Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


BuyVM: All data lost on nodes 27, 41, 52, 58 - Page 4
New on LowEndTalk? Please Register and read our Community Rules.

BuyVM: All data lost on nodes 27, 41, 52, 58

1246

Comments

  • FranciscoFrancisco Member, Top Host, Host Rep

    @prometeus said: We have 16A+16A or 32A+32A to our racks powered independently, so almost all our servers are dual powered, but as I understand buyvm has a single line of power...

    Not quite.

    Our racks can only have 40A in them, period. Be it we A+B it or otherwise. Sure, we could have got more racks and A+B, but then we'd be filling racks less than we are now.

    Hard shutdowns on mdadm don't always lead to loss data, it's just a very high chance of a rebuild required. You can get around this by enabling bitmap's for the array but you take a ~10% performance loss. the bitmap means whenever it comes up and does have to sync it's going to only be the little bits it left (a journal in other words).

    @prometeus - We push the service hard and we have a lot of people on it, around 800. While that's a high number it isn't nearly as high as it should be for what it provides. For $1.00/m more users can have their SQL on there and usually get a bit of a smaller plan or just a much faster running setup. The service also provides free, managed, DNS hosting.

    The one way we'll just have to drive the idea home is to either put together a monthly email reminding people they should be taking backups as well as promoting the service. It's one thing if we were promoting it to make a buck and being a pest, but it's a 100% free service minus the SQL.

    Francisco

    Thanked by 1Ellimist
  • jarjar Member, Patron Provider

    @Francisco Have you ever given any thought to sleeping? ;)

  • @jarland said: @Francisco Have you ever given any thought to sleeping? ;)

    Pony's don't sleep.

    Thanked by 1Boltersdriveer
  • prometeusprometeus Member, Host Rep

    @Francisco said: Our racks can only have 40A in them, period. Be it we A+B it or otherwise. Sure, we could have got more racks and A+B, but then we'd be filling racks less than we are now.

    Francisco, don't get me wrong I'm not discussing (or criticizing) your work or your choice.
    ;-)

    We got to lose power even with A+B setup some years ago because the power generator was out of fuel when UPS exausted batteries :-(

  • BoltersdriveerBoltersdriveer Member, LIR

    @Daniel said: Pony's

    Ponies* :3

    Thanked by 1DeletedUser
  • pcanpcan Member

    Another useful redundancy is RAID6, with dual redundancy disks. This way two disks can go bad togheter and the service will still be available. Very, very useful when a quake strikes. As a side note, no SAS disks were damaged, only the SATA disks took the hit.

  • jarjar Member, Patron Provider
    edited May 2012
  • FranciscoFrancisco Member, Top Host, Host Rep

    @pcan said: Another useful redundancy is RAID6, with dual redundancy disks. This way two disks can go bad togheter and the service will still be available. Very, very useful when a quake strikes. As a side note, no SAS disks were damaged, only the SATA disks took the hit.

    We had some failed drives but they aren't what caused the corruption.

    The replacement drives rebuilt quickly helping speed up the FSCK's where possible.

    I'll likely sleep in a bit, Anthony is pushing backups for those that we could salvage/people asked for.

    Francisco

  • RophRoph Member

    My backup came through, pretty much everything in /home was fine for me. Grabbed my old configs and reinstalled, now it's like nothing happened. Thanks :)

  • SpiritSpirit Member

    @prometeus said: We provide free ftp storage on request with our business plans, NO ONE asked for the access to his ftp area out of 50-60 accounts...

    Doing this would be unresponsible :P

  • prometeusprometeus Member, Host Rep

    @Spirit said: Doing this would be unresponsible :P

    What do you suggest to do then? ;-)

  • manmamanma Member

    I ended up getting the data on one of my VPS restored (the one that I didn't even mention, since nothing important was on it), but the one with all of my work ended up with nothing ;_; Was the data completely lost? I replied to the ticket and I'm hoping for the best, but I'm still super worried. Also, how did you guys end up finding the data for my other VPS when I didn't even mention any identifiable filenames/directories on that one?

    Again, not complaining or anything. I'm still surprised that an unmanaged VPS provider would go to these lengths to recover data -I- was responsible for.

  • AldryicAldryic Member

    @manma - We were able to put many VMs back together based on some of the networking configs we found in lost+found, etc. There are still others that we are trying to get restored, hence why asking for identifying files/etc to look for. We just came back from a short break, so one of us will be getting to the ticket soon.

  • AldryicAldryic Member

    @Jack said: @Aldryic sounds like a fun weekend!

    I think this is the first weekend that Anthony and I have both stayed sober in quite some time... that's never a good sign XD

    Thanked by 1djvdorp
  • manmamanma Member

    @Aldryic said: I think this is the first weekend that Anthony and I have both stayed sober in quite some time... that's never a good sign XD

    If I ever meet you guys, I'll buy you enough booze to make up for this weekend 1000x over.

  • FRCoreyFRCorey Member

    Those node numbers are my new lotto numbers ;)

    Thanked by 1raindog308
  • kalamkalam Member
    edited May 2012

    Only lost a DNS server, so luckily it was no big deal for me to reinstall and resync.

  • FRCoreyFRCorey Member

    Sure, I'd buy them a nice backup server :)

    I mean we just had this conversation about sending backups to S3. Murphy's Law strikes again. Or was it Mr. Mayhem.

  • yomeroyomero Member

    I haven't good any answer to my ticket u_u

    Apparently I got the recovery folder, but is empty, I don't know what this means :P

  • subinsubin Member

    I think someone should create LowEndDown.
    It will record major down of LowEnd provider and the reason behind it.

    IIRC this is 3rd/4th time buyvm has power failure like this, then fsck. then bla bla bla.

    Note : i don't say that another host didn't get power failure. I only following news about buyvm and few other providers.

    Thanked by 1[Deleted User]
  • FranciscoFrancisco Member, Top Host, Host Rep
    edited May 2012

    @subin said: IIRC this is 3rd/4th time buyvm has power failure like this, then fsck. then bla bla bla.

    Well this wasn't a fault of the DC this time. This was simply a strip damaging equipment. Now while at HE? Yea, we had 4 full DC power outages with them when we were in FMT.

    Thankfully no gear was ever damaged but I know Linode had to pretty much scrap full nodes because of it.

    @yomero said: I haven't good any answer to my ticket u_u

    We'll see what we can find. If you were given a .backup folder from 27 then we should have been able to find something. It's possible Anthony's copy script didn't complete yet or whatever, we'll see what he reports back. I've not been able to salvage node27 outside of the initial 95 CT's I restored by using some clever tricks + scripts.

    Francisco

  • SpiritSpirit Member

    @prometeus said: We provide free ftp storage on request with our business plans, NO ONE asked for the access to his ftp area out of 50-60 accounts...

    @Spirit said: Doing this would be unresponsible :P

    @prometeus said: What do you suggest to do then? ;-)

    How can you ask me this? :p As first not to backup vps with same host as this vps is hosted already :P Just because your "business plan" users don't use your ftp backup storage it doesn't mean that they don't do backups at all. Maybe they do it properly - like hopefully most of us here? (ie. 3rd party backup storage on different locations) :P

    To stay on topic - it's good to see an effort of buyVM guys in data recovery process.

    Thanked by 1Francisco
  • subigosubigo Member

    In the future, you guys (buyvm) should just create a simple backup tool that allows users to download a full restorable backup of their container (didn't SolusVM have something like that at one point?). Node crashes, user has a restorable backup and your life is a million times easier. Node crashes and they didn't take a backup? Oh well. I have no sympathy for users who can't be bothered to backup their own data.

  • FranciscoFrancisco Member, Top Host, Host Rep

    @Spirit said: To stay on topic - it's good to see an effort of buyVM guys in data recovery process.

    Thank you kindly.

    Yomero it looks like you got a boiler plate reply from Anthony since he scripted the data copies. Your CT got splattered it seems. Do you have a conf file/etc that you'd really like back? If so just PM me the name/contents.

    Francisco

  • FranciscoFrancisco Member, Top Host, Host Rep

    @subigo said: In the future, you guys (buyvm) should just create a simple backup tool that allows users to download a full restorable backup of their container (didn't SolusVM have something like that at one point?).

    They had their quick backup stuff but I didn't like how they did it so I never bothered implementing anything like it.

    If a user has a fatter CT, then I'd hate to have them pushing 20GB+ sized tar's around like a dead weight.

    What i'll likely do is add an extra check box inside stallion where users can opt-out of weekly reminders to have backups with a friendly reminder that we do have the backup space available at no cost.

    Francisco

  • FranciscoFrancisco Member, Top Host, Host Rep

    @yomero - Found ya'.

    Your interfaces file was nuked so I had to go by your debian_version file. You're the only debian 5 install still on there so it was easy to spot that and the gameserver you ran.

    Anthony is packing up/etc your stuff.

    Hope what you're after is in there :)

    Francisco

  • yomeroyomero Member
    edited May 2012

    @Francisco
    Thanks man, now I replied to Anthony about the folders that you listed.

    Oh, and about config files, probably I can recreate that without issues (hopefully :P )

    EDIT: LOLOOOOOOOOOL! The only Debian 5! I suck :D Was waiting for some free time to reinstall/update it... shame shame.

  • raindog308raindog308 Administrator

    @Francisco said: What i'll likely do is add an extra check box inside stallion where users can opt-out of weekly reminders to have backups with a friendly reminder that we do have the backup space available at no cost.

    If memory serves, the backup space is 5GB. I've never used it because it's too small.

    Something that scaled withe the size of the VPS would be a nice future improvement.

    You could call them...oh I don't know...storage plans or something :-)

  • FranciscoFrancisco Member, Top Host, Host Rep

    @raindog308 said: If memory serves, the backup space is 5GB. I've never used it because it's too small.

    Something that scaled withe the size of the VPS would be a nice future improvement.

    You could call them...oh I don't know...storage plans or something :-)

    We're boosting everyone to 15GB with the option of justifying more at no additional cost :)

    If you need 100GB to backup your porn collection then it's a no. If you need to backup some pretty important files then we'll do our best by you.

    Francisco

    Thanked by 1klikli
Sign In or Register to comment.