New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
BuyVM Catastrophic Data Failure - All data lost on a node!
BuyVM suffered a catastrophic failure on a node where multiple SSD controllers failed.
Anyone else effected by this? I am! Still waiting for the new server to be provisioned. Good thing I had a semi-recent backup.
BuyVM will also be restoring their backup, but it's a few days old and unsure if it contains everything.
Comments
Yeah, I have backups so nothing lost apart from the downtime.
Well is the best always to keep backups from your side also to keep a backup of the backup too if is possible.
Is this shared?
Massive hamster strike.
'Catastrophic'? That's a bit fucking dramatic.
Their words, not mine.
"I lost all my data and had no backup, I'm losing millions per minute"
Posts coming in 1..2..3...
Odd: so far, no announcement or tweet; network status is also good.
A bit like your reply.
Catastrophic is usual... "Colossal and unfathomable data failure" - this would be a bit better.
"Help guis I am loosing a lot money my client angry at me. I make only annual backup"
@Francisco, help us!
Well... new word for me.
I do not know how it's possible. It looks like a total amateurishness....
Anyone want to post the email?
Join their discord, that's got a lot more info.
I keep a weekly backup on the server... Is that lost as well?
How the hell do multiple controllers fail at he same time
Was is not raid10? How.the heck always fail 2 drives at once
Discord, I see, but one would think that Discord shouldn't be the main channel of communication about such an event.
Wanna be more dramatic?
The end is NIIIIIGH!!!
On a more serious note, is it for real? Could be a badly miss-timed April fool's joke.
Maybe they emailed customers. I'm not a customer of buyvm tho!
Yeah, I'm also affected, millions just being wasted every second...
Here's the email for those who asked for it:
Deadpool
The ponies strike again. Good luck, @Francisco! We're all pulling for a speedy and smooth recovery.
After some looking through our invoicing the 4 drives that made this array were part of the same order batch that the lv-shared03 drives were.
The same issue happened where the node crashed, rebooted, and the drives couldn't be seen by the BIOS.
We tried a half dozen different nodes and 4 different HBA's in hopes of something.
We do have recent backups so we're already restoring that. We're probably another 8 or so hours out before everything is back in action.
We took it as a good time to swap to some nice Intel NVME's as well as upgrade the CPU's in the box.
Not fun at all. It may very well spoil my experience with Samsung
Francisco
It's the proper word. The array was entirely lost and we lost active data on it.
We have backups, but in most cases it's a few days old.
Francisco
Bad batch of SSDs (maybe)
"Samsung" actually means three stars.
What do you expect from 3-star products?
Wasn't just 2 drives, was all 4.
All of them are the same batch from the same seller (amazon). I've not had a chance to dig through errata's to see if there's known issues with any batches or the likes, but so far its only been 850 1TB's that have done it.
We got other nodes on 840 1TB's and they never skip a beat.
Francisco