New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Yeah, he messed up quoting @sidewinder.
To be fair, was it really a known issue until Chicago happened? They had one occurrence on one node of many in LA without a definitive cause. Two instances starts to indicate a pattern and as they said they're going to accelerate plans to migrate everybody to new hardware as a result.
As with everybody before me I'm not defending them going dark. That part is fairly inexcusable.
i've booted into recovery mode on CentOs - any hail mary's i can throw at this ghetto VPS to fix the corruption?
what parameters should i run with fsck?
thanks for all the emotional support.
They 100% should of nuked the entire node and made everyone start over. I don't think anyone's container came back?
is their cloud on a different infrastructure?
I'm interested as to if everyone is actually on the same node, what CPU is showing up running 'cat /proc/cpuinfo'?
https://ibb.co/b62Cf12
Seems to be the same as last December in LA. Below from December 2021.
@tetech Wow, I was unaware of that. Sounds like STOR2.LAX, STOR4.LAX, and whichever node in Chicago it was, all failed with the same problem...
this place is a sinking ship. can someone recommend something in the same price range with a little better hardware?
This is very hard to find. Their prices are extremely competitive. For storage you could try Servarica or VirMach instead, but the prices aren't quite as good.
AFAIK their new hardware is a lot better than the old hardware. Not sure about the RAID hardware though. It'd be useful for them to tell us the model numbers of their old RAID cards vs the RAID cards used in the newer servers.
Very solid alternative.
I wonder if there have been any incident that involved disc failure and successful data recovery? Or all those have been "raid card failure"? This more and more feels like raid0 configuration. A VM wont be able to tell if the host storage is raid6 or raid0. That info is probably not available for virtual environment.
sounds like my LAX instance all over again...
So... is it safe to assume this might be another involucration?
What's this, penis sect?
The symptoms don’t really match a raid 0 failure
In case it's useful to someone, I took @joelby's idea of using
dpkg -V
to verify packages using their checksums, and ran it across all packages returned byapt list --installed
:differences in config files are expected if you modified that file, but differences in binaries likely means they're corrupted.
You can then use
apt install --reinstall
to reinstall the packages that are corrupted.I wasn't able to fix the booting issue but I managed to uninstal the os via this tool https://sourceforge.net/p/boot-repair-cd/home/Home/
After removing the os It kept my old files under 'deleted_os' folder. After that I mounted debian 10 iso and installed new os, and I chose not to format the drive.
Now I can boot to my server and can access my old files and hopefully transfer them to a new server soon.
Great argument! Oh wait...
RAID failure? No
Invoulcration? Yes
Em..
Chicago Node Status Down
I think this is all of Chicago as my NVME services went down at the same time.
I’m still moving all my data from Chicago to LA to recover from this recent LA data loss.
I have storage services in Chicago, LA, and London. Both LA services and my 10TB Chicago service have had to be migrated because of this issue. Both the 10TB in LA and Chicago resulted in data loss. Wonder how long it be until London suffers the same fate.
Mine is down too.
Yes my chicago storage new VM is down too.
This node is their new provisioned vm after the major chicago storage down.
Hosthatch is no longer reliable....
Chicago node down also
My Chicago node is up, looks like it just rebooted. Uptime is showing as 11 minutes. Also @cablepick I don't think they offer storage in London, just LAX, Chicago, Amsterdam, and Stockholm. Maybe the DC lost power, or we were all on the same node? I'm on an e5 server there, if anyone was on Epyc then it would confirm a larger issue.
They do, my storage vps is in London.
Super interesting, it's not listed on their site at all! I'll have to see if I can still order one.
It was a BF promo deal though so maybe that's why.