New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Little Creek Hosting Down
LittleCreek
Member, Patron Provider
in Outages
We have had a major malfunction. Server306 went down. Littlecreekhosting.com was hosted on that machine. I am running some recovery tools on it tryin to recover data but I don't yet know what the outcome will be. It may be another couple of hours before I know.
All the other servers and vps are working though.
Comments
Do you not have your website backed up?
what are backups?
What happened to it? Hardware failure?
And why didn't you keep backups?
@LittleCreek 3-2-1 is the minimum
Filesystem corruption on the VPS storage volume — not a hardware failure, all 5 NVMe drives are healthy. We're currently running e2fsck on the volume and making progress.
As for backups — we do have them. I have a backup of the littlecreekhosting.com VPS from June 21, and roughly 100 other VMs also have backups from the past week. However, I'm not ready to restore from backup yet because not every customer has one, and I don't want to wipe a potentially recoverable volume and lose data for those who don't.
Right now we're letting the filesystem repair run its course. Once we know what's intact, we'll have a clearer picture of who needs to be restored from backup versus who can be brought back directly from the repaired volume. I'll post an update as soon as we have more to share — appreciate your patience.
Incidents frequently occur while offline, yet there is no compensation whatsoever
the reasons given are meaningless—what matters is the incident itself
So you set up your main website on the server where the clients are?🤣 I like the thought process - you go down, I go down.
What I can tell you is that all 189 VMs have been successfully recovered with no data loss, and services are being restored now. The recovery took approximately 4 hours from identification to resolution.
Out of curiosity, what caused the corruption?
A kernel panic on the host node at 20:05 on June 28th. The panic caused an immediate ungraceful shutdown which left the ext4 filesystem on the volume in an inconsistent state. All NVMe drives showed zero hardware errors — it was purely a software/kernel issue. The root cause of the kernel panic itself is unknown as the crash destroyed the evidence before it could be logged.
The panic info will actually be logged via pstore and is exposed in
/sys/fs/pstoreafter a reboot, which systemd usually copies to a persistent directory. Check/var/lib/systemd/pstoreto see if it was saved there.Glad everything could be recovered. I always say
e2fsckis magic. It's recovered some incredibly damaged filesystems for me before!Wow 189 VMs on a single node...
Yes, Limited stock per node.
EPYC 7402 has 24 cores.
Assuming each of 189 VMs has 1 VCPU, every physical core has 7.875 VCPUs.
Host machine has 1 Gbps port.
On average, each VM has 5291 Kbps port speed.
Each VM has a gross income of $12/year if they all sold as limited deal.
The host has a gross income of $189/month.
When you don't all the facts conclusions can be way off.
Only assuming each VM is trying to saturate the port.
Mine server is not yet up. Should I worry ?
When you don't all the English, off way be can.
Sorry I have been up all night.
I am still working on some. Please submit a ticket.
Login and do manual restart. It worked for me.
your domain is offline again
i just pray for the best outcome
almost 15+ hours of downtime on my server now
Littlecreekhosting.com is working for the last 14 hours. If your server is still down then please submit a ticket or email [email protected]
Unless the link is going up and down, the average VM port speed will always be 1 (or 10) gigabit.
Have you tried turning it off and on?
I hope that things turn out smoothly for you in the end and hopefully you can get some sleep after this incident. Take care and hoping for best for both you and your customers
I think the issue is that folks are trying to ssh and if denied access, they are thinking the server is down. Actually post resolution of the downtime incident, the server is in an "inactive" state and needs to be turned on from the console and then ssh is working.