New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
@HostDZire any way of knowing if my VM volume was also encrypted ?
Was the data merely encrypted, or has it already been taken by hackers? I need to assess whether my backup data is also no longer secure.
.
If your server is down then it was encrypted. Unaffected nodes are still running
not fun, good luck with it everybody
I hope we will got some extension to our servers, post mortem ofc.
https://lowendtalk.com/discussion/209803/the-learning-by-thread
Mine 2 VPS at India Location also got affected. Waiting for the response. If there is a estimated time of rebuild by @HostDZire then it would be nice.
lost 1 month of my wife's business data. Gonna be a bad evening
Maybe buy flowers/gift to share bad news ?
nah. wont work. a holiday minimum is required.
Bummer that this happened. It looks like my AMS VPS might be down, but I can't confirm from the machine I'm using right now. Good news is that I haven't done anything with that VM yet, so no loss of data or time for me. Good luck to everyone that did lose something important and to HD who I'm sure is staring down some long nights at the office.
Hope our client information is safe. Luckily I did not have any important data on the server!
reminder to always have a disaster recovery plan in place
my servers at AMS are currently alive tho
I didn't had any important data, but there was a service running , serving our customers. Had to spawn a vm in OVH to redirect user to that. But its not fun seriously for the people having data in it.
Many seem to have lost quite a bit of data. Always reminds me that doing a incremental backup every day and a full backup every week is saving my ass.
Any other LET hosts use VMware ?
LOL. That's why always do multiple backup
@VeloxMedia
Phew, that was a close one!
I was about to move my precious vintage porn stash to HostDzire yesterday when I got delayed by unexpected events.
When I tried to resume I could not do it, it angered me at the time but now I'm glad it happened.
I only keep one copy as I don't believe in backups, I could have lost everything!!
Hello everyone,
If I am not responding to messages individually, it means I am fully focused on restoring services. This is a major incident, and at this stage the only viable solution is to reinstall all affected servers and rebuild the infrastructure from scratch.
This process will take time, but our priority is to provision fresh VPS instances and get everyone back online as quickly as possible.
At this point, we are unfortunately 100% certain that all data has been lost. Since we did not maintain backups for these servers, there is no backup available for restoration.
This incident has made it clear that our backup strategy needs to change. Going forward, we will be implementing multiple improvements to our infrastructure and disaster recovery processes to significantly reduce the impact of incidents like this in the future.
It's also important to point out that approximately 80% of the affected servers were using RAID. However, RAID did not prevent data loss because the failure was not caused by a disk failure. RAID protects against hardware disk failures, but it does not protect against ransomware, software corruption, accidental deletion, hypervisor compromise, or other infrastructure-level incidents.
Looking back, even weekly backups would have significantly reduced the impact of this incident. However, due to the extremely low pricing of these services and the associated infrastructure costs, maintaining provider-side backups was not financially feasible. This is why we have always strongly recommended that customers maintain their own backups.
The most reliable backup strategy is to keep an independent backup outside of the hosting provider's infrastructure. Even after we introduce weekly backups or other backup solutions in the future, no single backup system can ever be considered 100% safe. The highest level of protection is achieved when customers also maintain their own backups or store them in a completely separate location, such as another datacenter or an independent cloud storage provider.
Please also understand that this is the first time something of this magnitude has happened in our 13-year history. Throughout those 13 years, we have successfully protected our infrastructure against countless threats and incidents. Unfortunately, we became aware of this VMware vulnerability a few days after it was first publicly reported on July 29. By the time the attack was identified, the damage had already been done. While that does not change the outcome, we are carefully reviewing exactly what happened and will be strengthening our security monitoring, vulnerability management, and incident response procedures to reduce the risk of similar incidents in the future.
Over the past few weeks alone, we have seen serious security incidents affecting multiple virtualization platforms and operating systems, including KVM, Linux, and now VMware. As AI continues to accelerate vulnerability research, we expect new security issues to be discovered more frequently across the industry. This makes proactive patch management, independent backups, and rapid response more important than ever.
We sincerely appreciate your patience, understanding, and continued support during this difficult time. We will continue posting updates as progress is made.
Absolutely. People should have make backups. This wasn't the first time this happened to LET host. But people never learn
or maybe the provider can have some person to properly maintain their infra like not exposing everything to internet for example is a good start.
Interesting business decision: VMware but no backups. Doesn't VMware have full set of products designed around main product for centralized storage, DR, backups.. and only proper way of using VMware is to use full set of products ?
If you want my advice (which you should not): ZFS that shit. Even Proxmox is possible solution. ZFS with 7-days offsite snapshots would at least save VM data. Do it "pull", not "push" way also. That way when someone get's in your server they do not even know about backup server existence.
Most of it comes down to cost, not application features.
A customer cannot buy a low value plan but expect all premium features to be included.
If you require certain guarantees, you need to pay for that at the end of the day.
Like IP whitelist? Sure, but it's not foolproof either
Right, you do not buy Ferrari that takes 20L per 100KM if you can not fill it regularly and you only use it to take pictures. You buy 1.2L Polo and you RPM that thing to 5000.
In my country we have saying (which roughly translates to): Cover yourself as long as your quilt is long
I have seen a lot of people buying kinda ferrari coz they want to look cool. Stupid decision but well people do it in some countries.
I thought HostDZire offered snapshots.
Feeling bad now
I don't think I can continue to trust after this..