New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
If one of your nodes got hacked and encrypted, and they all have the same security setup, it's only a matter of time until it happens again...so you'll probably want to engage Incident Response to identify the root cause and implement a fix across all your nodes before another one gets popped
FDE is only a minor roadblock to an attacker at this level
Confidential computing (plus FDE) is the answer, but only major clouds offer it, hasn’t trickled down to low end hosts yet
Yelp. Is my data safe? We are all thinking it
Maybe one should think how can I have data backups so any issues with thr provider do not affect me.
Oh that part should be less concerning. I am referring to general privacy.
Why wouldn't it be safe?
I think onus lies on customer to implement measures to safeguard data that they deem critical, isn't that right?
Yes and no. Any level of encryption can be hacked over time. Storage is cheap.
You need to be able to trust your host though.
If your server is compromised and all your data is copied, that is on you.
If your host is compromised and your data is compromised, that is on your host.
💯 agree. Which is what I was driving to say. At the end of the day it's customers responsibility, unless the provider claims to be providing some hack/ ransomware proof service. Even in that case it's better to have some safety net.
Over time. If one is paranoid to such a level, they should be smart enough to figure out anything traversing over the wire can and will be eventually hacked. So one will ensure they don't use any services.
I have friends that do not trust the cloud for this exact reason. They create a small container encrypt and re encrypt with some tool and store their data locally on a system where they disable network connectivity. Seems too far for me, but normal to them. Everyone had a choice at the end.
This is the way. Except on the cloud.
Huh?
You're saying it's the customers responsibility to keep the hypervisor safe, unless the host provides a 'ransomware proof service' ?
Not sure that's a thing, and a pretty crappy take.
Sure, have backups, but GC has some serious investigating and explaining to do if I read things properly. A hypervisor apparently got ransomwared through an IPMI interface.
It's not customer's responsibility to secure the hypervisor. We expect failures and backups to be needed, but not because someone didn't set an IPMI password, or something alike.
Vested interest here - I was on that node too, but use luks, keeping an eye on this thread.
Never said customer should keep hypervisor safe.. heck, how would you in a shared environment like this? Sure it may have been a miss from the provider for not keeping safe but that also does not make the customer not protect data they care about.
Shit happens at all levels. Trying to ask is the data safe goes to imply that customer did not probably have good encryption set or did not expect to be hit with ransomware at hostnode level (if that even happened).
Sorry all users on that node are in trouble, but frankly common novice folks need to be better educated and aware that things could really go wrong, even if it is on the cloud...
There's some cheerleading going on for GC and I wonder if some incentives are involved in the process as measure for damage repair.
The whole issue was lack of transparency. If one ignores the obvious errors of translation one can see that one user had backups but he was surprised at the lack of disclosure from the host.
It's a simple expectations. If data is breached, or compromise.... Then be upfront about it to your customers. Because that sets up course for future events also. No host can provide 100% secured server because that's impossible. But at least let the customer know what actually happened.
Moreover, one needs to check for 5 minutes max to get that it was a security loophole.
GC has employed such people who has no idea about how to run a server and is depended upon only two senior technicians. So whatever data you are saving in the server, it's at the mercy of availability of these two senior engineers. If God forbid, these two engineers aren't available at the same time and something happens, these other incompetent employees can do nothing to help you.
Will one be happy with this set up?
I surely would not.
So, does every host you use have at least 10 employees? What are you doing at LET, where most companies don't even have two senior technicians, anyway?
It's about emmploying someone who have atleast basic ideas of how to run what. Just 2 minutes of checking and you can see what's wrong here but what the employees did?
That too could have been ignored but hiding the actual reason even after arrival of the senior technician? That's very ill intention from the host provider.
Transparency is the key 🔑 no doubt. Did something bad happen at the host level, we should give some time to provider to come back after their root cause investigation is complete.
I think it's a stretch to say the technicians are not knowledgeable. Not everyone coming in the industry will have experience to solve/ detect all problems.
Are you installing Proxmox on our VPS? Just make sure to have the network settings configured correctly. We have not seen any issues with network in Ho Chi Minh City recently.
Guys we have been in business for more than 11 years and we know how to do it, if it's that simple to break into our system then we cannot exist in the hosting business for that long. We are narrowing down the possible root cause of that node, we have also contacted Asus to see if they can reproduce the problem with the BMC firmware version which was released in mid-2023, there is only one newer version released in September 2023.
You haven't answered the simple question though from your own customer.
No one is saying that your system has to be perfect.
But why the lack of transparency and proper disclosure?
Can you answer this?
Check my previous replies on the first page please.
You didn't address it in the replies in the first page. You stated that there's an investigation why it has happened and will ensure that it won't happen again and etc. etc.
But you haven't mentioned the reason for non disclosure (until the customer himself found it).
If you have addressed it, kindly put the quoted words because I haven't seen it.
In my previous replies I already said that it was our fault and we will provide the disclosure notice once we finish our investigation.
It did.. The host's rep confirmed it on page 1 or 2 of this thread...
My issue with this situation was the host being dishonest with the customer on what happened (telling them it was a drive or RAID failure) when it clearly wasn't.
My first impression is that the staff/technician/whoever working on this first hand was freaked out and made the wrong move of cheating the customer and thought he could get away with it. Anyway @NDTN has since accepted it was a mistake/fault on their side and hopefully he will train his staff to be honest next time, at least discuss with your seniors first before sending out a "lie" (as it's irreversible). I'll certainly move on from this and keep a close eye patiently on the disclosure of the root cause later.
I had a 'those poor guys' feeling because I got the same mail, just didn't experience ransomware because luks was installed.
Tried logging into Solus but the VMs were gone by then.
Appreciated their response at first, but I'm in the same boat of... 24 hours later and it was still called a RAID card failure - seems off. Anyone could have ls -al on the host node and seen the readme they left. Or the encrypted files from the screenshots earlier.
Anyway, not jumping to conclusions either way, just hoping for a thorough investigation, explanation and changes made in the future. Usually if IPMI is internal and on it's own secure VLAN, firmware shouldn't matter. Which brought up that Fortinet active zero-day that people are using to make lateral movements on networks.
Only thing that really worried me after reading this was the quote that all of their systems security is setup the same way.
Still wish there was some more communication and less obfuscation, but that's my opinion.
I don't see it any different from a provider saying there was a extended power outage to the entire datacentre outside of their control and that is why x was down for y hours.
User regains access to x and uptime shows z days, that the system lost power.. They had screwed up their own router and took themselves offline..
I take issue with dishonest people..
Reminds me of their UK node outage earlier this year where people where specifically asking for more communication (yes, even something like "our initial plan didn't work out as we have run into additional issues, but we are still working on it and now expect it to be resolved by ..."). But looks like they really haven't learned anything from it.
What does Asus firmware have to do with the fact that you got your node encrypted by a ransomware gang via exposed/unsecured IPMI?