New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
This circus gets sillier by the day...
The CloudCone hackers replied to @justyy on Telegram, (taken from another thread), saying that they didn't hack the Virtualizor support system or they'd have used the credentials to hack everyone 🤣
https://lowendtalk.com/discussion/comment/4727400/#Comment_4727400
Their reply is so out of context that I honestly doubt they're lying, and only Gen-AI Script Kiddies would describe cookie stealing as "sophisticated" so I really believe they're being sincere.
Which means either @virtualizor has had their support system hacked by a different threat actor, (and they've emailed 1,500 customers claiming to have forensics identifying the hack), or they've lied about having proof of how these atatcks occurred and blamed them on their customer's incompetence instead!
Absolutely f*cking extraordinary!!!
I'm reading it as a second incident. It's on them to be more clear if that's not the situation.
I'm not sure how they wash off the stink of this incident and the previous incident especially with the tone they (and their security hero) set in their responses here. I guess as long as you win battlechat it's all good.
I don't sell vps so my opinion isn't super relevant but I was already avoiding hosts using virtualizor anyway so whatever.
any chance to recover vps with LUKS encryption set up?
@CloudHopper looks like there has been actually new breach:
https://www.virtualizor.com/blog/security-update-transparency-regarding-a-recent-support-ticket-incident/
@SecNinja any comments?
"SMS based 2FA for our VPN / Tunnels."
🤦♂️
"The attackers targeted plain-text root credentials that had been sent via email"
🤦♂️🤦♂️🤦♂️
This company is not serious.
Tbh, with rise of AI almost every company does some AI slop when shit hit the fan lol
I wonder if this thread will be hidden like the one related to @ColoCrossing being breached last year.
The PR company
The part that stands out
We have now redacted all ticket data containing sensitive credentials to ensure no further "historical" risk remains. We will also no longer take root details. Furthermore, our move to UEM-managed hardware and 3FA / MFA at multiple levels of access ensures our internal environment is resilient against the session-hijacking methods used in this attack.
Sensitive details should have been redacted when users were notified about the breach at that time not today. Explains why it allowed attackers months to go over it again and again hence they hit providers at different times. I’m pretty sure there’s either multiple different actors that access to the leak or a DB leak posted or sold on underground forums somewhere because our breach has had completely different ransomware and were more aggressive in our discussions with them, English not that good, many other characteristics whilst CloudCone breach sloppy ransomware, had good grammar etc. Despite this, there continues to be an emphasis on users not rotating passwords.
On our side, we changed our passwords the day after our support issue was resolved and re enabled 2FA. We also changed passwords 2-3 times within the months leading up to the breach. The only point I acknowledge is that we did not restrict access by IP. What remains unclear is how 2FA was bypassed, even after we changed passwords multiple times in the months before our breach.
The most reasonable conclusion is that during the breach, while our support issue was ongoing and 2FA was disabled for approximately two days, attackers gained access using valid credentials and placed something malicious. As a result, changing passwords and re enabling 2FA afterward did not fully resolve the issue.
Regardless of what exactly happened, it was a lesson learned and we have strengthened our security practices going forward.
That's damned scary, I don't envy your situation. And others affected of course. And others affected but don't know it yet. And.. Christ.
Dude, what is the difference between "remove that key" and "change that password"?
[Some] providers won't do it, I can also imagine that some providers will just by default add theirs keys to auth so they don't have to bother in future.
It will end the same way: our support hacked, they got access to our ssh private keys, you guys never rotated access, bye.
Well, the main difference is that leaking the public key doesn't give other people access to the system. Obviously, virtualizor could still be hacked and leak the private key, but in that case the situation is no worse than the password case.
Additionally, you can allow multiple keys at once, it's not all gated on one single password that everybody shares, and it's easy to log exactly which key is being used to connect with.
After you add their key and then remove it, you don't need to force a password to be recycled, because it was never revealed.
It's been a while since everything has happened. Did virtualizor patch all these vulnerabilities since in their panel aswell and the whmcs addon?
What vulnerabilities?
The hacker found a vulnerability in Virtualizor’s own support system, not in the panel or the WHMCS add-on the providers are using.
Only providers that had sent root passwords and had not rotated them later were affected.
See: https://www.virtualizor.com/blog/security-update-transparency-regarding-a-recent-support-ticket-incident/
Have you rotated your passwords?
It wasn't just password rotation either. I hear providers affected didn't whitelist their IPs for API and panel management access; they just had it open, and 2FA was not enabled.
It's bad what happened to Virtualizor's ticket system, but I do kind of feel bad for them. They are getting a lot of the blame for failures of people not securing their hosts correctly. Same could happen with Virtfusion if they got your password and didnt secure it correctly.
Reading the walls of texts from few last pages, these people really just yolo didn't think shit through before actual shit happens. Virtualizor with their requesting cleartext pw in support ticket, provider with their non rotating pw, non 2fa, no ip restriction to master server etc. Everyone played the pointing fingers, blaming game but at the same time also revealing own shittiness. A shitshow is really an understatement.
Yeah, I particularly think Virtualizor's response is laughable, saying their systems got compromised and everything was leaked, and somehow it's all the providers fault for having previously shared access credentials when requested by support.
There is so much wrong with that - getting hacked in the first place, only supporting password access not keys, requesting password access for root from customers, etc. All of that needs to change, regardless of the security practices of their customers.
Agreed, the Ticketgate scandal was a shambles on virtualizors part, but I do think it's unfair to say it happened because there was an exploit in their software.
At the end of the day, people are paying you to store their data. Why would you give your normal password out to anyone and leave everything wide open? It takes 2 minutes to make a temp password and not long to lock all nodes down.
It's a failure on both parts and as a mere customer I won't be dealing with either set of fellas.
edit: harsher than intended, I guess.
According to Virtualizor they leaked passwords from their support system in a "sophisticated attack", but then the hacks all stopped as if by magic.
But hackers usualy perform "sophisticated" attacks to breach a comparatively high-value target's emails and VPN, (bypassing their 2FA), and then only use that access to steal outdated passwords to perform low-value attacks on the target's clients.
It's also a complete coincidence that the CloudCone hackers announced they were taking a break to review their tactics at the same time as Virtualizor made their nonsense statement.
Which is what I hate about this sorry saga. We use Virtualizor, and it's put loads of people off it when it wasn't an exploit, so we will never get your custom cause some admin handed his password out.
Love to go with VirtFusion to support a fellow Englishman, but moving over 400 VPS when there is no software exploit will be to much of a headache.
I don't envy your position but as a drama-consumer I'm avidly awaiting the next screw-up we hear about on their part. If nothing else this was a wake up call for users of their software to tighten the bolts.
I'd also recommend reading Virtualizor's claim again...
They say security is a shared responsibility, yet they say nothing about what they've done to secure the email accounts and VPN that were breached.
Either they're telling the truth and YOLO'd their own security deficiencies, or they've lied and thrown their customers under the bus...and I'm not sure which is worse, but I know for a fact I would never trust their shit again, regardless!
Tagging administrators: @trewq / @jbiloh / @FAT32
Please introduce in LowEndTalk selling rules a recommendation for providers to not use Virtualizor. I understand the fact that community lleadership needs money from provider tag, but we also must think about the buyers. If Virtualizor is suspect of having issues, we should not necessarily block it or deny it from community, but at least an abstract recommendation towards providers should be made within our selling rules to not focus cloud infrastructures on it. If a provider insists on choosing Virtualizor, let it be so, but this community needs to care about its members using a small recommendation at least.
Actually, while I think Virtualizor needs to change a lot of their practices / attitude towards hack reports, this feels like it might be going too far.
Virtualizor had issues, and may still have, in relation to their own internal communication systems. The Tawk live chat, their helpdesk. It still seems that the major security flaw was that they (Virtualizor) had bad internal actors who copied details and used them to gain access on systems that were not updated after giving access to support staff.
I don't think anyone who was hacked answered the question, "The hypervisor(s) that were impacted, did you give Virtualizor support root access to these systems in the past?" and "If so, did you rotate credentials afterwards?" It seems odd that of the providers hit, only one or two or three hypervisors were impacted instead of all of them. That makes me think it was just whatever ones that may have previously had credentials shared in a ticket or live chat.
At least one of the providers admitted to having not using any sort of 2FA or admin IP restrictions.
It's still not perfectly clear if these new cases are really virtualizor's fault or just someone using the old details that never got updated and were leaked in the past. From both sides, everyone wants to save face and say it's not really their fault, which might be partially true for both sides as well.
So would it be okay to have providers that leave their management portal open for the world to see and have a crack at? Would it be okay for them to give passwords out to third parties and never rotate them, and play loose with clients' data, as long as they don't do it with Virtualizor?
Their ticket breach was a big incident, and they could do a lot better and need to, but I can't see how you can put 100% of the blame on them for it.
I feel really bad for the providers, I really do. What a horrible situation to be in, but everyone played their part to contribute to this incident to of happened.
Virtualizor claimed to have identified a new hack of their Support System in the email they sent to their customers. In the same email they also claimed to have "forensically analyzed" the affected customer's instances and proved that credentials hadn't rotated...even though some of those instances were already offline by the time Virtualizor's team supposedly "analyzed" them. 🤔
However, what's most notable for me is that nearly all the affected hosts have either ditched, (or are in the process of ditching), the platform. Which makes sense because in the best case scenario, Virtualizor has been hacked and leaked their customer credentials twice in 12 months...or worse they've lied about leaking credentials and blamed their customers to protect their own image.
I'm also certain nothing was hacked, but their lowly paid support agents try to earn extra bucks. But both are possible.
Both of those scenarios are terrible for their customers though...