New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Hey @Shakespeare,
Thanks for your question. SolusVM is an option along with a few others we have looked into for more variety for customers, in addition to having Virtualizor.
For any concerns you may have, I can say the @virtualizor team has been very proactive with us and helpful as well. The security protocols they had suggested to us were mostly in place already with a few minor varations. The final security updates are being implemented now to futher increase security.
Thank you
Hey @void ya we have had it disabled for a quite a while for security purposes. Thanks for the mention.
When you have more than one VPS, using the full Virtualizor panel is much more user friendly than using the embedded panel in WHMCS.
Does Virtualizor recommend disabling their own panel for security? If so, that doesn't inspire much confidence.
(not directed at you @DediRock
, many hosts disable the full panel so just talking in general)
And this is why we've decided to actually migrate the plans, but to redeploy the users: doing this process over 1000 times without any scripting to help nor guarantees it would work would be painfully hectic in terms of management, server-side and support-side. Things would be on fire.
But despite it all you're better than I was... Virtualizor » VirtFusion should be reasonably accessible if you were with Virtualizor because Virtualizor doesn't actually implement cloud-init and is reasonably close libvirt-wise to VirtFusion (with several differences, but still, close enough). So you can simply install cloud-init and qemu-guest-agent and it will take over the previous method and should work reasonably well, probably with some manual DHCPv6 magic for it to assume the IPv6 default formatting of VF.
SolusVM, which our company used, actually already implements cloud-init, has pre and post-boot specific scripting, used an entirely different networking stack, a different IPv6 IP formatting, a differently-built image on secondary disks... and to top it all off, several of the templates are different between each, with different default sizes, so matching would never be nearly equal. And yes, could be done with OpenVSwitch on VF's side, but when I already know that not only that doesn't allow IPv4 NAT (which was the main target all along), and also that it has ping issues past a certain scale at the dedicated server which customers would continue to suffer of... nope.
I wish you the best of luck on that process. It's not easy.
I work in... ugh I hate the term "cyber security"... I work in the information security industry as well and I have heard of them in reference to small random projects. They're certainly not industry leaders, but that doesn't mean they're garbage. Of course, passing an audit only means one thing: You passed an audit. It does not guarantee that your software is secure, especially considering how broad the term "audit" is. It can range from a quick look to make sure you're following industry standard best practice (just making sure you use prepared statements for example), or it could be an extremely deep, long-term, and in-depth analysis that requires complete familiarization with the entire codebase.
Honestly, that doesn't mean all that much without knowing what those bugs are in. At 20 years, that's less than one bug a week, which really is not that much unless you are almost exclusively auditing hardened, rock-solid code. If you're pentesting new panels all the time, surely you should be finding plenty of low-hanging fruit.
You're also mixing up exploits and security flaws, unless you're saying that you spend the time to develop an exploit for each vulnerability you find. Maybe that's common with web app exploitation. I don't know since all my experience is in binary exploitation / kernel exploitation where the gap between "oh that's a bad bug, it's almost certainly exploitable" and "here's a working exploit" can be vast. In fact, oftentimes you'll find that an exploit has to rely on multiple vulnerabilities to work.
That's not meant to throw any shade. Like I said, I don't do web app exploitation and I am not particularly familiar with the hosting industry. I just want to point out that "look how many bugs we've found" is as meaningless for assessing the quality of a pentester as "look how many lines of code I've written" is for a programmer.
The security of the hosting industry in general is kinda shitty. Just look at how many enable SMT and use truly ancient kernels on their nodes because rebooting an entire hypervisor is "scary". And I would be willing to be that not one single provider you have ever worked with has read any of the QEMU source code to determine how much attack surface they are adding by enabling certain features. While improving the quality of management frontends is commendable, there's a lot of remaining work to be done if the goal is to protect in any way against sophisticated threat actors.
I'm certainly no hosting expert, but I was contracted for a project where, among other things, I had to go and edit the KVM source code in the Linux kernel to reduce the attack surface area of a number of emulated instructions, as well as rip out some parts of QEMU that were unnecessary but could not be disabled. In that same project, I had to learn more about QEMU internals than I ever wanted to (https://airbus-seclab.github.io/qemu_blog/ was immensely helpful). That is the kind of thing, if done by people far smarter than me, that could truly improve the overall security of the hosting industry by a landslide.
Email From Virtualizor:
This notice provides the final technical details regarding an unauthorized access to our support ticket system, crucially, the factors contributing to the subsequent compromise of certain customer servers.
== The Source of the Exposure ==
A targeted session hijacking attack allowed an unauthorized party to access our support system. This was a sohphisticated targetted attack as we have 2FA for our ticket system, MFA enforced on our Google Workspace accounts and SMS based 2FA for our VPN / Tunnels. The attackers targeted plain-text root credentials that had been sent via email in tickets, rather than through our secure, encrypted submission forms. Approximately 1,500 Virtualizor support tickets were opened including this one. Not every ticket have passwords in them, but we are informing all these tickets.
== Why Certain Servers Were Impacted ==
Our forensic analysis of the impacted servers has identified two critical security lapses that allowed the stolen credentials to be used successfully:
Failure to Rotate Credentials: The servers that were compromised in this event were found to be using passwords that were, in some cases, over a year old and had not been changed i.e. these passwords were not rotated once the ticket was resolved and also a considerable time had passed.
Lack of Network Perimeter Security: Impacted nodes did not have a restricted firewall (IP Whitelisting) in place for the Virtualizor Admin Panel and SSH. This allowed the attackers to attempt logins from unauthorized external IP addresses.
== Immediate Mandatory Hardening ==
To prevent any further unauthorized access, we request these ticket Administrator(s) the following:
Rotate Passwords Immediately: If your root password is not rotated, and we have also emailed you about your ticket being accessed by an unauthorized agent, please change your root password.
Restrict Access: Implement firewall rules to allow access to the Virtualizor Admin Panel and SSH only from your known, trusted IP addresses.
Move to our new Support Access system: It uses SSH keys instead of passwords and the user created is also a temporary user which is deleted automatically after 7 days by default. This user account is also restricted to be accessed via specific tunnel IPs.
== Our Closing Actions ==
We have now redacted all ticket data containing sensitive credentials to ensure no further "historical" risk remains. We will also no longer take root details. Furthermore, our move to UEM-managed hardware and 3FA / MFA at multiple levels of access ensures our internal environment is resilient against the session-hijacking methods used in this attack.
We appreciate your cooperation in maintaining the Shared Responsibility required to keep the Virtualizor ecosystem secure. Also if you need any assistance, we are here to help.
We have been using Virtualizor since our company was founded. It's not great, but we haven't experienced any problems as long as the nodes are closed to external access.
In the past, support systems were hacked, but we never shared any node login credentials. Support was always provided via an Anydesk/Rustdesk connection through a temporary Windows VPS.
Screenshot as a proof: https://prnt.sc/8xpUeGJsDQAY
@raindog308 @angstrom @DP @FAT32 @jbiloh LEB Post or something?
A lot of companies will also pay a consultant to basically be very lenient and "help" them get certifications like ISO27001, etc.
Are they saying they got their Support System hacked again, or are they saying the affected providers didn't rotate keys/passwords after the breach last year?
@HOSTCAY you had 2FA enabled and bypassed, right?
And @HostSlick, you said you already had IP whitelists setup right?
Because if either one of those things is true then this can't have been caused by a cred leak...unless I'm misunderstanding something here? 🤔
I think its new attack as last time was live chat, now they say session hijacking attack on support system, probably support tickets?
I’m not entirely sure, but we had 2FA enabled and changed all passwords when everyone was notified at the time. Since then, we’ve rotated passwords two to three times, and the compromise occurred more than six months later.
The only explanation is that they either gained access via a hijacked session or accessed the account before we were notified and set up something malicious in advance, which would make sense. We did temporarily disable 2FA for about two days to grant support access, and in hindsight that likely explains it. Ultimately, it was our mistake, and we paid the price for it. This makes more sense as if there was something else more damage would have been done to more high-end providers.
It kind of boggles my mind a bit that they're even using passwords for this kind of access anyway.
If they want to allow a remote root shell to Virtualisor for support (although that itself seems crazy to me - why can't virtualisor just give you instructions on what to do instead?), then there's a very easy battle-tested method of doing so - using SSH keys.
You could simply remove that key when they no longer need access or if they say it might have been leaked.
I'm not buying it. If they really had 1500 tickets worth of root creds, there would be more providers getting hit and not those $10/year providers from last year let and black friday sales.
I read that as 1500 tickets in total, only some of which had creds, but they were replying to every ticket to make sure everyone took action. Which probably means it was just easier to script it to reply to everything.
Ah, I misread that. But this should be easy to verify then: Can the affected providers confirm that the specific nodes that were breached actually correspond to old support tickets where root creds were shared?
May I ask how did you recovered the data , I have storage vps and i don't want to lose my data , can you explain it step by step ?
That's what their statement mentions as their new method of access to client servers.
That they used to just request root passwords over a ticketing system, does raise serious questions about the security culture (or lack thereof) at Virtualizor, as it's completely insane this was ever deemed to be an acceptable way of gaining access.
It's good that they're changing it, but it frankly never should have been the case to begin with.
They can point to their clients not changing those passwords afterwards or not using IP based whitelisting for access to the underlying services, but they really need to do better and present a solid plan on improving their own security in general.
Fair play to you for taking ownership of this, but I don't think you'd be at fault if they lost your creds and you got hacked whilst the 2FA was disabled for their support to access the instance.
However, I still doubt it's a credential leak or re-exploitation through a backdoor. Both scenarios would use automation to ensure everyone gets hacked before the leak/backdoor is discovered, whereas this looks like a manual "hands on keyboard" scenario because the hacks have been happening hours/days apart.
I also have questions about their statement. For example, have they really "forensically examined" all the affected instances already?
A few days ago they were being evasive, but now they've done in-depth analysis of all of them, (including the ones we don't know about), and determined that none of them had rotated creds, implemented IP whitelisting or enabled 2FA... 🤔
But if they suspect a backdoor was installed from the last hack, did they also suggest that you completely wipe and reinstall the server from scratch too? If not, I'd suggest you do that anyway to be on the safe side.
Given the speed of their refutation duing the Ouiheberg situation, I think they have an automatic kneejerk response to deny all responsibility before they have access to any of the data necessary to make that assessment.
My take is that they're blaming old credentials because it's the easiest scapegoat and they've already suffered the reputational damage of all their internal communications being leaked, so it can be explained away as just fall out from that without having to publicly acknowledge they have other problems.
well at least we finally have an answer, what a shitshow.
i'd like to know if this relates to @ouiheberg issue a few months back though.
The providers involved who have blamed Virtualizor, but did not rotate their passwords after sending them to Virtualizor via tickets or email, right now:
It's an answer, but I'm really not sure if it's an honest answer or not...
For example, it's not clear whether they're saying they've been hacked again or whether they're blaming their previous hack/leak.
The start of their statement suggests they got hacked again, but then they say some credentials were a year old, which implies they were from the previous breach...although how they'd even know those year old creds were still valid and had no IP whitelisting or 2FA in place isn't clear.
We also know that at least some of the affected Virtualizor instances were taken offline immediately after they got hacked, so how they've "forensically" examined all those offline instances and determined that the affected hosts were at fault isn't clear either.
On the other hand, it could well be true because throwing your customers under a bus like that with no evidence would be a really shitty move!
I reinstalled the VPS, so the primary partition was wiped. I then ran testdisk on it. Exactly how to do it depends on how you set up the hard disk and I'm not very good at it myself (just enough to know when an LLM is going in the wrong direction)
I have already addressed this. We did change the passwords after initially providing access to Virtualizor. We then changed them again after being notified of a potential support-related password leak. Following that incident, the passwords were changed a further 2–3 times over the following months, before the compromise occurred.
This is why I stated in my earlier reply that the compromise happened during the period when access was provided specifically while access was granted and passwords were being rotated after the support issue was resolved. Something must have occurred during the two days they had access, which gave the attackers time to prepare or upload something that allowed persistent access.
As mentioned multiple times, after support access was removed, we changed the passwords again and re-enabled 2FA. This is also why we were confused about how access continued despite 2FA being enabled and multiple password changes.
The only logical conclusion is that something happened while access was granted, because enabling 2FA and repeatedly changing passwords afterward did not prevent continued access.
That’s why I am still all over the place and taking responsibility in terms of more security hardening should have been done. I’m not blaming Virtualizor anyone could have happened. I also admitted that only thing we did not do which would have helped is restrict IP access which makes sense. 2FA/password changes done but them accessing before the support leak was announced and uploading whatever to have continued access but this incident I’m still ask questioning myself how they bypassed 2FA and only logical reason I can think of is during the support breach they logged in and did some things so changing passwords and 2FA back on didn’t help whatsoever.
Anyway our Virtualizor master was reinstalled from scratch on a new server and location externally and security wise hardened on all levels and slaves resynced. We hired Mc2.dev for this as required importing couple sql tables subnets/vms ids to not break things as we don’t use any other vitualizor features.
This is quite magnanimous of you.
OR there is a 0-day exploit in Virtualizor.
It's good that you're stepping up and taking responsibility, but there's still a good chance there was actually nothing you could have done to prevent this. It certainly sounds like all the steps Virtualizor say should have prevented it were followed, and more, so at some point you have to consider the alternate hypothesis.
I cannot blame anyone else because once I start doing that, it raises even more questions. For example, why were no higher number other providers compromised? Why would the attackers upload fake, poorly written, AI generated ransomware? Their inexperience makes it hard to believe they were capable of writing proper ransomware code, let alone exploiting a system.
It does make some sense that it could be related to the support leak incident, but simply mentioning password changes did not help us, which is why I am questioning it. Perhaps we should have enabled IP restrictions as well, especially since we have not seen any admin login logs. We also may have needed to restrict firewall ports such as TCP 4084 and 4085. We assumed that enabling 2FA and changing passwords would be enough, but it seems that it was not (for our case).
Again, it's good that you have all these ideas about potential avenues that your system was compromised. At this stage you shouldn't be ruling any of them out.
If there is definitive, conclusive proof that they accessed the systems via old credentials, then fair enough. At the moment, that's just Virtualizor's best guess, and coincidentally the one that puts the least blame on them.
Any of the other things you mention could well be the root cause. But until they've all been investigated and ruled out, they're no more likely than an exploit in Virtualizor itself.
I ran testdisk and restored the partition, but I can't access or mount it , so I don't know how to recover the data