New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Did you audit anytime recently web terminal function?
Haven't you seen the screenshot is Cloudcone's master panel? 25207 VPS. Except admin and hacker, who do you think can cap this screenshot?
We found a Virtualizor bug before, if a client VPS record exists in WHMCS virtualizor plugin, but vpsid=0 (Happens when migrate from other panel or VPS delete callback failed sometimes) , if the user enter Enduser panel, he can will enter the panel with Master role directly.
We've reported to virtualizor and they fixed it in next version. It is very similar to Cloudcone's accident, both of them is user can bypass admin permission in Master panel.
I think their master panel is highly inadequate in terms of administrator authentication.
You said Virtualizor and related modules were heavily audited in the thread about the OuiHeberg hack...yet there's been at least six more incidents since then, (which is a rate of more than one confirmed incident every two weeks), and that's just the ones we know about.
@virtualizor told me privately that OuiHeberg only sent them an AI-generated report, so presumably you're implying they put the code through an AI audit and sent that in their report rather than a PoC, but other providers have also complained about the evasive responses they've received from their support so I'm spotting a pattern here.
But what's most notable about all of this is it just isn't happening to providers using VirtFusion or other panels at anything like the same rate, even though their environments would be just as prone to mistakes and security misconfigurations.
I accept there's currently no hard evidence of the root cause and it's possible this has nothing to do with Virtualizor, although I find that highly improbable even though I don't deny that it is possible.
This is true, but to clarify it only affects people that have installed the 2Checkout gateway. If it's not installed it can't be exploited. Still, we recommend upgrading to 5.13.3.
So the web terminal works like this:
An admin user logs into Virtualizor and opens the Server Terminal function.
In the background the following file is created on the node in question:
/var/virtualizor/ttyd/htpasswd
The contents will look like:
root-2862728:$apr1$$7SECBXP5AYND904zor.Lt.
The username and password are randomized.
Authorization: Basic cm9vdC0yODYyNzI4OlZLMkdRODltM2ZyMA==
Without someone knowing that Authorization string, it's a full stop. You cannot connect to /ttyd/ws to open the WebSocket connection and drop into a root shell. It will just give you a 401 error.
Virtualizor is using Nginx as their backend and this is the relevant config:
Location /tty {
auth_basic "ttyd";
auth_basic_user_file "/var/virtualizor/ttyd/htpasswd";
}
There is no realistic scenario that would get around it and magically drop into the Terminal as root without knowing the username/password.
Thanks for explaining the ttyd authentication flow. While that shows ttyd itself is protected, it doesn’t fully rule out a possibility of a flaw in the panel-side logic that generates or exposes those credentials (for example in the token endpoint, session handling, or API layer).
What remains unclear is why providers found the same ransomware script only on master nodes. That pattern seems unlikely to result from unrelated SSH compromises and points more toward a control-plane level issue especially if some of them had 2FA/OTP enabled, not trying to point fingers just want to combine all info and find out the problem since I didnt get yet for whole day any answer from @virtualizor and they are usually answering me under 30 minutes...
Are there any ongoing investigations into the token or terminal generation logic to rule that out?
Strange that there isn't a CVE assigned for the bug you mention there, but then there only seems to have been one CVE ever been assigned for a Virtualizor bug, (in 2017): https://app.opencve.io/cve/?product=virtualizor&vendor=softaculous
I wonder, was there a notification sent to clients warning them about the issue or was it silently patched?
Interestingly, the bug you describe above sounds like it's the same bug class to the CVE that has been assigned for the product. Specifically:
"Virtualizor does not verify the API calls correctly, which allows remote authenticated users to gain access of other users VPS"
https://gist.github.com/sedrubal/a83fa22f1091025a5c1a14aabd711ad7
In fact, the same bug class would also explain the current issues we're discussing now. And it's common for bug hunters to check old CVEs and try the same bug class against other endpoints.
But having gone around these circles about the same subject multiple times, you'll have to excuse me for repeating myself when I say I have zero trust in the platform or the company behind it.
Do you know how many thousands of Virtualizor installations exist? By your logic, if a hundred WordPress websites have been compromised in the last month it surely must mean there is a zero day in the software.
I think the fact that there have been 6 more incidents is not really relevant given how small of a pool that is compared to how many people are using Virtualizor.
There are many variables and factors that can ultimately lead to a Virtualizor installation being compromised, and of course the software itself is one but without any sort of proof we must speculate that it is bad security practices, 3rd party software being compromised or human error until proven otherwise.
We have assisted Virtualizor with dozens of forensic audits of hosting partners that have been compromised and with the exception of the "Live Chat Incident" in every case it has been proven to be the result of bad security practices, using outdated software/modules or third-party related.
As I said in the thread the other month, nothing will ever be 100% secure and I am sure Virtualizor accepts that. From my own experience, Virtualizor have always handled security reports with the most seriousness and have always patched any security flaws we found in their software faster than most.
I'll state again: If you are a hosting provider and believe you have been hacked due to using Virtualizor, don't touch anything, leave the logs, send me a DM or reach out through our website and we'll do our best to help you figure out what happened. I don't care if you have 5 clients or 5,000, we will help you out totally free.
( deleted )
And I'll state for the first time: What's your relation with Virtualizor?
If a security vulnerability DOES exist, hypothetically, would that affect you and make you look bad? Do you gain anything from there not being a vulnerability?
Many thanks,
If you really want to claim that the providers here who are using Virtualizor are less competent than those using other control panels, that's a bold argument but you're welcome to try.
But the fact remains that those who are using Virtualizor are experiencing far more security incidents than those that are not.
We periodically go over Virtualizor for security flaws and assist in forensic audits when their hosting partners have been compromised.
If a security flaw does exist, it absolutely does NOT make us look bad because the reality is we are the first to make it clear to anyone who uses our services that nothing will ever be 100% secure and there have been many times where a third-party have found security flaws in products we currently audit and/or have previously audited.
While I cannot list our clients, we work with many hosting platforms and have been doing this for longer than I can remember. We strive to find 99% of security flaws in software, but there will always be that 1% that slip by and it does happen from time to time. It sucks, but it is what it is.
Truth or dare
Update: He also confirmed on telegram that he’s watching this thread. He messaged me again saying he saw my posted screenshot
I just don't see why its being done at such a small scale if this is the case...
Are they trying to make it so they last longer or are they just unable to automate it?
🤔
He just messaged me again few seconds ago that he saw my screenshot on LET! So he’s active here.
He looks like this
Hostslick VPS still down, soon to be 24 hour downtime.
My hostslick vps is still up and running and no ransomware lock yet.
Edit: looks like Virtualizor and vps is down now.
honestly no it cant
this shoulnd't be hard, should just be raw disk files, basically copy and replace
done it many times between VF's, can't see this being different, maybe it is but would be surprised
I had to look at the UI for a few seconds to realize how shitty the software is, so yeah honestly it's a solid argument, not sure I'd make it myself but I can see it
Network devices might show up under different names (and break you network configuration), which might need to be manually adjusted (by logging into the VPS via VNC).
True
They're not just reading this thread, they're engaging on the forum with another well aged alt account...
https://lowendtalk.com/discussion/comment/4725216/#Comment_4725216
If its 5 servers then it’s easy, lets say you have 5000 VPS to migrate. You have to manually create a VPS, replace the disk and hope everything works. Add user creation and its few weeks of work, unless you automate it in some way + integrating new VF servers to billing system.
If Virtfusion will provide migration tool, then we can certainly think about it.
Only real way for a little bigger provider is to let legacy plan expires (like Incognet did) and force leftovers to move on it’s own. But this requires good enough growth and liquidity to get more servers.
That’s not hard to automate. Virtfusion WHMCS module stores relation to VF user & server in the SQL. Creating accounts is easy with the API.
Some things I found about oldenvale.ru domain:
Discord admin usernames
intelligent002 (idle status)
pafia076
I want to share some context that might be relevant here.
We experienced something very similar last year on one of our test environments. An attacker gained access to our Virtualizor master panel and used the terminal feature to deploy crypto miners across all connected nodes. They even created additional VPS instances to run more miners. The method is exactly the same as what's being described now, the only difference is they're now demanding ransom instead of just mining.
A friend of mine at another hosting company (who I won't name) had the exact same issue around the same time.
Here's what we suspected back then: Both of us had recently opened support tickets with Virtualizor and provided root credentials for troubleshooting. At that time, passwords submitted through their ticketing system remained visible in plain text indefinitely. This means any employee, or anyone who compromised their ticketing system, could go back and read old tickets containing master panel credentials.
We never accused Virtualizor directly, and they never admitted anything. However, shortly after our incidents, they changed their ticketing system. Passwords are now automatically redacted from tickets after submission. Whether this was because they suspected an insider threat or simply wanted to protect themselves from liability, I can't say. But the timing was notable.
Since then, we've implemented:
- 2FA on all administrator accounts including root
- Disabled terminal access on production nodes
- IP restrictions on admin panel access
My questions to those affected:
The terminal feature isn't the vulnerability itself. It's just a powerful tool that becomes extremely dangerous when someone gains unauthorized access to the master panel. The real question is: how are attackers getting master panel access across multiple different providers?
Either there's a critical auth bypass vulnerability, or there's an ongoing credential leak somewhere.
@HostSlick @HOSTCAY @Cloudcone Did any of you open support tickets with Virtualizor before the attack? This could help establish a pattern.
P.S. Now I'm auditing hundreds of servers to make sure everything is clean on our end 😅