New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
sorry we are not accepting this offer
And yes, we were listening to sigma girl while doing this;)
p.s. colocrossing, cloudcone, dedirock hackers
we are back with new hits, but much bigger this time
Sorry guys, I tried my best here.
According to some insider these guys already had @emgh 's nudes.
Also @jackgo before you get banned, what's your favorite anime?
How sure are you about this? By default, Virtualizor DOES store personal information from the costumer, and it has been accessed/used in previous breaches of other providers.
We don't watch anime, and we have a lot accounts to switch on
So, we will back soon after ban
you see how we are good to people???
why would you assume you'd watch it together is this some queer hacking group?
I agree though I don't watch anime either I find it a bit weird
@Murv likes it though
Can I join you guys and we all watch some anime?
By any chance, do you have an arabic version with the proper dressing? Maybe some hackers would like that more when receiving the ransoms.
well, we are the hackers;)
OuiHeberg sent a report to Virtualizor, (which both sides confirmed was received), and then they removed the product from their environment...which indicates their confidence in the vector.
https://lowendtalk.com/discussion/comment/4582812/#Comment_4582812
ColoCrossing was hacked and leaked data that generally gets stored in Virtualizor, but there was no indication of ransomware. VMs were deleted by the hacker, but that was because they controlled the environment. We don't know the root cause because they styled it out.
https://lowendtalk.com/discussion/205968/colocrossing-database-breach
GreenCloud also got hit with ransomware. Again, they completely styled it out so there was no analysis of the attack vector, although the payload was archaic and linked to IPMI-targeting campaigns from many years previously, but was almost certainly a bug in outdated firmware:
https://lowendtalk.com/discussion/198766/my-greencloud-vps-got-ransomwared-the-entire-mothership/
The GreenCloud ransomware incident was almost certainly a firmware exploit because HostCram experienced a very similar issue with the same archaic ransomware strain and was very open about:
https://lowendtalk.com/discussion/199608/junglesec-ransomware-9-linux-vms-are-affected-backup-your-data-ryzen-7000
@jackgo can you hack @emgh for hurting my feelings?
@jackgo hack @Murv for liking anime
If Virtualizor is indeed vulnerable, maybe we need to completely ban Virtualizor usage on LowEndTalk. Something like this can be introduced in selling rules so providers know not to use it.
Two hosts have plausibly identified an issue with Virtualizor, one of whom claimed to have reproduced the hack and the other has forensics pointing to that being the vector.
The fact that Virtuaizor denied they were at fault before they had any of the information also raises a LOT of red flags...as does the fact that they privately sent me a long denial, (including an attempt to recruit me to audit their platform), raises even more.
As I've said many times, people will believe what they want but all the evidence points to an issue in that platform and a developer who cares more about PR than fixing that issue 🤷♀️
I feel so betrayed rn.
From now on I'll shitpost with my new homie @jackgo instead of you.
Normally I wouldn’t reply to something like this, but I wanted to be transparent after seeing your comment and also help CloudCone in some way as it may or may not be their fault hence why we ourselves haven’t went public with this until now.
We were also victims of a similar incident around two months ago. Our main master node had Google Authenticator 2FA enabled, along with SSH terminal 2FA via PAM. Despite this, the attacker somehow managed to gain access to the master and use the server terminal (so we thought). To this day, we still don’t know how the 2FA was bypassed.
The attacker contacted us on Telegram demanding payment. During the incident, we lost around 300 VMs, as they continued to threaten further deletions. Eventually, we paid an undisclosed amount significantly less than what they initially demanded. They promised to explain how access was gained, but stopped responding once the payment was sent.
Since then, we’ve moved the master node to a different location and implemented additional security hardening. We haven’t experienced any further issues. We currently manage around 2,300 VMs, but we’re still questioning how access was possible with multiple layers of 2FA in place.
I strongly suspect a vulnerability in a Blesta or WHMCS module. Our Virtualizor installation is white-labelled and has no direct links or connections to our main website, yet the attacker still knew it was Hostcay’s node. This makes me believe the entry point may have been through a billing application vulnerability.
They were able to execute VM deletions and upload ransomware, which I initially assumed was done via the terminal but there’s still no clear explanation as to how 2FA could have been bypassed.
We chose to restore the data, apply further security measures, and move on. I’m sharing this in case it helps others. I won’t be commenting further on the matter.
I am backing up my data to a Host-C node, and I trust this man.
@ouiheberg claimed to have reproduced the hack and that the vector was the "Virtualizor-WHMCS addon".
https://lowendtalk.com/discussion/comment/4582812/#Comment_4582812
I don't have enough knowledge of the platforms, but presumably a billing panel hack wouldn't allow control of VMs unless it is somehow linked to the control panel?
Perhaps the device holding access to the 2FA is compromised?
Good to see you accept it and glad we can now find the right answer. Agreed. I searched on DDG and DDG AI got confused saying its open source but kimi was able to find that it wasn't open source and I then got even more confused so went to their github to literally see 3 small things or something.
Definitely a bit shady in my opinion tho. There's no denying about it. Although I agree with you that if virtualizor was truly (hacked?), we would see lots of providers down or hacked so I don't really know. Only more time will tell what really happened imo
Bit off topic, but my only question would be why, if the restoration of the VMs was possible to begin with, would you go ahead and pay the ransom? Am I missing something?
Either way, good to know.
The Google Authenticator app is only installed on an iOS device that is tied to a Gmail account. That Gmail account has multiple authenticator apps added to it and is protected with Google 2FA using a phone number. I am the only person with access to this account, and if anyone else had accessed it, I would have been notified. I am not making excuses, but there is no realistic way someone could have accessed the Gmail account without triggering sms notifications etc.
Even if they somehow did, that still does not explain how they obtained the Virtualizor login password in order to enter the authenticator code. Any such access attempt would have appeared in the login logs. More importantly, it does not explain how a Virtualizor instance with no hostname, links, VMs, emails, or identifying data could be associated with Hostcay at all.
All of our nodes use random domains as hostnames, and the same applies to subdomains in the settings. This is why I still strongly believe the issue originated from a billing module vulnerability. There are no access logs within Virtualizor itself showing unauthorized login, yet from the billing side it is still possible to execute actions such as suspend, delete, and other functions.
I am not claiming this is definitively the cause. As I have said, to this day I still do not know exactly how the access was gained.
If your dumb and use an easy hackable password, then you not really dream of sonething big, but fast cash.
That is exactly what we tried to do at first. We ignored them and focused on figuring out how the breach happened and how to lock them out. We were working against time, and during that period they kept deleting a few VMs at a time.
We do maintain backups through Hetzner, but restoring from them would have taken a significant amount of time. We would have needed to reinstall the nodes and master and then restore each VM individually. Given the number of affected VMs, this would have been very time consuming, and customer VMs would have been offline for much longer, generating even more support tickets.
At that point, we panicked and complied. Restoring the roughly 300 deleted VMs still took us two to three days. We compensated affected users by extending their services by an extra month, and that was the end of it.
The initial compromise and the first contact from the attacker via Telegram happened between 1 and 2 AM.
It’s not that easy to act and decide accordingly when your put in a tight spot. Just what we decided then and there.
Virtualizor as a software is not vulnerable, but people working as support on it are.
I think, the worst module could do, is leaking api keys and this can be mitigated by restricting IPs in Virtualizor settings which can use the keys. Also WHMCS/Blesta modules are open source, so it's checkable what could went wrong.
Yes, we questioned that ourselves. API keys were one of the main possibilities we acted on, but we are not able to find how they could have been exposed through billing system modules. I can confirm that we did not initially have IP restrictions enabled for API keys, but this has since been implemented.
As I mentioned before, I am not placing blame on anyone else. Ultimately, responsibility for security hardening falls on us. At that point, I had more questions than answers.
Could you please update the repair progress?
It seems they moved their gained 100USD to a mixer https://polygonscan.com/address/0xEf35250A9A2A763F87E406C2a9187A5a389c09AA#tokentxns
Indeed, it's very likely the plugin are the source of the problem.
WADR, I would avoid using authenticator apps tied to gmail account. Personally, I use air gapped device without radio transmitter plus paper backup stored in bank vault for peace of mind.
Was this recovery successful?