New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
How many people here actually run QEMU with
-nodefaults -chroot /var/empty -runas qemu -sandbox on, though? I'd be willing to bet that a good portion of providers here just run QEMU as root with all the insecure defaults in place.Is the fault in virtualizor itself or within the virtualizor whmcs addon?
And if the problem is virtualizor whmcs addon, then who created/maintains that addon? virtualizor right?
I'm still confused if those who were hit with this had their admin panel IP restricted or used Virtualizor support in the past and did not rotate passwords after giving temp access to their staff.
I don't think it's really a Virtuailzor "exploit" more than it is a combination of Virtualizor hiring some questionable characters with bad internal business security practices which allowed previous shared passwords to get saved by an employee(s) and later some live-chat leak or something.
That, combined with hosts not rotating their passwords after sharing them with virtualizor and/or also doing admin area IP restrictions make me thinks its a little bit of "Column A' and a little bit of "Column B".
If this was really a Virtualizor exploit of sorts, I think we'd have seen much more chaos in the industry. And the fact it seems relatively isolated to only one or two nodes with each provider instead of their entire infrastructure makes me think that perhaps, these nodes had issues in the past that they needed Vitualizor's support for. So likely shared creds with Virtualizor at some point but did not rotate passwords after. Why ransomware just one or two hypervisors instead of all of them? Hackers earned what, $200 in total so far from all this? Little gain.
I'm keeping our Virtualizor stuff offline (panels, not VMs) off for the time being just to be safe but the more I've read into this and based on what has been said (and what has been asked, but never answered) leads me to believe that this could very well be just a combination of Virtualizor's leaked customer root details and providers not rotating passes or restricting :4085 to internal staff IPs only.
Just my $0.02.
If a provider doesn't change root passwords after a data breach from a third party or doesn't rotate API keys, have no IP restrictions in place, VirtFusion and any other panel doesn't fix that.
I mean sure, but I'm not sure any providers should be trusting Virtualizor with any passwords if they can't be confident their staff won't be leaking / stealing keys.
If you can't trust them in the days after they have credentials shared with them, it's fair to say they shouldn't have been trusted before either.
About the last Virtualizor hacks etc. > @ralf said:
You shouldn't trust ANY thirt party when it comes to passwords, API keys etc. Remember, their last data breach was not caused by them but by their chat software they have used.
And it seems that some providers still use the old passwords, API keys etc.
Virtualizor recommends some security measurements like IP restrictions and still we see issues and all blame Virtualizor.
This was in an email I got from HostSlick, suggesting that they maybe didn't change credentials or similar after Virtualizor's breach last year:
and suggesting the VPS data is lost:
Is your vps running again?
I can't seem to boot at all, no vnc output in recovery env and almost every other action fails as well
I would expect to see atleast some "unable to boot" in the vnc terminal...
I did end up reinstalling it and it started running again. Obviously I lost my primary drive but it was a storage VPS so I wanted to make sure the data on the HDD was okay (it is). Before that it was just failing to do anything. Even now the panel is still very flaky.
My vps was on one of Hostslick nodes that weren't affected by this issue, but it just happened to have issues with the disk after a quick reboot. I'm guessing something got corrupted. I attempted to reinstall vps but it has issues resizing disk and mounting the disk
So everyone is still blaming Virtualizor without any proof.
How many of the providers that were hacked had given third-party access?
Example: They tested or provided services to CaaSify. CaaSify gets silently hacked, and the attacker extracts Virtualizor API credentials for several providers. Either directly or through CaaSify’s WHMCS module.
Another WHMCS module gets hacked and is used to dump tblservers, which contains Virtualizor API credentials. I pentested a third-party WHMCS module today, and I was able to extract information, not from tblservers, but this might be possible with another vulnerable module.
Both scenarios would fit, and you can’t blame either of them on Virtualizor.
If I were one of the affected providers, I would try to find out what I have in common with the other victims instead of blindly accusing Virtualizor, especially when Rack911 Labs has pentested both the panel and the WHMCS module.
Also storage VPS for me.
Before I try to reinstall, I want to make sure if you checked this box explicitly?
Goal is to not loose data on the secondary disk but I don't fully trust the process now
I did select that and it was fine. I did have to recover the partition as the superblock got corrupted but the data was all there. But it might be worth checking with them as I don't even think that should have happened (unless that was part of the exploit).
Edit: I should not be mad...
Are you able to please post the whole email? One of my @HostSlick VPSes was affected but no formal notification from them as yet.
So, we're finally getting prepped to move all of our older legacy VPS plans over to VirtFusion, where all of our new plans for the last year+ have been provisioned.
For those wondering why providers "don't just move to VirtFusion", it's not really that straight forward and requires a lot of manual work.
For importing servers into VirtFusion... You can see what is required here: https://docs.virtfusion.com/guides/importing-servers
TLDR:
This doesn't include linking the migrated VM to the user in WHMCS, or creating their account in VirtFusion either. So I guess you could create the "blank" VM in VirtFusion post-migration via WHMCS for the user by creating an "order" for them for that new product and manually approving it so that it's created via the API. That may make linking the new VirtFusion VPS easier to their WHMCS account as opposed to doing it all manually. I've not had my coffee yet today so perhaps there is an easier way. All ears.
Still, a massive pain in the ass. It's why we just decided to discontinue our old plans and mark them as legacy and basically provide no support for them in hope that people self-upgrade to new plans in VirtFusion... but with Virtualizor / Softaculous not being a company we wish to do business with anymore (even if this most recent incident isn't 100% their fault) it's time to move.
In any case, got new hardware live in WA to start the migration process and requests in for PA. Put in an order for new hardware in NL as well, but that is going to be a month + out, which it'll likely take me that long to just migrate WA manually.
@MannDude I legitimately hope you and your team get a well deserved break after migrating everything over.
That is not a friendly process at all. I actually don't know what's worse... the way I did my migration from Hyper-V to Proxmox(clonezilla cloning images baby, I had to do this 14 times) or what you're about to go through.
Sure, it wasn't an email they sent out... just a response to a ticket I raised.
That seems like an extremely irresponsible statement to make. The script kiddie had full access to the server contents. Just because he didn't gloat about having stolen data does not in any way indicate that the data was not stolen.
...
Thankyou!
I find it interesting that the boot message refers to cloudcone even though these were HostSlick services... were only LET hosts targeted?
yes, mostly same tactics to damage small hosts.
I've spent many years in the cyber security industry and I've only ever heard of Rack911 Labs on LET...and only since this nonsense with Virtualizor began...so I decided to take a little look.
Obviously they're an offshoot of Rack911, which is presumably how they acquire clients.
But they don't seem to have any reviews or recommendations on Google, their website hasn't had a meaningful update since 2021 and their only USP seems to be that they're dirt cheap.
"Our security services are some of the most affordable in the industry"
Aside from LET, literally nobody is talking about them and they barely have any kind of presence online at all, which is weird for a company that's been around since 2013.
They make a big deal out of being Bug Bounty Hunters, which isn't normal for a professional PenTesting company because bounties are paid out at a fraction of real work. Usually this only happens when you have little to no work and bills to pay.
They do have some visible research into exploits against Antivirus platforms from September 2020, but whether the people involved in that research still work there is anyone's guess. The fact they don't appear to have done anything since suggests not.
In short, I've found nothing to suggest that they're credible PenTesters, (in an industry full of charlatans), so I certainly wouldn't gamble my company security on one of their audits.
With all due respect, you have no idea what you are talking about.
You ask anyone who has been in the hosting industry long enough who RACK911 Labs is and they will tell you straight up. We are by far, by a mile, the most reputable and well known security company to the hosting industry. No one comes close and that is why we are in such demand constantly turning down new panels due to lack of time, for real.
We still work with a lot of the big names on a contracted ($$$) monthly basis. Companies I guarantee you use daily or have used. We have found more security flaws in cPanel and Plesk than anyone else to this date and take full credit for improving the state of their security over the last 10+ years. Most of our hosting panel clients have been with us for 5 to 10 years, that says something.
Our website hasn't been updated in years, too busy doing other more important things and very little of our time is on bug bounties... also side note, we actually have nothing to do with RACK911.com these days and lost access to that when Steven died. Some day we will rebrand but it's not a priority.
I have been doing this for almost 20 years, with approximately 13 or 14 of those years focused in the hosting industry and that is when we started sending off security flaws to cPanel and other big name hosting platforms.
We have come up with new exploitation techniques previously unheard of. I wish I could show you our master list of exploits, public and unpublic via contracts/audits, it's almost nearing 1000 to this date. Truly, almost 1000 security flaws!
Every month we send off security flaws to the companies who contract us. They are all hosting platforms or somehow connected to the hosting industry. I assure you, our work has improved hosting platforms you use on a daily basis and improved the overall security of the hosting industry by a landslide.
Just because you don't know us, frankly means nothing. You will never find another post by myself talking about our accomplishments, but you sir, have absolutely no idea what you are talking about when it comes to all of the good we have done for the hosting industry and who we work with on a constant basis.
why people waste time replying to "cloudhopper", is out of my understanding.
You're right, the person is clearly a troll and I should not have engaged.
You just gave yourself away to those of who know about these things, and the funny thing is you don't even realise it.
Working in the hosting industry, (even PenTesting infrastructure), is a completely different discipline to software auditing and code reviews....and as someone who regularly commissions both, I'm overly familiar with "Jack of all trades, Master of none" syndrome and that's why I have no problem calling bullshit on you and Virtualizor.
I'm honestly surprised that you report security vulnerabilities each month and yet you don't have a single CVE credit, even by accident, or any online footprint, any customer reviews or endorsements, even after 13 years of business and 1,000 reported vulns. It's really quite a feat!
It's also sad that you're offering such LowEnd security support, whilst blowing up one of your clients badly and causing a LOT of collateral damage in the process. It's classic Dunning-Kreuger. Pretending to be Jack Bauer whilst acting like Mr Bean.
But as we said after the OuiHeberg hack, time will tell. I'd say it already has, but some people are hard of learning so more hacks are bound to happen.
You can keep blaming customers, not addressing the core issues and offering bad advice to Virtualizor about root causes. But in the end reality always comes calling, so for now I'm happy to sit back and watch 🤷♀️
100% Agree