New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Isn't BMC an IPMI? :-D
But what does the firmware have to do with it? Junglesec ransomware has been around since 2018, so even if it was using an 'N day' exploit, surely that would have been patched by 2023?
Also, what exactly are Asus supposed to be trying to reproduce? If the provider has identified a new bug that's being exploited then maybe there's testing/patching for the vendor to do....but Occam's Razor suggests that weak credentials and an exposed endpoint are the culprit here.
This story makes a bad publicity
It has been 3 days. Still no official statement, no virtfusion node.
Still waiting too.
But you’ve got IPMI wired up to a separate internal network right? And you are now panicking that an attacker may have access to said critical network, right?
It's possible that this is the vector, (article from Summer 2023), which was presumably patched in the September 2023 firmware update: https://www.csoonline.com/article/647906/new-vulnerabilities-mean-its-time-to-review-your-lights-out-server-bmc-interfaces.html
The vulnerabilities described above would require access to IPMI, so either it's exposed to the world or, more likely, a private network is breached...and if it's a network breach then just fixing that specific vuln probably won't prevent further hacks, especially as weak passwords/bruteforcing would still be an option, (and they're the more likely root cause than someone using an 'N day' anyway).
But what's really interesting is how few recent results there are for JungleSec on Google, (barely any mentions in the last 2-3 years), so it's not clear if they're still active as a criminal group or whether someone, (maybe even an insider), has justed used one of their old encryptors.
Another interesting thing is that the email address "[email protected]" only appears on Google related to this incident, which isn't what you'd expect from a ransomware campaign, so I'm inclined to think this is a targeted attack, (presumably intended to disrupt rather than extort), rather than the usual "spray and pray" breaches associated with encryption extortion.
Any updates on this? As it stands this issue and the response ('RAID failure') paints greencloud in a bad light
The response RAID failure didn't stand.. The host's rep confirmed.
we all knew, just waiting for full story from GCV
Any update on this ?
Luks or not, I got out.
Not sure what's up with their NetActuate SJC location, if their IPMI network (or LAN was infected) and due to lack of any response, investigation, blog post, etc. just gives me a bad feeling.
At least they gave me a refund on one of the machines, I don't trust them honestly anymore.
You don't need 10 admins to figure out your shit got encrypted, just one with a fucking clue, don't be stupid.
You don't need an army of two lawyers to follow the discussion, understand what's written, and then respond to that, instead of taking things out of context and being a jerk about it.
He apologized for the situation and explained that due to specific circumstances, neither the two senior technicians nor he were available at that time, and a less experienced person handled the issue.
You've been around long enough to know that there are different tiers of support, and many LEB hosts don't even have tier 1 support available 24/7 (while this host clearly does...) let alone having tier 2/3 support. Don't be stupid.
I'm not saying this issue was handled correctly - even the host rep admitted it wasn't and apologized for it. But now, it is what it is. It's perfectly legitimate to not trust them if you feel that way.
My long-term experience with them (I'm hosted at five of their locations) has been nothing but great, and for the price I'm paying (which I'm not using as an excuse), this one-time occurrence won't turn me away.
The story doesn't pass the smell test. Level 1 should've been able to see the node was ransomwared, the rest was a lie. I logged in after I was tipped off and booting a rescue ISO on a second machine, I saw the same. My primary machine had luks. You simply need a brain and SSH login to figure this out, not the techs you suggested, or 'lack of senior techs' - your bias is showing.
I've been around long enough and owned a hosting company, was lead tech at HostDime almost two decades ago. Used Poornam and Bobcares for overnight 24x7. That used to be a thing. I understand the different levels of support, and they do have fast replies, it does NOT excuse blatant lies. Fast replies do not make up for straight up lies and zero communication.
I hope you use luks
Definitely not handled very well, that's the understatement of the year. For this to happen in the first place AND then for it to be handled the way it was means there are other more core systemic issues with the provider. You can see the red flags and decide to overlook them for a good deal (risk vs reward) but to ignore the red flags completely is problematic.
I understand they claim that Level 0 support screwed things up, I wasn't focusing on that.
More on the lack of any updates, lies, and burying this.
Was it IPMI, were other SJC nodes affected? Were they investigated? The machines only disappeared from SolusVM a few hours after this thread started.
I'll forgive the entire having Level 0 support derp it.
My issue is:
Lies about RAID, instead of the hypervisor being ransomwared. That was not the junior tech, but someone in management deciding to lie, and lettting it go 24 hours before updating the lie with 'sorry the RAID restore didn't work' - The only way we knew this happened is a few people tried their rescue CDs. You've clearly missed the point, and before this they were fine.
With zero notification to customers on the true issue, and them burying it - yeah I have a problem with that, and it has zero to do with the level of support. They mentioned the same security setup is across all their sites and threw in some bullshit about firmware update... was IPMI exposed to the net?
That's a good coherent way of putting it.
I went from feeling "boy what a stable brand" to "no one's steering the ship, or simply doesn't care" after this.
Level 0/1 may have screwed up, but what led to multiple emails from management with a straight up lie (multiple emails that the RAID restore failed) ?
That wasn't a junior admin, it's someone covering things up after the fact. GC never told anyone, someone posted a screenshot, a second person did, and I was able to confirm it on my second machine on that node before they yanked the VMs out of Solus due to this thread.
Choose your own host, I'm just staying far away due to the blatant lies of GreenCloud about their hypervisor being ransomwared, not their support or level 0 guy who didn't see it.
@Kris
I understand your point, but you quoted a post where I was responding to someone who had expectations of L3 support being available on-site 24/7/365. I mentioned that this is a bit of a stretch for a LEB host.
Your response to mentioned post, ending with "don't be stupid" seemed completely out of context, hence my further response.
I'll give you that, I didn't attach your quote to the context and yeah I doubt there's one or two seniors available 24/7. I'm more upset after a few weeks there wasn't a post-mortem, explanation, anything. Instead some new host is getting shit for a Virtualizor mistake over pages, when this was forgotten. Would've expected some statement by now, even boiler-plate on what happened, and steps taken to avoid it in the future... Instead nothing. If you didnt' read LET, you would think a RAID card failed.
That was to needing a staff of ten employees to realize it was ransomware, not being on 24/7. I tried to be as nice as possible with what I perceived to be your response, but I saw it later in the thread.
Everyone can choose their own host, just the fact not a peep from GreenCloud despite them claiming it's not their first rodeo, blah blah 11 years experience is disappointing from a host I used to like, and used for production. You're left to wonder if they did investigate SJC, or swept it under the rug. If it was access via IPMI, were they on the LAN. Did they yolo and admin/admin their SM pizza boxes.
Instead of a paragraph they choose to lie, twice, and leave users who don't browse this forum to think it was just hardware failure.
Honestly @Mumbly their support is fast and responsive. I can't even blame the person on-call if it went down like they claim.
It's management who really dropped the ball on this one and soured me on a brand I liked, and paid 3 years for by doubling down on a lie.
You're speaking as if people became clients of this host yesterday. How many years of being their client do you see red flags? Yes, shit happened and I am not looking for excuses for that, but apart from this, they have been pretty stable for years.
My decision to not abandon ship at the first problematic situation, after being their client for years, has the same legitimacy as your potential decision to leave immediately in doubt that they can provide you the level of service you need.
I don't blame you for feeling the way you do, but please give me the same courtesy. We'll see how things go from here onward. ;-)
I believe I have given you the same courtesy. I didn't say anything negative. I just said if you saw the red flags but decided to overlook them because of what you perceive as a good deal then that's certainly your prerogative. People are free to give anyone they want their money but I do hope that everyone is well aware of these glaring red flags regardless of how clean their record may have looked in the past. Character trumps history in my book. If someone is willing to cover up stuff now how do we know they haven't covered up stuff previously or will cover up more things in the future? That's just not a gamble I am willing to take when there are plenty of other great providers out there. I hope your gamble pays off for you though. Good luck!
Official Disclosure Notice from GreenCloud:
We’d like to provide an update regarding the recent situation as part of our commitment to transparency:
1. Internal Audits & Security Assessment:
We have completed a thorough internal audit and have also engaged an independent cybersecurity specialist to conduct both internal and external scans of our entire system. While the process has taken some time due to our scale, we can confirm that no breaches have been detected in our system or VPS nodes.
2. IPMI/BMC Vulnerability:
The issue stemmed from a vulnerability in the IPMI/BMC on certain Asus servers, which affected a specific node in our network. We have reported this to Asus and have implemented additional security measures to ensure this issue does not recur. While we are still uncertain whether we were targeted by hackers or competitors, this incident has been a valuable learning experience for us.
3. Staffing Changes:
The technician who sent the incorrect notification to customers will no longer be employed with GreenCloud, effective this month. We acknowledge that this situation should have been handled better, and to prevent similar issues in the future, we have made adjustments to our staffing structure, including ensuring that a senior technician is available on every shift, 24/7.
We understand that trust is hard to build in this industry, and we are committed to earning and maintaining that trust every day. If you were affected by this situation or have any concerns, please feel free to send me a private message.
Thank you for your understanding and support.
I randomly picked the second answer. I hope it's no. I don't speak Chinese.
Can you share more information about this vulnerability? Was it the issue patched in September 2023, (described in the CSO Online article below), or is it something new that you're expecting Asus to patch and that other Providers should be made aware of?
https://www.csoonline.com/article/647906/new-vulnerabilities-mean-its-time-to-review-your-lights-out-server-bmc-interfaces.html
JungleSec has been dead for years and the ransom email address has only ever been reported in relation to your breach, so it's almost certainly targeted at you...but if it's a Competitor with a 0-day then it's quite possible others will be similarly targeted
based on no 2, you should know who impacted by that issue’s right?
Ok so that's why I saw multiple service transfer threads for this provider.
Not really. There is a spike in increased transfers after their annual Birthday sale. People tend to get newer deals and let go previous years services which are upgraded. This is a one off instance and has no significant correlation to increased transfers.