ChicagoVPS - Update
This is an email sent out to all clients just a few moments ago:
The ChicagoVPS team has been working day and night over the past 48 hours to restore our environment and mitigate the impact of the issues we’ve experienced. This evening we have doubled our support staff to help better serve the ongoing support load and to make sure we are providing the highest level of customer service possible.
At this time all impacted nodes are back online and all customer VPS (containers) have been reinstalled using a fresh template. Our work to recover files continues; that effort is easiest for customers who made use of our centralized backup service. If you’ve used that feature please contact our support team so we may work with you to restore your files. Customers who did not utilize that method may also contact us and we will work on restoration as resources allow.
Please understand that this is now an all-hands event and we will work diligently until every customer is online and happy. We are willing to provide whatever assistance is required, without cost, to re-setup your environments as they were before this crisis.
Most importantly ChicagoVPS appreciates your business. We understand you have a choice when it comes to your VPS hosting and we promise to work as hard as possible to make our appreciation clear. Thank you for your patience and understanding.
Director Of Operations
The ChicagoVPS team will work 24/7 until all problems are resolved. Some of you may think we deserve this because of the way I acted in the past. Maybe I did get out of hand, and I apologize but our customers should not have to suffer because someone dislikes how I acted.
I will be on better behavior, like I have been for quite some time up until a week ago. For all those effected I am sorry and I promise we will do all that we can to get you back on your feet.
Best of luck
How is this an RFO?
It was just a follow up to the last emails we sent out. Sorry.
+1 Good Read Chris
Just kidding, good luck!
gl. would recommend changing the title in case anyone comes here an are all like wtf this isn't an rfo
Well done Chris. This is really classy.
PS: About time.
Just another update:
Our Support Team is working very efficiently right now and we invite any user with a problem to contact via a ticket and to expect a rapid response.
Thank you again for everyone's business, patience and understanding.
So, still no explanation about what happened or what this potential exploit is?
We will work closely with Solus to help them get to the bottom of what happened, when we are ready. We have decided to not discuss this aspect of the ordeal any further in public but please trust that we will do our part to talk more with Solus when we sort everything out.
What @NickM said, but also, what does "Customers who did not utilize that method may also contact us and we will work on restoration as resources allow" - what sort of restoration does that include?
If they have their own offsite backup, or specific packages they want us to install. Basically to take as much burden off the customer as possible.
So, while you take your jolly ol' time getting around to actually revealing the problem, you're OK with putting every other host at risk? I get that it's not really your business to care about other hosts, but would you want one of them sitting on information that could help you protect your business? Or are you keeping it a secret so that you can use whatever exploit there is against other hosts?
Yep, he should reveal it so everyone can use the exploit.
Chris just said he would work closely with SolusVM, I think that's pretty good. Given his recent change of attitude I'm inclined to believe him. The worst thing that could ever happen is for information to be posted publicly.
@NickM other hosts remain at risk until SolusVM fixes the problem. As long as SolusVM knows what needs to be done, whether Chris tells the public at large now or next week or next month doesn't matter.
He should reveal it so that hosts can effectively protect themselves.
No, the worst thing that could ever happen is for someone malicious to have the information, and no one else knows what the problem is so that they can protect themselves. That's the current situation. Security through obscurity is not security at all. Full disclosure allows hosts to take the necessary steps to prevent someone from using the exploit against them.
Other hosts can potentially block the exploit if they know what it is. They can write an effective snort rule to detect and block the attack, they can beef up their iptables rules, etc.
Exactly. Someone already is, so making people aware is key. If they're targeting LEB providers, any one here will (not might) be next.
I'm not sure if you chaps realize this, but at this point it seems extremely unlikely that any other host is at risk of getting attacked. This was very obviously an attack on his recent behavior. I don't find it likely that it was another host/employee of another host who did it, but just someone who either watches/uses these forums, or who was informed of the thread through other channels (reddit, 4chan) - not that I'm excluding the possibility that it was another host, I just find it unlikely.
Yes, it will be nice for Chris and his team to report and resolve the issue, in private, with Solus, but it's not very likely that any other host needs to worry about it. I'm also quite sure that the precautions that most other hosts (or, at least, the ones who use this site) have taken will be enough, for now, even if they are an inconvenience to business as usual.
So, to reiterate, "they" are not targeting anyone other than the person who was already targeted, and I'm sure that person has realized this.
@lzp If there's an exploit, chances are there's more than one malicious person who has access to it.
@CVPS_Chris I only got this email an hour ago, thought strangely it's been the only one I've had regarding this entire issue. Is your email server sending everything out correctly?
I got like 4 or 5 in total
As far as I know it should be
@NickM: It doesn't matter. If this guy had "access" to the exploit, and you haven't heard of this happening before this incident, it's just not happening (without specific reason), and there are no targets outside of this incident.
If this wasn't a very specific attack, do you really think any of the hosts that use this site would still be up right now?
Someone has to be first.
How many days has it been? Has anyone else noticed any abnormal traffic on their important servers? If this was "an attack on LEB providers," they'd all be down right now. Don't be foolish.
Just because it's not currently happening, that doesn't mean that this exploit isn't currently being auctioned off to the highest bidder, who certainly could use it to take out every single host...
You're right. That is, indeed, what is happening. Right now.