New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
ChicagoVPS hacked, bunch of VPS customers offline
This discussion has been closed.
Comments
I'd buy @CVPS_Chris a beer, but don't want the earlier week Chris coming back. Welcome to sobriety!
Lots of valid concern over the commonly used software pieces every provider seems to be addicted to. Would make me mighty weary to replicate the formula like this. Someone needs to make sure this never happens again to any provider.
Refugee coupon is mighty funny, only cause of what @CVPS_Chris did to SpotVPS' LEB offer.
I was nice and apologetic and open minded about this. Still sort of am. But I hate what I see in the archives and treatment of other providers. I don't think providers should be commenting on other providers or on their offers on LEB. This week has been a classic though. Months of pent up anti-CVPS sentiment and the sh!t literally hit the fan.
I like how Chris is just ignoring being called out re: lying about him contacting soluslabs.
Can be replicated.
I NEVER said we contacted them. They contacted us, and I know Jeremiah has talked with them about this.
I read this as "Too bad dudes, you all knew up front this is just a best effort thing, and you knew not to keep anything important on our infrastructure".
Let's see - before trusting you, I did read about the service on your website
All packages are "Semi Managed" , so not "unmanaged" - fine those are semantics , but the one thing I thought that would be handled would be some degree of data-loss prevention behind the scene. I don't need to know how you do that, but I think it would not be an unreasonable assumption. Also because of the statements below:
"they knew that no offsite backups would be made"
=> So why would I even need offsite backup?
SECURITY IS OUR TOP PRIORITY YOUR DATA WILL ALWAYS BE SAFE
From the homepage. Maybe change this to: "YOUR DATA WILL ALWAYS BE SAFE or gone"
I am not a l33t admin or full time tech person like most on the forum.
I'm more of an application person (instead of infrastructure), who knows enough to manage my VPS instance and the stuff that runs on it, and yes including backups. My mistake was not to export backups off the VPS service often enough, and now I have some major explaining to do to people who trusted me.
I consider myself a reasonably well informed person and I thought I was diligent and had my ducks in a row.
I trusted my data would be safe because of the claims on the website. Turns out I'm an underinformed idiot, and I will tell the people I disappointed as much.
Cheers Chris, thanks for confirming my stupidity!
Was not out fault, software issue as explained. It has been replicated and can still happen to anyone else here.
Seriously Chris, don't you think its time to contact solusvm
@CVPS_Chris - you really should update the marketing on your site. Saying "It's not our fault" and advertising "Security is our top priority, your data will always be safe" -- right on your homepage = Fail.
Nothing against you, just saying..
Have you checked out http://facebook.com/ChicagoVPS ?
Since Chris is less technically inclined, I take it Jeremiah handles all that stuff and it would probably be him who'd figure out what happened and how the exploit took place. As he's undoubtedly got his hands full right now, I'm sure he will once things quieten down.
Or can it?
http://i47.tinypic.com/32zm6i1.png
I always escalate one step too far :P
Who knows if deleting all API users is an effective mitigation against it? No one. Except maybe Chris, since he knows what the exploit is and won't tell anyone.
Live log viewer on my iPad... bowl of popcorn ready. I love my life.
I just got a funny email from ChicagoVPS. Something about me being a customer and backups being restored to my server that's been offline for days.
I got a bit scared!
Until I realized that I am not a customer because I found out how much of a shoddy service ChicagoVPS provides - and it only took one day, after I ordered an extremely oversold VPS service from them, to figure it out!
@NickM I doubt Chris knows anything, Jeremiah handles the technical stuff so he probably will check out what happened and report to Solus. I think he has his hands full at the moment undoing this mess and that's his number one priority, so I'd give them a while.
Now I understand such things happen and it doesn't automatically mean you're totally incompetent (although surely there is some strong correlation).
HOWEVER the fact that they focus on restoring and bla bla without telling at least the customers what is all about it means they are either ignorant of the potential issues with such security breach or chose to ignore it in order not to aggravate the customers any further (which is irresponsible).
This is not a matter of "we don't know yet" or "we're too busy". If you don't know then you need to say so:
Once/if you know more you have to know people, it doesn't make sense everybody goes into panic mode and it's a huge PITA to reinstall everything from clean backups, sometimes hardly possible:
@c32, are you ever going to give up? Were trying really hard here.
@CVPS_Chris, I do acknowledge that you're working hard to get everyone back up and running, but to be honest, you're worrying too much about making nice with everyone and not releasing the relevant information. @c32 does bring up an interesting point - if someone was able to delete a ton of VPSes, what else were they able to do? Were they able to read data that was on your customers' VPSes? You haven't said anything one way or the other regarding that, and that's important too - in fact, probably even more important to your customers than you releasing the actual exploit.
No, there was barely enough time for them to even delete the VPS's. This could have been a total melt down but we caught it before more damage was done.
@CVPS_Chris when will you address if customer billing records were lost or stolen, or whether customer data on VPSs was copied/stolen by the intruders? If you are in the process of making that assessment - fine - but would be good to communicate that. If you don't have sufficient situational awareness to know, that would also be good to know so customers can make the right choices for themselves (assume breach of personal data). All the email updates I've received so far have only talked about data destruction, but from experience, its rare intruders that gain significant access to an environment don't grab takeaways... Appreciate you covering this in your next email update.
@craigb,
Nothing on billing side of things was compromised, so no need to worry about credit cards etc. We do not believe that anything was copied from the nodes, it was a quick "destruction of data", just trying to hurt ChicagoVPS.
@CVPS_Chris are you storing credit card details in your billing system?
If the client decides to, yes. They add it on their own.
@CVPS_Chris interesting, i didn't find anything on your web site about PCI compliance. And nothing about PCI compliance on the nwnx.net website either. Do you (the company behind the ChicagoVPS brand) have PCI compliance?
Yes we do, or we wouldnt be able to store. When we got our merchant license we were approved for everything.
Well, I can't risk my CC details... 12.50$ replacement fee + 60$ experss DHL delivery from the USA.
Get one from Bulgaria, less problems with MaxMind i guess...
M
M
Never had Maxmind problems, neither with BG or US cards :P