New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
ChicagoVPS hacked, bunch of VPS customers offline
This discussion has been closed.
Comments
@pubcrawler i doubt any data retention laws says "keep the disk image of the customer". I think it would be also quite illegal to keep the customer's files after he has been terminated. At least in the EU.
What we do about data retention here - we keep all the data that is needed to identify who was the user of IP x.x.x.x, for 1 year period.
@rds100, Said perfectly, and something everyone still cannot seem to comprehend.
As for you guys thinking that we were compromised because of something I have set on the API such as "winning" or "password123" your wrong. I have nothing to do with installations or tech support for that matter.
Stop speculating and just take the facts as they are. At least @rds100 was able to come up with a conclusion logically here. As for why people are not posting here or WHT, Ive said it already, its because we are answering tickets very efficiently and taking care of people as best we can and as helpful as we can be.
Putting us down saying we are not competent is wrong, and simply just said to make us look bad. We know how bad the situation is and don't expect this to blow over like it was something as little as a DDoS. I expect to lose customers, and I expect to keep some.
Life will go on, this is not the end of the world. It hurts, yes, but it will only make us stronger.
Did they say they provide backups? I don't know, but I know I do not, it is stated in the terms, the customer is ultimately responsible for their data, even if it is the providers fault for losing the data. You don't get budget pricing and enterprise service, pick one or the other.
I am not on anyone's side, I am Switzerland, but going on about backups when everyone knows the customer is responsible for their data. The email posted here from ChicagoVPS said they will get fresh containers provisioned, then try to recover what data they could.
I agree @jeff_lfcvps.
With most sites and this market, the afford-ability of more servers is negligible. Having a dual located live site is obvious even if one is just a hot standby.
From those, you should also have the obligatory backups.
Redundancy is N+1. That means 3 sets, 3 live locations.
No clue what was on our VPS since we let it linger and do it's business. Ideally nothing important, but, we've had a few other nodes go nuts in the past month and changes really have systems mucked up.
If cancellations make stronger good luck.
Well said!
Queue providers offering coupons in 3...2...1?
@pubcrawler, I did just change our backups slightly though based on your one comment -- if a VPS was deleted, it's backups would be cleaned up the next time the deletion routine ran (daily, in the afternoon).
These backups are now kept for 2 weeks.
@jeff_lfcvps,
Glad to help
Take it that is a option in Solus?
I'd increase it to 30 days myself.
@serverbear,
Too bad SpotVPS doesn't return the favor for the original blow up with CVPS.
SpotVPS should offer a CVPS refugee special
I have mixed feelings on this. Chris has attracted the karma comments for his actions this week; however customers losing data is not good (even if they should have made backups). I also would never cheer on a provider suffering this problem. At the end of the day many of us here rely on this business for our livelihood.
Comments like "I expect to lose customers, and I expect to keep some. Life will go on, this is not the end of the world. It hurts, yes, but it will only make us stronger." Although quite true come across a little insensitive to customers who may have lost data
Some things do not seem to add up, in part with the conflicting info coming from CVPS and SolusVM. It is however CVPS right to release what info they want to who they want. I would only hope that customers get a sufficient RFO.
I wish both ChicagoVPS and their customers a speedy resolution to this situation
@pubcrawler, No -- our backups have nothing to do with SolusVM -- in-house system that our clients seem to like. You can queue a backup, restore your VPS, delete backups, lock a backup from being removed by the automatic clean up, and access a specific backup set using FTP-SSL.
From what I can see this points to either an exploit or a security failure. One of those two things is true in the case of CVPS.
Now when you consider SolusVM, you have to understand that even if an exploit occurred they would not be in a rush to acknowledge that publicly. Instead, they would keep quiet until a resolution was in place or contact providers privately. Certainly nothing in a public forum. At this point it's best that everyone stop speculating and stop talking about it.
Now wait a second, let's quantify what "taking care of people as best we can" means in this particular case. It means just answering tickets with something like "we're really sorry your box was down and sorry we can't recover it and sorry the backups are gone too".
If you're starting with a blank slate you can really start with any other provider around the world, in fact you'd be better off with anybody else really, for one thing they won't be swamped with trouble tickets, in panic mode and so on. And didn't manage to lose your data and backups this week.
What are $4 2GB deal, cant see it happening.
You act like were dead. A small portion of the business was effected yes. Not its entirety.
I do feel bad yes, but its an unmanaged service and they knew that no offsite backups would be made. If it was something I do personally to cause this it would be a different story, but what happened was out of our control.
You have emotion
Yeah, Im actually a nice guy when people are fueling the fire.
@CVPS_Chris so what actually happened?
Nifty in-house system @jeff_lfvps. That's the kind of stuff that sets a provider apart from the rest.
As I see it, too many folks competing for $0 price point and < $2 profit per customer.
Remember back in June @jeff_lfcvps and a little thread blowup
http://www.lowendtalk.com/discussion/2959/lfcvps-35-off-2x-ram-2x-bandwidth-11-free-backups
Damn someone has been piling up karma drowning points, hasn't he?
@CVPS_Chris, of course you aren't dead. But the mean Chris seems to have been exorcised
Maybe it's time to exit the VPS market? I mean you only do it because you can and most of your pocket jingle comes from your other businesses which are very profitable.
If the business is so successful then you ought to be running more than a three man miracle crew or whatever you have near that. Once this sort of thing happens, it is bound to happen again.
Yeah, no warranties, blah blah. This stuff happens with huge companies too.
After doing some more logical analysis - here is what i think happened:
Chris's billing system server / web server got somehow compromised. From there the attackers were able to get the SolusVM API ID/Keys and connect to the solusvm master and issue the API vserver-terminate calls (remember, the billing system IP is already whitelisted for API access, there is no need to spoof anything).
Now, i don't know if they accept direct payments by credit cards or if they store credit card details in their WHMCS. If they do - all users that have given them their credit card details should be warned to cancel and reissue their cards. This data has all been probably exposed.
@pubcrawler,
It doesnt need anymore than 3 people yet we have more than that. Im not going to exit the VPS market because of one incident.
We still plan on retaining a majority of the clients. This has been a bad 3 days, but will eventually get through it.
I liked everything you said up until that. Had nothing to do with WHMCS and you are just going to cause more chaos than there needs to be
>
Well, but it adds up. That IP is already authorized for API access, and the ID/Key are also there. I don't say it was WHMCS which was the entry points - i see there are other websites on that server too. Or could have been insecure ssh configuration or something else...
Will do next week. Nothing against ChicagoVPS, but I find the idea of a refugee coupon code for them to be a humorous gesture. I'm sure they will too, knowing full well that I don't have the capacity for 1/10th of their clients.
You are correct we did open a ticket with you. We opened it 1 Days 13 Hours 12 Minutes 49 Seconds Ago
That is also the 'last reply' time on the ticket.
LOL @Spencer, I was thinking of that song too