ChicagoVPS hacked, bunch of VPS customers offline

pubcrawler

Got an email 2+ hours ago directly from ChicagoVPS (am a customer):

[CRITICAL UPDATE]

re: Chicago VPS11, Chicago VPS12, Chicago VPS14, Chicago VPS16, Chicago VPS17, Chicago VPS26, Chicago VPS28, Chicago VPS29, Chicago VPS30, Chicago VPS31

ChicagoVPS experienced a brute force on the SolusVM API for the administrative section. This caused the above affected nodes to become compromised before we were able to stop the attack.

What does this mean? Currently the VM's on these nodes are being recovered to the fullest ability of Chicago VPS staff from the incomplete data destruction process and from central backups. Any VM's unable to be recreated from the remaining data or from backups will be created fresh.

ChicagoVPS is committed to customer satisfaction and any way in our ability will do what we can to get everyone back up and going as fast and as best as we can.

We will post additional updates on twitter and facebook and from time to time send out an email regarding the current status of the progress.

If you have any questions in the mean time, feel free to directly email me at jshinkle@chicagovps.net

Sincerely,

Jeremiah L. Shinkle Chief Networking Officer ChicagoVPS

ShardHost

Thats a pretty crappy thing to happen for both client and provider. Is this an exploit in SolusVM or something not locked down correctly?

Taz

Karma is a bitch. Hope it is not too bad and they can recover.

jarland

Anyone have a working theory that immediately comes to mind as to what happened here? Admin API would be the API used to connect billing software would it not? Is it not restricted by IP?

pubcrawler

Karma aside, I am wondering where the exploit is and if it's a SolusVM issue. An exploit in SolusVM could impact tons of folks.

ihatetonyy

@Taz said: Karma is a bitch.

Even if their.. marketing, I guess, ruffled some feathers, I don't think it deserves one node getting nearly-trashed, let alone ten.

With that said, still up over in LA.

Taz

@Liam @infinity please remove /hide ths thread. If this is a solusvm exploit, this can have hugee affect.

jarland

@Taz Nope. If there's an exploit and someone is targeting LEB providers this is the place it should be exposed.

Taz

But before solus releases a patch, you are welcoming more skiddies.

CVPS_Chris

@Taz said: But before solus releases a patch, you are welcoming more skiddies.

Your problem, should probably go deal with that.

jarland

@Taz said: But before solus releases a patch, you are welcoming more skiddies.

Nope, you're warning LEB providers to watch their butts. Otherwise you're keeping the info from them to let them get targeted if this is going to continue through the night.

NateN34

@Taz said: Liam @infinity please remove /hide ths thread. If this is a solusvm exploit, this can have hugee affect.

Not an exploit (according to ChicagoVPS):

"ChicagoVPS experienced a brute force on the SolusVM API for the administrative section. This caused the above affected nodes to become compromised before we were able to stop the attack."

jarland

@NateN34 said: Not an exploit:

But isn't that API locked to IP?

Jack

@jarland said: ut isn't that API locked to IP?

It should be yes.

ShardHost

@jarland said: But isn't that API locked to IP?

I know ours is, or at least SolusVM tells us it is, which is why I am asking about exploit ;)

Taz

API can only be accessed from whmcs IP I assume . Since someone was able to.bruteforce, something might not be right?

Jack

jarland

I'm sure when @CVPS_Chris gets this mess sorted he'll fill us in on whether the rest of us should be worried about it. Gonna be a long night for those guys.

CVPS_Chris

Confirmed that another host had the same issue. Everyone should be concerned.

Dont ask who, it is up to them to release it and not my job to tell.

seikan

Luckily my VPS with them are not affected. But this is real scary! Backup, backup, backup guys!

concerto49

Which version of SolusVM are we talking about? The latest?

PhilND

@CVPS_Chris

Ego aside, I hope you are being serious about what you have just posted?

Jack

Jesus Just spoke to Jeremiah via email this sounds nasty.

CVPS_Chris

@PhilND said: Ego aside, I hope you are being serious about what you have just posted?

Serious. I guess I can do one nice thing.

jarland

@Jack said: Just spoke to Jeremiah via email this sounds nasty.

What are your preliminary thoughts on the effect of revoking the API keys used for billing software? Assuming he explained more detail to you than we know. That's what I've done, as well as reduced stock to 0. I don't take chances.

Jack

@jarland said: What are your preliminary thoughts on the effect of revoking the API keys used for billing software? Assuming he explained more detail to you than we know. That's what I've done, as well as reduced stock to 0. I don't take chances.

Disabling the API's would be best yes.

However I wasn't 100% sure what was going on so I Just did :

:)

miTgiB

@jarland said: That's what I've done, as well as reduced stock to 0. I don't take chances.

I think you'd be safe selling stock, just revoke terminations and do those manually if there are any after the cron run

kbeezie

Though I'm wondering if an actual exploit occurred and if it's not say some kind of hardware failure at fault? I mean a brute force attack? Didn't have something as simple as Fail2Ban installed? Just curious cuz it doesn't seem to be adding up.

UGVPS

Yeah i have also followed up on this... This is BADD NEWS right now!

Jack

@UGVPS said: Yeah i have also followed up on this... This is BADD NEWS right now!

? You are the other host or what?

jarland

@kbeezie said: Didn't have something as simple as Fail2Ban installed?

I don't know if fail2ban or LFD would cover SolusVM API access without some tweaks that most people would probably consider overkill prior to knowledge of such an exploit.