GreenValueHost hacked, data stolen

Lee

Given all that has happened in the past 24 hours and you want a fast security fix. lol

XxNisseGamerxX

Dear Valued Client, 

It with our greatest remorse that we are sending you this email today to inform you regarding a breach of security that has occurred on GreenValueHost systems, resulting in a possible release of customer information and the rollback of data stored on our WHMCS client area and SolusVM VPS Control Panel. 

This breach occurred yesterday (June 23) around 7:00 PM CST (Central Time). Our technicians immediately began working on the issue at hand and was able to salvage data from recent backups to bring ourselves back online and running. 

Our WHMCS client area has been restored from a backup that was taken 10 minutes BEFORE the breach 

As far as we are aware, there is NO client sensitive data from our billing/support system being distributed. Our WHMCS database was 'dumped' into a publicly accessible domain under secure.greenvaluehost.com, however after after further investigation we've gotten to these conclusions regarding the sql dump: 

1. The dump was created and brought offline DURING THE SAME MINUTE. One of our ex-staff members Jack has managed to coincidentally (as verified by evidence) discover the dump URL, passed it onto Jon, which promptly resulted in the server being immedaitely shut down. A few hours later, the server was turned back on and the sql dump was immediately removed. 
2. After scanning logs, it appears that Jack and Jon were the only two people (the hacker, apparently, did not yet test the sql dump) that accessed the URL. Therefore it is concluded that client data from our billing/support system is SAFE. 

Our SolusVM VPS control panel/administrative area has been restored from a backup taken 10 hours BEFORE the breach occured. (New login URL: https://104.131.252.131:5656) 

There is a flurr of rumors out in public that a copy of our admin table database has been leaked to a public pastebin URL. At this time we cannot confirm that the admin table that was leaked is authentic, however it looks to be forged as the table is not completely accurate and is missing some information that should be there. From the information we were able to gather from the SolusVM server, we have concluded that the hacker whom compromised the server spent little time in this server as well and performed commands such as rm -rf to delete crucial folders. 

Although we are able to confirm that no data has been stolen/leaked from our WHMCS billing/support database, we are unsure of whether or not data has been stolen/leaked from our SolusVM database. It appears likely not, as what's been "leaked" appears inaccurate/incomplete on top of the fact that the entire operation of compromising and exploiting our systems was a "rushjob" -- The hacker knew that he had little time to do damage and thankfully wasn't able to do much damage at all (Nodes appear completely untouched from Solus; we are still verifying this) 

Despite our doubts that much has been done in the compromise in our systems, we don't want to take any chances. We care about your security, your privacy, and your safety. As of this notice, we are enabling the "Disable MD5 Clients Password" option in WHMCS which will force all clients to request a password reset before they are able to successfully log in to the client area. This, as a security precaution, will remain in place for another 24-48 hours. Clients are also advised to change their SolusVM VPS control password AS SOON AS POSSIBLE, as we are unaware of an option to force password resets for SolusVM. 

At this time we are still working on restoring SolusVM to full working order. We are aware that it is not fully usable right now, but we wanted to get an update out as quickly as possible for the well being of our clients. We can assure you that all data, although some may be rolled back, CAN be fully recovered and we have NOT lost control of any of our systems. They are more secured than ever, with additional heightened security measures still being put into place as we continue to sort things entire situation out. 

If any of your client data has been rolled back or not applied (such as invoices from automated subscriptions during site downtime, or SolusVM changes AFTER SolusVM is sorted), please contact us. Again, we are still working on getting everything sorted and would like to take this opportunity to let our clients know what has happened, and how we're proceeding to resolve things. We will be sending ANOTHER email shortly notifying clients of the completion of SolusVM data restoration, any further information from our investigation, and how we will be moving on from there. 

In the meantime, we are greatly sorry for any inconveniences these events have/may cause and we appreciate your continued patience, understanding, and patronage throughout this ideal. 

Any questions, comments, or concerns can be addressed to us through our helpdesk ticket system at https://secure.greenvaluehost.com/submitticket.php 

Thank You,

The GreenValueHost Team  

luissousa

@tr1cky said:
Hm... RamNode downtime for openvz fix was 5 seconds, prometeus downtime also 5 seconds, why do they take more than an hour?

@W1V_Lee said:
Given all that has happened in the past 24 hours and you want a fast security fix. lol

All nodes should be back up now. They say they didn't take more than an hour. If your VPS is still offline, please open a ticket.

Lee

His apology macro is in overdrive. Honestly the ones that stay with GVH after these recent events deserve everything coming to them.

tr1cky

@luissousa said:
All nodes should be back up now. They say they didn't take more than an hour. If your VPS is still offline, please open a ticket.

Server still down, raised another ticket.

Caveman122

LET, you come for a cheap VPS, and stay for the drama!

eddynetweb

@Caveman122 said:
LET, you come for a cheap VPS, and stay for the drama!

Mark_R

@WebSearchingPro said:
Frankly I wanted to help Jon because I felt bad for him, I saw the thread talking about his whmcs being down so I messaged Jon to see if he needed anything. That's actually how I helped out the first time, something was amiss and I wanted to help solve it. I would do that for any host here, I've lent a helping hand to a few people lurking the forums in times of need.

If you are speaking the truth here then that is beautiful stuff m8. I hope that you will be able to help GVH out and get their systems running again + get paid for all the effort you are putting in them but, just dont call out someone without solid proof, that will just cause another useless LET shitstorm. You know that it will not end up in anything good unless you are looking to destroy someone's reputation.

hostnoob

@raindog308 said:
Um, "Jaroslav" and "Jon" are the same person so the "move" was all in Jon's head.

lol, that made me think of this

http://www.imdb.com/title/tt0309698/

good movie.

Licensecart

raindog308 said: Yeah, the sames "sales manager" who is now restoring their web site & WHMCS.

You'd think a "sales manager" would be busy focusing on, well, sales, and leave this kind of technical operations work to, say, the new Interim Operations Manager.

Of course, in this case, they're all just pretend names Jon made up. But it does make you wonder...does Sales Manager Jon email Interim Operations Directory Jaroslav every hour?

Sales Manager Jon: "How's the server restore coming?"

(logs into different email account)

Interim Operations Director Jaroslav: "Hi Jon, it's taking a while."

(logs into a different email account)

Sales Manager Jon: "We're losing money here, buddy! Speed it up!"

(logs into different email account)

Interim Operations Director Jaroslav: "Look, your pressure isn't helping. You don't have to be a dick."

(logs into different email account)

Sales Manager Jon: "You're the moron who let this happen!"

(logs into different email account)

Tyler/Alan: "Guys, let's just focus on the problem."

You mate have made my day with that

Monsta_AU

Here come the emails - I have gotten two identical ones albeit with some markup changes (ooh, new guy has learnt to use bold! Look out!!):

Dear Valued Client,

This is a follow up email to the last email that we have sent earlier today regarding the recent security breach and proceedings.

SolusVM is now back online and accessible through the original URL https://solusvm.secureserverpanel.com. We have managed to conigure the settings so that clients will be required to perform a password reset before logging in (for security precautions).

Unfortunately as a result of untimely backup, data has been rolled back. If our VPS was hosted on one of the nodes that got renumbered recently, your SolusVM control panel will show incorrect IP addresses. Please use the IP addresses that you got after renumbering. We are working on a solution to update the SolusVM databases with the correct IP addresses as soon as possible, and is expected to be completed tonight (if we manage to get it done via automated script) or tomorrow (if we have to perform it manually). If you do NOT know your IP addresses, please submit a ticket to our technical support department.

We have reached the conclusion that none of our nodes have been touched during the SolusVM security breach. After examining the logs accessible to us, there were no modification commands made between SolusVM and the slave nodes, concluding that the hacker did not make the effort to compromise any of our nodes. Our nodes are safe. However just in case, additional security measures have been put into place and in addition, the kernel has been updated to the latest version as per the OpenVZ security patch that was released today.

TLDR; Basic summary of the entire situation

1. Our WHMCS and SolusVM systems were compromised, however NO data was stolen from WHMCS. Data (client/admin names, VPS IP addresses) may (although we suspect that it is unlikely due to the nature of the findings gathered) have been taken from SolusVM, however the necessary precautions and measures have been taken (such as forced password resets, and node password/auth key changing) to completely mitigate any further damage that could be done.

2. Our WHMCS client area support/billing system has been restored earlier today to 10 minutes before the breach. SolusVM has been restored to a point 10 hours before the breach. Our WHMCS installation was offline overnight during the process of fixing everything, and during this time there has been automated PayPal subscription payments sent to us that PayPal IPN could not send to our system. Clients who this is affecting will need to contact our Accounting department with their PayPal transaction ID # to have their invoice marked paid.

3. We have fully restored SolusVM to a fully usable state, however for a few nodes that have had their IP addresses renumbered, the incorrect IPv4 addresses are shown in the panel, however the correct IP addresses have already been assigned. We'll be updating the panel with the correct IP addresses, however if you do not remember the correct IPs, you will need to submit a ticket to technical support for us to provide you with the correct IPs.

4. Your sensitive data from our billing/support is confirmed to be 100% safe from falling into the wrong hands.

5. We've taken extra security precautions and measures to prevent something like from ever happening again, and backups of our systems will be made a lot more frequently (10 minute basis), with backups of VPS nodes being taken on a daily basis as well.

We still have more improvements to put into place. We're far from our goal of perfecting the company, however day-by-day we're improving things and doing all that we can to make your hosting experience better. This week and next week, we'll mainly be focusing on the consolidation of our U.S infrastructure into the ColoCrossing network. Clients will be informed over email regarding their virtual servers being consolidated to upgraded nodes (Buffalo NY first), with new IP addresses (these will stay permanent!) (We'll be sending out further updates regarding this as soon as possible!!)

Although we've tried to make this email as informative as possible, we understand that we may have left out things that our clients may still have questions about. If you still have any questions or concerns regarding anything, please feel free to contact us through our helpdesk and we'll be more than happy to clear up any confusion there may be and assist you any way possible.

Thank you everyone for being patient and understanding throughout this ordeal. We greatly appreciate it and we look forward to many more years of hosting yet to come!

Best Regards,

The GreenValueHost Team 

Meanwhile the LA4 node appears to still be down.....

EDIT: LA4 node back up now, my container is running. Now time to start planning backing up all the data and putting it elsewhere.

ricardo

I think they'll need to do better than that in regards to the IP renumbering.

I've found with almost all IP migrations, solusvm IP remains incorrect. GVH has renumbered before, so there's no obvious frame of reference for the correct IP of the servers at the moment. Logging in via console to find that out would be a pain in the @rse.

MSPNick

@Monsta_AU said:
Here come the emails - I have gotten two identical ones albeit with some markup changes (ooh, new guy has learnt to use bold! Look out!!):

Dear Valued Client,

This is a follow up email to the last email that we have sent earlier today regarding the recent security breach and proceedings.

SolusVM is now back online and accessible through the original URL https://solusvm.secureserverpanel.com. We have managed to conigure the settings so that clients will be required to perform a password reset before logging in (for security precautions).

Unfortunately as a result of untimely backup, data has been rolled back. If our VPS was hosted on one of the nodes that got renumbered recently, your SolusVM control panel will show incorrect IP addresses. Please use the IP addresses that you got after renumbering. We are working on a solution to update the SolusVM databases with the correct IP addresses as soon as possible, and is expected to be completed tonight (if we manage to get it done via automated script) or tomorrow (if we have to perform it manually). If you do NOT know your IP addresses, please submit a ticket to our technical support department.

We have reached the conclusion that none of our nodes have been touched during the SolusVM security breach. After examining the logs accessible to us, there were no modification commands made between SolusVM and the slave nodes, concluding that the hacker did not make the effort to compromise any of our nodes. Our nodes are safe. However just in case, additional security measures have been put into place and in addition, the kernel has been updated to the latest version as per the OpenVZ security patch that was released today.

TLDR; Basic summary of the entire situation

1. Our WHMCS and SolusVM systems were compromised, however NO data was stolen from WHMCS. Data (client/admin names, VPS IP addresses) may (although we suspect that it is unlikely due to the nature of the findings gathered) have been taken from SolusVM, however the necessary precautions and measures have been taken (such as forced password resets, and node password/auth key changing) to completely mitigate any further damage that could be done.

2. Our WHMCS client area support/billing system has been restored earlier today to 10 minutes before the breach. SolusVM has been restored to a point 10 hours before the breach. Our WHMCS installation was offline overnight during the process of fixing everything, and during this time there has been automated PayPal subscription payments sent to us that PayPal IPN could not send to our system. Clients who this is affecting will need to contact our Accounting department with their PayPal transaction ID # to have their invoice marked paid.

3. We have fully restored SolusVM to a fully usable state, however for a few nodes that have had their IP addresses renumbered, the incorrect IPv4 addresses are shown in the panel, however the correct IP addresses have already been assigned. We'll be updating the panel with the correct IP addresses, however if you do not remember the correct IPs, you will need to submit a ticket to technical support for us to provide you with the correct IPs.

4. Your sensitive data from our billing/support is confirmed to be 100% safe from falling into the wrong hands.

5. We've taken extra security precautions and measures to prevent something like from ever happening again, and backups of our systems will be made a lot more frequently (10 minute basis), with backups of VPS nodes being taken on a daily basis as well.

We still have more improvements to put into place. We're far from our goal of perfecting the company, however day-by-day we're improving things and doing all that we can to make your hosting experience better. This week and next week, we'll mainly be focusing on the consolidation of our U.S infrastructure into the ColoCrossing network. Clients will be informed over email regarding their virtual servers being consolidated to upgraded nodes (Buffalo NY first), with new IP addresses (these will stay permanent!) (We'll be sending out further updates regarding this as soon as possible!!)

Although we've tried to make this email as informative as possible, we understand that we may have left out things that our clients may still have questions about. If you still have any questions or concerns regarding anything, please feel free to contact us through our helpdesk and we'll be more than happy to clear up any confusion there may be and assist you any way possible.

Thank you everyone for being patient and understanding throughout this ordeal. We greatly appreciate it and we look forward to many more years of hosting yet to come!

Best Regards,

The GreenValueHost Team 

This email is yes informative but just so dam repetitive.

Pretty sure customers of gvh can find better vps elseware.

ricardo

Indeed. $2/m hosting... $5/m of time if you actually read these.

MSPNick

@ricardo said:
Indeed. $2/m hosting... $5/m of time if you actually read these.

You see. I've just seen some off there offers. Some plans offer 13ipv4. Must be having a heart attack at gvh.

ricardo

Well, at the kind of prices offered around here, IMO some are worth a look. Over the longer term though, the time-suck through all the various mishaps and nonsense do reduce that value greatly.

concerto49

This thread is getting huge as well.

Shoaib_A

@concerto49 said:
This thread is getting huge as well.

Why wouldn't it when GVH is the one of the biggest hosts on LET(surely not based on quality but based on number of clients they probably are) & considering the amount of drama they manage to create most of the threads about them would get huge unless admins close them.

Yes this is very unfortunate that data of their customers got compromised but they're humans & no matter how much precaution & security measures someone takes, things like this can happen to other hosts as well. Yes they provide plans which seem unsustainable, they lie & cheat as well but when a situation like this comes I think people should feel sympathy for them & try to help them. What happened tot hem can happen to other providers on LET as well. So please sympathize & help them in making sure that thing like this doesn't happen again instead of ridiculing & advising others to stay away from them just because the data got compromised. Surely i is a big thing but no one can claim his servers are 100% immune to such exploits.

gsrdgrdghd

K2Bytes said: So please sympathize & help them

I think Jonny's behaviour prevents many people from sympathizing with him

Nekki

gsrdgrdghd said: I think Jonny's behaviour prevents many people from sympathizing with him

If only it prevented many people from buying from him, we probably wouldn't be in this mess.

texteditor

Nekki said: If only it prevented many people from buying from him, we probably wouldn't be in this mess.

VPN

Offtopic, Nekki why ain't you moderator anymore?

Gunter

@VPN said:
Offtopic, Nekki why ain't you moderator anymore?

Wait what the hell? @Nekki was one of the only staff members operating on an honest pretense.

raza19

made many decisions in life but I couldn't be prouder of the one I made to cancel gvh within a week of subscription.

GVH was and always has been unreliable. Things like 100TB bandwidth and countless other unfathomable offers should have been an indicator of what was about to happen but sadly if you host with colocrossing everything is shoved under the carpet....

Nekki

@VPN said:
Offtopic, Nekki why ain't you moderator anymore?

No drama, I decided to call it a day. The last couple of days have not been enjoyable.

Gunter

Nekki said:

No drama, I decided to call it a day. The last couple of days have not been enjoyable.

So you resigned?

Gunter

It looks like GreenValueHost transitioned to Mandrill is at their hourly sending limit. I can't get a password reset email.

Nekki

Gunter said: So you resigned?

Correct.

concerto49

Gunter said: So you resigned?

Nekki said: Correct.

Where's the immediate press release thread?

Boltersdriveer

concerto49 said: Where's the immediate press release thread?

Immediate press releases about immediate press releases have been banned