Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Categories

In this Discussion

› › ChicagoVPS hacked - Page 9
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

ChicagoVPS hacked

1…6791112

Comments

  • DomainBopDomainBop Member
    edited June 2013

    This was posted on VPSBoard, from a ticket

    Just this at 2:26 AM PST in a ticket:
 
Quote
SolusVM was hacked, and a user started deleting data. We are not sure what the total overall damage is yet.

If you are offline, its because the data was deleted, not that we turned them off. If you are in any location other than Atlanta, we have backups

Regards

---------------
Chris Fabozzi
CEO / Director of Operations
  • lennierb5lennierb5 Member

    Any idea on what data these backups contain and when they were taken? Do they have all the data of the VPS on them or only configuration?

    Just trying to figure out what I need to prepare for restoring.

  • mpkossenmpkossen Member

    Cloudmin is really undervalued.

  • blackblack Member

    @seriesn said:
    Those who are yelling opensource/hypervm blah blah.

    http://www.itwire.com/business-it-news/security/25559-hypervm-boss-hangs-himself-after-exploit-damages-100000-websites

    Everyone and everything has issue. What makes you better is how you handle it. Look at whmcs, they had some intense hack in the past, wiping out providers such as K-disk. But they are still going strong.

    That's how serious he is about his project.

  • fapvpsfapvps Member

    @mpkossen said:
    Cloudmin is really undervalued.

    That is an understatement of the week!

  • PcJamesyPcJamesy Member

    @lennierb5 said:
    Any idea on what data these backups contain and when they were taken? Do they have all the data of the VPS on them or only configuration?

    Just trying to figure out what I need to prepare for restoring.

    If i remember right from the last hack in November you had to restore form the backup made from the solusvm panel backup option. But you only had backups if you manually clicked the create backup button. I don;t know if they had any scheduled vps backups. Based off the pingdom reports it looks like their backup servers didn't go offline.

  • CVPS_ChrisCVPS_Chris Member, Patron Provider

    @PcJamesy said:
    If i remember right from the last hack in November you had to restore form the backup made from the solusvm panel backup option. But you only had backups if you manually clicked the create backup button. I don;t know if they had any scheduled vps backups. Based off the pingdom reports it looks like their backup servers didn't go offline.

    Since the hack we implemented offsite backups of the entire VPS. IF there is data loss we can restore it to the latest backup.

  • chronos511chronos511 Member

    @CVPS_Chris said:
    Since the hack we implemented offsite backups of the entire VPS. IF there is data loss we can restore it to the latest backup.

    Can you give us an idea of when the most recent backup was? Thankfully I didn't have anything mission critical on the VPS yet but I imagine there are others who are in a serious panic.

  • csauvecsauve Member
    edited June 2013

    @chronos511 Their offsite backups are made every Thursday.

    edit: I should say that when I requested a restore a month ago from offsite backup it was from the previous Thurs. I don't know if they stagger them for different nodes or what.

  • DomainBopDomainBop Member

    update #2 just emailed:

    Chicagov VPS June 18th RFO Report #2

    This is a further status update to the recent security breach that ChicagoVPS has experienced. We have successfully restored some nodes, and the vast majority of our VPS customers are online, however we have a small percentage of nodes which still need to be worked on. Some of the nodes we are working on had data loss that we cannot restore. These nodes are LA18, ATL1, ATL4, ATL5. I you on are on one of these nodes you can safely start to rebuild, or open a ticket asking for this months refund.

    On a positive note, it seems like SolusVM has released a new update in light of the recent incidents ( http://blog.soluslabs.com/2013/06/19/security-updates-available-for-all-solusvm-versions/ ). However, at this time we do not feel comfortable enabling SolusVM access at this minute as we were a victim of their security vulnerabilities two times in the past 7 months. We are evaluating other alternative panels, but at the moment our priority is taking care of our customers and getting the impacted nodes back in working order.

    Please understand that we have all hands on deck working tirelessly to restore service connectivity for those impacted. Therefore, our ticket response times are affected to allow us to effectively work without anything slowing us down. We apologize for the delayed ticket response times but we assure you we are making progress and working hard to get everything back to normal.

    Our goal is to have everything 100% restored tomorrow. Those affected by this incident will recieve compensation.

    Regards,

    ChicagoVPS Team

  • blackblack Member

    My VPS in chicago is still down, I'll wait another 24 hours before I bug them with a ticket.

  • nikcubnikcub Member

    One of my servers is still down and looks to be lost. Ironically it is the node that I had moved after my outage incident a few weeks ago:

    http://www.lowendtalk.com/discussion/10760/chicagovps-outage#latest

    The server that I kept on the node stayed up. There was one database that I wasn't backing up since I renamed it - urgh.

    I spent the last 10 days preparing to switch all reliance completely off CVPS but even with all that preparation I still lost some data.

  • nikcubnikcub Member

    @CVPS_Chris

    You guys have to realize that WHMCS can only send emails so quick. With over 10,000 emails sent out it takes hours.

    sounds like a good time to signup for an email provider like Mailchimp or Amazon SES

  • nikcubnikcub Member

    @mpkossen

    Besides, mysqli_real_escape_string was a concept unknown to the person that wrote that code.

    mysqli_real_escape_string on its own is also not enough to stop exploits. there are plenty of ways and places you can inject SQL or XSS that don't rely on an apostrophe, quote mark, carriage return, null character or a backslash (mysql syntax is very flexible).

    the most common culprit is uncase integer field, for eg.

    $sql = "select username, password from users where id = " . mysql_real_escape_string($_GET['id']);

    Old school injections won't work, but something like:

    1 and 1=0 union select null,password,null from users limit 1,1-- -

    will. you really need to cast and in string filter out anything that isn't A-Za-z0-9 with regular expressions or the filter_var functions. developers almost always slip up, especially with more complex fields such as emails (nobody gets validating emails correct). unicode opens up an entire pandoras box. you need to bind parameters (all the db libraries support it).

    it is a real mistake to create a function like safe_var() that just escapes ' " etc. not all variables are the same and you can't have one filter function that fits all. there is a very popular commercial software application used by VPS companies that makes this mistake.

  • dgprasetyadgprasetya Member

    oh no.., now ChicagoVPS..

    @nikcub is it blind sql injection ?

  • nikcubnikcub Member

    @dgprasetya

    yep, in most cases you blind, double blind, timing or error-based to extrapolate the schema and then exploit by inserting a user record or updating an existing. for eg. you can usually update permission levels and tokens on an existing user using blind queries.

    it is more complicated, but there are tools like sqlmap that automate it. you'd be surprised at what type of data you can get even without there being an error page or any data output.

  • EvoEvo Member

    @Nick_A,

    Unfortunately sales are with higher priority than security.

    It's good to see that they started a "full audit" of the code ... ...after a few years of negligence, so:

    "Thank you for your patience and continued support."

  • BlackKnightBlackKnight Member

    My Customers threat with lawsuit. ChicagoVPS cant answer the tickets.

    When system up and running ?

    I need net time for system up and running!

  • nikcubnikcub Member

    These nodes are LA18, ATL1, ATL4, ATL5.

    I can't work out if I am on LA18. The details panel in the control panel simply says 'unavailable. anybody have a clue how I can find out which node a server is on?

  • nikcubnikcub Member

    @dnwk

    Is it possible to run a reverse proxy in front of Solus and do some escape for them?

    If somebody wants to send me the source, i'd be happy to build something like this. either as a separate server or a new front controller that parses all input through sane rules.

  • BlackKnightBlackKnight Member

    how i can find im which node ?

  • mpkossenmpkossen Member

    @nikcub: indeed, mysqli_real_escape_string on its own is not enough. However, SolusVM does escape passwords to some extent, by using single quotes around the variable. If you combine this with mysqli_real_escape_string, you prevent breaking out of it, AFAIK.

    How solus does it:

    $s = "SELECT * FROM table WHERE id = '" . $id. "';";

    What I would recommend if PDO is no option:

    $s = "SELECT * FROM table WHERE id = '" . mysqli_real_escape_string($id) . "';";

    Anyway, I shudder from both the above. I'm all for PDO.

  • RalliasRallias Member

    @BlackKnight said: My Customers threat with lawsuit.

    Call their bluff. They'll never file lawsuit.

  • erhwegesrgsrerhwegesrgsr Member
    edited June 2013

    I'm still down lol, in LA, node 18, who's up?

    [18-6-2013 21:33:08] Chris: 2 hours max maybe?

    (skype, One hour ahead of UK)

    Maybe he did get everyone else up?

  • BlackKnightBlackKnight Member
    edited June 2013

    what is chris skype address ?

  • lennierb5lennierb5 Member

    As of now, it looks like we have 10 nodes back online and 10 left to go.

    Any updates from those who have been restored as to what date the backup restored was?

  • liquidiceliquidice Member

    @CVPS_Chris Since the hack we implemented offsite backups of the entire VPS. IF there is data loss we can restore it to the latest backup.

    So this statement and the email sent this morning contradict each other. My VPS is in Atlanta - can you restore from the latest backup I made or did I lose data?

  • liquidiceliquidice Member

    Just saw this notice posted on chicagovps's website - I guess this means my data from the offsite central backup is still there, I just have to open a ticket and wait for it to be restored? If so - this is good news, I don't mind waiting - I just want to know my data is still there.

    After working all night and making progress that was unexpected and not to our liking, we have decided to change our process of getting everyone online. At this point, restoring the VPS' from backups is too time consuming and with our man power will just simply take too long.

    Our new plan is to give everyone a fresh VPS to work with. There have been many of tickets saying that our clients just want a VPS to work with and will restore them themselves. This does not mean we cannot restore your VPS, but we will require you to open a ticket and then we can help you individually. We expect this to really cut down on the downtime and find a medium where everyone is happy or as happy as then could be in this situation.

    We really value your patience and once again apologize for what has happened the past 24+ hours. Once this is all cleared up we take even more precautions and higher security so this will never happen again, along with finding a new Control Panel.

    Regards

    The ChicagoVPS Team

  • akzakz Member

    with light of all the recent lies and lack of communication from cvps, i think its time customers begin to demand refunds.. but I doubt this process will go smoothly..

  • lennierb5lennierb5 Member

    Still wondering what the holdup is, the 10 servers that were down earlier are still down even though they aren't restoring from backups?

Sign In or Register to comment.