Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


ChicagoVPS Update - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

ChicagoVPS Update

24567

Comments

  • He just copied from his own email to clients....

  • @MorningIris said:
    He just copied from his own email to clients....

    Is that an issue?

  • SpiritSpirit Member
    edited June 2013

    Those things demand time, I understand this but it's really annoying to read all those days how they suggest clean container reinstall to speed up proces instead restore from backups and yet they ignore those clean vps reinstall requests but constantly bomb us with more or less meaningless RFOs how many VPSs they restored from backup already.

    @CVPS_Chris why don't your guys get up after 5 days of downtime atleast those who requested clean VPS reinstall?

  • MaouniqueMaounique Host Rep, Veteran

    This page always gets unformatted to me. Seems the others are OK...

    Now, how can you know that no VPS data has been compromised if the database containing the passes has been leaked ?
    I think you cant.

  • @MrObvious why does it hurt you?
    Your love for cvps is obvious and it may hurt you, but nobody but you likes Chris.

  • ouijaouija Member

    What a shitshow of a recovery process. It's been 5 days of straight downtime, multiple support tickets unanswered for days, and I still have NO idea if my data has been lost or is recoverable.

    While I do appreciate all the effort they have put into getting their services back online, I cannot feel like they have a skeleton crew of people working for them given the amount of time this has taken.

    All I really want to know is if my data has been completely lost or if there is a chance of recovery. I have asked multiple times for a definitive answer to this, but have got nothing but the runaround and vague responses.

    I've already switched providers due to this ongoing downtime as I had no choice but to get my clientele back online and couldn't afford to wait around for CVPS' false promises and timelines to elapse.

  • SpiritSpirit Member

    @ouija not to mention that after 5 days of downtime they aren't capable to do even fresh VPS reinstall to speed up recovery process.

  • ouijaouija Member

    Well this is fantastic. Finally received a response and have been informed that it's going to be at least another 3-4 days (or let's say a week given CVPS' concept of time) before my data is at least attempted to be restored (if it can even be restored in the first place).

    Guess I'll just bite the bullet and begin rebuilding my databases from scratch :(

  • CVPS_ChrisCVPS_Chris Member, Patron Provider
    edited June 2013

    @Spirit said:
    ouija not to mention that after 5 days of downtime they aren't capable to do even fresh VPS reinstall to speed up recovery process.

    You have no idea the amount of work that went into getting everything back to where it is today. Any other provider damaged like this would take much longer, I can assure you that.

    Please keep your panties on. We are doing our best.

    @ouija said:
    Well this is fantastic. Finally received a response and have been informed that it's going to be at least another 3-4 days (or let's say a week given CVPS' concept of time) before my data is at least attempted to be restored (if it can even be restored in the first place).

    Guess I'll just bite the bullet and begin rebuilding my databases from scratch :(

    Where were you told this? It should not take that long

    All tickets are receiving fast personalized responses and that a tremendous effort is being dedicated to getting everyone back online and restored on request. Thanks everyone for your patience.

  • ouijaouija Member
    edited June 2013

    File restoration is possible, though must be done manually by our staff. Right now we only have 2 more nodes to fix, and once that process is completed we are going to begin working through the long list of requests for file restoration from our backups. That process will take 3-4 days before all requests have been fulfilled.

    My major issue is that I was in the process of a website redesign for one of my clients for a few weeks leading up to this fiasco, and had literally been working day and night on this right up until SolusVM was hacked, and unfortunately my most recent backup of this database is outdated by this week in which the majority of the work had been completed.

    I just want a definitive answer if my server data can be restored to a point where I can access this MySQL database and get the data I need. Otherwise, I will have to face the reality of needing to rebuild from scratch.. If it's going to take another 3-4 days to even find out if this is possible, then I might as well cut my losses here and now and write off your service entirely.

    Update: Just received another reply that restoration might be within 6 hours. That is quite an improvement over 3-4 days. Now let's just see if that holds up.

  • SpiritSpirit Member
    edited June 2013

    @CVPS_Chris I understand that things demand time but why lies? Your support guy keep repeating: "At this time virtually all customers are back online; some with all original files in tact and others with new containers.

    To clients who are offline 5 days already this answer don't really help to calm down, don't you think?
    Also you keep repeating over last few days how many clients data are restored and yet your team isn't capable to do explicitely requested fresh VPS reinstalls to speed up recovery process. Why is that? After 5 days of downtime? Because you work too hard to restore data and those who requested fresh VPS reinstall to start to work with VPS sooner can wait longer? I fail to see logic in that.

  • mikhomikho Member, Host Rep

    I keep thinking of Virpus and when they had 19 nodes wiped in 2011. They needed about a week to create new containers to their customers and all customers were given new containers. A restore always takes longer time then to create an empty new container. So I feel for all of you during this process.

  • CVPS_ChrisCVPS_Chris Member, Patron Provider

    We are actively restoring backups for customers who have requested it.......
    further updates to follow.

  • From FB:

    First, please allow me this opportunity to thank all of your for your incredible patience and understanding as we’ve worked through this very challenging scenario. We are absolutely dedicated to ensuring the happiness and recovery of every single impacted customer.
    
    
    
    Now that we’ve begun to get ahead of the support load I’m able to share more specific details on our current status. At this time all customer services should be online and available. A small subset of customers may still be offline as their containers will require further investigation and attention. A few nodes have been condensed/transferred/migrated so please do not be alerted if your node ID has changed. Direct access to SolusVM remains inactive as we wait for their internal and external security audits to be completed (as discussed here: http://www.lowendtalk.com/discussion/11327/solusvm-audit-update#latest). During the interim you are able to control your containers through our billing/support system and may request OS reloads via ticket.
    
    
    
    Initially the mitigation strategy called for all impacted nodes to be inspected for logs, reloaded, and then images from our backup database installed. This process proved far too timely and dramatically slowed down the recovery effort which is also partially to blame for our poor communication during the first 48 hours post-compromise. We altered the plan two days ago, deciding to instead install all impacted customer service from scratch; this revised strategy has allowed us to rapidly return to full online status and reduce pressure as we work diligently to recover data for all customers who require it. ChicagoVPS has two separate backup facilities, a free public facing system called Central Backup and a secondary backup, which automatically ran each week. Unfortunately a small group of nodes were not yet setup for the automatic/secondary system or the backups were corrupted. For customers on those nodes, if you ran a Central Backup your data is absolutely safe and you may request a restoration via ticket. Customers who were not on the nodes with corrupted secondary backups can request a restoration regardless of whether they utilized the free Central Backup feature. The restoration process currently has a 6-12 hour lead time once you’ve requested it.
    
    
    
    ChicagoVPS will be analyzing this event closely so we may implement refined plans to both protect against repeat issues and to ensure our communication and reaction strategies are improved. While SolusVM has released multiple updates in the past few days to fix vulnerabilities which allowed this event, and others, to occur we take full responsibility for our outage. We simply should have reacted more quickly, more effectively and provided better communication. We’ve already learned from this past week, and we will continue to learn.
    
    
    
    As we push towards final resolution on all fronts we will continue to update our customers via email, Facebook (https://www.facebook.com/chicagovps) and the “Low End Talk” message board (http://www.lowendtalk.com/discussion/11304/chicagovps-update#latest).
    
    
    
    Thank you again for your patience, loyalty and understand. We appreciate you. 
    
    Regards
    
    Chris Fabozzi
    Director of Operations
    ChicagoVPS
  • SetsuraSetsura Member
    edited June 2013

    @CVPS_Chris said:
    We are actively restoring backups for customers who have requested it.......
    further updates to follow.

    So as I was on an ATL node, I can assume that everything I had there was more or less lost? I want to ask if there is any compensation planned? I didn't really have anything important, but I sure don't feel like setting it all up again if there isn't any compensation planned for those who's data was destroyed.

    I appreciate your efforts, and I understand this was in many ways not exactly your fault.

  • ErikTErikT Member

    What a load of crap. I've been with ChicagoVPS for a long time. They lost my data in November and probably now too. I don't have access to my VPS unless you count a web control panel that says offline and has a "Boot" button that doesn't actually boot my vps. I have submitted several tickets in the past few days and today and never received a personal response. I'm sure ChicagoVPS makes a lot of money by not hiring a support staff. Great a marketing, just put out a bunch of BS saying how everyone is up and getting great support. Lies, lies, lies.

  • CVPS_ChrisCVPS_Chris Member, Patron Provider

    Erik,

    That is very untruthful as we have a 5 man support team, ALL American based. As for the control panel, most providers have it shut off because of the on going issues. I dont think anyone wants any chance of being vulnerable to a compromise.

    Both hacks both fall back and relate to the same vulnerability, if you dont believe me, you can search this forum and see how I was called a liar for the past 8 months.

    I ask that you have a little more understanding as I think myself and the support team got everything back up and running in a reasonable time considering the amount of work we completed.

  • DomainBopDomainBop Member
    edited June 2013

    All tickets are receiving fast personalized responses

    mine have received copy and paste replies

    The restoration process currently has a 6-12 hour lead time once you’ve requested it.

    I opened a ticket to requets it a couple of days ago (which is more than 6-12 hours). Instead of a restoration of a backup I got a new install with the wrong OS distro.

  • ErikTErikT Member

    But everything isn't normal as my VPS is still down. I've been down nearly a week and have received nothing but copy/paste canned responses to my support tickets so you ARE understaffed otherwise my vps would be up by now. Too many customers, not enough staff. Obvious to me. I guess this is acceptable to you since nobody really expects the 99.9% uptime that you guarantee for a low end vps.

    I'm aware of the issues with SolusVM and also I'm aware that many other providers haven't been down all week. Rumor has it the first time was caused by a weak brute forced password, and that this time you failed to delete a vulnerable file in time. I'm aware you deny these things, but you've lost credibility with me.

  • MaouniqueMaounique Host Rep, Veteran
    edited June 2013

    @ErikT said:
    But everything isn't normal as my VPS is still down. I've been down nearly a week and have received nothing but copy/paste canned responses to my support tickets so you ARE understaffed otherwise my vps would be up by now. Too many customers, not enough staff. Obvious to me. I guess this is acceptable to you since nobody really expects the 99.9% uptime that you guarantee for a low end vps.

    I'm aware of the issues with SolusVM and also I'm aware that many other providers haven't been down all week. Rumor has it the first time was caused by a weak brute forced password, and that this time you failed to delete a vulnerable file in time. I'm aware you deny these things, but you've lost credibility with me.

    Rumours are just rumours.
    One thing is certain, nobody in this market can have enough staff to make up for a general catastrophe like this.
    5-6 people should be enough to manage day to day operations even with so many customers.
    Nobody can have "reserve" staff for events like this, but I do agree that, even so, it took way too long so far.
    100 servers to restore with 5 people and 3-4 hours on a server (that much it took for ours when raid failed) a similar 32 gb ovz e3 would mean a couple of days, 3 at max, considering not all were wiped, considering those people need to eat (sleep might be skipped or done 1-2 hours a day for a couple of days), need to shower, etc, I would say 3 days is a very generous time in which it should have been expected to have all back up.
    After the second server you make a script, or know the drill so well that you can work on more at once.

    As for the staff being American based, what is that supposed to mean ? That Europe has bad technicians ? Can't have good support staff ? Doesnt know enough English ? Does this include South America, Carribean and central america ?
    If they are all American and this is such a great thing, then they dont need 3 days, will do the job in just two.

    Thanked by 1mpkossen
  • It's sad that the remaining downtime before my service is restored has to be measured in multiple DAYS.

  • @Maounique said:
    As for the staff being American based, what is that supposed to mean ?

    Most companies in America have/will advertise as "American" support -- telcos, utilities, cablecos, etc. because it plays into the "buy American" thing that used to be all the rage and because it's a way of saying that "our support isn't outsourced" -- I assume that's what he means here.

  • sundaymousesundaymouse Member
    edited June 2013

    I used to with them when they posted a $5 offer for 2048MB OpenVZ possibly last year. At first, the virtual machine and the network were really good in LA, then when more people take the deal, the oversale was a crap and there was an all day downtime later, due to "broken parts taken down". But that's only $5, so you should expect some downtime or even some data loss.

  • srichtersrichter Member
    edited June 2013

    @ihatetonyy said:
    because it's a way of saying that "our support isn't outsourced" -- I assume that's what he means here.

    But isn't their support "outsourced" since they had ColoCrossing help them out?

    Thanked by 1tux
  • FreekFreek Member

    @Jeffrey said:
    Oh, great. Now that ColoCrossing made it obvious that they own LET, CVPS_Chris comes back. I guess Chris can replace RobertClarke for us. :)

    I thought ColoCrossing said Chris was a good client of theirs? Sorry if I'm late to the party.

  • SpiritSpirit Member

    No need to start with that again. ChicagoVPS and their clients have this moment other worries. Lets keep this thread clean for downtime and data restore updates.

  • I got some personal replies at the very beginning. Now its all copy and paste crap about they are working hard to get to my request, blah blah blah. All I want to know is if my data was or was not lost. I do have a VPS now, with the wrong OS and cannot get it fixed. They restored it with the same root password that was leaked... SMART!

    Can I get CentOS 6 fresh on my VPS? Do you have my data or not? I don't feel like wasting anymore time. Since you have answered more customers on the public boards than in your own support system maybe I'll finally get some answers.

  • Every time I request a restore from Central Backup I get a canned response to use my own backups or wait 3-4 days. Then the ticket gets flagged as "answered" and nothing happens.

    Your post on facebook says this process should take 6-12 hours if used Central Backup. I did use the Central Backup facility for my VPS and want it restored. I have been asking this for days.

  • ouijaouija Member

    Chris,

    THANK YOU FOR REDEEMING YOURSELF IN MY EYES.

    My VPS has finally been restored, within the timeline that I was actually given to boot!

    Some of my data has been lost, however, but that was to be expected. But the data I was most concerned about seems to be unaffected, and I am relieved that I've (or should I say CVPS) managed to recover it!

    Thank you and your team for all your hard efforts over the past week. I know that a lot of your userbase (present company included) has been giving you a lot of grief that just adds to an already stressful and busy situation, and I want you to know that your efforts are TRULY appreciated and haven't gone unnoticed!

    Please, for the love of god, NEVER go back to using SolusVM (which seems to be the direction you are heading) to avoid a terrible scenario like this in the future, and hopefully all of your efforts won't go unrewarded! Really really appreciate this, and words cannot express.

    At lot of my frustrations were just with the mixed timelines and unknowing uncertainty if my data was compromised. I apologize for being impatient and adding to your levels of stress, and wish I could have waited longer for your support but was becoming frustrated with the ever increasing turn around time. I hope you understand, and wish you all the best in your remaining recovery process.

    Thanks again.

    Thanked by 1netomx
  • XTREMEXTREME Member

    to have a guarantee VPS the best offer is 4USD/month

Sign In or Register to comment.