Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


ChicagoVPS Update - Page 3
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

ChicagoVPS Update

13567

Comments

  • @CVPS_Chris said:
    You have no idea the amount of work that went into getting everything back to where it is today. Any other provider damaged like this would take much longer, I can assure you that.

    >

    Please keep your panties on. We are doing our best.

    You're right. I have no idea how much work you and your company have done. Really, I don't care. I would have preferred if your node provisioning process were automated and that it took one man two hours to restore your infrastructure. I'm sure you would have too. I would have preferred that you had a fail-over, disaster recovery, or scripted recovery plan in place that would have made this recovery easier for you. I'm sure you would have too.

    But that doesn't matter to me, because I pay you to handle the infrastructure portion of my hosting. I pay you for a service, that the amount I pay you is small doesn't matter, because you set your prices. You do this by overselling your nodes. I'm ok with that. I'm fine with the occasional unavailability of memory, cpu time, or io throughput. I compile my services to use minimal resources anyway. That's my choice, that is what is under my control and what I consider my end of the bargain. Unfortunately, over the past five days you have failed to hold up your end of the bargain.

    As for your claim that any other provider damaged like this would take longer to recover, I call bullshit. First, I've worked as a data center analyst in three enterprise level datacenters with thousands (not just hundreds) of nodes. I've experienced a critical failure of multiple air handlers that caused a quarter of infrastructure to overheat and crash, corrupting hundreds of random servers, many of them providing critical resources such as provisioning, sccm, data cabinets, dns, etc. We suffered zero downtime of customer facing resources from this outage. Zero downtime. It took us approximately eight hours to restore the resources, although our raid checks over the following two weeks were inundated with failed drive replacements. Second, YOU HAVE NOT RECOVERED YET. So to say that any other provider would take longer is bollucks, as you are still recovering and have no idea how long it will take you.

    The sad part is that I'm perfectly understanding about the downtime. I'm only marginally inconvenienced. It's the overpromising and the underdelivering that is getting to me. It's the half-assed PR. I don't care about personalized ticket responses that tell me nothing. In fact, they only piss me off since I feel your five man team could be working on node recovery instead of ticket response. It seems like a waste of resources to me.

    All I want from a provider is honesty. If you had said, "We were hacked. We're currently rebuilding all of our nodes. We're not sure how long it will take to restore our services, but we'll update you every couple of hours with our current status." I would have been happy. Instead I get gaslighted. Keep my panties on? How immature. Do the job you're being paid for and stop pretending that you're clever. It's very obvious that you're not.

  • FreekFreek Member

    @Spirit said:
    No need to start with that again. ChicagoVPS and their clients have this moment other worries. Lets keep this thread clean for downtime and data restore updates.

    I am not starting anything, I'm just asking a fairly simple question. I would like to know if my data is being handled by their techs or by their 'clients'.
    If they hadn't lied in the first place, I wouldn't have to ask this question anyway.

  • uhuuhu Member

    Someone seems to be confusing "enterprise class" with a LEB provider. For the latter, they provide a box with some resources, you decide how much money you want to throw at backups, geographical distribution, platform resilience, etc

    @slashtact said:
    It's the overpromising and the underdelivering that is getting to me.

    However, this I agree with.

  • slashtactslashtact Member
    edited June 2013

    @uhu said:
    Someone seems to be confusing "enterprise class" with a LEB provider.

    You're right. However, puppet is foss, not difficult to learn, and makes infrastructure management a breeze. There's no excuse for any hosting provider to not implement some version of a scripted provisioning process.

    My point with the enterprise anecdote was to point out that five days down time for a hosting provider is uncommonly long. Chris's attempts to claim that rebuilding an infrastructure of less than 100 nodes would normally take longer belie an inherent lack of technological expertise on his part.

    Effectively, he's trying to make the crappy service he is providing look like some sort of gold standard. It's not. I understand the incredibly stressful task he and his employees are faced with, and that it only gets more stressful as time passes. However, good effort is not commensurate with good service.

  • slashtactslashtact Member
    edited June 2013

    Thanks for the provision, Chris! It's too bad it didn't work.

  • Considering my VPS didn't suffer any data loss directly from the exploit and was up for ~2 days after it with no data loss, I'm still struggling to understand why it has been taken offline presumably by CVPS and still down 3 days later.

  • @ProxyChain

    Wow, at least my VPS is up. I guess I should consider myself lucky.

  • ouijaouija Member

    I did lose 3 sites with corrupted data after the recovery process, however my major concern was with a single database in particular that I hadn't backed up for about a week prior to the SolusVM exploit and THANKFULLY I was able to recover the data for this after they restored my server.

    At first, I was more angry with myself for not backing this up manually sooner (I had a weekly backup running, but as my luck goes, I did a ton of work the week leading up to the hack and it didn't get backed up in time) then I was just getting frustrated and annoyed with the long wait times and runaround in getting a solid response if the data would be recoverable and how soon this would take (not to mention catching heat myself from various clients that I host and develop for as well).

    In the end, I'm very lucky to have recovered this data and I understand that there are a ton of people in the same predicament as me, and I appreciate the efforts put forward by the team at CVPS, and hope they will learn from this experience and invest in a better disaster recovery plan to minimize downtime and impact. :)

    When it's all said and done, I hope the workers of CVPS take a week off to get some much needed rest and relaxation. I know first hand the pressures of working around the clock in this type of scenario (being a system admin myself) and know the toll it can take on your mental and physical well being..

    Thanks again guys, and all my best to those of you who are still dealing with this terrible situation.

  • @ouija said:

    When it's all said and done, I hope the workers of CVPS take a week off to get some much needed rest and relaxation. I know first hand the pressures of working around the clock in this type of scenario (being a system admin myself) and know the toll it can take on your mental and physical well being..

    Thanks again guys, and all my best to those of you who are still dealing with this terrible situation.

    Take a week off after they get everyone else online! :) I'm still waiting for my data to be restored. My VPS is up, but doesn't really do anything and unfortunately my 'backups' were via the CVPS Central Backup Service.

  • @CVPS_Chris - My ticket has been in "Restoration" status since Sunday at midnight - it has been more than the advertised ~6-8 hours for restoration of my data. Can you give me a status of my restoration? My ticket # is #KPT-413866. Thanks much for your help!

  • I feel like I'm going nuts ... my initial ticket submission (which is btw my 3rd or 4th ticket asking for file restoration, I closed the earlier ones so I could submit 1 clean ticket with less questions and just a simple request asking for file restoration):

    My VPS is now operational but my data is gone. I had my data backed up via Central Backup and would like it to be restored.
    If you need the root password to restore, let me know.

    Initial reply is a canned message about restoration and asking that I please restore from my own backup if possible.

    My reply:

    I do not have my own backup so please restore my VPS from my central backup that I made. Thanks!

    Their reply:

    No problem sir.

    Then almost 24 hours later (22 hours to be precise):

    Unfortunately backups for your container from our master backup repository are not available. If you utilized our free Central Backup feature to create a restore point for your service we can restore from that backup. If you did not utilize that free service we do not have backups and will be unable to restore any of your data.

    I said twice in my ticket already that I have a Central Backup that I made and I want restored. What the heck.

    I replied a minute later that I do have a Central Backup and please restore it - and I bet the ticket will go unnoticed again for another 24 hours - all because someone can't read the ticket for 2 seconds and notice that I said I made a central backup.

    I'm trying to be patient and understanding - but c'mon! The responses I've been getting are clear that nobody is reading the ticket.

    I mean - the title of my ticket is : "Please restore my central backup to my VPS"!

    Sorry - just frustrated as I'm sure many CVPS customers are.

  • oddcopteroddcopter Member
    edited June 2013

    @liquidice

    I have had the same issue. Has anyone had a successful restore done from their Central Backup?

    My current ticket is flagged "restoration" and they said it would be done by the end of the day. Fingers crossed.

  • @oddcopter Nobody will have, realize it, they, CVPS, don't care about you.
    they have so many customers that they can lose 10 here, 10 there, and will come with some cheap plan again and get more poor clients over, didn't you notice by now?

  • oddcopteroddcopter Member
    edited June 2013

    @MorningIris

    I have been very pleased with the service for the price but have learned I need to be more vigilant with my own local backups and have plans to make a quick temporary move if something like this happens.

  • One of my two VPS in Buffalo is still down.

  • A friend's $12/yr VPS has been brought back up without data loss.. why do clients such as me (paying for 3x Xen services) not get some sort of priority?

  • mikhomikho Member, Host Rep

    Because every customer is equally important?
    Probably because they are going node by node instead of "jumping" around.

  • Thank you very much Matthew and Jacob for getting my VPS back and restored.

  • @oddcopter said:
    Thank you very much Matthew and Jacob for getting my VPS back and restored.

    So CVPS was able to restore your VPS from Central Backup?

    Can you tell me - did you do anything other than put in a ticket?

    I'm still waiting for my data to be restored from Central Backup.

    As I suspected - I replied 1 minute after Luc put in a response asking if I had a Central Backup (which angers me since EVERY single response I put into the ticket including the title of the ticket and the initial submission says restore my Central Backup) -- and nobody has responded to my ticket since replying 1 minute after 10pm EST yesterday. Maybe folks went and got some well deserved sleep - but I really hope they are taking care of my restoration today.

  • Still waiting... quicker ticket response times my arse. I told Chris from the beginning that I had no problem waiting, I have a very small site and some testing domains for projects that I'd have to restore. But, if they have a backup I didn't want to have to go back in and try to remember all of the items I did to harden and secure my VPS. I have been beyond the restore request for DAYS now, all I want is CentOS 6. Period. CVPS obviously cannot handle more than that and I guess I got what I paid for. But damn, how long does it take to reimage a VPS? I am asking for this since I already lost my VPS I might as well upgrade and in addition to that since the root password that was used to recreate the VPS after this last attack was the same one that was leaked I do not feel confident that something is not already on there lurking. Give me a fresh image and a random root password... it takes 5 mins tops.

  • smccloudsmccloud Member
    edited June 2013

    I wish we still had the ability to do the following. But no, the Virtual Server Control section is gone. And leave it to me to not check before I started my firewall edits (that I messed up).

    You can now access your virtual server controls at https://billing.chicagovps.net/clientarea.php?action=products . Select the service, and under the "Virtual Server Control" section you can manage multiple aspects of your VPS, including reboot, start, shut down, serial console, change root password, or change hostname.

  • emgemg Veteran

    In addition to offsite backups, it might be a good idea to keep notes as you make configuration changes to your VPS. That way, you can remember how you hardened your VPS, and it may ease troubleshooting when you need to look back at recent changes or updates. I keep a "configuration log" on my home computer for each VPS, recording updates, changes, etc.

  • @Jack said:
    It would of been removed yesterday after an exploit.

    But no update to tell anyone. Not to happy with the "communication" from ChicagoVPS from this. And how am I supposed to get a serial console now?

  • My VPS has also not been restored and I have not got any replies from my tickets over the last couple of days either. Before that they said "we'll take a look", which really didn't sound so promising. At this point I think I would rather just have my money back. No matter how cheap something is, if it does not work then it is not worth anything. I did not expect it to be up an hour after it crashed but this long is ridiculous.

  • smccloudsmccloud Member
    edited June 2013

    @Jack said:
    When Chris gets on I'm sure he will update the module that was released 2 hrs ago.

    >

    Hope so. I'm job hunting so having my personal domain down doesn't look all that good.

  • nikcubnikcub Member
    edited June 2013

    @CVPS_Chris said:
    We are actively restoring backups for customers who have requested it.......
    further updates to follow.

    I requested a backup restore and my ticket was closed with a pro forma response stating "All servers will be back online in 24-36 hours .. We appreciate your support" etc.

    Just re-opened it because I was hoping to recover a snapshot from the server (I can live without it, would just prefer to have it).

  • akzakz Member

    @CVPS_Chris when do you guys think you will have a way to auto re install OS again?

  • DomainBopDomainBop Member
    edited June 2013

    Both of my VPS's that were down are now up.

    Buffalo: all files intact.

    Los Angeles: restored from one of their "weekly backups". Based on the last login date shown on the restored VPS I think CVPS might want to change their marketing material from "weekly node backups" to "quarterly node backups" because the restored backup is almost 3 months old.

    Last login: Sun Mar 31 17:01:14 2013

  • No emails or updates from CVPS today - no posts from @CVPS_Chris today -

    My ticket still hasn't been touched since the horrible response last night at 10pm.

    Maybe they all decided to take the day off.

  • @liquidice said:
    No emails or updates from CVPS today - no posts from CVPS_Chris today -

    My ticket still hasn't been touched since the horrible response last night at 10pm.

    Maybe they all decided to take the day off.

    I got a reply from Chris & Mark on my ticket. Still trying to figure out a way to resolve my firewall issue since there is currently no Serial Console access.

Sign In or Register to comment.