Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


ChicagoVPS Update
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

ChicagoVPS Update

CVPS_ChrisCVPS_Chris Member, Patron Provider

Hey everyone, just wanted to give a personalized update here. We are making good progress and a new email went out just minutes ago, you can also see the contents of the email posted on FaceBook if you have not received it yet.

All tickets are being responded to with an update, and further response for info are being answered with personalized responses and you can expect to recieve more good news shortly.

Thank you everyone for all your patience, it is really appreciated.

Chris

Thanked by 2mpkossen uhu
«134567

Comments

  • Thanks for the update Chris and best of luck in getting everything back to normal. Keep up the good work!

  • Good luck w/ everything Chris. Glad to hear you guys are making progress.

  • jcalebjcaleb Member

    When everything is up, create a good offer for celebration....

  • @jcaleb said:
    When everything is up, create a good offer for celebration....

    Yap, 1USD for 2giga RAM.

    Thanked by 2jcaleb Inglar
  • @SayangAlif said:
    Yap, 1USD for 2giga RAM.

    And first 6 months free!

  • blackblack Member

    Sorry this happened Chris and good luck with everything. Your services have been great before this incident.

  • CVPS_ChrisCVPS_Chris Member, Patron Provider

    Further good progress has been made and we are down to the final list of machines that were affected. More than 50% that were affected have been fully restored with files intact and the remaining list that needs more attention is what is left and we are working very hard to restore them fully with files intact.

    Thanks for all your patience!

    Thanked by 1Coffee
  • Good luck and hopefully a little bit of free time to rest.

  • DotMGDotMG Member

    @jcaleb said:
    When everything is up, create a good offer for celebration....

    Not a very good idea IMHO. Good offer => More customers => More work => Lesser quality. Better focus on satisfying existing customers and improving quality, on looking good alternative for Solus, on finding ways to suck Solus ... If you wish to make some celebration, set up some sort of lottery where only one or couple lucky winners will get 4 to 8 GB RAM for free for one or two years. My humble two cents.

  • jcalebjcaleb Member

    @DotMG said:

    => Lesser quality. Better focus on satisfying existing customers and improving quality

    =)

  • CVPS_ChrisCVPS_Chris Member, Patron Provider

    Hey everyone, just a quick update. Since our last email, all nodes that were still affected have since been reinstalled. Right now we are working on installing the new VPS for each server and we are still making good progress.

    When we reach final completion we will release another update.

    Thank you all for your patience.

    Regards

    Chris

  • @CVPS_Chris This doesn't really give a good explanation why every single time estimate given has been broken. The last email I got, which was about 16 hours ago, said that all VPSes would be installed within 6 hours and all data restored within 18. Since it seems that you're at least 10 hours behind schedule as of now, how long will it take to get data restored?

  • Oh, great. Now that ColoCrossing made it obvious that they own LET, @CVPS_Chris comes back. I guess Chris can replace @RobertClarke for us. :)

  • Awmusic12635Awmusic12635 Member, Host Rep

    @Jeffrey I don't really see it being a bad thing as bit of people here have servers with CVPS and keeping them updated via email and here is only a good thing.

  • @Fliphost that is true. :)

  • @Fliphost would be nice if the updates contained time estimates that were better than throwing darts blindfolded.

  • Awmusic12635Awmusic12635 Member, Host Rep

    @Wolframium Things like this take time.

    When you are restoring such a large number of servers things are bound to go wrong and can end up taking longer than expected.

    In a perfect world none of that would happen, but it is not perfect

  • @Fliphost I suppose I don't really mind too much, but I'd rather that if they give definitive estimates they are at least reasonably accurate. I know that things happen and especially very early estimates can have a large degree of variability if they don't know yet exactly what happened and exactly how to fix it, but considering its already three days after the initial issues, I would have expected a more fixed and workable time frame.

    On another note, maybe a next step after all this is over would be creating some sort of automated backup restore so if something like this ever happens again the response can be accomplished a lot quicker.

  • Awmusic12635Awmusic12635 Member, Host Rep

    @wolframium

    The way I understand it, ( I could be wrong about this) but the nodes themselves were wiped, not just the VPSs on the nodes.

    This causes more issues with restoring the servers. Most backups only backup hte specific vps and not the node as a whole as that is generally not needed and resquires more space.

    To what you are suggesting would require the whole node backup, most likely via BMR.

    While I do this with Fliphost it is not always practical in all situations.

    I do agree though it will be interesting to see what they do to help either fix or prevent this in the future.

  • @Fliphost I probably don't understand the software enough to know what's really possible and what isn't, but if each individual VPS is backed up, shouldn't it be possible to just restore them all via a script or something? Some of them might be missed or use the wrong backup, but making manual restores the exception rather than the rule would speed things up a lot.

    I agree, I'm very interested in how they prevent this in the future, and it would be very interesting if they could create any sort of automated restore system (though it would be nice if this was coupled with automated backups.)

  • Awmusic12635Awmusic12635 Member, Host Rep

    @Wolframium well if the node is wiped and reinstalled it doesn't have the same node ID so restoring the backup to the right node causes issues. It can be a bit annoying

  • @Fliphost Ah okay, I suspected based on the fact that no one had done it that there was something making it harder than I thought in my mind, thanks for the explanation.

  • AlexBarakovAlexBarakov Patron Provider, Veteran

    @Fliphost said:
    Wolframium well if the node is wiped and reinstalled it doesn't have the same node ID so restoring the backup to the right node causes issues. It can be a bit annoying

    Those things can be easily changed (nodeid, key and etc.). Just keep a backup of the conf file for each node. In case of a complete failure/wipe/root and etc. , install solusvm, replace the conf files properly and initiate a restore script to restore each of the tar.gz backups of the individual VMs from the remote location they are stored in. Should take literally minutes to get it working (excluding the actual restore, as it could be TBs of data). Each container will get provisioned with same ID, the node will have same ID, pretty much will be absolutely the same.

  • MaouniqueMaounique Host Rep, Veteran

    We all had to do it some time, mainly because hardware failure or something.

  • Awmusic12635Awmusic12635 Member, Host Rep

    @Alex_LiquidHost

    Thanks for the heads up. Though I have each node BMR backed up.

  • AlexBarakovAlexBarakov Patron Provider, Veteran

    @Fliphost said:
    Alex_LiquidHost

    Thanks for the heads up. Though I have each node BMR backed up.

    You're welcome.

    I haven't looked at bare metal restores. Reading about them now, sounds interesting, could be a time saver as well.

  • MaouniqueMaounique Host Rep, Veteran

    It depends.
    We were thinking to use CDP but the data changing on an OVZ node would be too much for it, for instance.

  • Awmusic12635Awmusic12635 Member, Host Rep

    @Maounique check out bacula4hosts

    One license for whole node. Individual VPS restores.

    Gzip compression
    Incremental BMR

    It does quite a good job

  • CVPS_ChrisCVPS_Chris Member, Patron Provider

    With the recent SolusVM exploits that have affected our company and others with a negative impact, many of our customers and us are not supportive of enabling public facing access to our SolusVM VPS CP as additional code could be exploitable. Let's not take a risk when it comes to security. At this time, we are releasing an alternative frontend solution to our customers to allow them to reboot, start, shut down, serial console, change root pass, or change hostname on their VPS. We hope to be making this more feature rich soon, however at the moment the only thing that you CANNOT do with this new frontend is: reinstall VPS, manage DNS entries, or create central backup. We are working on making these features available to you ASAP.

    You can now access your virtual server controls at https://billing.chicagovps.net/clientarea.php?action=products . Select the service, and under the "Virtual Server Control" section you can manage multiple aspects of your VPS, including reboot, start, shut down, serial console, change root password, or change hostname.

    No client's VPS data was leaked or accessed by a 3rd party during this hack. The hacker(s) did not directly access any VPS container or hypervisor, and simply used a SolusVM exploit to wipe out and cause damage to a certain number of VPS nodes. The intentions of the malicious hackers was cause mayhem within our company by wiping some of our servers. With this compromise, our SolusVM database was accessed by a third party. As such, there is a possibility that any passwords that were related with SolusVM could be at risk, for example your initial password you signed up with. For those clients VPS's that are now accessible and showing as an online state in the virtual server controls section in our client area, we urge that you immediately change your root password by clicking on the "Change Root Password" button.

    Let it be clear that this compromise did not impact our client area in anyway, so any billing information, etc stored in our client area at billing.chicagovps.net is safe.

    For good measure, please take a minute to change your client area password. Those who used the same SolusVM password as the client area should do this promptly. https://billing.chicagovps.net/clientarea.php?action=changepw

    On a related note, rest assured we're making great progress in our recovery. A further update regarding this matter will be sent out later today.

    We thank our customers for their continued support during this ordeal.

    Regards,

    Chris

  • SpiritSpirit Member
    edited June 2013

    @CVPS_Chris when can we realisticy see all OpenVZ VPSs reinstalled and ready for clients to restore their own data?
    (I am not asking about recovery from your backups but simple fresh VPS container recreation of those who are still down)

Sign In or Register to comment.