Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


What's up with Verelox being down? - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

What's up with Verelox being down?

24

Comments

  • ihadpihadp Member

    @Tom said:

    @Verelox said:

    @rajprakash said:

    Verelox said: online next hour

    Online with customer data? Or just online with fresh installs?

    All servers that are coming online have customer data.
    Servers that are wiped will not come online right now, we will e-mail those clients with more information.

    @Tom said:
    @Verelox best of luck. Did you have backups & the ex-admin wiped these too?

    We have a backup of the customer database. Right now we are using the information in this backup to bring up all dedicated servers and cloud servers.
    We are'nt sure yet if we are missing any data in our backup.

    Good luck. Why did dedicated servers come down anyway? Did you take your whole infra down?

    Taking down the network isn't unusual or unwarranted when you have an attack on your infra and having trouble locating the source.

    Pull the plug on your border routers, cut off all public Access and begin diagnostics using your out of band network.

  • @ihadp said:

    @Tom said:

    @Verelox said:

    @rajprakash said:

    Verelox said: online next hour

    Online with customer data? Or just online with fresh installs?

    All servers that are coming online have customer data.
    Servers that are wiped will not come online right now, we will e-mail those clients with more information.

    @Tom said:
    @Verelox best of luck. Did you have backups & the ex-admin wiped these too?

    We have a backup of the customer database. Right now we are using the information in this backup to bring up all dedicated servers and cloud servers.
    We are'nt sure yet if we are missing any data in our backup.

    Good luck. Why did dedicated servers come down anyway? Did you take your whole infra down?

    Taking down the network isn't unusual or unwarranted when you have an attack on your infra and having trouble locating the source.

    Pull the plug on your border routers, cut off all public Access and begin diagnostics using your out of band network.

    This admin installed in our servers backdours when we add the servers online it will be big risk, thats why we reinstalling all our servers and import client data's.

    We expect some VM's will coming up in some hours.

    We are keep updating our websites if there is any server done with restoring.

  • Verelox said: backdours

    Now what kind? Or do you have a clue? How did he take control of the dedicated servers? Something is not adding up!

    Thanked by 1k0nsl
  • @PieNotEvenEaten said:

    Verelox said: backdours

    Now what kind? Or do you have a clue? How did he take control of the dedicated servers? Something is not adding up!

    All dedicated servers are back online.
    It was not only the servers in network (switch/router) everything hi got access.
    We have configure complete everything again.

  • @PieNotEvenEaten said:
    How did he take control of the dedicated servers?

    gotcha! i'm really interesting to hear their answer

  • Verelox said: All dedicated servers are back online. It was not only the servers in network (switch/router) everything hi got access. We have configure complete everything again.

    I'm curious: Where is this ex-Administrator located? Have yours or his local police been notified?

    Thanked by 1Clouvider
  • Wow, even IT magazines now report on this.
    I'm also curious about the stuff people asked in posts before, but please, focus on restoring the network, etc. first. As soon as all is done, we are eager for you telling us your story. XD

  • FreakensteinFreakenstein Member
    edited June 2017

    Whatever the reason or whatever Verelox did to this guy (or whatever he thinks they did to him), he's a massive a-hole. Not only is he causing damage to all the customers who never did anything to hurt him, but also his former co-workers are now having to work around the clock, 24x7, to try and clean up the mess of this spiteful little crybaby. I'm a sysadmin myself and no matter how mad I would get at my bosses, the idea I would screw over my co-workers by having them have to fix the damage I caused, would be one of the first reasons not to do something like this. That and knowing right from wrong of course, and trying to be a decent human being.

    It's almost impossible to keep high level admins from doing stuff like this, especially if he had time to prepare. Yes, you can disable his accounts, but he'll know passwords from administrator service accounts or will have created a second shadow admin account for himself. He could have copied all passwords of all priviliged users in advance. He could have installed a remote control tool in advance. He could have installed a keylogger on a workstation of a co-worker. As an admin, you'll know all the ways in and all the backdoors, and if no backdoors exist, you can create them.
    There's no company who will change all the passwords for all the privileged accounts every time an admin leaves the company.

    Best of luck to his former co-workers at Verelox! I know how you must feel, and hope that people will understand the situation you're in and appreciate the work you're doing. Pressure must be intense. :(

  • MikeAMikeA Member, Host Rep
    edited June 2017

    @Freakenstein said:
    Whatever the reason or whatever Verelox did to this guy (or whatever he thinks they did to him), he's a massive a-hole. Not only is he causing damage to all the customers who never did anything to hurt him, but also his former co-workers are now having to work around the clock, 24x7

    That's why;
    A.) You don't give people control of your whole server/network without a means of removing access.
    B.) You make people sign legal contracts before giving them access to your whole business.

    I've never heard of them but it seems like a mistake a Minecraft server host would make.

    Thanked by 1HBAndrei
  • Yeah, but you can only remove the access you know of. I can think of a dozen ways I could access the network of my company/customers or destroy their servers after they have disabled all my accounts.
    A legal contract won't stop someone like this. Even without a contract, this is completely illegal. He'll be prosecuted and probably go to jail. He'll also never get a job in his line of work again. Every tech company is going to avoid him like the plague and won't let him near any of their servers. If that didn't stop him, why would a piece of paper do the trick? Sure, they can fine him, but as we say in my country (and probably his): You can't pluck feathers from a bald chicken.

    Thanked by 1netomx
  • drivexdrivex Member
    edited June 2017

    Some people are saying it's related to this: https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/accidentally_destroyed_production_database_on/?sort=confidence

    Don't know if it's true, but if it is the CTO should be fired yesterday. Horrible story anyway.

    Thanked by 1williclarkam
  • @bluesega said:
    Okay then. Can't tell about this because all browsers are redirecting to https:// always and then tell it's not reachable, probably it is with :80 and http://, but sadly can't tell. A mail from them would've been nice though. (Is a screenshot possible for you please?)

    Probably HSTS:
    https://tools.ietf.org/html/rfc6797

  • @Freakenstein said:
    Whatever the reason or whatever Verelox did to this guy (or whatever he thinks they did to him), he's a massive a-hole. Not only is he causing damage to all the customers who never did anything to hurt him, but also his former co-workers are now having to work around the clock, 24x7, to try and clean up the mess of this spiteful little crybaby. I'm a sysadmin myself and no matter how mad I would get at my bosses, the idea I would screw over my co-workers by having them have to fix the damage I caused, would be one of the first reasons not to do something like this. That and knowing right from wrong of course, and trying to be a decent human being.

    It's almost impossible to keep high level admins from doing stuff like this, especially if he had time to prepare. Yes, you can disable his accounts, but he'll know passwords from administrator service accounts or will have created a second shadow admin account for himself. He could have copied all passwords of all priviliged users in advance. He could have installed a remote control tool in advance. He could have installed a keylogger on a workstation of a co-worker. As an admin, you'll know all the ways in and all the backdoors, and if no backdoors exist, you can create them.
    There's no company who will change all the passwords for all the privileged accounts every time an admin leaves the company.

    Best of luck to his former co-workers at Verelox! I know how you must feel, and hope that people will understand the situation you're in and appreciate the work you're doing. Pressure must be intense. :(

    I agree - I've personally seen colleagues visibly angry with managerial staff but I haven't seen any of them take it to the level of disrupting the service out of anger. Hope I never do, but if you did it leaves you liable to being sued etc so I don't think it's worth it for the weird sort of enjoyment this individual got out of it.

    Unfortunately, as you mentioned sysadmins have access to almost everything so there's a fair amount of trust within the role not to do any damage.

  • @drivex said:
    Some people are saying it's related to this: https://www.reddit.com/r/cscareerquestions/comments/6ez8ag/accidentally_destroyed_production_database_on/?sort=confidence

    Don't know if it's true, but if it is the CTO should be fired yesterday. Horrible story anyway.

    Doubt it. That thread is 7 days old. The Verelox incident apparently happened 3 days ago. Besides, a single mistake can never delete all servers in different kinds of datacenters. According to the thread he just deleted a production database.
    But I agree. That CTO should not blame the new guy. It never should have been possible for this to happen. Who in heaven's name puts production admin credentials in a manual for setting up a dev enviroment?! That's waiting for an accident to happen. \0/
    Wait, I'm wrong. It's not even waiting, it's rolling out the red carpet and sending an invitation on a golden platter. "You are cordially invited to destroy our production data. Formal attire not required."

  • Interestingly, nobody posted the update from the last days here, so I'll do that:

    It seems that there wasn't any huge data loss after all, can also confirm that my server is up again without any data loss (at least didn't notice one yet).

    Now we're up for your story, @Verelox :)

    https://www.verelox.com

    Thanked by 1netomx
  • sinsin Member
    edited June 2017

    @verelox any idea what day the hosting/panel will be back? I wanted to start up a new ashburn server :-)

    -edit- nevermind, I see the new status update says the website and everything will be coming up next.

  • CoreyCorey Member

    If any sysadmin wanted to do some damage they can probably wreak havoc before being caught, it's just how much access the position needs to do work. You can't restrict a sysadmin too much or they wouldn't be able to do their job, especially if they aren't tier 1 phone support and do more advanced things on your systems.

    Thanked by 2netomx hostdare
  • Anyone having issues ordering from other providers after this incident? I'm a verelox customer, and now I can't seem to order from other hosts due to fraud report...

    Trying to pinpoint who and when, but I haven't had any issues prior to Verelox being down.

    Sent verelox a message and awaiting response.

  • jarjar Member, Patron Provider

    @unusual917 said:
    Anyone having issues ordering from other providers after this incident? I'm a verelox customer, and now I can't seem to order from other hosts due to fraud report...

    Trying to pinpoint who and when, but I haven't had any issues prior to Verelox being down.

    Sent verelox a message and awaiting response.

    I can tell you why you're on fraudrecord and it has nothing to do with verelox. It's fairly serious, I wouldn't accept an order if I saw this listing. PM me if interested.

    Thanked by 2Clouvider unusual917
  • bluesegabluesega Member
    edited June 2017

    Hi there,

    Does anybody know something new?
    The Verelox website seems to be under maintenance for days.
    I think about a little bit more than a week ago they took this maintenance website online, also telling that there was a DDoS attack on NL servers.
    Now, the last message is not there anymore, but since thursday it's being told website will be back shortly.

    I mean, my server is running, but it'd be interesting to know when the control panel will be available again.

  • sinsin Member

    @bluesega said:
    Hi there,

    Does anybody know something new?
    The Verelox website seems to be under maintenance for days.
    I think about a little bit more than a week ago they took this maintenance website online, also telling that there was a DDoS attack on NL servers.
    Now, the last message is not there anymore, but since thursday it's being told website will be back shortly.

    I mean, my server is running, but it'd be interesting to know when the control panel will be available again.

    I emailed them a few days ago asking if they might have an estimate of when the website/control panel would be back up but I never got a reply.

  • @jarland said: you

    @unusual917 said:
    Anyone having issues ordering from other providers after this incident? I'm a verelox customer, and now I can't seem to order from other hosts due to fraud report...

    Trying to pinpoint who and when, but I haven't had any issues prior to Verelox being down.

    Sent verelox a message and awaiting response.

    I can tell you why you're on fraudrecord and it has nothing to do with verelox. It's fairly serious, I wouldn't accept an order if I saw this listing. PM me if interested.

    Now I'm curious what did he do for it to be more serious than usual chargeback if he remembers nothing of importance. (Spam? DDoS? CP?)

    Thanked by 1ucxo
  • unusual917unusual917 Member
    edited June 2017

    @jarland and @stefeman
    0.0
    Is there any way I can check myself? Jarland please can you pm me the info? Honestly I think I was falsely listed. Don't ever recall an issue with any of my providers...

    Jarland are you the provider that had issues with me? If so can you pm me I would really like to resolve it.

    Perhaps it is someone else you are referring too?

  • anyone having issues connecting from UPC_Broadband after the Verolox incident? users from UPC_Broadband have been complaining about random disconnects. they noted it wasn't an issue before the downtime.

  • hostdarehostdare Member, Patron Provider

    With great power comes great responsibility or you can create havoc ! applies everywhere .

  • sinsin Member

    Well the new control panel is sorta up but I don't see my $ balance.

  • Yeah, my balance isn't there either :(

Sign In or Register to comment.