Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

VSYS Host: UA data centers. What happened and where recovery stands?

124»

Comments

  • LeviLevi Veteran

    @Hondura said:

    @Levi said:

    @Hondura said:

    @deafcon said:

    @Hondura said:

    @vsys_host said:
    Hi everyone,

    Many of you have been affected by the outage in Kyiv or have seen it discussed here, so we want to give you a clear picture of what happened and where recovery stands.

    What happened

    On September 23, the first Russian missile strike hit one of data centers. A few hours later, the second one was hit, and then hit again. Both are now out of operation for an indefinite period.

    These were not isolated hits. The strikes also affected other data centers where we have no equipment. This was a wider attack on data center infrastructure.

    Condition of the equipment

    In the first hours we had no physical access to our hardware and could only rely on information from the data center teams. Once our engineers reached the sites:

    • In data center 1, 100% of our equipment is physically intact.
    • In data center 2, our equipment also survived, but some hardware shows signs of water exposure. These servers need to be inspected before they can be powered on, otherwise recoverable hardware could be damaged.

    Virtual services

    Rather than waiting for the original facilities to reopen, we moved our virtual infrastructure to another data center selected for its security profile. The majority of affected virtual services are already running there.

    Work on the rest continues, and we expect 100% of them to be restored within the nearest few days.

    Dedicated servers

    This is the harder part. Dedicated servers are the hardware itself, so they have to be physically moved to another facility before they can run again, and we're talking about a very large fleet.

    The new facility needs enough space, power, cooling, connectivity and staff to take this volume. It also must not carry an unacceptable risk of a missile strike. We won't move customer infrastructure to the first facility that happens to have space.

    We haven't secured such a facility yet, so there is no ETA for dedicated server restoration. We'd rather not give a date we can't stand behind. Once a facility is confirmed, servers will be relocated and restored progressively.

    In parallel, we're working on a way for customers to copy their data from dedicated servers before full restoration. We'll contact affected customers directly once a workable process is ready.

    Other locations

    VSYS Host is fully operational. Support is available, and our locations outside Ukraine are running normally. We're seeing a lot of migration requests, so some dedicated configurations outside Ukraine may soon be temporarily out of stock. If you're planning to move, open a ticket and we'll check what's available.

    Full incident report: https://vsys.host/news/two-kyiv-data-centers-hit-within-hours-incident-report-and-recovery-status

    We'll keep this thread updated as things progress. Happy to answer general questions here.

    Thank you to everyone who has been patient with us.

    We've been working through these days without a break to bring everything back, and we'll keep going until every service is restored.

    Thank you for your patience, your support, and every kind word we've received. It means a lot to us right now.

    **What did you start this thread for? **
    There’s no news, and some customers are being ignored—there’s no ETA.

    Their website says they’ve been around since 2009; how does a company that’s been in business for nearly 20 years fail to release any updates?

    THEY WERE HIT BY A DRONE STRIKE.

    Even a five-year-old knows they were bombed—so what? I’m asking for updates on what the other companies are doing so they don't leave their customers in the lurch.

    Update 1: We have been bombed.
    Update 2: Ruins everywhere, lot of water from firefighting. Servers to be seen.
    Update 3: Still water and ruins, no sign of servers.
    Update 4: Still water and ruins, no sign of servers.
    Update 5: Still water and ruins, no sign of servers.
    Update 6: Still water and ruins, no sign of servers.
    Update 69: Water evaporated, ruins still in place. No servers.
    Update 169: Some metal sheets from servers emerge out of ruins. We should be able to assemble few servers from 20 server parts. We are bringing up diesel generators and solar panels, thought, there is little sun in autumn.
    Update X: To be continued...

    I think exactly the same thing—that the servers are destroyed. They’re just claiming it’s an inspection or something because they don’t want to admit outright that the servers—or rather, all the data—are gone.

    What will happen if they admit that? Will you sue them in Ukraine? Push your disaster recovery button and that’s it.

  • LeviLevi Veteran

    @duonghowuaaxtnw763 said:
    The excuses offered above are truly laughable. If all providers were in the same boat, then there would be nothing to criticize. The problem is that other service providers have already restored their dedicated servers. So, do you really have any grounds left to make excuses? :D

    So what? War is “force majore” and if you knew that - then you accepted the fact that at any moment bombs can fel in and that’s it.

    “Others already restored” - why are you not with them?

    Again - hosting critical data in country where active war action takes place is beyond idiotic.

  • @Levi said:

    @duonghowuaaxtnw763 said:
    The excuses offered above are truly laughable. If all providers were in the same boat, then there would be nothing to criticize. The problem is that other service providers have already restored their dedicated servers. So, do you really have any grounds left to make excuses? :D

    So what? War is “force majore” and if you knew that - then you accepted the fact that at any moment bombs can fel in and that’s it.

    “Others already restored” - why are you not with them?

    Again - hosting critical data in country where active war action takes place is beyond idiotic.

    Manager from Vsys and Assistant from elon kurvaovic

    If you don't have any useful information, don't go around telling people what they should or shouldn't do—it's none of your damn business.

  • @Hondura said:

    @Levi said:

    @duonghowuaaxtnw763 said:
    The excuses offered above are truly laughable. If all providers were in the same boat, then there would be nothing to criticize. The problem is that other service providers have already restored their dedicated servers. So, do you really have any grounds left to make excuses? :D

    So what? War is “force majore” and if you knew that - then you accepted the fact that at any moment bombs can fel in and that’s it.

    “Others already restored” - why are you not with them?

    Again - hosting critical data in country where active war action takes place is beyond idiotic.

    Manager from Vsys and Assistant from elon kurvaovic

    If you don't have any useful information, don't go around telling people what they should or shouldn't do—it's none of your damn business.

    Great advice for you, stop telling vsys what to do :)

    Thanked by 1AdiOnLET
  • dusstdusst Member
    edited September 30

    Any méi jī jī who wants to get a working metal server back up within 3 days of the drone strike needs to learn 2 new and useful words for them: backup and dolboyob

  • Overheard from friends in the music piracy community (not involved myself):

    Apparenlty, there are two prominent sites (pillows.su and leaked.cx) which are popular DMCA-resistant filehosts/file sharing platforms for leaked and pirated music. Well, a few days ago (I believe on the 25th) both sites went down with no notice.

    No error message on Pillows but on Leaked:

    https://web.archive.org/web/20260925142838id_/https://leaked.cx/

    https://web.archive.org/web/20260926184421id_/https://leaked.cx/

    I guess we now know who is hosting these sites!

  • forestforest Member

    @duonghowuaaxtnw763 said: The excuses offered above are truly laughable. If all providers were in the same boat, then there would be nothing to criticize. The problem is that other service providers have already restored their dedicated servers.

    It's almost as if high explosives cause more damage the closer you are.

    Did you ever think that, perhaps, VSYS got unlucky and had many of their servers closer to the explosion?

    Thanked by 1OpaqueRegistrant
  • @Hondura said:

    @Levi said:

    @Hondura said:

    @deafcon said:

    @Hondura said:

    @vsys_host said:
    Hi everyone,

    Many of you have been affected by the outage in Kyiv or have seen it discussed here, so we want to give you a clear picture of what happened and where recovery stands.

    What happened

    On September 23, the first Russian missile strike hit one of data centers. A few hours later, the second one was hit, and then hit again. Both are now out of operation for an indefinite period.

    These were not isolated hits. The strikes also affected other data centers where we have no equipment. This was a wider attack on data center infrastructure.

    Condition of the equipment

    In the first hours we had no physical access to our hardware and could only rely on information from the data center teams. Once our engineers reached the sites:

    • In data center 1, 100% of our equipment is physically intact.
    • In data center 2, our equipment also survived, but some hardware shows signs of water exposure. These servers need to be inspected before they can be powered on, otherwise recoverable hardware could be damaged.

    Virtual services

    Rather than waiting for the original facilities to reopen, we moved our virtual infrastructure to another data center selected for its security profile. The majority of affected virtual services are already running there.

    Work on the rest continues, and we expect 100% of them to be restored within the nearest few days.

    Dedicated servers

    This is the harder part. Dedicated servers are the hardware itself, so they have to be physically moved to another facility before they can run again, and we're talking about a very large fleet.

    The new facility needs enough space, power, cooling, connectivity and staff to take this volume. It also must not carry an unacceptable risk of a missile strike. We won't move customer infrastructure to the first facility that happens to have space.

    We haven't secured such a facility yet, so there is no ETA for dedicated server restoration. We'd rather not give a date we can't stand behind. Once a facility is confirmed, servers will be relocated and restored progressively.

    In parallel, we're working on a way for customers to copy their data from dedicated servers before full restoration. We'll contact affected customers directly once a workable process is ready.

    Other locations

    VSYS Host is fully operational. Support is available, and our locations outside Ukraine are running normally. We're seeing a lot of migration requests, so some dedicated configurations outside Ukraine may soon be temporarily out of stock. If you're planning to move, open a ticket and we'll check what's available.

    Full incident report: https://vsys.host/news/two-kyiv-data-centers-hit-within-hours-incident-report-and-recovery-status

    We'll keep this thread updated as things progress. Happy to answer general questions here.

    Thank you to everyone who has been patient with us.

    We've been working through these days without a break to bring everything back, and we'll keep going until every service is restored.

    Thank you for your patience, your support, and every kind word we've received. It means a lot to us right now.

    **What did you start this thread for? **
    There’s no news, and some customers are being ignored—there’s no ETA.

    Their website says they’ve been around since 2009; how does a company that’s been in business for nearly 20 years fail to release any updates?

    THEY WERE HIT BY A DRONE STRIKE.

    Even a five-year-old knows they were bombed—so what? I’m asking for updates on what the other companies are doing so they don't leave their customers in the lurch.

    Update 1: We have been bombed.
    Update 2: Ruins everywhere, lot of water from firefighting. Servers to be seen.
    Update 3: Still water and ruins, no sign of servers.
    Update 4: Still water and ruins, no sign of servers.
    Update 5: Still water and ruins, no sign of servers.
    Update 6: Still water and ruins, no sign of servers.
    Update 69: Water evaporated, ruins still in place. No servers.
    Update 169: Some metal sheets from servers emerge out of ruins. We should be able to assemble few servers from 20 server parts. We are bringing up diesel generators and solar panels, thought, there is little sun in autumn.
    Update X: To be continued...

    I think exactly the same thing—that the servers are destroyed. They’re just claiming it’s an inspection or something because they don’t want to admit outright that the servers—or rather, all the data—are gone.

    Wouldn't you do things the way they are saying they are doing things, if you were them?

  • the amount of lack empathy is something else. i refuse to acknowledge mjjs are human.

  • rpqurpqu Member

    @ScreenReader said:
    the amount of lack empathy is something else. i refuse to acknowledge mjjs are human.

    There's strong financial motivation. If they're able to put the nuance over the service rendered deemed not fit for purpose/described. They may file to a third party and do chargeback.
    Or it's just straight, "IDGAF what it takes, make it happen. I've paid the for the service"

  • LeviLevi Veteran

    @ScreenReader said:
    the amount of lack empathy is something else. i refuse to acknowledge mjjs are human.

    Well, the mjj is human at its core, but it is without penis. Peniless human. It may affect them and their perception of reality.

  • @Hondura said:

    @Levi said:

    @duonghowuaaxtnw763 said:
    The excuses offered above are truly laughable. If all providers were in the same boat, then there would be nothing to criticize. The problem is that other service providers have already restored their dedicated servers. So, do you really have any grounds left to make excuses? :D

    So what? War is “force majore” and if you knew that - then you accepted the fact that at any moment bombs can fel in and that’s it.

    “Others already restored” - why are you not with them?

    Again - hosting critical data in country where active war action takes place is beyond idiotic.

    Manager from Vsys and Assistant from elon kurvaovic

    If you don't have any useful information, don't go around telling people what they should or shouldn't do—it's none of your damn business.

    You've been given good advice already: Activate your disaster recovery plan and restore your backups to another location.

    There will be no useful information from this host for a long time for the reasons they've already explained clearly. Comparing them to other hosts is a worthless exercise, as every company might have experienced different levels of damage, and not all of them have spare capacity to restore to another location.

  • My dedicated servers are still showing as “Awaiting Inspection.”

    I completely understand that VSYS has many servers to inspect and that safety must come first. However, there is one important detail about my servers that I have already reported to support, and I would really appreciate it if VSYS could make sure this information is passed to the technical/recovery team performing the inspections.

    According to the timeline of the incident, the attack had already occurred by approximately 11:30 Kyiv time.

    Despite this, my servers continued to operate normally until approximately 16:30 Kyiv time, several hours after the attack. My understanding is that the facility continued operating on supplementary/emergency generator power until that power was eventually exhausted.

    Only after approximately 16:30 did my servers become unreachable.

    Now the servers are awaiting inspection because of possible water exposure.

    I am not claiming that this proves the servers or drives were not subsequently exposed to water. I simply believe that the fact that they remained fully operational for approximately five hours after the attack could be relevant information for the technicians inspecting the hardware.

    I have already provided this timeline to support, but I would just like confirmation that this information has been added to the technical notes for my specific servers and communicated to the recovery team.

    At this point, my main priority is not even restoring the complete servers. I primarily need access to my data, particularly one important SQL backup.

    If someone from VSYS here could please make sure this information is attached to my case before the hardware inspection, I would greatly appreciate it.

  • GN9101GN9101 Member

    That's good info. The problem is VSYS's responses they're not giving any clarity, not answering anyone, and not updating the threads they created and called official. I've always been in favor of waiting and being patient because this isn't their fault, but they aren't doing their part to give users peace of mind. They just keep sending canned responses to everyone. What happened is unfortunate, but the way they're handling the situation is even worse.

  • Exactly. I completely understand that this was an exceptional event and that inspecting potentially water-exposed hardware takes time. I don’t blame VSYS for what happened, and I am prepared to wait if necessary.

    The difficult part is having no visibility at all.

    My servers have been showing “Awaiting Inspection” for days, and support cannot tell me whether they are 10th or 500th in the inspection queue, whether inspections are currently taking place, or even approximately how many dedicated servers have already been inspected.

    Even something as simple as:

    “Inspections have started, X servers have been checked so far, and we are processing approximately X servers per day”

    would make a huge difference.

    I don’t need a guarantee or an exact ETR. I just need to know that the process is actually moving forward.

    In my particular case, I don’t even need the entire server restored. I mainly need one SQL backup from the storage.

    I really hope VSYS starts providing some concrete progress updates, even if the news is simply that recovery will take longer than expected.

  • GN9101GN9101 Member

    Your situation is the same as many others, and what you're proposing would be the smartest move. The problem and what worries me is that if VSYS isn't doing it, there must be a reason, and while I don't want to say it out loud yet, we all know what it is. Doing what you suggested isn't hard at all.

    I think everyone with a dedicated server would be willing to wait months for the servers to work again if they were at least given the option to recover their data, but they aren't offering any clarity on that either. The way they are handling this situation is just tragic.

  • You guys are dense. Vsys already said what is the plan. No STUF and wait, stop whining like a bunch of little pussboys.

Sign In or Register to comment.