All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
VSYS Host: UA data centers. What happened and where recovery stands?
Hi everyone,
Many of you have been affected by the outage in Kyiv or have seen it discussed here, so we want to give you a clear picture of what happened and where recovery stands.
What happened
On September 23, the first Russian missile strike hit one of data centers. A few hours later, the second one was hit, and then hit again. Both are now out of operation for an indefinite period.
These were not isolated hits. The strikes also affected other data centers where we have no equipment. This was a wider attack on data center infrastructure.
Condition of the equipment
In the first hours we had no physical access to our hardware and could only rely on information from the data center teams. Once our engineers reached the sites:
- In data center 1, 100% of our equipment is physically intact.
- In data center 2, our equipment also survived, but some hardware shows signs of water exposure. These servers need to be inspected before they can be powered on, otherwise recoverable hardware could be damaged.
Virtual services
Rather than waiting for the original facilities to reopen, we moved our virtual infrastructure to another data center selected for its security profile. The majority of affected virtual services are already running there.
Work on the rest continues, and we expect 100% of them to be restored within the nearest few days.
Dedicated servers
This is the harder part. Dedicated servers are the hardware itself, so they have to be physically moved to another facility before they can run again, and we're talking about a very large fleet.
The new facility needs enough space, power, cooling, connectivity and staff to take this volume. It also must not carry an unacceptable risk of a missile strike. We won't move customer infrastructure to the first facility that happens to have space.
We haven't secured such a facility yet, so there is no ETA for dedicated server restoration. We'd rather not give a date we can't stand behind. Once a facility is confirmed, servers will be relocated and restored progressively.
In parallel, we're working on a way for customers to copy their data from dedicated servers before full restoration. We'll contact affected customers directly once a workable process is ready.
Other locations
VSYS Host is fully operational. Support is available, and our locations outside Ukraine are running normally. We're seeing a lot of migration requests, so some dedicated configurations outside Ukraine may soon be temporarily out of stock. If you're planning to move, open a ticket and we'll check what's available.
Full incident report: https://vsys.host/news/two-kyiv-data-centers-hit-within-hours-incident-report-and-recovery-status
We'll keep this thread updated as things progress. Happy to answer general questions here.
Thank you to everyone who has been patient with us.
We've been working through these days without a break to bring everything back, and we'll keep going until every service is restored.
Thank you for your patience, your support, and every kind word we've received. It means a lot to us right now.
Comments
Dear @vsys_host
Sad to heard about this news! Please provide resolution of ticket #3944580
Great work, but you could post it earlier. At least a thread such as "we are doing our best"
I like it more when they focus on dealing with the situation first, like they chose to do. LET is not as important.
Thanks for the feedback. From day one, our entire team has been focused on recovery, and the essential updates were shared directly with our customers. We chose to post publicly once we had verified, specific information rather than general statements. We'll keep this thread updated from here on.
We have been working with you for 5 years and we are confident you will rise stronger. We have always been by your side. We hope you overcome this.
Thank you. Our team replied to your ticket about 1.5 hours ago, so please check it for the details on your services. If anything is still unclear, just reply there and we'll follow up.
Thank you for your understanding. We're doing everything we can to bring all services back as soon as possible.
Five years together means a lot to us, and so does your support right now. Thank you for standing by us. We will definitely come out of this stronger, more than you can imagine.
What will happen to the servers that come into contact with water?
Hi vsys,
Are your engineers and staff safe?
Would you be able to migrate VPS's to NL for those who would choose to (granted if there are no meaningful policy differences and/or the customers are correctly informed)?
This way both you and the custom have less worries about the infrastructure being safe.
The server hardware has been retrieved from the affected facility and is being relocated to a secure data center for technical assessment and recovery work.
Because the equipment may have been exposed to water, it cannot be safely powered on before it undergoes proper technical inspection and servicing. At this stage, we have no confirmed information indicating that data has been lost. However, until the hardware and storage devices have been inspected, we cannot confirm the condition of the drives or guarantee that the data is recoverable.
Due to the number of servers requiring individual inspection, this process will take time. Affected customers will be updated once the inspection of their server has been completed.
Thank you for asking, it means a lot. Yes, our team is safe and continues working on the recovery.
Customers who want to move can migrate to one of our locations outside Ukraine. Please open a ticket, and our team will check current availability and help you.
Fellas, you do realize that any data center located in a country currently in an active war zone carries an obvious risk to your services, right? ............... for love of all things, please say yes
Cut @vsys_host some slack. I am fairly confident they are doing everything they reasonably can under the circumstances.
And seriously, what exactly do you expect them to do in the first hours after a building has been hit? Send an email saying:
“Hey, sorry, your service is offline. We don't yet know what happened, we don't know the condition of the building, we don't know whether we're allowed inside, and we don't know when recovery will be possible.”
That would provide exactly zero useful information.
Also sending out a notification saying:
“The data center hosting your service has been hit by a missile attack.”
without knowing the actual extent of the damage, whether the equipment survived, whether engineers can access the building, or what the recovery options are, would probably create even more panic while providing almost no useful information.
In a situation like this, saying nothing for a few hours or until you actually know what happened can make far more sense than sending out an alarming message based on incomplete information or rumors.
If a building is hit or structurally damaged, it can take days before engineers are even allowed to enter it safely. Only after that can they inspect the equipment, power, networking, water damage, cabling, and everything else involved in bringing services back.
So repeatedly posting ticket numbers and asking “when will my server be back?” after they have already clearly stated that there is currently no ETA achieves absolutely nothing.
If you've somehow missed what has been happening in Ukraine since 2022, perhaps spend five minutes checking the news before treating this like a normal hosting outage or some presuming the fella packed he's bags and went to the Bahamas.
Or you could be a grown up and show some degree of compassion.
Also, a forum thread is not a ticketing system.
Posting:
“Ticket #123456, please check.” will not magically move your server/service to the front of a recovery operation involving damaged buildings, infrastructure or physical machines.
Let their engineers do their jobs. When they have confirmed information and realistic ETAs, I'm sure they will publish them.
To @vsys_host and to all other providers affected by the latest missile attacks on Ukrainian infrastructure, I wish you the best of luck and hope your teams and infrastructure recover as safely and quickly as possible.
Thank you, truly. It means a lot to read this!
These have been some of the hardest days in our company's history. Our team has been working without a break to reach the equipment, move services to safer infrastructure and bring customers back online. Messages like yours give us the strength to keep going.
We'll keep sharing updates here as soon as we have confirmed information.
So my vps which is still down didnt suffer file loss and will eventually be back with files intact?
This guy...... I swear, LET needs KYC, at least for age verification....
Take care of yourself and your team, focus on what is important right now, and please, completely ignore ticket numbers here or rivers made from crying
.
I think you have better things on witch your time is worth wasting then the angry mob around here.
I really hope that the damage is minimal for all of you involved in the latest incidents.
I am 34... I just want to know if my files were lost or not no need to be rude
thanks!
as someone who has been with Vsys 5 years or more and have alot of files on there i think its a fair concern and question i never asked them to hurry up or got rude I asked when it is back are the files safe. What we can gather here is to never use Host-c for any of our hosting needs if you care about professionalism
I am 41. If I don't want my files to be lost, I back them up
best of luck @vsys_host
I have a backup from a couple of months ago, and it is on me; I should back up more often, but I was simply asking if the files are safe. If not, I fully understand that it is on me. It was a simple question, and I am speaking to Vsys, with whom I do business with not fourm trolls like you who i could not be bothered to care less about. @luckypenguin
And out of a few thousand affected services, judging by the amount of IP space that went dark, how exactly do you expect them to know the status of your specific files right now?
Seriously, try to imagine the situation from their side for a minute.
They have multiple data centers out, physical access restrictions, potentially damaged equipment, possible water exposure, power and network issues, relocation logistics, and a very large number of affected customers.
With limited people and limited time, what would you prioritize?
A - Going through individual forum posts trying to identify whether one specific server and one specific set of files are okay.
Or:
B - Restoring the infrastructure as a whole, securing replacement capacity, inspecting hardware, bringing networks back, and then dealing with individual cases once they actually have reliable information.
This isn't about me being rude. I am not,
It is about whether the question can realistically be answered right now.
“Are my files safe?” is obviously a completely reasonable concern for you, yet, for that to be answered, we need to do option B above.
If nobody has physically inspected your specific server yet, there is literally no useful answer they can give you beyond “we don't know yet.”
And repeatedly asking the same question in a forum isn't going to make an engineer teleport into the building and inspect your disks faster.
In an incident of this scale, individual service investigation is naturally going to come after restoring basic operations and determining what infrastructure is actually recoverable.
That is triage, and it is the only viable option in my view.
I am not trying to be rude, dude. I am simply being realistic about what can and cannot be done during a major physical infrastructure incident, but feel free to prove me wrong.
I get it, but I thought they might know if they have seen any file loss yet. I haven't asked that question multiple times, and I didn't even submit a ticket because I didn't want to bother them, but they have been answering questions here and replying to thank folks, so I figured I would ask a general question; I don't feel in the wrong for that. @host_c
No backup == no worth.
No backup in an active war zone == lol
https://bgp.he.net/AS6698#_prefixes
https://bgp.he.net/AS30860#_prefixes
Looking at this is meaningless for a dedicated server; it only indicates that the gateway was restarted elsewhere to restore VPS connectivity, but the dedicated server itself was not moved there.
## kindly check Ticket #0896107
True. If there's ongoing war, I would have dig a 60ft hole, put my hdds on a safe, silicone seal it then buried it. And that's before I make copies to other countries