VSYS Host: UA data centers. What happened and where recovery stands?

Hondura · October 7

@vsys_host said:
Thank you all for your questions. There have been quite a few, so we’d like to take the time to answer them properly.

We’re gathering your questions into a single Q&A and will post the answers in this thread tomorrow, covering as much as we can.

If there’s something you’d like us to include, feel free to add it below. Please keep your questions thoughtful, clear, and relevant so we can make the Q&A as useful as possible for everyone.

Another day has passed. Even though it was announced that many open questions would be answered today, once again, nothing happened.

For more than two weeks, we have known that the data centers in Ukraine were attacked. Please stop repeating this information over and over again. Every affected customer already knows what happened. We do not need another explanation of what happened — we need concrete information about what has actually happened with our hardware and our data since then.

In more than two weeks, there have basically been only two updates. Even in the VSYS panel, there is hardly any real progress visible. The status has simply changed from “Waiting for inspection” to “Waiting for intact” or similar status messages.

For us as customers, the situation is still:

Waiting for inspection: No ETA.
Waiting for intact: More than two weeks with no visible progress.
Are the servers and data safe? “We can only determine this after inspection.”
When will the inspection take place? No ETA.
When will the servers be brought back online? No ETA.

The main problem is no longer only the original incident. The communication afterwards has now become a serious problem as well.

If, after more than two weeks, the affected servers have apparently still not been fully tested, while at the same time there is still no concrete ETA for inspection, relocation, or restoration, then we have to ask: What actual progress has been made during these two weeks?

We are constantly being asked to remain patient. At the same time, businesses and livelihoods depend on these servers and the data stored on them. Every additional day without clear information causes further business damage.

Therefore, we finally need concrete answers to the following questions:

Have the affected servers actually been physically inspected and tested yet?
Have the storage drives been checked, and can anything already be said about the condition and integrity of the data?
Are the servers still at the original location, or have they already been transported to another data center?
If they have not yet been transported, what exactly is currently preventing the relocation?
When will the technical inspection of the servers that are still waiting for inspection actually begin?
Is there at least an approximate timeframe — are we talking about days, weeks, or months?
What happens if the inspection shows that some or all of the data can no longer be recovered?
What compensation or account credit will be provided to customers who have been unable to access their paid Dedicated Servers for weeks, or who may have permanently lost their data?

We are not asking for unrealistic promises or an exact date if one genuinely cannot be provided at this time. However, after more than two weeks, it should be possible to transparently explain what has already been completed, what is currently being done, what still needs to be done, and what exactly is currently blocking the recovery process.

“No ETA,” “please be patient,” and repeatedly referring to the attack do not answer these questions.

I am writing this publicly because, at this point, I no longer expect these questions to receive concrete answers publicly anyway. Nevertheless, these questions need to be asked. We are paying customers, and we have the right to know what is happening with our hardware, our data, and the services we have been paying for but have been unable to use for weeks.

After more than two weeks, we do not need another general statement.

We need concrete answers, transparency, and a clear recovery plan.

vsys_host · October 7

Q&A: your questions from this thread, answered

We read the whole thread and collected the questions that came up most often. Here are the answers, grouped by topic.

Dedicated servers: timelines

You already have the physical servers, you've found a new facility, you know which servers were affected, and there's still no ETA?

@GN9101 @UserUk @viper877 @Hondura
Every server we have checked so far is working, so we expect the affected share to be small: more than 90% of servers should be unaffected.

What holds things up is space, not hardware. We did find space, and all of it went to bringing cloud and VPS back online. One facility is now completely full. New racks for dedicated servers are being prepared and we expect them in about two weeks, although that doesn't depend on us alone. Moving servers abroad also takes time and involves factors outside our control.

Some customers have already been restored or moved to a replacement server.

What several means? If just relocation might take 5-6-7 weeks, then what about the servers from DC2?

@cockli
By "several weeks" we mean two to three. About two weeks for the new racks to be ready, then about 5 days to install the servers. Not all of this depends on us, but on our side we are doing everything we can to speed it up.

How many servers from DC1 need relocation, how many can be relocated per day, do they manage a priority list?

@cockli

All DC1 servers need relocation. The exact number is internal, so we won't publish it.
One facility is already full, and we are waiting for two more.
Once a facility is ready, we need about 5 days to install all the servers.
There is no priority list for restoring service: we physically move everything in one go.
Replacements do follow an order: by the date of the request and as matching components become available.

If DC1 servers are intact as claimed, why are they still offline? At this point, DC1 and DC2 are in the same boat.

@GN9101 @tangying @Hondura
In DC1 the cooling system was destroyed. The servers are intact, but they can't run there. The DC2 facility was destroyed completely.

Does anyone actually have a server in DC1? Does that group even exist in the first place?

@GN9101 @caccamo
Yes, it does. See the answers above, and post in this thread about a DC1 server that has already been relocated and is working.

DC2: water, inspection, data

Part of the buildings burned down, and the remaining part was flooded with water and fire foam. What kind of restoration are you talking about?

@jenkki
All of our equipment has been taken out of that data center and moved to safe storage. Visually it is undamaged. There are traces of water, but that should not affect the data. We have already checked most of the storage media, and all of it works.

What will happen to the servers that come into contact with water?

@pawntruw @viper877
We will service them. If a customer needs their disks, we simply replace the server. We have enough hardware; what we are short of is data center space to put it in.

What if, after two weeks, they say, "Yes, they're damaged and can't be recovered"?

@Hondura
We have checked a large number of SSDs, and so far 100% of them work. We haven't reached the HDDs yet, but we expect the same picture there.

Has VSYS successfully accessed data from any of the recovered storage devices? Is there any known reason why customers may not be able to recover their data?

@caccamo
Yes, we have. Every storage device we have checked is undamaged and working. The rest are in our storage and visually undamaged. The wait comes down to space: we can't connect all servers at once, so customers are in a queue.

Does VSYS currently have a realistic expectation that customers will be able to recover their data, or not?

@caccamo
Yes, we do. We expect 99% of customers to get their data. We believe all SSDs are intact, and most HDDs. Some HDDs may have been affected by the shockwave, but we haven't come across any so far.

"X servers have been checked so far, and we are processing approximately X servers per day"

@caccamo
The queues are moving:

data handover: ≈90% of those who requested it
replacements: ≈46% of those who requested one
full VPS and Cloud services restoration: up to 10%
full dedicated services restoration: up to 10%

They're just claiming it's an inspection because they don't want to admit outright that all the data are gone.

@Hondura @GN9101
If the data were lost, we wouldn't be telling you we expect 99% recovery. Our entire virtual infrastructure has been restored, and not from backups. Customer dedicated servers were in the same racks as that infrastructure.

The provider will never tell you, "Sorry, we're closed, all your data is lost."

@jenkki
We're not closed. VPS and Cloud services are available to order, servers will be back in stock soon, we have three other locations, and a new one is on the way: Warsaw.

Getting your data before the server is back

If full server restoration will take months, will customers be given an opportunity to retrieve their data before then?

@caccamo @cockli @GN9101 @Hondura
Yes. The procedure is already running. Everyone who contacted us about their data is on the waiting list, and some customers have already received theirs.

I only need one SQL backup of approximately 300 MB.

@caccamo
Please send me here or via PM your ticket number. A request for a specific file is processed faster than a request for a full disk dump.

I submitted a ticket, their reply was identical to the one from the 26th… no estimate…

@longqin2021
Please send us a PM here with your ticket number. We will look into your case and do our best to sort it out.

Is it possible for the customer to verify the integrity ourself?

@rpqu
If you're in Ukraine, yes. Please contact us to arrange it.

Moving to another location

Would you be able to migrate VPS's to NL…?

@serverenthusiast
Yes. Everyone who asked for this has either been moved already or is waiting for components to become available for their request.
If you need it - let us know via ticket.

Colocation

Do you have any news for colo customers? Any hope for them?

@mudassar788
For colo it's simple: send a courier and we will hand over your hardware. We can have it ready the next business day.

Support hours

You still force us to wait for the Sales Department and only during business hours.

@psyco23
Our technical team is working on data recovery 24/7. Sales and billing are working extended hours as well.

About communication

We also see what many of you wrote here about communication: that updates were too rare, that ticket replies read like templates, that chat wasn't working, and that news about VPS was of little use to those waiting for a dedicated server.

We understand that waiting without news is hard. Here is how we approach it. We publish a public update when there is new information we can stand behind, and we would rather not post just for the sake of posting. General information goes on our website. Anything specific to your service is answered in your ticket, because that is where we can look at your exact server. Live chat has been available the whole time.

We said we would answer questions in this thread, and we are doing that: we follow it, and this Q&A is part of it.

Other providers

Other service providers have already restored their dedicated servers. So, do you really have any grounds left to make excuses?

@duonghowuaaxtnw763 @Hondura @caccamo
As @aj_potc pointed out above, every company faced a different level of damage, so a direct comparison doesn't tell you much. We'd add that smaller companies have an easier time handling an incident like this.

Before the incident

When did VSYS become aware of that increased risk?

@caccamo
The risk has been elevated since 2022, February. Our other locations have always been available.

I migrated out of vsys about a month ago… I feared something like this could happen

@snowman11
A pity it wasn't to one of our other locations.

The important stuff

Is the new facility drone-proof?

@rpqu
YES!

Provider: we were bombed. MJJ: any deals?

@fensuiji @Murv @zed
Yes, of course. There will be Black Friday discounts in all locations, and there will be a large reserve of servers for them.

What's next

Many more of you took part in this discussion: @RAVNIX @OpaqueRegistrant @TimboJones @Levi @forest @Webby123 @luckypenguin @MikeA @Mainfrezzer @lirrr @deafcon @Arirang @JohnFilch123 @oloke @stefeman @rcy026 @host_c @Saragoldfarb @ScreenReader @Obelous @VOR_SYS @Verunix @Zyra @itzsenu @tentor @Noah2002 @hesweb

If you, or anyone else reading this, still have questions, please post them here. We will collect them over the next few days and come back with another round of answers on anything that is still unclear.

Saragoldfarb · October 7

@Hondura said:

We need a clear recovery plan.

Just install your backups. Problem solved. Carry on.

zed · October 7

i think it would be brilliant if the 12 year olds could depart the thread and stop taunting people dealing with the hardships this circumstance is causing. just take the low-class shit to one of the other 9000 threads.

Saragoldfarb · October 7

@vsys_host said:

Q&A: your questions from this thread, answered

We read the whole thread and collected the questions that came up most often. Here are the answers, grouped by topic.

Dedicated servers: timelines

You already have the physical servers, you've found a new facility, you know which servers were affected, and there's still no ETA?

@GN9101 @UserUk @viper877 @Hondura
Every server we have checked so far is working, so we expect the affected share to be small: more than 90% of servers should be unaffected.

What holds things up is space, not hardware. We did find space, and all of it went to bringing cloud and VPS back online. One facility is now completely full. New racks for dedicated servers are being prepared and we expect them in about two weeks, although that doesn't depend on us alone. Moving servers abroad also takes time and involves factors outside our control.

Some customers have already been restored or moved to a replacement server.

What several means? If just relocation might take 5-6-7 weeks, then what about the servers from DC2?

@cockli
By "several weeks" we mean two to three. About two weeks for the new racks to be ready, then about 5 days to install the servers. Not all of this depends on us, but on our side we are doing everything we can to speed it up.

How many servers from DC1 need relocation, how many can be relocated per day, do they manage a priority list?

@cockli

All DC1 servers need relocation. The exact number is internal, so we won't publish it.

One facility is already full, and we are waiting for two more.

Once a facility is ready, we need about 5 days to install all the servers.

There is no priority list for restoring service: we physically move everything in one go.

Replacements do follow an order: by the date of the request and as matching components become available.

If DC1 servers are intact as claimed, why are they still offline? At this point, DC1 and DC2 are in the same boat.

@GN9101 @tangying @Hondura
In DC1 the cooling system was destroyed. The servers are intact, but they can't run there. The DC2 facility was destroyed completely.

Does anyone actually have a server in DC1? Does that group even exist in the first place?

@GN9101 @caccamo
Yes, it does. See the answers above, and post in this thread about a DC1 server that has already been relocated and is working.

DC2: water, inspection, data

Part of the buildings burned down, and the remaining part was flooded with water and fire foam. What kind of restoration are you talking about?

@jenkki
All of our equipment has been taken out of that data center and moved to safe storage. Visually it is undamaged. There are traces of water, but that should not affect the data. We have already checked most of the storage media, and all of it works.

What will happen to the servers that come into contact with water?

@pawntruw @viper877
We will service them. If a customer needs their disks, we simply replace the server. We have enough hardware; what we are short of is data center space to put it in.

What if, after two weeks, they say, "Yes, they're damaged and can't be recovered"?

@Hondura
We have checked a large number of SSDs, and so far 100% of them work. We haven't reached the HDDs yet, but we expect the same picture there.

Has VSYS successfully accessed data from any of the recovered storage devices? Is there any known reason why customers may not be able to recover their data?

@caccamo
Yes, we have. Every storage device we have checked is undamaged and working. The rest are in our storage and visually undamaged. The wait comes down to space: we can't connect all servers at once, so customers are in a queue.

Does VSYS currently have a realistic expectation that customers will be able to recover their data, or not?

@caccamo
Yes, we do. We expect 99% of customers to get their data. We believe all SSDs are intact, and most HDDs. Some HDDs may have been affected by the shockwave, but we haven't come across any so far.

"X servers have been checked so far, and we are processing approximately X servers per day"

@caccamo
The queues are moving:

data handover: ≈90% of those who requested it

replacements: ≈46% of those who requested one

full VPS and Cloud services restoration: up to 10%

full dedicated services restoration: up to 10%

They're just claiming it's an inspection because they don't want to admit outright that all the data are gone.

@Hondura @GN9101
If the data were lost, we wouldn't be telling you we expect 99% recovery. Our entire virtual infrastructure has been restored, and not from backups. Customer dedicated servers were in the same racks as that infrastructure.

The provider will never tell you, "Sorry, we're closed, all your data is lost."

@jenkki
We're not closed. VPS and Cloud services are available to order, servers will be back in stock soon, we have three other locations, and a new one is on the way: Warsaw.

Getting your data before the server is back

If full server restoration will take months, will customers be given an opportunity to retrieve their data before then?

@caccamo @cockli @GN9101 @Hondura
Yes. The procedure is already running. Everyone who contacted us about their data is on the waiting list, and some customers have already received theirs.

I only need one SQL backup of approximately 300 MB.

@caccamo
Please send me here or via PM your ticket number. A request for a specific file is processed faster than a request for a full disk dump.

I submitted a ticket, their reply was identical to the one from the 26th… no estimate…

@longqin2021
Please send us a PM here with your ticket number. We will look into your case and do our best to sort it out.

Is it possible for the customer to verify the integrity ourself?

@rpqu
If you're in Ukraine, yes. Please contact us to arrange it.

Moving to another location

Would you be able to migrate VPS's to NL…?

@serverenthusiast
Yes. Everyone who asked for this has either been moved already or is waiting for components to become available for their request.
If you need it - let us know via ticket.

Colocation

Do you have any news for colo customers? Any hope for them?

@mudassar788
For colo it's simple: send a courier and we will hand over your hardware. We can have it ready the next business day.

Support hours

You still force us to wait for the Sales Department and only during business hours.

@psyco23
Our technical team is working on data recovery 24/7. Sales and billing are working extended hours as well.

About communication

We also see what many of you wrote here about communication: that updates were too rare, that ticket replies read like templates, that chat wasn't working, and that news about VPS was of little use to those waiting for a dedicated server.

We understand that waiting without news is hard. Here is how we approach it. We publish a public update when there is new information we can stand behind, and we would rather not post just for the sake of posting. General information goes on our website. Anything specific to your service is answered in your ticket, because that is where we can look at your exact server. Live chat has been available the whole time.

We said we would answer questions in this thread, and we are doing that: we follow it, and this Q&A is part of it.

Other providers

Other service providers have already restored their dedicated servers. So, do you really have any grounds left to make excuses?

@duonghowuaaxtnw763 @Hondura @caccamo
As @aj_potc pointed out above, every company faced a different level of damage, so a direct comparison doesn't tell you much. We'd add that smaller companies have an easier time handling an incident like this.

Before the incident

When did VSYS become aware of that increased risk?

@caccamo
The risk has been elevated since 2022, February. Our other locations have always been available.

I migrated out of vsys about a month ago… I feared something like this could happen

@snowman11
A pity it wasn't to one of our other locations.

The important stuff

Is the new facility drone-proof?

@rpqu
YES!

Provider: we were bombed. MJJ: any deals?

@fensuiji @Murv @zed
Yes, of course. There will be Black Friday discounts in all locations, and there will be a large reserve of servers for them.

What's next

Many more of you took part in this discussion: @RAVNIX @OpaqueRegistrant @TimboJones @Levi @forest @Webby123 @luckypenguin @MikeA @Mainfrezzer @lirrr @deafcon @Arirang @JohnFilch123 @oloke @stefeman @rcy026 @host_c @Saragoldfarb @ScreenReader @Obelous @VOR_SYS @Verunix @Zyra @itzsenu @tentor @Noah2002 @hesweb

If you, or anyone else reading this, still have questions, please post them here. We will collect them over the next few days and come back with another round of answers on anything that is still unclear.

Yes actually. Dunno if you're physically in UA but if you are... How are you holding up? Hope you're doing fine. Stay safe!

vsys_host · October 7

@Hondura said:

@vsys_host said:
Thank you all for your questions. There have been quite a few, so we’d like to take the time to answer them properly.

We’re gathering your questions into a single Q&A and will post the answers in this thread tomorrow, covering as much as we can.

If there’s something you’d like us to include, feel free to add it below. Please keep your questions thoughtful, clear, and relevant so we can make the Q&A as useful as possible for everyone.

Another day has passed. Even though it was announced that many open questions would be answered today, once again, nothing happened.

For more than two weeks, we have known that the data centers in Ukraine were attacked. Please stop repeating this information over and over again. Every affected customer already knows what happened. We do not need another explanation of what happened — we need concrete information about what has actually happened with our hardware and our data since then.

In more than two weeks, there have basically been only two updates. Even in the VSYS panel, there is hardly any real progress visible. The status has simply changed from “Waiting for inspection” to “Waiting for intact” or similar status messages.

For us as customers, the situation is still:

Waiting for inspection: No ETA.
Waiting for intact: More than two weeks with no visible progress.
Are the servers and data safe? “We can only determine this after inspection.”
When will the inspection take place? No ETA.
When will the servers be brought back online? No ETA.

The main problem is no longer only the original incident. The communication afterwards has now become a serious problem as well.

If, after more than two weeks, the affected servers have apparently still not been fully tested, while at the same time there is still no concrete ETA for inspection, relocation, or restoration, then we have to ask: What actual progress has been made during these two weeks?

We are constantly being asked to remain patient. At the same time, businesses and livelihoods depend on these servers and the data stored on them. Every additional day without clear information causes further business damage.

Therefore, we finally need concrete answers to the following questions:

Have the affected servers actually been physically inspected and tested yet?

Have the storage drives been checked, and can anything already be said about the condition and integrity of the data?

Are the servers still at the original location, or have they already been transported to another data center?

If they have not yet been transported, what exactly is currently preventing the relocation?

When will the technical inspection of the servers that are still waiting for inspection actually begin?

Is there at least an approximate timeframe — are we talking about days, weeks, or months?

What happens if the inspection shows that some or all of the data can no longer be recovered?

What compensation or account credit will be provided to customers who have been unable to access their paid Dedicated Servers for weeks, or who may have permanently lost their data?

We are not asking for unrealistic promises or an exact date if one genuinely cannot be provided at this time. However, after more than two weeks, it should be possible to transparently explain what has already been completed, what is currently being done, what still needs to be done, and what exactly is currently blocking the recovery process.

“No ETA,” “please be patient,” and repeatedly referring to the attack do not answer these questions.

I am writing this publicly because, at this point, I no longer expect these questions to receive concrete answers publicly anyway. Nevertheless, these questions need to be asked. We are paying customers, and we have the right to know what is happening with our hardware, our data, and the services we have been paying for but have been unable to use for weeks.

After more than two weeks, we do not need another general statement.

We need concrete answers, transparency, and a clear recovery plan.

Our Q&A went up 3 minutes after your post. As far as we can see, it covers questions you raised here.

If anything is still unanswered after you read it, post it here and we will answer.

Noah2002 · October 7

Dear @vsys_host

Kindly check ticket #3944580

I’m still waiting for the server replacement, as the Ukraine server has been down since September 23rd.

Hondura · October 7

@Saragoldfarb said:

@Hondura said:

We need a clear recovery plan.

Just install your backups. Problem solved. Carry on.

@Saragoldfarb said:

@Hondura said:

We need a clear recovery plan.

Just install your backups. Problem solved. Carry on.

Thank you. I honestly would never have thought of that myself. Your information has been incredibly helpful. Of course, I’ll restore my backup right away.

It’s always nice to see that there are such “knowledgeable experts” out there who feel the need to help others so many thanks

vsys_host · October 7

@cockli said: One of my customers has 3 servers on DC1 all running a DB in master/slave. He is OK if VSYS can prioritize the relocation of at least one of the three servers so that he can get the data. Yes he has a backup but not up to date. His backup machine was in DC2 and his offsite backup is few weeks old. He can survive the incident but if it is possible to get the data, even in 5-10 days, he will be more than happy.

>
Please send us a PM here with the ticket number and we'll take a look.

Saragoldfarb · October 7

@Hondura said:

@Saragoldfarb said:

@Hondura said:

We need a clear recovery plan.

Just install your backups. Problem solved. Carry on.

@Saragoldfarb said:

@Hondura said:

We need a clear recovery plan.

Just install your backups. Problem solved. Carry on.

Thank you. I honestly would never have thought of that myself. Your information has been incredibly helpful. Of course, I’ll restore my backup right away.

It’s always nice to see that there are such “knowledgeable experts” out there who feel the need to help others so many thanks

More than welcome love, always happy to help.

vsys_host · October 7

@Noah2002 said:
Dear @vsys_host

Kindly check ticket #3944580

I’m still waiting for the server replacement, as the Ukraine server has been down since September 23rd.

@Noah2002
We checked ticket #3944580. Your replacement request has already been created and is in progress: our engineers will assemble the server in the coming days. The details are in our latest reply in the ticket.

vsys_host · October 7

@Hondura

As far as I can see, you had the most questions in this thread, and many of them made it into the Q&A. We tried to cover the ones that came up most often and mattered to the most people.

If there is still something we haven't answered that matters to you, please tell us here. We do want to hear it.

Noah2002 · October 7

Thank you, @vsys_host.

I’m really impressed with your server uptime over the past year. The reliability has been excellent.

Could you also please let me know the estimated server activation date? and can you please make my this server replacement request as higher priority?

vsys_host · October 7

@Noah2002 said:
Thank you, @vsys_host.

I’m really impressed with your server uptime over the past year. The reliability has been excellent.

Could you also please let me know the estimated server activation date? and can you please make my this server replacement request as higher priority?

Thank you!

Give us a few minutes: we are posting an update in your ticket with the expected timeframe, and you will see it there.

GN9101 · October 7

I understand there is no ETA for restoring the dedicated servers because it depends on rack availability, but is there an ETA for giving users access to their server data? Some of us, for whatever reason, didn't have offsite backups and are completely dependent on that data. Is it foolish not to have a backup? Yes, we know that, and lecturing people about it is pointless. So my question is: is there an ETA on retrieving our data so we can deploy it on another server?

NyXo · October 8

@vsys_host said:

Q&A: your questions from this thread, answered

We read the whole thread and collected the questions that came up most often. Here are the answers, grouped by topic.

Dedicated servers: timelines

You already have the physical servers, you've found a new facility, you know which servers were affected, and there's still no ETA?

@GN9101 @UserUk @viper877 @Hondura
Every server we have checked so far is working, so we expect the affected share to be small: more than 90% of servers should be unaffected.

What holds things up is space, not hardware. We did find space, and all of it went to bringing cloud and VPS back online. One facility is now completely full. New racks for dedicated servers are being prepared and we expect them in about two weeks, although that doesn't depend on us alone. Moving servers abroad also takes time and involves factors outside our control.

Some customers have already been restored or moved to a replacement server.

What several means? If just relocation might take 5-6-7 weeks, then what about the servers from DC2?

@cockli
By "several weeks" we mean two to three. About two weeks for the new racks to be ready, then about 5 days to install the servers. Not all of this depends on us, but on our side we are doing everything we can to speed it up.

How many servers from DC1 need relocation, how many can be relocated per day, do they manage a priority list?

@cockli

All DC1 servers need relocation. The exact number is internal, so we won't publish it.

One facility is already full, and we are waiting for two more.

Once a facility is ready, we need about 5 days to install all the servers.

There is no priority list for restoring service: we physically move everything in one go.

Replacements do follow an order: by the date of the request and as matching components become available.

If DC1 servers are intact as claimed, why are they still offline? At this point, DC1 and DC2 are in the same boat.

@GN9101 @tangying @Hondura
In DC1 the cooling system was destroyed. The servers are intact, but they can't run there. The DC2 facility was destroyed completely.

Does anyone actually have a server in DC1? Does that group even exist in the first place?

@GN9101 @caccamo
Yes, it does. See the answers above, and post in this thread about a DC1 server that has already been relocated and is working.

DC2: water, inspection, data

Part of the buildings burned down, and the remaining part was flooded with water and fire foam. What kind of restoration are you talking about?

@jenkki
All of our equipment has been taken out of that data center and moved to safe storage. Visually it is undamaged. There are traces of water, but that should not affect the data. We have already checked most of the storage media, and all of it works.

What will happen to the servers that come into contact with water?

@pawntruw @viper877
We will service them. If a customer needs their disks, we simply replace the server. We have enough hardware; what we are short of is data center space to put it in.

What if, after two weeks, they say, "Yes, they're damaged and can't be recovered"?

@Hondura
We have checked a large number of SSDs, and so far 100% of them work. We haven't reached the HDDs yet, but we expect the same picture there.

Has VSYS successfully accessed data from any of the recovered storage devices? Is there any known reason why customers may not be able to recover their data?

@caccamo
Yes, we have. Every storage device we have checked is undamaged and working. The rest are in our storage and visually undamaged. The wait comes down to space: we can't connect all servers at once, so customers are in a queue.

Does VSYS currently have a realistic expectation that customers will be able to recover their data, or not?

@caccamo
Yes, we do. We expect 99% of customers to get their data. We believe all SSDs are intact, and most HDDs. Some HDDs may have been affected by the shockwave, but we haven't come across any so far.

"X servers have been checked so far, and we are processing approximately X servers per day"

@caccamo
The queues are moving:

data handover: ≈90% of those who requested it

replacements: ≈46% of those who requested one

full VPS and Cloud services restoration: up to 10%

full dedicated services restoration: up to 10%

They're just claiming it's an inspection because they don't want to admit outright that all the data are gone.

@Hondura @GN9101
If the data were lost, we wouldn't be telling you we expect 99% recovery. Our entire virtual infrastructure has been restored, and not from backups. Customer dedicated servers were in the same racks as that infrastructure.

The provider will never tell you, "Sorry, we're closed, all your data is lost."

@jenkki
We're not closed. VPS and Cloud services are available to order, servers will be back in stock soon, we have three other locations, and a new one is on the way: Warsaw.

Getting your data before the server is back

If full server restoration will take months, will customers be given an opportunity to retrieve their data before then?

@caccamo @cockli @GN9101 @Hondura
Yes. The procedure is already running. Everyone who contacted us about their data is on the waiting list, and some customers have already received theirs.

I only need one SQL backup of approximately 300 MB.

@caccamo
Please send me here or via PM your ticket number. A request for a specific file is processed faster than a request for a full disk dump.

I submitted a ticket, their reply was identical to the one from the 26th… no estimate…

@longqin2021
Please send us a PM here with your ticket number. We will look into your case and do our best to sort it out.

Is it possible for the customer to verify the integrity ourself?

@rpqu
If you're in Ukraine, yes. Please contact us to arrange it.

Moving to another location

Would you be able to migrate VPS's to NL…?

@serverenthusiast
Yes. Everyone who asked for this has either been moved already or is waiting for components to become available for their request.
If you need it - let us know via ticket.

Colocation

Do you have any news for colo customers? Any hope for them?

@mudassar788
For colo it's simple: send a courier and we will hand over your hardware. We can have it ready the next business day.

Support hours

You still force us to wait for the Sales Department and only during business hours.

@psyco23
Our technical team is working on data recovery 24/7. Sales and billing are working extended hours as well.

About communication

We also see what many of you wrote here about communication: that updates were too rare, that ticket replies read like templates, that chat wasn't working, and that news about VPS was of little use to those waiting for a dedicated server.

We understand that waiting without news is hard. Here is how we approach it. We publish a public update when there is new information we can stand behind, and we would rather not post just for the sake of posting. General information goes on our website. Anything specific to your service is answered in your ticket, because that is where we can look at your exact server. Live chat has been available the whole time.

We said we would answer questions in this thread, and we are doing that: we follow it, and this Q&A is part of it.

Other providers

Other service providers have already restored their dedicated servers. So, do you really have any grounds left to make excuses?

@duonghowuaaxtnw763 @Hondura @caccamo
As @aj_potc pointed out above, every company faced a different level of damage, so a direct comparison doesn't tell you much. We'd add that smaller companies have an easier time handling an incident like this.

Before the incident

When did VSYS become aware of that increased risk?

@caccamo
The risk has been elevated since 2022, February. Our other locations have always been available.

I migrated out of vsys about a month ago… I feared something like this could happen

@snowman11
A pity it wasn't to one of our other locations.

The important stuff

Is the new facility drone-proof?

@rpqu
YES!

Provider: we were bombed. MJJ: any deals?

@fensuiji @Murv @zed
Yes, of course. There will be Black Friday discounts in all locations, and there will be a large reserve of servers for them.

What's next

Many more of you took part in this discussion: @RAVNIX @OpaqueRegistrant @TimboJones @Levi @forest @Webby123 @luckypenguin @MikeA @Mainfrezzer @lirrr @deafcon @Arirang @JohnFilch123 @oloke @stefeman @rcy026 @host_c @Saragoldfarb @ScreenReader @Obelous @VOR_SYS @Verunix @Zyra @itzsenu @tentor @Noah2002 @hesweb

If you, or anyone else reading this, still have questions, please post them here. We will collect them over the next few days and come back with another round of answers on anything that is still unclear.

hello, #2931665 please.

TimboJones · October 8

@Hondura said:
You dog. I'll fuck your keyboard—stop spamming the thread with your retarded behavior.

You must have the world's tiniest dick.

TimboJones · October 8

@caccamo said:
I’m not saying this proves anything. I’m simply saying that after two weeks, I think it’s perfectly reasonable for customers to start asking what is actually happening behind those identical answers.

And since some people here seem so interested in LLMs, here’s an idea: ask your LLM what it thinks when a company keeps giving almost identical answers to different customers for weeks without providing measurable progress.

Maybe this time you’ll find the LLM useful too.

LLM said you're an idiot for expecting a different answer to different customers for the same situation. Seriously, use your brain.

Levi · October 8

@Hondura said:

@Levi said:

@Hondura said:

As loyal customers, we are reaching out to you today with several urgent questions, as the current situation is causing significant uncertainty among our end customers as well.
Please provide us with direct answers regarding the following points:

Condition and Data Recovery (DC2): How high do you estimate the chances are that the affected servers from DC2 will return to regular operation? We understand that the hardware must be inspected for water damage. However, should the inspection reveal that the data cannot be recovered, what is your specific emergency protocol or subsequent course of action in this case?

Billing and Contracts: Our next server payments are currently due. How will the offsetting of downtime be handled, given that the systems cannot be utilized at this moment?

Timeline for Provisioning: More than two weeks have passed since the incident occurred. How much more time is the setup of the new infrastructure expected to take until we can rely on a stable environment again?
Since this is a business-critical inquiry, we kindly request an official response directly from the @vsys_host

Thank you for your efforts and your prompt reply.

Xexe lao ban!

You dog. I'll fuck your keyboard—stop spamming the thread with your retarded behavior.

MJJ, are you mad?!

vsys_host · October 8

@GN9101 said:
I understand there is no ETA for restoring the dedicated servers because it depends on rack availability, but is there an ETA for giving users access to their server data? Some of us, for whatever reason, didn't have offsite backups and are completely dependent on that data. Is it foolish not to have a backup? Yes, we know that, and lecturing people about it is pointless. So my question is: is there an ETA on retrieving our data so we can deploy it on another server?

Please send me a PM with your ticket number, and I will look into your case and see what can be done.

vsys_host · October 8

@NyXo said:

@vsys_host said:

Q&A: your questions from this thread, answered

We read the whole thread and collected the questions that came up most often. Here are the answers, grouped by topic.

Dedicated servers: timelines

You already have the physical servers, you've found a new facility, you know which servers were affected, and there's still no ETA?

@GN9101 @UserUk @viper877 @Hondura
Every server we have checked so far is working, so we expect the affected share to be small: more than 90% of servers should be unaffected.

What holds things up is space, not hardware. We did find space, and all of it went to bringing cloud and VPS back online. One facility is now completely full. New racks for dedicated servers are being prepared and we expect them in about two weeks, although that doesn't depend on us alone. Moving servers abroad also takes time and involves factors outside our control.

Some customers have already been restored or moved to a replacement server.

What several means? If just relocation might take 5-6-7 weeks, then what about the servers from DC2?

@cockli
By "several weeks" we mean two to three. About two weeks for the new racks to be ready, then about 5 days to install the servers. Not all of this depends on us, but on our side we are doing everything we can to speed it up.

How many servers from DC1 need relocation, how many can be relocated per day, do they manage a priority list?

@cockli

All DC1 servers need relocation. The exact number is internal, so we won't publish it.

One facility is already full, and we are waiting for two more.

Once a facility is ready, we need about 5 days to install all the servers.

There is no priority list for restoring service: we physically move everything in one go.

Replacements do follow an order: by the date of the request and as matching components become available.

If DC1 servers are intact as claimed, why are they still offline? At this point, DC1 and DC2 are in the same boat.

@GN9101 @tangying @Hondura
In DC1 the cooling system was destroyed. The servers are intact, but they can't run there. The DC2 facility was destroyed completely.

Does anyone actually have a server in DC1? Does that group even exist in the first place?

@GN9101 @caccamo
Yes, it does. See the answers above, and post in this thread about a DC1 server that has already been relocated and is working.

DC2: water, inspection, data

Part of the buildings burned down, and the remaining part was flooded with water and fire foam. What kind of restoration are you talking about?

@jenkki
All of our equipment has been taken out of that data center and moved to safe storage. Visually it is undamaged. There are traces of water, but that should not affect the data. We have already checked most of the storage media, and all of it works.

What will happen to the servers that come into contact with water?

@pawntruw @viper877
We will service them. If a customer needs their disks, we simply replace the server. We have enough hardware; what we are short of is data center space to put it in.

What if, after two weeks, they say, "Yes, they're damaged and can't be recovered"?

@Hondura
We have checked a large number of SSDs, and so far 100% of them work. We haven't reached the HDDs yet, but we expect the same picture there.

Has VSYS successfully accessed data from any of the recovered storage devices? Is there any known reason why customers may not be able to recover their data?

@caccamo
Yes, we have. Every storage device we have checked is undamaged and working. The rest are in our storage and visually undamaged. The wait comes down to space: we can't connect all servers at once, so customers are in a queue.

Does VSYS currently have a realistic expectation that customers will be able to recover their data, or not?

@caccamo
Yes, we do. We expect 99% of customers to get their data. We believe all SSDs are intact, and most HDDs. Some HDDs may have been affected by the shockwave, but we haven't come across any so far.

"X servers have been checked so far, and we are processing approximately X servers per day"

@caccamo
The queues are moving:

data handover: ≈90% of those who requested it

replacements: ≈46% of those who requested one

full VPS and Cloud services restoration: up to 10%

full dedicated services restoration: up to 10%

They're just claiming it's an inspection because they don't want to admit outright that all the data are gone.

@Hondura @GN9101
If the data were lost, we wouldn't be telling you we expect 99% recovery. Our entire virtual infrastructure has been restored, and not from backups. Customer dedicated servers were in the same racks as that infrastructure.

The provider will never tell you, "Sorry, we're closed, all your data is lost."

@jenkki
We're not closed. VPS and Cloud services are available to order, servers will be back in stock soon, we have three other locations, and a new one is on the way: Warsaw.

Getting your data before the server is back

If full server restoration will take months, will customers be given an opportunity to retrieve their data before then?

@caccamo @cockli @GN9101 @Hondura
Yes. The procedure is already running. Everyone who contacted us about their data is on the waiting list, and some customers have already received theirs.

I only need one SQL backup of approximately 300 MB.

@caccamo
Please send me here or via PM your ticket number. A request for a specific file is processed faster than a request for a full disk dump.

I submitted a ticket, their reply was identical to the one from the 26th… no estimate…

@longqin2021
Please send us a PM here with your ticket number. We will look into your case and do our best to sort it out.

Is it possible for the customer to verify the integrity ourself?

@rpqu
If you're in Ukraine, yes. Please contact us to arrange it.

Moving to another location

Would you be able to migrate VPS's to NL…?

@serverenthusiast
Yes. Everyone who asked for this has either been moved already or is waiting for components to become available for their request.
If you need it - let us know via ticket.

Colocation

Do you have any news for colo customers? Any hope for them?

@mudassar788
For colo it's simple: send a courier and we will hand over your hardware. We can have it ready the next business day.

Support hours

You still force us to wait for the Sales Department and only during business hours.

@psyco23
Our technical team is working on data recovery 24/7. Sales and billing are working extended hours as well.

About communication

We also see what many of you wrote here about communication: that updates were too rare, that ticket replies read like templates, that chat wasn't working, and that news about VPS was of little use to those waiting for a dedicated server.

We understand that waiting without news is hard. Here is how we approach it. We publish a public update when there is new information we can stand behind, and we would rather not post just for the sake of posting. General information goes on our website. Anything specific to your service is answered in your ticket, because that is where we can look at your exact server. Live chat has been available the whole time.

We said we would answer questions in this thread, and we are doing that: we follow it, and this Q&A is part of it.

Other providers

Other service providers have already restored their dedicated servers. So, do you really have any grounds left to make excuses?

@duonghowuaaxtnw763 @Hondura @caccamo
As @aj_potc pointed out above, every company faced a different level of damage, so a direct comparison doesn't tell you much. We'd add that smaller companies have an easier time handling an incident like this.

Before the incident

When did VSYS become aware of that increased risk?

@caccamo
The risk has been elevated since 2022, February. Our other locations have always been available.

I migrated out of vsys about a month ago… I feared something like this could happen

@snowman11
A pity it wasn't to one of our other locations.

The important stuff

Is the new facility drone-proof?

@rpqu
YES!

Provider: we were bombed. MJJ: any deals?

@fensuiji @Murv @zed
Yes, of course. There will be Black Friday discounts in all locations, and there will be a large reserve of servers for them.

What's next

Many more of you took part in this discussion: @RAVNIX @OpaqueRegistrant @TimboJones @Levi @forest @Webby123 @luckypenguin @MikeA @Mainfrezzer @lirrr @deafcon @Arirang @JohnFilch123 @oloke @stefeman @rcy026 @host_c @Saragoldfarb @ScreenReader @Obelous @VOR_SYS @Verunix @Zyra @itzsenu @tentor @Noah2002 @hesweb

If you, or anyone else reading this, still have questions, please post them here. We will collect them over the next few days and come back with another round of answers on anything that is still unclear.

hello, #2931665 please.

Thank you, got it. I'm checking the details of your request now and will get back to you in the ticket as soon as I have an answer.

caccamo · October 8

@vsys_host thank you for reply, this is my Ticket number #1338883

vsys_host · October 8

@caccamo said:
@vsys_host thank you for reply, this is my Ticket number #1338883

Got it! Answered

hyperblast · October 8

one of my favorite w****sites shows this info:

Dear Administrators,

Update October 07, 2026:
The server hardware has been retrieved and is being moved to a secure data center for technical inspection, 
as it may have been exposed to water. It can't be powered on before the inspection. There is no confirmed data
 loss, but the condition of the drives can't be confirmed until the inspection is completed. We'll reach out to you 
as soon as the inspection is done

Due to an impact of the drones, there were some hardware that suffered water damage. We have to check 
these first before we can start them.

We apologize for the inconvenience.

Unfortunately, it was almost impossible to prevent it, as two of our locations were affected.

pawntruw · October 8

@vsys_host Could you please check my ticket number #9387870

LEBUserJoe · October 8

Long Live VSYS

When I was a tech noobie, and only knew Hetzner and OVH they was the only one I knew that would host me, and not terminate me instantly for a false DMCA.

There services has always been little pricey to me since I evolved, but I remember them being an insanely rock-solid and reliable host.

For those still asking for status of data, legit fuck yourself. Let's see your home get him by a missile and the first thing i'll ask as your picking up the pieces is do you still have my toothbrush?

it's a business, your 2k+ server you pay $70/month for, there concern is the hardware water damage occured, sudden shutdowns. Your data may be gone expect this and nothing less until you hear more simple.

This is for you @Hondura I also want to educate you for free, VSYS never guranteed backups for you, therefore there are no backups. You data is gone if you hear different it's a bonus otherwise it's gone move on with ur life and let them recover.

vsys_host · October 8

@LEBUserJoe

Thank you for the kind words and the support, it means a lot to us.

One correction, though: the data is not gone. As we shared in our Q&A above, every drive we have checked so far is working and we expect 99% of customers to get their data back.

vsys_host · October 8

@pawntruw said:
@vsys_host Could you please check my ticket number #9387870

Answered

vsys_host · October 8

@caccamo said:
@vsys_host thank you for reply, this is my Ticket number #1338883

Answered

rpqu · October 8

@vsys_host If you have the time, please update your signature. The new DC isn't online yet, right?

Howdy, Stranger!

Quick Links

Categories

In this Discussion

VSYS Host: UA data centers. What happened and where recovery stands?

Comments

Q&A: your questions from this thread, answered

Dedicated servers: timelines

DC2: water, inspection, data

Getting your data before the server is back

Moving to another location

Colocation

Support hours

About communication

Other providers

Before the incident

The important stuff

What's next

Q&A: your questions from this thread, answered

Dedicated servers: timelines

DC2: water, inspection, data

Getting your data before the server is back

Moving to another location

Colocation

Support hours

About communication

Other providers

Before the incident

The important stuff

What's next

Q&A: your questions from this thread, answered

Dedicated servers: timelines

DC2: water, inspection, data

Getting your data before the server is back

Moving to another location

Colocation

Support hours

About communication

Other providers

Before the incident

The important stuff

What's next

Q&A: your questions from this thread, answered

Dedicated servers: timelines

DC2: water, inspection, data

Getting your data before the server is back

Moving to another location

Colocation

Support hours

About communication

Other providers

Before the incident

The important stuff

What's next