New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
today any one server got up or not ?
hey kristina, how you doin
No, the worst part is that you can only confirm whether the drives are damaged after the inspection is done. It’s driving me crazy. What if, after two weeks, they say, "Yes, they’re damaged and can’t be recovered"?
Reading through threads like this makes me wanna cry... People are idiots.
It won't matter, because if your data were important to you, by then you already would have restored your backups to an alternate location and be up and running.
And if your data wasn't important enough to back up, then nothing of value was lost.
Honestly, I’m getting tired — and frankly disgusted — by some of the comments in this thread.
Every time someone asks a legitimate question about recovering data, a few people suddenly appear pretending to be infrastructure architects, explaining backups and disaster recovery to people whose infrastructure they know absolutely nothing about.
You don’t know our networks. You don’t know our architecture. You don’t know what those servers were used for. You don’t know whether they were production servers, backup servers, replicas, secondary nodes, or part of a much larger disaster-recovery strategy.
Yet somehow you’re certain you know exactly what everyone should have done.
There are also perfectly valid reasons for operating infrastructure in Kyiv. Sometimes server location is not a random choice — geographical proximity and latency matter, especially when you need to efficiently reach users in a particular region.
And yes, many of us have disaster-recovery systems and backups.
But disaster recovery does not magically make the data on an affected machine irrelevant. A backup server can itself be affected. A replica can contain newer data. There can be specific files, databases or snapshots worth recovering even when the underlying service has already been rebuilt elsewhere.
None of you know the circumstances of every customer here.
So please stop turning a thread about an actual infrastructure incident into a competition over who can sound like the smartest sysadmin in the room.
Some people here actually work with these systems, have businesses to run, and are trying to obtain useful information about their hardware and data.
The useful questions remain incredibly simple:
Have the inspections progressed?
Has anyone moved beyond “Awaiting Inspection”?
Has anyone with potentially water-exposed hardware recovered their data?
If you have information about that, please share it. That’s useful.
If all you have to contribute is “you should have had backups” without knowing anything about somebody else’s infrastructure, you’re not solving anything. You’re just adding noise.
Maybe spend a little less time diagnosing other people’s infrastructure from a forum post and a little more time dealing with the problems you actually know something about.
We’re here trying to solve a real problem, not audition for the title of smartest sysadmin on LowEndTalk.
Lmao, you're funny.
And I have a billion dollars in my bank.
The team is too busy answering stupid ass comments and tickets. Maybe sometime in the next decade.
Xexe lao ban!
Yes, title of the smartest sysadmin on LowEndTalk belongs to @inaba
Mmmm sure buddy! You got fucked and your data is gone most likely so stop whining and learn how to not be a retard next time
Should have use bunker
We know all about that based on the posts here. People were running businesses out of a datacenter in Ukraine without backups.
People are whining about production data, not their backups.
Be definition, a replica cannot contain newer data. Tell your LLM to do better.
Database replication has been around for decades and is really easy to set up.
We're not talking about hobbyists here. One of the complainers has a business that relies on infrastructure with 100+ TB of storage. Others are clearly running businesses as well.
I'd argue the vast majority of the whiners know about backups, and they have amply demonstrated (through their use of an LLM) that they have the tools to implement them.
They chose not to do so.
And now they take out their frustration on the hosting company.
VSYS mentioned in their statement that DC1 was 100% physically recovered, while DC2 was exposed to water. My question is: does anyone actually have a server in DC1? I've talked to several people with dedicated servers, and curiously enough, nobody is in DC1. Everyone was told that their server is being inspected implying it's in DC2 or directly told they were in DC2. I'd hate to think that statement was just meant to calm people down, but does anyone know for sure if their server is in DC1? Does that group even exist in the first place?
I have servers in both DC1 and DC2, but the ones in DC1 are still offline.
Wanting to recover an original storage device does not demonstrate that no backups existed.
Backup, replication and disaster recovery are not synonyms.
A perfectly legitimate DR design can have:
If my last consistent off-site database backup is from 12:00 and the original server remained operational until 16:30, restoring the business from the 12:00 backup does not suddenly make the original disk worthless.
There may still be 4.5 hours of transactions or changes worth recovering.
That’s literally why RPO exists as a concept.
Similarly, having a disaster-recovery plan means you have a strategy for recovering from a disaster. It does not necessarily mean RPO=0, RTO=0, synchronous geographically distributed replication and zero possible data loss under every failure scenario.
Those are very different engineering and economic requirements.
And recovering the original storage after failover is completely normal. You may want to reconcile databases, recover transaction logs, compare application state, retrieve files created since the last backup, verify integrity, or simply obtain an additional copy before decommissioning the failed system.
So the inference:
“They want their original data → therefore they had no backups”
is technically invalid.
Some people here may indeed have had inadequate backups. I have no idea.
And that’s precisely my point: neither do you.
You cannot determine somebody’s backup topology, RPO, RTO or DR architecture simply from the fact that they want data recovered from an affected server.
As for VSYS, my question remains exactly the same and has nothing to do with this backup debate:
Are inspections progressing, and has anyone whose server required inspection successfully recovered data yet?
That’s the information I’m here for.
And regarding the DC1 question raised above: I personally haven’t seen or heard of any confirmed case of a DC1 dedicated server being brought back online and recovered yet.
We now have at least one person here confirming that they have servers in both DC1 and DC2, but they also report that their DC1 servers are still offline.
So if anyone here actually has a DC1 dedicated server that has already been restored, please say so. That would be genuinely useful information for everyone following the recovery.
I’ve checked both the DC1 and DC2 servers, and neither is online. Vsys is taking its sweet time—which is ruining our business even further—and has only released two updates in two weeks; it’s an absolute disgrace. They’ll likely reply to this thread in three days without providing any clear information. It’s a real shame! We’re only interested in the servers for the sake of the data—nothing more, nothing less. And I hope no other moron chimes in saying they should have used a backup or gone with a different provider; there are plenty of brainless people around here.
I think in an age where you can start up a VPS in a few minutes it's easy to forget that a team of people is having to walk in and out of a DC loading heavy servers onto a truck and drive them to a new location, then after inspection load them onto a new truck and unload them and rack them one by one into a new DC.
How fast could you do it? How fast could you do it 2000 times? Without being prepared for it?
Its a very demanding task, im not sure of VSYSs scale, but anything more than 400 servers is already going to take days
And there might not even be another datacenter able to take it all in, especially if tons of people are rushing in from another datacenter
Most datacenters dont leave a bunch of active power circuits and cabs unused
The main issue is the lack of transparency. If DC1 servers are intact as claimed, why are they still offline? At this point, DC1 and DC2 are in the same boat. People wouldn't be criticizing them if they were transparent about the real status; we would just wait. They're just being way too vague.
They might not have enough cabinets in the new datacenter, and they are most likely still having to check the drives for water damage. They said it will likely be a few weeks.
Truer words have never been written.
That’s the issue right there: we are having to assume everything. If they just spelled it out clearly, nobody would complain, but they’re keeping it all very vague.
One of the servers I manage that was on DC1 has already been relocated and working and I am glad for this. I also have several more servers from DC1 expecting to be relocated. My issue is that the time frame is not well defined. I got a reply in one of my tickets (I manage several servers from several accounts - one account per customer of mine) that relocation will take several weeks. What several means? If just relocation might take 5-6-7 weeks, then what about the servers from DC2 that also need inspection before relocating?
VSYS can be more informative on this matter. How many servers from DC1 need relocation, how many can be relocated per day, do they manage a priority list etc
And if relocation indeed needs several weeks, what about their plan of giving access to data before relocating - for those that need data access?
Lol you guys still struggling? My UA server from is*hosting is back online since a week.
vps or dedicated?
Yeah, that's my reaction to. They are fighting a war and literally got bombed, and people cry about their tickets not getting answered? What the fuck is wrong with this world?
I'm not a customer of or affiliated in any way with @vsys_host, but they have my deepest respect. These guys are not only working to get things back online in a country that is in fact in a raging war, but they also manage to keep a professional and courteous tone in their replies. That's nothing short of admirable.
Vsys server ?
There is one aspect of this discussion that I think deserves a serious clarification, because the repeated argument of “you chose to host in a country at war, therefore you accepted everything that happened” is becoming far too simplistic.
Yes, Ukraine is a country at war.
Nobody is denying that.
And yes, as customers, we accepted a certain level of geopolitical risk when choosing infrastructure located there.
But that cannot be the end of the discussion.
There is an important difference between knowing that a country is at war and knowing that the specific facility hosting your infrastructure has reached a level of risk where its continued operation, physical integrity or ability to protect customer data has become seriously questionable.
Customers do not have access to the same information as the provider.
We don’t know the physical protection of the facility, its internal risk assessments, evacuation plans, power redundancy, emergency procedures, agreements with the datacenter operator, hardware relocation possibilities, or what contingency plans exist if the facility becomes unusable.
The provider does.
And this leads to some legitimate questions.
If the risk surrounding these facilities had materially increased before September 23, when did VSYS become aware of that increased risk?
If they were aware of it, what additional precautions were taken?
Was relocating critical infrastructure considered?
Was customer hardware capable of being relocated proactively?
Were additional off-site copies of operational data considered?
Were customers warned that the risk profile of their location had materially changed?
Were customers advised to migrate, replicate elsewhere, or retrieve additional copies of their data?
Because there is another uncomfortable fact that cannot simply be ignored:
VSYS continued offering services in Ukraine, operating those services and accepting payments for them.
As a customer, when a professional hosting company continues selling and operating infrastructure in a particular location without issuing an exceptional warning that the facility itself has become critically exposed, it is not unreasonable to assume that the provider still considers the service operationally viable within the risk level it is prepared to manage.
That doesn’t mean the provider can guarantee protection against a missile strike.
Obviously it cannot.
And it doesn’t remove the customer’s responsibility to maintain appropriate backups and disaster-recovery procedures.
But responsibility is not binary.
It cannot simply become:
“Ukraine is at war → customer knew everything → provider has no questions to answer.”
There is another point that I think some people here are overlooking.
Being a systems administrator, developer, network engineer or business owner does not automatically make someone a military or geopolitical intelligence analyst.
A customer can reasonably understand that Ukraine is at war without being able to determine whether the threat level around one particular Kyiv datacenter has changed from an accepted operational risk to an imminent danger requiring immediate evacuation of infrastructure.
That is precisely where communication from the company operating the service becomes important.
If VSYS had sent customers an email saying:
“Due to a significant deterioration in the security situation around this facility, we strongly recommend immediately creating additional off-site copies or migrating critical workloads outside Ukraine,”
then customers would have been given specific information on which to make a decision.
If such a warning was never issued, I think it is legitimate to ask why.
And I want to be very clear about something before this is turned into another backup argument:
this is not an attempt to transfer all responsibility from customers to VSYS.
Customers have responsibilities.
Providers have responsibilities too.
What I am asking for is transparency about what happened before the incident, not only after it.
When did the provider’s assessment of the risk to these facilities change?
Were there warning signs that caused concern internally?
Were preventive relocation measures considered?
And if the facility was still considered sufficiently safe to continue commercially operating and billing customers there, then it seems unreasonable to now dismiss every affected customer with the argument that “you knowingly put a server in a war zone.”
You cannot simultaneously argue that customers should have known the facility was too dangerous to trust while the company professionally operating that infrastructure continued to offer the location as a viable commercial service.
Those two positions deserve to be reconciled.
Finally, none of this changes the immediate question many of us are still waiting to have answered:
How many servers requiring inspection have actually been inspected?
Has storage from any potentially water-exposed dedicated server been successfully read?
Has any customer moved from “Awaiting Inspection” to a completed inspection?
Those answers would do far more to help affected customers than another argument about who should have predicted the future better.
Lmao, you are using llm in the wrong way. It induce panic and desperation. The world does not spin around you. Sue them. Moralisation will not restore your data.
Should have sent someone to knock @vsys_host door and ask whether it's possible for customer to verify the integrity ourself