Anyone else affected by the server wipe at Khanwebhost?

houmie

The outage in their Istanbul data centre started at 5th June 21:22.

It seems there was a short circuit and everything has been lost.

After enquiring today about the state, they said:

New server is ready and already working we have deployed your VM and delviered it to you already

"Already?" Sorry for asking. When I tried it, I still couldn't access it via SSH with my public key.

A followup enquiry comes up as bizarre as:

yes due to server crash it losted and we have backups but we are unable to restore it because of virtualizor is not allowing us to restore we are intouch with there devloper team if they can fix issue we will restore it

What does it even mean? It seems all data is lost and they can't fix it. Is anyone else affected here?

Any other provider recommendations for Turkey, please?

febryanvaldo

@houmie said: Any other provider recommendations for Turkey, please?

@BatuCloud maybe.

emgh

Hi houmie

houmie

Hello @emgh
I think your message got lost. I can't see it.

neoh

@houmie Did you get in touch with @AK_KWH ?

emgh

@houmie said:
Hello @emgh
I think your message got lost. I can't see it.

No I just wanted to say hi

zGato

Same happened to me with KWH. I needed to explicitly open a ticket to know what was going on, to just get answered "Turkey server was crashed we deployed a new VM". They deployed me 2 VMs (Texas for some reason) and then followed to deploy them correctly. I don't expect data backups to prevent these failures for the price, but atleast an apology or a message to affected users would be bare minimum. They also for some reason seem to have upgraded the node and they're now offering super cheap promotions for Turkey? coincidence?

Levi

"Due to server crash it losted"... Massive incompetence. Avoid at all costs.

houmie

@neoh said:
@houmie Did you get in touch with @AK_KWH ?

As far as I can tell based on previous support tickets in the last six months, it's always the same person handling them. I'm pretty sure it's just a one man company. Those messages above that I quoted earlier should be from him.

Void

May I ask how much you were paying for that server ? @houmie

plumberg

@jmaxwell said:
May I ask how much you were paying for that server ? @houmie

Millions

JasonM

@houmie said: Anyone else affected by the server wipe at Khanwebhost?

yes. my entire vm data on Khanwebhost lost in this location. No notice. nothing.
I've had last month's backup, so I moved to other host in Europe.

SirFoxy

@plumberg said:

@jmaxwell said:
May I ask how much you were paying for that server ? @houmie

Millions

it losted

AK_KWH

Just turkey was affected because of short circuit our server got dead we have backups but we are unable to restore it there is many errors while restoring maybe its due to Virtualizor version change or backuply we already reported it

yoursunny

@AK_KWH said:
Just turkey was affected because of short circuit our server got dead we have backups but we are unable to restore it there is many errors while restoring maybe its due to Virtualizor version change or backuply we already reported it

The lesson is, you need to test a restore procedure, every time after Virtualizor upgrade.

AK_KWH

@yoursunny said:

@AK_KWH said:
Just turkey was affected because of short circuit our server got dead we have backups but we are unable to restore it there is many errors while restoring maybe its due to Virtualizor version change or backuply we already reported it

The lesson is, you need to test a restore procedure, every time after Virtualizor upgrade.

i dont know about this issues, we never face this before, i will try my best that it never repeats

houmie

The problem is I can't even reinstall it from scratch to use my own backup.
I get these errors on a fresh Debian 11 installation. Has anyone seen this?

root@TR-IST-W-1:~# apt update --fix-missing -y
Hit:1 http://security.debian.org/debian-security bullseye-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

root@TR-IST-W-1:~# apt full-upgrade -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@TR-IST-W-1:~# apt install sudo tmux vim -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package vim is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Unable to locate package tmux
E: Package 'vim' has no installation candidate
root@TR-IST-W-1:~#

Thank you

dahartigan

Losted is the new involucrated.

emg

@yoursunny said:

The lesson is, you need to test a restore procedure, every time after Virtualizor upgrade.

This applies to backups in general. You must test the entire backup/restore process prior to the disaster. Otherwise, you will be fighting two fires - the initial failure and the restore process.

You don't want to read about data losses that I have seen over the years. Many of them were businesses that had a great backup strategy in place, but never confirmed that it actually worked.

I have seen backup products that wrote corrupted data. Yes, real commercial backup products. If data is lost during the backup itself, it was never actually backed up. Would you know if it happened to you? Before you encounter a full-recovery situation?

I saw a failed system where an incompetent employee had destroyed ALL of the backups during a long, unsupervised restoration attempt, but I could bore you all day with similar cases.

Close your eyes and imagine this feeling in the heat of a service outage crisis: Having worked so hard all day and all night to reach the restore point with the boss screaming at you every 30 minutes, you realize that you cannot recover the data. The data is lost forever ... something went wrong in the long backup/restore chain.

Suddenly it seems very very quiet, despite the loud fan noise and phones constantly ringing. At 4:00 AM. :-(

-> Verify your backups! End to end!! Before the disaster happens. Yeah, it is hard work and nobody likes it. Nobody. But ... that doesn't matter. You just gotta do it or you will eventually get burned. Trust me.

anubhavhirani

@emg said:

@yoursunny said:

The lesson is, you need to test a restore procedure, every time after Virtualizor upgrade.

This applies to backups in general. You must test the entire backup/restore process prior to the disaster. Otherwise, you will be fighting two fires - the initial failure and the restore process.

You don't want to read about data losses that I have seen over the years. Many of them were businesses that had a great backup strategy in place, but never confirmed that it actually worked.

I have seen backup products that wrote corrupted data. Yes, real commercial backup products. If data is lost during the backup itself, it was never actually backed up. Would you know if it happened to you? Before you encounter a full-recovery situation?

I saw a failed system where an incompetent employee had destroyed ALL of the backups during a long, unsupervised restoration attempt, but I could bore you all day with similar cases.

Close your eyes and imagine this feeling in the heat of a service outage crisis: Having worked so hard all day and all night to reach the restore point with the boss screaming at you every 30 minutes, you realize that you cannot recover the data. The data is lost forever ... something went wrong in the long backup/restore chain.

Suddenly it seems very very quiet, despite the loud fan noise and phones constantly ringing. At 4:00 AM. :-(

-> Verify your backups! End to end!! Before the disaster happens. Yeah, it is hard work and nobody likes it. Nobody. But ... that doesn't matter. You just gotta do it or you will eventually get burned. Trust me.

GitLab.

AK_KWH

@zGato said:
Same happened to me with KWH. I needed to explicitly open a ticket to know what was going on, to just get answered "Turkey server was crashed we deployed a new VM". They deployed me 2 VMs (Texas for some reason) and then followed to deploy them correctly. I don't expect data backups to prevent these failures for the price, but atleast an apology or a message to affected users would be bare minimum. They also for some reason seem to have upgraded the node and they're now offering super cheap promotions for Turkey? coincidence?

@houmie said:
The problem is I can't even reinstall it from scratch to use my own backup.
I get these errors on a fresh Debian 11 installation. Has anyone seen this?

root@TR-IST-W-1:~# apt update --fix-missing -y
Hit:1 http://security.debian.org/debian-security bullseye-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

root@TR-IST-W-1:~# apt full-upgrade -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@TR-IST-W-1:~# apt install sudo tmux vim -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package vim is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Unable to locate package tmux
E: Package 'vim' has no installation candidate
root@TR-IST-W-1:~#

Thank you

i have replied your ticket with the possible solution B/W its not the issue from our END if any package is missing in template you should have to google it how to install it there are many quick solutions available

emg

@anubhavhirani said:

GitLab.

Laconic.

(... but what does the comment mean?)

JeDaYoshi

@houmie said:
The problem is I can't even reinstall it from scratch to use my own backup.
I get these errors on a fresh Debian 11 installation. Has anyone seen this?

root@TR-IST-W-1:~# apt update --fix-missing -y
Hit:1 http://security.debian.org/debian-security bullseye-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

root@TR-IST-W-1:~# apt full-upgrade -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@TR-IST-W-1:~# apt install sudo tmux vim -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package vim is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Unable to locate package tmux
E: Package 'vim' has no installation candidate
root@TR-IST-W-1:~#

Thank you

Looks as if the template is missing the default Debian repos in /etc/apt/sources.list. What are its contents?

JoeMerit

is this a new thing? servers short circuiting?

SirFoxy

@JoeMerit said:
is this a new thing? servers short circuiting?

short circuit make data losted

zGato

@AK_KWH said:

@zGato said:
Same happened to me with KWH. I needed to explicitly open a ticket to know what was going on, to just get answered "Turkey server was crashed we deployed a new VM". They deployed me 2 VMs (Texas for some reason) and then followed to deploy them correctly. I don't expect data backups to prevent these failures for the price, but atleast an apology or a message to affected users would be bare minimum. They also for some reason seem to have upgraded the node and they're now offering super cheap promotions for Turkey? coincidence?

@houmie said:
The problem is I can't even reinstall it from scratch to use my own backup.
I get these errors on a fresh Debian 11 installation. Has anyone seen this?

root@TR-IST-W-1:~# apt update --fix-missing -y
Hit:1 http://security.debian.org/debian-security bullseye-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

root@TR-IST-W-1:~# apt full-upgrade -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@TR-IST-W-1:~# apt install sudo tmux vim -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package vim is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Unable to locate package tmux
E: Package 'vim' has no installation candidate
root@TR-IST-W-1:~#

Thank you

i have replied your ticket with the possible solution B/W its not the issue from our END if any package is missing in template you should have to google it how to install it there are many quick solutions available

Not entirely sure why quoted but yeah, just use an apt sources generator with Turkey mirror and it's blazing fast. Tho I still think that your answer is a bit arrogant. Not providing templates with bare minimum default debian installation sources.list is a bit of a situation I've never experienced with over 30 providers