New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Data corruption on Kvm/Xen/Openvz vps container
Let's suppose that we have three vps containers with untouched operating system on different virtualizations such as KVM/Xen/OpenVZ
Each vps containers are hosted with different providers.
After a month all of the vps containers went down. All vps providers has reported that data on vps was corrupted for unknown reasons. Providers didn't find the issue on their side as they said that they are using Raid 10 setup or etc.
What's the possible reason that data was corrupted on all three vps containers ? Vps containers weren't used at all. Is virtualization type makes sense when a data is lost ?
Thank you.
Comments
What's possible reason of data loss. Is this virtualization or hardware issue?
Thank you.
Containers aren't well secured and has been compromised by hacker.
Hmm.. I experienced something like this with relatively known reputable US OpenVZ VPS provider some week ago. I am not bothered much as it was more or less idle vps (and I am not complaining) but this was their response in case you're interested:
Slightly more detailed anwer to my question what happened:
(OpenVZ/SolusVM)
Server was properly secured, did not host anything and I doubt about intrusion. Host also confirmed: "The host node is running normally. I checked other VPSs on the node and they are operating as expected."
Provider is competent and node isn't overloaded btw.
This could be hardware compatibility issue (raid + ssd) or buggy kernel
It's most likely the template. We had to drop centos 5 32bit due to template issues. Data corrupts on vps reboot.
Yup, @jack pointed out this url https://bugzilla.openvz.org/show_bug.cgi?id=2647
(Debian 7 template - I used)
[My Fault], playing with IPconfig tried some addon and blocked all the access to the VPS (OpenVZ), so no ssh or anything
But it was fixed for me from support Department.
If you are cut out from ssh you can still access the console. Any decent vps provider will give you console access of some sort.
There's also ipv6.
Yeah, but more than half providers seem not to have it...
What if I did'nt even touch the vps. Should I blame all those providers for loosing my data even if there were nothing ?
It's untouched, minimal linux OS. There is no chances that they were compromised and yes it was just an example.
Exactly, I was pointing to the issue like this.
@concerto49 @Spirit Is there any issues with other linux distros and versions?
@MikeIn @Maounique I'm sorry but all providers has reported that data on vps containers has been lost. It's untouched, minimal linux OS.
Thank you guys!
That does not mean the data is lost. At the very least can tar everything together and put it some place for the customer to take it, they do not need the OS files, but their data files, so, even if the container no longer boots, data should still be salvageable.
So what OS was on these containers? Is it possible that some update auto installed and screwed things (incompatible with the OpenVZ kernel, etc.).
I'm not sure if updates could cause data loss on different virtualizations and even on different operating system for example ?
For different operating systems the updates shouldn't be the common factor, true.
It happened at different providers all at once, so it can't be hardware.
Well, it was just an example
So far the only common factor i see is you. Is it possible that somehow your workstation was compromised and someone used it to login to all these VPSes and delete them?
Yes, its getting interesting and tingling :P
No, I mean all this didn't happen and was supposed to understand the reason if "the shit happens".
I guess this is the only one left, host machine virtualization type.
Well as this never happened and you have ruled out hacking hardware and software.... its not going to happen in 3 location and 3 os types with 3 virt types at the same time.
OpenVZ has the stuck init issue which requires a node reboot since you cant kill an init process.
The others it's possible the server crashed and your disks failed a fsck check, or you turned off VNC so of course nobody can really access it. Another one is Debian 7 if you encrypt the drive and it thinks it's on a new machine it requires a root password to finish booting.
data corruption is a known issue with xen, openvz, kvm
the only virtualization software that does seem to not randomly corrupt is VMWare.
huh?
Well After 4+ years of Xen I can say I have never faced or found a known issue that corrupts data, KVM uses essentially the exact same methods and OpenVZ has no storage specific methods, in fact for KVM+Xen+OpenVZ there is nothing specific to the virt type that handles the storage so to say it is a known issue is to say that Linux itself has a known issue that causes data corruption is its either using LVM or the flat file system of the host OS.
VMware on the other hand has a propitiatory storage system to which I absolutely have seen corruption on due to snapshot failures, driver related issues for raid cards etc and it is much harder to recover from due to its propitiatory nature.
@AnthonySmith - It's possible to corrupt inodes/data on Xen/KVM simply by initializing several requests to resize the main disk at once and then rebooting the virtual server several times to mess up the disk resizing process.
If you want to shoot yourself in the foot, isn't it easier to just cat /dev/zero > your disk?
I don't see how this is Xen or KVM fault.
Oh.. I thought we were listing ways to corrupt data .. maybe I should have read the entire thread
My bad sorry
haha well yes, if anything that re-enforces my point, that is not Xen/KVM/OpenVZ specific.
Probably your provider fooled you