KVM issue

Asim

I am supposed to be away from internet because I lost one of my family members a week ago but this problem just popped up and I have no idea how to fix it. I wish it was not critical and this was another one of my test-boxes but its not. Of course its backup'd but help me resolve the issue that I am highlighting below

My KVM is a 512MB box and it was running pretty smooth from a reputable provider. The host had to take it offline for some maintenance (the node) and when the node finally came back up, my VPS wont boot. Its responding to ping but I cannot access it (http, ssh etc).

I logged in to console to see this

[img] http://p.twimg.com/AcDMzBlCMAIYob-.png:large [/img]

Any help? I think its the host but the host says its not them and I should do a full reinstall (wtf!!)

P.S. PLEASE DO NOT ASK TO CONTACT THAT HOST, BECAUSE I JUST DID AND THEY SAY THEY WONT HELP, I SHOULD REINSTALL. ALSO PLEASE DONT ASK WHICH HOST IS THIS. APPRECIATE ANY HELP

rds100

Since you have your busybox shell, can you check ls -l /dev/disk/by-uuid/

Maybe the root filesystem uuid changed for whatever reason?

Asim

Thanks for replying, this is the output of the above command, I dont understand the file exists but still the boot-time error is that it 'does not exist'

http://labs.asimz.com/screenshots/10-18-2011 11-17-08 PM.png

Asim

One thing I noticed is that the VNC port has changed from *:5909 to *:6286
unsure if that is the issue

rds100

One more thing to try - /sbin/fsck.ext3 -f -v /dev/vda1

(i am assuming you were using ext3 file system for / . What distro is it? Could be ext4 instead)

miTgiB

VNC port is irrelivent. Mount system rescue cd so you can fsck your disk

sleddog

Asim said: Thanks for replying, this is the output of the above command, I dont understand the file exists but still the boot-time error is that it 'does not exist'

Actually it's a link, pointing to /dev/vda1

You need to check if /dev/vda1 exists.

ls -l /dev/vda*

Asim

Distro is Ubuntu 10.04 32bit

/sbin/fsck.ext3 -f -v /dev/vda1
returns '/sbin/fsck.ext3: not found'

mounting rescue disk

Asim

Output of
ls -l /dev/vda*

http://labs.asimz.com/screenshots/10-18-2011 11-32-43 PM.png

rds100

From wiki.ubuntu.com "The default file system for installations of Ubuntu 10.04 LTS is ext4, the latest version in the popular series of Linux extended file systems."
So when you can try with fsck.ext4

Asim

@rds100 thanks,

/sbin/fsck.ext4 -f -v /dev/vda1

is now in progress. What should be the next step? will fsck auto-repair volume if its corrupt?

rds100

For some errors it asks if you want a fix to be attempted and you must say yes (y)

Alternatively the fsck command can be started with -y switch, which assumes yes to all questions

If any errors are found and fixed (it should say FILESYSTEM MODIFIED at the end) then run the fsck command one more time. After all fsck is finished just reboot and lets see if the VPS boots normally.

Asim

Thank you @rds100 , so you assume that the issue would autoresolve (hopefully) after fsck. Its taking a bit longer and I will post results as I get them

rds100

Yes, if there were errors on the filesystem and it couldn't be mounted because of these errors then a fsck and repair of the errors should fix the problem.

Asim

Thanks a lot, I will keep this thread updated

Asim

fsck ran, did not prompted to fix anything NOR did it report anything bad/broken

I rebooted using HardDisk again and this time I get the same issue

http://labs.asimz.com/screenshots/10-19-2011 12-29-01 AM.png
It says that the disk file does not exist BUT its there (see screenshot above for the ls output)

rds100

Do you have access to the grub menu? Try to remove the "quiet" boot option of there is one - should give some more verbose output. Also could try directly root=/dev/vda1 instead of the long uuid thing.

NordicVPS

Maybe you should try and wait until your host is done rebuilding the disks and then try restart your VPS and fsck it?

It takes some (very long) time to rebuild RAID5 arrays!

kiloserve

Asim said: Any help? I think its the host but the host says its not them and I should do a full reinstall (wtf!!)

I'm guessing you never rebooted your VPS after the busybox install. The problem isn't your host specifically but rather your boot menu. You probably didn't notice the problem before because the VPS hadn't been rebooted.

Check your boot list to make sure you have the correct OS mounts.

/boot/grub/menu.lst

Most likely the problem is there or in /etc/fstab

Asim

The host later resolved the issue. A special thanks to @rds100

kiloserve said: I'm guessing you never rebooted your VPS after the busybox install.

I never installed busybox, I just dont know where it came from and why it was not booting properly even after a FSCK check on disk

NordicVPS said: Maybe you should try and wait until your host is done rebuilding the disks and then try restart your VPS and fsck it?

Yes, I thought the same but it does not take 5+ hours to rebuild raid array, right? Regardless the issue is resolved by the host.

miTgiB

Asim said: but it does not take 5+ hours to rebuild raid array

Longer sometimes, that is many many disks in software raid, a rebuild/verify can take upto a day or two.

Mon5t3r

Asim said: it does not take 5+ hours to rebuild raid array, right?

it took around 12 hours for me to rebuild 2 VPS Node when i updating virtualizor to a newer version and get the same problem like yours.

you can't do anything except wait the host to solve this problem.

miTgiB said: a rebuild/verify

yes. per LV. one by one