All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Recovered a Windows VPS after QEMU Guest Agent was removed — without reinstalling
Had a pretty annoying but interesting Windows VPS issue this week.
WHAT HAPPENED
Customer lost access to his VPS and asked us to reset the Administrator password.
Normally that's a 30-second job from the panel.
Except the reset kept failing because the QEMU Guest Agent had been removed from inside Windows, so the hypervisor couldn't communicate with the guest OS to change the password anymore.
Then we checked the logs:
- 33,706 failed login attempts
- Windows lockout threshold: 10 attempts
- RDP was still exposed on the default port
So even if we unlocked the account, there was a pretty good chance it would just get locked again almost immediately.
WHAT WE HAD TO WORK WITH
- No working Windows credentials
- No console
- No snapshot
- Panel password reset dead
- Administrator account constantly getting hit with login attempts
- Customer had data on the VPS and wanted to avoid a reinstall
At that point reinstalling Windows was looking like the obvious option.
Before wiping anything, we decided to try recovering the existing installation.
RECOVERY
Booted the VM into Debian Rescue Mode and:
- Mounted the Windows NTFS volume
- Backed up the Windows registry files
- Repaired the Administrator account offline
- Unlocked the account
- Reset the account so we could regain access
That created another problem.
Windows remote authentication doesn't particularly like the temporary state created during an offline account recovery.
So temporarily we:
- Adjusted the Windows remote authentication settings
- Allowed the temporary recovery login
- Moved RDP from the default port to 53991
Moving the port was mainly to stop the account getting instantly hit and locked again while we were trying to get back in.
ONCE WE HAD ONE WORKING CREDENTIAL
Things became much easier.
The VM still had SMB and Windows remote-management services reachable, so instead of continuing to fight everything through Rescue Mode, we could finish most of the repair remotely.
WINDOWS BEING WINDOWS
There were a couple stupid issues in between.
One recovery attempt failed with:
Service Error 1053
because of the way we initially tried to launch the recovery script through a Windows service.
Then another batch script got partially eaten because one of the Windows account-management commands decided to throw an interactive prompt halfway through execution.
That one was especially fun to figure out.
FINAL RESULT
Got everything back without reinstalling Windows.
- No reinstall
- No customer files lost
- Existing Windows installation survived
- Administrator access restored
- QEMU Guest Agent reinstalled
- Panel password reset working again
- Normal Windows authentication settings restored
- Temporary recovery settings removed
- RDP restored to the default port
Was fully expecting this one to end with "recover whatever we can and reinstall Windows", so getting the original machine back intact was pretty satisfying.
Anyone here had to recover a Windows VM in a similar situation where the guest agent was completely gone and there was no console access?
Comments
We always use system-rescue iso, mount the drive, reset the admin password, disable the network and use VNC to access the server change RDP port, lock the down with the firewall to certain IP addresses only such as the customers current one even as a short measure, change the admin username to something random that wont be on most automated lists and restore the network connection at node level.
We have been employed a few times to do this externally for others on VM and Dedicated servers
It was my first time handling something like this