All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Neutralizing the security risk of QEMU Guest Agent without removing it
TL;DR instead of removing the qemu-guest-agent package which can limit your VPS' ability to integrate with the provider dashboard, you can prevent it from performing malicious actions by running:
cat << EOF > /etc/qemu/qemu-ga.conf
[general]
block-rpcs=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-set-user-password,guest-exec-status,guest-exec,guest-ssh-add-authorized-keys,guest-ssh-remove-authorized-keys,guest-get-users
EOF
systemctl restart qemu-guest-agent.service
Be aware that this does not provide protection against a genuinely malicious host. It does not prevent them from making a memory dump, mounting your storage device directly, or even attaching to QEMU with a debugger.
Background
QEMU Guest Agent is a daemon that comes pre-installed on most OS templates and even auto-installs during installation from ISO on most Linux distros when running virtualized. It's a daemon that allows the host to send commands over an RPC interface to the guest (typically a virtual serial device). Usually this is used for benign purposes such as freezing the guest filesystem for migration, displaying detailed usage stats, and rebooting from the dashboard. However, there have been instances of providers spying on running processes to look for torrent clients and resetting the root password in order to log in without permission using the guest agent.
For security reasons, it's common practice to uninstall the package, qemu-guest-agent, as soon as you get a new VPS. This has a few downsides, though. Although shutting down from the dashboard can be done by sending an ACPI shutdown signal, some providers only use the agent to perform a clean shutdown. If sending the signal fails, the VPS will then be subject to a hard shutdown which can cause filesystem corruption. In addition, removing the agent can prevent you from seeing detailed usage information on the dashboard.
Securing QEMU Guest Agent
It's not actually necessary to fully remove the daemon. It's possible to apply a whitelist or blacklist the RPC commands that are allowed. The complete list of RPC commands and their descriptions are available in the QEMU Guest Agent protocol reference. A few random examples include:
guest-get-host-name: Get the guest's current hostnameguest-shutdown: Request a graceful shutdownguest-set-user-password: Set a new password for any user (including root)guest-exec: Execute an arbitrary command as root
It's immediately obvious that some of these are harmless (or even beneficial) whereas others allow arbitrary code execution. It's possible to arbitrarily restrict which RPC commands are allowed. To block the most dangerous RPC commands, create a file /etc/qemu/qemu-ga.conf with contents such as:
[general]
block-rpcs=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-set-user-password,guest-exec-status,guest-exec,guest-ssh-add-authorized-keys,guest-ssh-remove-authorized-keys,guest-get-users
This will prevent the agent from being commanded to open or operate on local files, change your password, add an SSH authorized key, list current users, or execute arbitrary commands. It will still allow other benign tasks to be performed, such as requesting a graceful shutdown or reconfiguring the number of vCPUs.
You can also whitelist commands instead, if you know which exact ones you need. For example, if all you want is to allow graceful shutdowns and to have detailed usage information available on the dashboard:
[general]
allow-rpcs=guest-sync-delimited,guest-sync,guest-ping,guest-info,guest-shutdown,guest-get-disks,guest-get-host-name,guest-get-osinfo,guest-get-diskstats,guest-get-cpustats,guest-get-load
And if you want to allow nothing at all beyond graceful shutdown support, you would do:
[general]
allow-rpcs=guest-sync-delimited,guest-sync,guest-ping,guest-shutdown
Now restart the daemon and your new restrictions should be applied. You can run qemu-ga -D to display the current config to verify that it was set correctly. If the host attempts to send an RPC command that is not allowed, the guest agent will refuse it. The host cannot, using just the guest agent, force a command to be executed.
Caveats
Host communication with the guest through QEMU Guest Agent is cooperative (it requires guest support) and the daemon can refuse to execute commands. By restricting the commands it can execute, you reduce the ability for a host to casually snoop through your stuff. It does not prevent a malicious host from messing with your VPS if they really want to. You can get data-at-rest security by using full-disk encryption (FDE), but even that does not protect from the host directly tampering with guest memory. It's still a good idea on its own, though!
To fully protect from a malicious host, you would need a host that supports running the guest in a confined trusted execution environment (TEE) where memory content itself is encrypted, such as a host that supports AMD SEV-SNP. You'd additionally need to set up remote attestation and use FDE. The big hyperscalars often support this (well, kind of...), but the only provider on LET which can do this, to the best of my knowledge, is Onidel (@onidel and @oloke). SEV-SNP can only be broken by sophisticated attackers with physical access.
If you don't need that level of cryptographically-verified security, it's still a good idea to limit QEMU Guest Agent's ability to execute arbitrary commands on your VPS, either by restricting commands or by removing it entirely.
And Cloudinit is another subject altogether...
Comments
Good suggestions.
If average LET hosts shared best practices like @forest, security incidents will be fewer
Just wait until I post a guide on how to tell if a host is leaking your traffic to other tenants and how to determine if it's possible to use ARP spoofing to completely redirect traffic from another VM to yours.
It's actually a lot of hosts...
Please make executable/auto config because some of the host probably don't understand half of what you said.
Then they can ask AI for help?
If they can ask AI for help, the complaints could get solved in a week
is this one the reason my vps seems to use fat amount of bandwidth in member area dashboard when in hetrixtools says it's idling?
Probably not, but that could be another issue like an ARP flood or unknown-unicast traffic. Try running:
If there's a huge flood of traffic that you're receiving despite the destination IP being not yours, that would answer it. Sometimes that happens on nodes with poorly-configured networks.
Sadly the solution to that is very platform dependent. Or just plain nftables.
I have one VPS where a) receive about 500kbit/s of ARP traffic(!) and b) not only ARP but STP and VRRP too. But hey, at least they have redundant routers, so not all is lost
That won't help much from the guest's side. It'll still get dropped by the networking stack, but it uses up your port's bandwidth, and often your monthly traffic allowance too. Plus it comes with security issues. Only real solution is to get the provider to fix their network.
I've seen quite a few providers that unintentionally expose L2 management plane traffic like that.
I meant on the host. I think ebtables is now also superseeded by nftables?
Ah, yeah the host will generally fix it through their Proxmox GUI though and won't know how to use low-level firewall configs. At least most low-end hosts that you find here. I'm still trying to convince one low-end provider (albeit not on LET) that he doesn't need a separate vNIC for IPv6.
And ebtables is indeed superseded by nftables. Pretty much all network filtering is, although sadly there are still some things iptables can do by virtue of xtables that nftables cannot.
Very informative
Easier to just remove it and not bother, graceful shutdown works anyway via ACPI. Just make sure to run
acpid. Won't get VPS metrics in provider panel, but why visit that more often than once a year only to pay for renewal anyway.Some providers won't even send an ACPI shutdown signal, they'll just send an RPC command to qemu-ga and then, if the system hasn't powered down, it does a hard power off.
It is unfortunate if those exist
Anyways typically shouldn't need provider-side shutdown more often than 1-2 times early in the VPS life as well.
Except when the node goes down for a reboot. Even if you don't use the panel, there can be reasons for shutting down the VM. Now, I personally don't use qemu-ga at all, I have it totally removed, but some people like managing things through their dashboard, I suppose.
Like once on a huge European provider with a three letter name where you would not expect this to happen....
Yeah... that is way too common.
Good OP and points, @forest!
I personally would not disable 'guest-file-flush' because it merely flushes (open files) which I don't consider dangerous but rather potentially useful.
Thanks!
That's true, but it only works on a handle returned by
guest-file-open, which is blacklisted. In theory you'd only need to blacklist the open command and that would neutralize all the other file-related commands.For syncing all files, you could use one of the fsfreeze-related RPC commands. Those sync the filesystem and then freeze it, which is useful when migrating between nodes without shutting down the VM.
It would be nice if the config allowed you to specify whitelisted shell commands that
guest-execwas allowed to run and restrict arguments and environment. That way you could permitguest-execbut only if it issues/bin/syncwith an empty environment. I guess you could do that using an AppArmor profile against the guest agent, but that would be more complex.@forest if you ever start a VPS host, let me know. You seem like you're more competent than most hosts here
I've actually been considering setting up a very small one. Not to earn money of course (no way I can compete in this market), just to learn. I was thinking of using a gimmick to increase the expert-to-MJJ ratio, like having the entire purchase process, tickets, and configuration be done exclusively over an interactive SSH session (no WHMCS or anything like that). No templates, just installation via an ISO URL and serial over SSH. Maybe I'll offer heavy discounts for anyone who runs a Tor middle relay on their home internet or something. But so far it's just a vague idea, no concrete plans.
That sounds like shellbox.dev. If you want to run a host, I'd recommend to lease hypervisor from:
US: @crunchbits / @DartNode / @TierNet / @georgedatacenter (For hot SSD backup, I have told you the host, right?)
APAC: @bbmmsvr4u / @trumvps / @Advin / @ddps
EU: too many lol
I was considering colo at a local datacenter if I end up doing it (so in the US), since I have hardware already.
lolololol. But, you won't be anonymous then
Could narrow me down to my state, sure. It'd take more than OSINT to get more than that, though.
But this is still just a vague idea.
Ok. But it's better if you could borrow identity from someone. Maybe @beanman109 's AS could be used for real purpose. Use more shell corps. Get a clueless single mother as the director
No. The average LET host would lead to more incidents.
Maybe @JoshR can help you here.