Neutralizing the security risk of QEMU Guest Agent without removing it

forest

TL;DR instead of removing the qemu-guest-agent package which can limit your VPS' ability to integrate with the provider dashboard, you can prevent it from performing malicious actions by running:

cat << EOF > /etc/qemu/qemu-ga.conf
[general]
block-rpcs=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-set-user-password,guest-exec-status,guest-exec,guest-ssh-add-authorized-keys,guest-ssh-remove-authorized-keys,guest-get-users
EOF
systemctl restart qemu-guest-agent.service

Be aware that this does not provide protection against a genuinely malicious host. It does not prevent them from making a memory dump, mounting your storage device directly, or even attaching to QEMU with a debugger.

Background

QEMU Guest Agent is a daemon that comes pre-installed on most OS templates and even auto-installs during installation from ISO on most Linux distros when running virtualized. It's a daemon that allows the host to send commands over an RPC interface to the guest (typically a virtual serial device). Usually this is used for benign purposes such as freezing the guest filesystem for migration, displaying detailed usage stats, and rebooting from the dashboard. However, there have been instances of providers spying on running processes to look for torrent clients and resetting the root password in order to log in without permission using the guest agent.

For security reasons, it's common practice to uninstall the package, qemu-guest-agent, as soon as you get a new VPS. This has a few downsides, though. Although shutting down from the dashboard can be done by sending an ACPI shutdown signal, some providers only use the agent to perform a clean shutdown. If sending the signal fails, the VPS will then be subject to a hard shutdown which can cause filesystem corruption. In addition, removing the agent can prevent you from seeing detailed usage information on the dashboard.

Securing QEMU Guest Agent

It's not actually necessary to fully remove the daemon. It's possible to apply a whitelist or blacklist the RPC commands that are allowed. The complete list of RPC commands and their descriptions are available in the QEMU Guest Agent protocol reference. A few random examples include:

• guest-get-host-name: Get the guest's current hostname
• guest-shutdown: Request a graceful shutdown
• guest-set-user-password: Set a new password for any user (including root)
• guest-exec: Execute an arbitrary command as root

It's immediately obvious that some of these are harmless (or even beneficial) whereas others allow arbitrary code execution. It's possible to arbitrarily restrict which RPC commands are allowed. To block the most dangerous RPC commands, create a file /etc/qemu/qemu-ga.conf with contents such as:

[general]
block-rpcs=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-set-user-password,guest-exec-status,guest-exec,guest-ssh-add-authorized-keys,guest-ssh-remove-authorized-keys,guest-get-users

This will prevent the agent from being commanded to open or operate on local files, change your password, add an SSH authorized key, list current users, or execute arbitrary commands. It will still allow other benign tasks to be performed, such as requesting a graceful shutdown or reconfiguring the number of vCPUs.

You can also whitelist commands instead, if you know which exact ones you need. For example, if all you want is to allow graceful shutdowns and to have detailed usage information available on the dashboard:

[general]
allow-rpcs=guest-sync-delimited,guest-sync,guest-ping,guest-info,guest-shutdown,guest-get-disks,guest-get-host-name,guest-get-osinfo,guest-get-diskstats,guest-get-cpustats,guest-get-load

And if you want to allow nothing at all beyond graceful shutdown support, you would do:

[general]
allow-rpcs=guest-sync-delimited,guest-sync,guest-ping,guest-shutdown

Now restart the daemon and your new restrictions should be applied. You can run qemu-ga -D to display the current config to verify that it was set correctly. If the host attempts to send an RPC command that is not allowed, the guest agent will refuse it. The host cannot, using just the guest agent, force a command to be executed.

Caveats

Host communication with the guest through QEMU Guest Agent is cooperative (it requires guest support) and the daemon can refuse to execute commands. By restricting the commands it can execute, you reduce the ability for a host to casually snoop through your stuff. It does not prevent a malicious host from messing with your VPS if they really want to. You can get data-at-rest security by using full-disk encryption (FDE), but even that does not protect from the host directly tampering with guest memory. It's still a good idea on its own, though!

To fully protect from a malicious host, you would need a host that supports running the guest in a confined trusted execution environment (TEE) where memory content itself is encrypted, such as a host that supports AMD SEV-SNP. You'd additionally need to set up remote attestation and use FDE. The big hyperscalars often support this (well, kind of...), but the only provider on LET which can do this, to the best of my knowledge, is Onidel (@onidel and @oloke). SEV-SNP can only be broken by sophisticated attackers with physical access.

If you don't need that level of cryptographically-verified security, it's still a good idea to limit QEMU Guest Agent's ability to execute arbitrary commands on your VPS, either by restricting commands or by removing it entirely.

And Cloudinit is another subject altogether...

MikeA

Good suggestions.

rpqu

If average LET hosts shared best practices like @forest, security incidents will be fewer

forest

@rpqu said:
If average LET hosts shared best practices like @forest, security incidents will be fewer

Just wait until I post a guide on how to tell if a host is leaking your traffic to other tenants and how to determine if it's possible to use ARP spoofing to completely redirect traffic from another VM to yours.

It's actually a lot of hosts...

rpqu

@forest said:

@rpqu said:
If average LET hosts shared best practices like @forest, security incidents will be fewer

Just wait until I post a guide on how to tell if a host is leaking your traffic to other tenants and how to determine if it's possible to use ARP spoofing to completely redirect traffic from another VM to yours.

It's actually a lot of hosts...

Please make executable/auto config because some of the host probably don't understand half of what you said.

atkl

@rpqu said:
Please make executable/auto config because some of the host probably don't understand half of what you said.

Then they can ask AI for help?

rpqu

@atkl said:

@rpqu said:
Please make executable/auto config because some of the host probably don't understand half of what you said.

Then they can ask AI for help?

If they can ask AI for help, the complaints could get solved in a week

ScreenReader

@forest said:

@rpqu said:
If average LET hosts shared best practices like @forest, security incidents will be fewer

Just wait until I post a guide on how to tell if a host is leaking your traffic to other tenants and how to determine if it's possible to use ARP spoofing to completely redirect traffic from another VM to yours.

It's actually a lot of hosts...

is this one the reason my vps seems to use fat amount of bandwidth in member area dashboard when in hetrixtools says it's idling?

forest

@ScreenReader said: is this one the reason my vps seems to use fat amount of bandwidth in member area dashboard when in hetrixtools says it's idling?

Probably not, but that could be another issue like an ARP flood or unknown-unicast traffic. Try running:

tcpdump -ni eth0 -Q in "not (ip dst $your_ipv4 or ip6 dst $your_ipv6 or multicast)"

If there's a huge flood of traffic that you're receiving despite the destination IP being not yours, that would answer it. Sometimes that happens on nodes with poorly-configured networks.

glueckself

@forest said:

@rpqu said:
If average LET hosts shared best practices like @forest, security incidents will be fewer

Just wait until I post a guide on how to tell if a host is leaking your traffic to other tenants and how to determine if it's possible to use ARP spoofing to completely redirect traffic from another VM to yours.

It's actually a lot of hosts...

Sadly the solution to that is very platform dependent. Or just plain nftables.

I have one VPS where a) receive about 500kbit/s of ARP traffic(!) and b) not only ARP but STP and VRRP too. But hey, at least they have redundant routers, so not all is lost

forest

@glueckself said: Or just plain nftables.

That won't help much from the guest's side. It'll still get dropped by the networking stack, but it uses up your port's bandwidth, and often your monthly traffic allowance too. Plus it comes with security issues. Only real solution is to get the provider to fix their network.

@glueckself said: not only ARP but STP and VRRP too

I've seen quite a few providers that unintentionally expose L2 management plane traffic like that.

glueckself

@forest said:

@glueckself said: Or just plain nftables.

That won't help much from the guest's side. It'll still get dropped by the networking stack, but it uses up your port's bandwidth, and often your monthly traffic allowance too. Plus it comes with security issues. Only real solution is to get the provider to fix their network.

I meant on the host. I think ebtables is now also superseeded by nftables?

forest

@glueckself said:

@forest said:

@glueckself said: Or just plain nftables.

That won't help much from the guest's side. It'll still get dropped by the networking stack, but it uses up your port's bandwidth, and often your monthly traffic allowance too. Plus it comes with security issues. Only real solution is to get the provider to fix their network.

I meant on the host. I think ebtables is now also superseeded by nftables?

Ah, yeah the host will generally fix it through their Proxmox GUI though and won't know how to use low-level firewall configs. At least most low-end hosts that you find here. I'm still trying to convince one low-end provider (albeit not on LET) that he doesn't need a separate vNIC for IPv6.

And ebtables is indeed superseded by nftables. Pretty much all network filtering is, although sadly there are still some things iptables can do by virtue of xtables that nftables cannot.

Cobime32

Very informative

rm_

Easier to just remove it and not bother, graceful shutdown works anyway via ACPI. Just make sure to run acpid. Won't get VPS metrics in provider panel, but why visit that more often than once a year only to pay for renewal anyway.

forest

@rm_ said: graceful shutdown works anyway via ACPI

Some providers won't even send an ACPI shutdown signal, they'll just send an RPC command to qemu-ga and then, if the system hasn't powered down, it does a hard power off.

rm_

@forest said: Some providers won't even send an ACPI shutdown signal, they'll just send an RPC command to qemu-ga and then, if the system hasn't powered down, it does a hard power off.

It is unfortunate if those exist

Anyways typically shouldn't need provider-side shutdown more often than 1-2 times early in the VPS life as well.

forest

@rm_ said: Anyways typically shouldn't need provider-side shutdown more often than 1-2 times early in the VPS life as well.

Except when the node goes down for a reboot. Even if you don't use the panel, there can be reasons for shutting down the VM. Now, I personally don't use qemu-ga at all, I have it totally removed, but some people like managing things through their dashboard, I suppose.

OhJohn

@forest said: If there's a huge flood of traffic that you're receiving despite the destination IP being not yours, that would answer it. Sometimes that happens on nodes with poorly-configured networks.

Like once on a huge European provider with a three letter name where you would not expect this to happen....