HostHatch disk zero fill via guest-exec

Daniel15

Trying to figure out if this legit or some sort of preparation for an attack. I'm going to reach out to their support but wanted to see if anyone else has seen something like this.

From 1:43 AM to 1:51 AM Pacific time, I saw a large amount of RAM usage, swap usage, disk usage, and 100% usage of one CPU core on one of my HostHatch systems, far outside normal usage from this VPS.

It turns out all the usage was coming from guest-agent:

Analyzing just the Netdata data with AI (GPT-6.1 Sol in OpenCode) told me:

This looks like a temporary bulk-write job launched through QEMU Guest Agent—likely free-space zeroing or a storage test
...

• Three additional processes appeared under the guest-agent service.
• Logical reads and writes were almost exactly equal, but physical disk reads were essentially zero.
• Disk space consumption grew continuously, then disappeared when the job ended.
• CPU time was almost entirely spent in the kernel.

That is highly consistent with a command resembling dd reading /dev/zero into a temporary file, followed by deletion. A hosting platform can launch commands through the guest agent’s guest-exec interface. Free-space zeroing before image compaction or backup is one plausible reason; a write benchmark is another.

This is exactly what I found in the logs. qemu-guest-agent.service logs show a process starting, then pings every 10 seconds or so to check its status:

2026-10-29T01:43:12-0700 hostname qemu-ga[870814]: info: guest-exec called: "/bin/bash -c echo CnNldCAtZQoKZWNobyAiIyMjIGhvc3RuYW1lIgpob3N0bmFtZSB8fCB0cnVlCgplY2hvCmVjaG8gIiMjIyBkZiBiZWZvcmUiCmRmIC1oVCB8fCB0cnVlCgplY2hvCmVjaG8gIiMjIyB6ZXJvIGZpbGwgdG8gOTUlIG9mIGF2YWlsYWJsZSBzcGFjZSBvbiAvIgpBVkFJTF9LQj0kKGRmIC0tb3V0cHV0PWF2YWlsIC8gfCB0YWlsIC0xKQpGSUxMX0tCPSQoKCBBVkFJTF9LQiAqIDk1IC8gMTAwICkpCkZJTExfQkxPQ0tTPSQoKCBGSUxMX0tCIC8gNDA5NiApKQppZiBbICIkRklMTF9CTE9DS1MiIC1sdCAxIF07IHRoZW4KICBlY2hvICJOb3QgZW5vdWdoIGZyZWUgc3BhY2UgdG8gemVybyBmaWxsLCBza2lwcGluZyBkZCIKZWxzZQogIGRkIGlmPS9kZXYvemVybyBvZj0vdG1wL3plcm8gYnM9NE0gY291bnQ9JHtGSUxMX0JMT0NLU30gc3RhdHVzPXByb2dyZXNzIHx8IHRydWUKZmkKCmVjaG8KZWNobyAiIyMjIHJlbW92ZSB6ZXJvIGZpbGUiCnJtIC1mIC90bXAvemVybwpzeW5jCgplY2hvCmVjaG8gIiMjIyBmc3RyaW0gYWxsIG1vdW50cG9pbnRzIgpmc3RyaW0gLWF2IC0tbWluaW11bT0xMzEwNzIgfHwgdHJ1ZQoKZWNobwplY2hvICIjIyMgZGYgYWZ0ZXIiCmRmIC1oVCB8fCB0cnVlCg== | base64 -d | bash"
2026-10-29T01:43:12-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:43:23-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:43:33-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:43:44-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:43:54-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:44:05-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:44:15-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503

Decoding that script shows that it's not malicious, but instead just a script to fill the root partition to 95% of available space:

set -e

echo "### hostname"
hostname || true

echo
echo "### df before"
df -hT || true

echo
echo "### zero fill to 95% of available space on /"
AVAIL_KB=$(df --output=avail / | tail -1)
FILL_KB=$(( AVAIL_KB * 95 / 100 ))
FILL_BLOCKS=$(( FILL_KB / 4096 ))
if [ "$FILL_BLOCKS" -lt 1 ]; then
  echo "Not enough free space to zero fill, skipping dd"
else
  dd if=/dev/zero of=/tmp/zero bs=4M count=${FILL_BLOCKS} status=progress || true
fi

echo
echo "### remove zero file"
rm -f /tmp/zero
sync

echo
echo "### fstrim all mountpoints"
fstrim -av --minimum=131072 || true

echo
echo "### df after"
df -hT || true

I can't find this script on GitHub which makes me think it's a custom script.

What's unclear is if HostHatch did this intentionally (e.g. if they want to TRIM their SSDs) or if someone had access to their control plane and is just doing this to test that executing arbitrary commands works. I'll reach out to their support of course, but I just wanted to check if anyone else has seen something similar.

In the meantime, I'm blocking guest-exec in /etc/qemu/qemu-ga.conf just in case, until I figure out the purpose of this.

[general]
block-rpcs = guest-exec,guest-exec-status

tenji

@forest care for comment ? any analysis, tip etc.

hosthatch

This is a part of our old live migration script that will be phased out soon and replaced with fstrim only.

We live migrate VMs between nodes from time to time (especially to do node maintenances with no customer impact). Live migration generally makes the disk non-sparse and this makes it sparse again.

equalz

uhm wtf