All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
HostHatch disk zero fill via guest-exec
Trying to figure out if this legit or some sort of preparation for an attack. I'm going to reach out to their support but wanted to see if anyone else has seen something like this.
From 1:43 AM to 1:51 AM Pacific time, I saw a large amount of RAM usage, swap usage, disk usage, and 100% usage of one CPU core on one of my HostHatch systems, far outside normal usage from this VPS.


It turns out all the usage was coming from guest-agent:

Analyzing just the Netdata data with AI (GPT-6.1 Sol in OpenCode) told me:
This looks like a temporary bulk-write job launched through QEMU Guest Agent—likely free-space zeroing or a storage test
...
- Three additional processes appeared under the guest-agent service.
- Logical reads and writes were almost exactly equal, but physical disk reads were essentially zero.
- Disk space consumption grew continuously, then disappeared when the job ended.
- CPU time was almost entirely spent in the kernel.
That is highly consistent with a command resembling
ddreading/dev/zerointo a temporary file, followed by deletion. A hosting platform can launch commands through the guest agent’s guest-exec interface. Free-space zeroing before image compaction or backup is one plausible reason; a write benchmark is another.
This is exactly what I found in the logs. qemu-guest-agent.service logs show a process starting, then pings every 10 seconds or so to check its status:
2026-10-29T01:43:12-0700 hostname qemu-ga[870814]: info: guest-exec called: "/bin/bash -c echo CnNldCAtZQoKZWNobyAiIyMjIGhvc3RuYW1lIgpob3N0bmFtZSB8fCB0cnVlCgplY2hvCmVjaG8gIiMjIyBkZiBiZWZvcmUiCmRmIC1oVCB8fCB0cnVlCgplY2hvCmVjaG8gIiMjIyB6ZXJvIGZpbGwgdG8gOTUlIG9mIGF2YWlsYWJsZSBzcGFjZSBvbiAvIgpBVkFJTF9LQj0kKGRmIC0tb3V0cHV0PWF2YWlsIC8gfCB0YWlsIC0xKQpGSUxMX0tCPSQoKCBBVkFJTF9LQiAqIDk1IC8gMTAwICkpCkZJTExfQkxPQ0tTPSQoKCBGSUxMX0tCIC8gNDA5NiApKQppZiBbICIkRklMTF9CTE9DS1MiIC1sdCAxIF07IHRoZW4KICBlY2hvICJOb3QgZW5vdWdoIGZyZWUgc3BhY2UgdG8gemVybyBmaWxsLCBza2lwcGluZyBkZCIKZWxzZQogIGRkIGlmPS9kZXYvemVybyBvZj0vdG1wL3plcm8gYnM9NE0gY291bnQ9JHtGSUxMX0JMT0NLU30gc3RhdHVzPXByb2dyZXNzIHx8IHRydWUKZmkKCmVjaG8KZWNobyAiIyMjIHJlbW92ZSB6ZXJvIGZpbGUiCnJtIC1mIC90bXAvemVybwpzeW5jCgplY2hvCmVjaG8gIiMjIyBmc3RyaW0gYWxsIG1vdW50cG9pbnRzIgpmc3RyaW0gLWF2IC0tbWluaW11bT0xMzEwNzIgfHwgdHJ1ZQoKZWNobwplY2hvICIjIyMgZGYgYWZ0ZXIiCmRmIC1oVCB8fCB0cnVlCg== | base64 -d | bash"
2026-10-29T01:43:12-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:43:23-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:43:33-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:43:44-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:43:54-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:44:05-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
2026-10-29T01:44:15-0700 hostname qemu-ga[870814]: info: guest-exec-status called, pid: 1965503
Decoding that script shows that it's not malicious, but instead just a script to fill the root partition to 95% of available space:
set -e
echo "### hostname"
hostname || true
echo
echo "### df before"
df -hT || true
echo
echo "### zero fill to 95% of available space on /"
AVAIL_KB=$(df --output=avail / | tail -1)
FILL_KB=$(( AVAIL_KB * 95 / 100 ))
FILL_BLOCKS=$(( FILL_KB / 4096 ))
if [ "$FILL_BLOCKS" -lt 1 ]; then
echo "Not enough free space to zero fill, skipping dd"
else
dd if=/dev/zero of=/tmp/zero bs=4M count=${FILL_BLOCKS} status=progress || true
fi
echo
echo "### remove zero file"
rm -f /tmp/zero
sync
echo
echo "### fstrim all mountpoints"
fstrim -av --minimum=131072 || true
echo
echo "### df after"
df -hT || true
I can't find this script on GitHub which makes me think it's a custom script.
What's unclear is if HostHatch did this intentionally (e.g. if they want to TRIM their SSDs) or if someone had access to their control plane and is just doing this to test that executing arbitrary commands works. I'll reach out to their support of course, but I just wanted to check if anyone else has seen something similar.
In the meantime, I'm blocking guest-exec in /etc/qemu/qemu-ga.conf just in case, until I figure out the purpose of this.
[general]
block-rpcs = guest-exec,guest-exec-status
Comments
@forest care for comment ? any analysis, tip etc.
This is a part of our old live migration script that will be phased out soon and replaced with fstrim only.
We live migrate VMs between nodes from time to time (especially to do node maintenances with no customer impact). Live migration generally makes the disk non-sparse and this makes it sparse again.
uhm wtf
Thanks for the explanation! Makes sense.
It would be great if you could add a comment to the top of the script explaining that, so that people that see it in their logs don't freak out