Weird kernel memory usage on VPS

totally_not_banned

Last night i decided to make one of my non-idlers (purchased and never booted) into an active idler and proceeded to dd a basic system image to the disk. That worked without any kind of surprises and the box boots/works fine but the RAM usage is pretty strange.

Since this is pre-built diskimage, which i'm also using elsewhere i know quite well what kind of numbers to expect but on this given host they are way off and weirdly fluctuate. I'm expecting to see ~55mb (or ~65mb using virtio disks) RAM in active (not cache etc) use right after boot but i'm getting ~90mb. At first i thought this might be due the host using ballooning and blacklisting the balloon driver actually brought usage down to ~81mb but this still not really what i'm expecting to see and there's no difference in relation to the running processes.

Another thing that irritates me is that disk access seems to cause the kernel to allocate large chunks of (not cache) memory. Running apt-get update pushes usage up to about ~115mb while apt-get dist-upgrade managed to bring it to about ~170mb.

Usually these allocations seem to persist. Once usage has grown it just stays there but that's seemingly not always the case. After my last boot that ended up with the (for that system) expected ~81mb RAM used i ran apt-get update to check if i could replicate the ~115mb i had seen before and it worked. Now that's not overly surprising but shortly after while i was still watching htop it suddenly and without any visible reason snapped back to 64mb (i.e. lower than right after boot).

What the fuck is going on here? I mean, it's not like i would be short on RAM on that box but this behavior just baffles me and i certainly don't like it... All i can think of is this being somehow related to virtio drivers (which my other box where i'm using the image doesn't have) but i can't replicate it locally with qemu using virtio disks (i'm getting ~65mb here after boot - pretty much what the VPS had after randomly snapping back). It's also kind of interesting that while there's a virtio SCSI controller on the VPS the disk still seems to use qemu's default SATA controller (disk at /dev/sda instead of the expected /dev/vda).

I might be a bit nosy/paranoid but least my gut feeling says that the host is doing something weird here. Balooning shenanigans are never a good sign and their panel is regularly way off too (RAM usage is always - at times 100+mb... - higher than in the actual system and running nothing but htop supposedly comes down to 10-15% use of the single vCPU). Any clues as to what could be going on here or how i'd stop RAM usage from fluctuating so widely?

rm_

I also have noticed once a large difference between idling VPS RAM usage across different hosts. But don't remember details right now, and likely both are not idling anymore anyway.

Try comparing content of /proc/meminfo, to see where exactly the larger usage box has increased numbers.

And reinstall yourself from ISO to see if this persists.

slowservers

I'm curious what your memory usage looks like after a reboot with the balloon module blacklisted.

totally_not_banned

@rm_ said:
I also have noticed once a large difference between idling VPS RAM usage across different hosts. But don't remember details right now, and likely both are not idling anymore anyway.

Try comparing content of /proc/meminfo, to see where exactly the larger usage box has increased numbers.

That's a pretty good idea! Maybe there'll be a at least a hint as to where the additional memory is going.

And reinstall yourself from ISO to see if this persists.

It's already all more or less ISO installs. It's just that the installer ran in a local qemu instance with the resulting diskimage later transplanted onto the various VPS. That's why the systems themself can be pretty much ruled out as far as differences are concerned. Aside from a few lines in /etc/network/interfaces and passwd/shadow they are basically byte for byte identical.

@slowservers said:
I'm curious what your memory usage looks like after a reboot with the balloon module blacklisted.

It's already blacklisted by now. Blacklisting it brought the RAM used right after boot down from ~90mb to ~81mb. This pretty much was my first thought too but it seems the balloon driver isn't the culprit (or at least not the only one).

forest

Post the contents of ps auxf, cat /proc/{meminfo,vmallocinfo,vmstat,slabtop}.

totally_not_banned

.

totally_not_banned

@forest said:
Post the contents of ps auxf, cat /proc/{meminfo,vmallocinfo,vmstat,slabtop}.

• ps auxf: https://pastebin.com/6Vh5cfQg
• cat /proc/meminfo: https://pastebin.com/kXp9jwrY
• cat /proc/vmallocinfo: https://pastebin.com/h4Y5nTjw
• cat /proc/vmstat: https://pastebin.com/eHghsfv6
• cat: /proc/slabtop: No such file or directory
• lspci: https://pastebin.com/qXLZ8r3a

Taken right after boot. Please don't ask me why it now decided to use 78mb. It just felt like it...

I'll try getting data for comparison too but i can't reboot one box since it's actually in use.

totally_not_banned

Basically identical system running locally in qemu (actually uses virtio disks):

• ps auxf: https://pastebin.com/MMvXMpGn
• cat /proc/meminfo: https://pastebin.com/R38UCKQG
• cat /proc/vmallocinfo: https://pastebin.com/GCFFvQHt
• cat /proc/vmstat: https://pastebin.com/Ry31RnEK
• cat: /proc/slabtop: No such file or directory
• lspci: https://pastes.io/MsC59t4y

Edit: added lspci output since i noticed the VPS is actually emulating way more devices even beyond the balloon device.

Saahib

OP feeling bored, lets dig into how linux manages memory as he thinks he knows more than guys how developed linux kernel.

DrNutella

Mythos has freed itself
Decentralized power

totally_not_banned

@Saahib said:
as he thinks he knows more than guys how developed linux kernel.

And you come to that conclusion exactly how? Let me guess, i'm not going to get an answer, right?

In any case i can assure you that i'm not bored. Just because you have no curiosity and don't want to learn how/why things work the way they do doesn't instantly mean that everyone else will want to stay ignorant too.

Edit: Oh, is it because i wrote "kernel memory usage" in the title and you didn't realize that if running processes are the same the additional memory likely got allocated by some kernel component? If that's the case... well, maybe you shouldn't assume to be more clever than the people around you.

Saahib

@totally_not_banned said:

@Saahib said:
as he thinks he knows more than guys how developed linux kernel.

And you come to that conclusion exactly how? Let me guess, i'm not going to get an answer, right?

There are simply too many factors, including host state, where you don't have access. Linux memory management is simple, ie what you need is in memory, what you don't, is in cache, if free ram then also use it as cache but if you deep dive into it, its rabbit hole with so many things.

In your exclusive case, vm first has lot more pci devices (double) than old vm, obviously kernel need more memory to keep record of those. Also the stats you shared, there is only 8-10 mb difference on average on both vm usage (- cache) .

I donno what you trying to solve.

Saahib

Eh.. you edited your post.. without waiting for my reply.. no worry, you were in hurry in jumping to conclusion.

totally_not_banned

@Saahib said:
Eh.. you edited your post.. without waiting for my reply.. no worry, you were in hurry in jumping to conclusion.

Yeah, and it seems i was pretty much on point. If you want to flame and then point out the obvious that's cool but you'll need to find other discussion partners. A little respect goes a long way but seems you don't need it. Oh well...

Saahib

You are totally_bored.
Sadly, I can't continue, have important things those need my attention. Things that really matter.

totally_not_banned

@Saahib said:
You are totally_bored.
Sadly, I can't continue, have important things those need my attention. Things that really matter.

Cool, cool