Can't allocate cached memory on OpenVZ

fsLeg · May 2014

Hello.

I ran into a problem that forced me to upgrade my plan to a higher one. I kept getting "out of memory" errors even though free command reported enough memory (if cache was included) to run a program. Today I remembered about this issue and did a little research. Here's what I found.

According to linuxatemyram.com, kernel should allow other programs to allocate cached memory. But on my VPS it doesn't. To prove it I ran a little program from here that allocates all available memory. Here are the results:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          2048       1030       1017          0          0        461
-/+ buffers/cache:        568       1479
Swap:            0          0          0

$ ./bin/munch
(...)
Allocated 961 MB

$ free -m
             total       used       free     shared    buffers     cached
Mem:          2048       1032       1015          0          0        464
-/+ buffers/cache:        568       1479
Swap:            0          0          0

As you can see, the program can only allocate unused memory but it should also be able to allocate cached memory.

I wrote about it to my hoster's support and here's their answer:

We use standard OpenVZ kernel with standard setting and standard SolusVM
installation for VPS providing, so our VPs servers runs in the same way
as any other provider OpenVZ based VPS servers.
This includes also cache memory usage etc. We have not received any
other complain about memory usage or cache memory etc in a whole active
period of our services. This means that if there is any memory issue
(your guess that there is any) then this may be isolated and affected
only in your VPS server.
There is no memory or cache memory issues from our side related to
your VPS.

So I think there are three possible answers to this issue:

That's a normal behavior for OpenVZ.
Something is wrong with my VPS.
My hoster is a douche and forces its clients to buy more expensive plans.

Which one is it? Or maybe it's something else? I honestly don't know.

Some details about my VPS:

Debian Wheezy 32bit template, most likely official and unmodified. I also had this problem with Squeeze template.
Kernel 2.6.32-042stab085.17.
2 Gb RAM (was 1 Gb, but I had to upgrade), no burst, no swap.
No system tweaks.

I want to hear opinion of people that are more experienced with OpenVZ VPS.

linuxthefish · May 2014

Running that script on my OpenVZ VPS:

root@de:~# free -m
             total       used       free     shared    buffers     cached
Mem:           512        508          3          0          0          0
-/+ buffers/cache:        507          4
Swap:         1024        486        537

tommy · May 2014

try ps_mem.py script

fsLeg · May 2014

@tommy said:
try ps_mem.py script

Tried it, it showed what I expected: there's enough memory even if I was running 1 Gb VPS.

# ./ps_mem.py 
 Private  +   Shared  =  RAM used   Program

  4.0 KiB +   8.0 KiB =  12.0 KiB   nes-server.sh
  4.0 KiB +  18.0 KiB =  22.0 KiB   upstart-socket-bridge
  4.0 KiB +  19.5 KiB =  23.5 KiB   upstart-udev-bridge
  4.0 KiB +  34.0 KiB =  38.0 KiB   tt-rss-updater.
  4.0 KiB +  36.0 KiB =  40.0 KiB   xinetd
  0.0 KiB +  54.5 KiB =  54.5 KiB   ntpd (2)
  0.0 KiB +  71.5 KiB =  71.5 KiB   udevd (3)
 60.0 KiB +  34.0 KiB =  94.0 KiB   sh
 64.0 KiB +  34.0 KiB =  98.0 KiB   pokemon-showdow
 88.0 KiB +  25.5 KiB = 113.5 KiB   vsftpd
104.0 KiB +  34.0 KiB = 138.0 KiB   mysqld_safe
132.0 KiB +  54.0 KiB = 186.0 KiB   anvil
148.0 KiB +  54.0 KiB = 202.0 KiB   log
196.0 KiB +  19.0 KiB = 215.0 KiB   kaillerasrv
300.0 KiB +  27.5 KiB = 327.5 KiB   dbus-daemon
244.0 KiB + 111.5 KiB = 355.5 KiB   cron
296.0 KiB + 113.0 KiB = 409.0 KiB   su
372.0 KiB + 109.5 KiB = 481.5 KiB   nmbd
436.0 KiB +  59.0 KiB = 495.0 KiB   dovecot
232.0 KiB + 269.0 KiB = 501.0 KiB   saslauthd (2)
576.0 KiB + 120.5 KiB = 696.5 KiB   sudo
548.0 KiB + 154.0 KiB = 702.0 KiB   init
796.0 KiB +  66.5 KiB = 862.5 KiB   freshclam
388.0 KiB + 479.5 KiB = 867.5 KiB   smbd (2)
  1.1 MiB + 257.5 KiB =   1.4 MiB   screen (5)
  1.5 MiB +  71.5 KiB =   1.6 MiB   named
  1.4 MiB + 266.5 KiB =   1.6 MiB   exim4
  1.5 MiB + 112.0 KiB =   1.6 MiB   irssi
856.0 KiB + 841.0 KiB =   1.7 MiB   bash (3)
744.0 KiB +   1.0 MiB =   1.7 MiB   sshd (3)
  1.7 MiB +  67.5 KiB =   1.7 MiB   config
  1.9 MiB + 125.0 KiB =   2.0 MiB   rsyslogd
  1.6 MiB + 680.5 KiB =   2.2 MiB   imap-login (3)
  2.7 MiB +   1.0 MiB =   3.7 MiB   imap (3)
  2.8 MiB +   1.5 MiB =   4.3 MiB   nginx (2)
  6.0 MiB + 341.5 KiB =   6.3 MiB   murmurd
  4.0 MiB +   5.0 MiB =   9.0 MiB   /usr/sbin/spamd
  9.3 MiB +   1.9 MiB =  11.2 MiB   php5 (2)
 12.9 MiB +   4.9 MiB =  17.7 MiB   php5-fpm (4)
 40.0 MiB + 131.0 KiB =  40.1 MiB   mysqld
 48.5 MiB +   8.8 MiB =  57.3 MiB   spamd child (2)
247.9 MiB +   3.9 MiB = 251.8 MiB   nodejs (5)
---------------------------------
                        423.9 MiB
=================================
# free -m
             total       used       free     shared    buffers     cached
Mem:          2048        908       1139          0          0        391
-/+ buffers/cache:        516       1531
Swap:            0          0          0

perennate · May 2014

Not sure what you asked hoster the first time, but have you tried asking them to investigate why your VM is unable to allocate cached memory. Also try sync; echo 3 > /proc/sys/vm/drop_caches (maybe won't work in openvz)

rds100 · May 2014

What program gave you the "out of memory" message? Maybe it was another resource limit that was hit, not the total memory.

fsLeg · May 2014

perennate said: Not sure what you asked hoster the first time

I pretty much asked them the first three paragraphs of my initial post.

perennate said: Also try sync; echo 3 > /proc/sys/vm/drop_caches

Tried that, didn't work. I also asked the hoster about that, support's reply:

Regarding cache clearing, then in OpenVZ virtualization VPS servers,
this operation is not possible at the VPS level. This may be possible
only at the node level to clear ALL node cache memory (not a single
VPS cache). Also, this is not a solution, because cache memory is not
really used and is freed when any other program needs this memory.

perennate said: have you tried asking them to investigate why your VM is unable to allocate cached memory

My VPS is unmanaged, so basically I'm on my own, and after a reply I quoted in my first post I doubt they would even bother.

fsLeg · May 2014

rds100 said: What program gave you the "out of memory" message

Any program that required more memory than was unused. Namely clamav and a node.js app. And they explicitly told me "out of memory", so I don't think it's anything else than actually "out of memory".

rds100 · May 2014

Still check /proc/user_beancounters especially the last collumn and see which resource could not be allocated.

fsLeg · May 2014

rds100 said: and see which resource could not be allocated

Too bad it's too late. Like I said, I upgraded from 1 Gb RAM to 2 Gb, so no "out of memory" errors anymore. But I'll check it if I start running out of memory again.

sman · May 2014

There is a possiblity the provider doesn't have the Vswap checked in SolusVM config for that node which would result in 2.6.18 kernel beancounter settings configured for VMs.

It's an easy mistake to make when setting up a new node. If they are even remotely competent I would imagine they would have noticed it fairly quickly and/or gotten multiple complaints and figured it out so I doubt that's the problem.

Is this a new VPS and possibly a new node you were put on when you had the problem?

fsLeg · May 2014

sman said: Is this a new VPS and possibly a new node you were put on when you had the problem?

I've been using the same VPS for about two years. No complains besides this one.

I noticed the problem a while ago but didn't give it any attention. I thought I squeezed all potential of my VPS. Only recently it actually became an issue, when I upgraded node.js and wasn't able to run an important service anymore.

I googled a bit and checked /proc/vz/vswap file's presence. There's no such file. So my guess is they don't use VSwap.

perennate · May 2014

fsLeg said: I googled a bit and checked /proc/vz/vswap file's presence. There's no such file. So my guess is they don't use VSwap.

I don't have either on VM I know uses vSwap, so..

fsLeg · May 2014

perennate said: I don't have either on VM I know uses vSwap, so..

Well, I don't understand that thing anyway.

I mentioned that there's no swap on my VPS, and googling revealed that swap is a mandatory parameter when when node uses VSwap. I dunno though if swap can be set to 0 when using VSwap.

Also I don't understand how VSwap is related to apps not being able to allocate cached memory.

perennate · May 2014

The issue isn't really vSwap feature itself. New version of OpenVZ supports better memory management. I think if hosting provider is using SolusVM, they need to check the box so that it uses the newer thing (which doesn't support old kernels).

I have never interacted with OpenVZ myself so not too clear on details.

AThomasHowe · May 2014

-what am I smoking? snip.-

fsLeg · May 2014

AThomasHowe said: Could it be that your cache is in active use though?

I don't think that's how cache works. Those services should only use memory they allocated.

I asked support about VSwap, and they said my node was using VSwap. They also said the problem could be that my VPS was created quite some time ago, back when 2.6.18 kernel was in use. They changed my server's config to VSwap style, and the problem seems to be gone.

Thank you, everyone, for your replies.

AThomasHowe · May 2014

Yeah I read your question wrong, that's why I edited out. Guess you saw it before that.

sman · May 2014

If it worked before and stopped working after you did something to your VM then it's your VM...duh. Occam's razor blah blah.

Never seen a node spontaneously start messing up VMs memory and beancounters.

Rallias · May 2014

If I remember correctly, what you're describing is the host node running out of memory, which OpenVZ advertised as 'unusable cache'.

perennate · May 2014

Rallias said: If I remember correctly, what you're describing is the host node running out of memory, which OpenVZ advertised as 'unusable cache'.

If it works after they changed it to use new memory management system, then it doesn't make sense that it would be caused by host node running out of memory, since the problem wouldn't have been fixed if that's all they did.

Howdy, Stranger!

Categories

In this Discussion

Can't allocate cached memory on OpenVZ

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

Can't allocate cached memory on OpenVZ

Comments