Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Can't allocate cached memory on OpenVZ
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Can't allocate cached memory on OpenVZ

fsLegfsLeg Member

Hello.

I ran into a problem that forced me to upgrade my plan to a higher one. I kept getting "out of memory" errors even though free command reported enough memory (if cache was included) to run a program. Today I remembered about this issue and did a little research. Here's what I found.

According to linuxatemyram.com, kernel should allow other programs to allocate cached memory. But on my VPS it doesn't. To prove it I ran a little program from here that allocates all available memory. Here are the results:

$ free -m
             total       used       free     shared    buffers     cached
Mem:          2048       1030       1017          0          0        461
-/+ buffers/cache:        568       1479
Swap:            0          0          0

$ ./bin/munch
(...)
Allocated 961 MB

$ free -m
             total       used       free     shared    buffers     cached
Mem:          2048       1032       1015          0          0        464
-/+ buffers/cache:        568       1479
Swap:            0          0          0

As you can see, the program can only allocate unused memory but it should also be able to allocate cached memory.

I wrote about it to my hoster's support and here's their answer:

We use standard OpenVZ kernel with standard setting and standard SolusVM
installation for VPS providing, so our VPs servers runs in the same way
as any other provider OpenVZ based VPS servers.
This includes also cache memory usage etc. We have not received any
other complain about memory usage or cache memory etc in a whole active
period of our services. This means that if there is any memory issue
(your guess that there is any) then this may be isolated and affected
only in your VPS server.
There is no memory or cache memory issues from our side related to
your VPS.

So I think there are three possible answers to this issue:

  1. That's a normal behavior for OpenVZ.
  2. Something is wrong with my VPS.
  3. My hoster is a douche and forces its clients to buy more expensive plans.

Which one is it? Or maybe it's something else? I honestly don't know.

Some details about my VPS:

  • Debian Wheezy 32bit template, most likely official and unmodified. I also had this problem with Squeeze template.
  • Kernel 2.6.32-042stab085.17.
  • 2 Gb RAM (was 1 Gb, but I had to upgrade), no burst, no swap.
  • No system tweaks.

I want to hear opinion of people that are more experienced with OpenVZ VPS.

Comments

  • Running that script on my OpenVZ VPS:

    root@de:~# free -m
                 total       used       free     shared    buffers     cached
    Mem:           512        508          3          0          0          0
    -/+ buffers/cache:        507          4
    Swap:         1024        486        537
    
  • tommytommy Member

    try ps_mem.py script

    Thanked by 1deptadapt
  • fsLegfsLeg Member

    @tommy said:
    try ps_mem.py script

    Tried it, it showed what I expected: there's enough memory even if I was running 1 Gb VPS.

    # ./ps_mem.py 
     Private  +   Shared  =  RAM used   Program
    
      4.0 KiB +   8.0 KiB =  12.0 KiB   nes-server.sh
      4.0 KiB +  18.0 KiB =  22.0 KiB   upstart-socket-bridge
      4.0 KiB +  19.5 KiB =  23.5 KiB   upstart-udev-bridge
      4.0 KiB +  34.0 KiB =  38.0 KiB   tt-rss-updater.
      4.0 KiB +  36.0 KiB =  40.0 KiB   xinetd
      0.0 KiB +  54.5 KiB =  54.5 KiB   ntpd (2)
      0.0 KiB +  71.5 KiB =  71.5 KiB   udevd (3)
     60.0 KiB +  34.0 KiB =  94.0 KiB   sh
     64.0 KiB +  34.0 KiB =  98.0 KiB   pokemon-showdow
     88.0 KiB +  25.5 KiB = 113.5 KiB   vsftpd
    104.0 KiB +  34.0 KiB = 138.0 KiB   mysqld_safe
    132.0 KiB +  54.0 KiB = 186.0 KiB   anvil
    148.0 KiB +  54.0 KiB = 202.0 KiB   log
    196.0 KiB +  19.0 KiB = 215.0 KiB   kaillerasrv
    300.0 KiB +  27.5 KiB = 327.5 KiB   dbus-daemon
    244.0 KiB + 111.5 KiB = 355.5 KiB   cron
    296.0 KiB + 113.0 KiB = 409.0 KiB   su
    372.0 KiB + 109.5 KiB = 481.5 KiB   nmbd
    436.0 KiB +  59.0 KiB = 495.0 KiB   dovecot
    232.0 KiB + 269.0 KiB = 501.0 KiB   saslauthd (2)
    576.0 KiB + 120.5 KiB = 696.5 KiB   sudo
    548.0 KiB + 154.0 KiB = 702.0 KiB   init
    796.0 KiB +  66.5 KiB = 862.5 KiB   freshclam
    388.0 KiB + 479.5 KiB = 867.5 KiB   smbd (2)
      1.1 MiB + 257.5 KiB =   1.4 MiB   screen (5)
      1.5 MiB +  71.5 KiB =   1.6 MiB   named
      1.4 MiB + 266.5 KiB =   1.6 MiB   exim4
      1.5 MiB + 112.0 KiB =   1.6 MiB   irssi
    856.0 KiB + 841.0 KiB =   1.7 MiB   bash (3)
    744.0 KiB +   1.0 MiB =   1.7 MiB   sshd (3)
      1.7 MiB +  67.5 KiB =   1.7 MiB   config
      1.9 MiB + 125.0 KiB =   2.0 MiB   rsyslogd
      1.6 MiB + 680.5 KiB =   2.2 MiB   imap-login (3)
      2.7 MiB +   1.0 MiB =   3.7 MiB   imap (3)
      2.8 MiB +   1.5 MiB =   4.3 MiB   nginx (2)
      6.0 MiB + 341.5 KiB =   6.3 MiB   murmurd
      4.0 MiB +   5.0 MiB =   9.0 MiB   /usr/sbin/spamd
      9.3 MiB +   1.9 MiB =  11.2 MiB   php5 (2)
     12.9 MiB +   4.9 MiB =  17.7 MiB   php5-fpm (4)
     40.0 MiB + 131.0 KiB =  40.1 MiB   mysqld
     48.5 MiB +   8.8 MiB =  57.3 MiB   spamd child (2)
    247.9 MiB +   3.9 MiB = 251.8 MiB   nodejs (5)
    ---------------------------------
                            423.9 MiB
    =================================
    # free -m
                 total       used       free     shared    buffers     cached
    Mem:          2048        908       1139          0          0        391
    -/+ buffers/cache:        516       1531
    Swap:            0          0          0
    
  • perennateperennate Member, Host Rep
    edited May 2014

    Not sure what you asked hoster the first time, but have you tried asking them to investigate why your VM is unable to allocate cached memory. Also try sync; echo 3 > /proc/sys/vm/drop_caches (maybe won't work in openvz)

  • rds100rds100 Member

    What program gave you the "out of memory" message? Maybe it was another resource limit that was hit, not the total memory.

  • fsLegfsLeg Member
    edited May 2014

    perennate said: Not sure what you asked hoster the first time

    I pretty much asked them the first three paragraphs of my initial post.

    perennate said: Also try sync; echo 3 > /proc/sys/vm/drop_caches

    Tried that, didn't work. I also asked the hoster about that, support's reply:

    Regarding cache clearing, then in OpenVZ virtualization VPS servers,
    this operation is not possible at the VPS level. This may be possible
    only at the node level to clear ALL node cache memory (not a single
    VPS cache). Also, this is not a solution, because cache memory is not
    really used and is freed when any other program needs this memory.

    perennate said: have you tried asking them to investigate why your VM is unable to allocate cached memory

    My VPS is unmanaged, so basically I'm on my own, and after a reply I quoted in my first post I doubt they would even bother.

  • fsLegfsLeg Member

    rds100 said: What program gave you the "out of memory" message

    Any program that required more memory than was unused. Namely clamav and a node.js app. And they explicitly told me "out of memory", so I don't think it's anything else than actually "out of memory".

  • rds100rds100 Member

    Still check /proc/user_beancounters especially the last collumn and see which resource could not be allocated.

  • fsLegfsLeg Member

    rds100 said: and see which resource could not be allocated

    Too bad it's too late. Like I said, I upgraded from 1 Gb RAM to 2 Gb, so no "out of memory" errors anymore. But I'll check it if I start running out of memory again.

  • smansman Member
    edited May 2014

    There is a possiblity the provider doesn't have the Vswap checked in SolusVM config for that node which would result in 2.6.18 kernel beancounter settings configured for VMs.

    It's an easy mistake to make when setting up a new node. If they are even remotely competent I would imagine they would have noticed it fairly quickly and/or gotten multiple complaints and figured it out so I doubt that's the problem.

    Is this a new VPS and possibly a new node you were put on when you had the problem?

  • fsLegfsLeg Member

    sman said: Is this a new VPS and possibly a new node you were put on when you had the problem?

    I've been using the same VPS for about two years. No complains besides this one.

    I noticed the problem a while ago but didn't give it any attention. I thought I squeezed all potential of my VPS. Only recently it actually became an issue, when I upgraded node.js and wasn't able to run an important service anymore.


    I googled a bit and checked /proc/vz/vswap file's presence. There's no such file. So my guess is they don't use VSwap.

  • perennateperennate Member, Host Rep

    fsLeg said: I googled a bit and checked /proc/vz/vswap file's presence. There's no such file. So my guess is they don't use VSwap.

    I don't have either on VM I know uses vSwap, so..

  • fsLegfsLeg Member

    perennate said: I don't have either on VM I know uses vSwap, so..

    Well, I don't understand that thing anyway.

    I mentioned that there's no swap on my VPS, and googling revealed that swap is a mandatory parameter when when node uses VSwap. I dunno though if swap can be set to 0 when using VSwap.

    Also I don't understand how VSwap is related to apps not being able to allocate cached memory.

  • perennateperennate Member, Host Rep

    The issue isn't really vSwap feature itself. New version of OpenVZ supports better memory management. I think if hosting provider is using SolusVM, they need to check the box so that it uses the newer thing (which doesn't support old kernels).

    I have never interacted with OpenVZ myself so not too clear on details.

  • AThomasHoweAThomasHowe Member
    edited May 2014

    -what am I smoking? snip.-

  • fsLegfsLeg Member

    AThomasHowe said: Could it be that your cache is in active use though?

    I don't think that's how cache works. Those services should only use memory they allocated.


    I asked support about VSwap, and they said my node was using VSwap. They also said the problem could be that my VPS was created quite some time ago, back when 2.6.18 kernel was in use. They changed my server's config to VSwap style, and the problem seems to be gone.

    Thank you, everyone, for your replies.

  • Yeah I read your question wrong, that's why I edited out. Guess you saw it before that.

  • smansman Member
    edited May 2014

    If it worked before and stopped working after you did something to your VM then it's your VM...duh. Occam's razor blah blah.

    Never seen a node spontaneously start messing up VMs memory and beancounters.

  • RalliasRallias Member

    If I remember correctly, what you're describing is the host node running out of memory, which OpenVZ advertised as 'unusable cache'.

  • perennateperennate Member, Host Rep

    Rallias said: If I remember correctly, what you're describing is the host node running out of memory, which OpenVZ advertised as 'unusable cache'.

    If it works after they changed it to use new memory management system, then it doesn't make sense that it would be caused by host node running out of memory, since the problem wouldn't have been fixed if that's all they did.

Sign In or Register to comment.