OpenVz memory Leak | any one with a permanent fix?

Taz

(solusvm) Master server with 16GB ram with 0 active vm using almost 12GB memory (cache???) for no reason. The only hack (not fix) i have seen is using "sync && echo 2 >/proc/sys/vm/drop_caches" Any one have any fix solution for this error?

This is not related to solusvm but OVZ instead.

rds100

Yes. If you don't want your memory to be used as cache, remove it from the server. "Free memory is wasted memory". Linux will use the free memory for cache. When memory is needed, some of the cache will be removed and there will be enough free memory. Don't worry, everything is OK.

jarland

2.6.32?

Taz

The Ram are there for a reason. So I find your answer a bit vague. how ever, I have never seen that much cached storage by default on any linux install (Centos, debian and ubuntu), So this is obviously a bug and not some de-facto linux issue.

Taz

@jarland yes.

TheHackBox

Ram is cached by default in linux. When the memory is being used, it is removed from the cache. FREE RAM IS WASTED RAM.

jarland

I hit this issue as well. It is a bug. An idle system will snowball in cache and it will eventually become unstable. It is absolutely not a normal function. Clearing the cache every few hours is a bandaid fix.

Taz

@TheHackBox and @rds100

So what you are saying is this will take up the whole unused memory and clear it up when it is needed? not a bug but *nix thing?

rds100

@NinjaHawk exactly! It's a feature, not a bug.

Taz

@Jarland this is not even a bandaid fix. It is more of a duct tape fix.

Taz

@rds100 if that is so, why no one ever mentioned that? Everyone keeps saying it is a openvz bug and I do agree with it. I can not Imagine what will happen if some how cron gets disabled.

rds100

@NinjaHawk i don't know who is everyone that keeps saying that using free memory as a cache is a bug, but this has been a feature in linux for the last 15 years at least, if not more.

jarland

@rds100 It has never been a feature that an idle system would consume 100% ram over the course of 24 hours for cache. This does not occur on 2.6.18 and it causes issues when it occurs under 2.6.32. This cache is also not always reading as cache. The OS will not freely release it, at least not in this same scenario that I experienced.

Taz

For reference, http://forum.soluslabs.com/showthread.php/3861-Centos-6-openvz-64bit-kernel-memory-leak http://forum.openvz.org/index.php?t=msg&goto=45030&

rds100

You aren't (or atleast don't need to) be running an OpenVZ kernel on your SolusVM master, so this is really a very different thing that is reported there.

Taz

@rds100 said: You aren't (or atleast don't need to) be running an OpenVZ kernel on your SolusVM master, so this is really a very different thing that is reported there.

What if I mention that the same is happening on slaves as well?? and care to clarify why I should not run ovz kernel on solus node?

rds100

@NinjaHawk you don't need to use OpenVZ kernel on the SolusVM master if it is a master only (i.e. no VMs there). Of course you can use any compatible kernel you want, including the OpenVZ ones. But you don't have to. And here is the output from one of my nodes:

Mem:  16375760k total, 15854684k used,   521076k free,   930464k buffers
Swap: 33792716k total,        0k used, 33792716k free, 10831816k cached

Notice the about 10GB cache. And i am not worried at all about this output. It is perfectly OK, linux is working as it is designed to work.

jarland

@rds100 What kernel and does the cache ever eventually consume your ram completely so that the free value (ref: http://www.linuxatemyram.com/) drops to 0?

Taz

So what you are saying is that, when more memory gets requested, it will free up the cache while not causing any issue with the node itself?

rds100

@jarland kernel 2.6.18-308.8.2.el5.028stab101.1 And no, the free RAM doesn't drop to 0. When it gets below some threshold. linux starts freeing memory (by removing some of the cache, etc.).

jarland

@rds100 Hmm. Yeah not the same as the issue I had with .32. If his is the same as mine, it actually consumes all available memory until the kernel begins locking up and crashes.

rds100

@NinjaHawk test it yourself. Start some big memory consuming jobs on the server - web browsers, some java shit, etc. And see how the cache gets freed and memory is made available to meet the demand.

Taz

@jarland said: Hmm. Yeah not the same as the issue I had with .32. If his is the same as mine, it actually consumes all available memory until the kernel begins locking up and crashes.

Yes indeed. couple of nodes crashed without warning.

rds100

@jarland then you had a buggy kernel. Normal linux kernel behaviour is to use almost all free memory as cache, and free it again when it is needed.

Taz

@rds100 said: yourself. Start some big memory consuming jobs on the server - web browsers, some java shit, etc. And see how the cache gets freed and memory is made available to meet the demand.

Once my basement DC starts up, I will start some real world test. @jarland thanks for the hint as well.

prometeus

@NinjaHawk said: , it will free up the cache while not causing any issue with the node itself?

yup :)

every byte is used for cach if no process allocate memory.

jarland

@rds100 It seems to be not that uncommon on .32 with OpenVZ. It can be fixed without replacing the kernel or constantly clearing the cache though. Long story short, I carpet bombed it and I don't have a clue which one hit the target, so I won't detail my fix until I figure that out. I told Ninja what I did though, just prefer not to air every little detail of my servers.

jcaleb

hello @prometeus your openvz's run on 2.6.32?

prometeus

@jcaleb said: hello @prometeus your openvz's run on 2.6.32?

yes they are all .32

eastonch

@Ninjahawk you're running the OVZ Kernel on the solus master, is it also hosting servers? Bad practice if you ask me.

Damian

@NinjaHawk: Can you tell us what version of 2.6.32 it is? (output of uname -a)