My linux server “Number of processes created” and “Context switches” are growing incredibly fast – Managing your servers can streamline the performance of your team by allowing them to complete complex tasks faster. Plus, it can enable them to detect problems early on before they get out of hand and compromise your business. As a result, the risk of experiencing operational setbacks is drastically lower.
But the only way to make the most of your server management is to perform it correctly. And to help you do so, this article will share nine tips on improving your server management and fix some problem about linux, process, threads, stats, context-switch.
I have a strange behaviour in my server :-/. Is a OpenVZ VPS (I think is OpenVZ, because
/proc/user_beancounters exists and
df -h returns /dev/simfs drive. Also
venet0). When I do
cat /proc/stat, I can see how each second about 50-100 processes are created and happens about 800k-1200k context switches! All that info is with the server completely idle, no traffic nor programs running.
Top shows 0 load average and 100% idle CPU.
I’ve closed all non-needed services (httpd, mysqld, sendmail, nagios, named…) and the problem still happens. I do
ps -ALf each second too and I don’t see any changes, only a new
ps process is created each time and the PID is just the same as before + 1, so new processes are not created, so I thought that process growing in
cat /proc/stat must be threads (Yes, seems that
/proc/stat counts threads creation too as this states: http://webcache.googleusercontent.com/search?q=cache:8NLgzKEzHQQJ:www.linuxhowtos.org/System/procstat.htm&hl=es&tbo=d&gl=es&strip=1).
I’ve changed to
/proc dir and done
cat [PID]status with all PIDs listed with
ls (Including kernel ones) and in any process
nonvoluntary_ctxt_switches are growing at the same speed as
cat /proc/stat does (just a few tens/second),
Threads keeps the same also.
strace -p PID to all process too so I can see if any process is crating threads or something but the only process that has a bit of movement is
ssh and that movement is
write operations because of the data is sending to my terminal.
After that, I’ve done
vmstat -s and saw that
forks is growing at the same speed
/proc/stat does. As http://linux.die.net/man/2/fork says, each
fork() creates a new PID but my server PID is not growing!
The last thing I can think of is that all process data that
vmstat -s show is shared with all the other VPS stored in the same machine, but I don’t know if that is correct… If someone can throw some light on this I would be really grateful.
In your case I would assume that your kernel is not virtualizing some or all values in /proc/stat, so what you’re seeing should be caused by activity in other containers or in the hardware node itself.
You haven’t mentioned your specific kernel version so we can’t check for sure, but a similar issue was discussed in http://forum.openvz.org/index.php?t=msg&goto=44507& .
PS: check out /proc/vz/vestat instead.
I have seen at least one similar question here about someone with 2 identical ubuntu servers behind a load balancer…and I will tell you what I told him:
syspro (“system profiler”)
or try stuffing some limits in /etc/security/limits.conf
Apply the limits to both machines and compare which executables show up.
It will at least flag a process or thread and narrow the list of suspects.
man limits.conf: hard & soft for file descriptors, mem locked, stack size,cpu time, nice levels, etc