Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Restart process if CPU usage goes to 100%
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Restart process if CPU usage goes to 100%

Is there a simple script in Debian that I can run to monitor my CPU usage and whenever the usage goes to a certain threshold, restart the process, without the need to use things like Monit or some other third party programs?

Unfortunately my PHP-FPM process tend to have CPU usage go over the roof from time to time on one of my servers and I know the best practice is to figure out why that happened however with around 15 Wordpress installations (and each have different levels of visitors numbers and plugin configurations) I don't think this is a fun task to do so I'd rather just have a script that would control the CPU usage somewhat passively. Unless someone could suggest a way for me to log the exact CPU usage for each individual PHP script, again, hopefully without a third-party program. Everything I've been trying so far seems to give me the CPU usage by process (so all I see is php-fpm using x amount of CPU) but not by the script that's invoking the PHP-FPM process.

Comments

  • @spammy said:
    Unfortunately my PHP-FPM process tend to have CPU usage go over the roof from time to time

    Then fix what is causing the spike rather than restarting your service. And this is certainly a "fun Task" to do

  • Maybe you should limit the connections per ip with delays, which may results in less load of your php-fpm

    Thanked by 1spammy
  • @racksx said:
    Maybe you should limit the connections per ip with delays, which may results in less load of your php-fpm

    I am behind Cloudflare though, and I only allow connections from the Cloudflare IPs, so I guess that won't work too well for me or there is a way to limit by the originating IPs?

  • racksxracksx Member
    edited November 2016

    Or maybe you should monitor it, and see from where you are getting the connections, how many and try to increase them.

  • @mehargags said:

    @spammy said:
    Unfortunately my PHP-FPM process tend to have CPU usage go over the roof from time to time

    Then fix what is causing the spike rather than restarting your service. And this is certainly a "fun Task" to do

    Care to give some clue on where to start?

  • @racksx said:
    Or maybe you should monitor it, and see from where you are getting the connections, how many and try to increase them.

    increase them? you mean increase the max-children?

  • lonealonea Member, Host Rep

    Running major control panel ?

    If so, just get a copy of cloudlinux and be done with it.

  • @lonea said:
    Running major control panel ?

    If so, just get a copy of cloudlinux and be done with it.

    Nope, no control panel.

  • racksxracksx Member
    edited November 2016

    Yes,

    this might be what you need:

    #!/bin/bash
    for I in 0 1 2 3 4 5; do
        check=$(uptime | tr -d ',.' | awk '{print $10}')
        if [ "$check" -gt 5 ]; then
            /etc/init.d/php-fpm restart
        fi
        sleep 10
    done
    


    if you are using nginx have a look at:

    https://tweaked.io/guide/nginx/

    Thanked by 2spammy coreflux
  • kazukenkazuken Member
    edited November 2016

    Check out monit.
    it can monitor processes for you.
    something like the following line can be used:

    if cpu is greater than 99% for 5 cycles then restart

    check it out: https://mmonit.com/monit/

    they have a decent presentation that goes over all their features

  • @kazuken said:
    Check out monit.
    it can monitor processes for you.
    something like the following line can be used:

    if cpu is greater than 99% for 5 cycles then restart

    check it out: https://mmonit.com/monit/

    they have a decent presentation that goes over all their features

    Thanks but I specifically mentioned :)

    without the need to use things like Monit or some other third party programs?

  • DraetheusDraetheus Member
    edited November 2016

    I'm going to assume you're running all your sites with the default PHP-FPM configuration, which means they run in a single pool with a single user. So by default, you have no way of knowing which PHP process handled a request for a site. The solution is to give each site its own PHP-FPM pool configuration under the pool.d folder, and make sure each pool runs as a different user. That way the next time you experience high load you'll instantly be able to know which site is causing it.

    Also have you done any general PHP performance tuning?

    • Make sure you have enough FPM workers in your pool, especially if you have free RAM.
    • Upgrade to PHP 7 if at all possible
    • Enable and tune opcache extension
    • Set sane defaults for php.ini values: memory limit, max execution time, max input time, etc
    Thanked by 1spammy
  • sesamsesam Member
    edited November 2016

    I spent some hours researching, then built a Ruby script:

    cpulimit = 9.0
    last = [nil,nil]
    badtimes = 5
    badcount = 0
    sleep_seconds = 2 # seconds
    1.upto(60/rate*60) do # 60 sec * 60 / sleep_seconds = 1 hour
      a = %x(ps -C searchd -o pcpu,pid) # -C works on Debian, not on MacOS
      good = []
      bad = ''
      a.lines.each do |k|
        k =~ /([0-9\.]+) *( \d+)/
        if $1.to_f>cpulimit
          bad<<$2
        elsif $2
          good<<$2
        end
      end
    
      silent = (last == bad)
      badcount += 1 if !silent and bad!='' # break out of silence
      if badcount > 0
    ...
    
    Thanked by 1spammy
  • continued:

        badcount += 1 # and continue complaining for badtimes
        puts "possibly BAD pids: %w(#{bad})"
        badcount = 0 if badcount > badtimes # stop repeating the complaint...
      end
      last = bad
      print "\nGOOD: #{good.join(';')} " if good != [] and !silent
      if bad.strip != ''
        print " BAD: " + bad unless silent
        # puts ' => killing...' unless silent
        print '.'
        puts unless silent
      end
      sleep sleep_seconds
    end
    
    Thanked by 1spammy
  • To use this yourself, replace -C searchd with something that works for you to filter out your suspect processes by name. Or don't filter ;-> and risk killing something important.

    Also adjust cpulimit ("percent") to something relevant to your machine. top / htop shows processes as 100% on a 4 core machine, when ps -o pcpu gives you 25.0 ("procent"). And likely the "25%" will keep rising up to "60%" even if it's still hogging exactly 100% of one of your cores. At least this is my experience - happy if anybody can correct me on this.

    Thanked by 1spammy
  • Cant systemd do this?

  • @sin said:
    Cant systemd do this?

    That's what I was thinking. According to its man pages it has the ability to limit CPU allotment, etc. I've never tried it but it seems like the best approach. On Debian you can drop files in /etc/systemd/system to override vendor conf settings for things like PHP.

    To be honest, the entire approach is a bandaid solution. The real solution is to find the source of the problem and fix it. But... ¯\_(ツ)_/¯

  • raindog308raindog308 Administrator, Veteran

    racksx said: #!/bin/bash

    Not sure what the countdown is for...aren't you really just comparing the current load avg as an int versus a number, in an infinite loop with sleep?

    Simpler:

    #!/bin/bash
    
    while [ 1 ] ; do
       if [ $(uptime | awk '{ printf "%i", $9 }') -gt 5 ] ; then
          systemctl restart php-fpm # or whatever
       fi
       sleep 10
    done
    
    Thanked by 1spammy
Sign In or Register to comment.