Service watchdog script - input wanted!

littleguy

This is a general-purpose monitoring script for any service. It will check if the service is running and restart it if it is not via cron. It can also email you when service restarts occur so you can investigate the cause.

Would love to have your input and suggestions on how to improve it.

Example usage in Crontab: */5 * * * * sh /root/watchdog.sh apache2 (Checks if apache is running every five minutes and restart it if it's down.)

*/5 * * * * sh /root/watchdog.sh mysqld mysql ("Loose" check for mysqld every 5 minutes, second parameter is the name of the service to restart, in case the application and service names differ.)

More examples are available at the top of the shell script.

Download: https://dl.dropbox.com/u/2758854/watchdog.sh

Source: https://dl.dropbox.com/u/2758854/watchdog.sh.txt

Requirements: pgrep service

gsrdgrdghd

@littleguy said: (Checks if apache is running every five minutes and restart it if it's down.)

You could also just do */5 * * * * /etc/init.d/apache start ;)

sleddog

@littleguy said: Would love to have your input and suggestions on how to improve it.

Dispatch an email to the admin when a restart is necessary.

Apache or MySQL or other services should never just 'stop' on a stable server. If they do there's a problem that needs investigating. Simply restarting the service is a temporary fix, not the solution. I'm not criticizing your script :) just saying that notification of a 'stopped' service is important.

dannix

You should check monit

littleguy

@sleddog said: Dispatch an email to the admin when a restart is necessary.

That's a great idea, I have updated the script, you can now optionally add an email as a fourth parameter, if the service needs to be restarted an email will be sent to that address.

Example: */5 * * * * sh /root/watchdog.sh mysqld mysql admin@domain.com

@dannix said: You should check monit

Looks like a great monitoring system! My script is somewhere between monit and the one-liner of gsrdgrdghd. :D

telephone

Suggestion for a 5th parameter: - Memory usage. While checking if it's running, make sure it's under the alloted MB of RAM. (e.g. Ensure Apache is under 1GB)

Try to cram as much as you can into 100 lines :)

littleguy

@telephone said: Suggestion for a 5th parameter: - Memory usage.

I think that would be non-trivial (adding memory usage over multiple threads, memory usage from external scripts such as php5-fpm, php-cgi etc)

But as a different, separate script, this could be an interesting idea!

@telephone said: Try to cram as much as you can into 100 lines :)

Haha, that's an interesting philosophy. :)

marrco

@dannix said: You should check monit

this

peepingnerd

@littleguy

This is a general-purpose monitoring script for any service. It will check if the service is running and restart it if it is not via cron. It can also email you when service restarts occur so you can investigate the cause.

take a look at restard ;)

btw, that's what i use to spawn max concurrent process (usage ./maxspawn.sh "matching_ps_process" 2 "/a/kind/of/concurrentscript/filename_matching_ps_process.sh")

#!/bin/sh
PSMATCH="$1"
MAXPROC="$2"
SPAWNCMD="$3"

CURRENT=`ps awjx | grep -F "${PSMATCH}" | grep -v grep -c -m "$MAXPROC"`

if [ "$CURRENT" = "$MAXPROC" ]; then
        echo "Max concurrent process, exit without running";
        exit;
fi

echo "$CURRENT concurrent process running"

`"${SPAWNCMD}" &`

no security consideration at the monent: if you are smart enough you can start a runloop ;)

Surge

http://supervisord.org/