How to identify what failed to load at bootup?

Ree

One of my VPSes was used to send out spam yesterday, and I think it may have been because iptables-persistent didn't restore my iptables rules when the server rebooted.

After realizing there was a problem the first thing I did was an iptables -L, which showed no rules were in place, so my stupidly configured squid proxy was allowing access to sendmail! (Now fixed, in case this ever happens again -- now I know relying on the firewall and ignoring application level security isn't just lazy, but dangerous!)

So I rebooted the VPS, and after it came back up iptables -L showed all my stored rules, which is what leads me to believe that the problem was due to iptables-persistent not restoring my rules on the previous reboot.

I think the host node rebooted, so if it's really oversold did it maybe run out of memory while all the containers booted at once and some of my processes (like iptables-persistent) failed to run?

Is there anything I can check to know if this was the case?

dhamaniasad

Well, maybe for the future, you should monitor the host nodes IP using Uptimerobot or something to know when it goes down. But that isn't exactly very reliable. The host node wouldn't run out of memory while the containers are booting up, booting up a normal Linux container usually takes ~40MB of RAM from what I've seen. However, there is no other way to know if the host node was rebooted apart from asking the provider in your current situation.

howardsl2

I wrote a script to check that IPTables is running correctly with the rules every 5 minutes. It has the following components:

1 Somewhere in your IPTables rules, add this:

-A INPUT -m comment --comment "CHECKME"

OR

-A INPUT -m recent --set --name CHECKME

2 Create a script /root/check_iptables.sh

#!/bin/bash
/sbin/iptables -n -L | grep "CHECKME" > /dev/null
if [ $? -eq 0 ]; then
  touch /var/run/ipt_timestamp
fi

3 Create a cron job /etc/cron.d/check_iptables

*/5 * * * * root /root/check_iptables.sh &>/dev/null

4 Create a script /root/iptablesload.sh

#!/bin/sh
/sbin/iptables-restore < /etc/iptables.rules
exit 0

5 Install Monit, set up email notification, and the rule:

 check file iptables-ts with path /var/run/ipt_timestamp
   if timestamp > 6 minutes then exec "/root/iptablesload.sh"
   if timestamp > 6 minutes for 2 cycles then alert

And you are done. For more IPTables security tips check out my tech blog in my signature.

Edit: Changed the iptables restart command to an iptablesload script.

Ree

@dhamaniasad said: Well, maybe for the future, you should monitor the host nodes IP using Uptimerobot or something to know when it goes down. ... However, there is no other way to know if the host node was rebooted apart from asking the provider in your current situation.

I use three different uptime monitoring solutions, so I was aware it went offline almost right away. I wasn't able to access it via the control panel for a few minutes though, then it took another minute or two before the Boot command worked, which is why I was thinking maybe the host node rebooted. After it finally did boot I didn't think to look whether everything (like iptables-persistent) was started correctly.

And I reworded my message without proof-reading before submitting. So the "any way to check if this was the case" was meant to be attached to the question of how to check whether a program failed to load at boot, not whether the host node rebooted (whether it did or didn't isn't too relevant, that was just my theory). Basically I'm wondering if iptables-persistent ran and failed to restore the rules, or if it failed to run altogether.

@hwdsl2 said: I wrote a script to check that IPTables is running correctly with the rules every 5 minutes. It has the following components:

Thanks for sharing that -- I'll definitely be incorporating that into my new server setup guide!

craigb

@ree another option would be to bind squid on loopback and use an iptables rule to forward traffic to it as appropriate. That way if iptables fails to load, no traffic hits your proxy at all. Or just use squid ACLs as a second layer of IP filtering.

Ree

Thanks for the suggestions -- still not overly familiar with iptables so it never even occurred to me that I could do something like that! I like that option, since that way even if I bung the squid configuration the proxy won't be made public again.