Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


How to investigate vps suspended due to 0 bandwidth?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

How to investigate vps suspended due to 0 bandwidth?

bustersgbustersg Member

my vps got suspended because it hit 750GB bandwidth in last 5 hours.
i checked grafana monitoring and notice spike and CPU and bandwidth that killed the bandwidth and provider suspended my vps for a good reason.

https://ibb.co/Jqnrdkf
grafana graph

provider was kind enough to activate my vps to let me extract logs.
so i ls -lt and grab last modified logs in /var/log

how to i investigate from here? i got kern.log syslog auth.log etc.

Comments

  • MaouniqueMaounique Host Rep, Veteran
    edited May 2023

    Those would not help in most cases.
    Most likely it was some DDoS attack which would not show up in such logs. In fact, logging DDoS packets is counter productive as it would lead to extremely high CPU and IOPS as well, so you would be booted for that too.

  • rcy026rcy026 Member

    I agree with above, seeing that the bandwidth shows a clear spike in incoming bandwidth this is most likely a ddos.
    It's kind of strange that the cpu spike seems to initiate almost a full hour before the bandwidth spike though.

  • my password is > 10 chars and with symbols and i was using non-SSH port 22
    i checked the auth.log and did not notice anything else.

    last 30days grafana is flat graph except for the last 5hrs of course.
    i only have ip:9100 for grafana and few small non-public sites that are only accessable behind Apache httpd basic auth (which is another layer of security).

    so, do i need to subscribe alert services like PagerDuty to prevent such incident in future?

  • NanjaNanja Member

    @bustersg said:
    my password is > 10 chars and with symbols and i was using non-SSH port 22
    i checked the auth.log and did not notice anything else.

    last 30days grafana is flat graph except for the last 5hrs of course.
    i only have ip:9100 for grafana and few small non-public sites that are only accessable behind Apache httpd basic auth (which is another layer of security).

    so, do i need to subscribe alert services like PagerDuty to prevent such incident in future?

    I can't comment on PagerDuty, never heard of it.

    Depending on how you have your non-public sites setup, you could use free ddos protection from a CDN. I use cloudflare, they protect HTTP/HTTPS ports for free. It might be helpful in the future.

  • MikeAMikeA Member, Patron Provider

    If this was a DDoS you're really out of luck if the provider doesn't have any mitigation and counts inbound traffic against your quota. Kinda unfortunate. Best thing would be to capture some traffic with tcpdump if it happens again to try to find the source. If you mainly have things running on a web server should definitely try CloudFlare or another CDN.

  • MaouniqueMaounique Host Rep, Veteran
    edited May 2023

    @MikeA said: Best thing would be to capture some traffic with tcpdump if it happens again to try to find the source.

    That is kind of hard when the attack is ongoing as:
    1. the ports might be saturated therefore the access (unless OOB) would probably be seriously hindered;
    2. the cpu might be overextended.

    Add to this that it is probably an UDP attack (most are, I mean, the ones trying to saturate the pipe) AND probably spoofed anyway, that would most likely not help with finding a source. Only the provider can help in some situations (multi-homed, good monitoring) but in most situations even that won't help.

  • MikeAMikeA Member, Patron Provider

    @Maounique said:

    @MikeA said: Best thing would be to capture some traffic with tcpdump if it happens again to try to find the source.

    That is kind of hard when the attack is ongoing as:
    1. the ports might be saturated therefore the access (unless OOB) would probably be seriously hindered;
    2. the cpu might be overextended.

    Add to this that it is probably an UDP attack (most are, I mean, the ones trying to saturate the pipe) AND probably spoofed anyway, that would most likely not help with finding a source.

    Here's a script I found years ago and had always saved to capture tcpdump if packets per sec go over a threshold. Can run it in screen to keep it running when not connected to ssh.
    https://pastebin.com/raw/8JxEsQEN
    Can modify it to send a notification with pushover or something but I can't find that one I modified.

    Thanked by 2eliphas Obelous
  • thanks, will take a look at the script.
    i'm using free tier cloudflare for all my domains.
    not sure if it include free ddos.

  • i might have found the culprit?
    these 3 ips are in abuse list - https://www.abuseipdb.com/
    also this is related to some "Microsoft Exchange Zero-Day Vulnerabilities"
    https://www.idappcom.co.uk/post/detection-of-microsoft-exchange-zero-day-vulnerabilities

    nginx access.log

    162.243.133.12 - - [25/May/2023:17:10:26 +0800] "GET /owa/auth/x.js HTTP/1.1" 404 143 "-" "Mozilla/5.0 zgrab/0.x"
    107.170.252.8 - - [25/May/2023:17:11:06 +0800] "GET /ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application HTTP/1.1" 404 143 "-" "Mozilla/5.0 zgrab/0.x"
    162.243.140.44 - - [25/May/2023:17:17:01 +0800] "GET /owa/auth/logon.aspx HTTP/1.1" 404 143 "-" "Mozilla/5.0 zgrab/0.x"
    
  • NanjaNanja Member

    If you are starting a new server somewhere or using this server you are on now.
    Will you be using a backup?

    If it really is a zero-day, I don't know what to tell you. They can gain access to your system without you knowing it. I think recovering from a backup might be difficult, since if what they put on your system is in the backup.

    If your lucky, there is nothing at all.

    Not trying to scare you, this is just speculation.

  • Added network RECEIVE email alert to Prometheus
    threshold = 3x my usual day to day bandwidth

    expr: node_network_receive_bytes_total{device="eth0",instance="xx.xx.xx.xx:91xx",job="node-exporter"} > 400000000

    Anyone know how to use rate() to convert bytes to MB ?

  • rcy026rcy026 Member
    edited May 2023

    @bustersg said:
    Added network RECEIVE email alert to Prometheus
    threshold = 3x my usual day to day bandwidth

    expr: node_network_receive_bytes_total{device="eth0",instance="xx.xx.xx.xx:91xx",job="node-exporter"} > 400000000

    Anyone know how to use rate() to convert bytes to MB ?

    Rate calculates average over time for the value passed to it, it doesn't really convert anything per se. But you can just divide with whatever to get the unit you prefer.

    Example:
    "rate(node_network_receive_bytes_total[5m]) / 1000 / 1000" should give you the average bandwidth used during the last 5 minutes, provided there are at least two datapoints collected during that time.

    Using rate and reacting on the resulting value would probably be a better solution since it will not trigger if your bandwidth peaks for a few seconds, but it will trigger if it is constantly high over a defined period of time.

    Keep in mind that you have to kind of adapt the timespan to somewhat match your scrape interval or you can end up with some funny calculations. Short timespan and/or high scrape interval gives higher margin for error.

  • eliphaseliphas Member

    @MikeA said:
    Here's a script I found years ago and had always saved to capture tcpdump if packets per sec go over a threshold. Can run it in screen to keep it running when not connected to ssh.
    https://pastebin.com/raw/8JxEsQEN
    Can modify it to send a notification with pushover or something but I can't find that one I modified.

    Thank you very much for that, I've taken that idea and modified it further to stay running indefinitely, have some buffer room so it won't trigger so easily and limit email spamming.

    In case it helps someone else here it is:
    https://gist.github.com/eliphaslevy/9b633899b227bd6964daef469755fb6a

    Thanked by 1MikeA
  • "running indefinitely" means NOT need to put in cron job and execute every 5 min?
    your version can just execute once and it will be 'monitoring' in the background?

  • eliphaseliphas Member

    @bustersg said:
    "running indefinitely" means NOT need to put in cron job and execute every 5 min?
    your version can just execute once and it will be 'monitoring' in the background?

    Yes, indefinitely until something kills it. In theory. I always like to put things like that in cron so if it happens to crash it will be running again at most one hour later. If it is already running, there is the locking code for keeping it from having duplicates.

Sign In or Register to comment.