All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How to debug exactly what is using bandwidth?
I have 20+ tiny VPSs at a variety of providers, they use almost no traffic and are usually idle. I noticed one provider however despite both of my VPS' being barely used the bandwidth usage shown in their control panel is significantly higher than expected (over 500GB/mo each).
Both VPS used SSH private key auth, password disabled, fail2ban ban after 5 attempts and didn't have anything external-facing installed on it, but out of caution in case they'd been compromised I reinstalled them.
With a fresh install, fully updated (Debian 12), nothing else installed other than UFW which has been set to block all incoming traffic except ssh port... the issue is still happening. In the last 3 hours their control panel has increased "traffic used" by 5GB, with the VPS just sat idle.
NetHogs shows nothing except my SSH connection (@ 0.1KB/s), vmstat does show excessive traffic for what is an idle VPS:
eth0 / 5 minute
time rx | tx | total | avg. rate
------------------------+-------------+-------------+---------------
2023-10-15
11:00 53.96 MiB | 24.11 KiB | 53.99 MiB | 1.51 Mbit/s
11:05 21.18 MiB | 30.67 KiB | 21.21 MiB | 592.98 kbit/s
11:10 14.92 MiB | 16.51 KiB | 14.93 MiB | 417.54 kbit/s
11:15 65.47 MiB | 18.65 KiB | 65.49 MiB | 1.83 Mbit/s
11:20 38.39 MiB | 15.57 KiB | 38.41 MiB | 1.07 Mbit/s
11:25 61.34 MiB | 26.82 KiB | 61.36 MiB | 1.72 Mbit/s
11:30 98.89 MiB | 12.54 KiB | 98.90 MiB | 2.77 Mbit/s
11:35 68.29 MiB | 22.66 KiB | 68.31 MiB | 1.91 Mbit/s
11:40 12.62 MiB | 14.36 KiB | 12.63 MiB | 353.15 kbit/s
11:45 7.27 MiB | 27.56 KiB | 7.29 MiB | 203.94 kbit/s
11:50 33.01 MiB | 13.02 KiB | 33.02 MiB | 923.40 kbit/s
11:55 9.48 MiB | 16.06 KiB | 9.49 MiB | 265.40 kbit/s
12:00 29.77 MiB | 22.18 KiB | 29.79 MiB | 833.06 kbit/s
12:05 8.16 MiB | 14.26 KiB | 8.18 MiB | 228.63 kbit/s
12:10 33.71 MiB | 19.01 KiB | 33.72 MiB | 943.01 kbit/s
12:15 110.01 MiB | 29.16 KiB | 110.04 MiB | 3.08 Mbit/s
12:20 81.75 MiB | 31.54 KiB | 81.78 MiB | 2.29 Mbit/s
------------------------+-------------+-------------+---------------
but I'm not really sure how to further work out what is causing this? It only happens at this one provider, both VPSs, at every other provider my idle VPS takes approx 2.5MB / 5 mins which is more in line with what I would expect. At the rate these ones go they'd be using ~500GB/mo without doing anything.
Any ideas how to get a bit more info about what's going on?
Comments
Have your tried tcpdump to inspect the packets on the network interface for your VM?
tcpdump is a little bit over my head atm, but I did have a quick look and didn't really see anything that stood out. Just the standard "ARP Request who-has" packets
edit: watching udp packets with tcpdump shows a lot of:
idk if that's normal, 157.240.251.63 is supposedly in an IP owned by Facebook (?).
Spoofed source IP? Someone is trying to make your VPS send network packets to Facebook's servers? DDoS?
got a bit similar problem on previous vps, easy and nice way using bandwhich
using ss [socket statistics from iproute2 packages]
watch -n1 sudo ss -ntupusing iptables/iptables-legacy logging everything, but carefull with 'huge' log file
iptables -A INPUT -p tcp -j LOG --log-level debugiptables -A INPUT -p udp -j LOG --log-level debugiptables -A OUTPUT -p tcp -j LOG --log-level debugiptables -A OUTPUT -p udp -j LOG --log-level debugThanks for the suggestions, bandwhich and ss didn't seem to show anything of note. Haven't tried iptables yet because it's only a 6GB drive VPS so I'll need to be careful with the log size but will try it later.
Interested in finding out what is going on, since the original post traffic has increased by 10GB... more than some of my VPSs use in a month.
Perhaps this providers IPs are just in a range that are attacked often?
Highly unlikely.
Look at the RX/TX ratio which strongly suggests that something on OP's VPS does (smallish) requests (and more than a few) and receives about 1000 x times more bytes than it sends out.
And yes, bandwhich should allow you to track down the culprit.
From the arrow between the IP address, you can see that it's traffic coming from facebook's HTTPS (port 443) to a random port on your machine. So your machine is currently fetching something from facebook.
If that's not you, then you can try running netstat on one of the ports (so 443 or 56243):
(apt-get in case you're using debian as it's not installed by default)
If it doesn't show up anything, then maybe the connection has already finished, or maybe your machine is proxying something at the time. Are you using it as a VPN?
Install
ntopng. It runs a webserver at port 3000. Open it in a browser, check both Applications and Hosts on the main dashboardTurns out... this might have just been me being naïve to how much bandwidth ssh brute forces can consume if you don't have an IP blocking mechanism set-up properly (I had fail2ban misconfigured, oops).
I wasn't paying much attention to the failed ssh attempts because I assumed it would only be a few kb/s but in fact after watching it in bandwhich more there were often 8 chinanet IPs (218.92.0.xx) consuming 25KB/s each making it ~200KB/s (18GB/day, > 500GB/mo). After setting up fail2ban (properly this time) the traffic so far has been more normal:
Going to monitor it overnight but hopefully that was the problem. Thanks everyone for the help
The issue is not closed. You have to investigate what was coming from facebook.
After that post the Facebook IP didn't show up again so not really sure what that was about, and I don't know enough to debug it any further so I'll just pretend I didn't see that
The "x.x.x.254" isn't the IP for my VPS, mine is "x.x.x.153" I assume it's something to do with their switch (?). It gets similar udp packets from google DNS to x.x.x.254:
As long as the properly configured fail2ban stops the SSH brute forces consuming the entire monthly traffic allowance I'm not too bothered. I'll just wipe it back to the stock debian 12 image once more and secure it better (ip ssh whitelist) just in case
Spoke too soon, it didn't fix it, still used 10GB from 2AM-10AM. Have no idea what's going on, that was again on a clean install of Debian 12, updated, fail2ban set up to ban after 2 failed attempts, keyed ssh auth, passwords disabled, ufw blocking all incoming connections apart from port 22, nothing else running.
Another odd thing is it has 3 IPs/interfaces and they all spike at the same time, by (almost) the exact same amount:
Logging a 30 minute period with bandwhich showed:
vnstat for the same period showed:
Happens on my other VPS (also clean install) on this provider too, that one only has 1 interface / IP but still uses ~300GB/mo. Don't know what's going on and it's a bit over my head think I'll just have to give up on this provider, checked all my other providers and this doesn't happen anywhere else
It is highly unlikely they would spoof some traffic in order to use up your quota (albeit nothing is impossible and I wouldn't be too surprised).
Yeah, I didn't mean to imply they were doing it purposely, but something is up and I don't have the time/knowledge to work out what it is. It's received 225GB in 5 days yet bandwhich shows nothing significant... it's using more in a day than most of my VPSs use in a month. I don't think it can be compromised either because it resumes happening immediately after a fresh install. Don't know if that means their IP ranges are just constantly being attacked/flooded or what but I give up, not worth it for a $2/mo VPS
Maybe have a look to see if there are any other interfaces, e.g. wg0, that might suggest someone is running a VPN on it.
You could also run
iptables(or whatever the fancy new version is that people use nowadays) to see if there are any rules to masquerade traffic, as these might otherwise not appear on a netstat, because they're not actually open sockets on your machine.Also, that
iptablessuggestion early on in the thread to log only appends logging to the rules, and it's possible if a rule explicitly ends early with -j ACCEPT or -j MASQUERADE that it might not get as far as that rule and so might not actually be logged. I've only used -j LOG followed by a -j DROP, so not really sure if a -j ACCEPT would pass back to the -j LOG, it might only get there if no rules match.Over that that, maybe just do a full OS re-install if you're at all worried the system has been compromised. To me, if it's doing that much traffic, and especially to facebook, somebody is actively using it for proxying, so if you're not doing that I'd be very, very suspicious of everything that's installed. For precaution, if this machine has any keys or credentials to other systems, you should probably change them as well after re-install this machine.
Thanks for the extra suggestions, nothing is standing out in the iptables rules no masquerade rules or postrouting, it's all just the standard rules added by UFW. No unexpected interfaces either.
This is already a clean install, I've reinstalled it multiple times during the last few days, at the moment the setup is just:
Literally nothing else done to it, and then after it being 'idle' for an hour vnstat looks like this:
For reference all the VPS's I have at other providers look like this:
Check tcpdump again, I've seen heavy Windows networks run the numbers up, very noisy servers.
Okay, I had tcpdump running this time when one of the random ~100mb transfers happened. A second of it:
https://pastebin.com/raw/xGNMdbPv
It mainly comes from Iranian IP addresses (5.214.226.166 + 31.56.162.148 + 5.119.228.179) but that Facebook owned IP was there again (157.240.0.63) too.
One of the many things I don't understand... why is traffic to x.x.x.254 passing through my VPS when I don't have the x.x.x.254 address? Mine is x.x.x.164, presumably this is why it doesn't show in things like bandwhich?
Is this just normal abusive traffic? I don't know whether I'm making something of nothing here
. Again, this is just a fresh install and it happens no matter how many times I reinstall so I don't think the VPS itself is compromised...?
you could install pmacct(d), export the data to json or a database.
Then pull up fancy graphs sorting that by country, asn, port, protocol.
Block all outgoing except SSH port and services you want public
Even though you said this a couple of times in different posts, I didn't spot that it wasn't your IP address before.
Knowing that now, it rather looks to me like you're just receiving packets for someone else that you shouldn't be. In this case, it's a good sign that all the traffic is one way, in this capture "5.214.226.166.55382 > x.x.x.254.443" as that simply shows someone connecting to HTTPS on x.x.x.254, and the fact there are no packets the other way is kind of good because it means your machine isn't sending any responses.
So the question then is why are you getting these at all? My first thought was that your network interface was sent to promiscuous mode and was eavesdropping on everything. However, that seems unlikely because I'd also expect you to be able to see the other direction too, unless for some reason that's being dropped because the destination is the wrong subnet. That wouldn't normally be the case, as far as I remember.
My next thought then is that "the router" is broadcasting the packet, maybe because it doesn't have an arp mapping for it. It's also interesting that something ending in .254 is sometimes used for the router's IP instead of .1, so maybe something is misconfigured there.
But then I was thinking that this is a VPS, so it seems pretty weird that it would be sending you traffic for another VPS anyway, although looking into a similar situation myself I don't understand how that works anyway! In my case, I have the host running
tcpdump -i virbr0 port not 22and I can see all non-SSH traffic to all my VMs via the bridge interface that all the VMs share. However, on one VM runningtcpdump -i enp1s0 port not 22doesn't see the traffic for other VMs that share the same bridge, so I'm not sure of the mechanism by which they're filtered out.In my head, a bridge interface should be shared by all the VMs and so it should see all traffic, but maybe libvirt also does something here to pre-filter so each host only receives packets for its MAC and broadcast. It looks like there's some documentation here that suggests this is the case here: https://libvirt.org/firewall.html and https://libvirt.org/formatnwfilter.html
I think in summary, no, it's just a misunderstanding that the x.x.x.254 wasn't yours and you're just seeing one-side of traffic that's meant for someone else, and I now don't think you need to worry about the system being compromised.
However, it's still a problem if you're receiving traffic for someone else that you have no control over, especially if it is counting towards your bandwidth allowance. If you're in danger of going over your allowance because of this, you should definitely raise it with them as an issue.
As you said that you see this on other sites from this provider, it's likely they've configured something weirdly but done it the same across all sites. But if it's just traffic from x.x.x.254 that you're seeing, and not for any IP, I wonder if whatever they've done is left over from a previous config that treated .254 as a router and so allowing all those packets instead of blocking them.
Who is provider?
Blocking anything doesn't mean you won't be receiving the traffic, just that it would be stopped before reaching any service but AFTER it already reached your machine so it can be discarded as blocked. Blocked or not, the traffic would still be counted and maybe even appear in log, depending on your logging paradigm.
I very vaguely remember having a similar issue a long time ago where at the end of the day it was an issue the provider had to fix on their end.
Thanks! All really useful info, I submitted a ticket today, will be interesting to see what their response is. Hopefully I explained it well enough to them
https://github.com/imsnif/bandwhich
Yeah bandwhich was mentioned a few days ago and its use and output were discussed...
@zako12 Any word yet? I realize it's soon still, but I'd hope that a host would go "hm this seems potentially like something that needs at least an initial investigation kinda soonly..."
No reply yet although to be fair they might just not have seen it. Not sure exactly how their ticket system works but I spoke to someone on live chat about the problem which automatically opened a chat log ticket for me, and then I replied there with the new details. Maybe I should have opened a separate ticket, if I don't hear anything back by Monday I'll msg them again