New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Strange traffic pattern - What could be the reason?
Dear all,
one of my servers shows some strange and unusual traffic pattern since some hours:
Normally, that server rarely sees incoming traffic and never in those amounts.
What would be your steps to find out about the reason and to possibly stop it?
Any hint would be appreciated!
Kind regards and thanks in advance,
Amitz
Comments
Have you run iftop on the server itself? Should be able to see the source of the connections /data flow.
I'm thinking access logs for any public facing service. Maybe xmlrpc attack? Where relevant of course.
Sorry for the lack of initial information:
The server hosts a website behind cloudflare. Most IPs (incoming) that I see in iftop are Cloudflare IPs. The only access logging could be done by nginx on the server, but it does that all the time without those traffic patterns. The website is not based on any CMS like Wordpress or others. I see not requests for xmlrpc files when going through the logs.
maybe have a look into darkstat to gather some other statistics...
You don't need Wordpress installed on your site to be attacked using XMLRPC, the attacker floods vulnerable Wordpress installation and abuses their XMLRPC to request your website multiple times (pingback flood).
Anyway, as you say you're not able to see this in logs. Is there any reason you're not fetching the connecting user's real IP instead of the Cloudflare IPs?
See here.
This way you should be able to see what IPs are actually consuming that bandwidth.
Did you purge the cloudflare cache?
Darkstat looks very interesting, I will check that! Thanks!
Thank you very much, I did that already and can see the original IPs in the nginx logs. It's iftop where I only see the Cloudflare IPs. I guess there is no way to circumvent this?
I indeed did that, good point! But that was around 10 days ago, so I fear that this might not be the reason.
The more I stare at the iftop output, the more I think of a small distributed flood. There are strange IPs from Turkey, China and Germany (Contabo) with spikes in incoming packets. I banned a lot of them already via the Cloudflare Firewall, but there are new ones popping up every couple of minutes. That site has never been prone to attacks like this and - to be honest - if it is an attack, then it is quite small...
Any further help is appreciated, thank you guys!
Are you using Websockets? You may not see anything meaningful in your Nginx logs.
Try a security audit. Maybe something like Burp, or IronWasp?
also just because your server is behind cloudflare does not mean you cannot get direct incoming traffic. something like iptraf is a good start.
https://support.cloudflare.com/hc/en-us/articles/200170706-How-do-I-restore-original-visitor-IP-with-Nginx-
Blame it on the provider and demand a refund.
Tried that already. Sent them a ticket and scared those losers to death by threatening to punish them with a very, very negative review here on LET to completely destroy their scam operation. But Leaseweb did not care. :-)
Just kidding, of course!
Blame it on the provider, demand a full refund, do a charge back.:p
It obviously is an UDP flood. I wonder whether there is anything that I could do against it...
Interesting read, is this added traffic causing any harm? If not, give it some time and see if it clears up on its own. Have you tried rebooting the server? Maybe a service unknowingly started?
nothing you can do unless your provider cut it on some kind of external firewall
Well, it's a Leaseweb VPS. They offer some kind of external firewall thing included - I will have to have a look at this.
No, it is just eating bandwidth at the moment and my monthly limit at Leaseweb is IN+OUT combined, so that sucks a bit... ;-)
Are you using their .de location?
I have their nl plan ( 6TB) and , us (6TB), sg 2x(8TB). I always max them all
No, the NL location... It sucks to have that much background noise...
NL performs VERY well . How about iftop you could manually block them or make a small app (?)
"tcpdump -i eth0 udp port not 53 and not icmp" will help you design a rule to block the traffic, or if you are still stuck you can -w write it to a file and analyse it offline using Wireshark to see the most common ports, protocols (without any filters) and hosts for both incoming and outgoing traffic.
So for something like this I could block UDP port 4434, or look into blocking any UDP port that gets a lot of traffic from a wide range of IP's - which would indicate spoofed traffic? It might even be possible to block 0 length packets, but that might break something else!
Why don't you setup iptables input chain with a default policy of DROP after only allowing ports that you use.
For HTTP traffic you could allow only cloudflare IP's from https://www.cloudflare.com/ips/ ?
Something like "iptables -I INPUT ! -s 103.21.244.0/22 -p tcp --dport 80 -j DROP" will block all tcp port 80 traffic not from 103.21.244.0/22.
Large uploads / POST requests?
Leaking UDP or NTP?
How exactly this prevents traffic from hitting the link? you drop it fine, but it still gets to the link and this is what matters.
Not if it's TCP
I have contacted Leaseweb and they answered as follows:
I can sleep well with that and will now simply see those UDP requests as unavoidable background noise...
Something is wrong with their design then
Was my first thought too, to be honest...
I wonder what other (negative) implications that design has.
What do they actually mean? You do receive traffic for your neighbors in the VLAN but as per their design they cannot prevent this from happening?
Or is it meant like you get spammed/flooded with traffic that is spamming you in particular but cannot be filtered appropriately?
That would be what I understand but I don't really see why it has to be like that. Did they give you any more information @Amitz? Very strange behavior indeed. Also it's strange because you didn't have that pattern before, how come did it start some day?