All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Question - a lot of packets flying around
Hi,
I'm not that network/virt knowledgable person, so I'm asking smarter people to confirm/deny my suspicions.
Situation: VPS at a small provider being hosted at a large company (using the big company's IP subnet). The big company having lots of different subnets.
My VPS being completely idle is showing network activity on rx side: ~4-5GB/day with peaks of traffic up to 10Mbit/s.
So, started tcpdumping.
tcpdump -vv -n host not MY_IP and not arp and not icmp and not udp
there's A LOT of packets flying around from/to the big company's subnets.
Grabbing few minutes of dumps gives me nice SMTP sessions, HTTP traffic, you name it.
Opened a ticket with the small company (as they are the ones I paid), but no response for now.
I checked my other VPSes, some don't show any traffic like that, some show, but way less.
I think this should not be the case (I have dumb Ethernet hubs in my head), but can you please confirm?
Being able to get some other service's (in a completely different subnetSMTP session by just running tcpdump on my VPS does not seem right to me.
Not including any company specifics (at least for now)...
Thanks!
Comments
Is this a NAT VPS and is it LXC/OpenVZ?
No and no.
Public IP and:
I've just noticed 15 Mbit/s on my interface with someone's ssh/scp session...
What could be happening is on the host they're just attaching every IP address they bought from the big company onto one network adapter, and passing it into the VM while assigning the VM a single one of those IPs using /etc/network/interfaces or something. This is obviously just an awful idea in general but...
Unfortunately I have a feeling that the IPs that the traffic that I see on my interface is directed to big company's clients, not the small one...
I mean I guess it matters what the traffic is, the direction it's flowing, etc. Plenty of times I've spun up a new server to find out it's IP was previously assigned to someone who used it as a significant part of their workflows, but they never changed their configs, so I was being hammered with traffic from their other servers nearby for various reasons. For example, their monitoring software trying to hit every major service port to check if online.
SMTP could be stale MX records and cron job notifications from nearby servers, for another example.
Basically I treat any undesired traffic the same way. No need to get caught in the details usually, just blackhole IPs that are generating traffic that you don't value. The answer as to why you have traffic you didn't ask for is usually not interesting, and that the traffic is local probably isn't interesting either. It could be, it's just usually not. You'd definitely need to know more to know that it's interesting.
But, both source and destination IPs are not the IPs I got with VPS. It's not someone hammering my new IP. It's traffic from a lot of different places directed to different IPs owned by the big company. It's not only smtp, there's a lot of http traffic, ssh/scp, you name it.
If I see 15 mbit on ssh session between some completely outside IP and an IP from my provider it means that the traffic is not someone hammering me because of stale dns entries
Suppose that could be interesting then
🍿
I think Hetzner (used to?) runs a kind of similar config with (some of?) their dedis. Running tcpdump you'll see tons of traffic directed at neighboring IPs. If my memory doesn't fail me you won't see any of the replies though. Still if i'd have cared to i could have probably found a bunch of unencrypted stuff among the dumps, so morale of the story: Encrypt your traffic kids. This isn't the 90s anymore.
I've seen some unencrypted bits, but not sure if there were replies as well, will take a look in a bit to make sure.
Yep, I see only packets TO the big company's IPs, nothing out. But that's still way too much...
Someone messed up bridge configuration. Properly scaling L2 networks while doing it in a secure way is a challenge. This is why I prefer routed over switching networking whenever possible.
The issue was escalated to the bigger company, but they don't see a problem:
"the traffic could be for other servers on the same switch"...
Luckily I was able to cancel the service and get a refund...
The big ISP are also offering their services here, I'll drop them a message, but apparently they think it's how it's supposed to be. A bit frightening when looking at the scale of their services.
They put all IPs into one VLAN and didn't setup VXLAN.
@yoyek
This could happen in a network configuration with asymmetrical load balancing when one of the switches expires mac entry and doesn't receive packets from the client due to asyterical config, but the router still sends frames through this switch because have an active arp entry of the client. The mentioned switch will forward frames with unknown dst to every port. Then, if such frames are received by the host machine with bridges, it will be forwarded them to every VM in this Vlan. If big provider didn't solve it, small host was able to fix it on host machine with ebtables, but possible he hadn't enough skills or just didn't want.
The small host does not seem to have much knowledge, it seemed that even running tcpdump was a mystery for them.
I did talk to someone from the big host - apparently they've put more than one customer in the same vlan (as a part of ongoing network upgrade/migration).
Maybe that's the case, luckily I was able to cancel the service and get the refund. I've owned the service for two days, and this was only one of few problems I've faced there.