XDP/eBPF DDoS filtering on VPS nodes: how do you avoid false positives across tenants?

Comments

• LEBUserJoe Member

  October 11

  dont do of which u have not researched simple

  Thanked by 2buggedout tentor
• slideFactor Member

  October 11 edited October 11

  Not a hosting provider, but I do some DDoS mitigation in-house for my own services and I've done a lot of troubleshooting for problems caused by some providers' DDoS mitigation.

  As a customer, the key thing is visibility and transparency:

  • The best thing is to document as specifically as possible how everything works
  • You may not want to do that for some security through obscurity (which we can argue for and against), but at least explicitly show to your users when you've dropped traffic so they can narrow down the problem to your DDoS mitigation. I've had to do so much troubleshooting to help a client with a packet loss issue he had with my service the other week which turned out to be caused by his provider's way too paranoid DDoS mitigation rate limiting traffic silently

  The next important thing is to be sensible about what traffic you drop:

  • My client's packet loss was caused by UDP rate limiting that was kicking in at around 4 Mbps / 500 PPS. That was virtually the entire traffic to the server at that time. How can that possibly be an 'attack' that should be mitigated. Yes, it might be sensible to rate limit things fairly low when under attack. But not when 500 PPS is the entire traffic going in.
  • If the volumetric protection actually works well, about the only thing that would be worth doing locally on the node would be stateful / L7 filtering for sensitive applications, which should only be applied if the user explicitly configures an application filter. Maybe also a TCP SYN proxy under attack.

  You should definitely have the option to customise rules per-destination-IP address.

  Always-on filtering, or rules that only kick in above a threshold?

  Always-on for explicitly configured L7 filters, threshold based for generic filters.

  UDP: blanket drop of common amplification source ports, or something smarter?

  Common amplification source ports should be blocked or rate limited by the volumetric mitigation. It's the most basic thing they should handle for you.

  Native XDP in the NIC driver or generic mode, and any NIC you'd avoid?

  Always native. I mostly use Mellanox NICs. Don't use anything older than ConnectX-4 for this, and prefer ConnectX-5 and newer. Be aware I've seen significantly worse PPS scaling with dual port NICs when using both ports, vs using a single port. RX buffer size and coalescing policy will make a pretty massive difference. Make sure that you don't have any implicit synchronisation via memory accesses or false sharing between the cores in your eBPF code, because that will absolutely destroy performance.

  I've heard pretty bad things about performance with Intel NICs.

  Happy to share what we've learned so far, including the mistakes.

  Okay, your turn