Mitigating the ongoing DDoS attack against Tor relays

jsg · July 23

@forest said:

@DataRecovery said:

@jsg said: But relays? I don't see why one (i.e. private persons) would sabotage relays.

IMO the statistics on which relays are DDoSed and which are not might be interesting (if that is possible at all).

It's possible to estimate if a relay is being DDoSed by looking at its traffic rate. If it goes down suddenly, it could be due to being overloaded. And if it goes into an official overload state (which happens if the overload is so bad that it needs to alert clients to de-prioritize building circuits through them), that's even more evidence.

Overloads happen from time to time under normal circumstances, but just recently something like 6 of my relays went into a severe overload state at once (all guards), whereas before maybe one would overload each month (usually do to a noisy neighbor on the node bringing up CPU steal). The attack most severely effects relays that have very little margin for overload, e.g. those with very little free memory and which are often at >80% CPU, because they don't need much to push them over the edge.

I doubt they're attacking every single relay but theirs. Maybe they're attacking whole groups of relays? They might be going after their own relay too, which wouldn't necessarily make their attempts unsuccessful.

The attackers are mostly (or entirely?) originating from Contabo IPs. I wish Contabo was active on LET so they could investigate and chime in.

I'm not sure but I might still have the contact data of one of their higher up techies. If I find those I'll try contacting them and bring the problem to their attention.

jsg · July 23

@OpaqueRegistrant said:

@Dyingcat said:

Is Stack Exchange alive and kicking ？
I thought they'd been replaced by GPT lol.

It's dead. https://data.stackexchange.com/stackoverflow/query/1926661#graph

The result of over a decade of ignoring what users wanted. Completely their own fault.

Oopsie

jsg · July 23

@forest

Done, email to Contabo contacts sent. Hopefully they'll react (although probably only tomorrow).

forest · July 24

@OpaqueRegistrant said: The result of over a decade of ignoring what users wanted. Completely their own fault.

1000%. Even the moderators despise what the staff do. And somehow every action the company takes makes things worse, as if that was their only goal.

jsg · July 24

@forest
Sorry, no response so far. But keep in mind that my last contact with the Contabo people was >= 5 years ago, so maybe the two I tried to contact don't work there anymore or the email addresses have changed.
If and as soon as I get a response I'll let you know here.

JohnFilch123 · July 24

Contabo is know for being 'friendly' towards spammers and other bad actors, I suppose they will ignore complaints.

forest · July 25

Some of the attacks are moving to OVH now.

I might test out blocking an IPv4 /24 if more than 6 IPs connect from it at once, or a /32 if it's connecting to more than 6 of my relays at once (there's no reason a single IP should have 6 guards, much less 6 guards from my relay family). Of course, exempting other relays which do have legitimate reasons to make many connections.

default · July 25

Fourplex · July 25

I run a Tor relay on a Spectrum cable modem and saw ~70 Mbps in sustained use (this is non-high-split symmetric).

The attackers abused the DOCSIS burst heavily so I had to throttle the relay to ~25 Mbps.

rm_ · July 25

@Fourplex said: I run a Tor relay on a Spectrum cable modem and saw ~70 Mbps in sustained use (this is non-high-split symmetric).

The attackers abused the DOCSIS burst heavily so I had to throttle the relay to ~25 Mbps.

Why do you think it was any kind of DDoS though, 70 Mbps sustained just sounds normal for an average Tor relay operation.

Of course you can reduce that if it's too much for your connection. But rather than throttling it via external means, set up your bandwidth in torrc.

forest · July 25

@rm_ said:

@Fourplex said: I run a Tor relay on a Spectrum cable modem and saw ~70 Mbps in sustained use (this is non-high-split symmetric).

The attackers abused the DOCSIS burst heavily so I had to throttle the relay to ~25 Mbps.

Why do you think it was any kind of DDoS though, 70 Mbps sustained just sounds normal for an average Tor relay operation.

Asymmetric use maybe? There have recently been some strange issues where relays have had nearly 10x outbound rate than inbound, which I can only assume is someone abusing the directory cache somehow.

forest · July 26

This updated tor-ddos.conf may do better in some situations. I'm still testing it:

define mask4_24 = 255.255.255.0
define mask6_64 = ffff:ffff:ffff:ffff::

set tor_ddos4 {
        type ipv4_addr
        flags timeout
        timeout 24h
        size 262144
}

set tor_ddos6 {
        type ipv6_addr
        flags timeout
        timeout 24h
        size 262144
}

set tor_connlimit4 {
        type ipv4_addr
}

set tor_connlimit6 {
        type ipv6_addr
}

set known_relays4 {
        type ipv4_addr
        flags interval
        auto-merge
}

set known_relays6 {
        type ipv6_addr
        flags interval
        auto-merge
}

define tor_trusted4 = {
        # snowflake servers
        141.212.118.18,
        193.187.88.42,
        193.187.88.43,
        193.187.88.44,
        193.187.88.45,
        193.187.88.46,
        # directory authorities
        45.66.35.11,
        66.111.2.131,
        128.31.0.39,
        131.188.40.189,
        171.25.193.9,
        193.23.244.244,
        199.58.81.140,
        204.13.164.118,
        216.218.219.41,
        217.196.147.77
}

define tor_trusted6 = {
        # snowflake servers
        2a0c:dd40:1:b::42,
        2607:f018:600:8:be30:5bff:fef1:c6fa,
        # directory authorities
        2001:470:164:2::2,
        2001:638:a000:4140::ffff:189,
        2001:678:558:1000::244,
        2001:67c:289c::9,
        2610:1c0:0:5::131,
        2620:13:4000:6000::1000:118,
        2a02:16a8:662:2203::1
}

chain tor_input4 {
        ip saddr $tor_trusted4 accept
        ip saddr @known_relays4 jump {
                add @tor_connlimit4 { ip saddr ct count over 16 } drop
                accept
        }
        meter tor_ddos_fast4 { ip saddr & $mask4_24 timeout 2m limit rate over 8/minute burst 8 packets } update @tor_ddos4 { ip saddr & $mask4_24 }
        meter tor_ddos_slow4 { ip saddr & $mask4_24 timeout 2m limit rate over 16/hour burst 16 packets } update @tor_ddos4 { ip saddr & $mask4_24 }
        ip saddr & $mask4_24 @tor_ddos4 drop
        add @tor_connlimit4 { ip saddr ct count over 4 } drop
        add @tor_connlimit4 { ip saddr & $mask4_24 ct count over 16 } drop
        accept
}

chain tor_input6 {
        ip6 saddr $tor_trusted6 accept
        ip6 saddr @known_relays6 jump {
                add @tor_connlimit6 { ip6 saddr ct count over 16 } drop
                accept
        }
        meter tor_ddos_fast6 { ip6 saddr & $mask6_64 timeout 2m limit rate over 8/minute burst 8 packets } update @tor_ddos6 { ip6 saddr & $mask6_64 }
        meter tor_ddos_slow6 { ip6 saddr & $mask6_64 timeout 1h limit rate over 16/hour burst 16 packets } update @tor_ddos6 { ip6 saddr & $mask6_64 }
        ip6 saddr & $mask6_64 @tor_ddos6 drop
        add @tor_connlimit6 { ip6 saddr ct count over 4 } drop
        add @tor_connlimit6 { ip6 saddr & $mask6_64 ct count over 16 } drop
        accept
}

chain tor_input {
        # requiring syn effectively disables loose connection tracking (nf_conntrack_tcp_loose) for this chain

        ct state new tcp flags syn jump {
                meta nfproto ipv4 jump tor_input4
                meta nfproto ipv6 jump tor_input6
        }
        drop
}

chain tor_kill_established {
        type filter hook input priority filter - 1; policy accept;

        ct state established tcp dport $tor_orport jump {
                ip saddr & $mask4_24 @tor_ddos4 drop
                ip6 saddr & $mask6_64 @tor_ddos6 drop
        }
}

This simplifies the blacklisting by only dealing with IPv4 /24 and IPv6 /64 so the hoster end-site prefix length doesn't need to be hardcoded. It also adds a per-/24 (and /64) max connection limit in addition to keeping the per-/32 (and /128) and additionally blocks established connections that have gotten blacklisted.

I'll be keeping an eye out for false positives.

itoffshore · July 27

drop nftables traffic in the ingress chain (i.e before conntrack) - not the input chain

I block 5-6000 ip's with fail2ban / nftables for zero cpu cost

forest · July 27

@itoffshore said: drop nftables traffic in the ingress chain (i.e before conntrack) - not the input chain

It's a layer 7 DDoS, so even at its most severe, it's not going to be stressing nftables at all. That's why I didn't put much effort into micro-optimizations. But conntrack is needed anyway because it has to update the blacklist timeout each time too many new connections are attempted (i.e. an IP that attempts to connect too fast while blacklisted will reset the timeout).

In the past, for heftier servers experiencing layer a layer 4 DDoS, I had to write BPF code directly and use xt_bpf which worked wonders. That was kept in the (iptables) raw table. I wish nftables had BPF support.

rm_ · July 28

I now see (and remembered it from before -- I even have scripts for crontab to deal with this) another symptom of the DDoS. RAM usage leak, i.e. it climbs for no reason and stays there, in a bit of unexplained way. Affected nodes will end up using a lot of swap over time, like 1.2 GB swapfile use on a 2 GB RAM VPS - even though Tor is only consuming like 1GB "RES" in top. There's also very little "avail" mem, so it's not simply in "buffers/cache". Where is it, who knows. And Tor performance drops off, since it constantly needs to fetch some data from swap and write out other data. A lot of "kswapd" in top, and iowait%.

The solution for me is to kill the Tor process, swapoff -a, swapon -a and start Tor again. This helps for a decent period, like 6-12 hours or more.

forest · July 28

@rm_ said:
I now see (and remembered it from before -- I even have scripts for crontab to deal with this) another symptom of the DDoS. RAM usage leak, i.e. it climbs for no reason and stays there, in a bit of unexplained way. Affected nodes will end up using a lot of swap over time, like 1.2 GB swapfile use on a 2 GB RAM VPS - even though Tor is only consuming like 1GB "RES" in top. There's also very little "avail" mem, so it's not simply in "buffers/cache". Where is it, who knows. And Tor performance drops off, since it constantly needs to fetch some data from swap and write out other data. A lot of "kswapd" in top, and iowait.

The solution for me is to kill the Tor process, swapoff -a, swapon -a and start Tor again. This helps for a decent period, like 6-12 hours or more.

Set DosCircuitCreationBurst 20 in the torrc. That helps significantly with memory growth from the current DDoS. The other solution (which should be done even if you aren't being DDoSed) is to switch to jemalloc2 (or mimalloc2 if it's available)

The problem is that Tor is naturally subject to a lot of memory fragmentation, particularly due to the consensus diff cache, and the native glibc memory allocator, ptmalloc3, suffers from fragmentation badly. The DDoS just exacerbates this issue.

Just using jemalloc2 alone reduces memory usage by more than 75%.

rm_ · July 28

@forest said: The other solution (which should be done even if you aren't being DDoSed) is to switch to jemalloc2

Isn't that something for Tor itself to do? It always felt that aside from running relays, we are also expected to crank their shit so much for them.

I could see adding a line to torrc, but hacking the initscript with a custom .so preload is beyond reasonable. And then keep ensuring that hack is reapplied after every Tor update, gl hf.

forest · July 28

@rm_ said: Isn't that something for Tor itself to do?

It wouldn't be really feasible for them to do that because they just use whatever malloc() the system makes available to them. It just so happens that glibc's ptmalloc3 has pathological behavior in this case.

Tor Project is rewriting Tor in Rust (the daemon is called Arti), so the Tor binary written in C is getting fewer updates for the time being. It's still being maintained of course, but major changes to the way it handles memory won't be necessary as soon as Arti is released. But it's still in beta so we're stuck with C Tor for now.

@rm_ said: but hacking the initscript with a custom .so preload is beyond reasonable

No hack needed, just two commands:

apt install libjemalloc2

systemctl edit --stdin [email protected] << EOF
[Service]
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2"
EOF

Then it'll be applied automatically on every upgrade without modifying Tor itself (which is the standard way for changing the memory allocator for an arbitrary service under systemd). I think OpenRC can do something similar.

If for whatever reason you aren't able to do that, an equivalent fix would be to add DirCache 0 to your torrc. That will disable the functionality that causes the most severe memory fragmentation, but the downside is that your relay will be disqualified from the guard position so it will only be able to act as a middle. Still better than OOMing all the time, though.

rm_ · July 28

@forest said: standard way for changing the memory allocator for an arbitrary service under systemd

I won't use systemd, hence I said initscript. I am not sure if /etc/init.d/tor is overwritten on Tor updates. Same as for your [email protected], technically it should.

@forest said: an equivalent fix would be to add DirCache 0 to your torrc

Added that, thanks. Always found that becoming a guard ruins the BW numbers in any case, at least for a while.

@forest said: It wouldn't be really feasible for them to do that because they just use whatever malloc() the system makes available to them. It just so happens that glibc's ptmalloc3 has pathological behavior in this case.

I mean they could preload jemalloc if it's available, or have a setting in torrc to do so.

forest · July 29

@rm_ said:
I won't use systemd, hence I said initscript. I am not sure if /etc/init.d/tor is overwritten on Tor updates. Same as for your [email protected], technically it should.

That's fair enough (I'm no fan of systemd myself, but I use Debian for simplicity and that's what it comes with). I believe the init script is overwritten, although usually it might source the defaults file. Check if the init script has something along the lines of [ -f /etc/defaults/tor ] && . /etc/defaults/tor or equivalent. If so, you can just export LD_PRELOAD in the defaults file and it will stay unmodified across updates (that's what the defaults file is for).

The [email protected] doesn't get overridden at update because I used systemctl edit command which creates a separate /etc/systemd/system/[email protected]/override.conf file.

@rm_ said: Added that, thanks. Always found that becoming a guard ruins the BW numbers in any case, at least for a while.

It's probably ruining the BW numbers because the memory fragmentation is creating an I/O bottleneck from swap (or from the diff cache being paged in over and over). Guards usually have higher bandwidth, but if being a guard is crippling the relay with memory pressure, then it could certainly make bandwidth worse.

@rm_ said: I mean they could preload jemalloc if it's available, or have a setting in torrc to do so.

Ideally they'd just fix whatever is causing the pathological memory fragmentation behavior.

Howdy, Stranger!

Categories

In this Discussion

Mitigating the ongoing DDoS attack against Tor relays

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

Mitigating the ongoing DDoS attack against Tor relays

Comments