All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Mitigating the ongoing DDoS attack against Tor relays
There is an ongoing DDoS attack against Tor relays, and there's some discussion of mitigations on the tor-relays mailing list. The attack mostly comes from Contabo and overload the relay's CPU (it's a layer 7 attack, not layer 4). Tor operators here may have noticed that their relays may be undergoing CPU spikes or overloads.
For those using relatively stock setups and iptables, there is a simple script made by toralf which generates a firewall to largely mitigate the damage. It works by:
- exempting directory authorities and snowflake distributors from any filtering
- blacklisting IPs that make more than 8 new connections over a 2 minute period for 24 hours
- blacklisting IPs that make more than 16 new connections over a 1 hour period for 24 hours
- dropping attempts to connect more than 8 simultaneous connections
If you don't want to use nftables or don't run a Tor relay, you can stop reading here. I've made a port of this mitigation tool to nftables. While it lacks persistence (blacklists are cleared on reboot), auto-updating of directory authorities and snowflake IPs (which should rarely change anyway), and only supports servers that run a single Tor process, it does additionally give more leniency to known relays, which allows it to be significantly more strict on non-relay connections which, unlike relays that may have multiple Tor processes per server with the same IP, never have a legitimate reason to open 8 simultaneous connections, so we limit it to 4.
Step 1 (create the filter)
The following is to be put in /etc/nftables.d/tor-ddos.conf:
# set of ipv4 addresses to be dropped for one day
set tor_ddos4 {
type ipv4_addr
flags timeout
timeout 24h
size 262144
}
# set of ipv6 addresses to be dropped for one day (may include whole subnets)
set tor_ddos6 {
type ipv6_addr
flags timeout
timeout 24h
size 262144
}
# simply used to keep track of existing simultaneous ipv4 connections
set tor_connlimit4 {
type ipv4_addr
size 262144
}
# simply used to keep track of existing simultaneous ipv6 connections
set tor_connlimit6 {
type ipv6_addr
size 262144
}
# known ipv4 relay addresses, populated from tor's cached-consensus
set known_relays4 {
type ipv4_addr
flags interval
auto-merge
}
# known ipv6 relay addresses, populated from tor's cached-consensus
set known_relays6 {
type ipv6_addr
flags interval
auto-merge
}
# trusted ipv4 addresses that should never be denied
define tor_trusted4 = {
# snowflake servers
141.212.118.18,
193.187.88.42,
193.187.88.43,
193.187.88.44,
193.187.88.45,
193.187.88.46,
# directory authorities
45.66.35.11,
66.111.2.131,
128.31.0.39,
131.188.40.189,
171.25.193.9,
193.23.244.244,
199.58.81.140,
204.13.164.118,
216.218.219.41,
217.196.147.77
}
# trusted ipv6 addresses that should never be denied
define tor_trusted6 = {
# snowflake servers
2a0c:dd40:1:b::42,
2607:f018:600:8:be30:5bff:fef1:c6fa,
# directory authorities
2001:470:164:2::2,
2001:638:a000:4140::ffff:189,
2001:678:558:1000::244,
2001:67c:289c::9,
2610:1c0:0:5::131,
2620:13:4000:6000::1000:118,
2a02:16a8:662:2203::1
}
# hosters that allocate /64 subnets to customers from attacks seen in the wild
define hosters64 = {
2a01:4f8::/32,
2a01:4f9::/32,
2a01:4ff:ff01::/48,
2a03:4000::/32,
2a06:be80::/29,
2a0b:f4c2::/40,
2a11:e980::/29,
2a12:2240::/29
}
# hosters that allocate /80 subnets to customers from attacks seen in the wild
define hosters80 = {
2001:67c:e60::/48,
2607:f1c0::/32,
2a00:1a68::/32,
2a00:da00:8000::/34,
2a0d:7f00::/29
}
chain tor_input4 {
# accept trusted tor servers (authorities etc)
ip saddr $tor_trusted4 accept
# limit filtering on known relays, never blacklist and allow up to 16 connections
ip saddr @known_relays4 jump {
add @tor_connlimit4 { ip saddr ct count over 16 } drop
accept
}
# blacklist any ip that makes more than 8 connections/minute over a two minute window or 16hour over an hour window
meter tor_ddos_fast4 { ip saddr timeout 2m limit rate over 8/minute burst 8 packets } update @tor_ddos4 { ip saddr }
meter tor_ddos_slow4 { ip saddr timeout 1h limit rate over 16/hour burst 16 packets } update @tor_ddos4 { ip saddr }
ip saddr @tor_ddos4 drop
# limit number of simultaneous connections
add @tor_connlimit4 { ip saddr ct count over 4 } drop
# accept the connection so we don't fall through
accept
}
chain tor_input6 {
# accept trusted tor servers (authorities etc)
ip6 saddr $tor_trusted6 accept
# limit filtering on known relays, never blacklist and allow up to 16 connections
ip6 saddr @known_relays6 jump {
add @tor_connlimit6 { ip6 saddr ct count over 16 } drop
accept
}
# is the connection from a host that offers an entire /64? (otherwise fall through)
ip6 saddr $hosters64 jump {
# blacklist any /64 that makes more than 8 connections/minute over a two minute window or 16hour over an hour window
meter tor_ddos_fast6_64 { ip6 saddr & ffff:ffff:ffff:ffff:: timeout 2m limit rate over 8/minute burst 8 packets } update @tor_ddos6 { ip6 saddr & ffff:ffff:ffff:ffff:: timeout 24h }
meter tor_ddos_slow6_64 { ip6 saddr & ffff:ffff:ffff:ffff:: timeout 1h limit rate over 16/hour burst 16 packets } update @tor_ddos6 { ip6 saddr & ffff:ffff:ffff:ffff:: timeout 24h }
ip6 saddr & ffff:ffff:ffff:ffff:: @tor_ddos6 drop
# limit number of simultaneous connections from a /64
add @tor_connlimit6 { ip6 saddr & ffff:ffff:ffff:ffff:: ct count over 4 } drop
# accept the connection so we don't fall through
accept
}
# is the connection from a host that offers an entire /80? (otherwise fall through)
ip6 saddr $hosters80 jump {
# blacklist any /80 that makes more than 8 connections/minute over a two minute window or 16hour over an hour window
meter tor_ddos_fast6_80 { ip6 saddr & ffff:ffff:ffff:ffff:ffff:: timeout 2m limit rate over 8/minute burst 8 packets } update @tor_ddos6 { ip6 saddr & ffff:ffff:ffff:ffff:ffff:: timeout 24h }
meter tor_ddos_slow6_80 { ip6 saddr & ffff:ffff:ffff:ffff:ffff:: timeout 1h limit rate over 16/hour burst 16 packets } update @tor_ddos6 { ip6 saddr & ffff:ffff:ffff:ffff:ffff:: timeout 24h }
ip6 saddr & ffff:ffff:ffff:ffff:ffff:: @tor_ddos6 drop
# limit number of simultaneous connections from an /80
add @tor_connlimit6 { ip6 saddr & ffff:ffff:ffff:ffff:ffff:: ct count over 4 } drop
# accept the connection so we don't fall through
accept
}
# blacklist any /128 that makes more than 8 connections/minute over a two minute window or 16hour over an hour window
meter tor_ddos_fast6_128 { ip6 saddr timeout 2m limit rate over 8/minute burst 8 packets } update @tor_ddos6 { ip6 saddr timeout 24h }
meter tor_ddos_slow6_128 { ip6 saddr timeout 1h limit rate over 16/hour burst 16 packets } update @tor_ddos6 { ip6 saddr timeout 24h }
ip6 saddr @tor_ddos6 drop
# limit number of simultaneous connections from a /128
add @tor_connlimit6 { ip6 saddr ct count over 4 } drop
# accept the connection so we don't fall through
accept
}
chain tor_input {
# requiring syn effectively disables loose connection tracking (nf_conntrack_tcp_loose) for this chain
ct state new tcp flags syn jump {
meta nfproto ipv4 jump tor_input4
meta nfproto ipv6 jump tor_input6
}
drop
}
Then, in your main /etc/nftables.conf, add include "/etc/nftables.d/tor-ddos.conf" at the beginning of your filter table and replace the rule that allows your ORPort with tcp dport 9001 jump tor_input (with 9001 being whatever your ORPort is). As usual, you should accept established and related connections prior. Here is an example minimal /etc/nftables.conf that allows SSH on port 22 and DDoS-protected Tor on 9001:
flush ruleset
table inet filter {
include "/etc/nftables.d/tor-ddos.conf"
chain input {
type filter hook input priority filter; policy drop;
# allow local traffic and replies to outgoing traffic
iif lo accept
ct state vmap { established : accept, related : accept, invalid : drop }
# allow tor (with ddos protection) and ssh
tcp dport 9001 jump tor_input
tcp dport 22 accept
# allow rate-limited icmp (other necessary icmp types like pmtu discovery already match ct state related)
icmp type echo-request limit rate 25/second accept
icmpv6 type { echo-request, nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } limit rate 25/second accept
}
chain output {
type filter hook output priority filter; policy accept;
}
chain forward {
type filter hook forward priority filter; policy drop;
}
}
Step 2 (prepare auto-updating of known relays)
You might notice that there are two empty sets that are only referenced but never populated. They are to hold an up-to-date list of all known relays. The script that populates them is /etc/cron.daily/update_tor_nft_sets:
#!/bin/bash
test -s /var/lib/tor/cached-consensus || exit 1
(echo "flush set inet filter known_relays4"
echo "flush set inet filter known_relays6"
runuser -u debian-tor -- awk '
/^r / { v4 = v4 (v4 ? ",\n" : "") $7 }
/^a / { match($2, /\[[^]]+\]/); v6 = v6 (v6 ? ",\n" : "") substr($2, RSTART+1, RLENGTH-2) }
END {
if (v4) print "add element inet filter known_relays4 {\n" v4 "\n}"
if (v6) print "add element inet filter known_relays6 {\n" v6 "\n}"
}
' /var/lib/tor/cached-consensus) | nft -f -
The awk command is run as the debian-tor user for security reasons (reduces the amount of untrusted data the unprivileged Tor process can force root to parse). If you aren't using Debian, change the name of the Tor user there. If your main nftables table isn't called "filter" (which is the default in Debian), you'll have to replace the hardcoded names in the above script with whatever you called it.
Now, this will run once a day to keep the list of relays up to date, but it'll be empty when nftables starts on boot. To fix that, run the following as root to trigger the script when nftables is first started and whenever it's reloaded:
systemctl edit --stdin nftables.service << EOF
[Service]
ExecStartPost=-/etc/cron.daily/update_tor_nft_sets
ExecReload=-/etc/cron.daily/update_tor_nft_sets
EOF
Note that this assumes you already have Tor installed and the consensus is already cached. If you've just started Tor for the first time, you might have to run /etc/cron.daily/update_tor_nft_sets manually after Tor has synced to the network, as it is only able to get the list of known relays by parsing Tor's own cached consensus.
If you ever want to reload the configuration, you have to use systemctl reload nftables. Do not reload it with nft -f /etc/nftables.conf as that will not restore the set of known relays from the consensus file.
Step 3 (apply configuration tweaks, optional)
Add the following to /etc/sysctl.d/tor-relay.conf to tweak network settings:
# remain in FIN_WAIT_2 for only 10 seconds
net.ipv4.tcp_fin_timeout = 10
# send keepalives every 20 minutes
net.ipv4.tcp_keepalive_time = 1200
# allow TIME_WAIT sockets to be reused for new connections
net.ipv4.tcp_tw_reuse = 1
# increase the source port range for outgoing connections
net.ipv4.ip_local_port_range = 10000 64999
# allow larger accept backlog
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
# keep inactive conntrack entries for 1 day instead of 5
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
# increase the number of conntrack entries that can be kept at once
net.netfilter.nf_conntrack_max = 131072
net.netfilter.nf_conntrack_buckets = 32768
Because you're writing to NetFilter sysctls, it's necessary to tell the kernel to load the module early. This can easily be accomplished by running this as root:
echo nf_conntrack >> /etc/modules-load.d/modules.conf
Step 4 (reduce memory pressure, optional)
Tor uses the system's default memory allocator (the library that implements memory-related functions such as malloc() and free()). For glibc systems, this is ptmalloc3. Unfortunately Tor's RSS (resident set memory) gradually grows due to memory fragmentation. Considering the ongoing DDoS attack also increases memory usage, it's important for low-resource relays to reduce memory usage. While you can reduce memory usage by setting DirCache 0 in your torrc, that disqualifies you from being a guard or serving as a directory cache. Luckily, tests were done that show that the jemalloc2 memory allocator is far better at resisting memory fragmentation. To enable it, install libjemalloc2 and then run, as root:
systemctl edit --stdin [email protected] << EOF
[Service]
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2"
EOF
If you aren't using Debian, verify that you've set the correct path for libjemalloc.so.2. That will automatically select jemalloc2 as a drop-in replacement memory allocator. You should also enable zswap (add zswap.enabled=1 to the kernel command line in /etc/default/grub and run update-grub) for further memory savings. Unlike zram, zswap has a physical backing device, but only uncompressible pages are swapped out there. Anything that compresses well will be kept in memory for faster access.
To ensure the stored swap is encrypted, install cryptsetup and systemd-cryptsetup and run, as root:
echo "swap SWAP_DEVICE /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=128,sector-size=4096" >> /etc/crypttab
echo "/dev/mapper/swap swap swap defaults 0 0" >> /etc/fstab
Replace SWAP_DEVICE with a swap file or existing swap partition. Make sure you disable any existing unencrypted swap in /etc/fstab first.
Once you've done all this, reboot. You should see a decrease in your CPU usage during times of DDoS and your relay will be able to handle more legitimate connections. This is especially important for low-end relays that are already only barely able to handle the load they're given.
Disclaimer: This is still a WIP. I'm in the process of deploying it on over 50 relays, so I may very well post updates with improvements or bug fixes. Additionally, it's not designed to protect against high-PPS floods. It's purely designed to limit the damage of a specific class of application-layer DDoS attacks against the Tor protocol. This is the reason why there are no micro-optimizations and why the filtering is not done in prerouting. Still, it should result in a significant reduction of CPU and memory strain caused by the ongoing attack.
Some miscellaneous ideas for the future:
- Periodically update the authority and snowflake list (see
__fill_trustlistin toralf's original scripts). - Save and restore the blacklist across reboot, perhaps with
ExecStopPreandExecStartPost. - Log a message (but take no action) if there are many connections from the same v4 /24 or v6 /64.
- Don't blacklist IPs if Tor was just installed and there's no consensus file to obtain known relays from.
Comments
Do you have consider have website or blog to preserve your article/though? I kinda like stuff like this but, I hopefully can following on my RSS (sorry, I am RSS Junkie).
I don't have any website. I just write things here and on Stack Exchange.
I don't always see these threads either but I like them. I'd donate $7 for a year of blog hosting!
Soldiers, raise your shields!!
I love to run tor relays, however, its using so much memory, it nearly feels like its written in java.
It buffers a lot of traffic before actually relaying it I believe? Also @forest mentioned that using jemalloc lowers RAM usage somewhat, have you tried it?
So, like since 2023?
https://lists.torproject.org/mailman3/hyperkitty/list/[email protected]/thread/JOLAHNC7HGENT6C34I5Q3GZ4MNTAOQZB/
I wonder why? though? I mean sabotaging exits in some way would be understandable, after all those exits create plenty problems and pain to websites due to trolls and marauders. I in fact had all exits blocked for quite a while and updated my list daily because it was so bad.
But relays? I don't see why one (i.e. private persons) would sabotage relays.
I also wonder why? What's the point of attacking Tor Relays?
Very nice @forest, thank you.
I don't run any tor relay but I have been writing my own nftables firewalls for a few years.
Actually it's what I enjoy most doing with my dedis and vps.. The cat & mouse of it plus learning more about nftables has substitued my computer game playing, which goes back to the zilog days and then on thrugh about all other cpu that came along.
That was a different attack. It subsided. This one has only been around for a short time.
It's definitely not an attempt to cripple or ruin the network. The average user won't see all that much in terms of performance degradation.
I'm guessing it's either (rogue) research or part of a deanonymization attempt against a specific individual or onion service. Attacking relays (and specifically guards) increases guard churn. Normally, the first relay, the guard, is something you stick with for months while the middle and exit rotate. But if your guard is overloaded, your client will try new guards. And if the new guard is also overloaded, it will switch again until eventually you hit a malicious guard just long enough for the attacker to confirm your identity. There are some defenses against this already (Tor won't just keep switching guards indefinitely without warning you that something is wrong), but that gives an idea of the kind of thing such an attack might be useful for.
But that's just a guess. There are a number of other attacks that can be made somewhat easier by attacking certain relays and causing them to overload. Thankfully this does go to show just how robust Tor is, since even such an attempt takes a long time, whether or not it will succeed, and is incredibly noisy (the entire Tor network becomes aware of it).
Yeah, my gut pointed in the same direction. Thanks, it's good to hear it from a pro!
Is Stack Exchange alive and kicking ？
I thought they'd been replaced by GPT lol.
If you run a forum, have you considered only blocking exits from registering? If the problem is that bad actors are using Tor to get around your moderation, you can prevent them from getting in while still allowing Tor users to view the site. I'm a big fan of minimizing damage to Tor legitimate users while still allowing webmasters to keep Tor-using trolls under control.
Another possibility if you want a perpetually up-to-date list is to run your own non-exit relay or bridge on the site and simply parse
/var/lib/tor/cached-consensusto create your ipset/nftables set blacklist.Ehh... Participation has dropped significantly (mostly because all the "easy but popular" questions have already been asked), but that drop began before GPT. It's still the go-to for high-quality answers and generative AI (still) gives horribly wrong answers for anything that can't be trivially verified. SE won't die any time soon but it's no longer "hopping".
sometimes people do shit for no reason or they're just bored
It's largely memory fragmentation and the directory cache. There is some buffering of network traffic to memory (controllable with
MaxMemInQueues) while the scheduler makes its decisions, but it doesn't add substantially to your memory use unless you become a very high-capacity relay.The real issue has something to do with the way relays (guards in particular) serve cached directory files.
For a Tor client to connect to the network, it has to have an authoritative and up-to-date copy of vital network information called the consensus which contains, among other things, the list of relays' IPs and ORPorts, their cryptographic fingerprint, and what type of relay they are. The central Directory Authorities are the ones that generate the consensus, but with millions of clients, the authorities could never possibly bootstrap every single client every time it connects.
One of the solutions is "directory caches". Every guard relay will, by default, offer its own copy of the consensus (alongside other directory files) to their clients so that it can keep up-to-date without bothering the DAs. This ensures that the DAs only need to be bothered by clients when they connect to the network for the very first time. Any updates to the consensus are instead distributed by the relays themselves.
The problem with that is that you have to maintain a large cache of diffs in /var/lib/tor/diff-cache that are all perpetually memory-mapped, and you have to constantly compress and serve them to new clients. For some reason, that causes severe memory fragmentation under ptmalloc3. You can reduce memory usage substantially by disabling directory caching (
DirCache 0), but then you are disqualified from being a guard. The best solution I've seen is to use jemalloc2, which allows you to serve as a guard serving a cached consensus to clients while still avoiding heavy memory use. The mimalloc2 allocator is even better, but it's not present in Debian (which only has mimalloc3 which does not help).Speaking of firewalls, recently once I got a ton of "conntrack table full, dropping connection" in dmesg on my relay, so I switched to a stateless set of iptables rules, and don't seem to get any problems since.
The default max limit for conntrack table entries is enough for a web server but not nearly enough for anything like Tor or any mixnets or large-scale P2P applications. You'll want to set the following sysctls:
Then ensure the NetFilter module loads before the sysctls are applied by running, as root:
Then you'll get rid of that issue. It'll be more secure than just using a stateless firewall.
I always use stateless on Tor relays anyway, just forgot this time. Main reason is I want to minimize CPU usage in this area as well. That comes from the practice of running relays on Atoms and VIA Nanos. Would not say it's less secure in any significant way, e.g. SSH is still only allowed for a whitelist of IPs that I connect from.
True, but you can't do DDoS mitigation like the type I posted about without conntrack. :P
IMO the statistics on which relays are DDoSed and which are not might be interesting (if that is possible at all).
It's possible to estimate if a relay is being DDoSed by looking at its traffic rate. If it goes down suddenly, it could be due to being overloaded. And if it goes into an official overload state (which happens if the overload is so bad that it needs to alert clients to de-prioritize building circuits through them), that's even more evidence.
Overloads happen from time to time under normal circumstances, but just recently something like 6 of my relays went into a severe overload state at once (all guards), whereas before maybe one would overload each month (usually do to a noisy neighbor on the node bringing up CPU steal). The attack most severely effects relays that have very little margin for overload, e.g. those with very little free memory and which are often at >80% CPU, because they don't need much to push them over the edge.
I doubt they're attacking every single relay but theirs. Maybe they're attacking whole groups of relays? They might be going after their own relay too, which wouldn't necessarily make their attempts unsuccessful.
The attackers are mostly (or entirely?) originating from Contabo IPs. I wish Contabo was active on LET so they could investigate and chime in.
Is it the NSA or CIA?
No way to know, but most likely not simply because there are so many threat actors out there, and the NSA and CIA usually have other ways of tracking someone down (e.g. simply sending a browser exploit to them).
I think netflow analysis still works too
That threat has been weakened by padding: https://spec.torproject.org/padding-spec/connection-level-padding.html
It sends periodic keepalives that "collapse" NetFlow records so that the record reports a continuously active stream so that it only reports packet numbers and total transferred data in 30 minute windows rather than potentially finer-grained windows.
That doesn't mean the analysis is impossible to perform, but it's not the treasure trove of metadata it once was.
always pleasant to read. i need to stop procrastinating to use nftables in my debians nodes
Some context written up by a major Tor relay operator: https://1aeo.com/blog/tor-network-dos-wave-june-2026.html
And a much deeper analysis of the attack: https://1aeo.com/blog/circuit-building-dos-anatomy-june-2026.html
He also discovered that guards that are running out of memory due to the DDoS can limit memory growth by setting the
DosCircuitCreationBurst 20torrc option (default is 60). This won't help with CPU overload but it will help with memory pressure for guards under attack. The [email protected] mailing list is good for keeping up to date.It's dead. https://data.stackexchange.com/stackoverflow/query/1926661#graph
The result of over a decade of ignoring what users wanted. Completely their own fault.