All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
CPU Routers
randvegeta
Member, Host Rep
Does anyone have much experience building software routers (on normal dedicated servers) using different kinds of CPUs?
Of course the network cards themselves are important, but I've never quite figured out how important the kinds of CPUs factor into overall performance. Certainly in the past, a lot of the router functionality was only single threaded, and so having multiple threads sometimes did not do anything to increase performance, and clock speed was an important factor. And if I'm not mistaken, routing doesn't use many of the latest optimisations of modern CPUs so often a 3Ghz CPU from 2010 would often perform very similarly to 3Ghz from 2017.
I'm just paraphrasing what I've read, and not speaking from 1st hand experience. Does anyone know how true or applicable this is?
Anyone run tests comparing an i3, to an i5, to an i7 to a Xeon E3 or AMD Ryzen 1700 or AMD EPYC?
If clock speeds are the most important factor, then the i3s should perform similarly well to an i7, depsite the doubling of cores and threads. But if routing can be easily parallelized, then the AMDs with 8 cores should have them beat. Anyone with first hand experience?
Comments
There was a thread on software routers a bit back. IIRC bottom line was a beafy CPU (single core performance) is somewhat OK. Not sure at what speeds but i think it was something along the lines of 10GBit is doable. Don't quote me on it but thats what i recall. As far as using the latest features goes i am quite certain routing won't gain much from stuff like AVX or AESNI. SSE might be useful for dealing with V6 addresses but that's pretty much stoneage.
As far as cores go you want max performance per RX/TX queue core. Any more cores for normal workloads are pointless as all your work will take place within softirqs. Desktop hardware is not going to be comparable to Xeon's and server grade Intel (!!) NICs.
What about AMD Epyc and Ryzen?
Intel Server NICs is what I'm using.
Don't need more than 2-3G of total routing capacity. Only dealing with 1G lines and typical thoughput would be only a few hundred Mbit. But it would be nice if the routers could handle small DDoS (upto 1G). Not sure if a CPU router is feasible for such things.
Not exactly what you are looking for. I guess he knows what he's talking about.
the mbit value is not necessary a problem, it's all about the packets per second. You can easily route 10 or more gbit/s with big packets but as soon as you get lots of small packets your cpu will lock on interrupts. It's the same for FreeBSD as for Linux, going into forwarding 14 million packets or beyond will make you suffer.
If you just want to get a cheap Router that is able to handle your traffic requirements go with mikrotik Cloud core router. they use special network processors (this processors are also used in ovh ddos filter) that can handle up to 94 millionen packets per second.
If an E5 can handle 20G then I suppose 2-5G would be easily handled by an E3 or Ryzen.
An E5 can handle a lot more than 20G if the system is configured correctly
And yes, and E3 can easily handle 5 gig, even 10 depending on your traffic as well.
As far as I know, the smallest Packets are around 8bytes. To hit 1G in 8byte Packets, that should be about 134M pps right?
So at 14Mpps, if all packets were small, it would start to suffer at about 100mbit?
https://kb.juniper.net/InfoCenter/index?page=content&id=kb14737
so 14m pps are 10g. You always need either special hardware or a solution to bypass the linux/bsd kernel stack.
https://reddit.com/r/networking/comments/6upchy/can_a_bsd_system_replicate_the_performance_of/
that's a good explanation of this topic. save the headache and go for something else
I am to tired to do much maths but those 8 byte packets also need headers so in reality they won't be all that small.
https://en.wikipedia.org/wiki/IPv4_header#Header
https://en.wikipedia.org/wiki/User_Datagram_Protocol#IPv4_Pseudo_Header
That's a dead end product, it uses the Tilera chips, and Tilera architecture support was just removed from the Linux kernel, with the following comment:
Mikrotik CCR router family. The products all use old (linux-3.3) kernels
with lots of patches and won't be upgraded by their manufacturers. There
have been efforts to port both OpenWRT and Debian to these, but both
projects have stalled and are very unlikely to be continued in the future. -- https://lkml.org/lkml/2018/3/14/514
Mikrotik runs on an ancient platform with no chance or no intent by the manufacturer for any upgrade. Even the Tile CPUs vendor has ceased using them in newer products:
It sounded cool (64 cores on a low power 1 GHz chip for massive paralellism) but ultimately turned out to be a dead end. ARM64 seems to be the way forward even for Tile's authors.
So where in those 8 bytes it includes the sender and destination IP (at the very least, not to mention a ton of other stuff)? And you even manage some kind of a network?... See the comment by @mksh above.
Mikrotik is running on 3.3.x so that is not actually a problem. As long as Mikrotik keep up the support I can't see any problem with the current products. JunOS was running on an old freeBSD for a long time as well.
I see your point but I am pretty sure that they are unable to scale beyond 90m pps anyway no matter if Linux support exists in 4.x and beyond.
If you need to route 90+ mpps and don't get the budget for an juniper mx204 you are having problems anyway.
I have used a lot of D525 atoms with 4-8g of ram running pfsense and multiple NiCs. These handle traffic fairly well, and are very low power consumption. They will work for most use cases.
I didn't hit issues until I was pushing about 1.5m PPS out of one location. Swapped out an E3-1220v2 with 16G and I was smooth sailing for a steady 2M PPS.
What is the smallest packet size for UDP?
Edit: nevermind. I forgot about the IP headers. + 20 bytes.
Yes my maths was way off before. So to handle 1G UDP packets of 28Bytes, looks like just 5Mpps, not 134.
The E3 Cpus should be more than sufficient for 2mpps. Anyone know if 5mpps is doable?
You're running a 2mpps router on an E3-1220v2? Is that average or peak?
removed.
You can’t compare Juniper Networks capabilities with Mikrotik however.
True you can't and if you read carefully you will see that I didn't
. Just an example to show that you don't need the newest os version to perform well.
I'm not going to touch Mikrotik, and I wouldn't touch any of Juniper's software routers either.
We have a few softwarerouters placed in various locations just running on x86 hardware, and they work great. I'm just curious about how much extra performance you can squeeze out of different CPUs, and how much of a difference it really makes. And how far you can actually push the routers in terms of DDoS. Ideally the actual port would actually get fully congested before the CPU falls over. A router under heavy load is a pain to deal with. If the smallest possible packets are 28Bytes, then I'm looking at something that can handle 5Mpps, and it should be able to handle DDoS attacks of upto 1G (port speed) without falling over. Which CPUs can handle 5, 10 or even 15Mpps?
Yeah, I’m merely pointing that Juniper has more resources to be able to backport on their own without community support.
With that in mind, naturally most Juniper routing engines would never see the traffic as this would be processed by the routing engine, contrary to Mikrotik.
@clouvider I totally agree on that. Juniper are real routers and Mikrotik not. It's not only about the asics, junos as it is pure pleasure to work with and you won't go back to cisco, brocade or similar after getting used to junos.
The start model of the Juniper MX480-960 runs on a 1 core 2Ghz with 4GB RAM (RE-S-2000-4096). I know companys that do around between 80-120 Gbps traffic and the CPU usage is around 10%
The heavy work is done in hardware though.
Desktop hardware will not perform as well, but does it perform well enough for your needs? Thats for you to decide. Personally I would go with server hardware unless you were unable to afford it.
MX480 routing engine is responsible for control plane only. Data plane (so actual forwarding) is handled by PFE(s) installed on MPC cards, which is a custom Juniper silicon.
On your PC router the control and data plane is done by CPU hence you can forget about similar results.
Average. I think I peaked around 3.5M, and at that point the DC told me to throttle.
That's pretty impressive. How much traffic is that actually? 5G?