New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Are they Tiny series boxes?
Could be HP Mini boxes.
Could be, and could be the dell optiplex mini, or maybe the chrome boxes who knows.
Gawd if your customer support wasn't bad, I'd be all over these. But $10 for bumping a ticket or wait 6 days for hard reboot, nah. I'd pay a small premium extra for support and a working panel.
1) VAT excluded
2) It's more on the infrastructure and SW feature set side. This is a fully custom platform, to best our knowledge no one is attempting the density we are targeting etc.
You clearly have no idea what FIB is ... .
In JunOS / Juniper terms (brocade calls it something else etc.) it IS the active routing table. You cannot put it in the DRAM because then it literally is NOT the active routing table.
Routing table has to be stored inside the ASIC for minimum latency, nanoseconds matter here. Every bit matters here.
FIB is essentially the lookup table to see where to route this packet, i don't know how many lookups it exactly needs but adding even 0.1ms of latency here is a no-go largely.
The FIB cache? It is the routes ingressed from various BGP sessions, and the FIB itself is formulated from all of these based on your filters, weights, net sizes etc., so no 2 routes for the same network goes to the FIB, but only the route you picked.
How much latency does that add?
Well Ryzen has 7-21nanoseconds core to core latency, cache is 0.8ns L1, 2.4ns L2, 10.6ns L3 and i think the last metric is for DRAM which is probs 100ns, logarithmic chart: https://images.anandtech.com/doci/17276/CacheRamp.png
Article: https://www.anandtech.com/show/17276/amd-ryzen-9-6900hs-rembrandt-benchmark-zen3-plus-scaling/2
Here people talk about 175 to 200ns: https://www.reddit.com/r/Amd/comments/w0pl6z/flck_vs_ram_cas_latency_5800x3d/
Regardless, router would not have that fast, these are typically a generation or two behind in their DRAM architecture etc. it takes years and years to make a new platform, low volume so the cutting edge they do now? It trickles down to mere mortal businesses maybe in a decade or so.
Juniper MX series: https://en.wikipedia.org/wiki/Juniper_MX_Series
It is literally decade ago release for latest version. So we are talking more about Core2Duo age platform here, which have had various upgrades on the road to here.
Because these are so expensive they are used for decade+.
Now tell me how much would have extra 150-200MiB of L1 cache cost back in 2013? Would there have even been large enough reticle size fabs to do that? I don't actually know the mm2 area of the chips on something like Juniper MICs. I should teardown a old brocade 1gig linecard to see what is hidden underneath it.
Even tho MX480 has atleast for some bundles already EOL, and End of support is slated for 2026, brand new linecards are still ordereable and not sure if even that old, 4x 100G port linecard is around 35k € mark by itself.
Empty chassis are 5-10k € used, new chassis is many times of that.
These not general purpose computers, these are very highly specialized application specific hardware.
So "just add routes" is not cheap. You cannot run of DRAM. ALL the routes have to be stored inside the silicon, or you just don't have that route, simple as that.
You can always decrease the number of routes inside the silicon, but that means you start degrading your network quality.
We've even noticed that some HW Router fault modes is that when you store enough routes your L2 MAC lookup tables start to go wonky, quietly without any error messages, and needs a 3-5min hardware reboot.
Networking at this scale is hard, it is infact super hard. Documentation is sparse, help is sparse because this is such a specialized field with not that many people working around it, and many want to closely guard their secrets unless they have something to gain. or simply that there is no one to talk to about this stuff, so things are not shared.
For example, you got 2 vlans on same TOR, both vlans go in same physical links as a trunk to next aggregation layer. You have Node A in VLAN #1 communicating with Node B in VLAN #2. Does the packet go from Vlan #1 to Vlan #2 inside TOR directly or does it go to next aggregation layer?
Yes, it goes through the uplink. Then what happens?
The hardware starts to go through lookup tables for MAC addresses found in all the ports. OOPS! HW Design mistake here, it does not lookup for it's OWN port at all.
Packet route is not found, so it goes to the CPU, which sends it back to the same uplink with different vlan tagged.
Throughput is only 5MiB/s because it visited the 10-15year old CPU.
Put Node A and Node B in different switches or same VLAN, and your throughput is solid 9.5+GiB/s on 10Gbps links.
This scenario is supposed to work flawlessly and be rather standard, therefore no one could suspect this kind of issue.
Thanks @PulsedMedia for this insight into that topic! I basically have no knowledge in that direction, however it fascinates me
@yoursunny this is the level of costs we are talking here:
2x100GE + 4x10GE: https://www.ebay.com/itm/304732022971
Empty Chassis: https://www.ebay.com/itm/324785452278
Congrats, you just spent 34k $ buying used hardware on Fleabay with no warranty, just to get some uplinks running and add enough FIB space add IPv6.
In case you have 1M routes or less router now.
The routing engine on that chassis is https://apps.juniper.net/hct/model/?component=RE-S-2000-4096
It is EOL, has 2Ghz Pentium CPU + 4GB RAM, 40GB HDD + 1GB CF card.
But oh yeah, you cannot rely your whole business on single used linecard + router, now double that to 68k $.
Then add interior distribution linecards.
For 10G you want a few of this or better, 4x10G for 283$ from china and used! :O
https://www.ebay.com/itm/224536129010
Or from USA: 200$ a piece: https://www.ebay.com/itm/275532027808
10x10G version 1600$: https://www.ebay.com/itm/234823248986
You'll need these to utilize MICs, MPC3 MIC carriers, 545$ from China: https://www.ebay.com/itm/225396763635
So 8x10G just cost you something like 900-1100$
Oh and you only have 8 slots to do anything. So if you want 4x100GE that's 56k $ and you now only have 6 slots to distribute it, with port density of 20x10GE for ~4k $ you have to add at least 3 of those, you want some aggregation there, and probably some 10GE peering links too etc.
Your total spend with questionable sources from Fleabay is now ~73k $, and maybe get some spares too so at least 80k $ maybe 90k $
This is the jump from 1M routes to 2M routes, maybe 4M routes if you get super lucky to find right parts.
Otherwise you have to limit IPv4 performance OR use for IPv6 default routes only.
And ironically enough, that's the cheap portion
It's the management & human time spent on managing it all which is the actual cost, but does not look like it because it's "just 30secs to 5minutes per node when setting up", or "just 20secs extra to copy & paste the address to connect to it", or "just use DNS instead of addresses" type of crap.
Even starlink does not support IPv6 other than spottily here and there.
Ultimately, you can always just use a tunnel too when you absolutely need it, and you are the 0.05% of userbase who has use for IPv6.
IPv6 zealotry is exactly that; emotionally driven religion.
IPv6 does not make sense to implement "just because". There has to be a solid use case for our customers for it. There is none. Most ruckus about IPv6 ever in our 13 year history is You @yoursunny
We will maybe implement it when there is a solid business case for it, no sooner, no later. For just a handful of setups it merely takes a month or two to implement it (all of which mostly bureaucracy, not technical limits).
In tickets it has ever been mentioned only 44 times over the past 13 years. Who knows how many tens of thousands of tickets from that even is. It has been almost a year since last mention.
1 customer who has enough of a use case to ask for it, but just used a tunnel instead. Now this 1 customer might be a business reseller, but their volume is too low to justify spending the effort to get the basic setup done and IPv6 just for them.
Now what if we add IPv6 for everyone then?
1) Well, all our software needs to go through QA of hundreds of hours effort to make sure adding IPv6 back is not causing conflicts like it did when accidentally enabled in past.
2) Make sure everything prefers IPv4 primary for known performance and routing.
3) Develop routines and mapping IPv4 to IPv6 systematically for procedures, develop quick access things for management to copy & paste IPv6 addresses.
4) build software where possible to automate this mapping.
5) Realize we need to spend 100k $ to upgrade our router to handle 4M+ routes (no sense to buy 2M route gear today, it will be deprecated in 2-3 years as IPv6 tables grow)
OR
5) Just use IPv6 default route and home upstream knows, or carefully balance to take only say 20k IPv6 routes in to just fit into 1M active routes. (~880k IPv4 + 20k*4==80k IPv6 routes == 960k -- safety margin of 4% retained)
6) Increase customer prices by 5-10% to pay off that extra overhead.
If you spend 80k $ for 400G IP Transit, over 5 years that is 1333$ extra, or +33,33$ per 10G. He.net asks 600$ per 10G, so that is direct 5% increase in transit costs. Without spares, and questionable sources.
But this extra cost has to be amortized over Dedis only, since seedboxes has less than zero benefit from IPv6.
The MiniDedi, which you have zero @yoursunny to best of my knowledge, price would be +5-10€/Mo each.
Since you are not paying for that, you'd say ofc, go for it who cares if price is increased by 20-30%.
But since we need to entice people to use the servers, we cannot do that, it would be too big of an increase.
how much value does this add?
At most 1% of traffic will go over IPv6.
Here is some very worrying data for IPv6 zealotry: https://labs.ripe.net/author/wilhelm/ipv6-10-years-out-an-analysis-in-users-tables-and-traffic/
"RIS sees a total of 155,000 IPv6 routes; a growth close to 1600%", that requires the space of 600k+ IPv4 routes. Meanwhile they see 935k IPv4 routes.
And here we get to the jist of IPv6 push: Sell more routers AND push small players out of the field due to increased cost only big players can afford
So while IPv6 does serve some benefit, we are talking best case of 16.4%, and more typical along the lines of 4% on average business. For seedboxes, that goes down significantly to perhaps 10% of that, maybe even less (so at most 0.4%)
Even here conclusion was emotional reasons to use IPv6 AND issues abound AND not really worth it: https://www.reddit.com/r/seedboxes/comments/krsfwr/torrent_clients_and_ipv6/
All that being said, In Conclusion
We will add IPv6 as soon as there is sufficiently big business case for it. Right now, the biggest benefit for IPv6 has been bumping this thread and the free publicity
That's not enough to justify extra investments in the 100k $ scale + on-going overheads in perpetuity.
We try to avoid making any decisions for emotional reasons, there has to be logic and data to support the decision. Data does not support adding IPv6 support, even when it is "free", since management/SW development is the actual biggest cost which is very difficult to quantify. My best guess is it would add 1-4% management overhead to any sysadmin duties, as an total average aggregate. People cost more than hardware, and that overhead is forever, we'd be paying for it after the next 13 years too.
Personally, i hope there will never be a business case for IPv6 due to various design issue reasons, IPv6 is overtly complicated for what could've essentially been a simple solution with far less effort, fraction of the cost, and improve performance rather than degrade it. Lack of addresses issue could've been solved a decade ago already, permanently, but that's not the reason for the IPv6 push.
Trivia, i was one of the early adopters of IPv6. Whenever first IRCnet servers supported IPv6 and first tunnels were available, i was already using IPv6
I think that was just after mid 90s
Funny thing; Just PSUs for the MX480 cost ~650$+ used, and they are not nothing fancy, infact efficiency of them was rather low, i think it was like 88% or so. You typically use 48V input with these regardless.
and thanks again for the publicity @yoursunny
Now guys, enough of a story time, go buy some minidedis from us ASAP!
Nice offers, good to see DDR4 and NVMe :-)
Then please get one
Ah man, I've just recently acquired a dedi but these look banging. Good luck with sales
Nice offer, I think I will buy one or two
I have a question about the crypto payment though, is it through some 3rd party like BitPay(if so which?) or self hosted solution(btcpay\etc), either is fine I just wouldn't mind knowing
Also can you have another OS instead of Debian?
I clearly know what FIB is because I design routers.
I design content routers not IP routers, but I understand how they work and I did experiments with them.
As long as the router is keeping up with line speed, nanoseconds do not matter for bulk transfer packets.
You operate download network, not voice network.
The cited paper tells you how to design a router that drastically reduces cost without increasing latency for majority of traffic.
The main idea is that, more than 97% of traffic would match the hot routes in SRAM.
The remaining traffic needs to access DRAM, but information in SRAM can inform where to read in DRAM so that it requires fewer DRAM accesses than what it would have if all of FIB is in DRAM.
Get a few grad students in your local university.
They'll build a nice router for you in a few semesters.
Restore my 450GB from €81/year to €18/year, and I'll have non-zero.
Awesome!
CoinPayments
Sorry no other distro has been automated yet. Next up on the line is latest Ubuntu.
Too bad, I personally like Ubuntu or Arch Linux, hopefully you can add those at some point. For now I've ordered and look forward to trying the server
Let's break this down:
So you design low level ASICs for hardware deep packet inspection equipment? Yet you do not know how they physically operate?
So many things wrong here, not sure where to begin.
Voice networks are actually very very fault tolerant. VOIP is fine with 150ms, and people tolerate upto 300ms. They are also fine with packet loss upto 5%, jitter in the range of 30ms. I actually know a VOIP service provider, and have worked myself on PBXs in the 00s.
So you are comparing a protocol, running on top of IP (everything is VOIP these days, no more analog! for probably decades), which is tolerant to 300ms latency, with high jitter and high packet loss of 5% -- to something which can tolerate barely any jitter, is highly dependant on latency, and can tolerate 0.00% of packet loss to maintain throughput.
If latency for IP networking does not matter, then explain these:
connections contribute approximately 10ns and 50ns latency, respectively"
Shall i continue, or do you still believe that 500ns of extra latency does not matter?
100% has to match, or it does not get routed, period.
This SRAM needs to be in the ASIC itself.
FIB AKA Route lookup table cannot be loaded to DRAM modules.
THERE we go, the ultimate reason for this zealotry.
I am sorry you are so broke you cannot afford services you clearly need and want, but it's not our job to sponsor your usage, nor succumb to what has now essentially turned into attempted extortion. So you are saying if we do not give you under cost//free services beyond what we have promised, you will continue with your rampage on trying to find anything bad possible to say about us?
You sir, just gained permanent ban on our services. If i knew what your account was, i would go in and ensure your billing profile is going to be closed the moment you hit renewal date.
You are the poster child of LE market difficulties, which we talked A LOT about in the price increase email and notification, the reason why we had to increase price on the bottom of the barrel priced stuff so heavily. Hope you are happy with yourself, you are truly exceptional individual. Well... we already knew you are!
Thanks for reminding us never to post any 18€/Year deals ever again. We are sure to never make that mistake again.
Thanks
Ubuntu will be next in-line, just need a bit more volume first.
It's unfortunate that low end markets are often filled with people who expect a lot for very little(not talking about the guy you're speaking to or anyone in particular, just in general), personally I'm just happy to get a good deal.
Regarding the arguing about hexadecimal lines you're talking about, I would enjoy having ipv6, but it won't really severely affect me to not have it either
maybe in the future you can add it when you're rich or something lol.
Hope you all can have a good day/night.
It seems to me if these routers cost $150k that there's some good money to be made from producing cheaper routers that can cope with bigger routing tables.
I massively disagree with this. Let's consider what would happen if we had no cache at all, and everything hit DRAM.
Modern DRAM takes something like 20-30 cycles maximum to return data, and assuming a VERY slow memory bus of 2000MHz, something that's not in a cache will take 30ns to obtain. In practice, most real world DRAM chips are under 10ns.
Sure, this means that there is some additional latency for the lookup, but let's consider how big this latency actually is...
At 10Gbps, that's 0.1ns per bit, so the absolute worst case we hypothesised earlier would introduce a delay of 300 bits, or 37.5 bytes. To put it into perspective, that's about double the length of the ethernet frame header and about the same as the IPv6 header itself.
Now, I kind of dispute those numbers, but anyway, let's go with it. 200ns is still only a delay of 2000 bits or 250 bytes at 10Gbps. Even a decade ago, the SRAM available on cheap FPGAs was an order of magnitude larger than this.
[NOTE: I was assuming 10Gbps, so obviously for 100Gbps, the buffer size will be correspondingly bigger. A few KB per port buffer would still be considered tiny IMHO].
And it's not like this is really a new problem, because the routers will be doing most of this anyway, as they already had to wait until at least 58 bytes into the packet before they could do any routing anyway.
So really, without a cache at all, the latency on routing would be 5 times worse than the best possible routing assuming everything was in a single-cycle cache.
Now, if you introduce cache as a cache, not as something that needs to hold the entire routing table, you experience this extra latency only on the packets destined for a new un-cached route.
I mean, you'd have better insight than me into how many different routes is needed for the working set in a data centre with many different customers, but honestly I can't imagine it's all that high. Even a million /48 routes would only need 8MB cache and that seems massively overkill for the working set, because you probably only have a maximum of tens of millions of packets per second in the absolute worst case.
So, even if a data centre's infrastructure was such that outgoing traffic had 5 such switches on outgoing traffic, assuming your pessimistic timings (which I think are an order of magnitude too slow), that's only 1ms extra. On the first packet. That route can then be added to the cache.
The whole problem seems to be that the hardware you've bought requires the entirety of the routing table to be in super-expensive cache memory. If this is true, and this is the state-of-the-art in networking infrastructure, then it seems a lot of money could be made by rethinking the problem.
Stats on my €5.5/year 50GB LowEndSeedbox from iHostART, over past 25 days:
The setup is:
To calculate IPv4 vs IPv6 traffic consumed by mainly qBittorrent:
IPv6 accounts for 6.5% on my LowEndSeedbox.
This server is set to prefer IPv4, and IPv6 usage is entirely chosen by torrent trackers and leechers.
Yes there is always demand, see Force10 for example, but these often get merged to the big players.
See earlier posts, you are off by order of magnitude there
But assuming 10ns -- which would match L3 cache of the utmost highest performance leading edge tech today, not decade ago, but also it's not just 1 fetch, it's a lookup table. It's just not "get this route", but the route has to be found first to get the answer to where to send the packet.
Regardless, does not matter, no such hardware exists which can route line-speed and have infinite routing tables in essence by storing it in DRAM.
You can't just add DRAM to pre-existing routing hardware.
You seem to be mixing packet buffer with routing lookup tables?
Yes that's how cache works, but the HW routers store the entire table in the hardware ASIC to ensure 100% of packets get routed fast, so it's all in the cache all the time. Infact, it is the cache.
You cannot change hardware spec after manufacturing.
Who knows, maybe the next generation will do this, but those on mere mortal price range? Not really.
Most likely will stick to everything on hardware, since process nodes have advanced and you can have more memory on the die -- or even stack.
Slightly more: https://networkengineering.stackexchange.com/questions/76562/how-much-ram-you-actually-need-to-keep-whole-global-bgp-routing-table
Again moot, since you cannot add memory to pre-existing hardware.
The hardware supports X routes, and that's it. You cannot just go and add a DDR4 module and off you go with infinity more routes.
For latency, it all happens inside the ASIC. At least on Brocade, and i think on Juniper too, each linecard hosts it's own copy. Each port needs their own packet buffer which is very large.
The actual switching circuitry is actually very small and cheap, it's all about the data physically.
But again a moot point, because such router you can just add memory to infinity does not exist, well atleast i don't know. Except in software, which does not perform well enough.
Brocade MLX series for example had variety of sizes, and caps out at 2M. Those parts which have 2M routes cost insanely much, and every single line card etc. has to be matching size i believe, at the very least management + linecards which need to route have to be at that spec.
On Juniper, if you mix older DPC cards with MPC cards, your available routes plummet down to what the DPC cards can do. If i recall right Juniper offers upto 4M routes.
Remember, IPv6 address requires 4x the routing table space than a IPv4 address does.
Again, all of this is moot because we cannot magically wish into existence a new router with zero cost. We live in the real world, even if you or yoursunny could imagine a router with infinite routing tables and infinite performance and huge packet buffers, and 100% uptime and decades of trust behind it, that does not magically appear to everyone's DC everywhere in the world by the magic of thinking.
1ms extra for every hop on the route, say 20 hops == 20ms extra, quite a drop in the throughput.
Also switching does not add much latency, it's in the order of hundreds of ns what typical switches add, per hop. they are optimized for that.
So is routing table too, and because of latency requirements it has to be in the asic.
ALWAYS rethinking a problem can make a lot of money
Reason for them to do that is because of latency. If top of the line DDR on desktop PCs without ECC and other fault tolerance metrics takes hundreds of ns to fetch single piece of data from DRAM ... and in the timespan you get the packet you need to probably do hundreds, not sure how the sorting is optimized.
If data fetch takes 175ns like on ryzen desktop from DRAM, the algo does probably something like this:
1) Fetch first octet list
2) Find pointer for our first octet to 2nd
3) Fetch 2nd octet list based on the A class
4) Find pointer for our second octet to third
5) Fetch 3rd octet list based on the B Class
6) Find the right subnet, Fetch the target MAC address, if no MAC then drop the packet, if MAC found forward the packet to the right port with right VLAN tags (VLAN tags another few memory fetches probably)
There's a paper about the lookups, but i don't have time to read it: http://yuba.stanford.edu/~nickm/papers/pankaj-thesis.pdf
It's pointless because what exists, what is available for us, at what price is what rules. We cannot go and design our own router.
We tried to utilize more routes than FIB holds, L2 started to drop from ARP table randomly and in hard to debug ways, and only way to solve that was to reboot the whole damn thing. In that mode, the FIB would've been more of an cache, the idea is that it then goes to the CPU asking for route. How L3 routing over commit messed with L2 ARP tables? I have no idea.
CAM partitioning modes did not help neither, and each change requires a reboot.
We are forced to use much less routes than documentation and data shows, because we did not have the foresight that each linecard needs to be at that minimum spec, or you are restricted to the lowest silently.
This stuff really is not documented well. So few have to play around with actual HW BGP in production environments with a lot of users and nodes, with a lot of throughput.
We could get our current edge router to support more routes, but it would involve spending probably 20-25k € on a EOL router which can only last for us so long, maximum linecard is 2x100G and we can only put 8 of them on the system, but also very expensive optics. The linecard is "cheap" at only 12k € refurb typically, but lasers are like 3k € a pop. So 18k € to get 2x100G ... 20x10G linecards are 3k € a pop and so forth.
We are already upgrading to a MX480 soon regardless because got an offer for one we cannot refuse, and had to postpone some upgrades since our current one cannot ingest more routes. Otherwise performance wise it's still perfect and really really really fast. Just not enough routes, and we would need to replace management engine PLUS all linecards to get more routes.
We might keep it around as an aggregation layer tho, if i recall right it had 576MiB of packet buffer per 10G port on the 8x10G cards, and something like 250MiB per port on the 24x10G cards. But does consume a few kilowatts, so i'll have to do the maths if it's worthwhile to keep as a switch OR should we buy a few generations newer deep packet buffer switches (probably 5-10k € in itself) as the final aggregation layer.
Routers need it all, that's why they cost: Deep buffers, ultra low latency, high route counts, ultra high reliability.
The opposite of true is on the consumer gear you typically see. Say your home switch might only have 8KiB of packet buffer or something silly like that, they don't advertise it at all and do not tell you, probably because they have the bare minimum to be able to even come close to linerates (and D-Link not even that). They also need "routing", but it's only MAC address tables, so those are small, just a few KiB is typically enough for that too. Maybe even less.
Then you move to SMB etc. switches, none of them have any packet buffers neither, can do limited L3 routing, but often happens in the CPU and throughput is abysmal.
Next tier up you get the enterprise and datacenter switches, these might cost you 2-5k € each brand new, but yet you are limited to like 100k L3 routes max and like 4-16MiB of total packet buffer.
and then stuff starts to get real expensive ...
Like i said earlier, the extra cost on router is not even the bad part, it's a one off cost ... Imagine having to pay an extra 8% tax on everything? 4% more cost == 8% more needs to be earned to get to the same final balance, if you have high margins. LE stuff does not, so that 4% cost might equal to 40% more revenue required for operators like cociu was (10% profit margin, which is very typical on retail actually!)
Something that benefits 0,05%-0,40% of our customers, but increases cost by 8% is not a good investment, period. No matter the level of emotional outrage and zealotry there is.
Only going to reply to a few points here.
No, I'm really not. Look at the specifications for DRAM and read what CAS latency is and then look at what CAS latency is for typical DRAM modules:
But here are some pointers:
https://en.wikipedia.org/wiki/CAS_latency
Here's a typical off-the-shelf DRAM module:
https://www.ebuyer.com/1265314-kingston-fury-beast-16gb-2-x-8gb-3200mhz-ddr4-ram-black-kf432c16bbk2-16
Speed 3200 MHz (PC4-25600)
Latency Timings CL16 (16-20-20)
This chip would take a maximum of around 60 clocks (19ns) to return data, assuming both RAS and CAS were different from the previous access.
Yes, but all this means is that the current packet just needs to be delayed (buffered) for as many clock cycles as is needed to get the routing information from the DRAM. Your existing router is already doing this even if it's not operating in store-and-forward mode, because it's not possible to route anything until you've got to the end of the header.
This was my point. If it doesn't exist, one could make a killing in this market.
If you feel the need to point this out, I think you've misunderstood the point I was making.
No.
If you're running in store-and-forward mode, your latency is already massive, so a few extra clock cycles to look up a routing table from DRAM isn't even worth considering.
If you're not running in store-and-forward, then every packet is still buffered by at least the size of the header, because it's simply not possible to route anything until the header has been received as until you have that, you don't know where the packet is going.
I'm talking about just increasing the size of that buffering to allow the extra time to fetch the routing information from DRAM if it's not already in the cache. This would increase the size of the buffer from around 50 bytes (assuming routes are available basically instantly from ultra-fast memory) to around 250 bytes for 10Gbps speeds to around 2K bytes for 100Gbps speeds.
And my point is exactly that it doesn't need to be available instantly for 100% of all packets.
I know. Again, you're missing the point of what I was saying.
Again, I'm NOT talking about the entire table. I'm talking about the working set of routes that a node actually sees.
Again, I'm not talking about just adding DRAM to an existing system. I'm talking about how to design a system that can handle the size of routing tables that's actually needed.
It doesn't really. The minimum internet routable size is /48, which is double an IPv4 /24. Sure, you might have a few internal routes of smaller blocks, but if you follow good practice of /64 to each host, the routing table is still not massive, and of course this granularity is entirely under your control.
Again, you failed to understand the point I was making. It's not a massive technical challenge to design such a system. It also doesn't need to be "zero cost" if the existing systems you have cost $150k each and aren't even able to support today's routing table sizes.
Wow, way to extrapolate. I was already accounting for 5 hops and assuming your worst case (200ns) when saying this. Besides, most routes on the internet aren't anywhere close to 20 hops and those that are are usually already 100ms+.
This is exactly the point I'm making. If these switches are already adding "hundreds of ns" per hop, adding 30ns or so for the worst case "not in cache, need a DRAM lookup" isn't going to make all that much difference in the grand scheme of things.
Sure, if you want to design it to be as slow as possible. Also, you seem to be stuck in a IPv4 old-mindset way of approaching this.
A simple approach would be look up top /32 to a 32-bit offset (fits comfortably in a 16GB module) and takes a single read cycle.
That 32-bit offset might be enough to say the route is invalid, or everything in that /32 routes only one upstream. For the rest, you can multiply that by 16 bits and look that up in a second level for any routes that need further routing information.
Of course, it's even better than that, as internet addresses are currently only 2000::/3, so
already for that first lookup you can lookup a /35 with only 16GB.
Again, the entire point of my argument is that it's entirely possible to design a router that doesn't have these limitations. Just because you're using 10-year old routes that have crappy limitations from 10 years ago, doesn't mean that IPv6 should be discounted. It just means that people have to design routers that are able to cope with the needs of today's routing tables as well as the anticipated sizes 10 or more years in the future.
The packet buffer size is only an issue if you're doing store-and-forward in the first place. And if you are, then the latency involved in looking up routing from DRAM is irrelevant as the routing can be done as soon as the header is received, in parallel with the rest of the packet being read into the buffer.
The only time a route lookup from DRAM would need to add latency is if you're directly forwarding a packet to the destination while it's still being received, and as I explained earlier you need to add at least as much latency as is needed to decode the header.
The fact that you keep talking about the packet buffer size being an issue suggests to me that you are in fact mostly concerned about store-and-forward, which would also explain why you're already seeing hundreds of ns latency per switch. In this case, looking up routing from DRAM would not make this noticeably slower.
PS back on topic, the servers look nice!
Indeed response time is that: https://www.memorybenchmark.net/latency_ddr4.html
Seems odd others reported 175ns-200ns, but then noticed this article: https://www.cgdirector.com/ram-memory-latency/
that latency is the time it takes for the memory to respond. There are other overheads too, such as sending the request.
Further we are looking at current leading edge speeds, even current leading edge servers wouldn't use the top of the consumer memory you are comparing to.
or there is a reason it does not exist.
Neither of us knows this stuff deeply enough to know what the reasons were, this has for certainty have been attempted.
Could also be possible that DRAM speeds are just now reaching point where it's feasible.
So you are talking about of a potential in the future architecture?
Btw, you'd need more than that, full packet + next packet too probably, since it takes time to get the response.
For enterprise, high end, the target audience of this gear; Yes, Yes it does.
Amazon will never allow that occasionally a packet might take extra 0.5ms to be routed, by design.
In the hardware it does take 4x, even if data size purely would not be that much different. For what reason? I do not know, but this is the case for Juniper at least.
Oh and one more thing; You need to partition the space, so you regardless loose significant amount of IPv4 route capability. This again depends off the make etc.
The big MX480 are able to do it, no problem. My original point was that you have to upgrade to bigger routing hardware, which costs big money. Those which cannot route full table are much much less expensive.
You don't design for best case, but worst case. 20 hops is not worst case.
regardless, i am not anywhere near convinced that only 200ns extra would be spent if routing table is in the DRAM.
If it were that fast, it would already be used on latest gen stuff.
30ns is wishful thinking.
To determine that, check up on the algos, then conduct an experiment to measure the actual latency of something like DDR4 3000 ECC memory, and do a proof of concept.
I'll be very very surprised if it's even in the 2 orders of magnitude ballpark. But pleasantly, that would tell us that within a decade router max routes tables could become much larger. Then again, latest models already have enough space on the ASIC for the needs of next 2 decades or so...
Crappy 10 year old routers
Ooookay, you do realize that MX480 is a very solid router? It may have been released close to a decade ago, but has had continuous updates. That's how chassis routers are.
These architectures just do not change every other year. These are not consumer stuff.
Datacenters, ISPs etc. keep buying new MX480s every week to grow or upgrade their networks.
This is MX480: https://www.juniper.net/us/en/products/routers/mx-series/mx480-universal-routing-platform.html
But yeah got to be shit because it was not designed to your liking, and the lineup is long lived, riiiight?
This thread got a little bit derailed
Looks like Arista has been doing some nice work on compressing the routes and make faster more efficient lookups: https://www.arista.com/assets/data/pdf/Whitepapers/FlexRoute-WP.pdf
The old paper showed that they could almost double the amount of routes supported, but looks like behind licensing regardless, a lot higher cost on the models with heavier optimization.
regardless if modern HW could do it in DRAM, the industry incentives are to charge more for more routes.
I might get one if Ubuntu becomes available and I can convince myself to ignore the no v6 support.
Apparently it's possible to route 300Mpps on cheap off the shelf hardware.
https://media.ccc.de/v/vnog-4170-bgpospf-with-100mpps-on-amd64
Mentally strong people build their own routers out of FPGA and don't give in to the industry.
Plenty of networks route IPv6 with full tables at 100G for far less than @PulsedMedia is claiming (or at least heavily implying) is needed to set up a routing stack capable of it.
Ultimately the problem with not adopting IPv6 now is that when no new v4 allocations can be found anywhere and all lessors are out of ranges you'll have shot yourself in the foot because if your routing stack is so expensive to upgrade then you'll be left footing that cost down the line at an unexpected point in time rather than having planned for it. So from a business standpoint not adopting IPv6 at all, and not planning to ever do so, as a content network is a risky move.
Adding IPv6 costs = 0$ in hardware.
It is just 1 static route at it's easiest, and making BGP announcements. Sucks, but at least is there.
Having full tables of both IPv4 + IPv6 starts costing.
Right now you have to get min. ~2M route hardware, even that is only going to last you something like 2-4 years only. That stuff is expensive
Management of it is the expensive bit. Everything surrounding it, every little bit of management.
But i guess, this thread concluded that you can design better routers & switches in your basement in between lectures, partying with no money and no experience than industry giants like Juniper, Broadcom, Cisco etc. knows how to make.
They are stupid for not realizing they can just use old x86 CPUs, some DDR, just add ports and off you go it seems. Latency, packet loss and packet throughput all are irrelevant. It's the same if you can do 300Mpps or 15Bpps, there's really no difference. Why spend 10kw in electricity alone to route your measly 1 500Gbps if mama's old computer could do it?
You guys should start your own business since you guys know better than tyhe industry encumbents and can do 100x better routing hardware, and 100x anything than established players.
We are dumbstruck how we have been this stupid to have actual real routing hardware when we could have just taken mom's old computer out of the basement, dust it off and have it just as good!
Or better yet, hire a bunch of random college kids and have them design and build one! I am sure with the same money they will build a better one with decades of reliabity testing and experience, and not just drink it all!
Come think about it, we've been going all wrong with these servers! Why bother spending all that money for a server when we could just give everyone a abacus to compute with and stone tablet to store their data into?!?!???? :O Wow! Such savings
No IPv6 provided, period.
Absolute Need IPv6? Use a tunnel broker.
Show me. I took public refurb prices for redundant network setup. I am certain you can save a little bit, but not like 1/10th.
So please show us, we'd be happy to save cash in a future setup if you can do it cheaper without sacrificing performance and reliability.
Seriously, if we end up using that we'll even give you a nice healthy bonus.
Minimum specs are 5Bpps+, ~250MiB+ packet buffer per 10G port/min 2x per 100G, 3M+ IPv4/IPv6 routes. I would prefer to target 4M route FIB sizing.
These big boys have to be stupid to be spending thousands of euros per 100G port in refurb market (or close to 10k per port new). Amazon, Meta, Google etc. must be full of idiots spending that much.