Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Dynadot hit by 150 Gbps DDOS attack
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Dynadot hit by 150 Gbps DDOS attack

risharderisharde Patron Provider, Veteran

I'm quite surprised such a theoretically small attack caused the outage. Of course, maybe this was a layer 7 attack (???). I kind of thought that they were big enough to have put much higher levels of protection in place.

From the CEO via email:

Dear ,

Yesterday the Dynadot website was hit with a 150Gbps DDoS attack. While we did have DDoS protection in place, it was not capable of blocking an attack of this size. Our team worked non-stop to mitigate the attack, and we have installed additional defenses against such attacks in the future.

I know many customers are concerned about their account security. I am happy to report that customer data was not impacted by this attack. Account security has been, and always will be, a top priority of ours. Additionally we will work on improving communication during service disruptions to make sure we keep you in the loop.

I am very sorry this happened, and I know it was extremely inconvenient. Thank you so much for your patience and understanding. We will continue to improve our systems and processes.

Best Regards,
Todd Han
President
Dynadot.com

«1

Comments

  • In no way 150gbps is small, 150gbps is in the top 2%, maybe even top 1% of all attacks. Specialised services that have thousands if not millions invested could filter this (ie. Path.net, cloudflare, akamai, fastly etc) but a party like dynadot, i’m not so sure as proven here. Attacks of this size are EXPENSIVE to be capable to filter without a 3rd party service.

    Thanked by 1Logano
  • PulsedMediaPulsedMedia Member, Patron Provider

    150Gbps is not small in anyway. To counter that it would mean they would need to have 100s of Gbps of just idle capacity, paid for just-in-case this kind of stuff happens. That is a VERY expensive.

    Thanked by 2Void Logano
  • @PulsedMedia said:
    150Gbps is not small in anyway. To counter that it would mean they would need to have 100s of Gbps of just idle capacity, paid for just-in-case this kind of stuff happens. That is a VERY expensive.

    Well maybe onddns has that kind of setup so its small on their view..

  • DPDP Administrator, The Domain Guy
  • skorupionskorupion Member, Host Rep

    150 Gbps is a lot. Just to get that bandwidth (not looking at what they normally use) on a server not even counting the equipment, ip and Colo costs with he.net for 9 cents per mbps, it would cost 13500 $ per month for 150gbps at minimum

  • @skorupion said:
    150 Gbps is a lot. Just to get that bandwidth (not looking at what they normally use) on a server not even counting the equipment, ip and Colo costs with he.net for 9 cents per mbps, it would cost 13500 $ per month for 150gbps at minimum

    dns amplification :smile:

  • skorupionskorupion Member, Host Rep

    @Cappuccino said:

    @skorupion said:
    150 Gbps is a lot. Just to get that bandwidth (not looking at what they normally use) on a server not even counting the equipment, ip and Colo costs with he.net for 9 cents per mbps, it would cost 13500 $ per month for 150gbps at minimum

    dns amplification :smile:

    and the servers got overloaded also i guess, so the price of defending themselves would be rocket high

  • ehabehab Member
    edited January 2022

    @DP said:

    hitting @yoursunny website i guess.

    Thanked by 1yoursunny
  • TamerciagaTamerciaga Member, Host Rep

    @Cappuccino said:

    @skorupion said:
    150 Gbps is a lot. Just to get that bandwidth (not looking at what they normally use) on a server not even counting the equipment, ip and Colo costs with he.net for 9 cents per mbps, it would cost 13500 $ per month for 150gbps at minimum

    dns amplification :smile:

    For sending (spoofing) yes, but in order to mitigate such large attacks you need to have enough bandwidth capacity in your network, we call this tanking capacity.

    Would like to see what type of attack they encountered, other people can learn from it.

  • ralfralf Member
    edited January 2022

    @DP said:

    Is it just me that thinks this looks like a screenshot from the DEFCON game?

    Thanked by 1Logano
  • ascicodeascicode Member
    edited January 2022

    Did everyone got that email?

  • DPDP Administrator, The Domain Guy

    @ascicode said:
    Did everyone got that email?

    Don’t know about everyone, but I did.

  • MrRadicMrRadic Patron Provider, Veteran

    @PulsedMedia said:
    150Gbps is not small in anyway. To counter that it would mean they would need to have 100s of Gbps of just idle capacity, paid for just-in-case this kind of stuff happens. That is a VERY expensive.

    This....no one seems to understand the real cost of mitigation.

  • @ascicode said:
    Did everyone got that email?

    I didn't :/

  • PulsedMediaPulsedMedia Member, Patron Provider

    @MrRadic said:

    @PulsedMedia said:
    150Gbps is not small in anyway. To counter that it would mean they would need to have 100s of Gbps of just idle capacity, paid for just-in-case this kind of stuff happens. That is a VERY expensive.

    This....no one seems to understand the real cost of mitigation.

    Real cost of bandwidth to begin with. We've seriously had people close to decade ago throw a hisfit tantrum forum rampage for why they did not get dedicated 1Gbps with dedicated server at 10€ a month, 100% guaranteed all the way to their home in India ... from europe. Sure thing, if you have a few trillion $ to throw at it just to study the scope lol

  • @PulsedMedia said:

    @MrRadic said:

    @PulsedMedia said:
    150Gbps is not small in anyway. To counter that it would mean they would need to have 100s of Gbps of just idle capacity, paid for just-in-case this kind of stuff happens. That is a VERY expensive.

    This....no one seems to understand the real cost of mitigation.

    Real cost of bandwidth to begin with. We've seriously had people close to decade ago throw a hisfit tantrum forum rampage for why they did not get dedicated 1Gbps with dedicated server at 10€ a month, 100% guaranteed all the way to their home in India ... from europe. Sure thing, if you have a few trillion $ to throw at it just to study the scope lol

    One thing I've learned from working in IT, the less someone pays, the more demanding they are and the more they expect.

  • @PulsedMedia said: Real cost of bandwidth to begin with. We've seriously had people close to decade ago throw a hisfit tantrum forum rampage for why they did not get dedicated 1Gbps with dedicated server at 10€ a month, 100% guaranteed all the way to their home in India ... from europe. Sure thing, if you have a few trillion $ to throw at it just to study the scope lol

    Yeah.

    "I'm going to go with X provider because they're offering full gigabit".

    "Ok yeah sure use them. I'm sure it's a full dedicated gigabit."

  • @ralf said:

    @DP said:

    Is it just me that thinks this looks like a screenshot from the DEFCON game?

    Nope, not just you.

  • NyrNyr Community Contributor, Veteran
    edited January 2022

    @ascicode said: Did everyone got that email?

    No.

  • PulsedMediaPulsedMedia Member, Patron Provider

    @Trav said:

    @PulsedMedia said:

    @MrRadic said:

    @PulsedMedia said:
    150Gbps is not small in anyway. To counter that it would mean they would need to have 100s of Gbps of just idle capacity, paid for just-in-case this kind of stuff happens. That is a VERY expensive.

    This....no one seems to understand the real cost of mitigation.

    Real cost of bandwidth to begin with. We've seriously had people close to decade ago throw a hisfit tantrum forum rampage for why they did not get dedicated 1Gbps with dedicated server at 10€ a month, 100% guaranteed all the way to their home in India ... from europe. Sure thing, if you have a few trillion $ to throw at it just to study the scope lol

    One thing I've learned from working in IT, the less someone pays, the more demanding they are and the more they expect.

    exactly. and quality of service is inversely proportional to the price paid no matter what are the technical merits. You can offer complete pile of steaming pile of shit but if price is high enough it's perfect and flawless.

    Sometimes it makes me think why do we bother with targeting good value per euro paid .... sigh

    But then we remember the 1000s of users who are happy happy joy joy

    Thanked by 1Trav
  • risharderisharde Patron Provider, Veteran
    edited January 2022

    I'm definitely no expert with mitigation but the way this thread has been headed, once again I feel misunderstood by some of the community (or simply just a little too personally involved because I posted). Dynadot being down and me posting this wasn't a complaint - I lost no sleep over this, I'm not angry, sad, upset with Dynadot. I might be the 10% of users with a different perspective or even understanding when it comes to the claims made these days so let my attempt (I failed miserably in my last thread where I had to defend a position on hiring) - but some providers claim to be able to tank higher levels than 150 GBIT worth so my CURIOUSITY was whether they did not have backup services with some of these providers OR as I said, a layer 7 attack, if not properly filtered would cause the servers themselves to fall to their knees versus it being a bandwidth issue. Also, maybe there's another scenario I'm missing that could have caused them to go down.

    Again folks, I understand from the provider responses the magnitude of cost as it be no walk in the park figures you guys are quoting (and I feel for providers because I myself have not gone into hosting etc because I don't think I can afford what others are able to provide). I assume (hopefully rightly) that it's a cost usually worth it (reluctantly even) to get customers AND it's a cost that is usually shared by every client or it just wouldn't be possible? (this is rhetoric assuming you still remain profitable).

    EDITED: Still thank you for all of you who posted, I actually appreciate your point of views on this!

  • MaouniqueMaounique Host Rep, Veteran
    edited January 2022

    We had a few over 200 (saturated our capacity).
    This can happen and rolling ones which change IP attacked at few seconds are not easy to mitigate, but bw is not that expensive at 95 percentile, sustaining such an attack for multiple hours in an adaptive way to evade active (someone actually watching everything and moving pipes, BGP, etc) mitigation is also not easy.
    Provides which say they can tank this and that are actually praying that the upstream will, at least the ones here, or that cloudflare will, for their site, etc).
    TBH, I was more pessimistic about attacks a few years back, but bw didnt continue to drop in price at the same speed and mitigation techniques evolved quite a lot.
    Of course, as a business oriented provider, we stay away of DDoS magnets (game servers, hate speech such as religion, racism, homophobic etc as well as other DDoS magnets) are not allowed on our regular plans (can be if legal in other places than in Milano or in Milano but only with a protected IP as an ADD-on which means separate pipe, etc).
    Also, to bust a myth, a layer 7 attack is not depending on the size of the payload, but on the ability to get to the target specially crafted commands which will saturate its capacity to function, not clogging the pipe so it can't be reached at all.
    So, 100kbps attack can (if well designed) put down most webservers or some API, etc. You don't need 150 G for that. Skids need some serious skills to design a targeted L7 attack and need to keep changing the commands in order to avoid active mitigation as well as having thousands of zombies available allover the world or at least in the area the target is serving, because writing a script to block such attacks one IP at a time can be done in 1 minute, therefore the brute force one is much more used by them.

    Thanked by 1risharde
  • risharderisharde Patron Provider, Veteran

    @skorupion said:
    150 Gbps is a lot. Just to get that bandwidth (not looking at what they normally use) on a server not even counting the equipment, ip and Colo costs with he.net for 9 cents per mbps, it would cost 13500 $ per month for 150gbps at minimum

    I might be naive about this but would this be the best way to approach it or was the point mainly that bandwidth is expensive. I'm not poking holes at your explanation, it's reasonable but the scenario I would think of is offloading to a provider that offers the tanking that has that kind of capacity. Thanks for the comment again!

    @Maounique said:
    We had a few over 200 (saturated our capacity).
    This can happen and rolling ones which change IP attacked at few seconds are not easy to mitigate, but bw is not that expensive at 95 percentile, sustaining such an attack for multiple hours in an adaptive way to evade active (someone actually watching everything and moving pipes, BGP, etc) mitigation is also not easy.
    Provides which say they can tank this and that are actually praying that the upstream will, at least the ones here, or that cloudflare will, for their site, etc).
    TBH, I was more pessimistic about attacks a few years back, but bw didnt continue to drop in price at the same speed and mitigation techniques evolved quite a lot.
    Of course, as a business oriented provider, we stay away of DDoS magnets (game servers, hate speech such as religion, racism, homophobic etc as well as other DDoS magnets) are not allowed on our regular plans (can be if legal in other places than in Milano or in Milano but only with a protected IP as an ADD-on which means separate pipe, etc).
    Also, to bust a myth, a layer 7 attack is not depending on the size of the payload, but on the ability to get to the target specially crafted commands which will saturate its capacity to function, not clogging the pipe so it can't be reached at all.
    So, 100kbps attack can (if well designed) put down most webservers or some API, etc. You don't need 150 G for that. Skids need some serious skills to design a targeted L7 attack and need to keep changing the commands in order to avoid active mitigation as well as having thousands of zombies available allover the world or at least in the area the target is serving, because writing a script to block such attacks one IP at a time can be done in 1 minute, therefore the brute force one is much more used by them.

    Thanks for this insight - this is indeed an insightful post not that others are not, I'm trying to keep up with the conversation after being away for a few hours.> @FoxelVox said:

    In no way 150gbps is small, 150gbps is in the top 2%, maybe even top 1% of all attacks. Specialised services that have thousands if not millions invested could filter this (ie. Path.net, cloudflare, akamai, fastly etc) but a party like dynadot, i’m not so sure as proven here. Attacks of this size are EXPENSIVE to be capable to filter without a 3rd party service.

    Fair statement but why then would such a domain provider not utilize such providers. I mean I'd pretty much put domain providers on a high list of being ddos just because some clown will hold them responsible for a domain even when it has nothing to do with them.

    @paijrut said:

    @PulsedMedia said:
    150Gbps is not small in anyway. To counter that it would mean they would need to have 100s of Gbps of just idle capacity, paid for just-in-case this kind of stuff happens. That is a VERY expensive.

    Well maybe onddns has that kind of setup so its small on their view..

    Never tested (I really don't know how and it would probably be illegal to perform such a test even it's for self purposes) but I would hope something would stay up even if the main site goes down and technically I think it would since the providers we use in multiple locations claims to have ddos protection - albeit, not all with the same tank levels of course. One would have to be a little creative I would think to take down all of OnDDNS. That's not a challenge, that's just how I hope I did it (correctly hopefully)

    @PulsedMedia said:
    150Gbps is not small in anyway. To counter that it would mean they would need to have 100s of Gbps of just idle capacity, paid for just-in-case this kind of stuff happens. That is a VERY expensive.

    True, but that isn't unheard of nowadays right - there are providers (not many of course) but the names are there

    @DP said:

    The best part of this image DP is some of the packets look like it's coming from nowhere as well lol nice gif!!

  • PulsedMediaPulsedMedia Member, Patron Provider

    True, but that isn't unheard of nowadays right - there are providers (not many of course) but the names are there

    Uhm, yeah that list is very short. very very short.

  • skorupionskorupion Member, Host Rep

    @risharde said: I might be naive about this but would this be the best way to approach it or was the point mainly that bandwidth is expensive. I'm not poking holes at your explanation, it's reasonable but the scenario I would think of is offloading to a provider that offers the tanking that has that kind of capacity. Thanks for the comment again!

    except for latency, you see people get horny over latency in the web building world, and people will complain about additional 7 ms latency.

    Thanked by 1risharde
  • HostSlickHostSlick Member, Patron Provider

    @PulsedMedia said:

    @MrRadic said:

    @PulsedMedia said:
    150Gbps is not small in anyway. To counter that it would mean they would need to have 100s of Gbps of just idle capacity, paid for just-in-case this kind of stuff happens. That is a VERY expensive.

    This....no one seems to understand the real cost of mitigation.

    Real cost of bandwidth to begin with. We've seriously had people close to decade ago throw a hisfit tantrum forum rampage for why they did not get dedicated 1Gbps with dedicated server at 10€ a month, 100% guaranteed all the way to their home in India ... from europe. Sure thing, if you have a few trillion $ to throw at it just to study the scope lol

    Wow That's even worse then cases we had in recent years on our dedicated-servers

    I love it in general when peoples think pushing alot of traffic is cheap......

  • PulsedMediaPulsedMedia Member, Patron Provider

    @HostSlick said:

    @PulsedMedia said:

    @MrRadic said:

    @PulsedMedia said:
    150Gbps is not small in anyway. To counter that it would mean they would need to have 100s of Gbps of just idle capacity, paid for just-in-case this kind of stuff happens. That is a VERY expensive.

    This....no one seems to understand the real cost of mitigation.

    Real cost of bandwidth to begin with. We've seriously had people close to decade ago throw a hisfit tantrum forum rampage for why they did not get dedicated 1Gbps with dedicated server at 10€ a month, 100% guaranteed all the way to their home in India ... from europe. Sure thing, if you have a few trillion $ to throw at it just to study the scope lol

    Wow That's even worse then cases we had in recent years on our dedicated-servers

    I love it in general when peoples think pushing alot of traffic is cheap......

    They also think they need at least Gbps dedicated for them alone, for their irc idling habits

  • NyrNyr Community Contributor, Veteran

    @risharde said: some providers claim to be able to tank higher levels than 150 GBIT worth

    Many providers claim that, and some providers can actually offer this, but the list is short. Only big (as in network size) providers are able to absorb sustained attacks of this size and even then, it is costly.

    Still, there are some claims in this thread which can lead to confusion on the costs, because for those providers DDoS mitigation is not THAT expensive. They usually have a heavy outbound bias in their network (most hosting providers do) and have lots of spare inbound capacity, which can be used for mitigation. Their upstreams can also help with volumetric attacks before they reach the destination.

    Thanked by 2Maounique risharde
  • MrRadicMrRadic Patron Provider, Veteran

    @Nyr said:

    @risharde said: some providers claim to be able to tank higher levels than 150 GBIT worth

    Many providers claim that, and some providers can actually offer this, but the list is short. Only big (as in network size) providers are able to absorb sustained attacks of this size and even then, it is costly.

    Still, there are some claims in this thread which can lead to confusion on the costs, because for those providers DDoS mitigation is not THAT expensive. They usually have a heavy outbound bias in their network (most hosting providers do) and have lots of spare inbound capacity, which can be used for mitigation. Their upstreams can also help with volumetric attacks before they reach the destination.

    Capacity isn't just transit, it's really expensive network hardware, network engineers who knows their stuff, major investment in anti DDoS tech, etc.

    Please don't downplay costs and abilities. No upstream wants to help you without getting paid.

  • MaouniqueMaounique Host Rep, Veteran

    @MrRadic said: major investment in anti DDoS tech

    This anti-ddos tech is a myth busted last decade.
    In theory, yes, you can have it, but in practice, the "tech" is too rigid to deal with many challenges of today and none can face the challenges of tomorrow because nobody has discovered them yet.
    TCP/IP is a mess, it is a protocol designed for openness, hardening it down, especially UDP (which, well, isn't exactly TCP) is not easy and can't be done properly with billions of hosts instead of tens as they were in the beginning.
    Using ASICs and other single-use gear is not ideal due to the evolving attacks, while using a general purpose CPU is also not practical so the current routing gear has a bit of both.
    Yes, it is expensive, but it is not really "anti DDoS tech", more like routers with some features which can be used to mitigate.

Sign In or Register to comment.