Massive Layer7 attack, more than 33 hours

Abd

@solari said:
P.S @Abd using a Chinese webpanel to serve up the client area isn't a good look, either, but I have noticed something in common with these providers. They're either Indian, Romanian, or come from some sort of shit nationality. It is for this reason, why I only go with Aryan-white ran providers.

can't believe you're in 2022 with such views.

DP

@HostSlick said:
The Problem is that OP is giving the Attacker attention with the thread.

Usually they stop if you are successful at mitigating and/or they dont get attention. Longest (BIG) layer7 attack here took 1 week non-stop.

Attacker always knows his next step watching this here and is probably amusing himself

And since he knows the next step he can prepare himself as well.

👇

@DP said:
Maybe it would be good to discuss this in private perhaps?

The last thing you'd want is your enemy(-ies) to know your next move(s).

Just a thought.

HostSlick

@DP said:

@HostSlick said:
The Problem is that OP is giving the Attacker attention with the thread.

Usually they stop if you are successful at mitigating and/or they dont get attention. Longest (BIG) layer7 attack here took 1 week non-stop.

Attacker always knows his next step watching this here and is probably amusing himself

And since he knows the next step he can prepare himself as well.

👇

@DP said:
Maybe it would be good to discuss this in private perhaps?

The last thing you'd want is your enemy(-ies) to know your next move(s).

Just a thought.

Correct. @DP a smart guy

sandoz

@HostSlick said:
The Problem is that OP is giving the Attacker attention with the thread.

Usually they stop if you are successful at mitigating and/or they dont get attention. Longest (BIG) layer7 attack here took 1 week non-stop.

Attacker always knows his next step watching this here and is probably amusing himself

And since he knows the next step he can prepare himself as well.

Correct... Follow this.
Giving attention who want to do shit only will give motivation to do it more.

Hostslick is correct, follow this advice. Stay in your business and let them away. Keep away anything with competitors etc.

Probably you are being attacked because you are exposing everything here. If you ignore probably attacker will stop. Until that he is probably seeing that you are being affected with this.

cybertech

he should be busy right now at the cybercrime office.

AndreiGhesi

@cybertech said:
he should be busy right now at the cybercrime office

Lol. If you don't know who is the attacker and you don't have solid proof that he is behind the attack, the police will do nothing.

I played my games with Romanian cyber police/layers and judges to know that they know nothing about this sector.

jsg

@NoComment said:
Are you suggesting installing one adapter per server? Don't forget that most of his hardware is second-hand and an additional $500 could mean another 6 mths to recover costs. When you do have an entire rack, what are you gonna do with these adapters? Install some adapters on a server and some network cards to connect to the whole rack?

When a provider has (an) entire rack(s) they could and should do what I said earlier: install dual firewall machines with such adapters. "Paying for itself" by using the (web or whatever) servers for their task only.

Re "second hand": those adapters can be had second hand too.

@all

There is another, non-technical aspect. The attacker - and frankly, @FlorinMarian's public asking for help and some of actions I'm afraid - might lead to (and possibly aim for) pushing Hazi.ro out of business. Simple approach: make them look really bad and incompetent; maybe after a while FlorinMarian will get it under control (with the help of professionals) but by then it's probably too late (actually it might already be too late as in "too much damage done"), because "[big number] DDOS protection", oh well, really? Doesn't look like it.

In other words, a win-win situation for the attacker, sadly to a large degree due to how FlorinMarian dealt with it (or not really so far, it seems).

That's just my view and I might be wrong but (a) most customers expect that providers somehow protect their products and are (or at least look) skilled and experienced, (b) a provider might have a short windows (hours, not days) to ask for help; if the situation is still active after max 48 hours customers and potential customers are likely to loose trust. Add to that that Hazi.ro (at least here) is not yet considered an established solid provider.

Sad. I like(d?) Hazi and FlorinMarian and I sincerely hope that I'm wrong with the above.

LTGT

@jsg said:

@NoComment said:
Are you suggesting installing one adapter per server? Don't forget that most of his hardware is second-hand and an additional $500 could mean another 6 mths to recover costs. When you do have an entire rack, what are you gonna do with these adapters? Install some adapters on a server and some network cards to connect to the whole rack?

When a provider has (an) entire rack(s) they could and should do what I said earlier: install dual firewall machines with such adapters. "Paying for itself" by using the (web or whatever) servers for their task only.

Re "second hand": those adapters can be had second hand too.

@all

There is another, non-technical aspect. The attacker - and frankly, @FlorinMarian's public asking for help and some of actions I'm afraid - might lead to (and possibly aim for) pushing Hazi.ro out of business. Simple approach: make them look really bad and incompetent; maybe after a while FlorinMarian will get it under control (with the help of professionals) but by then it's probably too late (actually it might already be too late as in "too much damage done"), because "[big number] DDOS protection", oh well, really? Doesn't look like it.

In other words, a win-win situation for the attacker, sadly to a large degree due to how FlorinMarian dealt with it (or not really so far, it seems).

That's just my view and I might be wrong but (a) most customers expect that providers somehow protect their products and are (or at least look) skilled and experienced, (b) a provider might have a short windows (hours, not days) to ask for help; if the situation is still active after max 48 hours customers and potential customers are likely to loose trust. Add to that that Hazi.ro (at least here) is not yet considered an established solid provider.

Sad. I like(d?) Hazi and FlorinMarian and I sincerely hope that I'm wrong with the above.

well so far the thread has been "buy protection xyz", florin implements said solution and he still gets knocked out. Ofc it would be better if this didn't face the public, but rather a thread "admitting defeat" than having the hosting site down for weeks.

This might not be the highest volume attack, but it's still interrupting his business, customers are unable to pay their current invoices for example.

he had cloudflare, cloudflare let it trough

he had path, path let it trough.

(besides the fact that path is build on a open source project from github)

jsg

@LTGT said:
well so far the thread has been "buy protection xyz", florin implements said solution and he still gets knocked out. Ofc it would be better if this didn't face the public, but rather a thread "admitting defeat" than having the hosting site down for weeks.

This might not be the highest volume attack, but it's still interrupting his business, customers are unable to pay their current invoices for example.

he had cloudflare, cloudflare let it trough

he had path, path let it trough.

(besides the fact that path is build on a open source project from github)

That may all be true but customers rarely say "it's not his fault" and buy. Again, I'm absolutely not against @FlorinMarian and Hazi.ro, quite the contrary and I sincerely wish them well and a speedy recovery.
But you see, there is another potentially deadly element in this, the "too many advisors" problem and its ugly consequences.

I sincerely hope that I'm wrong in my post above and I hope that Hazi.ro will do well again.

sandoz

@LTGT said:

@jsg said:

@NoComment said:
Are you suggesting installing one adapter per server? Don't forget that most of his hardware is second-hand and an additional $500 could mean another 6 mths to recover costs. When you do have an entire rack, what are you gonna do with these adapters? Install some adapters on a server and some network cards to connect to the whole rack?

When a provider has (an) entire rack(s) they could and should do what I said earlier: install dual firewall machines with such adapters. "Paying for itself" by using the (web or whatever) servers for their task only.

Re "second hand": those adapters can be had second hand too.

@all

There is another, non-technical aspect. The attacker - and frankly, @FlorinMarian's public asking for help and some of actions I'm afraid - might lead to (and possibly aim for) pushing Hazi.ro out of business. Simple approach: make them look really bad and incompetent; maybe after a while FlorinMarian will get it under control (with the help of professionals) but by then it's probably too late (actually it might already be too late as in "too much damage done"), because "[big number] DDOS protection", oh well, really? Doesn't look like it.

In other words, a win-win situation for the attacker, sadly to a large degree due to how FlorinMarian dealt with it (or not really so far, it seems).

That's just my view and I might be wrong but (a) most customers expect that providers somehow protect their products and are (or at least look) skilled and experienced, (b) a provider might have a short windows (hours, not days) to ask for help; if the situation is still active after max 48 hours customers and potential customers are likely to loose trust. Add to that that Hazi.ro (at least here) is not yet considered an established solid provider.

Sad. I like(d?) Hazi and FlorinMarian and I sincerely hope that I'm wrong with the above.

well so far the thread has been "buy protection xyz", florin implements said solution and he still gets knocked out. Ofc it would be better if this didn't face the public, but rather a thread "admitting defeat" than having the hosting site down for weeks.
he had path, path let it trough.

(besides the fact that path is build on a open source project from github)

Path is build with Open source project? Where? That is new.

NoComment

@jsg said:

@NoComment said:
Are you suggesting installing one adapter per server? Don't forget that most of his hardware is second-hand and an additional $500 could mean another 6 mths to recover costs. When you do have an entire rack, what are you gonna do with these adapters? Install some adapters on a server and some network cards to connect to the whole rack?

When a provider has (an) entire rack(s) they could and should do what I said earlier: install dual firewall machines with such adapters. "Paying for itself" by using the (web or whatever) servers for their task only.

That would require more work and is honestly quite silly because when you have an entire rack you are probably better off just buying a router. This adapter solution is in a weird position imo because in a colocation setting, it doesn't make sense for 1-2U colocations and when you have scale. Add to that the fact that someone in the same datacenter probably has spare capacity.

These adapters were not designed for scrubbing, you have to do a lot of work to make it happen and it may make sense if you are just doing some simple filters or if you have just the one server in an office setting instead of in a datacenter. Either way, it's clearly not the solution for the average small provider on LET.

LTGT

@jsg said:

@LTGT said:
well so far the thread has been "buy protection xyz", florin implements said solution and he still gets knocked out. Ofc it would be better if this didn't face the public, but rather a thread "admitting defeat" than having the hosting site down for weeks.

This might not be the highest volume attack, but it's still interrupting his business, customers are unable to pay their current invoices for example.

he had cloudflare, cloudflare let it trough

he had path, path let it trough.

(besides the fact that path is build on a open source project from github)

That may all be true but customers rarely say "it's not his fault" and buy. Again, I'm absolutely not against @FlorinMarian and Hazi.ro, quite the contrary and I sincerely wish them well and a speedy recovery.
But you see, there is another potentially deadly element in this, the "too many advisors" problem and its ugly consequences.

I sincerely hope that I'm wrong in my post above and I hope that Hazi.ro will do well again.

rather a website online with "help of the community" than a website that is off.

of course it hurts his/hazi's image, but if he doesn't know a solution, better ask for help than letting it die.

@sandoz said:

@LTGT said:

@jsg said:

@NoComment said:
Are you suggesting installing one adapter per server? Don't forget that most of his hardware is second-hand and an additional $500 could mean another 6 mths to recover costs. When you do have an entire rack, what are you gonna do with these adapters? Install some adapters on a server and some network cards to connect to the whole rack?

When a provider has (an) entire rack(s) they could and should do what I said earlier: install dual firewall machines with such adapters. "Paying for itself" by using the (web or whatever) servers for their task only.

Re "second hand": those adapters can be had second hand too.

@all

There is another, non-technical aspect. The attacker - and frankly, @FlorinMarian's public asking for help and some of actions I'm afraid - might lead to (and possibly aim for) pushing Hazi.ro out of business. Simple approach: make them look really bad and incompetent; maybe after a while FlorinMarian will get it under control (with the help of professionals) but by then it's probably too late (actually it might already be too late as in "too much damage done"), because "[big number] DDOS protection", oh well, really? Doesn't look like it.

In other words, a win-win situation for the attacker, sadly to a large degree due to how FlorinMarian dealt with it (or not really so far, it seems).

That's just my view and I might be wrong but (a) most customers expect that providers somehow protect their products and are (or at least look) skilled and experienced, (b) a provider might have a short windows (hours, not days) to ask for help; if the situation is still active after max 48 hours customers and potential customers are likely to loose trust. Add to that that Hazi.ro (at least here) is not yet considered an established solid provider.

Sad. I like(d?) Hazi and FlorinMarian and I sincerely hope that I'm wrong with the above.

well so far the thread has been "buy protection xyz", florin implements said solution and he still gets knocked out. Ofc it would be better if this didn't face the public, but rather a thread "admitting defeat" than having the hosting site down for weeks.
he had path, path let it trough.

(besides the fact that path is build on a open source project from github)

Path is build with Open source project? Where? That is new.

path has been removed now,

but their js challenge was the following:
https://pastebin.com/E1S41vFY

and you can find it here:

https://github.com/kyprizel/testcookie-nginx-module/blob/51137466b12b2b023f0208a0b6a1bc9c0e805618/README#L227

jsg

@NoComment said:

@jsg said:

@NoComment said:
Are you suggesting installing one adapter per server? Don't forget that most of his hardware is second-hand and an additional $500 could mean another 6 mths to recover costs. When you do have an entire rack, what are you gonna do with these adapters? Install some adapters on a server and some network cards to connect to the whole rack?

When a provider has (an) entire rack(s) they could and should do what I said earlier: install dual firewall machines with such adapters. "Paying for itself" by using the (web or whatever) servers for their task only.

That would require more work and is honestly quite silly because when you have an entire rack you are probably better off just buying a router. This adapter solution is in a weird position imo because in a colocation setting, it doesn't make sense for 1-2U colocations and when you have scale. Add to that the fact that someone in the same datacenter probably has spare capacity.

These adapters were not designed for scrubbing, you have to do a lot of work to make it happen and it may make sense if you are just doing some simple filters or if you have just the one server in an office setting instead of in a datacenter. Either way, it's clearly not the solution for the average small provider on LET.

Well, let's just agree to disagree (have quite different views and approaches).

@LTGT said:

@jsg said:
That may all be true but customers rarely say "it's not his fault" and buy. Again, I'm absolutely not against @FlorinMarian and Hazi.ro, quite the contrary and I sincerely wish them well and a speedy recovery.
But you see, there is another potentially deadly element in this, the "too many advisors" problem and its ugly consequences.

I sincerely hope that I'm wrong in my post above and I hope that Hazi.ro will do well again.

rather a website online with "help of the community" than a website that is off.

of course it hurts his/hazi's image, but if he doesn't know a solution, better ask for help than letting it die.

Theoretically yes, but practically clearly no. Simple reason: LET is mainly visited and viewed by buyers/customers/potential customers.

stefeman

@jsg said:

@LTGT said:.

he had cloudflare, cloudflare let it trough

He never configured it correctly. He never followed any of our guidance.

The first thing he did was disabling CF when the stock settings on his free plan failed.

Hence why this is waste of time.

Blazingfast_IO

We have offered our help here and anyone that is going through this for a limited time we have our reverse proxy for free something similar to cloudflare free plan for now.

Send me a PM if you have any questions, all you need to do is create an account in our customer panel, order anycast dns which is free, add your domain after that you will have a message to use one of our IP for your dns A records.
(This is a work in progress)

nanankcornering

so.. 2 days have passed.. and it's still going...?

LTGT

@nanankcornering said:
so.. 2 days have passed.. and it's still going...?

yes

NoComment

@LTGT said:

@nanankcornering said:
so.. 2 days have passed.. and it's still going...?

yes

They got brought down by 5 Mbps so it probably isn't costing the attacker a lot of money to ddos them.

LTGT

@NoComment said:

@LTGT said:

@nanankcornering said:
so.. 2 days have passed.. and it's still going...?

yes

They got brought down by 5 Mbps so it probably isn't costing the attacker a lot of money to ddos them.

well its a layer 7 attack so 5 Mbps isn't all that "meaningful"

yoursunny

Florin is my brother.
People need to stop bullying my brother.

Don't hide in the shadows behind DDoS toys.
Come out and duel at dawn.

Zigi

@LTGT said:

@jsg said:

@NoComment said:

he had path, path let it trough

We didn't let any of the attack through. He disabled the domain due to the domain was misconfigured. Which was fixed

@sandoz said:
Path is build with Open source project? Where? That is new.

As for the testcookie, testcookie is just a part of our mitigation stack for our HTTPs clients; however, we have other parts of our mitigation stack always watching for bad actors / malicious requests.

DP

LTGT

@Zigi said:

@LTGT said:

@jsg said:

@NoComment said:

he had path, path let it trough

We didn't let any of the attack through. He disabled the domain due to the domain was misconfigured. Which was fixed

@sandoz said:
Path is build with Open source project? Where? That is new.

As for the testcookie, testcookie is just a part of our mitigation stack for our HTTPs clients; however, we have other parts of our mitigation stack always watching for bad actors / malicious requests.

so the 503 that was shown was not cause path let trough too many r/s ?

nanankcornering

@FlorinMarian said: I disabled CloudFlare for 24 hours because it was useless.

>

well if he exposes the direct server ip.. it's useless using path right?

or did path announce his /24 too? @Zigi

if not, everything is useless then........................

Thundas

His site seems fine now.

Hotmarer

@FlorinMarian Why are you teasing @tinyweasel? Just apologize him ...

Boogeyman

@yoursunny said: Florin is my brother.
People need to stop bullying my brother.

Don't hide in the shadows behind DDoS toys.
Come out and duel at dawn.

He picked fight with the wrong Weasel.

I wonder what happened with @Francisco . Didn't he report those attack to cybercrime investigation agencies?

Francisco

@Boogeyman said: I wonder what happened with @Francisco . Didn't he report those attack to cybercrime investigation agencies?

I did I included some of his friends too.

Francisco

Hotmarer

@Boogeyman said: I wonder what happened with @Francisco . Didn't he report those attack to cybercrime investigation agencies?

@Francisco is not as stupid as a @FlorinMarian, he did not tease tinyweasel creating threads about attack on LET ...

nanankcornering

@Ahfaiahkid said:
His site seems fine now.

after 6 pages, 2 days, ovh is the choice..