Do better Linode/Akamai

SplitIce

"We have detected an Outbound Denial of Service attack originating from your Linode. This is most likely the result of a system compromise. Because of the serious nature of this activity, we have applied network restrictions to your Linode to prevent further abuse."

Strange, we nearly exclusively use them for running Kubernetes clusters, and that server they have shutdown is running latest* LKE image (minor release was announced wednesday for a Distro update, I havent validated this yet). Cluster has extensive auditable security logging etc all of which shows clear. Did see a warning about SYN cookies on the BGP port a while ago, was it that? no, that appears to have been noise.

Dig a bit deeper and get the abuse report given to us.

The reporter? Linode (self report)
target_ip: 172.232.31.233
traffic_stats_byte_count: 13558735

Err, we ddos'ed their S3 service with our system backups by uploading 13MB. Looks like we were uploading at around 100mbit at the time (same speed as usual).

I dont doubt this was 100% automated from report, response to action given that Linode doesnt usually much in the way of weekend staff (remains to be seen when this node will be returned to operation, its been 8 hours now).

Do better Linode / Akamai. You are supposed to be production grade. I had better things to be doing with my Sunday than dealing with entirely preventable issues. Fortunately we have redundancy...

forest

Still better than Hetzner.

rpqu

production env
can't withstand 13MB at 100mbps

MikeA

They were always overhyped anyway. Big money marketing.

trewq

@forest said:
Still better than Hetzner.

They don’t think abuse happens on the weekend.

SplitIce

@forest said:
Still better than Hetzner.

Extra funny considering that the DRP for if we need to leave Linode is Hetzner Cloud.

VTCuong

@MikeA said:
They were always overhyped anyway. Big money marketing.

My rule of thumb is always avoid these big YouTuber sponsorship businesses, because they prioritize marketing over user experience.

TimboJones

@SplitIce said:
"We have detected an Outbound Denial of Service attack originating from your Linode. This is most likely the result of a system compromise. Because of the serious nature of this activity, we have applied network restrictions to your Linode to prevent further abuse."

Strange, we nearly exclusively use them for running Kubernetes clusters, and that server they have shutdown is running latest* LKE image (minor release was announced wednesday for a Distro update, I havent validated this yet). Cluster has extensive auditable security logging etc all of which shows clear. Did see a warning about SYN cookies on the BGP port a while ago, was it that? no, that appears to have been noise.

Dig a bit deeper and get the abuse report given to us.

The reporter? Linode (self report)
target_ip: 172.232.31.233
traffic_stats_byte_count: 13558735

Err, we ddos'ed their S3 service with our system backups by uploading 13MB. Looks like we were uploading at around 100mbit at the time (same speed as usual).

I dont doubt this was 100% automated from report, response to action given that Linode doesnt usually much in the way of weekend staff (remains to be seen when this node will be returned to operation, its been 8 hours now).

Do better Linode / Akamai. You are supposed to be production grade. I had better things to be doing with my Sunday than dealing with entirely preventable issues. Fortunately we have redundancy...

Ask for a SCAR.

Levi

@VTCuong said:

@MikeA said:
They were always overhyped anyway. Big money marketing.

My rule of thumb is always avoid these big YouTuber sponsorship businesses, because they prioritize marketing over user experience.

Well, when you experience event when one morning your let provider starts selling hardware and ip space on olex.ro while your server still active… It is better to go with youtube advertised hosts.

nghialele

Hmmm, send it here to who? 😄 I’m sure that Linode CEO would spend time reading all this.

VTCuong

@Levi said:

@VTCuong said:

@MikeA said:
They were always overhyped anyway. Big money marketing.

My rule of thumb is always avoid these big YouTuber sponsorship businesses, because they prioritize marketing over user experience.

Well, when you experience event when one morning your let provider starts selling hardware and ip space on olex.ro while your server still active… It is better to go with youtube advertised hosts.

Ahh yes, our favourite YouTube advertiser, CONTABO.
We all know that these over-hyped products are all mediocre, over-priced at best and straight up a scam at worst. Also, you had a bad experience with a LET provider, and now you assume every other one are the same, what kind of mentality is this?

Levi

@VTCuong said: Also, you had a bad experience with a LET provider, and now you assume every other one are the same, what kind of mentality is this?

Welcome to LET I guess...