New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
And again, who says it needs (or was) a raw packet? I just tried this behaviour in my own network with a OpenVZ VM (which can't do it either) and can replicate it by just sending similar UDP packets (which is not raw nor L2) to the broadcast IP. Outside of the NAT (which in the DC would be the gateway of the L2) it does not work, L3 routed it works with redirect (not enabled in DC; only in my test setup). You could also explain why your application sends this traffic in a datacenter if it was not misused - Broadcast in general is very unliked there and useless as nobody locally will react to it (and if you have other local servers in a cluster they should connect by IP directly, not by broadcasting it's presence). I don't know any gameserver that requires this to be running and cannot use direct IP connects instead (at least none that would work on Linux or VRS).
This is also going on for days now at least (more likely weeks) - Once i stopped it my monitoring KVM lost a lot of problems (timeouts mainly) and since then runs fine (i now have a SSH session open for 90min, formely not even 10 were possible without disconnect). I just thought it was overloaded but... yea, no, does not seem so.
Yes. As said - It is investigated, thats what i do right now and since then.
And again, it is still investigated how this was possible at all - I have never seen this before and i see a lot of DDoS, Spoofing and general weird network traffic (Netbios anyone?) weekly.
So when will VM be back online? And can you clarify whether sending packets to broadcast address due to program that sends it there by default is something I am supposed to ensure none of my programs are doing, or are you going to fix something from your side to prevent broadcast or make it do something other than send it to all IPs?
Some people run it behind router, and they have two programs on different computers that communicate with each other through broadcast. The game is old one that still uses broadcast for LAN and that is the same kind of packet used to send from one program to the other saying that the game exists. The broadcast is only meant to be responded to by another program on my VM, and it doesn't even matter if broadcast is discarded since it sends to loopback too now (but on some people who run it the loopback doesn't work and only broadcast does, so it tries both..).
You can say "stop sending broadcast" and I can update my program, but instead you suspended it...
And it is sending regular UDP broadcast, not any other transport protocol broadcast packet.
This won't work over the internet, so it should be disabled on the VPS.
Once on the routers its L3 and dropped anyway.
(Thank you for the broadcast explanation also, i know very well how it works, how to exploit it for good and bad and how to prevent and cause this things.)
Again, this caused much issues - Just by accident to me instead of another customer - so i see the suspension as entirely correct, it is the same as we would have done if another customer would have reported it. Alone the PP/S after multiplication justified this. As said before - Once investigated it will be unsuspended until it happens again.
It is the default, I didn't update default before running it because it never caused problem except on your server.
How should I know your network doesn't handle broadcast packet properly without making it cause network problem for your server, like by simply dropping it? And VM is still suspended even though I already said, I have moved the program to another server and if I move it back I'll change it to only send to loopback instead of loopback+broadcast.
Let me ask again - Why do you think running broadcast in a datacenter is worth even doing? OVH suspends you automatically for this and Hetzner reacts very, VERY mad on it, others are similar.
And i already said: It is investigated. While it is investigated the VM stays off until we know how this happened, like any good ISP would do to prevent further damage/issues.
The broadcast is the default, the intention is simply for another program on the VM to receive it. wtf is "worth even doing" supposed to mean? Most dedicated server have their own IP range and sending UDP packet to 255.255.255.255 goes inside the subnet only, why would that cause problem to datacenter?
Again: It did go into the subnet only, where my KVM happened to be (along with my other IP space, it adds up.) - It never crossed any L3 barrier (which is impossible).
I am not sure what the problem here is, really - Your server caused issues (which were caused by it, this is undoubtable), it was suspended and the issue is investigated since 16:00 (thus not even 2 hours). Yes, this could take until Monday (as said in the Mail), but it's unlikely - Until you know at all if it does take that long complaining about the timeframe is simply not acceptable, (lets assume) 3 hours on a weekend is not much at all.
What's your point? If a single packet sent to 255.255.255.255 overloads network a lot then you should disable it. I can disable it on VM after it is booted back up. Sending to broadcast address shouldn't be problem on your VM's, many old programs use it instead of loopback.
It was not a single packet - Your VPS sent a continuous stream of packets to the broadcast, at least since today - Likely much longer already. I said this before and made it clear in my mail as well.
Yes, like less than ten per second, probably less than even one per second.
Anyway I wasn't told that UDP broadcast packet is not permitted and causes problem on your network. It is just UDP packet sent to "255.255.255.255", not like it was made to mess with system; most providers handle it by dropping packet or something similar, not doing something that allows any broadcast packet to cause network problem.
Sorry but where did i ever state this or anything even remotely like this? Please show me. I stated that you were likely NOT doing this intentionally, 2 (and now 4) times. It caused issues and was suspended for investigation. It caused issues and was suspended for investigation as any ISP would do.
More like 10k/PP/S (unicast) - And even that with much higher bursts. At the same time a lot of inbound PPS as well (again, likely with spoofed headers) originating from out of country (non VIX peer group) via Retn.
Again, likely from Ecatel/Voxility, i will know at least the country on Monday when i get the frozen paths from Retn NOC, dumped before and after i stopped it.
I did this many times before, it's not like this is my first attack either as target or ISP, i run a lot of services that get attacked many times and have a lot experience in this (not as much as DDoS experts and the protection providers - 100% sure, obviously - But clearly much more than average even in this business).
For references:
I host a few very large german satirical websites behind Cloudflare and my own protection (which means BW, BW and more BW in the cheapest possible good location, NL) (sponsored for friends mainly), these get attacked heavily weekly in the range of 30-50G reflection attacks (that CF and my infra independently eat like it is nothing).
I host some streaming services that are attacked by "others" all the time (by now 10G UDP 24/7, often high bursts of very sophisticated TCP based attacks as UDP is dropped upstream wise, also got s)
I do this in my day job and emergency support since multiple years as well
Etc. etc.
How long is investigation going to take? You are saying 10k pp/s after it gets broadcasted or 10K broadcast packets per second? I already migrated the application I run that uses UDP so why can't you unsuspend the VM now, it is only hosting TCP application now that shouldn't cause problem on your network; I send a few broadcast packets from normal program and you don't handle them properly by dropping them or otherwise not sending them to all IP on the host node and then it gets suspended for 48 hours?
Again, as said multiple times, worst case until Monday.
10k outbound before amp, with an amp factor of (with my current data) around x30 (thus 300k PPS, 90% to my KVM -> massive issues, rest to other customers -> Less but still noticeable issues)
Because i am investigating how this attack works and won't unsuspend it before i know how it can be prevented - Again, this is what any ISP should do.
And how did you remove it if the vm is suspended...? Autostart? All things i have to consider.
We handle them very properly, in fact we handle them as they are designed by the standards. It is also dropped behind the switch obviously - I never said anything otherwise and clearly stated that it only affected this network segment.
I never said this either - I said it is suspended until Monday, which is an open timeframe to be unsuspended before - This now also multiple times already.
Sorry but i repeat myself here, this is not a useful discussion at all.
I clearly explained what happened, i clearly explained that it is investigated, i clearly explained everything you wanted to know multiple times for everyone to understand, sent you proof of the attack/"problem" and so on.
I verified that it was caused by your vm alone before suspension, ran a test in my private network setup to verify this is feasible at all before going into deeper analysis, organized dumps of current routes from the upstreams to analyze origin and report it later if possible ( nobody else seems to do that), analyze logs and research preventive measures as well as sideband attacks that could be used on a similar way - All that without even knowing what application it is. To protect other customers, as well as you, from further attacks in the future and knowing how to efficiently resolve them (if it cannot be prevented at all).
Other ISPs (like Linode) would simply nullroute you for 24hours without any chance of unsuspension before (we all know this happened before to LEB...), some providers would even cancel you straight away, and a few terminate you.
It doesn't autostart. I migrated it to different VM from backups. It cannot be started anymore since the panel that my client sees is redirecting to different server; and then it'll be deleted when VM is booted back up.
Anyway I will wait for unsuspended.
Other ISPs also violate RFCs. And now again, for the last time: This is EXACTLY one of the reasons WHY it is suspended while i investigate. Please read my posts, i stated this before. You cause much more delay by this here also, it's not helpful in any way.
This discussion is over - I made mine and our company point very clear and explained the entire process and what happened/what happens currently/what will happen.
I see nothing wrong with our/my reaction to the immediate reocurring (well, permanent for days, because i do work while writing here and know now it was going on for days at least) threat (that i in this case even got to heavily feel myself, again) and the suspension timeframe is still open and not even 3 hours yet.
End of thread. You will get further informed by email like any other customer that gets attacked or causes such issues/attacks.
2 hours 57 minutes - Investigation done, VPS booted, network monitoring for this situation implemented.
On a saturday. While i don't even work and just did this because it affected myself, free.
Case closed until further issues.
From my view it looks like one provider abusing another provider.
Well, I was considering to try Luna Node VPS but after this will stay away from them as nobody knows what they're doing.
Any comment about the attack? I'm curious
Most providers (both VPS and dedicated server) filter 255.255.255.255 or at least break down the broadcast domain to small units so that UDP packets sent there will not be forwarded. Common reasons for applications to send UDP broadcast packets include peer detection (as in my case), so that when the same program is executed in a LAN network it can find peers who can connect to each other after receiving each other's broadcast packets. Obviously LAN doesn't apply on servers, but these applications are designed to work in any environment.
For example (application that I'm running): see http://ghostplusplus.googlecode.com/svn/trunk/ghost/game_base.cpp line 425. Other applications that would use broadcast packets include things like samba.
The applications I am running send about 1-2 packets per second to 255.255.255.255, relating to game server (so in LAN, the purpose would be for LAN peers to see the game information and be able to connect to the game; on server it is useless but it is the default configuration for the application). This is not at all abnormal, as the game is about a decade old (nowadays multicast and such are used). The router then apparently multiplies these packets and sends them to all the IP's in the subnet, apparently there are 5000+ targets; so the couple of broadcast packets per second become 10000 packets per second.
William's characterization of this as an "attack" is misleading. UDP packets sent to 255.255.255.255 broadcast address are common in many older programs and do not cause problems unless the network is set up in such a way that they would be forwarded to so many hosts. I am running the same program with the same setup on several other providers (10+ others) and none of them have this issue (packets to 255.255.255.255 are either filtered away, or in one or two of the ones I tested only are received by a small number of other machines). If the network were setup as it should be, to filter the packets, there would be no problem; there is no reason why packet to 255.255.255.255 should be redirected by the router to every machine on the subnet on a server router.
Additionally, I was never told that the packets to broadcast address were causing any sort of issue on there side. When I learned of the issue, I immediately said that I would reconfigure the program so that it would only send packets to 127.0.0.1 instead of using the default 255.255.255.255. However, the VM was suspended for three hours while some kind of "investigation" was done of what they characterized as an "attack" on EDIS network.
He says other people would cancel service but that doesn't make sense because no one else has same network configuration which broadcasts to 5000 machines as EDIS does.
I run the program happily on other machines (and I've since migrated the particular instance that was running on EDIS VM to a Prometeus VM). There are hundreds of other users who also run it as a server and have not experienced any problems. This is without special configuration to use 127.0.0.1 for broadcast target instead of 255.255.255.255.
Void.
OVH, Hetzner, FDCServers (in all DCs)... i could go on and on. Basically any provider that has no dedicated VLANs per server (to save IPs mainly on dedicated, not feasible technically on vitualization often), and on virt platforms next to no one does this (hint: ebtables and mac filtering works very well, OVH does this for 100k+ servers in just a few network segments).
This behaviour, as in amplification, is expected with a broadcast storm (broadcast...storm...right.).
How it was generated (or sent) is in the end not relevant at all, if it can be amplified (which is just broadcasting, which is the usage of broadcast) it will be, again (i just repeat myself...) it never left the switch/vlan.
Void.
As said it was much more sent by it, continuos.
Void.
It caused issues for my VM and other users in the network, as said.
Void.
It was not discovered before.
Oh come on, seriously? Thats a very low point by now.
3 hours are nothing for a network threat. I just repeat myself again with things i don't even need to, 3hours are perfect on a weekend. Try to get this on Hetzner or Online.net (or nearly any provider in Germany, Austria and Switzerland or rather most of the EU). For a not even 10EUR product with zero guarantees in any way except RAM, neither uptime wise nor in any other imaginable way (see T&C and below).
Void.
Prometeus uses VLANs (at iwstack at least), further they use an entirely different network design and different routing hardware than we or for example FDCServers and OVH do.
Void.
Not in our network.
Void.
This is likely due to VLANs in smaller ranges and so on - Without providing the network design, used hardware and their configurations you have no point, each network - even if it looks similar - varies very much internally.
Next. I gladly discuss this with you forever (/sarcasm) - Until now your (multiple times repeated) arguments had no substantial point at all. Further, you might want to read the AGB/Tos (that you confirmed to be read and understood at order, so "read again") which clearly state what we can do and what we are liable for and the support guarantees (7.1, 7.2, 9.1, 10.3 (yes, it's not worded as such but legally valid for VPS), 12.3 for the future).
Hmmm, really? I don't see any change like this you said you have done on the current network - It still sends this packets the same as before, just not getting amplified anymore (and now, compared to the screenshot i sent in my email, in a much slower way - Thus the normal, non attacking behaviour, because the booter likely has removed it from his amp target list) http://i.imgur.com/6HF2CNk.png
Yes, we have a VLAN setup more suitable for small and corporate networks that need isolation and some protection from ARP and broadcast storms.
This is because the prometeus offer is mostly intended for corporate networks and clusters for streaming it has serious drawbacks, one being that quite a few IPs are wasted and we have some crazy offers at times to get those filled (such as, but not limited to-low end gems)
There is no perfect solution, in the end, each provider does what they consider best (safety, cost, maintenance ease, protection, etc), our solution has a lot of issues of it's own but it does have a few advantages as well...
It looks like on your OpenVZ plans any UDP packets to broadcast address are filtered entirely.
Edit: anyway I'm sorry William, I wasted your time posting here and three hours downtime is not much.