Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Datashack Wholesaleinternet Packet Loss - A warning to dedicated server providers
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Datashack Wholesaleinternet Packet Loss - A warning to dedicated server providers

earlearl Member

Been experiencing some packet loss issues past couple of days on my Datashack server.

If anyone is interested in the reason why, here is a re post from WHT where Aaron of Wholesaleinternet explains a little bit about the issue.

http://www.webhostingtalk.com/showthread.php?t=1370854

I'm not normally one to air my dirty laundry publically or post rants but I felt this issue was serious enough and, from what I hear, is affecting several providers in the states.

So here's the story:

Back in January we started to see a huge influx of Chinese customers. We do a lot of business with China so it wasn't any massive surprise but normally what we see is Chinese resellers, we don't get a ton of individuals. Our average Chinese customer has 100+ servers but these guys were buying 1 or 2 and that was it. And they were using a lot of bandwidth. Some 40-50Mbits but others were using 100-500Mbits constant. They quickly saturated our links and we started seeing packet loss and latency across the network.

Fortunately we maintain good relationships with our providers so we just added a circuit. and another, and another...

From February through March we had a ritual, I would call one of our providers on Wednesday and they would turn up another 10G on Friday. Every week. We were adding 10G of usage every week.

A couple weeks ago I called our provider on Wednesday but this time was told there was no bandwidth left for me. I had it all and will have to wait until June when they will have another 100G line brought in... That sucks. We added a couple lines from our other provider but their provisioning times are much longer. (like 20-30 days)

So we decided to look inward to see what all this bandwidth was about. We checked our flow data again, sniffed a couple ports... everything looked legit but the way it was working almost smelled like a botnet. 500-1000 servers all from different customers firing up at the same time... again, the traffic, as far as we could tell, was all legit.

The problem for us was we were outgrowing our circuits faster than we could add them and then... que dramatic music... people couldn't pay their overage bills. 100TB on a server that comes with 20TB is $1600.

As fast as those people would get canceled because they owed us $1600 on their $25 server new people would take their place so then churn started to climb as well.

So, again, since we do a lot of business with China, I have some friends in the DC business over there. I reached out to one of them last week to see if he knew anything about what I was seeing.

He did.

If you guys remember the old SETI@Home project or some of the newer folding projects, what they do is use excess CPU cycles when you're not using your system to do their work. Essentially what a couple Chinese companies have come up with is the same concept in a CDN network that uses excess bandwidth. So if you've got a gig port but you're only using 10Mbits it will try to use the other 990Mbits for it's CDN traffic and for this, they promise to pay you. The problem is, from what my friend tells me, it's a scam. Half the people get paid less than they thought and the other half don't see anything.

Apparently the DCs in China got hit with this the last half of last year. All of them have banned it now so the people are looking to the US to continue.

We've figured out, for the most part, how to detect the practice and are rate limiting our customers doing it so they don't incur overages. We've also changed our TOS to ban the practice in our DC. We've been able to bring our traffic levels back down and resolve our saturation issues.

Like I said at the start. I normally wouldn't go into this detail on an issue we've seen but I felt it was worthwhile to describe the problem so whatever DC is next, and they're all going somewhere, might be a little more prepared than we were.

«1

Comments

  • wychwych Member

    Interesting...

  • And thats why you don't allow overages unless they are prepaid. Like with prepaid mobile phones or mobile data plans, once you've used up your included minutes/bandwidth - top up.

    No-one wants to get the end of the month and find they have a $1000 overage bill.

  • Yes I read that yesterday. Well concept looks like bitcoin mining they use CPU +GPU power.Here it is uses Uplink power.

  • earlearl Member
    edited April 2014

    @instatech said:
    Yes I read that yesterday. Well concept looks like bitcoin mining they use CPU +GPU power.Here it is uses Uplink power.

    Someone on WHT mention it was called "LIULIANGKUANG" similar to Bitcloud where they mine for bandwidth, but LIULIANGKUANG is corporate owned and used specifically to stream their media.

    Here is a bit more about LIULIANGKUANG :

    https://bitcointalk.org/index.php?topic=415100.msg4658487#msg4658487

  • @earl said:

    Seen that LIULIANGKUANG on RamNode's AUP and was wondering what the hell it was, guess that clears that up.

  • earlearl Member

    @MarkTurner said:
    And thats why you don't allow overages unless they are prepaid. Like with prepaid mobile phones or mobile data plans, once you've used up your included minutes/bandwidth - top up.

    No-one wants to get the end of the month and find they have a $1000 overage bill.

    So Mark is this the default for all the servers from delimitervps?

    So when your monthly bandwidth is consumed you could suspend the server till next month if you don't wish to pay for overages?

  • wychwych Member

    @earl said:
    So when your monthly bandwidth is consumed you could suspend the server till next month if you don't wish to pay for overages?

    A few seem to drop BW to like 5mbps. Send them an email or check the ToS/AuP for details for a specific provider.

    Thanked by 1earl
  • @earl said:
    So when your monthly bandwidth is consumed you could suspend the server till next month if you don't wish to pay for overages?

    I've gone over, I'm ashamed to say. As far as I can tell, you don't get charged unless its an 500-1TB overage. Never gone over that so don't quote me on that.

    Thanked by 1earl
  • @Earl - of course, thats why there is a clearly stated included amount of transit. There is none of this 'unlimited' double-talk or end of month sting for excess bandwidth.

    If a customer wants to have more bandwidth then they can buy it upfront or as they need it, there is no risk for the customer suddenly being landed with an unplanned massive bill.

    About three years ago Delimiter use to let people run past their included transit and then backbill them for excess, but customers were far from happy because they were not expecting it.

    Thanked by 1earl
  • earlearl Member

    @wych said:
    A few seem to drop BW to like 5mbps. Send them an email or check the ToS/AuP for details for a specific provider.

    Ah ok that would make sense.. similar to OVH I guess.

    I tried to find this on their TOS, but could not find anything on it maybe I missed it.. anyways Mark is pretty active here so I will wait for his response.

  • 0xDragon - its not setup to be punitive, most of the time you'll end up with an email saying you're over your inclusive bandwidth, you need to cut back or buy some bandwidth. It all depends on how much over it is :)

  • earlearl Member

    @MarkTurner said:
    0xDragon - its not setup to be punitive, most of the time you'll end up with an email saying you're over your inclusive bandwidth, you need to cut back or buy some bandwidth. It all depends on how much over it is :)

    So do you suspend or just rate limit the port?

  • @Earl when the bandwidth runs out the port is shutdown. But in most cases where the customer is just a few % over their quota and its near month end they are left to it.

    Thanked by 1earl
  • earlearl Member

    Somewhat odd thought, my server with DS is still getting packet loss but my server with WSI has been fine all along, then again its not the $25 server.. wonder if they put the cheapo servers in a separate DC.

  • We too see that same thing from chinese. Most of that also used server to scan network, try floods etc. We was forced to stop orders from that country. Verify chinese orders is quite difficult and most of them take server for 1-2 month then leave it.

  • rm_rm_ IPv6 Advocate, Veteran
    edited April 2014

    said: people would get canceled because they owed us $1600 on their $25 server new people would take their place so then churn started to climb

    Nothing to see there, just one provider publicly confessing what an enormous-scale idiots they actually are.

  • NingNing Member
    edited April 2014

    @earl said:

    Yes, it means BW mining, mainly used for media steaming( p2p torrent) ,the company have to shut down its servers recently because of copyright enforcement by authorities this month.

  • rm_ said: Nothing to see there, just one provider publicly confessing what an enormous-scale idiots they actually are.

    and the fact they where spying on traffic.

  • rm_rm_ IPv6 Advocate, Veteran
    edited April 2014

    TarZZ92 said: and the fact they where spying on traffic.

    Nothing wrong with checking out the nature of traffic if you see abnormal usage.

    But the fact that they sheepishly kept "adding 10G after 10G every week", thinking they are going to collect those 1600 USD overage fees on a $25 server rented by someone in China, that's... that's... I don't even

    Thanked by 1c0y
  • @rm_ said:

    That's the bit that gets me. Keep adding 10G links without making sure you're getting paid for the ones already in place.

    Thanked by 1daxterfellowes
  • smansman Member
    edited April 2014

    @Virtovo said:

    ...buh...but I thought bandwidth was free. It just falls out of the sky like rain. I mean how else can people sell $1/month VPS's with unlimited bandwidth....because they get it for free of course.

    Just need to keep emailing their data center to up their limits is all. The data center just keeps emailing their provider, that provider just keeps emailing their provider and eventually the flying spaghetti monster steps in and gives his hair pasta angels more bandwidth to spread around.

  • @sman said:

    Really? I best get talking to my upstreams and get that flying spaghetti monster invoked.

  • MunMun Member

    Hmm, maybe I should point these users to GVH XD

  • MicrolinuxMicrolinux Member
    edited April 2014

    @TarZZ92 said:
    and the fact they where spying on traffic.

    Any responsible network operator is going to collect flow data, it's moronic not to. When a sane person sees unusual flow data long with network problems, they are going to then investigate.

  • Microlinux said: Any responsible network operator is going to collect flow data, it's moronic not to. When a sane person sees unusual flow data long with network problems, they are going to then investigate.

    I Don't. simple policy of bursting ports if one uses more than say 100Mbps for a large amount of time then that server is placed on a cap.

  • MicrolinuxMicrolinux Member
    edited April 2014

    @TarZZ92 said:
    I Don't.

    Sometimes I feel its amazing the Internet manages to operate as well as it does.

    If you're talking about a commercial network, it's a little scary you have no idea what is going on in your network other than port speeds.

    Thanked by 3Cakey Wintereise Dylan
  • TarZZ92 said: I Don't. simple policy of bursting ports if one uses more than say 100Mbps for a large amount of time then that server is placed on a cap.

    Are you even in a position to do the collecting, i.e: have you ever actually run a properly vetted network?

    If not, I feel this discussion is moronic and you probably have no idea what you're talking about.

    It is absolutely crucial to know exactly what is flowing through your network at what time to actually keep it up, you need to do this to know when to step in, and when not to.

    "It's their port, they can do whatever they want" does not work in 2014, and if this is your policy, I highly recommend reconsidering.

  • Microlinux said: If you're talking about a commercial network, it's a little scary you have no idea what is going on in your network other than port speeds.

    It's not my business to see what customers are doing. if they are breaking the rules ( and we get a abuse notification action would be taken with the exception of DMCA notices which we ignore)

  • MicrolinuxMicrolinux Member
    edited April 2014

    @TarZZ92 said:
    It's not my business to see what customers are doing. if they are breaking the rules

    Incredible. Because it's best to sit clueless with your head stuck in the sand until someone else notifies you.

    Good god, when generation Y completely takes over operation of the Internet we're all screwed. All network admins will do is tell each other it's not their problem.

    Can't tell if really that clueless, or troll.

  • I heard a story about a company that didn't collect flow data. One month they accrued $1500 in overages, sadly they didn't have anything to inspect to figure out why.

    The second month they again accrued $1500 in overages, their supplier let them defer payment and advised them on how to collect flow data.

    The third month they accrued $2000 in overages and had failed to heed the advice of their supplier.

    They weren't around anymore for the fourth month.

Sign In or Register to comment.