Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Providers, how common is node abuse by consumers today and what do you do to combat it? - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Providers, how common is node abuse by consumers today and what do you do to combat it?

24

Comments

  • AnthonySmithAnthonySmith Member, Patron Provider

    @KuJoe said: We would like to do this and some day

    Should be fairly simple with some if's and then's if you want to chuck over what You have got I will add that, it sounds like you have a tight run ship, and this gave me an idea, would be fairly simple to run this and instead of vzctrl shut down it could do a count and check again in xx minutes after an email has been sent.

    Might have a look to see what I can do with Xen in this regard too.

  • AnthonySmithAnthonySmith Member, Patron Provider

    @spycrab101 I think what he meant was that it could potentially kill a node process rather than a container process, obviously some trepidation is needed.

  • Vzctl enter $ctid
    Kill -9 $process
    Exit.

  • AnthonySmithAnthonySmith Member, Patron Provider
    edited December 2012

    you can pass variables from a script on the host in to a container after enter?

  • @spycrab101 I think what he meant was that it could potentially kill a node process rather than a container process, obviously some trepidation is needed.

    So killing a node process would just kill the VPS? and not the node?

  • Hmm that I didn't think of. Well it is the thought that counts :P

  • AnthonySmithAnthonySmith Member, Patron Provider

    Well I have not seen the script but what I suspect from what has been said it that the script runs based on process loads and that could potentially kill a process vital to the host, obviously if you issue a vzctl shutdown to a process id it will just do nothing which makes it safe.

    I am guessing here :)

  • Also I'm interested to know if openvz can be setup to email a client when an admin is attempting to run commands on their VPS.

  • So how is restarting someones VPS better than killing a process on their VPS on the off chance it may be a node process?

  • KuJoeKuJoe Member, Host Rep
    edited December 2012

    @spycrab101 @Taz
    So instead of using my 10 lines of code that works perfectly well, I am expected to code a completely new program that will track down the process ID, determine the vzid, confirm it is in fact a process being run by a OpenVZ container, and then hope that vzctl enter will work before killing the process?

    vzctl enter $ctid

    enter into CT $ctid failed
    Unable to fork: Cannot allocate memory

    All of a sudden you're running kill commands as root on the host node with the assumption that the process ID is correct.

    No thanks. If you would like to share the scripts you running on your production VPS nodes with hundreds of clients I'll be interested in taking a look at them but for our production environment we will always play it safe and err on the side of caution when it comes to node stability and our clients.

  • I agree with you
    My shit was well just shit. Ignore it :)

  • AnthonySmithAnthonySmith Member, Patron Provider

    because if your scripts action is 'vzctl shutdown' it will just shut down a container, if your scripts action is 'kill' and it works on process id's rather than container id's it has the potential to kill a process on the node instead of on a container.

  • KuJoeKuJoe Member, Host Rep
    edited December 2012

    @spycrab101 said: So how is restarting someones VPS better than killing a process on their VPS on the off chance it may be a node process?

    The script is based off of the VPS load, not a single process. If the VPS has 300 processes using 1% of CPU then the VPS as a whole is using 300% CPU but the processes would not be noticed in an automated script because they are using only 1% each.

    @spycrab101 said: Also I'm interested to know if openvz can be setup to email a client when an admin is attempting to run commands on their VPS.

    Where would OpenVZ get the e-mails? No client details are stored anywhere on the server.

  • AnthonySmithAnthonySmith Member, Patron Provider
    edited December 2012

    @spycrab101 said: Also I'm interested to know if openvz can be setup to email a client when an admin is attempting to run commands on their VPS.

    probably.

    @KuJoe I think he meant an anti-snooping alarm that he could set up himself if the admin ever enters the container unannounced :)

  • MaouniqueMaounique Host Rep, Veteran
    edited December 2012
    1. Abuse happens.
    2. Abuse tends to get lower in time as the troublesome customers leave either kicked or throttled to death and look for another victim.
    3. Abuse will never go away, will just go down in percent.
    4. Abuse is of 2 kinds, resources and ToS breaching, for resources an agreement can be reached in almost all cases, for ToS violations it is usually the door, unless old and serious customer where hacking story might hold some water, but usually if the customer gets hacked once, will probably get hacked again...

    how often ?
    well, a few days ago had to warn 3-4 people and shut down (not suspend) 2 other VPSes, all in one night. Usually, during my watch of about 14 hours, nothing like that happens. Say an average of 1 incident per day.
    On the other hand our tolerance for resource abuse is quite high, with SSD and sas2 big arrays and some of them ssd cached, there are many IOPS to spare usually, even if this means lower space on plans.
    Pure CPU abuse happens rarely with ppl running grid computing, bit coin miners we nice them, poor scripts get looked into together with the owner and usually can be solved, IO is not taking cpu (hardly over 0.1-0.2 % at peaks except backups) so CPU stays below 30% and has plenty of room to spare.
    Network abuse is also rare, some ppl gobble 10 tb+ a month but the 1 gbps port on the node can usually cope with it unless some nasty spikes that at times go up to 800 mbps amounting to small DDoSes. Less than 1 a week at most and is usually a torrenter with a bad setting.
    The only real problem is with ppl running scripts on desktops with firefox and flash to increase views for youtube and sell/trade them. They do that in 128-192 mb ram and it is hammering the CPU. That is the main reason we dont give very small VPSes to everyone, just to proven customers or as they become available from the older batches. That and the IP prices.
    There is no real cure for that other than kicking the customer and we dont do that, they just leave by themselves after the extra-special deals they got expire.

    The other kind of abuse, ToS violation, well, that is another matter. Regularly checking for blacklisting and there is no recourse to that. Suspension occurs, not only for the blacklisted IP, but also for all the VPSes of the customer and there is no going back, not even files backup so the spammer cant get his harvested CCs or emails from crawlers, whatever.
    DMCAs come for "replica" shops usually, Uncle even had to answer in person in front of stupid cops that dont know that having a shop on your ip space does not mean that you are operating it, especially when your business is hosting. Also hosting links to tv shows or something.
    Most other ToS breaching means suspension till expiring. Especially if the service hosted was a forbidden DDoS magnet on that plan and DDoS happened.
    For DDoSes happening where the DDoS magnet is allowed, at first strike we nullroute till we get a reply and second time we ask the customer to prepare the movement because we do not offer DDoS protection of any kind.
    Outgoing DDoS mean suspension without ifs and buts, all other VPSes he may have too. Just a notice for the reason.

    I have found out that friendly warnings work out well, ppl know we are watching and they are more careful in the future. Explaining how they might hurt other neighbors with charts and details is working much better than threats or suspension.
    At most, first time a shutdown if reboot is not working. Second time the same, if it continues to happen, we must part ways because the customer is either a cheap guy which does not want to purchase a dedi or a reasonably sized VM or has no idea how to manage and needs a managed service.

    EDIT: Windows 2008RC2 Server does not seem to be a magnet for abusers, at least we had only one case which was running crawlers to harvest emails.

  • @KuJoe

    It seems like you are overly paranoid? That's the feeling I got when I had to deal with you taking down my VPS. Maybe someone who knows more than I do on this can tell me if KuJoe says is a valid reason for restarting someones vps? I'm thinking restarting someones VPS should be the last resort. It's like someone deleting a big document you've worked on for hours and asking you to start again.

  • KuJoeKuJoe Member, Host Rep

    @spycrab101 I openly accept any provider who can offer a better solution that is as safe and effective as my script. :)

  • @Maounique said: Abuse tends to get lower in time as the troublesome customers leave either kicked or throttled to death and look for another victim.

    no.

  • AnthonySmithAnthonySmith Member, Patron Provider

    @spycrab101 said: It seems like you are overly paranoid? That's the feeling I got when I had to deal with you taking down my VPS. Maybe someone who knows more than I do on this can tell me if KuJoe says is a valid reason for restarting someones vps? I'm thinking restarting someones VPS should be the last resort. It's like someone deleting a big document you've worked on for hours and asking you to start again.

    what he is saying is totally valid, no offence (And I really mean that) you clearly have no idea when it comes to managing a VPS node from a hosts perspective.

    If a single VPS is causing issues for other customers or compromises stability then it needs to be stopped fast, not 24 hours later when you pick up your email and then start an argument about how you don't agree etc by which time the host has had to deal with 20 other tickets complaining about performance..

    I do agree with you however that some notice could be scripted e.g. 10 - 15 minutes but those things do not currently exist without further development.

    your document analogy is not really valid, think of it more like driving at 30 mph on a single lane road with a speed limit of 70 mph and leaving a tailback of traffic behind you.

  • @spycrab101 said: I'm thinking restarting someones VPS should be the last resort. It's like someone deleting a big document you've worked on for hours and asking you to start again.

    How do the two compare? restarting a VPS, if it is setup even half-assed, will restart all the services on boot, your document is not even similar.

  • Never understood, why Firefox on vps if you can do the same with vpn?

  • MaouniqueMaounique Host Rep, Veteran

    @Maounique said: The only real problem is with ppl running scripts on desktops with firefox and flash to increase views for youtube and sell/trade them.

    @Taz said: Never understood, why Firefox on vps if you can do the same with vpn?

    You cant do that for more VPSes at once and why use all that bw at home and pay electricity if you can pay an Eur a month VPS ?

  • @AnthonySmith

    I'm not offended :) I don't have the slightest idea on how openvz works from a host perspective. I would like to know what is defined as "safe" ? Why can't a process be stopped and the client emailed? Why is this so complex? Is this an openvz problem?

  • @spycrab101 said: Why can't a process be stopped and the client emailed? Why is this so complex? Is this an openvz problem?

    No client data is on the node, @KuJoe already said he wrote a 10 line script, your solution is 10 tons of work, maybe more, maybe less, but work to fix a problem that doesn't exist, a 10 line script already does it well.

  • joepie91joepie91 Member, Patron Provider

    @Taz said: Vzctl enter $ctid

    Kill -9 $process
    Exit.

    I think you mean vzctl exec.

    As for automatically killing processes, this can in fact be a problem and lead to nasty situations (race conditions, data loss, unreasonable resource consumption by other processes, etc) if another process depends on the killed process and does not properly handle a sudden termination.

  • @miTgiB

    Are you kidding me? If someone restarted my VPS right now there would be problems, simply because I've not had the chance to fix them yet. If you are doing production stuff your aren't going to be setting things up perfectly until what you are working on is complete.

  • Are all these "nasty situations" openvz related?

  • KuJoeKuJoe Member, Host Rep

    @spycrab101 said: Are you kidding me?

    As a service provider we have to put our clients first. If 1 client is impacting 100 clients then that 1 client gets the short end of the stick. We can't realistically ask 100 clients to deal with the actions of 1 client and expect them to remain clients with us.

    While every client is important and special to us, if that 1 client is impacting 100 equally important and special clients then we have little say in the matter. 1 upset client is 100 times better than 100 upset clients. There is no question about this. Any company who is willing to upset 25% of their clients so that less than 1% is happy will not be in business very long and is not a company I would want to be a client of.

  • KuJoeKuJoe Member, Host Rep
    edited December 2012

    @spycrab101 said: Are all these "nasty situations" openvz related?

    It's worse in KVM and Xen because the ONLY option you have with Xen and KVM is to forcefully reboot the VPSs (i.e. pull the power with possible data loss/corruption), at least with OpenVZ you can do a graceful restart.

  • @Kujoe how about virsh suspend? we use that quite well for containers that are trashing disk + cpu.

Sign In or Register to comment.