Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Long outage at Racknerd LA DC02
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Long outage at Racknerd LA DC02

ralfralf Member
edited July 2022 in Outages

There was a quarter of an outage that affected LA a couple of weeks ago, and my server in LA has just been inaccessible for about an hour and a half. Although as I was writing this, it just came back online.

Any idea what's going on here @dustinc? There was nothing about it on the status page.

Comments

  • Your outage was doubled

  • ArkasArkas Moderator

    And what may I pray ask makes you think that posting this here will solve this?

    Thanked by 1JasonM
  • mmuyskensmmuyskens Member, Host Rep

    this end is.....

  • henixhenix Member

    Go to virmach

  • NeoonNeoon Community Contributor, Veteran

    @Arkas said:
    And what may I pray ask makes you think that posting this here will solve this?

    If he posts the Ticket number, he gets his bandwidth doubled again?

    Thanked by 1WSWD
  • ralfralf Member

    Well, it's already "solved" in that connectivity has been restored. But maybe @dustinc might know more about the outage than anyone else, especially it was quite long but not picked up by their status page.

  • ralfralf Member

    @Neoon said:

    @Arkas said:
    And what may I pray ask makes you think that posting this here will solve this?

    If he posts the Ticket number, he gets his bandwidth doubled again?

    Double of 0? :p

    Thanked by 1WSWD
  • NeoonNeoon Community Contributor, Veteran

    @ralf said:

    @Neoon said:

    @Arkas said:
    And what may I pray ask makes you think that posting this here will solve this?

    If he posts the Ticket number, he gets his bandwidth doubled again?

    Double of 0? :p

    Given x is his bandwidth and usually its higher than 0 except @dustinc fucked up, it should be doubled.

    Who says there is a limit on doubling, its infinite.

  • sotssots Member
    edited July 2022

    [deleted]

  • @henix said:
    Go to virmach

    Go to HostHatch

    Thanked by 2rogerwilco Azenot
  • dustincdustinc Member, Patron Provider, Top Host
    edited July 2022

    @ralf said:
    There was a quarter of an outage that affected LA a couple of weeks ago, and my server in LA has just been inaccessible for about an hour and a half. Although as I was writing this, it just came back online.

    Any idea what's going on here @dustinc? There was nothing about it on the status page.

    Hi @ralf -- appreciate you reaching out.

    We prefer tickets through our helpdesk as our response times are quite fast, considering our availability is truly 24x7 you'll often times experience responses/solutions within a few minutes.

    This does sound like it could be an isolated incident (i.e. could be caused by a configuration issue within the OS level or a firewall set up within the OS such as iptables rules or CSF). Without looking at your account specifically, it’ll be hard to say at this point in time.

    Feel free to e-mail me at [email protected] with the VPS IP in question — I will then verify if we can see any visible issues with that VPS IP from our end, and if not will assist you further with troubleshooting this from your VM’s OS level (as long as you’d provide us with temporary root credentials).

    Thank You for doing business with RackNerd and for being our valued customer.

  • LoganoLogano Member
    edited July 2022

    Yeah, for Racknerd, I'd definitely try sending a ticket first. I got myself IP-banned from WHM and cPanel a couple of days ago (was fooling around with WebDAV), asked for whitelisting, and it was done and replied to in 4 minutes.

    I also haven't noticed any lengthy downtime on my 3 VPS and 1 reseller at DC02 since getting them last November. I am monitoring just the 1 reseller and 1 VPS, though.

  • ralfralf Member

    It wasn't my VPS specifically - hetrixtools showed the machine inaccessible from 3 different locations in the US and from Europe, so obviously just a network issue. I just wondered as last time there was an outage, there was a very clear problem with the upstream that Dustin explained about here. And this was for quite a long time, so I figured I should flag it in case it was something serious.

    I guess mostly it's noticeable because my connectivity to my RN machines hardly ever goes down.

    Thanked by 1Logano
  • Yep.
    Out of my 3 boxes at LA DC02, 2 of them were down for about a hour and half. But I have nothing critical there. Also these are free VPS won last BF, don't want to complain or ticket anyway.

    Thanked by 1Logano
  • @ralf said:
    There was a quarter of an outage that affected LA a couple of weeks ago, and my server in LA has just been inaccessible for about an hour and a half.

    I have a VPS in LA DC02 and I didn't even realize it.

  • dustincdustinc Member, Patron Provider, Top Host

    @ralf said:
    It wasn't my VPS specifically - hetrixtools showed the machine inaccessible from 3 different locations in the US and from Europe, so obviously just a network issue. I just wondered as last time there was an outage, there was a very clear problem with the upstream that Dustin explained about here. And this was for quite a long time, so I figured I should flag it in case it was something serious.

    I guess mostly it's noticeable because my connectivity to my RN machines hardly ever goes down.

    Hi @ralf -- when you mention it wasn't your VPS specifically, what do you mean by that? I assume on Hetrix, you are monitoring your VPS IP address itself, right?

    We have not had any widespread outages in Los Angeles other than the incident you speak of from weeks ago (which was upstream and has since been fully resolved) - which leads me to believe this is an isolated issue. Either way, happy to look into this for you if you can send me an e-mail with your VPS IP - [email protected]. From there, I'll double-check to see if we can see any visible issues with that IP from our end, and if not we will go from there and assist you with the next troubleshooting steps.

    Thanked by 1ralf
  • dustincdustinc Member, Patron Provider, Top Host

    @rattlecattle said:
    Yep.
    Out of my 3 boxes at LA DC02, 2 of them were down for about a hour and half. But I have nothing critical there. Also these are free VPS won last BF, don't want to complain or ticket anyway.

    Hi @rattlecattle -- Thank You for being our customer :)

    Was this as of yesterday, or? It sounds like you're referring to the outage from weeks ago (which was already discussed here previously and on our status page), but just wanted to make sure as we haven't experienced any recent widespread outages after that one.

  • dustincdustinc Member, Patron Provider, Top Host

    @Carlin0 said:
    I have a VPS in LA DC02 and I didn't even realize it.

    Hi @Carlin0 -- Glad to hear your service is working seamless :) We aim towards to providing service that our customers can rely on. And if/when outages occur, it is shared with full transparency on our status page.

    We hope that you expand with RackNerd, as your operation/needs grow 👊

  • tuctuc Member

    @ralf said:
    There was a quarter of an outage that affected LA a couple of weeks ago, and my server in LA has just been inaccessible for about an hour and a half. Although as I was writing this, it just came back online.

    Any idea what's going on here @dustinc? There was nothing about it on the status page.

    By posting here you get doubled your time for the response.

    Thanked by 1ralf
  • JasonMJasonM Member
    edited July 2022

    @dustinc said: And if/when outages occur, it is shared with full transparency on our status page.

    this what i like with Racknerd. Very few hosts put up a notification on status page. It saves time of customers wondering and logging into the VMs.

    Thanked by 1Logano
  • rattlecattlerattlecattle Member
    edited July 2022

    @dustinc said: Was this as of yesterday, or? It sounds like you're referring to the outage from weeks ago (which was already discussed here previously and on our status page), but just wanted to make sure as we haven't experienced any recent widespread outages after that one.

    Yes this was on 9th of July.

    HetrixTools reports a downtime of 1hr 40 mins.

    Monitor: RackNerd-LA3
    Target: xx.xx.xx.xx[ping]
    Downtime: 1 hr 40 min
    Noticed at: 2022-07-09 21:50:38 (UTC+00:00)
    

    Same for both VPS at LA DC02.

    The actual downtime is quite likely less than that as the ping monitors are set at 10min interval.

  • dustincdustinc Member, Patron Provider, Top Host
    edited July 2022

    @rattlecattle said:

    @dustinc said: Was this as of yesterday, or? It sounds like you're referring to the outage from weeks ago (which was already discussed here previously and on our status page), but just wanted to make sure as we haven't experienced any recent widespread outages after that one.

    Yes this was on 9th of July.

    HetrixTools reports a downtime of 1hr 40 mins.

    Monitor: RackNerd-LA3
    Target: xx.xx.xx.xx[ping]
    Downtime: 1 hr 40 min
    Noticed at: 2022-07-09 21:50:38 (UTC+00:00)
    

    Hi @rattlecattle -- the good news is that we utilize Hetrix as well (in addition to our internal monitoring scripts) for the monitoring of our primary hypervisors (we don't monitor individual VM's, but we do monitor primary host nodes). This will make comparing stats simple and transparent between your VM and the host node. Please also make sure you've added HetrixTools' monitoring probe IPs to your firewall allow list as well, if you are running a firewall (to avoid false positives): https://docs.hetrixtools.com/uptime-monitoring-ip-addresses/

    Feel free to e-mail me at [email protected] with your VPS IP so I can review what node you're on, and once we identify that, I'll be happy to compare/share the stats we see from Hetrix from the host node level as well. We’ve extended these reports with @ralf as well as he has reached out via email (at this time, that issue does appear to be isolated/unrelated to the host node itself, could have been be a firewall or something within the OS etc, but we're continuing to monitor and keeping a close eye nonetheless).

  • ralfralf Member
    edited July 2022

    Thanks @dustinc for looking into this. I've sent him some more details, but interestingly looking at my logs it's clear during the period that hetrixtools was reporting an outage to me (from every location), I was still getting a periodic ssh scan (about once every 5 minutes, about the same as normal), so it's clear I definitely had at least some, if not full, connectivity at this time, so it's entirely possible the problem was with hetrix not networking connectivity in general from RN. Although also interestingly, I had some logs about NTP connection errors in that time period, which I never normally see, but also some after I managed to log into the VM to check it. So who knows. By the time I noticed the hetrix report, I could log into the server fine, although hetrix was already reporting it as up again so that doesn't prove much.

    Anyway, props to @dustinc for looking into it, especially after I'd said it's not a machine as I use much, so it wasn't really a big deal.

    EDIT: I just realised too, especially on a Sunday when he should be relaxing!

    Thanked by 1Logano
  • dustincdustinc Member, Patron Provider, Top Host

    @ralf said:
    Thanks @dustinc for looking into this. I've sent him some more details, but interestingly looking at my logs it's clear during the period that hetrixtools was reporting an outage to me (from every location), I was still getting a periodic ssh scan (about once every 5 minutes, about the same as normal), so it's clear I definitely had at least some, if not full, connectivity at this time, so it's entirely possible the problem was with hetrix not networking connectivity in general from RN. Although also interestingly, I had some logs about NTP connection errors in that time period, which I never normally see, but also some after I managed to log into the VM to check it. So who knows. By the time I noticed the hetrix report, I could log into the server fine, although hetrix was already reporting it as up again so that doesn't prove much.

    Anyway, props to @dustinc for looking into it, especially after I'd said it's not a machine as I use much, so it wasn't really a big deal.

    EDIT: I just realised too, especially on a Sunday when he should be relaxing!

    Thank You for recognizing that @ralf

    We're an entire team here committed to providing competitively priced products, along with support that our customers can count on, especially in the times most critical. Rest assured we'll always be here to help.

    Thanked by 2Logano ralf
  • sangatsetiasangatsetia Member
    edited July 2022

    but calm, an advantage RackNerd fast response that I feel. even when I had a problem with LA, asking to move to SJ didn't wait long, a matter of minutes can be used on a new VPS. they respond to customers with the same portion, do not differentiate between low price customers or small customers, always respond quickly. succes RackNerd @dustinc keep improving to stay the best..
    ** edit I hope there will be a new server added for LA (eg LA DC-06, LA DC-07 and so on hehehehe)

    Thanked by 2suut ralf
  • suutsuut Member

    @sangatsetia said:
    but calm, an advantage RackNerd fast response that I feel. even when I had a problem with LA, asking to move to SJ didn't wait long, a matter of minutes can be used on a new VPS. they respond to customers with the same portion, do not differentiate between low price customers or small customers, always respond quickly. succes RackNerd @dustinc keep improving to stay the best..
    ** edit I hope there will be a new server added for LA (eg LA DC-06, LA DC-07 and so on hehehehe)

    Tokyo DC-01 <3

    Thanked by 1sangatsetia
  • VoidVoid Member

    @Arkas said:
    And what may I pray ask makes you think that posting this here will solve this?

    It probably won’t solve anything but a chance to complain about a provider, who here would miss that?

Sign In or Register to comment.