Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Longest outage you experienced? - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Longest outage you experienced?

2

Comments

  • @dallascao said:

    @hassancent said:

    @rcy026 said:

    @hassancent said:
    Racknerd Strasbourg still down. And im on their yearly. I hope they offer any refunds. Its been barely a month. Would move in a heart beat. My start up is greatly affect by racknerd

    If your company is greatly affected by downtime, you should not rely on a single lowend provider.

    Not really a company. Just started a start up. It was getting popular. I was gona buy better servers if it reached > 20k visitors daily. But downtime made me lose out on customers

    Again it's more about redundancy and fail-over than better servers. Always have a fail-over in place no matter how "good" or expensive the provider is.

    How do you even achieve this? Having two servers with same files+config? so that we can manually change dns record in case on fails?
    Or maybe its something that can be fixed using load balancers?

  • Yes. You don't have to manually change DNS though. You can write a monitor script and change the DNS settings through name servers API.

  • You can also use smart DNS services like "Azure Traffic Manager" to achieve auto fail-over.

    How do you even achieve this? Having two servers with same files+config? so that we can manually change dns record in case on fails?
    Or maybe its something that can be fixed using load balancers?

  • uekixuekix Member

    123system, almost a decade

  • rcy026rcy026 Member

    @hassancent said:

    @dallascao said:

    @hassancent said:

    @rcy026 said:

    @hassancent said:
    Racknerd Strasbourg still down. And im on their yearly. I hope they offer any refunds. Its been barely a month. Would move in a heart beat. My start up is greatly affect by racknerd

    If your company is greatly affected by downtime, you should not rely on a single lowend provider.

    Not really a company. Just started a start up. It was getting popular. I was gona buy better servers if it reached > 20k visitors daily. But downtime made me lose out on customers

    Again it's more about redundancy and fail-over than better servers. Always have a fail-over in place no matter how "good" or expensive the provider is.

    How do you even achieve this? Having two servers with same files+config? so that we can manually change dns record in case on fails?
    Or maybe its something that can be fixed using load balancers?

    It depends on what kind of server it is of course, but yes, dns and loadbalancers are common ways of dealing with it.

  • dallascaodallascao Member
    edited March 23

    Do both. And you don't have to run load balancers like haproxy or maxscale on a separated vps. Install it on each of your production server and put your all productions under each. And then point your DNS to each haproxy.

    It depends on what kind of server it is of course, but yes, dns and loadbalancers are common ways of dealing with it.

  • carp969carp969 Member

    Poor Conceptboard. It looks like the dns for their eu side are all hosted in that dc.

    https://conceptboard.statuspal.eu/incidents/164060

  • wii747wii747 Member

    When OVH DC burnt down

    Thanked by 1yoursunny
  • techdragontechdragon Member
    edited March 24

    @raza19 said: leasewe

    @jtk said:

    @dallascao said:
    Racknerd's Strasbourg servers are still down after 19 hours:-)

    19 hours is nothing. I see systems go unavailable for longer than that regularly. The two most obnoxious, unrelated extended outage events I can recall were more than 3 months. One from a former a provider that no longer visits these forums and the other from a now defunct provider that was fairly well known for providing KR hosting.

    Wrong. When you are selling three nines or above, it is unacceptable. 99.9% equates to 8 hours and 46 minutes downtime a year. Five nines is no more than five and a half minutes downtime a year.

    I think LowEndHosts really do exaggerate their stability and availability which is often overlooked. Claims are made that they are on par with HA providers, some even going as far as to say they're cheaper. A lack of SLAs and it's clear why certain customers would never touch them.

  • Sounds like my racknerd service is back since 37"
    Better weekend ever :p
    And not one customer has complained ... Surprised I am ;)

  • tdwuktdwuk Member
    edited March 25

    Racknerd France 100 hours of downtime - longest I can recall for me with any host :neutral:
    https://app.screencast.com/ZIdr6080g9W1C

  • DataWagonDataWagon Member, Patron Provider

    One time our upstream rebooted their router for scheduled maintenance, we lost $300,000 during the 3 minute outage.

  • MrEdMrEd Member

    @wii747 said:
    When OVH DC burnt down

    This. I thought it would be mentioned much earlier in this thread :)

  • dustincdustinc Member, Patron Provider, Top Host

    @tdwuk said:
    Racknerd France 100 hours of downtime - longest I can recall for me with any host :neutral:
    https://app.screencast.com/ZIdr6080g9W1C

    Hi @tdwuk -- Most clients should have seen affected services come back online throughout the past 24 hours, and as of a few hours ago, everything in Strasbourg should be 100% back online from an infrastructure standpoint.

    To recap and out of transparency, this was a larger facility-wide issue at the DataDock SXB facility, which was beyond our immediate control. On the morning of March 21, GoDaddy's section on the first floor of DataDock SXB experienced a small fire in a battery-backup room, which triggered/caused a water leak. The fire was quickly contained, and all of RackNerd's equipment was free of water damage. To avoid additional damage and to comply within the limits of local authorities, power was then subsequently cut for the first floor, which impacted roughly ~30% of RackNerd's infrastructure (as the majority of our infra is on the second floor).

    We took proactive measures within our power to restore affected services as quickly as possible. Our datacenter partner has successfully relocated our affected equipment from the impacted 1st floor to the unaffected 2nd floor, avoiding the need to wait for the facility to resolve the issues with the 1st floor (they worked hard to move racks of equipment, not only for us but many others). Every step of the way we were committed to keeping our customers in the loop with our progress, as well as sharing frequent updates via the status incident page. Ultimately, this situation could of played out a lot worse but we're glad that it's all behind us now.

    At this point, we are showing everything to be back online, so if your service in France is still offline, it may be an isolated issue. If anyone is still facing issues please open a support ticket and we can investigate/troubleshoot accordingly :)

    Thanked by 1commercial
  • Did anyone get any refund? I was using the host for barely 1month+ and this happened. Moved my sites to hetzner. I asked the support for partial refund, but rejected with a response that they will compensate "later".

  • dustincdustinc Member, Patron Provider, Top Host

    @hassancent said:
    Did anyone get any refund? I was using the host for barely 1month+ and this happened. Moved my sites to hetzner. I asked the support for partial refund, but rejected with a response that they will compensate "later".

    Hi @hassancent - to add clarity by default RackNerd doesn't provide refunds, as outlined within our terms of service. So if you're requesting a full refund, our team is trained to deny these requests. Though we work closely with our customers to ensure we help solve outstanding technical issues they may be facing.

    From what you're commenting, it does sound like terrible timing (as a new customer of RackNerd) and the duration of an unexpected outage.

    We are most known for our solid service, backed by solid earned reputation for the service level we provide. This event was rough for everyone, and not something you should expect of RackNerd.

    We're still working with our upstream and gathering our side of things to issue SLA compensation along with the RFO.

    Thanked by 1wwwroots
  • @dustinc said:

    @hassancent said:
    Did anyone get any refund? I was using the host for barely 1month+ and this happened. Moved my sites to hetzner. I asked the support for partial refund, but rejected with a response that they will compensate "later".

    Hi @hassancent - to add clarity by default RackNerd doesn't provide refunds, as outlined within our terms of service. So if you're requesting a full refund, our team is trained to deny these requests. Though we work closely with our customers to ensure we help solve outstanding technical issues they may be facing.

    From what you're commenting, it does sound like terrible timing (as a new customer of RackNerd) and the duration of an unexpected outage.

    We are most known for our solid service, backed by solid earned reputation for the service level we provide. This event was rough for everyone, and not something you should expect of RackNerd.

    We're still working with our upstream and gathering our side of things to issue SLA compensation along with the RFO.

    Cool. Should have read t&c, that's my fault. btw i asked partial refund.
    Well I wouldn't use racknerd for even hobby projects after what happened. I was expecting max 8-10 hours of downtime, maybe once a month. This was way to much.
    I know this incident sounds like it might not have been in hands of racknerd, But seems there was no backup plan in mind if something like this happens.

  • @hassancent said:
    I know this incident sounds like it might not have been in hands of racknerd, But seems there was no backup plan in mind if something like this happens.

    Regardless of the provider, you are responsible for your own backup plan. Unless you're paying extra for management and an SLA it's always going to be 100% on you.

    Thanked by 1commercial
  • SplitIceSplitIce Member, Host Rep
    edited March 26

    @carp969 said:
    Poor Conceptboard. It looks like the dns for their eu side are all hosted in that dc.

    https://conceptboard.statuspal.eu/incidents/164060

    If you are doing anything important you should always host your DNS servers on two continents (or at-least east/west) and two different external (not you) providers.

    Its also one of the few things that even if you can, you shouldnt put on your own network. Or any single network.

  • @dahartigan said:

    @hassancent said:
    I know this incident sounds like it might not have been in hands of racknerd, But seems there was no backup plan in mind if something like this happens.

    Regardless of the provider, you are responsible for your own backup plan. Unless you're paying extra for management and an SLA it's always going to be 100% on you.

    Yes.
    All i meant was that this incident made racknerd not even worth being considered as backup server.

    Thanked by 1Morphmaster
  • rcy026rcy026 Member

    @carp969 said:
    Poor Conceptboard. It looks like the dns for their eu side are all hosted in that dc.

    https://conceptboard.statuspal.eu/incidents/164060

    When you claim to have companies like Amazon, Walmart, HP and FedEx as customers you do not host everything on a single Racknerd vps. No offence to Racknerd but they are a lowend provider, nobody with a brain would host something vital there without backup. Someone is either lying or is stupid as fuck!

    Also, spinning up a new dns somewhere else and point the domain to that should take 30 minutes, max. Instead they moved everything from .com to .eu, there is no way that could have been faster, cheaper or easier.

  • coldcold Member

    I'm pretty sure some still hope that @cociu will come back 🤣

  • Wenlee8Wenlee8 Member

    OVH SBG, my dedi even never come back :#

  • tdwuktdwuk Member
    edited March 27

    Service down again today for approx. 2hrs so far. Not ideal. I had to put in a cancellation request. It's not worth the headache. Luckily I only had one landing page company site on the vps. Moving that asap.
    Same Strasbourg datacentre, ironically I am paying extra to host there :D

    Thanked by 1commercial
  • @tdwuk said:
    Service down again today for approx. 2hrs so far. Not ideal. I had to put in a cancellation request. It's not worth the headache. Luckily I only had one landing page company site on the vps. Moving that asap.
    Same Strasbourg datacentre, ironically I am paying extra to host there :D

    All's fine for me now with Racknerd
    Here are the last data:

  • @cold said:
    I'm pretty sure some still hope that @cociu will come back 🤣

    Don't hope too much, after a ticket I had a refund of $8 for a $36 annual plan,
    I've learned how his business could be profitable now :->

  • @dustinc said:
    At this point, we are showing everything to be back online, so if your service in France is still offline, it may be an isolated issue. If anyone is still facing issues please open a support ticket and we can investigate/troubleshoot accordingly :)

    All's well that ends well, and you handled the situation very well. Nobody is immune from a data center incident, nobody.
    I'm delighted to renew my confidence in you.

    Thanked by 1dustinc
  • dustincdustinc Member, Patron Provider, Top Host

    @tdwuk said:
    Service down again today for approx. 2hrs so far. Not ideal. I had to put in a cancellation request. It's not worth the headache. Luckily I only had one landing page company site on the vps. Moving that asap.
    Same Strasbourg datacentre, ironically I am paying extra to host there :D

    Hi @tdwuk -- We sincerely apologize for any inconvenience. There was a routing issue that impacted several nodes in this location earlier this morning, which has since been resolved. As of now service in France should be restored. To recap everything, a more detailed email will be going out to all affected customers next week (including compensation details), once we receive an official RFO from our datacenter provider.

    While most customers should be back online, and everything is online from a power perspective, it's inevitable that some isolated issues may still occur after a broader outage incident like this -- our team is still working with customers to resolve any isolated cases.

    Although we fully expect our France datacenter operations to stabilize back to how it was pre-incident once these individual cases are finished being sorted out, we understand if some customers would have better peace of mind with a migration to another location all together. Therefore, we are providing our France-based clients the option to migrate their services free of charge to any of our other available VPS datacenter locations, including our other European location, Amsterdam. @tdwuk -- if interested, please submit a support ticket, and our team will assist you with the migration process accordingly.

  • dustincdustinc Member, Patron Provider, Top Host

    @commercial said:

    @dustinc said:
    At this point, we are showing everything to be back online, so if your service in France is still offline, it may be an isolated issue. If anyone is still facing issues please open a support ticket and we can investigate/troubleshoot accordingly :)

    All's well that ends well, and you handled the situation very well. Nobody is immune from a data center incident, nobody.
    I'm delighted to renew my confidence in you.

    Hi @commercial -- Thank You so much for your understanding -- it is true that not even the best of us can be immune from any sort of incidents in the world of IT/technology. We always strive to address issues transparently, learn from them, and continuously improve our processes and infrastructure resilience - while most importantly keeping our customers in constant communication/in the loop.

    We look forward to working together for many more years to come 👊 Thank You again for the understanding and renewed confidence.

    Thanked by 1commercial
  • zGatozGato Member
    edited March 27

    @dustinc said:

    @tdwuk said:
    Service down again today for approx. 2hrs so far. Not ideal. I had to put in a cancellation request. It's not worth the headache. Luckily I only had one landing page company site on the vps. Moving that asap.
    Same Strasbourg datacentre, ironically I am paying extra to host there :D

    There was a routing issue that impacted several nodes in this location earlier this morning, which has since been resolved. As of now service in France should be restored.

    I mean, it's kinda still happening

    (45.x subnet)

    Outside of that, and seeing that you offer migrations to AMS, any plans on providing that location for the yearly deals? even if it costs extra?
    (just wondering, would support allow me to buy a similar server that I have in FR but in NL instead?)

Sign In or Register to comment.