New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
As said already by me you are just relying on a single fiber link provider that is why dual cut now caused a whole DC down.
You should have an alternate link from other provider as well same as we host backup server on other DC.
What the benefit of just monitoring? You should work on alternate solution now as almost 12 hours and counting down time...
MTRs show a lot of HE.net to ReliableSite IPs, perhaps they got some emergency BW to get things back online?
Backhauling from Piscataway to Newark with only Zayo... That's a bold move cotton.
450 minutes down, and counting. i don't see any update from Zayo.
I doubt Zayo can solve this problem immediately.
They are still having trouble even though it seems like the cable break problem has been resolved.
This is so bad.
I hope you think that "zayo can't fix it in time". So you should be able to find the best solution from now on, regardless of whether Zayo can fix this problem or not.
i think, you should think of alternative way, since experience 4 hours downtime.
This is something that hurts, but I hope we all learn about this, especially the trusted.net site
The longer this is delayed, the harder it is to explain to our respective clients.
Not a single fiber link....
We had a redundant ring, multiple paths. One was damaged a little over a day ago, the second was cut by this maintenance.
We're bringing online alternate transit asap.
Were both to 165 Halsey? Would figure path one would go to 111 8th, if only to pickup more peers.
Godspeed, things like this are hell, especially when it's upstream.
Unfortunately for me and my clients in South America we are 100% offline, since the beginning of the day, some clients are asking to migrate to another location. For now I'm resisting but I won't be able to wait much longer, because here in Brazil we are 100% offline. operation
My sites using IPv4-only are mostly down but IPv6 is still up. Cloudflare tunneled sites are also up. Hopefully this can be resolved soon.
Time continues and we are accumulating more tickets from angry customers, hopefully this can be resolved as soon as possible. I can't see the reliablesite.net twitter account because I was banned, and only for commenting on its deficiencies in a respectful manner.
I find it ironic that folks on LET are complaining about outages and the lack of peering redundancy. As a designer of fiber networks I will refer you to the graphic below. Here's your sign.
Might be just a cut in your redundant fiber, however, I see prefixes partially not longer announced.
Something is broken.
People are forgetting how cheap this service is, expecting flawless uptime on extremely inexpensive services is a little silly. It looks like they're working on fixing the issue though.
Sure, but I think it’s probably a lack of proper planning or a very unlucky day, or both.
Most great DC’s that have been around long enough has had big periods of downtime. I think, at least.
Additionally, I don’t think RS is cheap enough to justify their uptime regularly going down for hours and hours because of cheaping out due to so cheap pricing.
So I’m neither attackong or defending RS, I don’t think they’re cheap enough to justify this being a regular thing, and I don’t think it is either.
Expecting the temporary transit to come online any minute now.
Yeah I moreso meant they provide cheap and somewhat adequate service, generally for the price you pay you're not getting insane triple replicated infrastructure just in case multiple things fail at once. Infrequent downtimes due to the stars aligning in a super unfun way just sort of happen with smaller providers like this, it's part of the cost of doing business with cheaper hosts. I don't expect this to be a regular thing and fiber cuts are never fun, good luck Radic!
Seeing GTT and a lot of NYIIX / Tata. Up from all locations.
So you means to say cheap service equal cheap quality?
Yeah I don't understand this. Were Zayo still announcing the prefixes so the traffic was effectively being blackholed or did RS just not have any other transit providers? This screenshot seems to imply that they did have another transit provider because some packets went through though...
If they didn't have any other transit providers, are these just taking some IX route on some other provider's backbone?
This is very weird indeed...
Next time use a network outside of your own premises for hosting ,
Hetzner's Ashburn DC is recommended
exactly
Sometimes it seems like too cheap = bad
But after awhile the effect seems to go away (and not even after too long)
I mean sure, DO >>> Virmach
But BlueHost >>> Hetzner?
Talking raw compute here, but the examples are less important
It’s more like
Too cheap = trying to save money across the board
Cheap enough = trying to save money on things that might not be of value to a potential client (e.g. TV ads)
My point earlier was that I’m not 100 % sure RS falls within the ”too cheap to be very good” category
Anyway, I guess this answers it:
@MrRadic - will you be keeping something equivillent to the new temporary transit once things are back up, or could this very well occur a second time (i.e. the price paid dosen’t justify the added redundancy)?
Also yes, to me it seemed like they, after a while, deployed a static version of their site on another network (because the static stuff was loading, but it seemed that some dynamic content as well as the WHMCS wasn’t
Clients being able to access first-party information during incidents is important
Our costs have nothing to do with cutting corners. We've spent years investing in automation in deployment, network, and common tasks. We have almost a decade of experience with manufacturing hardware. These two things along with a few other things we've been doing internally have allowed us to keep costs low with massive scaling and pass that on to get costs to where they are now.
Currently the temporary transit is not active, but we can flip it on if we need to.
We're knee deep in diversifying our transit between 165 Halsey and 111 8th Ave. It really shouldn't be needed, but between what happened today with Zayo's cut of our redundant fiber and the power outages that 165 Halsey had earlier this year, there doesn't seem to be a choice.
@MrRadic thanks for the answer. This was actually what I thought and meant by ”not cheap enough to justify..”
I feel like with optimized processes, your prices should have some margin to them, even with a reliable network and other infrastructure
Not that I find the services expensive, but far from so cheap that reliability shouldn’t be expected
Final question: Enabling the temporary transit, now that it’s put in place but isn’t active, how long would that take should this occur again? Are we talking minutes or hours?
Just need to flip on a BGP session. It's already configured.
Honestly, to me, thah settles it: Not cheap enough to not be reliable
Everyone should claim SLA credit and if they refuse, all just post review here and WHT so atlest new potential clients at least know about how "unreliable" resbilesie it?
Oooh you're in 111 8th Ave? Any option to get something inside that building specifically?