OVH Canada Down?

singsing

sin said: Just wish they would get their payment system up soon, keep getting 500 error when trying to pay for my new orders.

Well, hopefully they'll have the sense not to shut off servers when their payment system is down ...

ATHK

@singsing said:
Well, hopefully they'll have the sense not to shut off servers when their payment system is down ...

It'll more than likely be automatic .. So .. Yea..

@sin also said new orders, I'd assume the servers haven't been provisioned yet and sin chose a hell of a day to order anything ha-ha.

SplitIce

any status alerts regarding these incidents (VAC & BHS) that confirm this issue is fully resolved?

coolice

http://travaux.ovh.net/?do=details&id=15238

they claim so on French version

G Translate
The repair of the cable by the western route is complete. All links are still cut now reassembled (bhs <> chicago, bhs <> montreal, internal network to Europe and vrack). Traffic rerouted peering at the Bell / Videotron was standardized to return to normal performance. The new fiber to the new Montreal East road had to be put into production the same week. Bad timing. Although not planned and carried out entirely in the urgency of the situation, it goes into production has allowed us to avoid several hours of extra redonder failure and permanently the ability to Newark. We will in the coming days to migrate to other links on this link to secure the capacity to Chicago, Montreal and inter-data center network to Europe. We also expect the making of a 3rd next available completely independent road BHS to Newark that will further secure data center connectivity BHS. We present again our apologies for this incident.

krishnap

ovh is scam?

/s

sin

What caused the issues

doghouch

@joshin said:
Yeah, there was a fiber cut & they didn't have all their backup links in place.

I don't believe that OVH Canada runs only on one major backbone ._.

doghouch

@Kris said:
Remember they're backhauling the lot of fiber from Montreal, etc. to random part of Quebec, not exactly located in a telco city. Makes it cheap for a data center, but we run into the issue of the Wholesale Internet drop outs with 'redundant links' that go across the street but were run in parallel next to each-other.

IMO the parallel uplinks aren't worth dick when they're run on the same side of the street, bundled together, etc. What if an anchor drops by some random ship? You have 40% packet loss to BHS. Still not seeing 100% by any means, if you're going to SLA them for this, you picked the wrong DC. Bring that shit to Rackspace or something, they are budget, major fiber cuts, and still up. But just my opinion.

I am wondering if lake + many fibers cut = drunk fisherman sleeping being approached by angry OVH techs.

I'm pretty happy using the Google Cloud platform as they have a 100% SLA guarantee for my databases

William

doghouch said: I don't believe that OVH Canada runs only on one major backbone ._.

BHS is in a pretty rural location - i am surprised that the town even HAS 2 fiber paths.

gestiondbi

@William said:

As I previously said, this path is the main one for every DC/ISP/Transit provider to newark. Not only for OVH. This city was almost out of internet before OVH install itself and build multi-fiber path.

ricardo

OVH'll be glad they've stemmed the flood of customers heading to GoodHosting.co

HostHoney

Glad they seem back to normal, I know my kimsufi server is not having issues with connections from north america now.

perennate

Hello,

Thank you for contacting us about this concern. We did experience a major

service interruption yesterday, which impacted many of our customers. We
apologize for any impact this may have had on your services. Currently the
issue has been resolved, and we are now in the process of assessing the impact
in order to compensate our customers accordingly.

An official communication will be issued containing detailed information about

the service interruption, as well as the compensation that will be offered.

We thank you for your patience and understanding, and thank you for choosing

OVH.

I remain at your disposal for any further questions.

Have a great day,

sman

Didn't they learn anything from the backhaul fiber cut they had over some bridge going out to their farmhouse/datacenter awhile ago?

If they are going to be out in the sticks they should have planned for (at least) 2 separate paths taking different routes. Sounds to me like this was basically the same thing again. You save money being out in the boonies but you have to properly plan for the extra cost of other things like a proper redundant backhaul.

netomx

sman said: Didn't they learn anything from the backhaul fiber cut they had over some bridge going out to their farmhouse/datacenter awhile ago?

I think this question answers itself.

On the other hand,

BASEMENT.

Amitz

BASEMENT RAGE!!!

sman

@netomx said:
BASEMENT.

How is your packet sniffing going there Mr. VPN security expert keyboard commando.

FalconInternet

I opened a ticket for each server that was affected and they closed all of them, except one. But when you open a ticket, they want you to select the server affected and you can't pick more than one.

gestiondbi

@FalconInternet said:
I opened a ticket for each server that was affected and they closed all of them, except one. But when you open a ticket, they want you to select the server affected and you can't pick more than one.

You have the option... "Others"

Setsura

@perennate

Dear customer,

Due to the incident in our Beauharnois datacentre, you may have experienced a

downtime of your services yesterday evening, 2nd November. This has happened
due to a 6x10G damaged fibre link between Montreal and Beauharnois, 9 km from
the Beauharnois datacentre.

The fibre cable has been replaced and the incident was successfully resolved.

We apologise for any inconvenience caused by this incident.

The commercial gesture of 1 month as a compensation offered by OVH will be

automatically added to your account.

If you have any more questions, please do not hesitate to contact our OVH

support.

Yours sincerely,

OVH Management Team

I think this means they are offering me a full month of my services free? Rather than the 25% or so by their network SLA of 5% per hour downtime. I don't see anything on my account yet though.

FredQc

@Setsura said:
perennate

I think this means they are offering me a full month of my services free? Rather than the 25% or so by their network SLA of 5% per hour downtime. I don't see anything on my account yet though.

I really doubt that.

Setsura

@FredQc said:
I really doubt that.

What I posted was from my ticket, I don't see why they'd blatantly lie. OVH doesn't have the best support ever, but they usually aren't liars.

perennate

Setsura said: I think this means they are offering me a full month of my services free? Rather than the 25% or so by their network SLA of 5% per hour downtime. I don't see anything on my account yet though.

Cool, I hope that happens, still waiting for the "official communication" that they mentioned in my ticket.

inthecloudblog

No one asks unsatisfied customers to remain clients of ovh and those who are not clients why do they speculate?

perennate

inthecloudblog said: No one asks unsatisfied customers to remain clients of ovh and those who are not clients why do they speculate?

huh?

krishnap

@inthecloudblog said:
No one asks unsatisfied customers to remain clients of ovh and those who are not clients why do they speculate?

keyboard experiencing packet loss?

dragon2611

@sin said:
"The links between Newark and our datacenter of Beauharnois are down:
6x100G
20x10G
We will update this taso as soon as we have more information."

I'd have hated to be one of the NOC engineers on shift when that happened, the moment the alarms start you know it's going to be a bad day/week

gestiondbi

@dragon2611 said:
I'd have hated to be one of the NOC engineers on shift when that happened, the moment the alarms start you know it's going to be a bad day/week

However, is always the most happy week in terms of reactivity and knowledge. I hate routine!

Also, for those who cares, the official Post Mortem is out: https://forum.ovh.com/showthread.php/107019-Incident-Datacentre-BHS-2-novembre?p=654617#post654617 (FRENCH)

Gulf

I had servers in BHS two years ago and I was happy with their service.
Have they invested more in support, network and connectivity here?
What can you say about connectivity to Asia, Middle East, Europe?

perennate

On Monday November 2, 2015, the BHS datacenter suffered a new network problem
impacting all BHS Datacenter customers. This incident was similar to 5 months
ago, when 3 pairs of our optical fibers were cut between Beauharnois and
Montreal on the North route.

Two years ago, we began adding redundancy on the 103 KMs of fiber on the South
loop. Since then, there have been long delays concerning this project, bringing
the time of completion to 24 months instead of the planned 12. Work was finished
this week and only in response to the urgency of the situation. We forced the
installation of the new South fiber optic line, thereby reducing the duration of
downtime of the public network from 10 hours and 51 minutes to 4 hours and 55
minutes.

Now, the datacenter is connected in total redundancy by the North loop and the
South loop. We have 5 directions: 2 towards New York, 1 towards Chicago and 2
towards Montreal.

Three years after starting the project, we are continuing the work on improving
the 2nd route towards New York where all that remains is to solder the optic
fiber at the Canada/USA border. Negotiations are underway to connect Toronto and
Detroit, MI.

Here is what our dark fiber and DWDM network connecting BHS looks like:

We own:

• 4x10 Gbps between BHS and Montreal via the North loop
  
• 4x10 Gbps between BHS and Montreal via the South loop
  

• 6x100 Gbps towards New York/Newark via the South loop
  
  We rent:
  

• 10x10 Gbps towards Chicago via South loop
  

• 20x10 Gbps towards Newark via the North loop
  
• 4x10 Gbps towards London (through Canada) via South loop
  
• 4x10 Gbps towards Paris (through USA) via the North loop

http://mj.ovh.com/img/4umv/b/xj/0k.png

Within 3 to 4 weeks, we will finish the work on our own 2nd route towards New
York/Newark and we are going to activate an optical failover of 6x100 Gbps
through the following 2 routes: BHS/NYC-NWK. This will allow us to terminate the
rental of the 20x10 Gbps through Newark.

http://mj.ovh.com/img/4umv/b/xi/0h.png

We are going to increase the capacity of Montreal/Chicago to 2x100 Gbps then
4x100 Gbps while waiting for our own route to be finished between BHS and
Chicago by Toronto/Detroit. We cannot yet provide an ETA, knowing that we lack
800 KMs between Toronto and Detroit, MI. We already have in place BHS towards
Toronto and Detroit, MI towards Chicago, IL.

http://mj.ovh.com/img/4umv/b/xi/0i.png

At the same time, we are working on the network in the USA between New York and
Newark (NY) then Chicago (IL) and Ashburn (VA). This new USA network is planned
to be deployed in Q2 2016 and will not only provide redundancy from BHS towards
Chicago through New York but also allow for the connection of a datacenter in
the USA.

We sincerely apologize for the latest BHS downtime. We underestimated the time
and the complexity of the fiber optic network construction projects in Canada,
thus creating an unreasonable delay in securing each kilometer connecting the
BHS Datacenter. Even though today it's done and we have 100% redundancy, we are
really sad to announce this under such circumstance.

Within the next 10 days, we will offer compensation which conforms to our SLA
for the month of November.

The exact times concerning the incident:

• down at 11:53 AM GMT-5 (5:53 PM GMT+1)
  

• at 4:48 PM GMT-5 (10:48 PM GMT+1), traffic to the BHS datacenter from the
  outside returned to normal thanks to the optical fiber by the South
  

• in contrast, the links between BHS and the other OVH datacenters and the VAC
  (Anti-DDoS) were affected until the complete repair of the North fiber. Traffic
  was able to resume at 10:44PM (04h44 GMT+1).

Best Regards,
Octave