Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Leaseweb having troubles, vps offline for 2 days now
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Leaseweb having troubles, vps offline for 2 days now

raza19raza19 Veteran
edited September 2015 in General

Tue, Sep 1, 2015

>

Dear LeaseWeb customer,

Yesterday evening, August 31st, around 20:00 CEST we encountered an unexpected event on one of our storage platforms in the Netherlands, resulting in a serious degradation of the availability of your Public Virtual Server.

LeaseWeb engineers and our storage supplier started immediate investigation, after receiving alerts regarding high load on the nodes, caused by the storage platform. Since then, several corrective actions were carried out, in attempt to normalize the load on the nodes. As these actions were not successful, investigation continued with the highest priority.

Please be assured that all our resources are determined and available to solve the incident adequately as soon as possible. We understand the impact on your business. Despite ongoing efforts and the complexity of this issue, we don’t expect the incident to be resolved within the next 12 hours. We will inform you frequently through email.

The current situation and the general stability of this platform does not meet our company standards and the reliability that you may expect from LeaseWeb. Parallel to the focus on this incident, we already started preparations last month to replace this platform. More information about this replacement will follow.

Sincerely,

LeaseWeb Operations

.

[UPDATE 17:15 CEST]

>

Dear LeaseWeb customer,

>

The investigation is still ongoing, but unfortunately no significant improvement of the performance of the platform has been achieved. LeaseWeb engineers and vendors will therefore continue the investigation during the evening and throughout the night if needed, to restore the platform to normal operations. We fully realize that this incidents persists too long, but we are doing everything in our power to counter these performance issues.

.

[UPDATE 23:00 CEST]

>

Dear LeaseWeb customer,
This is an update about the earlier communication regarding an unexpected event on one of our storage platforms in the Netherlands, resulting in a serious degradation of the availability of your Public Virtual Server.

LeaseWeb engineers, together with our storage supplier are currently stabilizing the platform. Our engineers are bringing individual virtual servers back online in the next hours. To ensure platform stability during these operations, we have temporarily disabled start/stop and reboot functionality in the LeaseWeb Customer Portal. Please be assured that all our resources are determined and available to solve the incident adequately as soon as possible.

The current situation and the general stability of this platform does not meet our company standards and the reliability that you may expect from LeaseWeb. Parallel to the focus on this incident, we are therefore also rolling out the replacement of this platform. More information about this will follow.

.

Wed, Sep 2, 2015 [UPDATE 09.30 CEST]

>

Dear LeaseWeb customer,

We hereby want to update you on the current status of the platform. LeaseWeb engineers worked throughout the night to stabilize the platform. It is currently functional again, and a significant part of the affected Public Virtual Servers are now back online.

The remaining Public Virtual Servers are currently returning to operational status through an automated script. Our engineers are also manually double checking the functionality of each and every affected servers.

In addition we will start the migration of the Public Virtual Servers to the new storage platforms today. Customers will be individually contacted with more details regarding this event and the migration.

«1

Comments

  • vfusevfuse Member, Host Rep

    Cloud providers always seem to link centralised storage with HA and failover but somehow the centralised storage always fails.

    Local storage > nas

    Thanked by 1Jeffrey
  • Yep, adding complexity does not always add reliability. Sometimes it's the opposite. The simplest things are the best and most reliable.

    Thanked by 3coolice samc 4n0nx
  • ClouviderClouvider Member, Patron Provider

    They key is to have redundant storage behind the cloud ;-)

  • Frankly this is disappointing, one pays services like leaseweb more than usual to avoid low end uncertainty. Turns out its all for nothing...One's certainty it seems like life is primarily a factor of luck.

  • joepie91joepie91 Member, Patron Provider
    edited September 2015

    @raza19 said:
    Frankly this is disappointing, one pays services like leaseweb more than usual to avoid low end uncertainty. Turns out its all for nothing...One's certainty it seems like life is primarily a factor of luck.

    Really, this just reinforces the thing that several people on here (including me) have been repeating for a while now: technical failures happen, and relying on your provider is a bad idea. It doesn't really matter whether you're paying $5/mo or $50/mo.

    You should always implement redundancy on the application level, involving multiple providers.

  • Some good news:

    Dear Customer,

    Since yesterday, we have been working to help a group of customers to recover from a storage failure. As we continue to strive for the highest level of quality and reliability, we are taking immediate steps to improve the platform’s performance and availability by upgrading to a full-SSD storage system.

    You can find the latest updates at http://leasewebnoc.com/en/networkstatus/unexpected-event-on-our-storage-platform-in-the-netherlands. If you have any questions, please contact [email protected] and we will give you the help you need.

    We know how critical our infrastructure is to your business and intend to use this incident to drive improvements right across our platform going forward. Above all, we apologize for any inconvenience it may have caused and would like to offer you one of the following compensation options:

    • No charge for your existing Virtual Server for 3 months (October, November and December). To apply for this option, please contact us by phone or email [email protected].

    -or-

    • Any new Virtual Server free of charge for 3 months. To take advantage, please enter the code “FREEVPS” when you checkout on our website. Valid from 02 Sep 2015 12:00 CEST until 04 Sep 2015 12:00 CEST.

    We appreciate your patience and cooperation during this time.

    Kind regards,

    LeaseWeb Sales

  • ClouviderClouvider Member, Patron Provider

    After you published this code, lets see how many new 'Customers' they get :-).

    Thanked by 1vimalware
  • @Clouvider said:
    After you published this code, lets see how many new 'Customers' they get :-).

    Then again its not like that would be one thats hard to guess if you were just randomly entering codes.

  • Man I feel for those of you in their NL datacenter, I'm subscribed to their network status emails (I'm in their Virginia datacenter) and I'm forever getting a shit ton of emails about something going wrong in NL - but at least they do provide regularly updates to issues instead of keeping people in the dark.

  • ClouviderClouvider Member, Patron Provider

    @VPSSoldiers said:
    Then again its not like that would be one thats hard to guess if you were just randomly entering codes.

    Yeah, but not many would try to brute force it.

  • You'd be amazed how many times I've seen threads about people entering random strings until they found a valid code.

  • @madtbh said:
    Some good news:
    No charge for your existing Virtual Server for 3 months (October, November and December). To apply for this option, please contact us by phone or email [email protected].

    LeaseWeb Sales

    I paid three years in advance! So I get @$#&;()

  • @Clouvider said:
    After you published this code, lets see how many new 'Customers' they get :-).

    I thought it would only apply to the affected customers like myself, oops.

    @sin said:
    Man I feel for those of you in their NL datacenter, I'm subscribed to their network status emails (I'm in their Virginia datacenter) and I'm forever getting a shit ton of emails about something going wrong in NL - but at least they do provide regularly updates to issues instead of keeping people in the dark.

    One of my boxes is okay, the other was running in read only, and on top of that they limited the performance. Now I'm just waiting to be moved to the full SSD storage platform.

  • So almost 2 weeks down the lane my vps is still barely usable, their support told me it could take another week for all the vps to move !

    dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 37.6176 s, 28.5 MB/s

  • 28.5MB/s is barely usable?

  • @tr1cky said:
    28.5MB/s is barely usable?

    I caught this on a good day, there have been worse.

  • NyrNyr Community Contributor, Veteran

    raza19 said: I caught this on a good day, there have been worse.

    Is this one of those 3€/month VPS? If so, I would consider even 10 MB/s reasonable.

  • rds100 said: Yep, adding complexity does not always add reliability. Sometimes it's the opposite. The simplest things are the best and most reliable.

    This.. all my dedicated servers literally have had pretty much 100% uptime. Meanwhile my VPS's..

  • ru_tldru_tld Member, Host Rep

    LSW cloud offer is certainly not bad if only storage was much stable and better.
    They are currently migrating their storage from SAS to SSD. I hope that they also have changed storage vendor.

  • @Nyr said:
    Is this one of those 3€/month VPS? If so, I would consider even 10 MB/s reasonable.

    their smallest plan starts from 5 euros a month

  • raza19 said: their smallest plan starts from 5 euros a month

    Yep, it seems their SAVE40 promo is no longer valid.

    Thanked by 1vimalware
  • sinsin Member
    edited September 2015

    Nyr said: Is this one of those 3€/month VPS? If so, I would consider even 10 MB/s reasonable.

    I get around 75-80 MB/s on all my Leaseweb VPSes and they have been hella stable too, but I'm at their Virginia location. $5.97 for 2 cores, 2GB, 60GB Disk, and 6TB bandwidth is awesome.

    I've been receiving customer surveys from them and it looks like they plan to offer SSD and snapshot features on their VPSes in the near future.

  • Chat between me & a tech rep at leaseweb :

    Lennard Stam: Hello How can i help?

    >

    Me: I was told on Sep 11 that vps migration wasn't complete and it would take another week for all the migrations

    >

    this is what I had on Sep 11

    >

    dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 37.6176 s, 28.5 MB/s
    .................................

    >

    this is what shows right now

    >

    dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync
    16384+0 records in
    16384+0 records out
    1073741824 bytes (1.1 GB) copied, 41.4216 s, 25.9 MB/s

    >

    >

    so basically things have gone worse ?

    >

    Lennard Stam: It is migrated. But performance is capped until total move is done

    >

    Me: how long will the total move take ?

    >

    Lennard Stam: It is almost done. more then 60% done. for this node.

    >

    Me: but should I wait another week for it to complete ?

    >

    Lennard Stam: Hard to tell precisely. It should be around a week indeed. Maybe sooner

  • bersy said: Yep, it seems their SAVE40 promo is no longer valid.

    Sad news, that was a good deal, with their normal pricing other providers are more interesting..

  • zeitgeistzeitgeist Member
    edited September 2015

    @raza19 said:
    Chat between me & a tech rep at leaseweb :

    Lennard Stam: Hard to tell precisely. It should be around a week indeed. Maybe sooner

    Strange. I have a NL VPS as well, and dd showed somewhere ~ 90MB/s pre- and post-migration. Not exactly SSD speeds, and I expected them to be capping the performance, but it's good enough for me given the low price and the excellent bandwidth.

  • scy said: Sad news, that was a good deal, with their normal pricing other providers are more interesting..

    True. FWIW, the promo is still good (and automatically applied) for the L and XL plans.

    Thanked by 1scy
  • @zeitgeist said:

    maybe you are on a different node in NL

  • raza, is your vps migrated to ssd node? what's the io like?

Sign In or Register to comment.