Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Serverica down - Page 4
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Serverica down

124»

Comments

  • iNK79iNK79 Member
    edited July 2021

    @default said: EDIT: Also, why did you want to move in the first place? There is a saying: "If it works, don't fix it."

    from the email

    New Products Soon:

    Since we have moved to our own space, we will be able to offer services that we didn’t offer in the past mainly we will start offering dedicated servers and even colocation services

    For dedicated servers, we may even offer storage dedicated servers

    Thanked by 1servarica_hani
  • My downtime was perfectly reasonable, no need for a refund.

    A few hours update ahead of the masses when mammoths or even more impressive storage offerings are made available would be awesome though.

    Thanked by 1servarica_hani
  • Daniel15Daniel15 Veteran
    edited July 2021

    Got this via email. I wish all providers were this transparent. Kudos @servarica_hani - downtime sucks, but you handled the issues extremely well. I won't be asking for a refund.


    As you know from 30th of June and for 2 days after that we have been moving servers to new location.

    The move took much more than expected for storage vps plans, and in the email we would like to explain what went wrong ,  explain the compensation for the downtime and some good news about new product lines

     

    How our storage works

    As some of you know we are running our storage servers as SAN storage, that means we have separate server that have the disks and another server that run the VPS (VMs)

    The servers that have disks will call them in this email as storage servers and the servers where VMs run will call them compute servers.

    The benefit of this system is that it allow us to run compute servers in pools with N+1 redundancy,

    So for a compute pool with 4 servers, we will run VPSes(VMs) that fill the RAM  of only 3 servers keeping 1 free (we run the vms in all 4 servers but we have enough free ram in the pool to free 1 server)

    We can move VPS(VM) live between compute servers of the pool and we use this feature to apply security updates and other maintenance to the servers (we free the servers by moving all its VPS to the other compute servers in the pool live then we can reboot that server)

     

    It is a lot of work to do maintenance this way but it allow us to keep client vms up and running even when we need to reboot the host they run in.

     

    We have several compute pools and several storage servers and any vm on any pool can have its storage in any storage server.

    This many to many relationship proved to be huge issue while moving.

    As we are forced to move all compute servers and all storage servers in one shot

     

    Factors that affected the move

    Now add to that other factor that make this move much slower.

    Disks

    our top priority was not actually speed, our top priority was delivering the disks safe and keep them secure through the journey since user data was on them. We cannot move the servers while disks are in due to very heavy weight and due to the fact it is too risky on the disks so all disks have to be removed from servers and packed, and on the new datacenter the disks have to reinstalled again on the servers

    In order to reduce move time, we decided to create the network in the new datacenter from scratch We did all the cabling in the new DC before the move. Although having the network ready on the new dc was done to save a lot of time it didn’t end up saving much due to mistakes in connecting the servers to wrong network ports (even with correct labeling)
     

    Manpower:

    The Move was estimated based on the fact that several movers will be moving the servers. The move of the first SSD and NVMe servers went fast but we faced few networking issues in the new location, and we decided to fix them before moving the remaining servers. The network fix and testing that everything on first 2 patches is good took almost all 30th of June. We didn’t agree with anyone for the 1st of July and non of the movers was available for 1st of July That means the biggest part of the move (physically not necessarily in the number of clients) has to be done by single person 1st of July
    Date:

    30th of June and 1st of July are the moving days for every home in Montreal (here all home renters contract finish on 30th of June and in the 1 week before and after it is very hard to find trucks and movers ) That is the reason why no one was available on 1st of July The contract with old datacenter end for most of the racks on June 1st and to avoid gray situations I decided to proceed with the move alone
     
     

    Due to those factors the work of moving storage servers was done by single person, now with all those disks and servers moving it took very long time to be done  

    By the time the server installation done few mistakes were done while connecting the network which caused few extra hours of delay as well  

     
     

    SLA REFUND:

    Storage servers availability were between 95% to 97% due to the move and as a result of that based on our SLA every storage user is entitled to 50% refund

    For SSD and NVMe servers since their uptime was between 99% to 99.9% their refund will be 15%

    For IPv6 only storage servers (mouse storage with IPv6 only) the refund is 100%

    So please answer this email or open a ticket for the refund to be applied.

     

    Few notes:

    if total refund value was under 6$ then it will be just added as credit to your account which will be applied to next invoice if last payment cannot be refunded due to the payment method (crypto currencies) or due to being long time since the payment made (6+ months for paypal) the refund will be issued as credit for non monthly payments the refund is calculated based on the total payment for the term / number of months in the term * refund percentage
    So for yearly plan that cost 120$ with 50% refund total refund will be 120/12 * 0.50 = 5$

     
     

    New Products Soon:

     

    Since we have moved to our own space, we will be able to offer services that we didn’t offer in the past mainly we will start offering dedicated servers and even colocation services

    For dedicated servers we may even offer storage dedicated servers

     

    Thanks 

    Hani

    Team ServaRICA


  • @default said:

    @default said:
    @servarica_hani - If I may ask, why did you not hire more people, so the impact would be minimal?

    EDIT: Also, why did you want to move in the first place? There is a saying: "If it works, don't fix it."

    The email (I quoted it above) explains why he couldn't get more people for the move.

    AFAIK the move was to relocate to a new facility with more space, as the old space was completely full (or close to full) and didn't have enough space to add new servers.

  • I also won't be applying for any credit for the (in my opinion) minimal downtime. Although the move took longer than expected, it was announced in advance.

    It's not like my storage server disappeared without warning or explanation.

    I'm happy with the service and with the updates provided.

  • @Daniel15 said:

    @default said:

    @default said:
    @servarica_hani - If I may ask, why did you not hire more people, so the impact would be minimal?

    EDIT: Also, why did you want to move in the first place? There is a saying: "If it works, don't fix it."

    The email (I quoted it above) explains why he couldn't get more people for the move.

    Not really, no. It's not like it's unexpected to have peak movers on the 1st of the month, he even points it out.

    July 1st was Canada Day, a statutory holiday. Doing moves like this on a public holiday is either very smart or very dumb, depending on the circumstances. I'm sure that exacerbated any issues.

    I'm sure he's learned to lock down movers ahead of time if this ever happened again.

  • hotsnowhotsnow Veteran

    @servarica_hani said:

    @hotsnow said:
    any updates on IPv6? still can't connect...

    it is up for some time now

    you cant connect to google though as current uplink does not peer with them , this will be fixed when we get back the other uplink

    id you still cant access your ipv6 open ticket as it should be workin for all by now

    thank you for the update :)

    and it seems also no peer with HE.NET? can't connect to and from HE

  • JabJabJabJab Member
    edited July 2021

    From what I understand they had people on 30th and planned to do everything on that one day, but because of "issues" it was delayed to 1st of next month and they couldn't hire anyone because everyone was already booked in advance. Current datacenter contract was ending on 1st so it was choice between a) next month contract if they agree, probably premium; b) hope that they won't cut power/internet on 00:00 1st of month and let it fly for extra 24 hours (?); c) move everything solo

    Yeah, planning wasn't great, but that is combo of issues + deadlines (like always ;)).

    Thanked by 1TimboJones
  • ServaRica is a premium one..

    Thanked by 1servarica_hani
Sign In or Register to comment.