Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


HAZI.ro - New problems, new solutions
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

HAZI.ro - New problems, new solutions

FlorinMarianFlorinMarian Member, Host Rep

Hello friends!

The first message was sent by email yesterday to all active customers, but in order not to SPAM, I leave here the update with the solution to the respective problem.

Hello!
In the last few days there have been multiple reboots at the level of hypervisors located in our own location in Romania. Although there was no data loss, we know that not all customers have autostart set for their critical software, and these reboots can be very annoying.
So, we are aware that we have a problem and we are ready to solve it as soon as we find out the source of this problem.
What we know for sure at the moment is that at the grid level (the public electricity supply network) there are interruptions in the supply of energy lasting less than a second, the effect being visible only when there are light bulbs on and there are small fluctuations of light. The fact that these interruptions are so short, puts a strain on our systems which should ensure us up to 100% uptime:

  • the generator starts automatically in 10 seconds -> useless in the present case
  • the batteries can support the load for up to 2 hours -> useless in the present case
  • The OFF-GRID inverter in UPS mode should switch the electricity source from the grid to the batteries in less than 100ms -> this may be the problem
  • The PDU that shares the load from the inverter to the IT equipment has in turn different degrees of protection -> this may be the problem.

Since we don't know exactly if the PDU or UPS is failing in this scenario, we will need to continue monitoring the infrastructure until we know which component will need to be replaced.

We know that the current situation makes it difficult to use the services normally, but we assure you that we are doing everything we can to solve this problem as quickly and efficiently as possible. We do not want to buy a new PDU or a UPS for nothing as the costs vary between 1000 and 5000 EURO and an unnecessary investment would later lead to unjustified price increases to cover this new investment.

Thank you for understanding!

UPDATE

Hello!

We are happy to announce that today we purchased a new UPS that will solve the problems of the last few days, problems that we talked about in the previous announcement.

The delivery of the new UPS will be made in a maximum of 5 working days.

We apologize in advance for the problems you will face in the next period until the new UPS is put into operation.

Thank you for understanding!

Thanked by 3host_c davide ikibsys
«13

Comments

  • dosaidosai Member

    OK

  • LeviLevi Member

    @FlorinMarian said: but in order not to SPAM

    You just spamed here. 0 f's given. I didn't even read the text...

  • @FlorinMarian said: would later lead to unjustified price increases to cover this new investment.

    Can't wait to see the new new basement hosting price

    Thanked by 2Void salkoay
  • davidedavide Member
    edited June 25

    What new location?

    In the old location, the tutorial map, aka The Basement, you had some trouble understanding that the problem was the undersized batteries. How did that one go?

  • therawtheraw Member

    7$ 32gb ram vps and i promise i will read whole thing

  • HOSTCAYHOSTCAY Member, Patron Provider

    Succes mariane

  • Tldr
    @Calin has gone clear

    Thanked by 2Calin Frameworks
  • AndruAndru Member
    edited June 25

    Keep spamming LET threads, maybe more users will read your signature.

  • kheng86kheng86 Member

    $7

  • coldcold Member

    @davide said:
    What new location?

    In the old location, the tutorial map, aka The Basement, you had some trouble understanding that the problem was the undersized batteries. How did that one go?

    new location : garage, all servers were migrated

    Thanked by 1JasonM
  • xvpsxvps Member
    edited June 25

    @FlorinMarian said:
    ... We do not want to buy a new PDU or a UPS ... price increases to cover this new investment.

    One day later

    We are happy to announce that today we purchased a new UPS ...

    Your broom closet vm's currently cost approximately the same as Hetzner cloud and then you plan price hikes. :D

  • FlorinMarianFlorinMarian Member, Host Rep

    @xvps said:

    @FlorinMarian said:
    ... We do not want to buy a new PDU or a UPS ... price increases to cover this new investment.

    One day later

    We are happy to announce that today we purchased a new UPS ...

    Your broom closet vm's currently cost approximately the same as Hetzner cloud and then you plan price hikes. :D

    For your sake, I invented a new quote: I can offer you a hosted server at Hetzner, but Hetzner cannot offer you a hosted server in my house.

    Thanked by 2yoursunny gbzret4d
  • davidedavide Member

    @FlorinMarian said:
    For your sake, I invented a new quote: I can offer you a hosted server at Hetzner, but Hetzner cannot offer you a hosted server in my house.

    +1, love it!

    Here's another one!!

    It's better to remain silent and be thought a fool than to speak and to remove all doubt

  • edited June 25

    @FlorinMarian said:
    For your sake, I invented a new quote: I can offer you a hosted server at Hetzner, but Hetzner cannot offer you a hosted server in my house.

    Now if your house manages to match Hetzner's support that's quite a feat ;)

    (support in this case meaning that i can open a ticket at 01:30 in the morning saying that my box won't come online anymore and 25-30 minutes later have a reply stating that it won't POST, so the disks were transplanted to a new server with exact same specs and the box is back up running on the original IP - TL;DR: 25-30 minutes from report of dead box to diagnose and full restoration of service in the middle of the night)

  • FlorinMarianFlorinMarian Member, Host Rep

    @totally_not_banned said:

    @FlorinMarian said:
    For your sake, I invented a new quote: I can offer you a hosted server at Hetzner, but Hetzner cannot offer you a hosted server in my house.

    Now if your house manages to match Hetzner's support that's quite a feat ;)

    (support in this case meaning that i can open a ticket at 01:30 in the morning saying that my box won't come online anymore and 25-30 minutes later have a reply stating that it won't POST anymore, so the disks were transplanted to a new server - exact same specs - and the box is back up running - TL;DR: 25-30 minutes from report of dead box to diagnose and full restoration of service in the middle of the night)

    The type of clients I hunt for don't use Hetzner because they don't know it exists, but because they need something else.
    Let's not prolong an endless story. Hetzner does what it does well and I do what I can.

    Thanked by 1gbzret4d
  • yoursunnyyoursunny Member, IPv6 Advocate

    @xvps said:
    Your broom closet vm's currently cost approximately the same as Hetzner cloud and then you plan price hikes. :D

    Yes but does Hetzner have two very scary guard dogs?

    Thanked by 3xvps inthecloudblog 1q1
  • edited June 25

    @FlorinMarian said:

    @totally_not_banned said:

    @FlorinMarian said:
    For your sake, I invented a new quote: I can offer you a hosted server at Hetzner, but Hetzner cannot offer you a hosted server in my house.

    Now if your house manages to match Hetzner's support that's quite a feat ;)

    (support in this case meaning that i can open a ticket at 01:30 in the morning saying that my box won't come online anymore and 25-30 minutes later have a reply stating that it won't POST anymore, so the disks were transplanted to a new server - exact same specs - and the box is back up running - TL;DR: 25-30 minutes from report of dead box to diagnose and full restoration of service in the middle of the night)

    The type of clients I hunt for don't use Hetzner because they don't know it exists, but because they need something else.
    Let's not prolong an endless story. Hetzner does what it does well and I do what I can.

    That's not really the point. It's rather why either Hetzner or their clients would care about them being able to sell boxes at your house when the main thing drawing people to Hetzner isn't there.

  • AndreixAndreix Member, Host Rep

    @totally_not_banned said:

    @FlorinMarian said:

    @totally_not_banned said:

    @FlorinMarian said:
    For your sake, I invented a new quote: I can offer you a hosted server at Hetzner, but Hetzner cannot offer you a hosted server in my house.

    Now if your house manages to match Hetzner's support that's quite a feat ;)

    (support in this case meaning that i can open a ticket at 01:30 in the morning saying that my box won't come online anymore and 25-30 minutes later have a reply stating that it won't POST anymore, so the disks were transplanted to a new server - exact same specs - and the box is back up running - TL;DR: 25-30 minutes from report of dead box to diagnose and full restoration of service in the middle of the night)

    The type of clients I hunt for don't use Hetzner because they don't know it exists, but because they need something else.
    Let's not prolong an endless story. Hetzner does what it does well and I do what I can.

    That's not really the point. It's rather why either Hetzner or their clients would care about them being able to sell boxes at your house when the main thing drawing people to Hetzner isn't there.

    I'm not taking Florin's side here, however I find it funny. We also receive a lot of tickets with "OVH is cheaper/Hetzner is cheaper/Can you price-match Kimsufi"... etc.

    And the simplest question that comes to my mind is: If OVH/Hetzner fits your needs better than me, why are we having this conversation in the first place?
    It's clear that you are not eligible for those providers or they don't provide you with something that I do. So, that's why you pay me for the service and not OVH/Hetzner.

    Thanked by 3FlorinMarian ralf jsg
  • edited June 25

    @Andreix said:

    @totally_not_banned said:

    @FlorinMarian said:

    @totally_not_banned said:

    @FlorinMarian said:
    For your sake, I invented a new quote: I can offer you a hosted server at Hetzner, but Hetzner cannot offer you a hosted server in my house.

    Now if your house manages to match Hetzner's support that's quite a feat ;)

    (support in this case meaning that i can open a ticket at 01:30 in the morning saying that my box won't come online anymore and 25-30 minutes later have a reply stating that it won't POST anymore, so the disks were transplanted to a new server - exact same specs - and the box is back up running - TL;DR: 25-30 minutes from report of dead box to diagnose and full restoration of service in the middle of the night)

    The type of clients I hunt for don't use Hetzner because they don't know it exists, but because they need something else.
    Let's not prolong an endless story. Hetzner does what it does well and I do what I can.

    That's not really the point. It's rather why either Hetzner or their clients would care about them being able to sell boxes at your house when the main thing drawing people to Hetzner isn't there.

    I'm not taking Florin's side here, however I find it funny. We also receive a lot of tickets with "OVH is cheaper/Hetzner is cheaper/Can you price-match Kimsufi"... etc.

    Um, i wasn't the one talking about pricing. That was just the guy starting the whole HAZI/Hetzner thing. I wrote about their insanely reliable support. Sure their prices are nice too, no question. As far as matching is concerned, well, i guess with older hardware and cheap energy it might still be possible to get close even for smaller operations. A paid off system is a paid off system after all. Newest Ryzen or something like that though...

    And the simplest question that comes to my mind is: If OVH/Hetzner fits your needs better than me, why are we having this conversation in the first place?

    That's pretty much what i'm saying to. People drawn to Hetzner likely won't care about them not being able to rent out out spots in Florin's house ;)

    It's clear that you are not eligible for those providers or they don't provide you with something that I do. So, that's why you pay me for the service and not OVH/Hetzner.

    Well, i can't really comment here as i'm not one of those guys. If i don't like your prices i'll 99% just leave and look elsewhere. If i actually contact you there's probably really something of interest to me and i'll tell you directly which part of the configuration is the interesting one and which i feel are useless overhead ;)

    Edit: This ultra lowballing is also a bit of a cultural thing for some people. I've spend quite a bit of time selling stuff at fleamarkets and you learn to recognize your customers quite quickly. There's this type of guy which you always give the tripled or quadrupled price when asked as this is what will take the least amount of time to actually sell at your target (yeah, he'll counter your 1€ with 25cents and will probably walk away when you insist on anything above 50 - he might come back later and try the same shit again though...). It's kinda stupid and obviously won't work on a website.

    Thanked by 1Andreix
  • Not_OlesNot_Oles Moderator, Patron Provider

    Initially,

    @FlorinMarian said:

    • The OFF-GRID inverter in UPS mode should switch the electricity source from the grid to the batteries in less than 100ms -> this may be the problem
    • The PDU that shares the load from the inverter to the IT equipment has in turn different degrees of protection -> this may be the problem.

    Subsequently,

    @FlorinMarian said: today we purchased a new UPS that will solve the problems

    Greetings @FlorinMarian!

    Any chance you might want to share details on how you determined that the UPS was the cause of the problem? Or did you decide to just try replacing the UPS first? If the latter, why? What are the relative costs of the two items?

    @FlorinMarian said: not all customers have autostart set for their critical software

    Why would some customers not have Autostart set for their VMs? Why isn't Autostart the default?

    Thanks!

    Tom

  • bootboot Member

    I read all that and I want a $7 refund.

  • wanna see DC pics!

    Thanked by 1Calin
  • FreekFreek Member

    Thanks for your feedback

  • @SillyGoose said:
    wanna see DC pics!

    What DC dude? A rack, a switch and 4 servers are not a DC.

    @anyNewProvider read all @FlorinMarian posts and do the oposit. You will have succes.

    Thanked by 2sasslik cybertech
  • emghemgh Member

    @totally_not_banned said:

    @FlorinMarian said:
    For your sake, I invented a new quote: I can offer you a hosted server at Hetzner, but Hetzner cannot offer you a hosted server in my house.

    Now if your house manages to match Hetzner's support that's quite a feat ;)

    (support in this case meaning that i can open a ticket at 01:30 in the morning saying that my box won't come online anymore and 25-30 minutes later have a reply stating that it won't POST, so the disks were transplanted to a new server with exact same specs and the box is back up running on the original IP - TL;DR: 25-30 minutes from report of dead box to diagnose and full restoration of service in the middle of the night)

    As I’m skimming on mobile, I first thought you changed to a huge signature

  • FlorinMarianFlorinMarian Member, Host Rep
    edited June 25

    Hey, Tom!

    @Not_Oles said:
    Initially,

    @FlorinMarian said:

    • The OFF-GRID inverter in UPS mode should switch the electricity source from the grid to the batteries in less than 100ms -> this may be the problem
    • The PDU that shares the load from the inverter to the IT equipment has in turn different degrees of protection -> this may be the problem.

    Subsequently,

    @FlorinMarian said: today we purchased a new UPS that will solve the problems

    Greetings @FlorinMarian!

    Any chance you might want to share details on how you determined that the UPS was the cause of the problem? Or did you decide to just try replacing the UPS first? If the latter, why? What are the relative costs of the two items?

    Initially, it was difficult to share the blame, not knowing initially what configuration exists between the IT equipment (6 servers and two switches), knowing that I have two PDUs behind me and some experiences that showed that:

    • the Orange switch never goes off
    • my Cisco switch never turns off
    • my storage server never goes down
    • the dell server and the four QuantaPlex nodes always restarted when those very short power outages occurred.
      After finding that there is no direct connection between "PDU A resists and PDU B produces these effects", I concluded that some equipment in both PDU A and PDU B never shut down Why this happens is quite simple. They have a lower energy consumption compared to the value they have available in the capacitors and are newer.
      The PDU would have cost me about 500 EUR, being like the current one with included management (the sockets are automatically closed/opened/restarted) and have a temperature sensor that can make decisions based on certain thresholds:

    • to send me an email

    • to make loud sounds
    • to stop the supply to consumers
      The UPS cost only 500 EUR (including the tray for the rack) because we opted for a refurbished UPS produced by the Eaton company, having 3000VA / 2700W (in the conditions where we currently consume up to 2.2KWh with the servers and another value considerable for cooling), having new batteries and a 12-month warranty. (the same product costs 3000 EUR new)

    @FlorinMarian said: not all customers have autostart set for their critical software

    Why would some customers not have Autostart set for their VMs? Why isn't Autostart the default?

    Some clients do not have autostart for their applications (game servers, blockchain peering), not for VMs. The VMs all start automatically, I made sure of that.

    Thanks!

    Tom

    Thanked by 1Not_Oles
  • emghemgh Member

    AutoBoot™

    Thanked by 1totally_not_banned
  • Not_OlesNot_Oles Moderator, Patron Provider

    Thanks for the additional details! Interesting to read! :) I wish more people posted with additional details. Thanks again!

    Thanked by 1FlorinMarian
  • CalinCalin Member
    edited June 25
  • KrisKris Member

    ITT: bro learned to get a APC Back-UPS Pro 1500 due to... running a data-center in Romania.

    I remember teaching Manny @ HostDime when he leased a penthouse abondoned UUNET DC in Florida why we needed the same. He learned and in time we ditched it (AC was too heavy for roof there) and got an abandoned aircraft hangar in Maitland. I coined the term DimeNOC, and we had our first DC. OC3 with TWTelecom, calling every week to upgrade the line... the good days.

    Anyaway, UPS sort of crucial, as generators take a few minutes, and most will be brownouts of 1-2 seconds due to grid design. Downtown Orlando's power grid during the summer 20 years ago was on par with a Romanian residential power grid on a good day I imagine.

    Put all the smaller stuff on separate smaller UPS, and 1 per server to ride it out.

    Good luck and may you become as stable as host_c and... well Calin one day.

Sign In or Register to comment.