Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


HostHatch Los Angeles storage data corruption (Was: HostHatch Los Angeles storage down) - Page 5
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

HostHatch Los Angeles storage data corruption (Was: HostHatch Los Angeles storage down)

12357

Comments

  • DPDP Administrator, The Domain Guy

    @LiliLabs said: Genuinely amazed there's no status page for HH yet.

    https://status.hosthatch.com/

    Thanked by 2Logano the_doctor
  • LoganoLogano Member

    Unfortunately, that this is by location and not by node, so they won't say a location is offline for the puny little 5% that lose connection. Pfffft

  • @DP said:

    @LiliLabs said: Genuinely amazed there's no status page for HH yet.

    https://status.hosthatch.com/

    This is only useful if an entire location goes down, practically useless IMO. Needs to be like how most providers do it, with status per node.

    Thanked by 1Logano
  • DPDP Administrator, The Domain Guy

    @LiliLabs said:

    @DP said:

    @LiliLabs said: Genuinely amazed there's no status page for HH yet.

    https://status.hosthatch.com/

    This is only useful if an entire location goes down, practically useless IMO. Needs to be like how most providers do it, with status per node.

    Yeah clearly that status page could use more details :smiley:

    Thanked by 3webcraft skorous Logano
  • @LiliLabs said:

    @DP said:

    @LiliLabs said: Genuinely amazed there's no status page for HH yet.

    https://status.hosthatch.com/

    This is only useful if an entire location goes down, practically useless IMO. Needs to be like how most providers do it, with status per node.

    Looks like ramnode are using 'Node Ping', while HostHatch are using 'Uptimerobot'

    Thanked by 2the_doctor Logano
  • @Logano said:

    @hosthatch said:
    we have to decide how much time we should spend sharing information as compared to actually solving it.

    Seeing as how you spent zero time "sharing information as compared to actually solving it", I guess you see zero importance in timely customer notification because

    we have several PB of raw storage at the moment, and these failures represent a very small % of it

    I guess this part says it all:

    @hosthatch said:
    Not sure why there needs to be a thread, but sure :)

    5 users at a location? Okay, I can agree. But 5%?

    You don't need to write up a 10 paragraph status notification for these things until you've tried to solve an issue and learned that you can't, at which point the customers deserve one since they'll have to clean up after your own failures (which you ended up doing here, I'm not criticizing that). A simple one-liner in your status page saying "We are investigating issues with some storage nodes on XXX" within 15 minutes of seeing the issue helps a lot. Then another one-liner update a few hours later. It takes 30 seconds, which is probably less time than you spend trying "to decide how much time we should spend sharing information....". Simply refusing to communicate will just bring frustrated people to LET since your support seems to be a big cobwebbed black hole.

    UPDATE: Examples from RamNode
    https://twitter.com/NodeStatus
    http://status.ramnode.com

    One might assume bringing these critical issues to the two head honchos would get some traction and some appearance of getting off their asses.

    Nope

    HostHatch will be the company that ruined two year prepayment deals for me. Any company I see that offers two year deals I'll think, "oh shit, they're looking to check out like HostHatch. Time to move on".

  • wpyuelwpyuel Member

    I have been unable to use LA for 5 days

    Thanked by 1foitin
  • Been down for a week or so - anyone else? VPS says it's running but won't boot...

    Hosthatch deadpool incoming?

    Uptime was good until now - no help from support, a VPS that won't boot. No status updates. Fuck you maybe?

  • risharderisharde Patron Provider, Veteran

    @sidewinder said:
    Been down for a week or so - anyone else? VPS says it's running but won't boot...

    Hosthatch deadpool incoming?

    Uptime was good until now - no help from support, a VPS that won't boot. No status updates. Fuck you maybe?

    Just assumed mine was up, I need to check, thanks for the heads up...

  • Daniel15Daniel15 Veteran
    edited May 2022

    @sidewinder said: Been down for a week or so - anyone else? VPS says it's running but won't boot...

    If you're just trying to connect using SSH, connect to VNC via HostHatch's control panel and see what it says? It's likely something is corrupted - either the kernel, the GRUB config, the fikesystem itself (fsck needed), etc. Post a screenshot of what it says.

  • @sidewinder said:
    Been down for a week or so - anyone else? VPS says it's running but won't boot...

    Hosthatch deadpool incoming?

    Uptime was good until now - no help from support, a VPS that won't boot. No status updates. Fuck you maybe?

    you may have to go fsck yourself

  • sidewindersidewinder Member
    edited May 2022

    @Daniel15 said:

    @sidewinder said: Been down for a week or so - anyone else? VPS says it's running but won't boot...

    If you're just trying to connect using SSH, connect to VNC via HostHatch's control panel and see what it says? It's likely something is corrupted - either the kernel, the GRUB config, the fikesystem itself (fsck needed), etc. Post a screenshot of what it says.

    "no setup signature found"

    Trying to boot into old kernels seems to work but every command i issue gives me a segmentation fault.

    Did anyone's VPS come back or was this a fake restore where literally every node is down and out?

  • DPDP Administrator, The Domain Guy
    edited May 2022

    @sidewinder said: Hosthatch deadpool incoming?

    Even if their entire LA location went poof, impacting both storage and non-storage VPSs, that doesn't mean they're deadpooling.

    That's just 1 site out of 14?

    @sidewinder said: Uptime was good until now - no help from support

    "No help" and "no response" are two totally different things.

    I'm expecting the latter to be the actual case - please correct me if I'm wrong.

    @sidewinder said: No status updates

    Ok, but this has already been highlighted somewhere in this thread and clearly it's something they'll need to work on and improve.

    @sidewinder said: Fuck you maybe?

    I know you're frustrated, but sorry, cursing won't revive your VPS so maybe have some respect for other human beings, especially to the ones who could actually help you in this situation.

    I have several services with them too and it's a known fact that their ticket response time is horrible so the best thing for you to do is either attempt to find out yourself what's going on, as suggested by @Daniel15, or just wait for them to respond to your ticket.

    No guarantees that they'll respond here, or even want to, but I'll just tag them for the sake of helping.

    @Emil @hosthatch

    ✌️

    Thanked by 2bdl bulbasaur
  • Daniel15Daniel15 Veteran
    edited May 2022

    @sidewinder said: Trying to boot into old kernels seems to work but every command i issue gives me a segmentation fault

    The segfault most likely means the program is corrupt and it's trying to execute invalid code or access an invalid memory location.

    If it's Debian or Ubuntu, does apt work? If so, you can use apt --reinstall install whatever to reinstall the program, which should give you a working version. If apt is corrupt then I'd guess you'd need to boot from a live CD and copy the corrupted apt files across manually. Once apt is working, you can reinstall any damaged apps.

    I don't know much about CentOS or other distros, sorry!

    In any case, your data is likely at least 1% corrupted (I found 1-5% corruption across all my data via checksum mismatches compared to backups), so I'd recommend restoring your latest backup. In my case I only back up the data and not the operating system, so I've also had to manually reinstall a bunch of programs.

    Rule #2 of low end boxes is to always have backups. Borgbackup is efficient enough that you can run it daily - it'll likely only take a minute or two to run, as it only has to back up the differences since the last backup.

    (rule #1 is "no more than $7/month")

    @sidewinder said: Did anyone's VPS come back or was this a fake restore where literally every node is down and out?

    They restored it inasmuch as the VPSes power on now. Previously they weren't even doing that. However, it seems like many of them are corrupted. As far as I'm aware they haven't attempted any data recovery on the corrupt data.

  • MikePTMikePT Moderator, Patron Provider, Veteran

    @hosthatch hey, there seems to be several complaints with no reply at all. Do you guys have any information you can share?

    Thanked by 1ElonBezos
  • joelbyjoelby Member

    @sidewinder said:
    Trying to boot into old kernels seems to work but every command i issue gives me a segmentation fault.

    Did anyone's VPS come back or was this a fake restore where literally every node is down and out?

    This is essentially what happened to me (though I couldn't even get it to boot). You can use your Linux distribution's package manager commands to verify system packages - most likely some libraries are corrupted. If you can't even run the package manager tools, boot from a rescue distribution, and then use tools like md5sum/sha1sum to compare checksums on important files and then copy them over. It's easiest to do this if you have another system (e.g. a VM on your computer) running the same distribution and package versions as the broken system, though in general downgrading libraries by a few minor versions won't cause any problems.

    Once your package manager is working, you can then run something like dpkg --verify or rpm -Va, which will check your installed files against the checksum files. If any packages report errors, reinstall them. If your distribution's checksum files are missing/broken for any packages, reinstall those packages anyway.

    Once the OS is back up and running, you'll probably have a similar issue with your data files, but unlike system packages there's not much you can do about those unless you have backups of the data elsewhere.

    I'm not sure if Hosthatch will provide this level of support and reconstruction - I think other people in this thread took the DIY approach.

    Thanked by 1Daniel15
  • Daniel15Daniel15 Veteran
    edited May 2022

    @joelby said: I'm not sure if Hosthatch will provide this level of support and reconstruction

    Honestly I'd be surprised if any unmanaged host provided this level of support. Some (including Hosthatch) might do it as a one-off courtesy, but the cost to a low-end host in terms of support hours would end up costing more than any revenue gained from the customer, so it's not really sustainable if a lot of people also ask for the same thing.

    The idea with low-end hosting is that you're intentionally giving up some amount of support in exchange for cheaper hosting. This applies even moreso in HostHatch's case since their sale prices are a lot lower than many other hosts. For people not comfortable with the low-end-lifestyle, I'm sure the $50/m fully managed VPS providers would be happy to have them :tongue:

    I'm not defending HostHatch or saying their approach is good - they really should be more active in terms of messaging and replies on this thread. I'm just saying that it's somewhat expected to get less hand-holding and less support when things go wrong the lower in price you go, in general across all low end hosts.

  • @MikePT said:
    @hosthatch hey, there seems to be several complaints with no reply at all. Do you guys have any information you can share?

    At this point, @hosthatch should be in damage control mode and bring his ass here to "face the music". Right now, it's not looking good at all.

    We've got people (including myself) considering other options when the 2-year renewal falls due, and this thread is proving to show a good case for letting it expire.

    Thanked by 3MikePT foitin msallak1
  • @DP said:

    @sidewinder said: Uptime was good until now - no help from support

    "No help" and "no response" are two totally different things.

    I'm expecting the latter to be the actual case - please correct me if I'm wrong.

    A quarter after 2020's BF I gave away my account with them because of having many small problems here and there but not getting a support reply (or just a useless one like "We're looking into it" without something that has changed in the end). Last BF they promised everything has improved so I gave them another try and it first appeared to be running better but now there's again a problem here and another one there (not storage line) and support is again non existent. This was my last try. Using their service is just a waste of time for me now even though it's cheap and if it's running, it's a good service. The latter one is timely limited unfortunately and support abilities to restore this state are too limited to make it any fun (in terms of usefulness) to use their services.

  • webdevwebdev Member

    mine is still working

  • MumblyMumbly Member
    edited May 2022

    @Daniel15 said: I'm just saying that it's somewhat expected to get less hand-holding and less support when things go wrong the lower in price you go, in general across all low end hosts.

    I am afraid we have different idea of what "slow support" and "hand-holding" mean.

    1. 3 days of waiting for rDNS change as example is slow support
    2. help request with my application, my script, my setup, etc ... as example is hand-holding

    But that's not the case with hosthatch company.

    Weeks of downtime because of their broken system, broken control panel, and so on without a single support ticket response is not just slow support. It's pure ignorance, incompetence, incapability to manage own system, to manage own userbase, to provide them what the paid for.

    Once THEY fuck up you're pretty much on your own. When they fuck op, not when you fuck up.
    You may be able to put OS back online, able to reinstall system. Or you may not be able, is out of your control because of their broken control panel. It doesn't matter. You're own your own and you're still paying for it.

    You may find yourself in a situation with hosthatch where control panel wont let you to reinstall operating sistem. You will open support ticket. And there won't be a single response for a several weeks. This means several weeks of downtime you're still paying it. And that's not slow support response, that's just rip off.

    Some people says "for what I am paying they are good". I am wondering what those same people will say next time when they won't be able to use already paid but suddenly inaccessible VPS for a month or more and hosthatch won't give a shit about it. Literally won't give a shit, because they won't bother to respond even simple support ticket inquiry.

    Thanked by 2Logano quicksilver03
  • skorousskorous Member

    @Samael said: I am afraid we have different idea of what "slow support" and "hand-holding" mean.

    Considering that the context of your quote was someone discussing whether or not HH would help them rebuild their OS, I think your definitions line up pretty well. It definitely would've been hand holding. Glad we're on the same page.

  • caracalcaracal Member
    edited May 2022

    I was moved to cloud.hosthatch.com and effectively have less disk space (around 800GB less) now due to a difference in calculation of 10TB.

    root@la:~# df -h
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/vda1       8.9T  1.8G  8.9T   1% /
    

    Compared to

    Disk       : 9.7 TiB
    

    On the other hand, I got a reply in 5 days. Not too bad.

  • MumblyMumbly Member
    edited May 2022

    Nah, I quoted directly "I'm just saying that it's somewhat expected to get less hand-holding and less support when things go wrong the lower in price you go, in general across all low end hosts" and responded that instead of less hand-holding and less support, there isn't any hand-holding or any support even when host fuck up and solution depend solely from them.
    This storage data corruption is just one of many hosthatch fuck-ups if you read this forum and my response isn't limited to the present one.

  • ralfralf Member

    @Daniel15 said:
    I'm not defending HostHatch or saying their approach is good - they really should be more active in terms of messaging and replies on this thread. I'm just saying that it's somewhat expected to get less hand-holding and less support when things go wrong the lower in price you go, in general across all low end hosts.

    From the sounds of it though, people's disk images are corrupt and it's being blamed on a known RAID controller issue. With no real confidence that the disk corruption won't happen again, it seems less about people asking for support and more about being hosted on a server that actually works.

    Thanked by 1risharde
  • Daniel15Daniel15 Veteran

    @ralf said:

    @Daniel15 said:
    I'm not defending HostHatch or saying their approach is good - they really should be more active in terms of messaging and replies on this thread. I'm just saying that it's somewhat expected to get less hand-holding and less support when things go wrong the lower in price you go, in general across all low end hosts.

    From the sounds of it though, people's disk images are corrupt and it's being blamed on a known RAID controller issue. With no real confidence that the disk corruption won't happen again, it seems less about people asking for support and more about being hosted on a server that actually works.

    Yeah I absolutely agree with you.

  • ralfralf Member

    @caracal said:
    I was moved to cloud.hosthatch.com and effectively have less disk space (around 800GB less) now due to a difference in calculation of 10TB.

    root@la:~# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/vda1 8.9T 1.8G 8.9T 1% /

    Have you tried using "tune2fs -m" to reduce the amount of reserved space in the filesystem? Personally I'd never set it below 1%, but it's up to you. The default is 5%, which probably accounts for some of the difference, and the rest is probably 1000 vs 1024 in the calculations...

  • risharderisharde Patron Provider, Veteran

    TRIMMED

    @risharde said: Fuck you maybe?

    I know you're frustrated, but sorry, cursing won't revive your VPS so maybe have some respect for other human beings, especially to the ones who could actually help you in this situation.

    @DP I didn't say that - so either my account here is hacked or you messed up the quoting. Please confirm or point me to the post where I said this.

    Thanked by 1DP
  • DPDP Administrator, The Domain Guy
    edited May 2022

    @risharde said:
    TRIMMED

    @risharde said: Fuck you maybe?

    I know you're frustrated, but sorry, cursing won't revive your VPS so maybe have some respect for other human beings, especially to the ones who could actually help you in this situation.

    @DP I didn't say that - so either my account here is hacked or you messed up the quoting. Please confirm or point me to the post where I said this.

    Sorry but I probably quoted your quote, LOL.

    I'll update that.

    EDIT: Fixed and thanks for pointing it out :smiley:

    Thanked by 1risharde
  • risharderisharde Patron Provider, Veteran

    @dp thanks, I flagged it because I wasn't sure you were on. Please see flag for details if necessary. Thanks for adjusting.

Sign In or Register to comment.