Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


I need advice! I might of messed my server up.
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

I need advice! I might of messed my server up.

MagniPhiCatMagniPhiCat Member
edited January 2022 in Help

Hi there!

Our main (well, only) server crashed (partially, but still... no ssh, no apt-get, no rsync, no emails... fortunately web is still ok, but for how long?) and now we are looking to urgently create a new, more agile, infrastructure (spoiler alert: I am not a sys admin, please be tolerant...)

I contacted NexusBytes because I have read here that they are very nice people always happy to help, and that they reply in a matter of minutes. And I like kind people :)

But now almost 2 days have passed and I sent a couple messages, but still no answer.

Do you think we should look into another provider? This one was so promising... :(
I know very well they absolutely don't owe us an answer quickly, or even at all...

... But this is somewhat urgent because our partially-crashed server has been badly impacting our customers for 2 days now (yes, I also know it is on us because we didn't have a backup server with everything setup on it... I'm precisely trying to make things better).

Our needs:
- several small servers around the globe
- each server with its own task/responsibility (mail, database, files, web...)
- for a website with around 5k users / day worldwide
- budget of 60-80$ per month (for around 8 servers)
- of course: root ssh access, dedicated IP, port 25 for the mail server (on which I will install MailCow)
- if possible: guaranteed/dedicated CPU

Followup question: we need the different servers (worldwide) to share the same storage space/disk/server (I don't know which term applies). I am having a (very!!) hard time finding information on this kind of setup. How should we proceed so that all webservers share a "network-drive" (or something, instead of just a local path on the local drive) to save user-uploaded-files on it? (without needing each webserver to have 500GB of storage and to rsync everything both ways between all webservers)

Should we create something like a private network? I remember having seen this in one of my previous jobs. But I don't know how it's done and more importantly I don't know if it's even possible across multiple providers (in case we have one provider for the EU and another for the USA).

Any link to documentation/tutorials/tips on the topic would be so very welcome!

Thank you kindly in advance :)

Best,

R.

«13456

Comments

  • @seriesn dear, someone is looking for you.

    Thanked by 1NobodyInteresting
  • Thank you both for the ping :) but I don't want to be someone invading the tranquility of people (that's why I didn't send Jay a message here on LET when I first contacted NexusBytes through their website).

    Any advice for the other points? Maybe a popularization website of infrastructure setups matching my initial idea? I try to dabble at sys admin to help my client, but it is not my core job (at all). Anything too convoluted/complex would be hard for me to grasp I fear.

  • yoursunnyyoursunny Member, IPv6 Advocate
    edited January 2022

    If it's urgent, go with Azure, which guarantees immediately availability.
    When you have time later, you can move back to cheap servers.

    billions per hour

  • jarjar Patron Provider, Top Host, Veteran

    If the server is up and you can't fix the software on it, you really do need a managed VPS. It's the only way you're going to be able to sleep at night.

  • "a new, more agile, infrastructure"

    So what old, less agile, infrastructure are you currently using?

    You may have too high expectations for your budget

    (Congrats on your first post)

    Thanked by 2bdl hyperblast
  • yoursunnyyoursunny Member, IPv6 Advocate

    @jar said:
    If the server is up and you can't fix the software on it

    I have an OpenVZ server that still responds to ping and SSH port is open, but I lost the SSH private key and password authentication was disabled.
    Moreover, I don't have any access to the Virtualizor account.
    Is this server still "up"?

  • FAT32FAT32 Administrator, Deal Compiler Extraordinaire

    Try other providers, NexusBytes was great but I doubt it is still so

  • @yoursunny said:
    If it's urgent, go with Azure, which guarantees immediately availability.
    When you have time later, you can move back to cheap servers.

    Thank you for the feedback! Not sure I want to do it all over again next week though... ^^

  • @jar said:
    If the server is up and you can't fix the software on it, you really do need a managed VPS. It's the only way you're going to be able to sleep at night.

    Thank you for the input, I'll look into it but I'm not sure it will be in the same price-range.

    I don't mind doing some sys admin even if it's not my job, but having a somewhat disposable and self-contained small server (separation of concerns) is a peace of mind too. At least, I think so.

    But it doesn't matter what I think, actually, my client asked me for this kind of new setup...

  • lentrolentro Member, Host Rep

    @MagniPhiCat said: But I don't know how it's done

    It seems like going to a managed provider would be better based on what seems like a lack of knowledge. But, you're running a tech operation -- do you happen to have a tech team (or single developer), and if so, could you get your tech team to hop on this forum? Otherwise, I doubt the help and suggestions that people post will be helpful.

    Thanked by 1bdl
  • @angstrom said:
    "a new, more agile, infrastructure"

    So what old, less agile, infrastructure are you currently using?

    A dedicated server with 8 cores and 32GB ram. Everything runs on it: custom-made mail server, database (mariadb), redis, web (nginx/php), and a few other things.

    My client is now asking for a more compartmentalized infrastructure.

    You may have too high expectations for your budget

    What do you mean? I am genuinely interested, because sys admin is not my job. But if you're knowledgeable about sys admin, then I'm sincerely all ears, eager to learn things!

    This is the budget my client gave me, not much of a choice here either.

    But I was thinking it would be enough...

    (Congrats on your first post)

    Thanks :smiley:

  • @FAT32 said:
    Try other providers, NexusBytes was great but I doubt it is still so

    Thank you for the reply!

    What do you mean by that? I thought I did read a few topics praising NexusBytes here. It is also (very) well rated on both HostAdvice and TrustPilot (although I did not look deep about which reviews are recent and which aren't)

    Which providers are you advising me?

    Oh... And... Would you happen to have anything for me regarding my last question (in my original post)? Since you are an admin here, surely you know a thing or two! :blush:

  • In the worst case, no chance of rebooting the server?

    You make it sound as though there were no chance of fixing things, at least temporarily

  • @lentro said:

    @MagniPhiCat said: But I don't know how it's done

    It seems like going to a managed provider would be better based on what seems like a lack of knowledge.

    Indeed it is, no offense taken :)

    But, no matter where I go, nobody seems able/inclined to give me the proper directions to find tutorials on the topic.

    I might not use the proper keywords, but my googling did not yield anything.

    But, you're running a tech operation -- do you happen to have a tech team (or single developer), and if so, could you get your tech team to hop on this forum? Otherwise, I doubt the help and suggestions that people post will be helpful.

    I am the tech guy. The lone one :) I am a web dev, and I dabble at sys admin to help my client. I previously manually configured the mail server from scratch, following tutorials, and it went well! But as you said, I lack some knowledge. But if you're not using too complicated tech words, I might very well understand...?

    Again, not sure a managed hosting would be good, because it would cost too much I think.

    Thanked by 1lentro
  • DataIdeas-JoshDataIdeas-Josh Member, Patron Provider

    @MagniPhiCat If welcome to work with a couple of different providers. Here is the top 4 on my list of providers

    Myself. DataIdeas - https://my2.dataideas.com
    @Francisco - BuyVM https://buyvm.net
    @seriesn - NexusBytes https://nexusbytes.com/
    @jonathanspw KnownHost https://www.knownhost.com/

    Otherwise I would say look into Digital Ocean/Linode/Vultr

  • @yoursunny said:

    @jar said:
    If the server is up and you can't fix the software on it

    I have an OpenVZ server that still responds to ping and SSH port is open, but I lost the SSH private key and password authentication was disabled.
    Moreover, I don't have any access to the Virtualizor account.
    Is this server still "up"?

    As you describe it, it's still up, but you've lost any way of accessing it

    But the OP's issue appears to be different

  • @angstrom said:
    In the worst case, no chance of rebooting the server?

    You make it sound as though there were no chance of fixing things, at least temporarily

    No believe me there is not. I won't touch the server now, because it is barely standing on its trembling feet right now. Rebooting it would only give it a chance to make things worse (e.g. reloading a lib that was broken beforehand) and to break the only thing that is still standing: the website.

    The only thing I can do, is access it using the (lame) OVH console (iKVM or something). But everything is broken because I messed up some upgrade...

    If I were a sys admin guru, I might (or might not) find some fix. But honestly, with glibc broken, I don't have any slight hope. I'm just waiting for the new infrastructure, and then I delete this server :(

  • @angstrom said:

    @yoursunny said:
    Is my server still "up"?

    As you describe it, it's still up, but you've lost any way of accessing it

    But the OP's issue appears to be different

    Yes. Because glibc is broken, I can no longer:

    • use SSH
    • use rsync
    • use apt

    I can only use the OVH iKVM console, which is a pain to use, and hope that I can zip some files and download them using HTTP as a last resort backup. Really, nothing more.

  • @MagniPhiCat said:

    @angstrom said:
    In the worst case, no chance of rebooting the server?

    You make it sound as though there were no chance of fixing things, at least temporarily

    No believe me there is not. I won't touch the server now, because it is barely standing on its trembling feet right now. Rebooting it would only give it a chance to make things worse (e.g. reloading a lib that was broken beforehand) and to break the only thing that is still standing: the website.

    The only thing I can do, is access it using the (lame) OVH console (iKVM or something). But everything is broken because I messed up some upgrade...

    If I were a sys admin guru, I might (or might not) find some fix. But honestly, with glibc broken, I don't have any slight hope. I'm just waiting for the new infrastructure, and then I delete this server :(

    Ah, so the current situation is the result of a (perhaps customized?) upgrade that went wrong. Yes, this could be tricky.

    Thanked by 1MikeA
  • deankdeank Member, Troll

    The end is nigh.

    I've yet to detect PMS.

  • @DataIdeas-Josh said:
    @MagniPhiCat If welcome to work with a couple of different providers. Here is the top 4 on my list of providers

    Myself. DataIdeas - https://my2.dataideas.com
    @Francisco - BuyVM https://buyvm.net
    @seriesn - NexusBytes https://nexusbytes.com/
    @jonathanspw KnownHost https://www.knownhost.com/

    Otherwise I would say look into Digital Ocean/Linode/Vultr

    Thank you Josh for the suggestion!

    Thanked by 1DataIdeas-Josh
  • @MagniPhiCat said:

    @angstrom said:

    @yoursunny said:
    Is my server still "up"?

    As you describe it, it's still up, but you've lost any way of accessing it

    But the OP's issue appears to be different

    Yes. Because glibc is broken, I can no longer:

    • use SSH
    • use rsync
    • use apt

    I can only use the OVH iKVM console, which is a pain to use, and hope that I can zip some files and download them using HTTP as a last resort backup. Really, nothing more.

    How did glibc break?

  • MagniPhiCatMagniPhiCat Member
    edited January 2022

    @angstrom said:
    Ah, so the current situation is the result of a (perhaps customized?) upgrade that went wrong. Yes, this could be tricky.

    Precisely :) (I am using a smiling face here, but I have been quite stressed these last couple days...)

    I tried to upgrade openssl following a (quite old) tutorial, and failed miserably. Fortunately I have a database slave with real-time backup, and a very recent files (user-uploads) backup. But I still have a few things to (try to) back up before losing the server.

    And for the last couple of days, I have been waiting for an answer from NexusBytes, refreshing my email box every 10 minutes, and researching VPS providers in the mean time.

  • @deank said:
    The end is nigh.

    I've yet to detect PMS.

    Sorry I am not English. What does it mean?

    Thanked by 1boka003
  • @MagniPhiCat said:

    @angstrom said:

    @yoursunny said:
    Is my server still "up"?

    As you describe it, it's still up, but you've lost any way of accessing it

    But the OP's issue appears to be different

    Yes. Because glibc is broken, I can no longer:

    • use SSH
    • use rsync
    • use apt

    I can only use the OVH iKVM console, which is a pain to use, and hope that I can zip some files and download them using HTTP as a last resort backup. Really, nothing more.

    Sounds more like a Linux issue more than a hosting company issue. Can you use VNC to log into the terminal and try to boot it into single mode or something to troubleshoot the broken Linux install?

    If not, if the host provides a way to boot from ISO or open a ticket and have them attach an ISO for you to boot, you might be able to rescue your system that way.

    As the old adage goes, always do backups, in fact backup the backup, this holds especially true for production boxes.

  • @angstrom said:
    How did glibc break?

    When trying to upgrade openssl so that I could install Zabbix agent.

    Silly me...

  • MikeAMikeA Member, Patron Provider

    @MagniPhiCat said:

    @angstrom said:
    Ah, so the current situation is the result of a (perhaps customized?) upgrade that went wrong. Yes, this could be tricky.

    Precisely :) (I am using a smiling face here, but I have been quite stressed these last couple days...)

    I tried to upgrade openssl following a (quite old) tutorial, and failed miserably. Fortunately I have a database slave with real-time backup, and a very recent files (user-uploads) backup. But I still have a few things to (try to) back up before losing the server.

    And for the last couple of days, I have been waiting for an answer from NexusBytes, refreshing my email box every 10 minutes, and researching VPS providers in the mean time.

    What OS were you using? If you deploy a new server and migrate you should use the latest LTS or use something like AlmaLinux instead of you're using an older RHEL based OS.

    If one of my plans work for your needs and are within your budget I'll give you a discount on the first month on any server to help you get things back up and running well.

  • @deltatux said:
    Sounds more like a Linux issue more than a hosting company issue.

    It is a man-issue ^^ (mine)

    If I'm here it's not because I need help fixing glibc. It's because my client asked me to create a more compartmentalized infrastructure.

    The current hosting company has nothing to do with my misfortune. Forgive me if it's what (wrongly) transpired from my original post, it was clearly not intended at all.

    Can you use VNC to log into the terminal and try to boot it into single mode or something to troubleshoot the broken Linux install?

    I can use iKVM yes. But if I reboot the machine, I (immensely) fear the web server will not start again. And since it's still working right now, and I still don't have the new infrastructure, I prefer not to touch anything.

    If not, if the host provides a way to boot from ISO or open a ticket and have them attach an ISO for you to boot, you might be able to rescue your system that way.

    As the old adage goes, always do backups, in fact backup the backup, this holds especially true for production boxes.

    I have backups of the database and of the user-uploaded files. Not from the system itself. Which is probably my fault, as I am not knowledgeable concerning sys admin... I should have done something like a "system restore point" like on Windows 🤷‍♂️

    Anyway I'm here to find a new infrastructure, and, if possible, advice on where to find tutorials on the infrastructure setup I would like to do (mostly: having a shared "drive/server" between the geographically distant web-servers), because I can't seem to find anything on my own.

  • @MikeA said:
    If one of my plans work for your needs and are within your budget I'll give you a discount on the first month on any server to help you get things back up and running well.

    Sorry Mike, I cannot access your website, it is telling me that my IP is blocked...

Sign In or Register to comment.