Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


I need advice! I might of messed my server up. - Page 5
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

I need advice! I might of messed my server up.

1235

Comments

  • I have an OpenVZ server that still responds to ping and SSH port is open, but I lost the SSH private key and password authentication was disabled.
    Moreover, I don't have any access to the Virtualizor account.
    Is this server still "up"?

    If a server falls in a DC, and there’s no one around to hear it, does it make a sound?

    Thanked by 2grep yoursunny
  • grepgrep Member
    edited January 2022

    @MagniPhiCat said:

    Your intervention really is golden. Keep up the good work!

    The irony in you saying that to a stranger on the internet when your client should be saying that about YOU (when it is actually way more accurate) is insane. I can't believe you keep saying the shit you do and insist that anyone else who says otherwise (especially with mean words T_T) is wrong.

  • @MagniPhiCat said: 300 messages / hour

    iirc..Thats quota per email account not the whole mxroute account/domain

  • jsgjsg Member, Resident Benchmarker
    edited January 2022

    First, while seeming to be unnecessarily harsh and unfriendly, some criticizm directed at you isn't totally off. I myself also had thoughts along the lines of "$80? What a generous budget /sarcasm off" in my mind. And frankly, I also think that you've allowed your client to put you in a quite ugly position. Your client would be well advised - and seems to need being reminded of that - that there is a relationship between desires and cost. In this case you are the "poor idiot" who has to take a lot of work on himself to somehow balance your clients unbalanced desires vs costs desires. Just saying...

    @MagniPhiCat said:

    Because that' a rather loose and superficial "group" term behind which there are quite many and diverse actual technologies and even paradigms.

    I understand. Would you happen to have a link to a good website of your knowing explaining all these solutions? 🙏 It would allow me to learn many things and better know what I am talking about and doing.

    Sorry, no. It took me years to arrive where I'm now and I don't know a web site where you can quickly gain even a moderately adequate overview of that complex field. Plus, frankly, if I wanted to be a teacher I'd have become one, but I think that I'm quite patient with you and honestly try to guide you through the maze.

    Step 1 (asap): USA only (70% of global traffic is in eastern USA)

    To be fair, you'll likely get more valuable advice on those from Americans. I'm a rather Europe centric person (although I benchmarked quite a few asian providers/VPS). But ask experienced LET users

    That's what I did in my original post... 😭

    I'm cautious because that's your main location and server, but if you push me, well I'd pick a typical european route and simply get a Contabo VDS in their St. Louis location which is the best, especially in terms of connectivity (I benchmarked all of them extensively). But again, be aware! There quite likely will be some saying that they know far better and or (equally good but) cheaper providers in the USA - and some of them will be damn right.

    If you're Europe centric, what do you think of netcup? It is just perfect (dedicated resources, very low prices), except for the scores on the websites that you don't want to hear about :)

    (a) it's not about not wanting to hear about; it's about knowing that the vast majority of words there simply amount to BS and nonsense.
    (b) Yes, netcup would be very high up on my list for Europe. "Perfect"? No; I do not know a perfect provider. But they are damn attractive, very established, and offer very decent products at almost insane prices.

    Did you say that hostinger was bad? They also have datacenters in Europe.

    No, I did not. But then why would I choose a non-european provider for Europe when there are plenty providers from here?

    Frankly, get rid of that [email] trouble and have it handled by someone who does it professionally since years. AFAIC there is not even a question, simply wait for @jar's response and be done.

    My client said that 300 messages per hour is not enough for their need.
    I cannot used MXroute or NexusBytes for that reason.

    Then ask @jar if he has a product for massive users - but be sure to provide evidence that your client isn't a spammer (frankly, if I were asked for a mail service sending more than max. a couple of dozen emails per hour, I'd simply turn around and walk away without even discussing or listening, but as jar is in that field he might have a solution for you).
    Btw, I'd also turn away because your client seems to be the kind who wants a lot for pennies.

    • one database master server (mariadb ; database is 2GB without log/event tables) (I already have a slave, that I will have to reconfigure obviously)

    That's a baby database. No worries. Just put it on a VPS with plenty of memory and a decent NVMe. As it's highly likely also the back end of your web server I'd put both of them on one decent VDS (or very good VPS). Something like 4 vCores, 8 or 16 GB memory and 50ish GB (decent!) NVMe should be a good starting point.

    4 dedicated cores and 16GB of RAM for the DB? But... Today I'm not even using 3.5GB for the whole server (that includes the mariadb database, redis, the mail server, and the web server). At least that's what htop is telling me. clamd (for checking uploaded files) being the most RAM hungry process (or so it seems)

    (a) I said "or 16 GB", not "16 GB min".
    (b) memory translates to caches of all kind and big ones, so it translates to faster DB results, faster web responses, etc.
    Sure, it'll highly likely work with 2vCores and 2 GB too, but I remember reading about Redis and CDN and "fast" in your posts ...

    Also 16GB memory for a single nginx instance? Please see my answer above.
    It really seems waaaayyy oversized to me... 🤔

    Well, then buy a smaller one.

    Step 2 (in a few weeks): EU + Asia

    Basically repeat and rinse the above - with a but: Does your web stuff need synchronization? If yes then the whole job gets trickier.

    What do you mean synchronization? Related to what?

    Example: a community site spread over 3 servers. If you do not somehow synchronize them then users will see different sites, comments will be visible on one site but not on others, etc.

    • (CloudFlare?) as a DNS with geo-routing
    • (CloudFlare?) as a CDN for accessing (i.e. retrieving using HTTP) media from the NFS

    My advice: stay away from CDNs unless you really really need it.

    Why? They offer a great boost of speed for far away visitors, no?

    (a) so does a fast server with good connectivity, and (b) you already (want to) have 3 server spread over the planet and if your servers and setup aren't crappy that should be easily good enough; after all we're not talking about a FinTech site, right? (c) CDN also add complexity and problems, plus very often they are but "cosmetics" that is, used to cover up a slow site that should be optimized in the first place. Keep it simple.

    • maybe adding a database slave in each geo region for read queries

    Don't get me wrong but your approach sounds as if you read too many "how real pros do it" articles ...

    That's the problem... I didn't. It's just things I heard here and there. It is wrong to get the database closer, for faster read queries?

    No, not at all - but it risks to be a Pandora's box, especially for less experienced sysadmins.
    Quick question: how "live" are your sites? Is it OK when say the European site reflects the state (of the USA main server) with some delay, be it 1 second or a minute or anything in between?

    What are we talking about in terms of requests per second and avg. and max connected users? 10? 50? 500? 5000? Or what? Keep in mind that all those "nice" "add a load balancer here and a redis there" ideas add complexity and for someone with your knowledge level (sysadmin) complexity is the enemy.

    I don't have all the numbers right here... I'll check this when I can, and tell you :)
    In general, I try to put as much cache as possible. But there are still like a dozen very small queries per page that I cannot easily remove.

    Those numbers are decisive.

    Uploaded files will be centralized (just like they are today, actually... but then a CDN will allow for better latency).

    Oh, why not adding in anycast plus a distributed file system plus some flying frogs, too?
    KEEP IT SIMPLE. And always ask "what for? Do we really need that and what will it bring to the table?"

    Haha ^^
    It will bring (much) faster load times of media (static) files for people who are far away from the NFS server (roughly 30% of users)

    So do cheap regional mirrors or caches.

    My client wants a server in each of the three regions. The only possibility is to use a DNS with geo-routing.

    Good luck and have fun with geo-DNS

    Thanked by 1AlwaysSkint
  • @MagniPhiCat said:
    Who am I, and who is anybody for that matter, to judge my client's orders? 😐

    I think this is a big part of the problem here.
    Any consultant, employee, friend or whatever that does not question a client when they are wrong will find themself in a shitload of problems later on.
    If your clients demands are unreasonable, it is up to you to make your client understand this. If you don't, you will find yourself in a position where it is impossible for you to make your client happy.
    It seems like your clients have made some impossible demands, and now you try to live up to them which you never will be able to do which means you can not win.

    I promise you, any consultant worth a shit will frequently judge and question their clients, and most likely often simply tell them "no".

  • @MagniPhiCat said:

    @lentro said:

    @MagniPhiCat said: But I don't know how it's done

    It seems like going to a managed provider would be better based on what seems like a lack of knowledge.

    Indeed it is, no offense taken :)

    But, no matter where I go, nobody seems able/inclined to give me the proper directions to find tutorials on the topic.

    I might not use the proper keywords, but my googling did not yield anything.

    But, you're running a tech operation -- do you happen to have a tech team (or single developer), and if so, could you get your tech team to hop on this forum? Otherwise, I doubt the help and suggestions that people post will be helpful.

    I am the tech guy. The lone one :) I am a web dev, and I dabble at sys admin to help my client. I previously manually configured the mail server from scratch, following tutorials, and it went well! But as you said, I lack some knowledge. But if you're not using too complicated tech words, I might very well understand...?

    Again, not sure a managed hosting would be good, because it would cost too much I think.

    First of all: Please don't take this as an offense, mate. I am a Webdev myself and I just had to laugh, because this situation reminds me of the job offers from many Startups, where it says you will be able to grow fast and have lots of responsibility. Fast forward when you are hired: You are the one guy for EVERYTHING. You applied as a webdev, but now, please manage Linux servers. We don't want to pay for a sysadmin, you see.. not saying it's like that in your company, but this happens all too often.
    Regardless, with the challenges thrown at you, you'll learn a lot, I am sure. Still, a managed vps would probably be the option to go to. Otherwhise, people here are usually happy to help, so you could try all the tech you need and see what people say. There's also @Cloudcone who offer managed vps even for their small instances from 5$/mo iirc. Afaik you can just open tickets and ask them to install/configure whatever software and they'll try to help :)
    They also have a "managed" option that costs extra where the vps is actually managed and proactively updated etc. But if it's just configuring mail etc, perhaps that'll also work.
    @mrTom also is known for offering managed vps. He's a good guy :)

    Thanked by 2yoursunny Cloudcone
  • @MagniPhiCat said: netcup: not well rated at all (on HostAdvice and TrustPilot), but excellent prices (dedicated resources only for 2nd tier servers) (only EU) (although I did read some high praise here I think)

    A number of us use and like netcup: they are a good, reliable provider.

    This said, netcup are very careful about who they accept as their customers, sometimes doing extensive verification of personal details, and not everyone likes this.

    Furthermore, netcup are very strict about contracts, and (with few exceptions) a customer needs to cancel their contract at least 31 days in advance, and not everyone likes this. (So for the most part, no last-minute cancellations are possible.)

    A glance at TrustPilot suggests to me that many of the complaints concern one of these points and much less the technical, server side of netcup. (Even so, they have an overall rating of 3,9/5, which is better than I would have anticipated.)

    Anyway, I'm inclined to think (as others and I have already said) that you need a fairly quick interim solution before deciding on a long-term solution, and (I suspect) that something like Hetzner Cloud would be better suited than netcup for this.

    Alternatively, for an interim solution, just stay with your current provider (whoever that is): just get another server and move things over.

    (But, yeah, I know that you want a nice dinner and dessert quickly ...)

  • NeoonNeoon Community Contributor, Veteran

    @MagniPhiCat said:
    I can't choose my budget... :(

    But you can choose the right clients.

    Thanked by 1yoursunny
  • angstromangstrom Moderator
    edited January 2022

    @Neoon said:

    @MagniPhiCat said:
    I can't choose my budget... :(

    But you can choose the right clients.

    True, but in this case, I would be inclined to blame @MagniPhiCat .

    It sounds like things were working satisfactorily on the one dedicated server until @MagniPhiCat decided to experiment on the system without having a full backup.

    In this situation, the client can rightfully expect @MagniPhiCat to (try to) repair the damage done.

    As for further development (multiple servers, customization, etc.), the client's budget is by all signs too small, but it's not clear (to me) whether these ideas really come from the client themselves or rather from (a kind of wishful thinking on the part of) @MagniPhiCat . (Basically: "if I have multiple servers across the planet, then if I mess up one, there will still be others".)

    Thanked by 1bulbasaur
  • chorongchorong Member
    edited January 2022

    I normally just lurk on LET, but this thread is just.. something.

    @MagniPhiCat - I don't know why you're trying to set up a "new, more agile, infrastructure" when your current server could crash at any minute. You should absolutely follow what @jsg said and keep your server running, but be aware that at any moment that something on the NexusBytes side could cause the server to reboot leaving you with... not much at all.

    It seems like server management isn't even in your remit to begin with. Offloading management to someone more experienced might be the play, or picking an unmanaged VPS with the option to buy managed hands for an hour or two to upgrade things might be the more cost effective measure.

    What I don't get is this:

    But this is somewhat urgent because our partially-crashed server has been badly impacting our customers for 2 days now

    Customers are badly impacted, yet no-one seems to care that the server is hanging precariously by a thread?

    Take this free consulting advice if you want:

    Focus on rebuilding your service first so that customers don't have to be impacted while you plan, buy, and set up your distributed frontend nodes.

    Once service is back to normal / stable, set up your backups to somewhere that isn't the same server. When that's all done, start looking into your $80/m 4 VPS node architecture.

    Clients are not always right, they'll always want the "north star". You need to set some boundaries around your responsibilities and guide them in the right direction

  • yoursunnyyoursunny Member, IPv6 Advocate

    @angstrom said:
    As for further development (multiple servers, customization, etc.), the client's budget is by all signs too small, but it's not clear (to me) whether these ideas really come from the client themselves or rather from (a kind of wishful thinking on the part of) @MagniPhiCat . (Basically: "if I have multiple servers across the planet, then if I mess up one, there will still be others".)

    Let's mess up the database server.
    Website is down instantly.

    Let's mess up the NFS server.
    Website is down instantly.

    Having more servers without rewriting the app in a fully distributed fashion would not solve any problem.
    It would only make the problem worse because every page view depends on three servers instead of one server.

  • @yoursunny said:

    @angstrom said:
    As for further development (multiple servers, customization, etc.), the client's budget is by all signs too small, but it's not clear (to me) whether these ideas really come from the client themselves or rather from (a kind of wishful thinking on the part of) @MagniPhiCat . (Basically: "if I have multiple servers across the planet, then if I mess up one, there will still be others".)

    Let's mess up the database server.
    Website is down instantly.

    Let's mess up the NFS server.
    Website is down instantly.

    Having more servers without rewriting the app in a fully distributed fashion would not solve any problem.
    It would only make the problem worse because every page view depends on three servers instead of one server.

    Not gonna work, because the client said so.. lol

  • jsgjsg Member, Resident Benchmarker

    @yoursunny said:

    @angstrom said:
    As for further development (multiple servers, customization, etc.), the client's budget is by all signs too small, but it's not clear (to me) whether these ideas really come from the client themselves or rather from (a kind of wishful thinking on the part of) @MagniPhiCat . (Basically: "if I have multiple servers across the planet, then if I mess up one, there will still be others".)

    Let's mess up the database server.
    Website is down instantly.

    Let's mess up the NFS server.
    Website is down instantly.

    Having more servers without rewriting the app in a fully distributed fashion would not solve any problem.
    It would only make the problem worse because every page view depends on three servers instead of one server.

    That's too harsh a view, but yes, I had reasons to mention synchronization although I wanted to wait with mentioning that his or his clients funny "fast Geo availability and low latency" desires highly likely will require some rewriting (well, actually, some redesign too).

  • drizbodrizbo Member
    edited January 2022

    @MagniPhiCat What's your bussiness primarily btw? Since this is your client.

    And you said provider is nexusbytes, but then you say its OVH server and you have access to ovh control panel? So nexusbytes just resold you ovh server or what? Or did I misread?

  • @drizbo said:
    @MagniPhiCat What's your bussiness primarily btw? Since this is your client.

    And you said provider is nexusbytes, but then you say its OVH server and you have access to ovh control panel? So nexusbytes just resold you ovh server or what? Or did I misread?

    You misread. @MagniPhiCat is not with Nexus Bytes. The title of this thread is a bit misleading because Nexus Bytes have nothing to do with the problem that @MagniPhiCat has.

  • deankdeank Member, Troll

    The problem is OP himself.

    No one here can help him. Only he can help himself. He just needs to realize that he can't do what he is supposed to do.

    What is he supposed to do anyway? He wants to do something - that I know. I just don't know what.

  • @deank said:
    The problem is OP himself.

    No one here can help him. Only he can help himself. He just needs to realize that he can't do what he is supposed to do.

    What is he supposed to do anyway? He wants to do something - that I know. I just don't know what.

    The OP just want to look less bad on his client's eyes by doing what he did not understand at all in just 4 days, then he seek advice here but ignore all of suggestion just because the client said so, anything else are unacceptable.. even the current broken by him server and the data inside is not relevant anymore..

  • Background: The OP carries out a risky experiment on a production server without a backup. The experiment fails

    1st-stage panic: The OP realizes that the experiment has failed and that he has broken glibc

    2nd-stage panic: The OP writes to Nexus Bytes, a popular low-end provider who sells unmanaged servers, and hopes for quick advice and a quick solution

    3rd-stage panic: After not receiving a reply from Nexus Bytes, the OP signs up on LET and starts a thread, hoping for quick advice and a quick solution

    4th-stage panic: Despite receiving a lot of quick advice from LET participants, the OP doesn't receive a quick solution and decides to ignore the advice

    5th-stage panic: The OP disappears from LET

    (Something like this)

  • @angstrom said:
    Background: The OP carries out a risky experiment on a production server without a backup. The experiment fails

    1st-stage panic: The OP realizes that the experiment has failed and that he has broken glibc

    2nd-stage panic: The OP writes to Nexus Bytes, a popular low-end provider who sells unmanaged servers, and hopes for quick advice and a quick solution

    3rd-stage panic: After not receiving a reply from Nexus Bytes, the OP signs up on LET and starts a thread, hoping for quick advice and a quick solution

    4th-stage panic: Despite receiving a lot of quick advice from LET participants, the OP doesn't receive a quick solution and decides to ignore the advice

    5th-stage panic: The OP disappears from LET

    (Something like this)

    great tl;dr

    Thanked by 1angstrom
  • deankdeank Member, Troll
    edited January 2022

    Pretty sure OP is right now blaming his host for their incompetence and whatnot to his client.

    He doesn't strike me as someone who accepts responsibilities.

    Really, he should change his career. A politician sounds okay for him.

  • @deank said:
    Pretty sure OP is right now blaming his host for their incompetence and whatnot to his client.

    I would not be surprised if he actually blames the evil users of LET since they we did not provide a simple one-line solution.

  • @rcy026 said:

    @deank said:
    Pretty sure OP is right now blaming his host for their incompetence and whatnot to his client.

    I would not be surprised if he actually blames the evil users of LET since they we did not provide a simple one-line solution.

    There is actually.. managed service

  • deankdeank Member, Troll

    As mentioned by some idiot, LET users do not understand the difference between managed and unmanaged.

    To them, their hosts must assist no matter what since they are paying. How much they pay doesn't matter. They are paying. Therefore, their hosts must assist.

    Thanked by 1chocolateshirt
  • AlwaysSkintAlwaysSkint Member
    edited January 2022

    Tries to bypass using a $100k/year Infrastructure Engineer. Hmm. >:)

  • servarica_haniservarica_hani Member, Patron Provider

    I didnt read every message here but since you were asking about starting point with tutorials to understand what to do you can go here

    https://github.com/donnemartin/system-design-primer

    This will give you an idea of how big this field is and how much time , effort and EXPERIENCE you need to make right decisions

    That being said your number 1 priority is to migrate your services/site from the current vm to another vm with same provider or another provider ASAP

    good luck

  • @yoursunny said:

    Let's mess up the database server.
    Website is down instantly.

    Let's mess up the NFS server.
    Website is down instantly.

    Having more servers without rewriting the app in a fully distributed fashion would not solve any problem.
    It would only make the problem worse because every page view depends on three servers instead of one server.

    NFS is not good solution to serve files even if the 2 VPS are in the same datacenter. I can't image what troubles he will have with this setup if the NFS and Nginx / PHP are in different datacenters.

  • yoursunnyyoursunny Member, IPv6 Advocate
    edited January 2022

    @CyberCr33p said:
    NFS is not good solution to serve files even if the 2 VPS are in the same datacenter. I can't image what troubles he will have with this setup if the NFS and Nginx / PHP are in different datacenters.

    @MagniPhiCat wants to accept file uploads from multiple web servers, where each web server runs the same application and all states are stored in a shared database.
    It is valid to store uploaded files in NFS.
    However, the NFS server is a single point of failure.

    When I was an operations intern at Microsoft MSN, I learned Distributed File System (DFS) feature on Windows Server 2008R2, and convinced the company to switch several websites to use this feature for uploaded files.
    It's like NFS but there are at least two replicas storing the same files, with multi-primary replication coordinated by the Domain Controller.
    When you reboot any one server, the folder is still readable & writable, and will re-sync later.
    Backups are still necessary, to protect against buggy website deleting files by error.

  • @yoursunny said:

    @jar said:
    If the server is up and you can't fix the software on it

    I have an OpenVZ server that still responds to ping and SSH port is open, but I lost the SSH private key and password authentication was disabled.
    Moreover, I don't have any access to the Virtualizor account.
    Is this server still "up"?

    That wasn't the question nor his conclusion. His conclusion is that if you get in that state, you suck and should have someone who knows wtf they're doing.

  • @MagniPhiCat said:
    but this again is out of budget. It would cost 50$ per month, when my client's overall budget is 80$.

    Math is hard, but there's a rumour that $50 is less than $80.

  • yoursunnyyoursunny Member, IPv6 Advocate

    @TimboJones said:

    @yoursunny said:

    @jar said:
    If the server is up and you can't fix the software on it

    I have an OpenVZ server that still responds to ping and SSH port is open, but I lost the SSH private key and password authentication was disabled.
    Moreover, I don't have any access to the Virtualizor account.
    Is this server still "up"?

    That wasn't the question nor his conclusion. His conclusion is that if you get in that state, you suck and should have someone who knows wtf they're doing.

    How I lost SSH

    I typed this command to install new SSH key and delete old SSH key on all my servers:

    for B in host0 host1 host2; do
      ssh $B bash -c "echo NEW-SSH-KEY > ~/.ssh/authorized_keys"
    done
    

    For some reason that I still don't understand, this command creates a blank authorized_keys file, so that I cannot login anymore.

    The consequence

    I spent several hours going into VNC or serial console of each server, re-entering the SSH public key.

    I have to delete and re-create Oracle Cloud instances.
    This includes the in-demand 4C24G, which took me several weeks before getting one again.

    I have to pull the SD card on my Pi Zero in order to put back the SSH key.

    The lesson

    Correct command is:

    for B in host0 host1 host2; do
      ssh $B "echo NEW-SSH-KEY > ~/.ssh/authorized_keys"
    done
    

    I should have tested the command on one or two servers and verified its result, before running it on all the servers.

    I thought I ran the command on two servers and didn't receive any error message.
    However, I didn't properly verify the result (i.e. I can login with new key).

    How I lost Virtualizor

    It's Boomer's fault.
    They migrated Virtualizor and I don't know the password.
    Reset password doesn't work.

    Why I can't open ticket

    It's a server received after donating to charity.
    I don't have billing account.

Sign In or Register to comment.