New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
You must be (or have been) using an old (no longer supported?) distribution.
Normally, one shouldn't need to upgrade openssl outside of the distribution.
And upgrading openssl required glibc to be upgraded?
Unless you really knew what you were doing, you were playing with matches/fire.
Ah, sorry, nothing I can do about that unless you are using a VPN with a static IP assigned to you only. I don't allow VPN networks.
rm -rf does horrible things. or try moving/renaming certain lib folders to backup...
solution is simple and has already been suggested by @yoursunny , get some tenner out of your a*se and deploy whatever cloud vm somewhere and move your stuff onto it instead of debating two or three more days over which provider should be the next.
after everything is running there take a deep breath. then take backups. then sit down and think over your next steps including a possible new provider and change of infrastructure, disaster recovery strategy and whatnot.
on the other hand, if your holy grail still isn't worth even 10 bucks for some cloud VM and a few hours work to move it back and forth, then what's all the ado about it?
be responsible to your client even if that means you have to pay up from your own pocket this time for issues you caused in the first place.
Yes, Debian 8. I never dist-upgraded because I never had any backup server.
I asked my client several times, but they had (and I can understand) other issues at hand.
I guess it did 🤷♂️
I know... Well, I was only using apt, nothing from source or anything. So it was supposed to be fine. But actually I got distros mixed up in sources.list, that's what caused the issues.
Yes I am using a VPN because my ISP is giving me crap IPs which prevent me from browsing normally (e.g. captchas all over the place, amazon not even working properly, etc.)
Sorry that I am not allowed to check your website then 🤷♂️
One can install certain userland packages from (e.g.) Debian 9 on Debian 8, but unless you really know what you're doing, you don't want to install core packages such as openssl or (much less) glibc this way.
In principle, you should be able to do a dist-upgrade "to complete" the upgrade to Debian 9 (since it sounds like you've already begun the upgrade to Debian 9), but depending on how broken the system currently is, this may be less than straightforward.
What makes you think I did that? And what does it matter how my server crashed?
It's not what I am asking here...
I asked two things (and two things only):
I don't mind answering questions and chatting with people, I do actually enjoy it
But for now, nobody answered any of these two questions.
I can accept being lectured up to some point only, especially since I don't answer to you.
I have already addressed this with my client, and the decision was that I had to wait some more, until I find a reliable VPS provider to work with.
This is indeed some topic on which I would love need to dive deeper
10 bucks won't allow me to recreate the same thing I did on our current server.
Plus, it has already been decided by my client that I had to wait to find the best possible provider before doing things.
That's precisely what I am here to ask.
Who am I, and who is anybody for that matter, to judge my client's orders? 😐
I understand what you're saying, it makes sense (to some length). But it is not what I was asked to do, nor what I did ask here, if I may remind it. You may have your opinion, but please do respect my client's decision. I cannot do otherwise.
I prefer not to answer this. Thank you for taking the time to lecture me reply to my original message (even if you did not actually reply to anything I was asking 🤷♂️)
I did not install glibc, only openssl because I was at version 1.0.1 and needed 1.1.0 (IIRC)
Thank you for genuinely trying to help me fix the server
However, I was installing using "Sid", not "Stretch".
Your idea is not bad at all! But you forgot what I said: apt no longer works. It cannot fetch anything because of glibc incompatibilities 🙁 One solution would be to compile glibc and ln-s . Or to dpkg force all. But it is very complicated and also very risky. I advised my client to no longer touch the server, while the website is still functioning.
You seem to love dabbling with servers
Any chance of feedback on my original questions though? 🤩
I think that @Falzo was simply suggesting an interim solution before you decide on a long-term solution. (In other words: recreate the server elsewhere -- get it working again -- and then consider your options for a long-term solution, because the latter may take a while.)
Oh, wow: a partial upgrade from Debian 8 to Sid? What could possibly go wrong?
Definitely tricky!
Honestly based on your subsequent replies in this thread, the fact that you were running a very outdated Debian install and trying to run software that required newer packages as their dependencies was the root of the issue, not taking a full backup made it worse. Even if you went with another VPS provider, you're likely going to run into the same problem, especially since what you bought was an unmanaged service, which means the server owner has to troubleshoot all issues.
If you weren't going to upgrade Debian to the next version past its 5 year support lifecycle, I would have recommend sticking to CentOS (or now Alma or Rocky Linux) instead as they have 10 year lifecycles (closer to Windows Server's lifecycle). Installing packages from Sid made things worse.
I would have personally gone one step further and used Docker so that you wouldn't have had to deal with forcibly upgrading core packages to get an application going. Zabbix does have Docker containers available and the containers would have had all the required dependencies included.
For your infrastructure question, I would personally recommend cloud instances at this point, there's redundancy and easy scalability which you won't get as easily with VPSes. There's a reason why the hyperscale providers like Amazon, Google, Microsoft, Oracle and etc. are successful. Even smaller providers like Linode, Vultr, Digital Ocean et. al. are gaining popularity as they can scale better than traditional VPS providers.
I like you, and don't mind the folks here in the thread. Most do troubleshooting 24/7 for a living and focus only on the technical problem they think they can solve instead of the architecture problem they can't really solve.
The short answer is it is hard to give you advice that is accurate without a large discussion.
If I was to venture a guess, I'd agree with the Azure recommendation. Use Azure with shared storage between multiple servers. Or Amazon to the same effect.
One off services from LET providers you can maybe make work - take some BuyVM servers with a storage slab and wire them together. Using a private network with hard coded IPs is just a good practice so you don't get hijacked (much like how you would hard wire an API to only allow a localhodt website to contact it, to make it harder to hack).
Generally though, I'd need a lot more information. Exactly what you want installed on which servers. Do you need more than 1 mail server? Are the nodes just runing a website? Is it using an API? Is that a single instance of the API, or one locally per machine? What user data is being stored? Lots of big files, small files? How fast does that storage and retrieval need to happen?
I absolutely hate building web services for people. But when you do, architect and define it as much as you can on paper. Those designs will let you see heat servers you need and where. And from there how the storage, IPs and security should be laid out.
The quick answer is you should take a good solution designer focused on networks and servers out for a lunch, and put some designs together with them. Done wrong this can do a lot of damage and run a lot of risk for security compartmentalized unless it is well designed and planned out. So my long answer is you need to research many individual pieces of technology and learn how to bridge them together. And that the amount of design documents and QA stuff I'd need to ask from you is more than what I would be able to do with you over LET.
I will say that your general goal, decentralized services and clusters or nodes with geo redundancy is very common. But each set is usually somewhat unique as it will have different requirements and goals.
Usually you have full redundancy (servers, storage, everything) not too spaghetti style or it becomes very difficult to maintain and scale as it grows.
In the future though, you need a development and production environment. VirtualBox works in a pinch. Docker if you want to learn another thing on top of it. If a client says no to that, don't take the project! And it needs to be identical. Usually if it is any kind of critical you want a fail over or at least a stable back up solution.
That said, I agree. The immediate moment is to design, decide and build the replacement. From what it sounds like from your messages however this will be complex and tricky, and you likely are not going to be able to build that quickly. Even for knowledgable folx - all the installs, patching and configs are going to take a full day just to prepare and sync up from what it sounds like. Add to that it's unfamiliar to you and it should take you quite a while. Hence the recommendations to use a temporary interrum server is a good one.
Nailed it. Trying to set up your own distributed scaling infra with 8 servers is going to leave you with a lot of sysadmin to do, and it doesn't sound like they're up to the task. For what the client wants, you probably need to go with one of the big boy cloud services, it will be easier to set up and admin and there's going to be a ton of info out there to help you through setup.
Sorry, maybe I am on the defensive... With all the "ars*" words and reproach in the message, I tried to defend myself.
So you both think that finding a reliable VPS provider is a long task?
I could totally grab the first server I see (hell, even a dedicated server, I am lucky enough to be able to pay for a server) and create everything on it, but then I would have to do it again a few days later.
Of course it would have allowed me to have a functioning server in a day.
But I'm not entirely convinced it's the best solution. And at the very least, it's not what my client asked... (but I'm sure if I told them it was the best solution, they would have agreed to do it, but I was not expecting to wait that much in the beginning, for the record
)
Indeed 😅 I have a been a fool... But I did not clearly comprehend that I was doing something dangerous, otherwise I would never have done it, obviously, I don't want to sabotage my baby...
Yes. Then if it takes more than a few days, you won't be pressured to rush the architecture this time around. I'm not convinced you'll be able to rebuild this the way you want in a few days - there is a lot of research you'll need to do during this process if it's a new to you.
I would have built this all on Azure and let the owner migrate it later if they wanted to.
As I also told: sys admin is not my job, my client does not have money for this, I am doing my best, I told them several times to get another server for redundancy, but they decide their own priorities and I cannot and won't judge, and I never wanted to dist-upgrade without a backup server on the side.
Having smaller servers with separation of concerns would allow me more easily to spin a new one, link it, unlink the old one, upgrade it, and then link it again, and destroy the temporary server.
This is a great idea. I am not used to Docker, but I heard that it is great especially when you have the same binaries to install/configure on "many" servers (before that, people were using Puppet if I'm not mistaken)
Thank you for this @deltatux
Which ones would you recommend? Reliable and affordable (I could ask my client for a budget increase, maybe I could get it)
Do NOT reboot! Just keep the box alive and running for the moment.
Can it be repaired? Chances are 98% that yes, it can - but not by you. You'd need someone with considerable experience and know-how.
"In a matter of minutes"? I don't think so, at least not always, but still they have a very decent response time.
Unfortunately it seems that @seriesn is going through a troublesome period - the person, not the company from what I know (and I know a bit).
Obviously, yes although that's unfortunate.
No problem.
No problem if VPS are good enough. If however you want VDS ("dedicated resources") you'll probably have a hard time finding those boxes.
No problem, although some providers won't like port 25 (outgoing). How about using @jar's MXroute service? Not expensive and I hear only good things about them.
Highly likely not.
No problem but might get a little bit more tricky if it's sensitive data.
Now some thoughts/ideas/suggestions off the top of my head
I hope this helps. If you want to continue: your turn.
Maybe I asked on the wrong forum then. But people here seem nice! (even though I got lectured a lot... but if it's done kindly, I don't mind ^^)
Isn't an "NFS" a good fit here? Really this must sound like a dumb question, for professionals like you all. I am sorry... I really need help figuring this all out.
Yes, I was advised these services several times actually. But it is out of my client's budget.
People also advised me Google for Business for emails (instead of using MailCow on a server), but this again is out of budget. It would cost 50$ per month, when my client's overall budget is 80$.
This is some actual, nice advice!! 🤩 Thank you for this @KermEd !
Do you reckon it is possible to create a private network between different hosting providers? e.g. I rent a storage server from ProviderA, and a webserver from ProviderB, and I create a private network with the two servers?
If so, where should I configure this network? (you don't have to answer all my noob questions, do not waste too much of your time with me, you can just give me some link to a tutorial or something, if you happen to have one?! and I'll try to understand things by myself as much as possible)
We can have more than one? oO
Sorry for the dumb question. I think one is great, and enough!
Ideally, yes, the webservers will only be running nginx+php (and probably some redis cache, as I tend to use it as much as possible)
Niope, it's a classic PHP web application: http request from client, nginx redirects to PHP, routing in PHP, fetching data from the mariadb database (or redis when applicable), using a twig template, sending it to the client. JS can call some async URLs too. Nothing more, really.
Not much at all. It is in a database (identifier, professional headline and summary, avatar... that's about it)
All kinds of files are uploaded by the users. Documents, images, videos... It is a professional b2b webapp with all kinds of user-uploaded-files. We have roughly 200GB of files right now.
Do you mean files? I think that the storage will happen with some kind of NFS? And the retrieval will happen thanks to a CDN I guess (like CloudFlare, but we are not yet using it)
I already did. For me, and also to show my client. But this is not my core job. At all. I tried to "guess" what would be fine.
I still have to find one 😅 I have a friend... But we haven't talked in years.
Then why can't I find some general-use-case tutorials, from which I could start and build something? And why can't people give me links?
It's like those things don't exist, or something
We don't need too much. Otherwise it would be a "rich person problem" and we would be using some services like you and other people advised (e.g. AWS or Azure).
We only need:
I don't think it justifies going the AWS way, especially since my client does not have the budget. But, again, it is not my core job at all. I'm doing the best I can with the requirements I am given.
Already have one
But the issue is double: 1) we only have one production server, and 2) the testing server is not completely identical to the production server
No to Docker, or no to a testing server? My client is someone who helped me a LOT in a time of despair. I cannot say no to anything (except lying/stealing/etc. but it's not my client's type, so we're good
)
Oups... 😅
Even for the mail server? If so, it means twice each server, which means 8 servers if I don't count either the other geographical areas or the redundancy in the USA + the load-balancer. It will be outside of the budget, for sure.
I still don't get this one... Why would it be good to create a server, put everything on it, just to do it again a few days later because "it will take me too long to do it"? If it takes me 2 days (for example) to reinstall everything, it doesn't matter that the server(s) I am doing it on are temporary or definitive... It's the same work.
I really don't get this one 🙁
Anyway, thank you a lot for all your input and advice @KermEd !! 🤩👍
Yeah, actually the 8 servers is step 2.
Step 1 is one mail server, one web server, one storage server, one database server.
If I really wanted to throw everything later, I could use only one server and put everything on it.
But it's not what I was asked to do (at least for now ; my client set the deadline for a new VPS provider to 4 days top).
You are right that I might not be up to the task. But I already did many things. And although I admit I never created an infrastructure of 4 or 8 servers, I am not afraid. I don't like sys admin, and I will sweat... but I'm not afraid.
The client does not want to allocate that kind of budget, unfortunately.
Believe me, if I could have the whole thing managed and not worry about it anymore, I would gladly do it. I just can't 🤷♂️
(and I also think/agree the budget is real tight, but I cannot and won't judge my client ; we did do it with success up until now by using 20% of this budget, we will do it successfully now as well! 😎💪)
You really think I will need more than a few days to install mailcow + mariadb + nginx/php/redis + a NFS? (this one might be the trickiest part, actually)
I don't understand why it would. I am not a seasoned sys admin. Agreed. But I have already set up (from scratch) a few servers.
Ok, never mailcow, and never an NFS. But it can't be that difficult?
Maybe the NFS part could wait, then, because it would require a private network, and I have no idea how to do it, I admit.
But that's also why I was hoping for a hosting provider (namely NexusBytes, initially) to help me set it up... 🙁
I believe they only provide unmanaged services. And now you asked them to manage your server for free. If that is the case, you should buy a managed services, dude!
It's all good, I'm not good at explaining things today. It's not a dumb question and most professionals don't seem to know how to cluster things properly anyway imho.
Yes, so this is where it gets a tad different. Think of a private network much like a VPN service. Some companies will sell you VPN for private networks specially so you can customize the VPN on all your sporadic servers. They then are all working on the same virtualized network and think they are enclosed.
This is just done to stop external traffic from trying to access servers they shouldn't - and to get a speed boost in the same data centers.
All in one services like Azure make this stupid easy within their service. Outside of this, you'd want to google something akin to 'VPS Private Networks for Businesses' or maybe 'VLAN for Cloud Servers'. Most cloud heavy companies though can afford to do this kind of stuff in house. And so good documentation is hard to find at times.
Usually a single provider would be ideal for all your single instance servers and the Private Network
Then you only need to do special configurations for the web servers connecting to that back end.
In this case you'd google something along the lines of Apache configuration for an external Private Network. It's not done often I imagine, and documentation may be sparse.
This helps a lot, what I'd probably do is:
Main Provider:
** These can all be different VPSes, but will be easier to configure being on one provider ad the private network is likely much easier to configure
Then individual providers:
You then would setup a CDN, and a configuration for the web instances to connect to the private network. It's likely going to involve a lot of whitelisting of IPs and some VPN style configuration setup. I'd also document this really well so in a month you can add a node without needing to think about it much.
I hope this helps a bit!
The reason I recommended recreating it as-is and breaking it apart later, isn't because you can't do it all in two days. But so you don't have to if you get a big bandaid in place. But it's just down to choice really
Being a sys admin is a full time job, if your client can't afford a sysadmin, they should be offloading them into the cloud. For example, email hosting can be had for very cheap via services like MXRoute, Hostinger or Zoho Mail. They should also look into PaaS solutions so that you can focus on being a developer while letting others handle the infrastructure.
Services like Google's App Engine may be that solution that works best for you & your client for instance.
Docker lets you spin up containerized applications modularly and allows you to connect one container with another. Due to its portability, I'm a big fan of Docker containers, you should really look into it. If you're deploying and maintaining a lot of containers & servers, look into Kubernetes.
You can't go wrong with big players like AWS, Azure & Google Cloud Platform. Personally, I haven't used the small players like Linode & Digital Ocean as my employer uses the hyperscale providers instead and my personal requirements doesn't justify the cost of Linode/Digital Ocean, they are cheaper than the big players. However, if the big players are too costly, I've heard great things about Linode & Digital Ocean for their cloud instances.
That being said, Oracle Cloud provides an Always Free Tier and their ARM-based instances are great value, you can spin up to 4 ARM-based VMs with up to 24 GB of RAM for free, for many web apps, they are architecture-agnostic, so running on ARM CPUs shouldn't really matter. I personally pair my very cheap VPS with Oracle Cloud's Always Free tier to cover all my project needs so far.
Oracle Free Tier does also provide a basic AMD Epyc-based micro-tier cloud VM with up to 1 GB of RAM, but honestly, I'd rather not waste that slot on it and just go ARM unless you have an application that must run on x86.
@MagniPhiCat
Keep things simple.
You have only 1 server currently. It is not working properly. You want a managed VPS provider, ok.
-First find that provider to replace that single server.
-Then after that take your time to analyze your needs and your proposed solution. For example you said you want 8 servers, each for one task but then you want all the servers to share a storage and the servers are in different locations. Basically you went from something simple to something really complex with basically no decent budget.
In general all you need is an OpenVPN server (your own), and assuming I understood you correctly and servers are in different location, then your servers connect as clients to that VPN server, and then configure the VPN so that client can talk to each other.
But I get this feeling you want this big infrastructure just because you want to throw servers to the problem. In reality probably all you need is one server, and make sure you have good backups.
Assuming your stack is LAMP or LEMP, my recommendation is to contact KnownHost, pinging @jonathanspw, they have the balance between budget and good managed VPS server. Their replies are very fast, so this will help with your anxiety. You can also contact BigScoots, they might not be within your budget but they also take care of customers that want managed VPS.
But essentially if your stack really needs that many servers with such custom setups you probably will not find any budget managed VPS provider that will help you with that. That's more of a custom sys admin (specialist) or infrastructure expert job and it will cost you a few hundreds at minimum since hours are not cheap when it comes to custom admin jobs.
Keep the server count low if your budget is ... low.
For nostalgia...
. I liked the brand.
If this was 2015 I would say contact WiredTree they were probably one of the best managed VPS server company at the time. But sadly it got purchased by LiquidWeb and dissolved
Agreed with @Hxxx simple is best..
You could just go dedicated server, and throw everything there, setup proper backup server for failover, use cdn for that geo something.. it will work just fine for 5k user/day
This guy is complaining about @seriesn inactivity, when he is being inactive regarding his client and cries when @Falzo basically said "get off your ass and do something". Crazy.
I can't seem to be allowed to post my reply... A partial CloudFlare CAPTCHA popup shows up and I can't fill it or anything. Other times, an "reply" popup shows up, allowing me to (again) leave a comment. It feels very buggy. Tried in incognito. With or without VPN. Same thing.