New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
OK I looked at that wall of text a little bit further. You have a very lightweight site but you should only use HDD for infrequently accessed bulk storage. A mechanical HDD can do about 150 iops and this hasn't substantially changed in decades, even though capacity is 100x higher than before. So the amount of iops you get for 100GB of capacity is now almost useless.
I'm going to suggest spinning up a 4GB ram, 2cpu, 80GB SSD DigitalOcean or Vultr instance and trying your stuff on it ($20/mo). BuyVM is also great except they are out of stock all the time. Avoid HDD except for stuff like backups or occasionally served media files. DO also has an object store for media files and Vultr might have something similar. DO tech support is reportedly very good. Don't know about Vultr. I use both of them but haven't needed much support from either.
DO and Vultr both have referral codes where you get some free credit to try stuff out with (and the person who refers you gets credit if you end up becoming a paying customer). Also they bill by the hour instead of the month, so you can do a quick test and see if you like it without being on the hook for $20 or whatever.
I'm sure there are people here who can supply you with referral codes: I'm happy to supply mine but since I'm giving you a hard time I can understand if you use someone else's.
Thanks for clarifying. My understanding is that the box would have to be rebuilt and you cannot adjust this on the fly like OpenVZ. This actually is great news for me to think about my next project because I was not too thrilled with the idea of reinstalling everything if I need to scale up resources.
You can't easily and safely downgrade disk on KVM but upgrades are fine. There's just a few more steps.
https://stacksetup.com/KVM/resize
OK so I get a better understanding of how this all work..
if a user leases a KVM VPS. and the stated specifications are 4 vCPU's at 2.3 GHZ and 4.0GB RAM 100mbit uplink and 100GB of disk space.
Running the setup above. is it normal to have 2-3 cores pegging at 100% more then 5 times in a 10 minute window with the process that is causing the high CPU load being completely random.
getting only 0.5MB/s transfer speeds from a 100mbit link?
I do not know how much the hypervisor allows for changing these things. The information I have found is conflicting. as i stated I also set up a duplicate VM and it works without any of the above issues. So I know it is not a software issue in the VPM it's self.
I did manage to get a rather irritating response from my host. they stated to me that they will "turn up" the network speed... which leads me to believe that what I paid for I am in fact not getting they are able to tune down everything at the hypervisor layer. I am not 100% certain on what can or cannot be changed without the user knowing.
There is a provider on here that lists IO access priority for each of the levels of they VPS's which is another indication of throttling. they are honest and upfront about it which is a good thing. From those 2 things I am gathering it can be done. But the question is how to know when it is being done?. is there anything that can be used to check?
If it is being done and the user has no knowledge of it that's fraud.
Although I can agree it isn't a pleasant practice to heavily throttle your cpu and io without getting in contact, unless you're guaranteed a certain MHz/GHz or a certain iops etc that is dedicated to you - i don't think it could be considered fraud.
At the end of the day if you want dedicated resources, you can always just get a dedicated server.
Try looking at your database's slow query log and see if there is a specific query causing this. The usual problem is a missing index on some table. Although if you really copied everything (preferably restored a snapshot of your production vm) it's a little bit strange that it didn't reproduce on your home setup.
Yes cheap VPS will not allow you to hose cpu like that. They will eventually either suspend or throttle your server. You can get VPS with dedicated CPU but as usual they cost more. But you should not need that and it's a poor solution. Something is wrong with your setup and you should figure it out and fix it. After that your stuff should run fine on a small VPS.
I suspected this as well and I have checked it. nothing seems out of place from the rest of the queries.
The thing is the CPU use happens with all processes running. Not just MySQL or Apache.
If I am running htop that will ding them to 100% for 20-30 seconds. and the random number of cores that peg is what baffles me. the process that seems to be the most frequent is /sbin/init
I just checked right now and there is only a single connection to MySql and that is me. and the cpu use keeps on spiking to 80%+ and I am only just sitting there in MYSQL workshop. I am not running any queries. and MYSQL shows very little data going out. (probably because of the monitoring the workshop is doing)
another really odd thing is that when some of these spikes are taking place if I add up the processes over 1% (in some cases might be 4 processes. it is not even close to using 80%
only 10% at best. 95% of the process are at 0.0 and there may be a couple at 0.1.
I am clueless as to what it can be that is causing the problem. if i make an exact replica i do not get these issues.
In that case I'd say reinstall everything on a new VPS and see if the problem goes away. Again I suggest Digital Ocean or Vultr. Regarding the cpu spikes, do the process id's keep increasing? sbin/init may be inheriting the runtime or processes that spin up and exit, i.e. there is some kind of accidental fork bomb happening.
In order to fix this you are going to have to get really good at moving your server. I would find an offer that looks like it will work, shutdown your forum, move it, point-DNS and see what happens. Keep doing this until you find a provider that works.
Is it possible using a XEN provider would be better?
I don't think the virtualization type is the culprit here. It would help to be able to reproduce the CPU spikes, then do some aggressive debugging as needed. If moving the forum to another VM makes them stop, then maybe shrug and stay on the new VM. If they recur there, now you have two installs where it happens so you can use one of them to debug with, by replaying the workload through it.
I am thinking the same thing as @willie. It's a very illusive problem. and one I am not able to identify a source.
Now I can find information about older versions of Ubuntu having an issues with init (systemd) and having high CPU use but nothing about Ubuntu 16.04. and nothing that includes other processes being included in the spike as well.
and the fact that the processor use total does not add up to what is being used is also strange.
I was looking at Launch VPS and they seem to have decent offerings for VPS. and I have not found any horrible reviews about their service. I did as instructed and sent a message over the chat. only be to told that they will email me because no one was available to speak with .. I have not received an e-mail back and this was done at 1:00 PM MST (GMT - 6:00) which I find rather strange for a Friday. I do not know where they are located. I thought it was the east coast which would have made to 3:00PM there. Maybe they leave early on Fridays, who knows. I was Hoping to get a VPS for this weekend to start working on it. looks as tho I am going to have to wait.
I was going to get the 6GB RAM 4 vCPU 60GB SSD package they have. It is the only package they have listed with the 4vCPU's I want to make sure that everything is going to run OK on it. I would rather spend a little more per month and know that the processing power is there.
I am going to take a look at Vultr in a little bit.
This should get you a $50 vultr credit on signup (it is a referral code, as described above): https://www.vultr.com/?ref=7979992-4F
I'm not sure how quickly you have to use the credit. There is typically a 1 month limit. But that lets you try quite a lot of stuff.
That's right I did look at them. No where could I easily locate the information as to wht kind of a VPS it is, Container based Virtual Machine (OpenVZ) or Kernel based Virtual Machine (KVM).
The name of the VPS line they offer service "Cloud Compute" would lead me to think it is a Container based VM. I didn't do any huge amount of digging on their site to try to attain this information. They should have it right there and easily found.
Also there pricing structure kind of skips a step. and it would be the one I would want.
they go from 2 vCPU 4GB RAM 80GB SSD @ 20.00USD to 4vCPU and 8GB RAM 160GB SSD @ 40.00.. what happened to the 4vCPU and 4GB RAM for 25ish? and the 4 vCPU 6GB RAM for 30ish?
their pricing structure is double the price double the resources available.
4 vCPU and 4GB RAM and 80 GB SSD (preferred but 60 GB will work) is what I have decided on going with. that will leave me enough resources when i decide to expand and add the file repository. Anything over that would be severe overkill and anything under I feel might fall short when I add the file repository.
Do you mean Vultr? Vultr and DO and similar services are always KVM except a few of them are Xen. OpenVZ is a low end thing. This is slightly higher class stuff and it costs a bit more but it can save you headaches.
You are actually probably fine with their $5/month plan except you won't have enough disk space. They have SSD block storage at $0.10/GB so you could add that, but if you need 100GB of it that's $10 already, so the bigger instance looks attractive. Vultr used to have a promo where you'd get 50GB free in NY/NJ, don't know if it's still available.
With DO you could use their object store for media files but there is a 1TB bandwidth limit ($.01/GB after that) which might be insufficient for your site.
@sidewinder
moving the server is not an issue so much as it is an inconvenience. I wrote a script to do all of the dirty work actually it does the whole process except for the OS install. It updated and installs all of the software and it will move all of the data as well. All I have to do is to run it and come back in an hour or 2 or 3.. depending on how fast the data can transfer. (took 3 hours to move 3.5GB when I did a backup the other day). So I am not all that bother by the actual process of moving so long as no problems occur.
I am no where close to using 1TB of bandwidth. that is why I have not even included that as any kind of a need. almost all offer at a minimum of 1TB. The program that i offer is only a 25MB download. and it doesn't even get downloaded from our server. it gets downloaded from GitHub. The file repository on the other hand because the file sizes are so tiny (90% are 2K-3K) this would not be an issue. I want to have the extra space for any tests I may run. log files. Things of that nature. I want to have some wiggle room in the event something happens and say a log file grows to some bonkers size (had this happen already 8GB access log for Apache). so padding by 20GB is almost a must. if I wanted to only give myself maybe 5Gb of wiggle room I could go with a 40GB plan.
the forum is a support forum for my software. the almost 7000 attachments are again very tiny. it's not media of any kind. Nothing to take up a vast amount of disk space. I can tell you what the average size file is for the attachments but it will take me a minute to calculate it.
average attachment size is 131K
I also averaged the files that would be in the repository and that is 38K around 1000 files to start off with.
I also did want to note that I am currently consuming about 1.3 GB of ram. and that is about the only thing that seems to be accurate with my current host.
i would say try another VPS environment like Vultr, DO with free credits and if you cannot replicate the problem, it suggests that your current host node is the culprit. Testing on your own computer is not a very apples to apples comparison. You probably spend more unnecessary time with other methods.
And what software would that be
Read:
Get a Slice. With a slab.
Francisco
doesbuyvmhavestock.com is now domain squatted
Do you even have slabs in NY? It would be a good combo, once stock is available. NY slices are currently out of stock. It will be great when new hardware arrives there.
I thought you were using around 100GB of disk already and was wondering where it went. If those attachments are not frequently downloaded then putting them in HDD storage is reasonable. I was imagining a forum with images showing up in posts, so they would get accessed and served very frequently.
8gb of logs is an awful lot unless something is going crazy. I'd suggest some kind of automatic log rotation that compresses old logs and moves them to secondary storage.
Anyway it sounds like your main issue is your existing server hanging/spinning (and hours to transfer 3.5GB of files is ridiculous). I'd say set it up on another server and try to run the real workload through it for a day or so and see what happens.
Would you mind sharing who your current host is? It's not alpharacks or w00thosting is it?
I have been using ChicagoVPS for years now and am very satisfied with performance.
However, if you really want completely dependable performance, go with Amazon AWS EC2, or rent a dedicated server. Reliability and reserved resources for dependable performance costs real money to provide.
The cheap deals here rely on over-selling and under-provisioning resources to keep prices low and still make a profit.
I see that you have been smoking a lot of potassium.
I have been toying with the idea of getting a dedicated server. the thing is that would be extreme overkill for what I need.
The name of the software is EventGhost. You may have heard of it.
The 8GB log file was a single time that it happened. it was because I didn't have a log rotation set up on the Apache access log. and some ass munch decided they were going to run some kind of a program that tried to gain access to portions of the website that are restricted. This problem has now been solved with log rotations and with an automatic IP block if they try more then "n" times. I didn't think that someone would want to try and do that to out site. so i never really had put much thought into it when i first set up the server. There are things that get overlooked and or a new bug or hole can cause issues like this. so it is best to have some wiggle room.
I am not exactly sure how many of our attachments are images. I would have to run a query against the database to find out. If i had to take a guess it would be between 30% and 50% of the attachments are inline images. I would also prefer to not alter the current directory structure. phpBB is somewhat temperamental when it comes to changing paths. and the database holds the information about the filenames of the attachments. I have never looked at the database tables for the attachments. I do not know how it stores the path and filename.
I just checked and the total disk space currently being used is a tad over 30GB for everything. (without the file repository.)
This I know. This is the single largest reason I am trying to move. we have had cases where 200 + people will be accessing the forum at the same time, it is not often this happens but it does. And with the server performance being what it is performance wise it's not going to keep up. I would never be able to add the file repository. This is something that is going to get done on the next 6 months. I need to have the ability to supply the files and not kick back "Try again later" messages or have it take a week to transfer 30K of data.
at this point I have to wait until Monday to be able to move forward. I want to ask Vultr if they can provide something that is in between the 20.00 and the 40.00 plans.
I doubt that Vultr will do a custom config for you though of course you can ask. The model with their style of product is that VM's are like cattle rather than like pets. A big user would run 100s or 1000s of them, spin them up and down from an API, etc. It sounds to me like their $20 plan should be enough though. It has 80GB of SSD and you could put your apache logs into add-on block storage without messing up the directory structure for your PHP files.
A dedi would be total overkill to run a non-enormous forum but they do get rid of the whole noisy neighbor question.
@MikeA do you have anything for this guy? 2GB ram, 2 cores, and 100GB SSD in Quebec, say.
Nah, not on OVH. In Dallas I might but he wants east coast.