New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
run top and post that
https://goo.gl/maps/fLnHSajWHV1atNF56
Probably I/O,
vmstatoriotop.Yeah probably an io component to start.
Other possibilites
enable slowlog on php-fpm config
@yokowasis
Well for a start a load avg. of 18 on a 12 cores system isn't exactly low.
Plus have a look at the IO too, both disk and net.
I am referring to the processor graphs. No core seems utilized at 100% yet the load avg is very high.
It takes into account I/O wait.
Don't put too much significance into those graphs. Look at the numbers, not at the eye candy.
Two quick side notes:
(a) are you sure you want swap active?
(b) your setup uses only a very small part of the plenty memory you have. You might want to think about trading processor load for memory. I suggest to considerably extend your caching (on all levels, from mysql up to web pages).
This is definitely an I/O issue and tetech rightly pointed out that load average includes I/O. It's a measure of how long processes are waiting for a chance to run.
Post an
iotop, really difficult to help you otherwise.Increase
innodb_buffer_pool_sizeto about 15G and restart mysql serverYour processes are going to D state, which usually means they are fighting for IO. In other words, your disks are overloaded.
You guys were right, it was an io issue. I added 2 more nvme with raid-0. The problem is solved.
Thanks Noted.
fast-forward a month
"Guys, how can I restore my system from backup?"
Mysql_tunner.pl that sql server. Apply recommendations. Second, try to adjust php-fpm settings. If you have nvme, use more disk cache.
That load is very high for that CPU usage... looks like a lot of time is waiting on IO
Hmmm, if a (reasonable size, reasonable quality) NVMe isn't good enough than one has an app with rqs in the tens of thousands - and/or one has an application design and a caching problem and that should be addressed. "moar nvmes!" usually doesn't solve the problem but rather covers the symptoms and buys some time. Just saying ...
YOLO.
So long the nvme doesn't break, it should be fine, right ? I have faith in hetzner nvme. Most of the data isn't that important anyway. They all got deleted once a month.
It's AX41. The default setup is 2 x 2TB HDD with RAID1. Since it's just mainly wordpress sites with a total under 1K page views / day, and most of the time, the server is just idling. I though it's fine.
Sorry, no, it's absolutely not fine when such a beast dedi (6 core Ryzen 3000, 64 GB memory) with a paltry less than 1k page views per day gets down to snail speed and/or up to high CPU load. There's either something wrong with your box or config or you have had hackers visiting your system/site.
And btw. before shelling out more money for NVMes you really should think about my advice and use what you already have (plenty memory) and what is way faster than NVMes too.
Frankly, I think you should also inspect your system and do your best in terms of looking for rootkits, malware, etc.
nginx php-fpm, dynamic content data on ramdisk 32GB (tmpfs)
1200 - 1500 req/s
1K page views / day, $15 / y VPS is enough for that...
Your WP page full of bloatware..
The page views is 1k at the same time as in 1k visit in 5 a seconds, not through the day. And you know the human. If a page doesn't load within 5 sec, they will reload the page. Which makes it even worse.
Tried with a normal vps and shared hosting. It doesn't work.
Are you suggesting I am moving the database to ram? The WordPress has very specific use case. 1k visitor at the same time, and all of them write to database at the same time. An image uploader website if you could say.
What's the normal writing speed of RAID - 1 spinning HDD? Is RAID - 1 could make the performance worse for write intensive website? I mean the file get written into 2 place at the same time.
It's a custom made theme. The function mainly for uploading article and a lot of images. The article will then be sent to Google firestore, the image will be sent to another cdn.
It doesn't serve the visitor directly. So nothing fancy or plugin going on.
When I said 1k page views, it's a 1k of websites Admin doing stuff. Not normal visitor.
Are you actually writing images to a database or just writing a reference to an image on the filesystem?
Well, that's not helpful since that's not what page views means.
Basically, people can probably improve the efficiency of your setup at every step if you provide enough details.
Doing something repetitive in a 5 second window type activity is begging for RAMDISK. And you got tons.
You also should have SSD, not HDD'S. How much space do you actually need? You'd probably be better off using a single NVMe and daily backup to local HDD.
Has to be IO. Do a IOPS check. Also do some caching, that will help a lot
Just a reference. But admin can upload 100 images at one batch, or no images at all.
Yeah that's where is was wrong. I though HDD is enough. Because the website is mostly used by admin, not user. It works fine since January.
Throwing more hardware is never the right long term solution.
Optimal use of your RAM, using several cache layers and tuning your code and config files is better in the long run. If you are not using your server to its full potential, why bother upgrading?
Sometimes it's THE only solution. it's called scaling. No matter how much you optimize there is so much you can do. In my case is a spinning HDD.
No matter how much I optimize the RAM for cache, there is not much you can do if the purposes of apps is to to upload a bunch of files.
Nope, I wasn't yet suggesting anything concrete, because I know very little about your use case and setup/config. But one thing is clear: if your AX41 is very slow plus it seems to be IO related plus you have tons of unused RAM you're doing something wrong.
No, Raid 1 (even software Raid 1) has no noticeable performance impact.
Some remarks based on info presented since my last post:
We can't really help you/provide good concrete advice with the very limited info piecemeal you provide. So, if you want to get good and useful advice for your situation you must tell us much more.