New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Serving static media content through NGINX
Hey-hey
I want to serve static mp4 files through NGINX, I was wondering if there is an efficient way to approach this. Maybe some modules or configs I should look into?
I looked into NGINX+Varnish yesterday and I couldn't seem to make it any faster. I was using malloc assigned with 25GB (I think it uses memory after looking into docs).
My server specifications are:
40 GB RAM
6 Cores CPU
24TB HDD
Any suggestions or ideas? I want to be able to serve the content as fast as possible, if it requires caching etc. I dont mind.
Comments
Did you check your hit rate with varnish and stuff? What did you see using varnishncsa, varnishadm etc? I think varnish doesn't by default support mp4 streaming, see; https://www.varnish-software.com/developers/tutorials/example-vcl-template/#10-piping-other-non-http-content
You could also try NGINX upstream caching;
https://www.nginx.com/blog/nginx-caching-guide/
https://www.tecmint.com/cache-content-with-nginx/ (read thoroughly)
If the files are on local disk, nginx itself can serve it efficiently, and you don't need other software.
The kernel will use spare RAM as filesystem cache.
You're limited by your disk read speeds as always then even more, also check your iowait when streaming the content and such (iostat -x 1).
Thanks, I'll firstly look into NGINX upstream caching, then I'll take a look into the Varnish section
Depending on system different things help
For serving static start with (add it to your config)
And work from there. Play with on/off on first two settings, delete directio etc. and find what works the best for your server. I think this config is the best, but play with sendfile on/off to see which is better.
Also try different filesystems, ZFS with 20GB ARC cache can do wonders.
My file system is already ZFS, but I'll give those NGINX settings a try and see if it does anything. Thanks once again!
So you dont need additional caching. You ahould be able to saturate 1Gbps with my settings. With stripping 2Gbps.
Enabling aio and directio should massively help, Im not sure about sendfile because it depends on maasny things in system/files.
Thanks a lot, will give these settings a try in around an hour. Will update this thread on how that goes!
The recommendations made by @AXYZE are very good and should help a lot. Here is my /etc/nginx/conf.d/io.conf file with thread pooled AIO:
And on the global level of /etc/nginx/nginx.conf:
thread_pool default threads=4 max_queue=16384;
@Jari That is very specific, I doubt it can be a blanket application to many Nginx installs.
@Arkas What exactly is "That"?
Naturally, one should experiment with the numbers (buffer sizes) depending on the hardware and file size. We are talking about larger static files here (MP4 videos), not a "blanket application to many Nginx installs".
I get,
"aio on" is unsupported on this platform
When adding in that part, I'm using Ubuntu 18.04.05 LTS
nginx must be compiled with the option
--with-file-aio
, check usingnginx -V
. You may have to compile nginx from source. This would allow you to remove any modules that are not necessary for serving static files, as well as using more aggressive optimization flags (if you care about every little bit of performance). You may run into other bottlenecks on the way.Much faster after enabling the other configs and turning off sendfile, did do a bit of port monitoring and I see it spike to around 280mbit/s when seeking on videos. Is this a convenient way to go about this? If there are multiple users seeking and its hitting the max port speed, wouldn't this be bad?
Rate limit on firewall
You can also configure BBR for your network. Great write up is here: https://www.linuxbabe.com/ubuntu/enable-google-tcp-bbr-ubuntu
Consider adding SSD cache to your HDD array. That should help with IO seek time.
Network usage is very good! It means it can read and send necessary data fast!
Now you can limit connection speed. You can do it via firewall (limit per IP) or via nginx (limit per connection)
First line will limit single connection to 600kB/s.
Second will delay the limit so it activates after 500kB is downloaded (file header) and that will make sure that "play" button will work instantly.
Now you need experiment with these values.
limit_rate should be 30-40% higher than highest bitrate video in your collection.
limit_rate_after should be 500k or 1m.
You can setup another vps and check how it works with many streams (you can play with load balancing tools like JMeter, see online how yo configure them for mp4 files)
And then see from your local connection if you can still watch videos with no problem. If it lags then try with different rate limit, different settings in config that I mentioned above...
So if one of my videos is 3300kB in total for video bitrate, does it need to be 30-40% higher than this value, or does it need to be calculated in a different way?