Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Serving static media content through NGINX
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Serving static media content through NGINX

sgno1sgno1 Member

Hey-hey

I want to serve static mp4 files through NGINX, I was wondering if there is an efficient way to approach this. Maybe some modules or configs I should look into?

I looked into NGINX+Varnish yesterday and I couldn't seem to make it any faster. I was using malloc assigned with 25GB (I think it uses memory after looking into docs).

My server specifications are:
40 GB RAM
6 Cores CPU
24TB HDD

Any suggestions or ideas? I want to be able to serve the content as fast as possible, if it requires caching etc. I dont mind.

Comments

  • Did you check your hit rate with varnish and stuff? What did you see using varnishncsa, varnishadm etc? I think varnish doesn't by default support mp4 streaming, see; https://www.varnish-software.com/developers/tutorials/example-vcl-template/#10-piping-other-non-http-content

    You could also try NGINX upstream caching;
    https://www.nginx.com/blog/nginx-caching-guide/
    https://www.tecmint.com/cache-content-with-nginx/ (read thoroughly)

  • yoursunnyyoursunny Member, IPv6 Advocate

    If the files are on local disk, nginx itself can serve it efficiently, and you don't need other software.
    The kernel will use spare RAM as filesystem cache.

  • @yoursunny said:
    If the files are on local disk, nginx itself can serve it efficiently, and you don't need other software.
    The kernel will use spare RAM as filesystem cache.

    You're limited by your disk read speeds as always then even more, also check your iowait when streaming the content and such (iostat -x 1).

  • @FoxelVox said:
    Did you check your hit rate with varnish and stuff? What did you see using varnishncsa, varnishadm etc? I think varnish doesn't by default support mp4 streaming, see; https://www.varnish-software.com/developers/tutorials/example-vcl-template/#10-piping-other-non-http-content

    You could also try NGINX upstream caching;
    https://www.nginx.com/blog/nginx-caching-guide/
    https://www.tecmint.com/cache-content-with-nginx/ (read thoroughly)

    Thanks, I'll firstly look into NGINX upstream caching, then I'll take a look into the Varnish section

  • AXYZEAXYZE Member
    edited January 2022

    Depending on system different things help
    For serving static start with (add it to your config)

    sendfile off; 
    aio on; 
    directio 8m; 
    directio_alignment 4k;
    output_buffers 1 2m;
    

    And work from there. Play with on/off on first two settings, delete directio etc. and find what works the best for your server. I think this config is the best, but play with sendfile on/off to see which is better.
    Also try different filesystems, ZFS with 20GB ARC cache can do wonders.

    Thanked by 3pbx yoursunny jugganuts
  • @AXYZE said:
    Depending on system different things help
    For serving static start with (add it to your config)

    sendfile on; 
    aio on; 
    directio 8m; 
    directio_alignment 4k;
    output_buffers 1 2m;
    

    And work from there. Play with on/off on first two settings, delete directio etc. and find what works the best for your server. I think this confog is the best, but play with sendfile on/off to see which is better.
    Also try different filesystems, ZFS with 20GB ARC cache can do wonders.

    My file system is already ZFS, but I'll give those NGINX settings a try and see if it does anything. Thanks once again!

  • @sgno1 said:

    @AXYZE said:
    Depending on system different things help
    For serving static start with (add it to your config)

    sendfile on; 
    aio on; 
    directio 8m; 
    directio_alignment 4k;
    output_buffers 1 2m;
    

    And work from there. Play with on/off on first two settings, delete directio etc. and find what works the best for your server. I think this confog is the best, but play with sendfile on/off to see which is better.
    Also try different filesystems, ZFS with 20GB ARC cache can do wonders.

    My file system is already ZFS, but I'll give those NGINX settings a try and see if it does anything. Thanks once again!

    So you dont need additional caching. You ahould be able to saturate 1Gbps with my settings. With stripping 2Gbps.
    Enabling aio and directio should massively help, Im not sure about sendfile because it depends on maasny things in system/files.

  • @AXYZE said:

    @sgno1 said:

    @AXYZE said:
    Depending on system different things help
    For serving static start with (add it to your config)

    sendfile on; 
    aio on; 
    directio 8m; 
    directio_alignment 4k;
    output_buffers 1 2m;
    

    And work from there. Play with on/off on first two settings, delete directio etc. and find what works the best for your server. I think this confog is the best, but play with sendfile on/off to see which is better.
    Also try different filesystems, ZFS with 20GB ARC cache can do wonders.

    My file system is already ZFS, but I'll give those NGINX settings a try and see if it does anything. Thanks once again!

    So you dont need additional caching. You ahould be able to saturate 1Gbps with my settings. With stripping 2Gbps.
    Enabling aio and directio should massively help, Im not sure about sendfile because it depends on maasny things in system/files.

    Thanks a lot, will give these settings a try in around an hour. Will update this thread on how that goes! :)

  • The recommendations made by @AXYZE are very good and should help a lot. Here is my /etc/nginx/conf.d/io.conf file with thread pooled AIO:

    # Use sendfile by default
    sendfile on;
    
    # Prevent one request from starving I/O, limit sendfile I/O utilization
    sendfile_max_chunk 4m;
    
    # Use sendfile for small files, AIO for larger files than this
    directio 4m;
    # XFS requires 4K alignment
    directio_alignment 4k;
    
    # Use async thread pool for reads and writes
    aio threads;
    aio_write on;
    
    # Increase default output buffers & buffer size
    output_buffers 4 1m;
    

    And on the global level of /etc/nginx/nginx.conf:

    thread_pool default threads=4 max_queue=16384;

    Thanked by 1jugganuts
  • ArkasArkas Moderator

    @Jari That is very specific, I doubt it can be a blanket application to many Nginx installs.

  • @Arkas What exactly is "That"?
    Naturally, one should experiment with the numbers (buffer sizes) depending on the hardware and file size. We are talking about larger static files here (MP4 videos), not a "blanket application to many Nginx installs".

  • @AXYZE said:
    Depending on system different things help
    For serving static start with (add it to your config)

    sendfile off; 
    aio on; 
    directio 8m; 
    directio_alignment 4k;
    output_buffers 1 2m;
    

    And work from there. Play with on/off on first two settings, delete directio etc. and find what works the best for your server. I think this config is the best, but play with sendfile on/off to see which is better.
    Also try different filesystems, ZFS with 20GB ARC cache can do wonders.

    I get,

    "aio on" is unsupported on this platform

    When adding in that part, I'm using Ubuntu 18.04.05 LTS

  • I get,

    "aio on" is unsupported on this platform

    When adding in that part, I'm using Ubuntu 18.04.05 LTS

    nginx must be compiled with the option --with-file-aio, check using nginx -V. You may have to compile nginx from source. This would allow you to remove any modules that are not necessary for serving static files, as well as using more aggressive optimization flags (if you care about every little bit of performance). You may run into other bottlenecks on the way.

  • @AXYZE said:

    @sgno1 said:

    @AXYZE said:
    Depending on system different things help
    For serving static start with (add it to your config)

    sendfile on; 
    aio on; 
    directio 8m; 
    directio_alignment 4k;
    output_buffers 1 2m;
    

    And work from there. Play with on/off on first two settings, delete directio etc. and find what works the best for your server. I think this confog is the best, but play with sendfile on/off to see which is better.
    Also try different filesystems, ZFS with 20GB ARC cache can do wonders.

    My file system is already ZFS, but I'll give those NGINX settings a try and see if it does anything. Thanks once again!

    So you dont need additional caching. You should be able to saturate 1Gbps with my settings. With stripping 2Gbps.
    Enabling aio and directio should massively help, Im not sure about sendfile because it depends on maasny things in system/files.

    Much faster after enabling the other configs and turning off sendfile, did do a bit of port monitoring and I see it spike to around 280mbit/s when seeking on videos. Is this a convenient way to go about this? If there are multiple users seeking and its hitting the max port speed, wouldn't this be bad?

  • Rate limit on firewall

  • You can also configure BBR for your network. Great write up is here: https://www.linuxbabe.com/ubuntu/enable-google-tcp-bbr-ubuntu

  • exception0x876exception0x876 Member, Host Rep, LIR

    Consider adding SSD cache to your HDD array. That should help with IO seek time.

  • @sgno1 said:
    Much faster after enabling the other configs and turning off sendfile, did do a bit of port monitoring and I see it spike to around 280mbit/s when seeking on videos. Is this a convenient way to go about this? If there are multiple users seeking and its hitting the max port speed, wouldn't this be bad?

    Network usage is very good! It means it can read and send necessary data fast!
    Now you can limit connection speed. You can do it via firewall (limit per IP) or via nginx (limit per connection)

    limit_rate 600k
    limit_rate_after 500k
    

    First line will limit single connection to 600kB/s.
    Second will delay the limit so it activates after 500kB is downloaded (file header) and that will make sure that "play" button will work instantly.
    Now you need experiment with these values.
    limit_rate should be 30-40% higher than highest bitrate video in your collection.
    limit_rate_after should be 500k or 1m.

    You can setup another vps and check how it works with many streams (you can play with load balancing tools like JMeter, see online how yo configure them for mp4 files)
    And then see from your local connection if you can still watch videos with no problem. If it lags then try with different rate limit, different settings in config that I mentioned above... :)

    Thanked by 3niknar1900 sgno1 kkrajk
  • @AXYZE said:

    @sgno1 said:
    Much faster after enabling the other configs and turning off sendfile, did do a bit of port monitoring and I see it spike to around 280mbit/s when seeking on videos. Is this a convenient way to go about this? If there are multiple users seeking and its hitting the max port speed, wouldn't this be bad?

    Network usage is very good! It means it can read and send necessary data fast!
    Now you can limit connection speed. You can do it via firewall (limit per IP) or via nginx (limit per connection)

    limit_rate 600k
    limit_rate_after 500k
    

    First line will limit single connection to 600kB/s.
    Second will delay the limit so it activates after 500kB is downloaded (file header) and that will make sure that "play" button will work instantly.
    Now you need experiment with these values.
    limit_rate should be 30-40% higher than highest bitrate video in your collection.
    limit_rate_after should be 500k or 1m.

    You can setup another vps and check how it works with many streams (you can play with load balancing tools like JMeter, see online how yo configure them for mp4 files)
    And then see from your local connection if you can still watch videos with no problem. If it lags then try with different rate limit, different settings in config that I mentioned above... :)

    So if one of my videos is 3300kB in total for video bitrate, does it need to be 30-40% higher than this value, or does it need to be calculated in a different way?

Sign In or Register to comment.