Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


High CPU usage Issue
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

High CPU usage Issue

emdademdad Member

Guys I need a little help. I've been troubled understanding CPU usage of the server I'm working on.

As per bellow image, the load average is 20 and some processes taking about 600-900% CPU but when I check using 1 in top it shows me only 20-30% of every CPU are being used on average.

How do I know, what is the real usage condition of my server?

**I'm running 10 ffmpeg instance to take input from live stream, encoding each stream in 3 different bitrate (one is passthrough) & pass them to Wowza Server. My output streams are lagging. This is a Dual Intel Xeon E5-2620 v3 @ 2.40GHz CPU.

Comments

  • exception0x876exception0x876 Member, Host Rep, LIR
    edited March 2016

    check the niceness column, your CPU load is maxed. Perhaps you should consider some sort of hardware accelerated encoding.

  • E5 = crappy single thread, go for E3 for encoding

  • Assuming you are using ffmpeg with libx264, multi-thread is already enabled by default, so your 10 instances are using all 24 threads. Most of the encoding takes place in the nice column, so libx264 is using low priority cpu slices to perform the encoding, and will gracefully degrade quality if there is not enough cpu.

    Output latency is to be expected when transcoding content. You can however try to optimize ffmpeg for latency if you look around for guides on that, e.g. https://trac.ffmpeg.org/wiki/StreamingGuide

  • emdademdad Member

    @exception0x876 said:
    check the niceness column, your CPU load is maxed. Perhaps you should consider some sort of hardware accelerated encoding.

    But nice column is not showing maxed out yet, please suggest.

  • Can you check iostat so we can see your iowait?

  • FalzoFalzo Member
    edited March 2016

    man nice

    your ffmpeg processes seem to run with a niceness of -3 which gives them more priority not less! if you do set this explicitly somewhere and want them to be more nice (aka less priority) you need to use a positive value in your nice command.

    edit says: that you see cpu percentages like 600% or 800% on a single process is right, as it simply sums up the usages of the multiple threads that process uses. 24 equals a max of 2400% overall.

    I'd say your system is somewhat stressed yet, but not overloaded.

  • emdademdad Member

    @BradND said:
    Can you check iostat so we can see your iowait?

    Thanks, Please check my iostat output.

  • emdademdad Member

    @Falzo said:
    your ffmpeg processes seem to run with a niceness of -3 which gives them more priority not less! if you do set this explicitly somewhere and want them to be more nice (aka less priority) you need to use a positive value in your nice command.

    Thanks for your response. yes, It was intentional. I was trying to use maximum CPU for encoding.

  • Iowait looks OK - so not waiting for IO

    What codecs are you running? Only some of them will work with multi-threading correctly...

  • emdademdad Member

    @wwabbit said:
    Assuming you are using ffmpeg with libx264, multi-thread is already enabled by default, so your 10 instances are using all 24 threads. Most of the encoding takes place in the nice column, so libx264 is using low priority cpu slices to perform the encoding, and will gracefully degrade quality if there is not enough cpu.

    Output latency is to be expected when transcoding content. You can however try to optimize ffmpeg for latency if you look around for guides on that, e.g. https://trac.ffmpeg.org/wiki/StreamingGuide

    Thanks for your suggestion. As per my understanding, if I specify which threads to use for ffmpeg, will libx264 get more CPU to use? I can increase priority using nice for ffmpeg, can I setlibx264 encoding priority high?

  • emdademdad Member

    @BradND said:
    Iowait looks OK - so not waiting for IO

    What codecs are you running? Only some of them will work with multi-threading correctly...

    I'm using libx264 for video & audio is pass-through. This is ffmpeg command I'm using -

    ffmpeg -re -i input -acodec copy -codec:v libx264 -profile:v high -b:v 700k -maxrate 2000k -bufsize 2000k -vf scale=960:540 -threads 0 -strict experimental -f flv ouput

  • FalzoFalzo Member

    @emdad said:

    I don't think this will always help. if you priorize the encoding itself you may hinder other tasks to get the cpu power they need to assure lagless processing.

    I can only guess, but I'd try leaving standard niceness or even lowering priority a bit amd instead have a look at your lot of free memory and try to put it to a good use.
    I don't exactly know which options ffmpeg will offer regarding this, but maybe there are possibilities to get a bigger buffering to avoid lags.

    And as I see there is some java I'd suggest to have a look in the performance tuning of that too, if you haven't already done yet.

  • emdademdad Member

    Falzo said: And as I see there is some java I'd suggest to have a look in the performance tuning of that too, if you haven't already done yet.

    Java being used to run Wowza Server with Wowza recommended performance tuning. My ffmpeg encoded streams are input source of another Wowza Server, buffering shouldn't be big issue in here. I'm not sure though.

  • FalzoFalzo Member

    emdad said: ffmpeg -re -i input -acodec copy -codec:v libx264 -profile:v high -b:v 700k -maxrate 2000k -bufsize 2000k -vf scale=960:540 -threads 0 -strict experimental -f flv ouput

    try raising that bufsize. AFAIR the buffer is filled depending on maxrate setting though, but I'd say buffer should be a fair amount bigger than maxrate can fill in one second.

    you're screenshot shows only five ffmpeg processes running, it seems reasonable that the system may not be capable of handling concurrent ten of these.
    hence why I say you need to find a way to relieve on cpu power and I'll stand by my first point: giving even more priority to ffmpeg will make things worse... ;-)

  • Your ffmpeg instances are using most of the 24 CPUs already, changing nice is not going to change that. You need to stop assuming that the latency is the caused by high CPU usage. I am pretty sure it isn't. You can confirm this by running only one stream on one instance of ffmpeg.

    Try reducing your -bufsize to 700k to match your bitrate, this should reduce latency from 3 seconds to 1 second.

  • FalzoFalzo Member

    @wwabbit said:
    Your ffmpeg instances are using most of the 24 CPUs already, changing nice is not going to change that. You need to stop assuming that the latency is the caused by high CPU usage. I am pretty sure it isn't.

    this!

    Try reducing your -bufsize to 700k to match your bitrate, this should reduce latency from 3 seconds to 1 second.

    reducing latency yes, but getting rid of occasional lagging no. I'd assume it'll probably make things more jerky...

  • exception0x876exception0x876 Member, Host Rep, LIR

    starting from ffmpeg 2.8.0 there is QuickSync support for x264 encoding. You may try it as well.

  • emdademdad Member
    edited March 2016

    @wwabbit said:
    Try reducing your -bufsize to 700k to match your bitrate, this should reduce latency from 3 seconds to 1 second.

    @Falzo said:
    reducing latency yes, but getting rid of occasional lagging no. I'd assume it'll probably make things more jerky...

    Thanks guys, but my issue isn't latency, video gets distorted,jerky & doesn't play smoothly.

    @exception0x876 said:
    starting from ffmpeg 2.8.0 there is QuickSync support for x264 encoding. You may try it as well.

    Will try that, just for clarification, do I have to use any hardware acceleration to use Quicksync?

  • You will just have to reduce the number of streams and/or the quality of the encoding until you get acceptable performance.

    Start with -preset ultrafast and see how high you can increase the quality without issues.

  • FalzoFalzo Member

    @wwabbit said:
    You will just have to reduce the number of streams and/or the quality of the encoding until you get acceptable performance.

    Start with -preset ultrafast and see how high you can increase the quality without issues.

    agreed.

  • exception0x876exception0x876 Member, Host Rep, LIR

    @emdad said:
    Will try that, just for clarification, do I have to use any hardware acceleration to use Quicksync?

    Yes, that is the entire point of it. You need an access to Intel integrated GPU

Sign In or Register to comment.