Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Error 502 Bad gateway
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Error 502 Bad gateway

xaitmixaitmi Member
edited April 2016 in Help

Hi.

I haven't made any changes to my webserver setup but all of a sudden this morning I get 502 Bad Gateway. I tried restarting nginx on all my webservers but still the same issue.

Basically my setup is

Cloudflare --> Nginx Reverse Proxy VPS --> Actual Website VPS

Anyone have any suggestions as to what to check?

I made sure all servers have disk space available, nginx restarted etc.

I even reinstalled nginx on the reverse proxy.

If I access each server directly by IP in my browser the site loads, but when I use the domain cloudflare just says 502 bad gateway.

Update: after checking error_log there are thousands of these

2016/04/19 09:05:52 [error] 17324#17324: *28 connect() failed (111: Connection refused) while connecting to upstream, client: 108.162.216.239, server: domain.com, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "domain.com", referrer: "https://domain.com/"

How do I fix this

Comments

  • If you use centos check /etc/php-fpm.d/www.conf find user and group check is it have permision to use your web root, and also check /var/lib/php

  • Update: For some reason php-fpm stopped running on multiple VPS's hosting different sites, after restarting that on all servers it all started working.

    Very odd

  • Php could be down or there could be too many php connections.

    Check the php logs.

  • elgselgs Member
    #!/bin/bash
    
    err502=`curl -s https://your_website.com/ | grep 502`
    
    if [[ "$err502" == *"502"* ]]; then
      /usr/sbin/service php5-fpm restart
      date >> /tmp/php5-fpm-restart.log
    fi
    

    I put the script above into a sh file and run it with crontab every minute.

    Thanked by 2budi1413 ATHK
  • @elgs said:
    #!/bin/bash

    err502=curl -s https://your_website.com/ | grep 502

    if [[ "$err502" == "502" ]]; then
    /usr/sbin/service php5-fpm restart
    date >> /tmp/php5-fpm-restart.log
    fi

    I put the script above into a sh file and run it with crontab every minute.

    Hi I tried using your script today because same thing happened today but it's not working for me.

    When I run curl -s https://domain.com/ | grep offline or curl -s https://crypticvpn.com/ | grep 502 nothing happens.

    Maybe because im using cloudflare?

  • elgselgs Member

    I apologize. I also found crontab does not like /usr/sbin/service. I changed to /sbin/restart, and it works on Ubuntu.
    #!/bin/bash

    err502=`curl -s https://your_website.com/ | grep 502`
    
    if [[ "$err502" == *"502"* ]]; then
      /sbin/restart php5-fpm restart
      date >> /tmp/php5-fpm-restart.log
    fi
    
  • xaitmixaitmi Member
    edited April 2016

    @elgs said:
    I apologize. I also found crontab does not like /usr/sbin/service. I changed to /sbin/restart, and it works on Ubuntu.
    #!/bin/bash

    err502=curl -s https://your_website.com/ | grep 502

    if [[ "$err502" == "502" ]]; then
    /sbin/restart php5-fpm restart
    date >> /tmp/php5-fpm-restart.log
    fi

    When I run

    curl -s https://site.com | grep 502

    shouldn't it output something? When I run it nothing happens.
    I use Centos 6 I just need it to do

    service php-fpm restart
  • elgselgs Member
    edited April 2016

    @xaitmi said:
    When I run

    curl -s https://gocryptic.com | grep 502

    shouldn't it output something? When I run it nothing happens.
    I use Centos 6 I just need it to do

    service php-fpm restart

    It depends on if the website gives back 502 error or not. If it outputs nothing, most likely the website is working properly. You can only catch it when the website actually is in 502 error.

  • Update: When I do curl without the -s tag it says SSL Error

  • elgselgs Member
    edited April 2016

    @xaitmi said:
    Update: When I do curl without the -s tag it says SSL Error

    If you don't care about the SSL error, add -k.

    err502=`curl -sk https://your_website.com/ | grep 502`
    
  • @elgs said:

    @xaitmi said:
    Update: When I do curl without the -s tag it says SSL Error

    If you don't care about the SSL error, add -k.

    err502=`curl -sk https://your_website.com/ | grep 502`
    

    still says curl: (35) SSL connect error
    ( i did it without the -s tag)

  • elgselgs Member
    edited April 2016

    still says curl: (35) SSL connect error
    ( i did it without the -s tag)

    Ah! I see. Normally it's not a problem. If you don't like it, add 2>&1. Make it look like this:

    err502=`curl -sk https://your_website.com/ 2>&1 | grep 502`
    

    By adding 2>&1, you explicitly redirected the stderr to the stdout. Before By adding 2>&1, stderr is not filtered by grep while it goes to the same terminal as stdout.

  • xaitmixaitmi Member
    edited April 2016

    @elgs said:

    still says curl: (35) SSL connect error
    ( i did it without the -s tag)

    Ah! I see. Normally it's not a problem. If you don't like it, add 2>&1. Make it look like this:

    err502=`curl -sk https://your_website.com/ 2>&1 | grep 502`
    

    By adding 2>&1, you explicitly redirected the stderr to the stdout. Before By adding 2>&1, stderr is not filtered by grep while it goes to the same terminal as stdout.

    still outputs nothing even tho site is offline.

  • Found this online and it works

    curl -1IsS --ciphers ecdhe_ecdsa_aes_128_sha https://domain.com | grep 502

    outputs HTTP/1.1 502 Bad Gateway

  • iptables okay ?

  • Ya everythings ok now I got the script working with that command i found above.

  • FalzoFalzo Member

    @xaitmi: out of interest would you mind sharing which control panel you are using (if so) or what you host there and maybe which provider you are with (not to shame but to check if this may be a link on the the issue).

    I had the same issue(s) at probably the same time window as you, a magically disappering php-fpm service... first time may be a coincidence, second time - hmmm.

    I am using a setup which involves VestaCP running only nginx+php-fpm which serves a xenforo install. thats all on a 768MB vm with hostus in DC (which achieved 3 month uptime so far without any issues)

  • @elgs said:
    I apologize. I also found crontab does not like /usr/sbin/service. I changed to /sbin/restart, and it works on Ubuntu.
    #!/bin/bash

    err502=curl -s https://your_website.com/ | grep 502

    if [[ "$err502" == "502" ]]; then
    /sbin/restart php5-fpm restart
    date >> /tmp/php5-fpm-restart.log
    fi

    This is just a temporarily fix, this doesn't fix the source of your issue. I recommend you to dig further and fix it permanently.

  • xaitmixaitmi Member
    edited April 2016

    @Falzo said:
    @xaitmi: out of interest would you mind sharing which control panel you are using (if so) or what you host there and maybe which provider you are with (not to shame but to check if this may be a link on the the issue).

    I had the same issue(s) at probably the same time window as you, a magically disappering php-fpm service... first time may be a coincidence, second time - hmmm.

    I am using a setup which involves VestaCP running only nginx+php-fpm which serves a xenforo install. thats all on a 768MB vm with hostus in DC (which achieved 3 month uptime so far without any issues)

    I don't use a control panel but I use CentMinMod
    The host I am using is HostUs their $12/year 786mb ram package (theres a special link on LET)

    I have the same setup on 4 VPS's, and 2 of the HostUs always have a disappearing PHP-FPM at the exact same time, the other VPS's are DigitalOcean and Ramnode and those never have this issue.

    It's only been happening on HostUs so far.

    Maybe @alexanderM can take a look since you are also having the same issue on HostUs?

    by the way I am very happy with HostUS, never had a single downtime yet since I joined months ago.

  • FalzoFalzo Member
    edited April 2016

    @xaitmi said:
    by the way I am very happy with HostUS, never had a single downtime yet since I joined months ago.

    same as me, but as said before I doubt it being coincidence esp. if it is not exactly the same software setup. I have the same product $12/year 768MB being on node WDC01-VZ48.

    please do note: I am far from complaining or blaming hostus as I my expectations are already more than satisfied given the price ;-) (even if it stays this way randomly dropping php-fpm)

    but it may be interesting though, why sometimes processes get killed and seemingly not random ones but php-fpm.
    on a side note: I recognized that this not only happened to the normale php-fpm procs but those vestacp uses with its own/secondary nginx-setup too.

    so maybe some kind of forgotten garbage collection on the node which tries to remove dead or stalled php-fpm processes? hopefully that has been some kind of manual task and not based on schedule. ;-)

    @AlexanderM if you'd like to look into that issue and need more information just let me know, how to help.

  • xaitmixaitmi Member
    edited May 2016

    @falzo this has gotten really out of hand this week, in the last 24 hours alone it's been killed and restarted 51 times...

    I sent @AlexanderM a PM about this.

    The script that we wrote from the help in this thread comes in very handy

    If I didn't have this script I'd have to manually restart it each time...

  • FalzoFalzo Member

    @xaitmi said:

    thanks for tagging as I am for sure interested in the progress of such things.

    but I am afraid I can't be much of help anymore as I did move my xenforo install to another VM on one of my dedis elsewhere in the end of april (KVM, far more RAM involved) ... as it got more into production state and I didn't wanted to rely on a small dev box any further ;-)

    just took a quick ps at my hostus and it looks like:

    root     27405  0.0  0.3 229684  2456 ?        Ss   Apr23   1:33 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
    admin    27406  0.0  1.9 242800 15352 ?        S    Apr23   0:30  \_ php-fpm: pool 
    admin    27407  0.0  1.7 240476 14076 ?        S    Apr23   0:30  \_ php-fpm: pool 
    admin    27408  0.0  1.8 240876 14548 ?        S    Apr23   0:29  \_ php-fpm: pool 
    admin    27409  0.0  1.4 238600 11480 ?        S    Apr23   0:03  \_ php-fpm: pool 
    admin    27411  0.0  1.9 242880 15340 ?        S    Apr23   0:30  \_ php-fpm: pool 
    admin    10409  0.0  1.8 242464 14808 ?        S    Apr23   0:15  \_ php-fpm: pool 
    admin    20597  0.0  1.9 242800 15252 ?        S    Apr23   0:04  \_ php-fpm: pool 
    admin    23373  0.0  1.3 237980 10540 ?        S    Apr23   0:00  \_ php-fpm: pool 
    

    I don't use any monitoring/restart script on this, so suprisingly there are just old php-fpm processes running without any issues.
    keep in mind, that those probably haven't served anything, as the content is gone and the nginx/php-fpm isn't used for anything else since then.

    this makes me guess, that probably those terminations only happen when you hit some unknown limits maybe in terms of process count or something like that - which only will happen if there is some reasonable traffic to your site and which would make it more likely that things can become worse, if you have some noticable increase in visitors or something like that.

    can't think of things with xenforo itself which should make php-fpm terminate otherwise... neither do I know what kind of scripts you are running on your install at all ;-)
    hopefully @AlexanderM can shed a light on this for you!

  • WeikangWeikang Member

    You should check the logs.

Sign In or Register to comment.