502 Gatway error Every 2 days or so

Read 29546 times
Okay I am going to start this at the Top and do everything suggested to me 1 at a time in hopes that Someone will be able to figure out what is going on.

I got Centova Cast on a Debian Server installed and working Fine. Every day or 2 I get the 502 Gateway Error and i have to re-install it everytime. you can see how I am very frustrated since Centova is unhelpful in that all they say is it's Something other then the software. I got with the server provider and they can not find any logs about anything not working.

Centova Cast gives me error after error about the cron job, and nothing i do will fix it. so i re installed this morning and by tomorrow i will have the same Gateway error. and i need help fixing it by then.
Please any and everyone help me
We also had this problem and found out that Crond was running twice... And because of that, it would clash with itself causing ours to give the 502 error...

Once we fixed that issue, we havent had the 502 error since then  (knock on wood).

That was "our" issue anyways.   But check to make sure that Crond isnt running twice.  If it is kill 1 of them off, and the error should stop.
CrossFire-Hosting LLC.
Co-Owner
Thank you DJ fire
How did you kill one of them? I ran the "Killall -9" something script but it did not work for me. I don't recall the error but when this system crashes, which it will. I will do it again and post the result here.
Do this ONLY if you see that you have 2 crond running.

Do this through SSH logged in as root.


first run:

ps aux | grep cron


If you see 2 crond running... Look at the time stamp, and the one with an earlier date on it, get the PID of that one.

then  use

kill -9 PID

Which you would replace PID with the actual PID number

Then run

 ps aux | grep cron

If you only see 1 crond running now...  Then stop and start crond 

service crond stop
service crond start


And then 1 final

 ps aux | grep cron

And it should all be good.
CrossFire-Hosting LLC.
Co-Owner
so i did check the Crond Job and there was only 1 running and this is also what i got on it when i tried to restart it.



Last login: Sun May 26 14:37:28 2013 from 67.161.119.249
[ds5950]$ ps aux | grep cron
asapcpu1  6181  0.0  0.0 125160   844 pts/0    S+   02:36   0:00 grep cron
[ds5950]$ service crond stop
-bash: service: command not found
[ds5950]$ service crond start
-bash: service: command not found
[ds5950]$ root service crond stop
-bash: root: command not found
[ds5950]$ sudo service crond stop
[sudo] password for asapcpu1:
crond: unrecognized service
[ds5950]$ sudo service crond start
crond: unrecognized service
[ds5950]$  ps aux | grep cron
asapcpu1  6200  0.0  0.0 163156   844 pts/0    S+   02:37   0:00 grep cron
'service' is a script that just points you back to the /etc/init.d directory and the scripts within it.  Not all distros of Linux have the script present.  It's perfectly safe and permissible to add the script to your server yourself, though you'll need to be root to do it.

The script should be called 'service', and goes here:  /user/sbin/service

Here's the contents of that script:

#!/bin/sh

. /etc/init.d/functions

VERSION="`basename $0` ver. 0.91"
USAGE="Usage: `basename $0` < option > | --status-all | \
[ service_name [ command | --full-restart ] ]"
SERVICE=
SERVICEDIR="/etc/init.d"
OPTIONS=

if [ $# -eq 0 ]; then
   echo "${USAGE}" >&2
   exit 1
fi

cd /
while [ $# -gt 0 ]; do
  case "${1}" in
    --help | -h | --h* )
       echo "${USAGE}" >&2
       exit 0
       ;;
    --version | -V )
       echo "${VERSION}" >&2
       exit 0
       ;;
    *)
       if [ -z "${SERVICE}" -a $# -eq 1 -a "${1}" = "--status-all" ]; then
          cd ${SERVICEDIR}
          for SERVICE in * ; do
            case "${SERVICE}" in
              functions | halt | killall | single| linuxconf| kudzu)
                  ;;
              *)
                if ! is_ignored_file "${SERVICE}" \
                    && [ -x "${SERVICEDIR}/${SERVICE}" ]; then
                  env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/$                  {SERVICE}" status
                fi
                ;;
            esac
          done
          exit 0
       elif [ $# -eq 2 -a "${2}" = "--full-restart" ]; then
          SERVICE="${1}"
          if [ -x "${SERVICEDIR}/${SERVICE}" ]; then
            env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVI                  CE}" stop
            env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVI                  CE}" start
            exit $?
          fi
       elif [ -z "${SERVICE}" ]; then
         SERVICE="${1}"
       else
         OPTIONS="${OPTIONS} ${1}"
       fi
       shift
       ;;
   esac
done

if [ -x "${SERVICEDIR}/${SERVICE}" ]; then
   env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" "${SERVICEDIR}/${SERVICE}" ${OP                  TIONS}
else
   echo $"${SERVICE}: unrecognized service" >&2
   exit 1
fi

how do i add it using putty?
There are a couple of ways.

I'd recommend making a text file and using FTP to upload it, that's the easiest way to get the script up there to start with.  Then use Putty to move the file into its correct location and set the permissions.

Otherwise, you could use a text editor like Vi, Emacs or something.  Open the 'service' file in place in /sbin/, copy the text from the web page, and then switch to Putty and right-click to paste into the editor.  Then save the file.  That can be a little tricky, though.  But now you have two ways to do it.

Good luck.
So I'm working on getting the script on the server now as I type this, but as i am I am thinking to myself. What is causing more then one Crond Job running at the same time? is it a fault in the software? a Delayed script. what? :o
Also /usr/sbin/service is NOT a directory, rather a binary file correct? so i need to txt editor it and add it? please explain a little bit :-[
From the looks of that out put you gave, you don't have crond running... Either 1. your system uses something else.. or 2.  you dont have crond installed.

If you have cron job error, that could be why.

You could try this out:

https://secure.centova.com/pages/faqs/display/diagnosing_cron_job_problems_113

And see where that takes you.  And go from there.


but the reason 2 crons could be running could be from anything.  Ours was from having CCv2 and upgrading to CCv3... Seems a cron got stuck and ended up running twice.
CrossFire-Hosting LLC.
Co-Owner
/usr/local/centovacast/bin/ccmanage cronjob all --debug

Resulted in


[INF] Completed job: checkprocesses
[INF] Starting job: hourlymaint
[INF] Performing hourly maintenance...
[INF] Completed job: hourlymaint
[INF] Starting job: pollhosts
[DBG] Polling host Local server ...
[DBG] Updated status cache
[INF] Completed job: pollhosts
Result: OK Complete


and yesterday when it was running just fine I saw 2 Crond Jobs running and i was able to close 1 of them but when i restarted the server 2 times, the server had to be manually rebooted. since yesterday it's now a waiting game to see when it goes down next. let's hope it don't
Yes, you're creating a file, not a directory. 
So using Midnight comander i got the Service.txt into the /usr/sbin/ but i'm not sure how to get it into the *service file from there

http://prntscr.com/17b0lz

And this morning when i got up the server was Down again with a 502 Bad Gateway.

And running: ps aux | grep cron
only showed 1 Crond job running

Aslo trying to Fixprims led to:

[ds5950]$ sudo /usr/local/centovacast/sbin/fixperms
[sudo] password for asapcpu1:
chown: invalid user: `root.ccuser'
chgrp: invalid group: `centovacast'
chgrp: invalid group: `ccuser'
chgrp: invalid group: `ccuser'
chown: invalid user: `root.ccuser'
chown: invalid user: `ccuser.ccuser'


Running the /usr/local/centovacast/bin/ccmanage cronjob all --debug resulted in this:

[INF] Completed job: logprocess
[INF] Starting job: hourlymaint
[INF] Performing hourly maintenance...
[INF] Completed job: hourlymaint
[INF] Starting job: pollhosts
[DBG] Polling host Local server ...
[INF] Error contacting host: Cluster host connection failure for Local server: C                                                                             onnection refused (111)
[INF] Completed job: pollhosts
[DBG] Cluster host connection failure for Local server: Connection refused (111)                                                                              at /usr/local/centovacast/system/class_RPCDaemonClient.php:1063
[DBG] #0 /usr/local/centovacast/system/class_RPCDaemonClient.php(1063): RPCDaemo                                                                             nClient::send()
[DBG] #1 /usr/local/centovacast/system/class_RPCDaemonClient.php(1009): RPCDaemo                                                                             nClient->send()
[DBG] #2 /usr/local/centovacast/system/class_RPCDaemonClient.php(900): RPCDaemon                                                                             Client->command()
[DBG] #3 /usr/local/centovacast/system/class_UserSession.php(222): RPCDaemonClie                                                                             nt->apply_quota()
[DBG] #4 /usr/local/centovacast/system/class_SystemControl.php(216): Identity->r                                                                             pc()
[DBG] #5 /usr/local/centovacast/system/class_SystemControl.php(341): SystemContr                                                                             ol->pop_usercontext()
[DBG] #6 /usr/local/centovacast/system/ccmanage.php(293): SystemControl->__call(                                                                             'cronjob', Array)
[DBG] #7 {main}
Result: ERROR Cluster host connection failure for Local server: Connection refus                                                                             ed (111)


Centova says: it's likely that a past cron job has frozen up for some reason.  This can usually be fixed by logging in via SSH as root and running "killall -9 ccmanage".

so running that gets me

[ds5950]$ sudo killall -9 ccmanage
ccmanage: no process found

so far this is the only thing i can do that fixes it BUT ONLY for a day or 2

root #!/usr/bin/env bash
# pull in database configuration values
. /usr/local/centovacast/etc/centovacast.conf

# uninstall
echo 'UNINSTALL' | /usr/local/centovacast/sbin/uninstall --i-want-to-delete-all-my-data

# clear the database
echo "DROP DATABASE $DB_NAME; CREATE DATABASE $DB_NAME;" | mysql -h$DB_HOST -u$DB_NAME -p$DB_PASS

# clean up any remaining data from the old installation (shouldn't be any, but just in case)
rm -rf /usr/local/centovacast /etc/centovacast.conf

# automate the reinstallation; tweak the parameters to your tastes
sudo ./install.sh --icecast-all --channel=unstable --admin-email=tech@asapcpu.com --admin-pass=newcastle --dbname=$DB_NAME --dbuser=$DB_USER --dbpass=$DB_PASS --dbhost=$DB_HOST --force
Last Edit: May 29, 2013, 02:41:10 pm by asapcpu
Code: [Select]
[ds5950]$ sudo /usr/local/centovacast/sbin/fixperms
[sudo] password for asapcpu1:
chown: invalid user: `root.ccuser'
chgrp: invalid group: `centovacast'
chgrp: invalid group: `ccuser'
chgrp: invalid group: `ccuser'
chown: invalid user: `root.ccuser'
chown: invalid user: `ccuser.ccuser'

A permission fix that worked for us, when we had the same problem with the fixperms, in receiving the invalid group message.

Code: [Select]
usermod -G psacln,centovacast centovacast


cd to /var/log/

Code: [Select]
su - centovacast -s /bin/bash

It should allow you to do the following:

Code: [Select]
/var/log# su - centovacast -s /bin/bash

Then run
Code: [Select]
/usr/local/centovacast/bin/ccmanage cronjob all

which should result in:
Code: [Select]
[INF] Cron job startup at (date / time will show here)
[INF] Starting job: checkprocesses
[INF] Checking for crashed applications...
[INF] Completed job: checkprocesses
Result: OK Complete
CrossFire-Hosting LLC.
Co-Owner