I just got a mail from the crondaemon with this subject:
Cron <centovacast@server9> /home/centovacast/system/runascc/runascc exec ccmanage cronjob all >/dev/null

The content of the mail was this:
kill 3292: Operation not permitted

Earlier this week, this cronjob kept being busy for a long long time which caused my firewall to notice me that a process was running for longer then normal.

What is going on here?
It's not the license file because it's up to date. I did a search on the forums but nothting to be found.

Just to be sure I tried a manual debug (I have root access to the server) and this is what came out, looks good to me:

sudo -u centovacast /home/centovacast/system/runascc/runascc exec ccmanage cronjob all --debug
INF Cron job startup at 2012-12-13 20:48:40
INF Checking for crashed servers...
DBG Begin account check
DBG Checking server process for centrum (expected: up)
DBG Checking PID 3326
DBG /proc/3326 state: State:    S (sleeping)
DBG Detected as up
DBG Check complete
DBG Source disabled for this stream, skipping
DBG centrum is broadcasting at 128
DBG Checking server process for kabel (expected: up)
DBG Checking PID 3331
DBG /proc/3331 state: State:    S (sleeping)
DBG Detected as up
DBG Check complete
DBG Source disabled for this stream, skipping
DBG kabel is broadcasting at 192
DBG Checking server process for kerst (expected: up)
DBG Checking PID 3336
DBG /proc/3336 state: State:    S (sleeping)
DBG Detected as up
DBG Check complete
DBG Source disabled for this stream, skipping
DBG kerst is broadcasting at 128
DBG Checking server process for pirate (expected: up)
DBG Checking PID 3341
DBG /proc/3341 state: State:    S (sleeping)
DBG Detected as up
DBG Check complete
DBG Source disabled for this stream, skipping
DBG pirate is broadcasting at 128
DBG Checking server process for plusradio (expected: up)
DBG Checking PID 3292
DBG /proc/3292 state: State:    S (sleeping)
DBG Detected as up
DBG Check complete
DBG Source disabled for this stream, skipping
DBG plusradio is broadcasting at
DBG Checking server process for rtvpresident (expected: up)
DBG Checking PID 3350
DBG /proc/3350 state: State:    S (sleeping)
DBG Detected as up
DBG Check complete
INF Log rotation frequency is 12, log process frequency is 12, last processed 2012-12-13 20:30:01, last rotated 2012-12-13 20:30:01, currently 2012-12-13 20:48:41

Why is it giving this "operation not permitted" error? And what operation is not permitted?

I just got a mail from the crondaemon with this subject:
Cron <centovacast@server9> /home/centovacast/system/runascc/runascc exec ccmanage cronjob all >/dev/null

The content of the mail was this:
kill 3292: Operation not permitted

That means Centova Cast tried to stop (aka "kill") a process, but the process was not running under the 'centovacast' user account so the "kill" operation was not permitted by the Linux kernel.

Usually this means someone/something has been tampering with your Centova Cast file ownerships/permissions and now Centova Cast is running in a damaged state (specifically, for the technically inclined: the setuid binary has lost its setuid bit).  If that's the case, this is the first of many weird symptoms you're going to see.

You'd need to terminate all ices/sc_trans/sc_serv/icecast processes, then follow the steps in the disaster recovery article in our KB to repair this.

Oke thank you for the advice. I will do that.

Just a little additional question.

I went to have a look and see that in the main centovacast webroot directory (which is located at /home/user/domains/domainname/public_html/centova), all userfiles are owned by user and user's group like user:user.
All directory's are centovacast:centovacast.

However, in all subdirectory's all files are owned by centovacast:centovacast.

Should Centovacast's main webroot directory not also be owned by centovacast?

You'd need to terminate all ices/sc_trans/sc_serv/icecast processes, then follow the steps in the disaster recovery article in our KB to repair this.

Now I'm pissed. The streams at least restarted automatically before this.

Now I followed these instructions and as a result sc_serv is not starting anymore. This was the only one we were using for the clients.

When I run the debug I get this now:

sudo -u centovacast /home/centovacast/system/runascc/runascc exec ccmanage cronjob all --debug
INF Cron job startup at 2012-12-19 03:07:02
INF Checking for crashed servers...
DBG Begin account check
DBG Checking server process for centrum (expected: down)
DBG No PID: 0
DBG Check complete
DBG Source disabled for this stream, skipping
DBG centrum is broadcasting at
DBG Checking server process for kabel (expected: down)
DBG No PID: 0
DBG Check complete
DBG Source disabled for this stream, skipping
DBG kabel is broadcasting at
DBG Checking server process for kerst (expected: down)
DBG No PID: 0
DBG Check complete
DBG Source disabled for this stream, skipping
DBG kerst is broadcasting at
DBG Checking server process for pirate (expected: down)
DBG No PID: 0
DBG Check complete
DBG Source disabled for this stream, skipping
DBG pirate is broadcasting at
DBG Checking server process for plusradio (expected: down)
DBG Checking PID 3292
DBG /proc/3292 does not exist; process down
DBG Check complete
DBG Source disabled for this stream, skipping
DBG plusradio is broadcasting at
DBG Checking server process for rtvpresident (expected: down)
DBG Checking PID 3350
DBG /proc/3350 does not exist; process down
DBG Check complete
DBG Source disabled for this stream, skipping
DBG rtvpresident is broadcasting at
DBG Checking server process for webpres (expected: down)
DBG Checking PID 3356
DBG /proc/3356 does not exist; process down
DBG Check complete
DBG Source disabled for this stream, skipping
DBG webpres is broadcasting at
DBG End account check
OK Check complete
INF Log rotation frequency is 12, log process frequency is 12, last processed 2012-12-18 21:15:01, last rotated 2012-12-18 21:15:01, currently 2012-12-19 03:07:03

I already rebooted the complete server, but that would not help.

Oke, the rights in the disaster recovery must be wrong somewhere.

I managed to start te servers, don't ask me how. I logged in as the client, started the server, got notice that it started succesfully.
Then I had to go via SSH, see that the server was started under the admin account. Kill that sc_serv pid and after about a minute, the server restarted automatically under the centovacast user.

I had to do that for all clients seperately. This was not the case before the disaster recovery, then all worked fine except for that crontab kill error.

However especially in client sections:
1.) Logfiles can not be read ("Unable to access log file: Could not open log file" is the error notice)
This is an example of how logfiles are present in the vhosts/*/var/log directory.
-rw------- 1 centovacast centovacast  967 Dec 19 01:29 access.log
2.) Servers is displayed as being offline, while it's online
Pidfile is looking like this:
-rw------- 1 centovacast centovacast 4 Dec 19 03:30 server.pid
3.) Source connected NO (while it is in fact yes)
4.) Listeners: none (while in fact there are listeners)

So it must have to do something which changed in the wrong way when doing the disaster recovery which I did very exactly as shown.

I'm afraid we have to do a reinstall, but something must be wrong with that disaster recovery manual, otherwise my server wouldn't have gone down.

So I still would appreciate a fix.

Fixed it myself.

The disaster recovery does NOT set the setuid bit either.
So I did it myself by using chmod 4555 on runascc and castd and all my problems mentioned were over.

Just that easy.

Disaster recovery should have done that though.

And to inform you and other users, the fixperms.sh is not working correctly.

I use DA and gave this as webroot directory (I masked the domain here):
/home/admin/domains/domain.nl/public_html/centova
also tried this with a trailing slash. But neither worked, the script kept telling me that the webroot was not existing there, while it is.
So maybe the check for DA users can be fixed.

The disaster recovery does NOT set the setuid bit either.

It sure does, if you follow the instructions correctly.

Step 4 of the "Manual recovery" section of the Disaster Recovery article rebuilds both castd and runascc, and the Makefile automatically sets the setuid bit.

And to inform you and other users, the fixperms.sh is not working correctly.

Check the "NOTE:" under step 1 which explicitly states that it may fail on some configurations and explains how to proceed if that happens.

It sure does, if you follow the instructions correctly.

Well I followed the steps correctly, read: I copyed and pasted every command!

And yes the make was done, and one of the files was created with owner root, the other one was created as owner centovacast, but the setuid bit was not set by make.
Otherwise I did not have to do it manually.

Check the "NOTE:" under step 1 which explicitly states that it may fail on some configurations and explains how to proceed if that happens.

LoL... Ofcourse I read that. Otherwise I could not have done the procedure. Fixperms.sh did not work for me so I had to do it manually like the "NOTE" said.

So I followed every step, like I said, copy'd and pasted every line to be sure to make no mistakes, and the setuid bit was not set by make.