Streams Restart every 12 hours and 3 minutes apart

Read 32410 times
I have this as a Ticket already, but it is taking a while for them to get back to me and I am rather put out by this problem so I am hoping the Forums will help me some for the time being.

Every day just after 6 and it is every 12 Hours and a few minutes, so AM and PM,  a Bundle of stream will Restart. I get a Mass Email from the streams that I have. (by default I make them with my email)
and a Bundle of them just randomly restart. This is the error I get from them.

----------------------------------------------------------------------
 2014-01-02 18:40:46   WARN   [CONFIG] Invalid item on line 22 of ../stream_074/etc/server.conf -> `yp1debug'
2014-01-02 18:40:46   WARN   [CONFIG] Deprecated statement found on line 88 of ../stream_074/etc/server.conf -> change autodumpsourcetime_1=30 to autodumptime_1=30
2014-01-02 18:40:46   WARN   [CONFIG] Invalid item on line 102 of ../stream_074/etc/server.conf -> `yp2'
2014-01-02 18:40:46   WARN   [CONFIG] Invalid item on line 164 of ../stream_074/etc/server.conf -> `specialfiletmpdir'
2014-01-02 18:40:46   INFO
2014-01-02 18:40:46   INFO   ***************************************************************
2014-01-02 18:40:46   INFO   ** SHOUTcast Distributed Network Audio Server (DNAS)
2014-01-02 18:40:46   INFO   ** Copyright (C) 1999-2013 Nullsoft, Inc.  All Rights Reserved.
2014-01-02 18:40:46   INFO   ***************************************************************
2014-01-02 18:40:46   INFO   [MAIN] SHOUTcast DNAS/posix(linux x64) v2.2.1.109 (Nov 29 2013)
2014-01-02 18:40:46   INFO   [MAIN] PID: 4683
2014-01-02 18:40:46   INFO   [MAIN] Loaded config from ../stream_074/etc/server.conf
2014-01-02 18:40:46   INFO   [MAIN] Calculated CPU count is 4 -> using all available CPUs
2014-01-02 18:40:46   INFO   [MAIN] Limited to 1024 file descriptors [relates to ulimit -n]
2014-01-02 18:40:46   INFO   [MAIN] Starting 4 network threads
2014-01-02 18:40:46   INFO   [MICROSERVER] Listening for source and client connections on port 8432
2014-01-02 18:40:46   INFO   [MICROSERVER] Listening for legacy source connections on port 8433
2014-01-02 18:40:46   INFO   [MICROSERVER] Flash policy file server not enabled
----------------------------------------------------------------------------------------

If the Stream is being used or not this happenes, and it is every 12 hours and a few minutes apart.
Any Kind words of sympathy is appreciated.

Rodger from Centova Cast told me in the ticket:
After reviewing your server log we found that your current version of Shoutcast is not fully compatible with Centova Cast, specifically you are using Shoutcast 2.2.1 which is NOT officially supported at this time.

According to your logs from the latest couple of stations that were restarted by Centova Cast, ShoutCast is not recognizing some of the lines in your configuration file (below). This could be due to the fact that you're using ShoutCast 2.2.1 which is not officially supported at this time, however I'll have to escalate your issue to our developers for confirmation

But that was almost a Week ago now, and I still have the Problem.
Thank you for replying Groovemind
I have been using 2.2.1 since it's release. and this has only started say mid December.

My ticket is sitting in the dev dept as well. Still waiting to hear back from them.

and I don't know how to revert to 1.9 without recreating the accounts, which I have quite a few of.
suggestions are helpful
Every day that I check the SSH I have 2 Cron jobs running.
And I noticed yesterday that when I stopped 1 of them that the streams did NOT restart.

[sudo] password for asapcpu1:
root 2937 0.0 0.0 179936 1068 ? Ss 04:18 0:00 /usr/sbin/cron
asapcpu1 5149 0.0 0.0 128964 844 pts/0 S+ 17:03 0:00 grep cron

Which one should I be stopping and how do I keep it from starting back up again?
I just ran a /usr/local/centovacast/bin/ccmanage cronjob all

and this is what followed.

[ds5950]$ sudo /usr/local/centovacast/bin/ccmanage cronjob all
[INF] Cron job startup at 2014-01-07 19:59:16
[INF] Starting job: updatenews
[INF] Checking for news...
[INF] No news updates
[INF] Completed job: updatenews
[INF] Starting job: checkprocesses
[INF] Checking for crashed applications...
[INF] Server demo is down and should be up; restarting

Warning: chdir(): Permission denied (errno 13) in /usr/local/centovacast/system/ class_Mailer.php on line 520
[INF] Server stream_001 is down and should be up; restarting
[INF] Server stream_008 is down and should be up; restarting
[INF] Server stream_009 is down and should be up; restarting
[INF] Server stream_010 is down and should be up; restarting
[INF] Server stream_011 is down and should be up; restarting
[INF] Server stream_031 is down and should be up; restarting
[INF] Server stream_033 is down and should be up; restarting
[INF] Server stream_039 is down and should be up; restarting
[INF] Server stream_044 is down and should be up; restarting
[INF] Server stream_064 is down and should be up; restarting
[INF] Server stream_074 is down and should be up; restarting
[INF] Server stream_075 is down and should be up; restarting
[INF] Server stream_089 is down and should be up; restarting
[INF] Server stream_098 is down and should be up; restarting
[INF] Server stream_100 is down and should be up; restarting
[INF] Server stream_101 is down and should be up; restarting
[INF] Server stream_123 is down and should be up; restarting
[INF] Server stream_134 is down and should be up; restarting
[INF] Server stream_140 is down and should be up; restarting
[INF] Server stream_150 is down and should be up; restarting
[INF] Server stream_190 is down and should be up; restarting
[INF] Server stream_192 is down and should be up; restarting
[INF] Server stream_193 is down and should be up; restarting
[INF] Server stream_195 is down and should be up; restarting
[INF] Server stream_196 is down and should be up; restarting
[INF] Server stream_403 is down and should be up; restarting
[INF] Server stream_412 is down and should be up; restarting
[INF] Server stream_502 is down and should be up; restarting
[INF] Server stream_505 is down and should be up; restarting
[INF] Server stream_510 is down and should be up; restarting
[INF] Server stream_513 is down and should be up; restarting
[INF] Server stream_515 is down and should be up; restarting
[INF] Server stream_521 is down and should be up; restarting
[INF] Server stream_528 is down and should be up; restarting
[INF] Server stream_529 is down and should be up; restarting
[INF] Server stream_530 is down and should be up; restarting
[INF] Application process sctrans2 for unleashed_003 is down and should be up; restarting
[INF] Loaded playlist: Standard Rotation
[INF] 28 track(s)
[INF] General rotation weighted at 2
[INF] Pool of 1 playlists
[INF] Pool of 28 tracks
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked Spring Break.mp3 (29) from Standard Rotation; 27 tracks remain; cursor=1/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked angelinmylife-USB Audio Device-USB Audio Device.mp3 (30) from Standard Rotation; 26 tracks remain; cursor=2/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked beautifulgirltake1.mp3 (31) from Standard Rotation; 25 tracks remain; cursor=3/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked black thunderbird.mp3 (32) from Standard Rotation; 24 tracks remain; cursor=4/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked blueeyes.mp3 (33) from Standard Rotation; 23 tracks remain; cursor=5/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked dontletgo.mp3 (34) from Standard Rotation; 22 tracks remain; cursor=6/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked doudreamoflove.mp3 (35) from Standard Rotation; 21 tracks remain; cursor=7/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked dreaming of the ocean.mp3 (36) from Standard Rotation; 20 tracks remain; cursor=8/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked fabiannia-USB Audio Device.mp3 (37) from Standard Rotation; 19 tracks remain; cursor=9/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked freedom.mp3 (38) from Standard Rotation; 18 tracks remain; cursor=10/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked hollywooddreams.mp3 (39) from Standard Rotation; 17 tracks remain; cursor=11/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked hollywoodversionoflove-USB Audio Device.mp3 (40) from Standard Rotation; 16 tracks remain; cursor=12/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked insideyourmind.mp3 (41) from Standard Rotation; 15 tracks remain; cursor=13/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked jujufunk-USB Audio Device.mp3 (42) from Standard Rotation; 14 tracks remain; cursor=14/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked livin'thedream-USB Audio Device.mp3 (43) from Standard Rotation; 13 tracks remain; cursor=15/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked metaverse-USB Audio Device.mp3 (44) from Standard Rotation; 12 tracks remain; cursor=16/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked midnightmoon.mp3 (45) from Standard Rotation; 11 tracks remain; cursor=17/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked miracles-USB Audio Device.mp3 (46) from Standard Rotation; 10 tracks remain; cursor=18/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked on your side.mp3 (47) from Standard Rotation; 9 tracks remain; cursor=19/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked robotpoop-USB Audio Device.mp3 (48) from Standard Rotation; 8 tracks remain; cursor=20/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked sandsoftime.mp3 (49) from Standard Rotation; 7 tracks remain; cursor=21/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked saturdaynight-USB Audio Device.mp3 (50) from Standard Rotation; 6 tracks remain; cursor=22/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked sending my love.mp3 (51) from Standard Rotation; 5 tracks remain; cursor=23/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked soulshine-USB Audio Device.mp3 (52) from Standard Rotation; 4 tracks remain; cursor=24/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked summermemories-USB Audio Device.mp3 (53) from Standard Rotation; 3 tracks remain; cursor=25/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked tendays.mp3 (54) from Standard Rotation; 2 tracks remain; cursor=26/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked whatdoesyourheartdesire.mp3 (55) from Standard Rotation; 1 tracks remain; cursor=27/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked whenimdancinwithyou.mp3 (56) from Standard Rotation; 0 tracks remain; cursor=28/28; unfinishedgeneral=1
[INF] Playlist finished (unfinishedgeneral is now 1)
[INF] Playlist exhausted, resetting
[INF] Completed job: checkprocesses
[INF] Starting job: hourlymaint
[INF] Performing hourly maintenance...
[INF] Completed job: hourlymaint
[INF] Starting job: pollhosts
[INF] Completed job: pollhosts
Result: OK Complete
Centova tells me that it is an issue with the server, yet my server provider won't do anything because it has 3rd party software on it. GRrrrrrrrr.

I am finding a TON of files titled logrotate_local.rotate.2013-05-10.18454.lastresult.

the Dates are different per Log.

this is the content of most of them

/bin/ls: cannot access /dh/lighttpd/servers: No such file or directory
httpd-argon : begin rotate
httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
asapcpu.com : *** need to rotate logs
rotating 1 domains
httpd-argon : reloading apache (1368256140)
Reloading httpd-argon...USR1.
httpd-argon : checking pid (need 5 passes)httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
1, httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
2, httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
3, httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
4, httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
5,
httpd-argon : passed 5 pid checks

I am feeling like I am getting the run around on both sides.Any help is appreciated :o :o :o
I was curious i you ever resolved this issue?

I am running the latest release and all worked perfectly until I asked Centova to do some work for me (paid) to set a hostname.  From that day on all the streams were randomly and regularly restarting, sometimes all off them, sometimes just a few with the same errors you have said.  I contacted Centova who, like you said it was a server problem.  I found this odd as the server had run for 6 months trouble free until their intevention.  Anyway I ignored it as I was migrating to a new server soon.

Fast forward... bought new server and paid Centova to install and migrate for me and told them about this problem but even on the new server it's still happening so it can't be a server problem!

I did contact the server host who couldn't help with Centova obviously but they ran some routine tests on the server and said its performing perfectly with no issues.  The new server also has Plesk on for emails (this is not causing the problem as the old server didnt have this) etc and Plesk monitors the actual server and all the logs within plesk are clean and fine... I even paid plesk for support as they offer phone support which centova doesn't and at the end of it they told me that they can look into centova but everythingg on the system is fine and this is 100% isolated problem within the offending application.

Just emailed support again about it but their last response was "don't blame us, blame the server" so I thought I would see if anyone here has any ideas!

Thanks!
It seems to be a problem of the DNAS 2.2.1.109 of shoutcast2. Is not from Centova. I've wrote to DrO (the shoutcast2 developer) and told him about this issue. It happen exactly when shoutcast2 rotate the logs (every 12 hours). The server crash and so Centova Cast in max 5 minutes restart it.
The problem is: DrO at the moment seems to not have time to work on it. But this is a serius problem to solve.
The only solution for now, is to switch back to the previus version of Shoutcast2 (2.0.0.29). There were other problems in the old version... but it doesn't crash every 12 hours at log rotate.
Newradio Streaming & Radio Tools
http://www.newradio.it
The problem is not for all installations. It seems depend from kernel or linux version.
Newradio Streaming & Radio Tools
http://www.newradio.it
Here's the solution: install 32 bit version of Shoutcast 2.2, it fixes the restart problem :)
http://www.radioboss.fm - stream hosting
http://www.djsoft.net - radio automation software
It seems to be a problem of the DNAS 2.2.1.109 of shoutcast2. Is not from Centova.

Just so everyone knows, we're not ignoring this issue.  Our silence in this thread is just because Roberto is exactly right (above) -- it's a bug in DNAS2 that affects ALL DNAS2 users, not just those running under Centova Cast.  Accordingly we have no recourse and can't do anything about it -- we, like all of you, are at Radionomy's mercy here, waiting for an update to DNAS2.


Here's the solution: install 32 bit version of Shoutcast 2.2, it fixes the restart problem :)

Interesting.  We'll investigate that and see if it solves the issue for us as well.  If so, we'll post a sticky with instructions for running the 32bit version on a 64bit platform.  Thanks for the tip!
Yes Steve. It works.
I try to install the 32 bit version on all my servers and the issue seems solved.
Newradio Streaming & Radio Tools
http://www.newradio.it
instructions for switching to the 32-bit version:
http://forums.centova.com/index.php?topic=3192.0

Thank you heisenberg for the hint
we've released a new release (2.2.2) which may resolve this issue -> http://forums.winamp.com/showthread.php?t=373139 (though as none of the people i reached out to test this issue responded, it's unknown if we have actually fixed it or not).
Hi DrO,
Last night I replace the new sc_serv on all my centova cast systems (working on Debian squizee).
After upgrade and restarts of all the servers (about 200 istances on several different machines) I still have continuos crash of the server and so restarts from centova cast panel.
So it seems the problem still remain. If this continue for all the day, next night, I'll try to switch to the 32 bit version of the new one. And I will see.
There is something I can do for you to help to understand what happen?
Thanks.
Newradio Streaming & Radio Tools
http://www.newradio.it
i ideally need access to a machine which is experiencing the issue so i can debug / try out changes to try to track down the issue (is easier and quicker if am able to have remote access to some sort of test account on a setup showing the issue i.e. so can ssh into it or along those lines).

though i don't know how easy that is to do without possibly having to do it on a working setup which leads on to the next question... does it also happen for you if using the the log rotate actions via the server admin page in the DNAS itself ?