Centova Technologies Forum
Centova Cast v3 => General discussion => Topic started by: asapcpu on January 02, 2014, 07:40:04 pm
-
I have this as a Ticket already, but it is taking a while for them to get back to me and I am rather put out by this problem so I am hoping the Forums will help me some for the time being.
Every day just after 6 and it is every 12 Hours and a few minutes, so AM and PM, a Bundle of stream will Restart. I get a Mass Email from the streams that I have. (by default I make them with my email)
and a Bundle of them just randomly restart. This is the error I get from them.
----------------------------------------------------------------------
2014-01-02 18:40:46 WARN [CONFIG] Invalid item on line 22 of ../stream_074/etc/server.conf -> `yp1debug'
2014-01-02 18:40:46 WARN [CONFIG] Deprecated statement found on line 88 of ../stream_074/etc/server.conf -> change autodumpsourcetime_1=30 to autodumptime_1=30
2014-01-02 18:40:46 WARN [CONFIG] Invalid item on line 102 of ../stream_074/etc/server.conf -> `yp2'
2014-01-02 18:40:46 WARN [CONFIG] Invalid item on line 164 of ../stream_074/etc/server.conf -> `specialfiletmpdir'
2014-01-02 18:40:46 INFO
2014-01-02 18:40:46 INFO ***************************************************************
2014-01-02 18:40:46 INFO ** SHOUTcast Distributed Network Audio Server (DNAS)
2014-01-02 18:40:46 INFO ** Copyright (C) 1999-2013 Nullsoft, Inc. All Rights Reserved.
2014-01-02 18:40:46 INFO ***************************************************************
2014-01-02 18:40:46 INFO [MAIN] SHOUTcast DNAS/posix(linux x64) v2.2.1.109 (Nov 29 2013)
2014-01-02 18:40:46 INFO [MAIN] PID: 4683
2014-01-02 18:40:46 INFO [MAIN] Loaded config from ../stream_074/etc/server.conf
2014-01-02 18:40:46 INFO [MAIN] Calculated CPU count is 4 -> using all available CPUs
2014-01-02 18:40:46 INFO [MAIN] Limited to 1024 file descriptors [relates to ulimit -n]
2014-01-02 18:40:46 INFO [MAIN] Starting 4 network threads
2014-01-02 18:40:46 INFO [MICROSERVER] Listening for source and client connections on port 8432
2014-01-02 18:40:46 INFO [MICROSERVER] Listening for legacy source connections on port 8433
2014-01-02 18:40:46 INFO [MICROSERVER] Flash policy file server not enabled
----------------------------------------------------------------------------------------
If the Stream is being used or not this happenes, and it is every 12 hours and a few minutes apart.
Any Kind words of sympathy is appreciated.
Rodger from Centova Cast told me in the ticket:
After reviewing your server log we found that your current version of Shoutcast is not fully compatible with Centova Cast, specifically you are using Shoutcast 2.2.1 which is NOT officially supported at this time.
According to your logs from the latest couple of stations that were restarted by Centova Cast, ShoutCast is not recognizing some of the lines in your configuration file (below). This could be due to the fact that you're using ShoutCast 2.2.1 which is not officially supported at this time, however I'll have to escalate your issue to our developers for confirmation
But that was almost a Week ago now, and I still have the Problem.
-
2.2.1 was just released not to long ago and yes, centova does not support it. personally I think that should be #1 priority since its the most stable version.
I use 2.2.1 and my stream is fine. It did however crash after a week a little while ago and I sent my issue to the ticket system which has been escalated to their dev dept.
if you dont want to deal with these issues I suggest you revert back to 1.9.8
-
Thank you for replying Groovemind
I have been using 2.2.1 since it's release. and this has only started say mid December.
My ticket is sitting in the dev dept as well. Still waiting to hear back from them.
and I don't know how to revert to 1.9 without recreating the accounts, which I have quite a few of.
suggestions are helpful
-
I could be wrong but all you have to do is put the shoutcast 1.9 files in your shoutcast directory in your server. Basically the reverse of what you did when upgrading to 2.2.1
-
Every day that I check the SSH I have 2 Cron jobs running.
And I noticed yesterday that when I stopped 1 of them that the streams did NOT restart.
[sudo] password for asapcpu1:
root 2937 0.0 0.0 179936 1068 ? Ss 04:18 0:00 /usr/sbin/cron
asapcpu1 5149 0.0 0.0 128964 844 pts/0 S+ 17:03 0:00 grep cron
Which one should I be stopping and how do I keep it from starting back up again?
-
I just ran a /usr/local/centovacast/bin/ccmanage cronjob all
and this is what followed.
[ds5950]$ sudo /usr/local/centovacast/bin/ccmanage cronjob all
[INF] Cron job startup at 2014-01-07 19:59:16
[INF] Starting job: updatenews
[INF] Checking for news...
[INF] No news updates
[INF] Completed job: updatenews
[INF] Starting job: checkprocesses
[INF] Checking for crashed applications...
[INF] Server demo is down and should be up; restarting
Warning: chdir(): Permission denied (errno 13) in /usr/local/centovacast/system/ class_Mailer.php on line 520
[INF] Server stream_001 is down and should be up; restarting
[INF] Server stream_008 is down and should be up; restarting
[INF] Server stream_009 is down and should be up; restarting
[INF] Server stream_010 is down and should be up; restarting
[INF] Server stream_011 is down and should be up; restarting
[INF] Server stream_031 is down and should be up; restarting
[INF] Server stream_033 is down and should be up; restarting
[INF] Server stream_039 is down and should be up; restarting
[INF] Server stream_044 is down and should be up; restarting
[INF] Server stream_064 is down and should be up; restarting
[INF] Server stream_074 is down and should be up; restarting
[INF] Server stream_075 is down and should be up; restarting
[INF] Server stream_089 is down and should be up; restarting
[INF] Server stream_098 is down and should be up; restarting
[INF] Server stream_100 is down and should be up; restarting
[INF] Server stream_101 is down and should be up; restarting
[INF] Server stream_123 is down and should be up; restarting
[INF] Server stream_134 is down and should be up; restarting
[INF] Server stream_140 is down and should be up; restarting
[INF] Server stream_150 is down and should be up; restarting
[INF] Server stream_190 is down and should be up; restarting
[INF] Server stream_192 is down and should be up; restarting
[INF] Server stream_193 is down and should be up; restarting
[INF] Server stream_195 is down and should be up; restarting
[INF] Server stream_196 is down and should be up; restarting
[INF] Server stream_403 is down and should be up; restarting
[INF] Server stream_412 is down and should be up; restarting
[INF] Server stream_502 is down and should be up; restarting
[INF] Server stream_505 is down and should be up; restarting
[INF] Server stream_510 is down and should be up; restarting
[INF] Server stream_513 is down and should be up; restarting
[INF] Server stream_515 is down and should be up; restarting
[INF] Server stream_521 is down and should be up; restarting
[INF] Server stream_528 is down and should be up; restarting
[INF] Server stream_529 is down and should be up; restarting
[INF] Server stream_530 is down and should be up; restarting
[INF] Application process sctrans2 for unleashed_003 is down and should be up; restarting
[INF] Loaded playlist: Standard Rotation
[INF] 28 track(s)
[INF] General rotation weighted at 2
[INF] Pool of 1 playlists
[INF] Pool of 28 tracks
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked Spring Break.mp3 (29) from Standard Rotation; 27 tracks remain; cursor=1/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked angelinmylife-USB Audio Device-USB Audio Device.mp3 (30) from Standard Rotation; 26 tracks remain; cursor=2/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked beautifulgirltake1.mp3 (31) from Standard Rotation; 25 tracks remain; cursor=3/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked black thunderbird.mp3 (32) from Standard Rotation; 24 tracks remain; cursor=4/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked blueeyes.mp3 (33) from Standard Rotation; 23 tracks remain; cursor=5/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked dontletgo.mp3 (34) from Standard Rotation; 22 tracks remain; cursor=6/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked doudreamoflove.mp3 (35) from Standard Rotation; 21 tracks remain; cursor=7/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked dreaming of the ocean.mp3 (36) from Standard Rotation; 20 tracks remain; cursor=8/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked fabiannia-USB Audio Device.mp3 (37) from Standard Rotation; 19 tracks remain; cursor=9/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked freedom.mp3 (38) from Standard Rotation; 18 tracks remain; cursor=10/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked hollywooddreams.mp3 (39) from Standard Rotation; 17 tracks remain; cursor=11/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked hollywoodversionoflove-USB Audio Device.mp3 (40) from Standard Rotation; 16 tracks remain; cursor=12/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked insideyourmind.mp3 (41) from Standard Rotation; 15 tracks remain; cursor=13/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked jujufunk-USB Audio Device.mp3 (42) from Standard Rotation; 14 tracks remain; cursor=14/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked livin'thedream-USB Audio Device.mp3 (43) from Standard Rotation; 13 tracks remain; cursor=15/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked metaverse-USB Audio Device.mp3 (44) from Standard Rotation; 12 tracks remain; cursor=16/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked midnightmoon.mp3 (45) from Standard Rotation; 11 tracks remain; cursor=17/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked miracles-USB Audio Device.mp3 (46) from Standard Rotation; 10 tracks remain; cursor=18/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked on your side.mp3 (47) from Standard Rotation; 9 tracks remain; cursor=19/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked robotpoop-USB Audio Device.mp3 (48) from Standard Rotation; 8 tracks remain; cursor=20/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked sandsoftime.mp3 (49) from Standard Rotation; 7 tracks remain; cursor=21/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked saturdaynight-USB Audio Device.mp3 (50) from Standard Rotation; 6 tracks remain; cursor=22/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked sending my love.mp3 (51) from Standard Rotation; 5 tracks remain; cursor=23/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked soulshine-USB Audio Device.mp3 (52) from Standard Rotation; 4 tracks remain; cursor=24/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked summermemories-USB Audio Device.mp3 (53) from Standard Rotation; 3 tracks remain; cursor=25/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked tendays.mp3 (54) from Standard Rotation; 2 tracks remain; cursor=26/28; unfinishedgeneral=1
[INF] weightedplaylist: 1; playlistindex: 0
[INF] Picked whatdoesyourheartdesire.mp3 (55) from Standard Rotation; 1 tracks remain; cursor=27/28; unfinishedgeneral=1
[INF] weightedplaylist: 0; playlistindex: 0
[INF] Picked whenimdancinwithyou.mp3 (56) from Standard Rotation; 0 tracks remain; cursor=28/28; unfinishedgeneral=1
[INF] Playlist finished (unfinishedgeneral is now 1)
[INF] Playlist exhausted, resetting
[INF] Completed job: checkprocesses
[INF] Starting job: hourlymaint
[INF] Performing hourly maintenance...
[INF] Completed job: hourlymaint
[INF] Starting job: pollhosts
[INF] Completed job: pollhosts
Result: OK Complete
-
Centova tells me that it is an issue with the server, yet my server provider won't do anything because it has 3rd party software on it. GRrrrrrrrr.
I am finding a TON of files titled logrotate_local.rotate.2013-05-10.18454.lastresult.
the Dates are different per Log.
this is the content of most of them
/bin/ls: cannot access /dh/lighttpd/servers: No such file or directory
httpd-argon : begin rotate
httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
asapcpu.com : *** need to rotate logs
rotating 1 domains
httpd-argon : reloading apache (1368256140)
Reloading httpd-argon...USR1.
httpd-argon : checking pid (need 5 passes)httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
1, httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
2, httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
3, httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
4, httpd-argon : checking pid file (server_version apache2)...
httpd-argon : pid check passed
5,
httpd-argon : passed 5 pid checks
I am feeling like I am getting the run around on both sides.Any help is appreciated :o :o :o
-
I was curious i you ever resolved this issue?
I am running the latest release and all worked perfectly until I asked Centova to do some work for me (paid) to set a hostname. From that day on all the streams were randomly and regularly restarting, sometimes all off them, sometimes just a few with the same errors you have said. I contacted Centova who, like you said it was a server problem. I found this odd as the server had run for 6 months trouble free until their intevention. Anyway I ignored it as I was migrating to a new server soon.
Fast forward... bought new server and paid Centova to install and migrate for me and told them about this problem but even on the new server it's still happening so it can't be a server problem!
I did contact the server host who couldn't help with Centova obviously but they ran some routine tests on the server and said its performing perfectly with no issues. The new server also has Plesk on for emails (this is not causing the problem as the old server didnt have this) etc and Plesk monitors the actual server and all the logs within plesk are clean and fine... I even paid plesk for support as they offer phone support which centova doesn't and at the end of it they told me that they can look into centova but everythingg on the system is fine and this is 100% isolated problem within the offending application.
Just emailed support again about it but their last response was "don't blame us, blame the server" so I thought I would see if anyone here has any ideas!
Thanks!
-
It seems to be a problem of the DNAS 2.2.1.109 of shoutcast2. Is not from Centova. I've wrote to DrO (the shoutcast2 developer) and told him about this issue. It happen exactly when shoutcast2 rotate the logs (every 12 hours). The server crash and so Centova Cast in max 5 minutes restart it.
The problem is: DrO at the moment seems to not have time to work on it. But this is a serius problem to solve.
The only solution for now, is to switch back to the previus version of Shoutcast2 (2.0.0.29). There were other problems in the old version... but it doesn't crash every 12 hours at log rotate.
-
my server doesnt restart every 12 hours and I use DNAS 2.2.1.109 of shoutcast2
-
The problem is not for all installations. It seems depend from kernel or linux version.
-
Here's the solution: install 32 bit version of Shoutcast 2.2, it fixes the restart problem :)
-
It seems to be a problem of the DNAS 2.2.1.109 of shoutcast2. Is not from Centova.
Just so everyone knows, we're not ignoring this issue. Our silence in this thread is just because Roberto is exactly right (above) -- it's a bug in DNAS2 that affects ALL DNAS2 users, not just those running under Centova Cast. Accordingly we have no recourse and can't do anything about it -- we, like all of you, are at Radionomy's mercy here, waiting for an update to DNAS2.
Here's the solution: install 32 bit version of Shoutcast 2.2, it fixes the restart problem :)
Interesting. We'll investigate that and see if it solves the issue for us as well. If so, we'll post a sticky with instructions for running the 32bit version on a 64bit platform. Thanks for the tip!
-
Yes Steve. It works.
I try to install the 32 bit version on all my servers and the issue seems solved.
-
instructions for switching to the 32-bit version:
http://forums.centova.com/index.php?topic=3192.0
Thank you heisenberg for the hint
-
we've released a new release (2.2.2) which may resolve this issue -> http://forums.winamp.com/showthread.php?t=373139 (though as none of the people i reached out to test this issue responded, it's unknown if we have actually fixed it or not).
-
Hi DrO,
Last night I replace the new sc_serv on all my centova cast systems (working on Debian squizee).
After upgrade and restarts of all the servers (about 200 istances on several different machines) I still have continuos crash of the server and so restarts from centova cast panel.
So it seems the problem still remain. If this continue for all the day, next night, I'll try to switch to the 32 bit version of the new one. And I will see.
There is something I can do for you to help to understand what happen?
Thanks.
-
i ideally need access to a machine which is experiencing the issue so i can debug / try out changes to try to track down the issue (is easier and quicker if am able to have remote access to some sort of test account on a setup showing the issue i.e. so can ssh into it or along those lines).
though i don't know how easy that is to do without possibly having to do it on a working setup which leads on to the next question... does it also happen for you if using the the log rotate actions via the server admin page in the DNAS itself ?
-
Hi Dro,
I answer you in the Winamp forum.
-
for those not following things on the SHOUTcast forum, please see http://forums.winamp.com/showpost.php?p=3003610&postcount=29 on what you can do to resolve the log rotate crash issues as best as possible before a new fully fixed v2 DNAS is released. that should allow you to at least update to the current DNAS build for the high client connection stability fixes.
so this now explains why it's been such an intermittent issue to track down and why some builds were more susceptible to the issue than others (and why the 32-bit build of 2.2.2 now showed the issue unlike the 32-bit 2.2.1 build due to code clean-up made for the 2.2.2 release).
so there's a temporary solution (clearing out 0-byte log files) and a permanent fix for the issue to follow (newer DNAS build).
-
I see there is a workaround in the new release of Centovacast out in these days.
One question for Centova Staff:
With NO_ROTATE_DNAS2 enabled, it means that that the log file continue to grow endlessy (and what happen if it become too big?). Or there is another behavior to control the logfiles?
This is important to understand. Waiting DrO fix.
-
With NO_ROTATE_DNAS2 enabled, it means that that the log file continue to grow endlessy (and what happen if it become too big?). Or there is another behavior to control the logfiles?
DNAS2 supports rotating its own log files when they get to a certain size. The NO_ROTATE_DNAS2 option tells Centova Cast not to also try rotating them, which is basically just a duplication of effort.
It'd be more ideal if DNAS2 could be told not to rotate its own logs at all and just let CC do it, since it's most efficient for us to process the log file in a single pass and then rotate it out, rather than processing a bunch of separate files generate by DNAS2... but it probably doesn't amount to much difference in terms of real-world performance.
-
DNAS2 supports rotating its own log files when they get to a certain size. The NO_ROTATE_DNAS2 option tells Centova Cast not to also try rotating them, which is basically just a duplication of effort.
it's actually based on how long since the last rotate (which is currently set at 24hrs). the size doesn't have any effect on things (though i can see cases where it would be helpful if it could be roughly controlled by filesize).
It'd be more ideal if DNAS2 could be told not to rotate its own logs at all and just let CC do it, since it's most efficient for us to process the log file in a single pass and then rotate it out, rather than processing a bunch of separate files generate by DNAS2... but it probably doesn't amount to much difference in terms of real-world performance.
noted
-
Ha! I keep forgetting you're on our forums, DrO... Despite how it may look, I swear my comments aren't meant as passive aggressive criticisms of DNAS2. :) We should probably just make NO_ROTATE_DNAS2 the default setting in Centova Cast... The only reason it isn't presently the default is because our DNAS2 module is derived from our old DNAS1 module, which required us to rotate the logs ourselves.
-
i know, just clarifying things :) though it's definitely valid for the DNAS to provide a bit more control over how the log rotates are handled anyway. so is valid to add probably 2 more config options to allow for the default behaviour to be overridden.
-
there'll be a 'rotateinterval' option (default is 86400 seconds i.e. 24 hours) and setting it to 0 will disable all log rotations. though like you note, it might be simpler to just leave the DNAS to do it but the option for you or the DNAS to maintain things will be there as of the new build.
either way, this stability issue should at last be resolved which is good for everyone involved.
-
we've just released v2.4.0 which has the log file handling fixes in. so that should be end of the crashes on log rotation (along with the work Centova have been doing from their side).
-
How to update? Or better to wait till Centova will build update?
Thx in advance
-
Thanks for the heads-up DrO. I'll contact Steve about this ASAP
-
TESTING... on Debian squeeze and centos...