Centova Technologies Forum

Centova Cast v3 => Bugs and issues => Topic started by: Todd73NJ on November 18, 2013, 08:28:09 am

Title: CONSTANT Server Restarted Emails...
Post by: Todd73NJ on November 18, 2013, 08:28:09 am
Anyone else ever have this problem?   Ive been getting them since Day 1 of my install...

I have about 20 personal streams on my account, and then some of clients. They are generating constant server restarted emails from the Centova program.

I get about 100 server restarted emails a day.  Not all at one... 7.. then a few hours 5 more... then some time passed 9 more..

Any ideas on what could be causing this?

My server has been up for 7 days straight now.. so I have no idea what could be causing this!

Any info/fixes would be appreciated!

Thanks

Todd
Title: Re: CONSTANT Server Restarted Emails...
Post by: DJFire_CFR on November 18, 2013, 09:53:56 am
Actually, just recently, we have started getting this too.
Title: Re: CONSTANT Server Restarted Emails...
Post by: Todd73NJ on November 18, 2013, 10:00:50 am
I have rebooted my main server... started and stopped all the streams in question... but it just keeeeeeeepppsssss generating those emails.

Glad Im not alone in this.. makes it seem like its a big that can be fixed!
Title: Re: CONSTANT Server Restarted Emails...
Post by: GroovinMind on November 18, 2013, 10:11:46 am
Steve said this was fixed via 3.0.5

Are you 100% sure you have the most up to date version of centovacast?
Title: Re: CONSTANT Server Restarted Emails...
Post by: Todd73NJ on November 18, 2013, 10:50:35 am
Ahh maybe I havent installed the new version. I was installed 3 weeks ago... so unless something changed since then.

Can you reference this post please? I searched and only found a few things that were kind of off topic.

Is there a post for the install instructions on 3.0.5?
Title: Re: CONSTANT Server Restarted Emails...
Post by: GroovinMind on November 18, 2013, 01:59:10 pm
Not going to try and dig for the thread but here is the command to update centovacast.

/usr/local/centovacast/sbin/update

It shouldn't interrupt your streams and if you are up to date it wont do anything.
Title: Re: CONSTANT Server Restarted Emails...
Post by: DJFire_CFR on November 18, 2013, 07:10:45 pm
I have the most recent update.. as we run the update script once a week..
Title: Re: CONSTANT Server Restarted Emails...
Post by: singfa on November 19, 2013, 07:07:31 am
Unfortunately we have the same problems with some of our clients. And this after we passed on the new version of centova cast 3.x
On Centova Cast 2.x with the same customers and the same media files we had these problems.
It seems that this new system that use the new types of Icecast 2 shoucast 2 but with shoutcast v.1 too (that is a strange thing) have some bugs and even after I asked for help from  centova.com technicians this issue is not resolved.
Title: Re: CONSTANT Server Restarted Emails...
Post by: Todd73NJ on November 20, 2013, 09:02:58 am
Well I needed to do the update anyway... but I guess listening to the "knowledge" of a forum is never a good idea...

Had the new update installed by Centova staff... and the issue still exists.

I was told that the change was not mentioned under the changes list, but that the problem was not a bug but a result of my users kicking the autoDJ or uploading invalid mp3 files that causes ices-cc to crash.

Which is strange.. bec it had not been happening at all up until recently - and now it happens to almost every account.

Title: Re: CONSTANT Server Restarted Emails...
Post by: Todd73NJ on November 20, 2013, 09:21:36 am
Just testing the theory on a bad MP3...

1st issue - I have sctrans2, as well as ices-ss auto DJ accounts generating the restart emails, and stopping and starting.

Looked at a few of the specific accounts - some only have 2 or 3 hours of music in the Auto-DJ... but they may go 8 or 10 hours without generating a restart email - then all of a sudden a few restart emails within the next few hours.  If it were a bad MP3 - wouldnt it stop/start every time it hit that MP3?

Seems to me there has to be more to this...
Title: Re: CONSTANT Server Restarted Emails...
Post by: GroovinMind on November 20, 2013, 11:36:17 am
I dont know about centova with miss-matched encoded mp3s but I know when I had whmsonic it would buffer and act really weird if they were not all encoded the same.


I always encode my mp3s via LAME ( I have a mac so I use - http://markvapps.com/mp3encoder/ )


even if thats not the cause of your prob, I suggest you do encode all your mp3s the same anyway,


Good luck
Title: Re: CONSTANT Server Restarted Emails...
Post by: Todd73NJ on November 20, 2013, 12:01:21 pm
I have a support request in .. they will be checking my server to see what they find.

Ill post the results here. Maybe they willl be helpful to others as hell.
Title: Re: CONSTANT Server Restarted Emails...
Post by: Reinaldo C. Andrade on November 22, 2013, 10:42:01 am
Friends, the problem reported is nothing more than a bug in the migration.
I noticed that in new accounts the problem does not happen.
Do not know how, but migration can not correctly compile the existing files on the server.
Do a test by creating a new account and sent the files via upload, do not use the transfer route server.
Advise delete all "MP3" files of all accounts and redo uploads, preferably at Centova panel, without using external FTP since. in this case the library update could take hours to complete. ;)
Title: Re: CONSTANT Server Restarted Emails...
Post by: cybermix on November 24, 2013, 02:50:55 am
Hi Guys,

I have similar issue, new activated account, MP3 files uploads and are correct encoded in MP3, i get continue and constant Server restarted emails. This have to be a bug. This happening only after i upgrade to the last release few days ago, before didn't happen. Server reboot do not solve this issue and to amny documentation about how to debug this issue are not available.

Thanks and hoping this bug can be fixed, again i'm blocked on Centova2 and can not migrate to Centova3 because it's not 100% stable, from this reason i continue to test Centova3 till i consider stable and free of major bugs.

Title: Re: CONSTANT Server Restarted Emails...
Post by: Todd73NJ on November 25, 2013, 11:30:30 am
Hey Cyber,

I would have to disagree with your theory - bec if you read the thread you will see that I had NOT updated... I was using what they had installed for me, and then did the update as a proposed fix.  So its not the upgrade that caused it.. this has to be an ongoing bug!


But to everyone else.. who is having the problem let me share with you my experiences..

Support logged into my panel, and checked it out... they came back with this:

Hello Todd,

You have two accounts which were restarting on a regular basis (every 5 minutes) during Nov 17 and Nov 18, but they didn't seem to have any problems after that.
The accounts are: james_sherwood and ayslynn

The autoDJ logs from 'james_sherwood' show a huge number of lines like this one: 'Error during send: Mount failed on http://94.23.250.16:8034/stream, error: Couldn't connect'
and after a few tries they end in 'Too many stream errors, giving up Ices Exiting...'
These are entirely backing up our initial diagnose that the autoDJ had been kicked. The logs simply show that it is trying to connect but it fails (because there is another source broadcasting) and after that the autoDJ it's giving up, only to retry and fail again every 5 minutes, while also sending you restart e-mails about this.

The 17-18Nov logs from 'ayslynn' had already been rotated, so I couldn't check for that account but I am fairly confident the problem is exactly the same. It is far more plausible that the autoDJ was failing to connected every 5 minutes than that it was actually crashing every 5 minutes.

Regards,
Centova Technologies Inc.



To that reply I asked again, how I fix the problem....

And received this:

Hello Todd,

Unfortunately this is not something you can fix, because it is been caused by your clients.

You can find a detailed description of the problem at http://www.centova.com/faq/general/troubleshooting/kicking_the_autodj


Please feel free to contact us again if you have further inquiries.

Regards,
Centova Technologies Inc.



Now, from my observations, I did have a few accounts that were restarting way more than others - but all of thew accounts were restarting..


BUT  - after the visited my panel - on 11/20/13 - I have only received 1 (yes, one in 4 days) server restart email. And not from either of the problem accounts.

How is that possible that it is 99% better after they visited the panel but didnt make any changes?

But Im not complaining.

Id recommend putting in a ticket - and see if thats a fix for all of you also.
Title: Re: CONSTANT Server Restarted Emails...
Post by: cybermix on November 28, 2013, 03:17:05 am
Hi,

have no sense what they told you, i use SC_TRANS2 not ICES, and i didn't have this bad experience till last update, i test before and works all well, without get 1 million restarted email notification. The bad part is as they close Centova2, and they release Centova 3.0.5 as a stable version, with is not so stables as we can see. I have no problem with centova2, but migration to centova3 it's more than impossible till is not 100% with bugs solved, major bugs. Also i do not understand why they refuse to continue to support shoutcast1 to work with SC_TRANS2, as we know it's not so sure the situation with AOL and SHOUTcast, if they close YP we will see son many people crying with SHOUTcast v2

Anyway, back to this issue, as i say before, i didn't have this issue till last upgrade to 3.0.5. i run 2 servers with same audio files on test, one stream is AAC+ and works normal, the second one MP3 send me frequent restart notification email.

I hope this will be a point for debug.


Regards,
CyberMIX
Title: Re: CONSTANT Server Restarted Emails...
Post by: Reinaldo C. Andrade on November 29, 2013, 07:26:45 am
Friend, do not use the migration feature of Centova or via server commands to copy the MP3 files.
Make the uploadas centovacast by the panel and perform new tests.
Again, the bug is in the migration.
With new uploads and new accounts the problem does not happen.
Title: Re: CONSTANT Server Restarted Emails...
Post by: DJFire_CFR on December 06, 2013, 08:44:40 am
So we got a couple new servers, just added accounts on them, and nothing but restart emails.
Title: Re: CONSTANT Server Restarted Emails...
Post by: DJFire_CFR on December 09, 2013, 05:38:38 pm
I think we got down to why this is happening...

I found today that one of the resellers was receiving these emails over and over (as we were also getting them).

The account it was doing it on was a Shoutcast v1.x stream.  When they stopped "Auto DJ" and went live, This is when the emails started flowing out.

Does Centovacast think when auto DJ is turned off or is "kicked", it thinks that there is something wrong with the stream itself ??

IF so, there must be something that can be done there to get that fixed up?
Title: Re: CONSTANT Server Restarted Emails...
Post by: kahfluie on December 23, 2013, 05:28:58 am
@DJFire - wondered if you were able to get any resolution. I too have the same problem when someone stops the autoDJ and goes live. Like clockwork every 5 or 10 minutes I get the server has been restarted email.
Title: Re: CONSTANT Server Restarted Emails...
Post by: My Auto DJ on December 24, 2013, 04:32:26 am
I have ran across this before, do any of the mp3s have non-english characters? If so I think that's the reason it keeps restarting, simply changing the character set should correct the problem but If there is a mix of English and non-English characters mp3s (which can be common if your client has a mix of different music, I think the last client I found that had this problem had some Russian mp3s mixed with USA songs so I just deleted the few Russian songs and the problem was gone! 
Title: Re: CONSTANT Server Restarted Emails...
Post by: audiobras on January 08, 2014, 07:44:31 pm
But the issue could be in mp3 name file or mp3 ide tags or both? Because if was the mp3 name centovacast could make something to filter (delete) the anormal caracters after upload or after some media library update.
Title: Re: CONSTANT Server Restarted Emails...
Post by: sam604 on January 12, 2014, 12:28:16 pm
Any new possible solution.. as I also have this issue.. and none of the above corrects it... :(
Title: Re: CONSTANT Server Restarted Emails...
Post by: Centova - Steve B. on January 21, 2014, 12:31:59 am
Just to put an official end to the speculation here: in all but a few rare cases, "server restarted" emails basically mean SHOUTcast DNAS or IceCast crashed.  Centova Cast is not the problem here -- it's just reporting what happened, and telling you that it restarted SHOUTcast/IceCast so that it didn't remain down.

Our clients often mistakenly believe that Centova Cast is the cause of the problem since Centova Cast notified them of the problem, but it's important to avoid shooting the messenger, as the saying goes.

DNAS, DNAS2, and sctrans2 all have known crash bugs at this time, with DNAS2 and sctrans2 being the most commonly affected.  Since DNAS, DNAS2, and sctrans2 are NOT our products and we do NOT have any access to their source code, there is nothing we can do about any bugs in their code -- your options are to either wait for a fix from Radionomy (who recently purchased the SHOUTcast intellectual property rights) or switch to IceCast.

To complicate matters, there are a few other rare situations (usually hardware or kernel issues) that can cause ALL processes to terminate, and this will be reported identically by Centova Cast.  Odds are, 95% of you do NOT fall into this category but I'm mentioning it for the inevitable 5% who will complain that my diagnosis must be wrong simply because you're not one of the 95% I mentioned above. :)

In either case, our helpdesk staff can usually help you identify which category you're in.  Feel free to open a ticket if you're in doubt.
Title: Re: CONSTANT Server Restarted Emails...
Post by: audiobras on January 21, 2014, 10:14:15 am
Like the major problem could be bugs in shoucast, sc_transcv2 or other software not related with centova would be good identify what could be the commom factors that cause that. If users of centova identify some possible causes I believe that help to guide users to not make the same. I don't know if that is possible, because must be more than one thing that explore the possible bugs, but each user with the problem could try to test the panel deleting musics and adding to see what is the file or folder that could start to cause the issue. After identified some causes maybe centovacast could make something to prevent some cases. I suggested in feature requests a way to centova filter "strange" caracters from folders or files to see if that could cause something. If not, the problem could be related with some mp3 corrupted or something...

But steve,  sc_transcv2 could only be used with shoucastv2? is not possible give other option to aac streams?
Title: Re: CONSTANT Server Restarted Emails...
Post by: audiobras on January 21, 2014, 11:10:52 am
I did read other post about shoutcast v.2.2.1 and I make a update in server to see if a problematic account will stopt to restart. I will post here the test result.
Title: Re: CONSTANT Server Restarted Emails...
Post by: Centova - Steve B. on January 21, 2014, 10:29:58 pm
Like the major problem could be bugs in shoucast, sc_transcv2 or other software not related with centova would be good identify what could be the commom factors that cause that.

Excellent suggestion.  The problem is just that it's not always easy to determine exactly what causes a crash in a closed-source program that's made by another company.  In most cases, without the source code, unless you can reproduce the crash consistently every time you perform a specific action, you have no way of knowing what's causing it.

To be clear, we're not trying to say "it's not our software so it's not our problem" -- we hear that a lot from clients who don't understand why we can't help them with DNAS2/sctrans2 crashes, but that's not the case.  We've had clients encounter crashes under IceCast before, for example, and while IceCast isn't our product either, it's open-source so it's not hard for us to run it under a debugger and get an idea of why it's crashing.  And then we can suggest a workaround, or modify Centova Cast to avoid triggering the crash.

With DNAS2, the only crash that we've been able to consistently reproduce (and thus diagnose) is caused when using sctrans2 with MP3s with certain accented characters in their ID3 tags.  In some cases sctrans2 does not properly UTF8-encode the accented characters, and when it sends them to DNAS2, DNAS2 crashes.

Anyone experiencing crashes should first look to see if it's always the same set of MP3s that causes this problem.  And if so, they should try removing the accented characters from the MP3's ID3 tags and seeing if that fixes the problem.

Again, this was easy enough for us to diagnose because *every* time our client used a specific MP3, DNAS2 crashed.  With that information we were able to narrow down the cause.  Other crash situations are not so easy for us to diagnose, and there is at least one other DNAS2 crash scenario that a lot of users are reporting that we haven't been able to track down yet.  (And unfortunately, we may never be able to unless a clear trigger is found.)

But steve,  sc_transcv2 could only be used with shoucastv2? is not possible give other option to aac streams?

Yes, we're working on it!  We should hopefully have a preliminary release with support for an sctrans2 alternative (including AAC support) by the end of the month.
Title: Re: CONSTANT Server Restarted Emails...
Post by: audiobras on January 22, 2014, 04:24:53 am
In one case here, after I changed to the new SHOUTcast Server v2.2.1.109 the the constant restarts stopped. The same mp3 and account that was restarting each 30 minuts now works fine. So I believe that the new Shoutcast v2 have fixed some bugs. I don't know if that is a solution for all cases, but is a very good news to me.

And goog to hear too that centova will add more options to aac stream early.
Title: Re: CONSTANT Server Restarted Emails...
Post by: Reinaldo C. Andrade on January 31, 2014, 02:27:35 pm
Unfortunately, I have to agree with you Steve, but it was hard.
I realized that the problem only happens with the files with  specials chars if it is renamed after shipping (within the panel), if sent via FTP third party programs, or if they are migrated from one account to another, regardless of the process used.
If you upload these the same files of MP3  via panel, the problem does not occur.
What I do now is to guide my clients to uploads via  panel and not rename uploaded files. With that, I have no more problems with rebooting the server and the flood of emails with this notice in my inbox.