Problems with Auto DJ not Deactivating when you click Deactivate Source from the centova panel

Read 23789 times
I have a Centova cast reseller account and My Host and I have been trying to figure out what is going on with the Auto DJ *NOT* Deactivating when you click the Deactivate source button on the centova panel.

My host has purchased the upgrade to the latest centova version, I have made sure that NONE of my customers try to kick the auto off using the shoucast DDNS page (as I read it here that will cause server issues) I have used the Auto DJ API mentioned here and even that some times wont work right but the biggest issue I have is the centova panel not being able to deactivate source, it says it does but the auto DJ just continues to play like it never even tried to turn off...

It seems that when I had only a few customers every thing worked fine, but when I started getting more I currently have 34 customers and out of that I'd say 1/2 of them had had the issues all at random times.

I tried to contact Centova cast directly just to tell them what I was experiencing from the reseller side but they refused to listen to me as my host is who own the centova license not me. I just wanted to let them know something wasn't working right and I didn't care if they contacted my host to trouble shoot and fix it. I didn't want any info from them (unless they knew of a fix or course for this issue so I can could tell my host) but any way back to the issue at hand...

The only want to Temporarily fix the problem I have found you have to stop and start the server all over and it will work fine for a while but the problem always comes back.

My opinion here: I'm thinking that there an IRQ problem that develops and when you click deactivate source it says it did it, but its not actually interrupting the ICES part of the centova auto DJ.

Like I said I have seen the problem get worse as I pick up more Customers, its gotton so bad I have stopped selling Centova Auto DJ's and have purchased a WHMSonic panel with auto dj to move some of the people who have had this issue happen to them over too, I'd prefer centova but if it dont work like it should then I cant keep selling servers with auto on it.....

Has any one else had issues with the Centova Panels Deactivate Source failing to work like it should? Both me and my host would be interested in hearing about it! We have done just about everything we can think of to over come this issue and so far have failed to get it fixed.....maybe someone from centova will read this and come up with a reason for this issue and fix too!

Thanks for your time,

Rick
Ive had this issue as well, I have one clicnt that complains about it 1 per week, centova has told me that the client must be doing it wrong...

Dylan T
HostRail.com
Yea I heard the same thing they were claiming the customer(s) were using the shoutcast DDNS to try to kick the auto with but they were not. I have personally tried to deactivate the source using the Centova Deactivate button only to have it say it did it but the server never stopped the auto DJ.

It seems is you only have a few Auto Dj's then its not an issue but when you grow beyond certain point the problem seems to get worse.

I am hoping that someone at Centova will read this and start looking into it because it is a problem for us resellers and since centova wont listen to what I tried to tell them maybe they will listen here to us!
I would also welcome improvements in this area. Having the ability for DJs to stream live is essential for my station but we've had repeated problems with the source not stopping and stream passwords no longer being recognised from one day to the next...

Thanks!

Hugh
Quote
Yea I heard the same thing they were claiming the customer(s) were using the shoutcast DDNS to try to kick the auto with but they were not.

Actually, I don't think anyone on our end ever claimed that this was the ONLY situation that could cause this.  It's just the most common situation that causes the problem, not the only one.  

In way of a lengthy explanation, when Centova Cast invokes ices, it records its PID and keeps an eye on that PID to know whether or not the autoDJ is active.  When you see that the autoDJ is running but Centova Cast is refusing to terminate it and showing it as "Offline" in the control panel, that means that the ices process invoked by Centova Cast is, for some reason, no longer running, and another one is in its place.

Most commonly that happens when kicking the autoDJ, because when you kick the autoDJ, it gets disconnected from the server.  Then Centova Cast will then detect the source disconnection and try to start another copy of ices in its place.  What Centova Cast doesn't realize, though, is that the kicked ices process is still idling in the background, periodically trying to reconnect (that's a feature of ices).  If that kicked process eventually successfully reconnects, you get the situation you described -- Centova Cast is looking for the PID of the newly-launched ices process, whereas the old process (formerly thought to be dead) is now connected to the server.

There are certain other cases where this can happen, but all of them involve the source being disconnected unexpectedly and Centova Cast launching a new one in its place.  For example, if something goes wonky on your server for a bit and the server load spikes for, say, 30 seconds, ices may not be able to transmit audio to ShoutCast/IceCast during that time.  ShoutCast/IceCast will disconnect idle sources after awhile (usually 30sec, but it's configurable) so that can have an effect identical to kicking the autoDJ.  

That's just one example; there may be other ways it could be disconnected as well, but in any event, this situation is ALWAYS caused by ices getting unexpectedly disconnected from the server.  If you're not kicking it, then something else (possibly the above) is causing it to be disconnected... you just need to figure out what.

v3 adds some extra checks to try to catch situations like this... if CC detects that ices is disconnected, it will try to detect any "half-dead" ices processes and do its best to terminate them before starting another copy which should hopefully put this issue to rest.


Quote
It seems is you only have a few Auto Dj's then its not an issue but when you grow beyond certain point the problem seems to get worse.

See my comment above about load spikes.  More accounts often equals more load.  With re-encoding enabled, more accounts equals a LOT more load.


Quote
centova wont listen to what I tried to tell them maybe they will listen here to us

This isn't a matter of listening or not; we do already know that some users experience this issue.

In regard to the response you specifically received, as we explain time and time again, we don't provide technical support to end-users because end-users do not administer the server.  For all you know, your host could be running some kind of CPU/IO-intensive maintenance cronjob at 3am that grinds all processes to a halt and results in the autoDJ disconnections.  Or maybe your host is running the machine on an old P3 server that just can't handle the autoDJ load.  (Extreme examples, but you get the point.)  That's why we ask that if end-users have problems, they contact their hosting provider.  If the hosting provider believes it's a CC issue, they can contact us for help so that we have direct communication with the folks in charge of the server.

If your host is not willing to contact us, then either 1) they already know why the issue is occurring (i.e., the 3am cron job :) ), or 2) they just don't care, in which case it may be time to find a new host.
My Host has me a totally different server then I was on before and I'm gonna quote what they told me about this server:

"We have you on our strongest server that has 8 cpu's and almost 8 gigs of ram, and their is only like 6 customers on it so their is no way that server is bogged down, the only thing running on it is centovacast, and the only cron jobs active are to update the stats for centovacast"

So I don't buy that itis my Host Especially since Other Hosts are having the same Issue, if it was just me maybe then I would say it was the host but I have looked around and found a few articles where people have having the same issues....

Now as for the Comment that centova will only talk to the host, You thats fine and dandy but when I emailed centova I did it only to give yall the information on what was happening I did not want yall to go fix it or anything like that I was just basically passing a bug report to you so you would be aware of it. I was not asking for tech support and if you had read it and maybe already had this issue and found a fix for it then you could have said refer to this link, or theres a fix for then have your host contact us and we'll get them the fix, but instead you just said and a para phrase here..... You dont own a centova license so were not gonna talk to you at all! Thats bad customer service.... Again if I asked you to fix it then I would understand saying that I needed to have my host get in touch with you, but what I sent was just a bug report asking if you were aware of the issue and if there was something I or my host could do to fix it thats all....

Maybe ALL hosts need to start selling WHMSonic servers instead of Centova at least they wont blame the hosts and tell people to go find aniother host!

My host has bent over backwards trying to help fix something that centova should have fixed to begin with!!!!

Quote
So I don't buy that itis my Host Especially since Other Hosts are having the same Issue

I believe I was fairly clear in stating the nature of the problem and explaining that any number of things could cause it.  I did provide the CPU load issue as an example, but it was just that -- one example.

I did not "blame" anything on your host; the only comment I made about your host is that there's a possibility that something *could* be going on with their server, and the only way to diagnose the issue is for them to contact us.


Quote
you could have said refer to this link, or theres a fix for then have your host contact us and we'll get them the fix, but instead you just said and a para phrase here

Let's not paraphrase, because doing so tends to lead to hyperbole.  As I explained in my last post there is no specific "fix" or link to refer to because there is no one circumstance that cause this problem.

What you were told is that you needed to contact your host for support, and that they could contact us if they believed it was a software problem.  I'm sorry if you feel that asking our customers to contact us for assistance is poor customer service(?), but the bottom line is that (while I do respect that the success of our product ultimately depends the level of demand from, and satisfaction of, the end-user) you are not our customer -- you are our customer's customer.  You do not run our software on your server, nor do you administer the server, so you do not have the means to diagnose/resolve this issue.

To be perfectly clear, as we've explained time and time again: If your host needs help with the software, we're happy to provide it.  But with all due respect to you as an end-user of our product, complaining loudly here on the forum and arguing every point in my posts does not change the fact that we cannot help you without the involvement of your host.

And for the other folks following this thread, please note a key point that I made in my last message -- v3 is presently being tested by several large hosting providers, and it includes some preventative measures that will hopefully curb this issue.  As this is something of a transient problem we can't yet definitively say whether it's gone for good, but it hasn't been reported in v3 *yet*, so... so far so good. :)
An update on this issue -- we recently received access to a server exhibiting this problem and tracked down a bug in an advanced configuration setting (AUTODJ_FAILSAFE) which we implemented in v2.2.4.  The bug can trigger a problem that looks very similar to the "Kicking the autoDJ" and "High CPU load dropping source connections" issue.

If you are/were experiencing this issue with v2.2.3 or earlier, or if you have not enabled AUTODJ_FAILSAFE, then this bug will NOT affect you and your problem is as described in my earlier posts in this thread.

But if you (or your host) are using v2.2.4 and have turned on the AUTODJ_FAILSAFE setting in config.php, this very well may be the issue, and the support department has a patch available.

On a related note, I should mention that this highlights the importance of having your *host* contact us if you have a problem -- in this case, an affected host provided us with root access to an affected server, and that allowed us to diagnose and patch the problem in all of about 30 minutes.  My thanks to the host in question. :)