Troubleshooting ices problems

Question:

I'm using ices/ices-cc and my source process occasionally exits unexpectedly. How can I diagnose this?

Answer:

The first step in diagnosing the problem is reviewing the ices log for problems. To review the log, you can either log in via SSH and review the log file directly:

/home/centovacast/vhosts/USERNAME/var/log/ices.log

Or, log in to the affected account in Centova Cast, click View logs, and then click AutoDJ log.

Search for the approximate date and time at which ices exited. In most cases, you will find detailed information in the log explaining why the exit occurred.

Nothing was recorded to the log

Most commonly that means you are not looking at the right section of the log. If you are 100% certain that nothing was recorded to the log at the time of the incident, then this indicates that ices crashed.

A crash in ices virtually always means that you're trying to play an MP3 which is badly corrupted. ices is quite good at recovering from bad MP3s, however in rare cases the occasional bad MP3 may cause it to crash.

The easiest way to diagnose this (and other ices problems) is to run ices manually, however this needs to be done in a particular way to avoid damaging your filesystem permissions. DO NOT ATTEMPT TO RUN ICES AS ROOT. Instead, please follow these steps:

Login to the affected Centova Cast account.
If you are having problems with ices crashing: Start the stream if it is not already started, then click Deactivate autoDJ in the lefthand column to stop ices.

If you are having problems getting ices to run at all: Edit the stream settings, click the AutoDJ tab, and set the autoDJ status to "Disabled". Then start the stream.
Login to your server as root via SSH and run the following commands:

su - centovacast -s /bin/bash cd /home/centovacast/vhosts/USERNAME/ /usr/local/ices/bin/ices -vvv -c etc/source.conf

Replace USERNAME with the username of the appropriate Centova Cast account, of course.
Leave the stream running (and your SSH terminal window open) until ices terminates. ices will continue displaying its progress (including the songs it is attempting to play) on your terminal.

If you are experiencing crashes, this will allow you to determine which MP3 was being played at the time of the crash, and possibly determine what other factors may have been involved in the crash.

When finished, simply type exit to return to the root account.

Common log and console output from ices

Certain entries in your source log or console output have specific meanings that may not be obvious at first glance. Please check your output for each of the below.

DEBUG: Script playlist handler serving: ============================= [Error: could not execute query]

If you see a number of lines containing equals signs (=) and/or notices to the effect of could not execute query, this indicates that your MySQL server is reporting an error when Centova Cast attempts to retrieve the next track to play. This is a MySQL problem and has nothing to do with Centova Cast nor ices.

Most commonly this either means your MySQL server is down, or that one of your tables is damaged. If you need to see the full error message, you can view /home/centovacast/vhosts/USERNAME/etc/source.conf and look for a line that looks like:
```
<Module>/home/centovacast/system/runascc/runascc exec ccmanage icesnext username 0123456789abcdef</Module>
```
Copy the command inside the <Module> tags and run it as a shell command, i.e. for the above example, you might run:
```
/home/centovacast/system/runascc/runascc exec ccmanage icesnext username 0123456789abcdef
```
This should output the complete MySQL error message in detail.
double free or corruption

If you receive an error similar to the following:
```
*** glibc detected *** /usr/local/ices/bin/ices: double free or corruption
```
...this typically means one of a few things:
1. Your streaming server has reached its source limit. If you're using SHOUTcast DNAS, this likely means another source is already connected. If you're using IceCast, it likely means that you've reached your "max sources" limit.
2. Centova Cast's dynamic playlist system is not working, which would indicate a damaged Centova Cast installation and/or a problem with your PHP CLI installation.
  
  You can test this by checking /home/centovacast/vhosts/USERNAME/etc/source.conf and looking for a line that looks like:
```
<Module>/home/centovacast/system/runascc/runascc exec ccmanage icesnext username 0123456789abcdef</Module>
```
  Copy the command inside the <Module> tags and run it as a shell command, i.e. for the above example, you might run:
```
/home/centovacast/system/runascc/runascc exec ccmanage icesnext username 0123456789abcdef
```
  This should output precisely two lines. The first line should be the path to an MP3 file from your media library, and the second line should be the artist and track name for that MP3 file.
  
  If any other lines or errors are displayed, you'll need to determine why and resolve the problem. You may need to refer to the Disaster Recovery article.
3. Something else is preventing a successful connection to the server. Check that you're using the correct IP address, port, and so-on.

NOTE: This article is for Centova Cast v2 only; an alternate version exists for Centova Cast v3.