Disaster recovery

Question:

My Centova Cast installation is damaged and needs to be recovered. How do I go about doing so?

Answer:

The process for recovering your Centova Cast installation depend on the nature of the damage that has been done. More information is included in the sections below.

Procedures described hereunder are dangerous and unsupported

Please note that we cannot provide support for anything explained below; this article is intended to assist advanced users in recovering a damaged installation, but this is outside of the scope of the Centova Cast support agreement. Please also note that the procedures described below are advanced and risky, and may result in data loss if performed incorrectly.

Recovery by a Centova Technologies Technician

If you have any doubts about your ability to perform a recovery, but are sure your data is intact, you may wish to contact Centova Technologies to have an experienced technician perform the recovery for you. In most cases, this requires less than an hour of time and thus will only incur a fee of one hour of professional services (priced at USD$70 at the time of this writing).

Permissions Problems

When encountering permissions problems, many naive administrators tend to try to set file/directory permissions to 0777 to make everything world-writable. This will badly damage Centova Cast and cause further permissions problems which may be difficult to resolve.

If you are encountering permissions problems of any kind, do NOT attempt to make your files or directories world-writable. Instead, run the following commands:

/etc/init.d/centovacast stop
killall -9 sc_serv sc_trans icecast icecast2 ices ices2
/usr/local/centovacast/sbin/fixperms
/etc/init.d/centovacast start

Note that the above procedure will interrupt all of your streams, although they will be brought up again automatically within a few minutes by the Centova Cast process monitoring system.

Reinstallation and recovery

If your Centova Cast installation is damaged beyond repair, the best approach is often to reinstall Centova Cast, then restore your old data. Please note that this is a particularly risky procedure; be sure to backup your data carefully in advance.

To perform a reinstallation and subsequent recovery:

Back up your Centova Cast database and account data.

Note that this is not simply a precautionary procedure; you will need to restore this backup at the end of the recovery process, so do not skip this step.

If your server is otherwise intact, you can keep the backup on the same server on which Centova Cast is currently running. This is by far the fastest option. To do so, run:
```
mkdir -p /usr/local/castbackup/
mysqldump -uUSERNAME -p DATABASENAME > /usr/local/castbackup/centovacast.sql
mv /usr/local/centovacast/var/vhosts /usr/local/castbackup   
```
Replace USERNAME with your MySQL database username and DATABASENAME with your MySQL database name. Enter your MySQL database password when prompted. If successful, your complete backup will be stored in /usr/local/castbackup, and you can continue to step 2.

If you need to decommission or reimage the server that is hosting Centova Cast, you will need to copy your backup to a remote server.

If your Centova Cast installation is still intact enough to do so, you may be able to use the simplebackup script to push a backup to a remote server automatically. To try this, run:
```
/usr/local/centovacast/sbin/simplebackup --hostname=foo.example.com
```
If successful, this will login to foo.example.com as root via SSH, and save a complete Centova Cast backup in /root/centovacastbackup/ on foo.example.com, and you can continue to step 2. Note that you can run /usr/local/centovacast/sbin/simplebackup --help for help with custom backup options.

If the simplebackup utility is unusable, you will need to create a backup in /usr/local/castbackup as described above, then copy /usr/local/castbackup to another computer via other means (rsync, scp, etc.)

Note that if you have lost either your Centova Cast database or your account data directory, then your installation has sustained permanent data loss and recovery will not be possible.
Uninstall Centova Cast.

Note that this is, of course, only necessary if you do not intend to reimage the server or recover your Centova Cast installation to a different server.

If your Centova Cast installation is still intact enough to do so, you may be able to use the uninstall utility remove Centova Cast automatically. To try this, run:
```
/usr/local/centovacast/sbin/uninstall --i-want-to-delete-all-my-data
```
When prompted, enter the word UNINSTALL to confirm that you do indeed want the uninstaller to remove Centova Cast and delete all of your data.

If the uninstall utility is unusable, you will need to remove Centova Cast manually. To do so, run the following commands:
```
/etc/init.d/centovacast stop
killall -9 sc_serv sc_trans icecast icecast2 ices ices2
rm -f /etc/cron.d/centovacast
rm -rf /usr/local/centovacast
rm -f /etc/centovacast.conf /etc/init.d/centovacast
userdel centovacast
groupdel centovacast
userdel ccuser
groupdel ccuser
```
Centova Cast should now be uninstalled and you're ready to install a new, fresh copy.
Install Centova Cast as usual, as described in the installation manual. Ensure that you install it exactly as you did the first time (using the same server/source software and so-on) otherwise your streams will be unable to start after recovery.
Stop Centova Cast:
```
/etc/init.d/centovacast stop
```
Temporarily disable the Centova Cast cron jobs by moving /etc/cron.d/centovacast to /tmp:
```
mv /etc/cron.d/centovacast /tmp
```
Remove all tables from the newly-installed Centova Cast database, and restore the backup you made of your Centova Cast database.
Restore your backup of the /usr/local/centovacast/var/vhosts/ directory.
Fix the file ownerships/permissions on the recovered files:
```
/usr/local/centovacast/sbin/fixperms
```
Restore the Centova Cast cron job:
```
mv /tmp/centovacast /etc/cron.d
```
Start Centova Cast:
```
/etc/init.d/centovacast start
```

At this point, you should have a working Centova Cast installation.

NOTE: This article is for Centova Cast v3 only; an alternate version exists for Centova Cast v2.