Hi Steve,
today I try to upload abot 900 songs on an account media folder with ftp.
After I use the command: /usr/local/centovacast/bin/ccmanage reindex [name account]

Song where re-index corretly but when the script start to retrieving covers from Itunes, after about 400 songs it stops with this error:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 62125 bytes) in /var/local/centovacast/system/albumds/includes/itunes.php on line 35

And the web server of cc3 stop to work.
After a cc3 restart all is working fine.

The server has 24 giga of RAM.

What do you think about it?

try increasing memory_limit in php.ini (now it is set to 128 M)

Hi Steve,
Song where re-index corretly but when the script start to retrieving covers from Itunes, after about 400 songs it stops with this error:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 62125 bytes) in /var/local/centovacast/system/albumds/includes/itunes.php on line 35

If you try running the reindex command again, does the same thing happen again? eg: is it a consistently reproducible problem?

Something is definitely going very wrong if you're getting that error -- the PHP processes don't use much memory under normal operating conditions, and I've just recently successfully imported 5000+ tracks in 30GB+ of music in a single batch with no problems.

I just have another import job running right now with all album data sources except iTunes turned off, so I'll see if that makes any difference, but at the moment this doesn't appear to be something I can reproduce.

And the web server of cc3 stop to work.

This is a very strong indicator that something more serious is wrong with your installation.  The media import process, if it's performed via the commandline, has absolutely nothing to do with the Centova Cast web server.

try increasing memory_limit in php.ini (now it is set to 128 M)

No, please don't -- Centova Cast is not nearly memory-hungry enough to require that much memory per PHP process, so if you're running out of memory, something is going wrong.

Just an update -- my new import job just finished with no errors.  I tried to reproduce the conditions you described -- 948 tracks in my case, with all album data sources except iTunes disabled to ensure that it gave the iTunes module a heavy workout.  So I'd need further information to help reproduce the issue.

If you try running the reindex command again, does the same thing happen again? eg: is it a consistently reproducible problem?

If i run the command again the script skip songs just re-indexed and start from the file where stop in the error... After about 400 or 500 songs get the same error end stop.
The third time run till the end and all songs covers are re-indexed with success.

I try to reindex about  2000 songs on another account with cover retrieving disabled. The script worked fast and end with success with no errors.

And the web server of cc3 stop to work.

It happens only the first time, not others times.

The cc3 installed is very clean. Is a debian squeeze with 24 gb of ram with no control panel (plesk, cpanel etc..)
There is only cc3 and cc2 installed.
Everything is going fine and all the panels are working without any other problems.

Witch other informations cold it be util for you to indagate?

I am having the same issue when trying to import around 50,000 tracks into the database. I have run the reindex command a number of times but it always eventually returns an error similar to:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 10241 bytes) in /usr/local/centovacast/system/class_RPCDaemonClient.php on line 852

I have tried unticking the album art options in the control panel in case it's that which was causing the memory issue but it doesn't seem to have made any difference.

I don't know if it's relevant but I was able to import this same collection into v2 without issue.

Any assistance would be much appreciated.

Many thanks,

Adam

I am having the same issue when trying to import around 50,000 tracks into the database.

It makes sense that you could encounter this with something on the order of 50,000 tracks in a single import job, but I still don't see how it'd be possible with 900 tracks.

In your case, the issue here is that PHP is rather inefficient when it comes to the memory utilization of its data structures, and we do a lot of in-memory caching to make things as fast as possible.  Turns out, it's just not practical to keep that amount of information in memory when importing tens of thousands of tracks in a single batch.

We've made some changes to the code to (internally) process smaller batches of files at a time rather than taking an all-at-once approach, as well as flushing the caches periodically when they start to grow too large.  I've tried a simulated import job of 100,000 files and memory utilization never exceeded 100MB with these changes, so you should see better results in the next build.  (Beyond 100,000 files, which is an atypical use-case for a single import job, you'd have a legitimate reason to increase the memory limit in cc-system.conf.)

Your explanation about the memory makes complete sense, thanks for explaining how it works.

It's great to hear that you can come up a solution to handle large imports in the next release.

In the meantime upping the memory limit to 1GB temporarily was sufficient to import all 50,000 tracks in one go.

Thanks so much for your help with this, it really is much appreciated.

Cheers,

Adam