Centova Technologies Forum
Centova Cast v3 => Bugs and issues => Topic started by: alex46 on November 12, 2012, 12:33:26 pm
-
Hello! :
2 running, 175 sleeping, 0 stopped, 104 zombie
:~# ps -el | grep -e '^.\sZ'
0 Z 1001 6864 2477 0 80 0 - 0 - ? 00:00:00 nextsong <defunct>
1 Z 1001 6865 2477 0 80 0 - 0 - ? 00:00:00 ices <defunct>
0 Z 1001 7647 2477 0 80 0 - 0 - ? 00:00:00 nextsong <defunct>
1 Z 1001 7648 2477 0 80 0 - 0 - ? 00:00:00 ices <defunct>
0 Z 1001 8411 2477 0 80 0 - 0 - ? 00:00:00 nextsong <defunct>
1 Z 1001 8412 2477 0 80 0 - 0 - ? 00:00:00 ices <defunct>
0 Z 1001 9155 2477 0 80 0 - 0 - ? 00:00:00 nextsong <defunct>
what i can fix it? (it was in centova v2 too)
-
Based on what you quoted this is an ices issue, not a Centova Cast issue.
In any case, we don't receive many (read: none as far back as I can remember) reports of ices freezing up, so most likely there is a problem of some sort on your server... various kernel issues can do this, as can an NFS share becoming unreachable, etc.
-
Hm... ;)
Steve, what linux OS you recommend for centovacast v2 and v3 ?
-
we use kernel: 2.6.32-5-686-bigmem
-
Same problem here on CentOS 6.3 - 64bit
psa|grep defunct
29428 pts/0 S+ 0:00 | \_ grep defunct
25124 ? ZN 0:00 \_ [ices] <defunct>
25164 ? ZN 0:00 \_ [php] <defunct>
25166 ? ZN 0:00 \_ [ices] <defunct>
25222 ? ZN 0:00 \_ [php] <defunct>
25224 ? ZN 0:00 \_ [php] <defunct>
25226 ? ZN 0:00 \_ [ices] <defunct>
25275 ? ZN 0:00 \_ [php] <defunct>
25277 ? ZN 0:00 \_ [php] <defunct>
25279 ? ZN 0:00 \_ [ices] <defunct>
......
Any solution?
-
Same problem with CentOS 6.3 64bit. Any Solution CentovaCast?
I'm thinking that the problem is in ices-cc, modified for Centova.
500 32732 0.0 0.0 0 0 ? Z 09:33 0:00 [php] <defunct>
500 32734 0.0 0.0 0 0 ? Z 09:33 0:00 [ices] <defunct>
http://icecast.imux.net/viewtopic.php?t=7951&sid=173faa06063214cbb8e9c97caee7c27f
-
top - 18:32:41 up 1 day, 11:01, 1 user, load average: 4.60, 5.08, 9.48
Tasks: 307 total, 1 running, 178 sleeping, 0 stopped, 128 zombie
-
Same problem with CentOS 6.3 64bit. Any Solution CentovaCast?
No, and FWIW I cannot reproduce this under CentOS 6.3.
What does the output of `ps -eo ppid,pid,state,cmd` look like while this is going on? As Karl stated this should only happen if the parent process stops reaping its children, but the parent process should be init for any processes spawned by Centova Cast (incl. ices).
If you see a 1 in the PPID field for your ices zombie processes, then init is not reaping its children and there is something very wrong with your system which has nothing to do with ices *or* Centova Cast. If you see something else there, then Centova Cast is for some reason unable to reparent the ices process to init (which would also likely indicate a problem with your server since it works everywhere else, and we're using a standard fork/setsid/fork to daemonize, but it'd at least be something we could further investigate).
-
ps -eo ppid,pid,state,cmd
3845 2446 Z [ices] <defunct>
3888 2449 Z [nextsong] <defunct>
3888 2450 Z [ices] <defunct>
3767 2451 Z [nextsong] <defunct>
3767 2452 Z [ices] <defunct>
3819 2474 Z [ices] <defunct>
30217 2476 Z [ices] <defunct>
no ppid with 1....
-
Your filtering (grep or whatever) makes this somewhat useless... without the rest of the output we can't identify processes 3845, 3888, etc. to see which parent processes are failing to reap their children.
-
1 889 S /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/va
889 978 S /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user
1 1070 S /usr/libexec/postfix/master
1070 1077 S qmgr -l -t fifo -u
1 1080 S proftpd: (accepting connections)
1 1088 S /usr/sbin/httpd
1 1096 S crond
1 1110 S /sbin/mingetty /dev/tty1
1 1112 S /sbin/mingetty /dev/tty2
1 1114 S /sbin/agetty /dev/hvc0 38400 vt100-nav
1 1115 S /sbin/mingetty /dev/tty3
259 1117 S /sbin/udevd -d
1 1118 S /sbin/mingetty /dev/tty4
1 1120 S /sbin/mingetty /dev/tty5
1 1122 S /sbin/mingetty /dev/tty6
1088 1123 S /usr/sbin/httpd
1088 1124 S /usr/sbin/httpd
1088 1125 S /usr/sbin/httpd
1088 1126 S /usr/sbin/httpd
1088 1127 S /usr/sbin/httpd
1088 1128 S /usr/sbin/httpd
1088 1129 S /usr/sbin/httpd
1088 1130 S /usr/sbin/httpd
1088 1141 S /usr/sbin/httpd
1 1210 S /home/centovacast/shoutcast/sc_serv /home/centovacast/vhosts/radio
1 1284 S /home/centovacast/shoutcast/sc_serv /home/centovacast/vhosts/orbit
1 1294 S /home/centovacast/shoutcast/sc_serv /home/centovacast/vhosts/radio
1 1303 S /usr/local/ices/bin ices -v -c /home/centovacast/vhosts/radioebene
1 1310 S /home/centovacast/shoutcast/sc_serv /home/centovacast/vhosts/radio
1 1319 S /usr/local/ices/bin ices -v -c /home/centovacast/vhosts/radiofasip
1 1327 S /home/centovacast/shoutcast/sc_serv /home/centovacast/vhosts/radio
1 1334 S /usr/local/ices/bin ices -v -c /home/centovacast/vhosts/radiometod
1 1343 S /home/centovacast/shoutcast/sc_serv /home/centovacast/vhosts/radio
1 1350 S /home/centovacast/shoutcast/sc_serv /home/centovacast/vhosts/radio
1 1357 S /usr/local/ices/bin ices -v -c /home/centovacast/vhosts/radiosomdo
1 1365 S /home/centovacast/shoutcast/sc_serv /home/centovacast/vhosts/skyle
1 1372 S /usr/local/ices/bin ices -v -c /home/centovacast/vhosts/skyler/etc
1 1380 S /home/centovacast/shoutcast/sc_serv /home/centovacast/vhosts/tomso
1 1388 S /home/centovacast/shoutcast/sc_serv /home/centovacast/vhosts/yahwe
1 1395 S /usr/local/ices/bin ices -v -c /home/centovacast/vhosts/yahweh/etc
1 1403 S /usr/local/bin/icecast -c /home/centovacast/vhosts/93fm/etc/server
1 1410 S /usr/local/bin/icecast -c /home/centovacast/vhosts/agenciasindical
1 1417 S /usr/local/bin/icecast -c /home/centovacast/vhosts/araranguaam/etc
1 1424 S /usr/local/bin/icecast -c /home/centovacast/vhosts/culturafm/etc/s
1 1432 S /usr/local/bin/icecast -c /home/centovacast/vhosts/energiafm/etc/s
1 1439 S /usr/local/bin/icecast -c /home/centovacast/vhosts/fm97/etc/server
1 1446 S /usr/local/bin/icecast -c /home/centovacast/vhosts/iradio/etc/serv
1 1454 S /usr/local/bin/icecast -c /home/centovacast/vhosts/meridional/etc/
1 1461 S /usr/local/bin/icecast -c /home/centovacast/vhosts/meridionalcolor
1 1468 S /usr/local/bin/icecast -c /home/centovacast/vhosts/meridionaljaru/
1 1475 S /usr/local/bin/icecast -c /home/centovacast/vhosts/meridionalvilhe
1 1482 S /usr/local/bin/icecast -c /home/centovacast/vhosts/ondasulfmhd/etc
1 1489 S /usr/local/bin/icecast -c /home/centovacast/vhosts/radioa4pma/etc/
1 1497 S /usr/local/bin/icecast -c /home/centovacast/vhosts/radiocidade/etc
1 1504 S /usr/local/bin/icecast -c /home/centovacast/vhosts/radioguaruja/et
1 1511 S /usr/local/bin/icecast -c /home/centovacast/vhosts/radiomoc/etc/se
1 1518 S /usr/local/bin/icecast -c /home/centovacast/vhosts/radiorecordrj/e
1 1525 S /usr/local/bin/icecast -c /home/centovacast/vhosts/radiozoom/etc/s
1 1532 S /usr/local/bin/icecast -c /home/centovacast/vhosts/rdireitodeviver
1 1539 S /usr/local/bin/icecast -c /home/centovacast/vhosts/regionalfm/etc/
1 1546 S /usr/local/bin/icecast -c /home/centovacast/vhosts/ssmhd/etc/serve
1 1553 S /usr/local/bin/icecast -c /home/centovacast/vhosts/testehd/etc/ser
1 1560 S /usr/local/bin/icecast -c /home/centovacast/vhosts/transamericafm/
1088 5498 S /usr/sbin/httpd
1070 15981 S pickup -l -t fifo -u
1 16109 S /usr/local/bin/icecast -c /home/centovacast/vhosts/tocantinsfmarag
1088 16126 S /usr/sbin/httpd
1 16145 S /usr/local/bin/icecast -c /home/centovacast/vhosts/tocantinsfmguru
1088 19483 S /usr/sbin/httpd
1088 19509 S /usr/sbin/httpd
1395 19914 Z [php] <defunct>
1395 19916 Z [ices] <defunct>
1357 19934 Z [ices] <defunct>
1357 19935 Z [php] <defunct>
1357 19937 Z [ices] <defunct>
1372 19983 Z [php] <defunct>
1372 19985 Z [ices] <defunct>
754 20058 S sleep 60
853 20069 S sshd: root@pts/0
20069 20076 S -bash
20076 20092 R ps -eo ppid,pid,state,cmd
-
Steve i can provide you you shell access ...
-
OK that confirms basically what I've received from another client as well. This does in fact appear to be an ices issue. As Karl noted in the thread carlosjpr linked to previously, a defunct process indicates a problem in the *parent* process.
Taking two of your processes as an example:
1395 19914 Z [php] <defunct>
1395 19916 Z [ices] <defunct>
...these zombie processes are owned by PID 1395... which is:
1 1395 S /usr/local/ices/bin ices -v -c /home/centovacast/vhosts/yahweh/etc
...ices. So ices is the parent, and ices is failing to reap its child processes when they exit.
As for why, I don't have a clue... it's pretty certain to be a bug of some sort, and it appears to happen exclusively on CentOS 6.3 according to the reports we've received, so clearly something has changed (perhaps in the default kernel used by CentOS 6.3 or somesuch) to bring this bug to light.
FWIW, I've created several CentOS 6.3 VMs and ices has worked fine on all of them, which is somewhat puzzling. We'll continue to investigate as best we can but just a reminder to everyone that ices is NOT our product and we are NOT the developers of ices, so our recourse is somewhat limited here.
-
Steve,
Thank you for your attention to this bug. Would you like to access one of my VM`s with CentOS 6.3 to check the difference of config of your VM`s?
I use XenServer
-
Thank you for your attention to this bug. Would you like to access one of my VM`s with CentOS 6.3 to check the difference of config of your VM`s?
Thanks but I've already obtained access to an affected server and I've found no perceptible difference between the affected machine and one of our own (which does note exhibit the problem), right down to the exact kernel version.
Clearly *something* has changed in CentOS 6.3 as this doesn't seem to affect users of ANY other Linux distribution, but it doesn't appear to be kernel-specific so I'm stumped.
I did make some changes in the latest version of the control daemon to change how the daemon forks new processes though, so if anyone wants to update and give that a try (being sure to kill off all ices processes after upgrading to ensure a "clean start") I'd be interested to hear if that makes a difference. Technically the changes *shouldn't* make a difference, but they safeguard against a similar problem that affects certain oldschool UNIX environments so I figured it was worth a shot anyway.
-
I would like to test Steve.
-
I would like to test Steve.
By all means, go for it.
-
OK folks, some interesting developments. While testing a client's server that was affected by this issue, I grabbed a precompiled copy of ices from one of our Debian Squeeze servers, copied it to the affected server, and reconfigured Centova Cast to use it instead of the copy that had been built on the CentOS 6.3 machine.
I've been monitoring the server for about an hour now, and not a single defunct process has shown up. (Previously, they'd pop up every few minutes.)
Given that ices binaries compiled using the buildices.sh script provide custom versions of all of their own non-system libraries (vorbis, shout, lame, xml, etc.) and do not depend on anything from the CentOS repos, my best guess at this point is that this is some kind of issue with the build toolchain used by CentOS 6.3.
I realize that that in itself doesn't help much for those of you experiencing this issue, but at least it's a step toward a diagnosis -- and confirmation that the problem is definitely not within the Centova Cast code base.
-
Steve, so ... that is, it can be concluded that there are no errors on centos?
Can you provide me 3-4 days free new Centova license to test on second VPS server with CentOS ? Please :)
So i can provide you ssh access to our host server (his stats see below).
top - 23:42:45 up 4 days, 15:43, 1 user, load average: 0.84, 2.54, 1.88
Tasks: 323 total, 1 running, 184 sleeping, 0 stopped, 138 zombie
And.. can you fix ezstream CC3 module? (errors with creating), and i found some errors with ices 0.4 (no cc) module.
-
Steve, so ... that is, it can be concluded that there are no errors on centos?
I can't say that as I don't know whether it's a problem in the ices code/Makefile, or a problem in the version of gcc/glibc used by CentOS 6.3, or just some kind of bad interaction between the two.
Can you provide me 3-4 days free new Centova license to test on second VPS server with CentOS ?
You're welcome to order a trial.
And.. can you fix ezstream CC3 module? (errors with creating), and i found some errors with ices 0.4 (no cc) module.
We can fix just about any issue that's reported. If you find bugs, please report them to the bugs/issues forum.
-
OK folks, some interesting developments. While testing a client's server that was affected by this issue, I grabbed a precompiled copy of ices from one of our Debian Squeeze servers, copied it to the affected server, and reconfigured Centova Cast to use it instead of the copy that had been built on the CentOS 6.3 machine.
Steve,
Could you send the precompiled copy of ices to I copy it to my affected server, and how I reconfigure Centova Cast to use it instead of the copy that had been built on the CentOS 6.3 machine?
Thanks!
-
What need to be reconfigured in Centova Cast (v2) to use a ices compiled in other server? Only replacing the ice file not works?
I am with the same issue in a server CENTOS 5.9 x86_64 xenpv - 2.6.18-164.11.1.el5xen