Cyrus dies quite frequently

[robert78]

Hi,
 Cyrus dies quite frequently when the server gets some load (copying public mail folders from an Exchange 5.5 server...)
Restarting cyrus fixes the problem in such a case.

Any ideas on how to debug this?

TIA,
 Robert

[robert78]

The processes seem to hang when this happens:
 ~ # ps aux | grep cyr
cyrus    13450  0.0  0.1  14548  1616 ?        Ss   13:58   0:00 /usr/lib/cyrus/master
cyrus    13564  0.0  0.3  60164  3736 ?        S    13:59   0:01 imapd -C /etc/imapd.conf
cyrus    13597  0.0  0.3  60164  3724 ?        S    14:01   0:00 imapd -C /etc/imapd.conf
cyrus    13723  0.0  0.3  59412  3460 ?        S    14:11   0:00 pop3d -s -C /etc/imapd.conf
cyrus    13746  0.0  0.3  59228  3412 ?        S    14:13   0:00 pop3d -s -C /etc/imapd.conf
cyrus    13748  0.0  0.2  58508  2900 ?        S    14:15   0:00 lmtpd -a
cyrus    13750  0.0  0.2  58508  2904 ?        S    14:15   0:00 lmtpd -a
cyrus    13753  0.0  0.2  58512  2960 ?        S    14:15   0:00 lmtpd -a
cyrus    13757  0.0  0.2  58512  2896 ?        S    14:15   0:00 lmtpd -a
cyrus    13761  0.0  0.2  58512  2920 ?        S    14:15   0:00 lmtpd -a
cyrus    13765  0.0  0.2  58512  2912 ?        S    14:15   0:00 lmtpd -a
cyrus    13776  0.0  0.2  58512  2896 ?        S    14:15   0:00 lmtpd -a
cyrus    13777  0.0  0.2  58508  2884 ?        S    14:15   0:00 lmtpd -a
cyrus    13781  0.0  0.2  58508  2884 ?        S    14:15   0:00 lmtpd -a
cyrus    13784  0.0  0.2  58512  2868 ?        S    14:15   0:00 lmtpd -a
cyrus    13789  0.0  0.2  58512  2848 ?        S    14:15   0:00 lmtpd -a
cyrus    13797  0.0  0.2  58508  2864 ?        S    14:15   0:00 lmtpd -a
cyrus    13799  0.0  0.2  58516  2900 ?        S    14:15   0:00 lmtpd -a
cyrus    13807  0.0  0.2  58508  2864 ?        S    14:15   0:00 lmtpd -a
cyrus    13820  0.0  0.2  58512  2860 ?        S    14:15   0:00 lmtpd -a
cyrus    13824  0.0  0.2  58512  2852 ?        S    14:15   0:00 lmtpd -a
cyrus    13825  0.0  0.2  58512  2856 ?        S    14:15   0:00 lmtpd -a
cyrus    13856  0.0  0.2  58512  2812 ?        S    14:15   0:00 lmtpd -a
cyrus    13857  0.0  0.2  58512  2808 ?        S    14:15   0:00 lmtpd -a
cyrus    13861  0.0  0.2  58512  2824 ?        S    14:15   0:00 lmtpd -a
cyrus    13863  0.0  0.2  58512  2808 ?        S    14:15   0:00 lmtpd -a
cyrus    13869  0.0  0.2  58512  2816 ?        S    14:15   0:00 lmtpd -a
cyrus    13871  0.0  0.2  58512  2808 ?        S    14:15   0:00 lmtpd -a
cyrus    13901  0.0  0.1  46612  1720 ?        S    14:15   0:00 lmtpd -a
cyrus    14078  0.0  0.1  15556  1600 ?        S    14:28   0:00 ctl_cyrusdb -c
root     14155  0.0  0.0   2624   564 pts/0    R+   14:32   0:00 grep cyr
m2 ~ # /etc/init.d/cyrus restart
 * Stopping kolabd ...                                                                                                                       [ ok ]
 * Stopping cyrus imapd ...                                                                                                                  [ ok ]
 * Starting cyrus imapd ...                                                                                                                  [ ok ]
 * Starting kolabd ...                                                                                                                       [ ok ]
 ~ # ps aux | grep cyr
cyrus    14433  0.2  0.1  14548  1616 ?        Ss   14:32   0:00 /usr/lib/cyrus/master
cyrus    14830  1.0  0.2  44680  2464 ?        D    14:32   0:00 ctl_cyrusdb -c
cyrus    14832  0.0  0.2  58508  2876 ?        S    14:32   0:00 lmtpd -a
cyrus    14833  0.0  0.2  58964  2884 ?        D    14:32   0:00 lmtpd -a
cyrus    14834  0.0  0.2  58660  2920 ?        D    14:32   0:00 lmtpd -a
cyrus    14835  0.0  0.2  58512  2888 ?        S    14:32   0:00 lmtpd -a
cyrus    14836  0.0  0.2  58512  2840 ?        S    14:32   0:00 lmtpd -a
cyrus    14837  0.0  0.2  58956  2876 ?        D    14:32   0:00 lmtpd -a
cyrus    14838  0.0  0.2  58940  2864 ?        D    14:32   0:00 lmtpd -a
cyrus    14839  0.0  0.2  58508  2844 ?        S    14:32   0:00 lmtpd -a
cyrus    14840  0.0  0.2  58384  2864 ?        S    14:32   0:00 lmtpd -a
cyrus    14847  0.0  0.2  58888  2864 ?        D    14:32   0:00 lmtpd -a
cyrus    14848  0.0  0.2  58972  2856 ?        D    14:32   0:00 lmtpd -a
cyrus    14849  0.0  0.2  58508  2856 ?        S    14:32   0:00 lmtpd -a
cyrus    14850  0.0  0.2  58516  2896 ?        S    14:32   0:00 lmtpd -a
cyrus    14853  0.0  0.2  58512  2848 ?        S    14:32   0:00 lmtpd -a
cyrus    14854  0.0  0.2  58512  2892 ?        S    14:32   0:00 lmtpd -a
cyrus    14855  0.0  0.2  58512  2864 ?        S    14:32   0:00 lmtpd -a
cyrus    14856  0.0  0.2  58512  2844 ?        S    14:32   0:00 lmtpd -a
cyrus    14857  0.0  0.2  58512  2876 ?        S    14:32   0:00 lmtpd -a
cyrus    14858  0.0  0.2  58516  2852 ?        S    14:32   0:00 lmtpd -a
cyrus    14859  0.0  0.2  59324  2812 ?        D    14:32   0:00 lmtpd -a
cyrus    14860  0.0  0.2  58980  2860 ?        D    14:32   0:00 lmtpd -a
cyrus    14861  0.0  0.2  58508  2848 ?        S    14:32   0:00 lmtpd -a
cyrus    14862  0.0  0.2  58512  2856 ?        S    14:32   0:00 lmtpd -a
cyrus    14863  0.0  0.2  59544  2856 ?        D    14:32   0:00 lmtpd -a
cyrus    14864  0.0  0.2  58512  2848 ?        S    14:32   0:00 lmtpd -a
cyrus    14866  0.0  0.2  58384  2836 ?        S    14:32   0:00 lmtpd -a
cyrus    14867  0.0  0.2  58512  2852 ?        S    14:32   0:00 lmtpd -a
cyrus    14868  0.0  0.2  58964  2808 ?        D    14:32   0:00 lmtpd -a
cyrus    14869  0.0  0.2  58548  2820 ?        R    14:32   0:00 lmtpd -a
cyrus    14870  0.0  0.2  58512  2864 ?        S    14:32   0:00 lmtpd -a
cyrus    14879  0.0  0.2  58980  2812 ?        D    14:32   0:00 lmtpd -a
cyrus    14881  0.0  0.2  58500  2592 ?        D    14:32   0:00 imapd -C /etc/imapd.conf
root     14904  0.0  0.0   2628   600 pts/0    S+   14:32   0:00 grep cyr

After the restart, it works for a while....

[robert78]

The last relevant log enty is
master[21146]: about to exec /usr/lib/cyrus/imapd
After that, Cyrus is dead until a restart

[Gunnar Wrobel]

I guess it is most probable that your machine hits some fixed limits. But without knowing anything about the hardware and the configuration it is hard to judge that. If it only happens during load it is at least unlikely that the problem is caused by a problem with a library or some strange compile options with cyrus itself.

I would start by monitoring the hardware itself and check if there are any ressources that are too low.

[robert78]

The load is actually very low.
The machine is an Athlon64 3700+ with 1 GB of RAM
So, the hardware itself is probably not the limit. However, I am monitoring it.
I will, BTW, post details about the Exchange 5.5 migration to Kolab, once everything runs smoothly...

 Robert

[Gunnar Wrobel]

Ok, the machine hardware should not really be a problem. But I guess that it might be hard to identify the problem then. Does it have any effect if you modify the number of times the lmtpd process is being reused (-U flag for the lmtpd process in /etc/cyrus.conf). You may also specify a debugging command for lmtpd using the -D flag but I never tried that so far.

Does the lmtpd problem relate to your previous post about problems with "/usr/bin/php" timeouts? This would indicate that the problem somehow relates to the kolab mail filters that are written in php. On the other hand that is not very probable since the php filters are only called by postfix.

Any information about migration from exchange 5.5 are certainly welcome. If it is not Gentoo specific I guess the best repository for that is the kolab wiki at http://wiki.kolab.org

Cheers,

Gunnar

[robert78]

Hi Gunnar,

the PHP time limit problem seems to be gone since I put
max_execution_time = 120     
max_input_time = 120   
memory_limit = 64M     
in php.ini

I just realized that the load goes over 10 when copying mail folders from Exchange (during IMAP append). Since CPU usage stays low, I asked the guys at the data center to give me another GB of RAM...
I'll report back when the upgrade was done.

Thanks, Gunnar!

[robert78]

Ok, the machine hardware should not really be a problem. But I guess that it might be hard to identify the problem then. Does it have any effect if you modify the number of times the lmtpd process is being reused (-U flag for the lmtpd process in /etc/cyrus.conf). You may also specify a debugging command for lmtpd using the -D flag but I never tried that so far.

Can you give me an exaple of the syntax?
I could not find a description.

Thanks,
 Robert

[robert78]

OK, I found it.

Set to -U 20 and -U 400, but still no go...

BTW: Is there a standard / safe way to update Kolab on Gentoo?
Maybe I should update the packages (currently running kolabd-20060523)

Thanks,
 Robert

[robert78]

I'm not sure if lmtpd is actually the problem:
 ~ # ps aux | grep imapd
cyrus    20010  0.0  0.1  59436  3492 ?        S    14:26   0:00 pop3d -s -C /etc/imapd.conf
cyrus    20196  0.0  0.1  60036  3828 ?        S    14:26   0:00 imapd -C /etc/imapd.conf
cyrus    20361  0.0  0.0  17772  1692 ?        S    14:26   0:00 pop3d -s -C /etc/imapd.conf
cyrus    20394  0.0  0.0  17976  1672 ?        S    14:28   0:00 imapd -C /etc/imapd.conf

imapd -s -C /etc/imapd.conf seems to be gone or is at least not restarted...

[Gunnar Wrobel]

I am not certain I fully grasp the conditions under which the error occurs.

Are you currently certain that the problem is caused by a low amount of memory? Did the upgrade in memory change anything about the situation?

What procedure are you using for copying the mails? I'd assume imapsync - is that correct? In any case you should use a tool that does not route the mails over the postfix server but copies the messages into the imap server directly. If the mails are routed through the MTA at a massive rate this might cause problems with the spam/virus filters and the kolab php scripts. This would be at least a procedure that might kill a significant amount of memory.

Is the copying procedure the only process that leads to failure of the cyrus server? Or does this also happen on a server with standard load?

[robert78]

The memory upgrade (2 GB now), keeps ther server from swapping, but it did not solve the problem.

Imapsync turned out to be too heavy on our Exchange server, so we use KMail to copy the mails.
The problem, however, does not only occur when copying mails, but also (a couple of times a day) under normal usage load (25 clients, 4 of which are Kontact, the rest is Outlook 2003 with Kolab connector. They will be switched to Linux once everything runs smoothly)
When Cyrus, crashes / hangs, the load goes significantly up for a relatively short period of thime (lots of PHP processes), and only restarting Cyrus will bring it back.

I have not yet tried to update to the newest version. (See my question about updating). Maybe that would even solve the problem.
I just wanted to be sure what is considered the safest way before I do it.

[Gunnar Wrobel]

Hm, the machine should be able to handle this type of load.

I am somewhat confused by the combination with the PHP processes. Can you confirm that these jobs originate from the Postfix MTA? I still assume that the sporadic failure for cyrus indicates some hardware limitation which could be the case if php consumes irrational amounts of resources. On the other hand I do not understand why postfix should be involved if you just copy mails from IMAP to IMAP.

But I am able to run Kolab on virtual servers with PHP set to

max_execution_time = 30;
max_input_time = 60;
memory_limit = 8M;

so the new memory limit you had to set on your php might indicate that the problem originates from you php setup.