DB Memory Issue

[jayk]

I have been having an issue on one of the kolab servers that I admin. Every now and then, cyrus stops working and my log starts getting multiple errors like this one:

Code:

Aug 20 15:43:37 mainsrv master[3417]: about to exec /usr/lib/cyrus/imapd
Aug 20 15:43:37 mainsrv imaps[3417]: executed
Aug 20 15:43:37 mainsrv imaps[3417]: sql_select option missing
Aug 20 15:43:37 mainsrv imaps[3417]: auxpropfunc error no mechanism available
Aug 20 15:43:37 mainsrv imaps[3417]: _sasl_plugin_load failed on sasl_auxprop_plug_init for plugin: sql
Aug 20 15:43:37 mainsrv imaps[3417]: DBERROR db4: Logging region out of memory; you may need to increase its size
Aug 20 15:43:37 mainsrv imaps[3417]: DBERROR: opening /var/imap/mailboxes.db: Cannot allocate memory
Aug 20 15:43:37 mainsrv imaps[3417]: DBERROR: opening /var/imap/mailboxes.db: cyrusdb error
Aug 20 15:43:37 mainsrv imaps[3417]: Fatal error: can't read mailboxes file

Through some searching, I came across a blog post that told how to solve the db4: Logging region out of memory error by increasing the memory available for the Berkley DB logging region and Cyrus's caching region.

The blog post is here: http://www.cjc.org/blog/archives/2006/08/22/cyrus-imap-log-and-cache-settings/

The Cyrus bdb files are at /var/imap/db, not /var/lib/imap/db as in the post (they were using Fedora). This isn't an issue. The problem I'm running into is that I don't have a db_recover and db_stat utility. I have db4.2_recover, db4.5_recover and db4.2_stat, db4.5_stat. I don't know whether to run the 4.2 or 4.5 utility.

If anyone can point me in the right direction, it would be much appreciated.

Thanks,
Jason

[Gunnar Wrobel]

Hm, I've seen that error on a Kolab2/Gentoo server before and I have been unable to solve it. From the configuration of the machine it might have been possible that it was indeed a memory issue but in the end I didn't believe so since modifying the DB_CONFIG did not help. Finally it made more sense reinstalling the machine. But in your case I'd suggest trying to backup the mailbox file and recreate the db. maybe that fixes the db4 issue.

Cheers,

Gunnar

[Gunnar Wrobel]

Funny, had the issue on a server today...

Try the following:

Code:

echo "set_lg_regionmax        1048576
set_lg_max              10485760
set_lg_bsize            2097152" > /var/imap/db/.db_config

Followed by:

Code:

/etc/init.d/cyrus stop && mv /var/imap/db/.db_config /var/imap/db/DB_CONFIG && db4.3_recover -h /var/imap/db -v && chown -R cyrus\:mail db && /etc/init.d/cyrus start

Does that help?

Cheers,

Gunnar

[xehqter]

I had the same problem today, I tried your method Gunnar, lets see if it is fixed.

[gardeniik]

Hi,
I have this thing on my kolab installation on amd64 arch (2GB RAM). And this didn't help. Still the same error;o( But the time between the errors is longer.

Gardenik

[gardeniik]

Do You think using skiplist instead of berkeley backend would solve this? Anny suggestions? Any known problems? I have a problem that server is in production ;o(

Gardenik

[xehqter]

The fix above solved my issue.  The only thing I can think of would be to double the suggested log size values or try to dump the database and then re-import it as in http://pybsddb.sourceforge.net/ref/log/limits.html

make backup's first, these are only guesses.

[xehqter]

as per http://tnt.aufbix.org/tnt/doku.php?id=cyrus

DB_CONFIG from one of the largest cyrus sites

Code:

set_cachesize 0 2097152 1
set_lg_regionmax 2097152
set_lg_bsize 2097152
set_lg_max 16777216
set_tx_max 100

if you run db_stat -l (db4.5_stat -l in my case) from /var/imap/db what is the output?

[Gunnar Wrobel]

Reverting to skiplist should also be a solution.

The Kolab guys moved away from skiplist since they had some strange errors that seemed to be loosely connected to using skiplist. But I believe this was never really proven.

Anyhow if people get problems with using berkeley db they should switch back to skiplist for now. I'll have to investigate this further. There might be other problems involved and Berkeley db is not the easiest thing to debug.

[gardeniik]

I used new configuration You wrote me to use. It crashes again but after nearly 3 days.

db4.2_stat -l lists this:

40988   Log magic number.
8       Log version number.
2MB     Log record cache size.
0644    Log file mode.
16Mb    Current log file size.
181MB 496KB 521B        Log bytes written.
        Log bytes written since last checkpoint.
1936    Total log file writes.
0       Total log file write due to overflow.
1936    Total log file flushes.
12      Current log file number.
5864010 Current log file offset.
12      On-disk log file number.
5864010 On-disk log file offset.
4       Max commits in a log flush.
0       Min commits in a log flush.
4MB     Log region size.
471     The number of region locks granted after waiting.
3370082 The number of region locks granted without waiting.

And I also found that imapd segafults with error 4
dmesg part

imapd[3648]: segfault at 0000000000000078 rip 0000000000413e13 rsp 00007fff5b47e910 error 4
imapd[3650]: segfault at 0000000000000078 rip 0000000000413e13 rsp 00007ffff94538f0 error 4
imapd[3651]: segfault at 0000000000000078 rip 0000000000413e13 rsp 00007fffd808e530 error 4
imapd[3652]: segfault at 0000000000000078 rip 0000000000413e13 rsp 00007fff290af550 error 4
imapd[3654]: segfault at 0000000000000078 rip 0000000000413e13 rsp 00007fffcb098530 error 4
imapd[3655]: segfault at 0000000000000078 rip 0000000000413e13 rsp 00007fff52dcb260 error 4

Any Ideas? I already have no idea....

Gardenik

[Gunnar Wrobel]

@gardenik: this is using "skiplist" now? This should not be affected by db issues. The cyrus-imapd segfault might be caused by a problem in one of the Kolab patches. Can you try to upgrade to 2.3.9-r20?

[gardeniik]

no, this is using still bdb bud DB_CONFIG  posted by xehqter. I'm using cyrus 2.3.8-r20 all higher versions are blocked

Calculating dependencies... done!
[ebuild     U ] net-mail/cyrus-imapd-2.3.9-r20 [2.3.8-r20] USE="kolab pam ssl tcpd -autocreate -autosieve (-drac) -idled -kerberos -nntp -replication% -snmp (-afs%)" 2,237 kB [1]
[blocks B     ] >net-mail/cyrus-imapd-2.3.8-r20 (is blocking net-mail/kolabd-2.1)

G.

[gardeniik]

Do You think I should first try to upgrade it or revert back to skiplist? when upgrading shall I use FEATURES="-collision-protect" ?
I' m very stressed about all changes cause server is in in use ;|
Gardenik

[Gunnar Wrobel]

Do You think I should first try to upgrade it or revert back to skiplist? when upgrading shall I use FEATURES="-collision-protect" ?

Upgrade first and change it to skiplist then.

FEATURES="-collision-protect" is incorrect. You need to resolve the block => man emerge

I' m very stressed about all changes cause server is in in use ;|

I must admit I have no pity. The warning states clearly enough that Kolab2/Gentoo is no production quality. I added this warning for a reason. If you need a production quality Kolab server you will have to run Kolab2/OpenPKG.

[xehqter]

Out of curiosity how many users are on this box? My only suggestion would be to run the following command every few hours (daily maybe)

su -s /bin/bash -c "/usr/lib/cyrus/ctl_cyrusdb -r -x" cyrus


As a byproduct of recovering the databases it seems to reset the log file (or at least the stats)

If that doesn't work try backing up your kolab install, wipe the box, load kolab, and restore your data (ldap, cyrus store, etc)

If that isn't an option, listen to gunner, he knows more then me when it comes to kolab.