DSPAM on Kolab

[xehqter]

I'm almost done integrating DSpam with kolab on gentoo and would like to post a HowTo, I need to attach some patches/templates/scripts and wondered where the best place to do so is since I don't have a webhost.

[jayk]

I put in a request to Gunnar to make the attachments directory on the forums writable.

http://forum.pardus.de/index.php/topic,128.0.html

Because of the hour in Germany now, I don't expect he'll post back on the subject until tomorrow. That might be a good option if he's willing to do it.

Jason

[jayk]

Another place might be the kolab wiki. The howto could be tied in with the other Gentoo related docs. I'm planning to modify the imap store backup document for kolab on Gentoo and put it up there when I get a chance. I don't know if you can upload files there though.

Jason

[xehqter]

since I couldn't figure out how to implement dspam the way I like it (spam, block, unblock folders) with amavis-new I created another smtpd port w/ postfix which calls cyrus's deliver command and moves the message to the spam folder. I then modified kolabfilter.php to run dspam and use that port if a message is flagged as spam rather then amavis's port. I was wondering if anyone had any other ideas?

[Gunnar Wrobel]

In principle you should be able to upload stuff to the forum now. But in any case the kolab wiki is definitely the best place for storing extension information at the moment.

It does not have to be perfectly structured or anything. Just dump it in there. I might pick it up later or others will. I'll close the p@rdus wiki at some point anyhow once I transferred all knowledge over to the kolab wiki which is now the only point of reference.

[SteveB]

I know DSPAM very well. What kind of setup did you wanted to implement? Can you describe what you want to do? I probably can help you.

// SteveB

[xehqter]

This is the basic info, i'll add more later

http://wiki.kolab.org/index.php/Gentoo_-_dspam_-_Installation

[Gunnar Wrobel]

I hope to be able to work on improved spam rejection soon and I will certainly take a look at this then. Thanks for contributing it!

[Gunnar Wrobel]

Hi Jeremy,

I started looking at the wiki page you generated. Looks very good.

I'm currently considering moving into the direction of a separate spam box that does all the spam analysis.

With the current Kolab configuration this would simply mean to outsource amavisd, spamassassin and clamav to a separate machine. Postfix would then route incoming mail to the spam box rather than to localhost:10024 and receive the messages back from the spam box via port 10026.

Now I found some references that gave me the impression that I could hang dspam also into amavisd-new and I was wondering why you choose to let postfix directly send stuff to dspam.

For the configuration I have in mind I believe it would be easier to let amavisd-new handle all the various filters.

Cheers,

Gunnar

[xehqter]

could you give me the URL's? I'll try them out, all the searching I did lead me to be believed I was married to SpamAssassin with amavisd-new

[xehqter]

I’m finding sparse information regarding setting up dspam w/ amavisd-new, all of it seems to overlook how you retrain the email. Dspam doesn’t work out of the box, it must be trained and retrained.

I had mixed results when I tried it, maybe due to configuration but I can’t find enough information to narrow it down. I tried the $dspam = 'dspam'; method (vs smtp forwarding). It launches dspam and classifies email, however it seems to override values in the dspam.conf file.

it seems to override virtual users, so it stores all token data under the amavis user rather then creating individual databases for each user. Another oddity is that it inserts the ISO character map into the results.  Ie: X-DSPAM-Result: =?iso-8859-1?Q?Innocent=0D?=

I’m also not sure how it would handle plus addressing, what I’ve found says it puts spam in the amavis quarantine, which my method avoids, it dumps suspected spam into an IMAP spam folder. 

It would be interesting to see how it is integrated into amavis, maybe that would solve some of my issues. Until then I’ll stick with forwarding because it supports all the features I need.

[Gunnar Wrobel]

This is a very recent thread:

http://dspam.nuclearelephant.com/dspam-users/6551.html

It also highlights that there are apparently still some problems when trying to integrate the two.

amavisd-new-2.5.2 does have a config line concerning dspam though:

> grep dspam amavisd.conf
$dspam  = 'dspam';

not much

I don't dspam that well yet... can you judge if the drawbacks mentioned in the thread I posted above are really problematic?

If embedding dspam in amavisd-new is not useful, do you thinkg it would be possible to let amavisd-new pipe its output directly into dspam rather than redirecting through postfix again? Dspam could then perform the final delivery back into postfix so that one would be able to have the spam handling running on just one box?

This would also have the advantage of not changing the current situation too much so it might be easies to wrap the whole thing into one template ebuild with a "dspam" use flag.
At least that is what I'm aiming at.

Cheers,

Gunnar


... darn, you posted while I was writing too anyhow, I'll post this one and answer your new post then

[Gunnar Wrobel]

okay, thanks for the infos.

That is already a good basis to start playing around with this. I'll probably have to get to know dspam better at this point.

Concerning the training I'm probably going to provide a new script that will fetch spam that the users marked as such. I'll see how this might work out with dspam.

Cheers,

Gunnar

[xehqter]

DSpam can be configured in many different ways, so there are a plethora of guides all accomplishing the same thing in a different way. Many of them are directed towards POP3. With pop3 you'll often have a web interface with all mail quarentened as spam for ham retraining and a email address to send your spam for spam retraining. I find that IMAP is a much better alternative since you can have folders and have all the work done behind the scene. Just drag and drop

I have two cron scripts running every minute. One checks for new users and creates the spam, block, and unblock folders, then assigns the correct permissions. The other scans each users block & unblock folder for corrections, retrains dspam, and cleanup work.

I've only had three issues thus far.
Toltec's MIME parser has issue with incorrectly headers which causes it to download messages but never display or modify them. I solved this problem by automatically deleting spam older then 30 days. I'd think there would be a way to fix malformed headers in postfix but thats a project for another day.

Postfix Content Filters Scan incoming and outgoing email, I need to figure out how to only have it scan incoming

Thunderbird doesn't expunge mail automatically so if you drag a message into block, then move it to unblock it gets trained twice.. not sure how to overcome that one.

Otherwise besides those three issues it's been smooth sailing with dspam once it was configured.

I don't know if my method is what you had in mind for spam filtering, but let me know.

[Gunnar Wrobel]

DSpam can be configured in many different ways, so there are a plethora of guides all accomplishing the same thing in a different way.

Yes, looks like it If somehow possible I want to find a setup that also fits as much as possible into the general upstream structure. It is clear that people are bound to modify their Kolab servers but the base configuration should - if at all possible - always be sound and simple.

Many of them are directed towards POP3. With pop3 you'll often have a web interface with all mail quarentened as spam for ham retraining and a email address to send your spam for spam retraining. I find that IMAP is a much better alternative since you can have folders and have all the work done behind the scene. Just drag and drop

Yes, I agree.

I have two cron scripts running every minute. One checks for new users and creates the spam, block, and unblock folders, then assigns the correct permissions.

This is very useful and I would like to integrate something similar upstream. perl-kolab is currently doing the user setup anyhow and I think it might be possible to automatically create the spam folders upon user creation. I will have to discuss this on kolab-devel though and get feedback.

In any case these spam folders should get folder annotations. The Kolab format currently provides "mail.junkemail" as annotation. Maybe we should just change that to "mail.spam" and "mail.ham".

Using folder annotations rather than your fixed structure has the advantage that the sysadmin can also deviate from the per-user scheme and easily create shared folders for the same purpose.

The other scans each users block & unblock folder for corrections, retrains dspam, and cleanup work.

Did you post that somewhere? I was aiming at generating a python script for the bayes training. I already have one for the spamassassin bayes stuff but this needs to get rewritten.

I've only had three issues thus far.
Toltec's MIME parser has issue with incorrectly headers which causes it to download messages but never display or modify them. I solved this problem by automatically deleting spam older then 30 days. I'd think there would be a way to fix malformed headers in postfix but thats a project for another day.

I don't fully understand where these "incorrect headers" come from?

Postfix Content Filters Scan incoming and outgoing email, I need to figure out how to only have it scan incoming

Ok.

Thunderbird doesn't expunge mail automatically so if you drag a message into block, then move it to unblock it gets trained twice.. not sure how to overcome that one.

Your script should disregard expunged messages. So I assume that the script is working on file level rather than using IMAP access?

The script I had in mind should scan the Kolab IMAP server for folders of type "mail.spam" and "mail.ham" than can be accessed by "anyone". It would then fetch any existing messages, pass them through the training and purge the folders.

Otherwise besides those three issues it's been smooth sailing with dspam once it was configured.

I don't know if my method is what you had in mind for spam filtering, but let me know.

Besides the stuff mentioned above the only other two things where I'd like to deviate from your approach would be:

• no MySQL db
• integrated into amavisd-new

A MySQL db on a Kolab server is a no-go because of philosophy reasons The server should stay as lean as possible.

The amavisd-new approach helps to isolate the spam handling on a single spam box that can be accessed by different servers. I still don't know if having dspam in amavisd-new will cripple it too much but I'll have a look.

Thanks for all your input!

Cheers,

Gunnar