Brightmail versus Mail-Filters for accurate spam filtering

12/23/2005

I've had the same email address for more than a decade. My email address is also used by pretty much every person named Josh in the world when they fill out a web form. I get a lot of spam, often thousands of spams per day. For me a good spam filter is a necessity.

 I've been using BrightMail for the past few years and have been happy with it. My contract was expiring so I thought this was good opportunity to check out the competition.

Technology

Brightmail and Mail-filters spam filtering systems are unique in that they rely on human editors to craft rules targeted toward individual spam attacks in real-time. Neither product requires any training; the rules are automatically downloaded from servers hosted on the internet several times an hour.

The rules are very specific so false-positives are almost eliminated when compared to more general rules-based systems. For me, this is very important because I get way to many spams to be able to ever go back through my Junk Email folder looking for false-positives.

There is no other technology that offers similar effectiveness rates without needing continuous training and without incurring costly false positives.

Effectiveness

Both systems are effective, but Brightmail is more effective than Mail-filters.

It is interesting to note that there was significant non-overlap between the two systems. If money were no object, the most effective solution would be run incoming emails through both systems in series. This setup would have yielded a cumulative effectiveness of more than 90% in this test - about 6% more effective than running Brightmail alone and about 10% more effective than running Mail-Filters alone.

Note that my email account is pretty close to a worse-case for evaluating spam filter effectiveness. I get spam attacks that no email address less than 5 years old would likely ever see. My experience with running both these filters on "normal" email accounts is effectiveness rates in the middle to high 90%'s.

False-positives

Both Brightmail and Mail-Filters have extremely low false-positive rates.

The false-positive generated by Mail-Filters was a spam that a friend had forwarded to me prefixed with a little note asking "How can I get rid of these?".

Installation & Operation

Mail-Filters' setup is much simpler and more elegant than Brightmail's.

A Brightmail installation is complex. It requires Microsoft IIS and MS SMTP services to run, and installs its own copies of Apache Tomcat and MySQL. The MS SMTP service acts as a full SMTP server and internally queues incoming messages and attempts to send bounce messages. This makes it heavy and fragile. There have been about a half dozen times in the past few years when I noticed that no external mail was coming though to my account.  Restarting the Brightmail server machine fixed the problem. The MS SMTP server was continuing to queue incoming messages even though it was not passing them through so no email was lost, but this is a little scary.

Mail-Filters runs as an SMTP proxy in front of your normal SMTP server. After installing the program files and setting up the XML config file, you can start the service and begin filtering emails. The Mail-Filters software is also capable of running as a POP3 proxy, which might be handy in some situations.

Note that both products can run under Linux and Brightmail can integrate with Sendmail. I did not evaluate these configurations. I'd expect both products to perform similarly under Linux, although the Brightmail installation on Linux would benefit by not needing IIS/MS SMTP.

Pricing

For large numbers of users, Brightmail is much more expensive than Mail-Filters, as much as 10x as expensive per user per year. For installations with less than about 100 users, Brightmail is less expensive per user.

Keep in mind that Brightmail on Windows needs IIS and MS SMTP, so it will also require a server operating system like Windows 2003 which could increase total installation cost.

Conclusions

Both products do an excellent job of filtering out spams while passing legitimate emails. There are no other filtering products that can deliver similar effectiveness rates without incurring false positives.

Brightmail is the gold-standard choice and it processes a large fraction of all the email in the world. If you are willing to pay the high per user price and can accommodate its system requirements, you'll enjoy its slightly higher effectiveness rates. If you have a small number of users, then the price per user is less of an issue making Brightmail more attractive. Additionally, if you already need to have an IIS and MS SMTP server running (probably because you are using MS Exchange), then Brightmail is also a more attractive choice.

Mail-Filters is relatively new and small compared to Brightmail, but they are in the same ball park as in terms of effectiveness and false-positive rates. Additionally, their software is much lighter and more flexible than Brightmail. If you have a large number of users and want a simple solution, then Mail-filters is an economical and effective choice.

Methodology

I ran the test from 9/19/2005-12/02/2005 using my personal email account in the josh.com domain.

I used Brightmail for Windows version 6.0.2. The Brightmail spam threshold was set to "72".

I used Mail-Filters SpamCure Server release 20050720.

Both systems were configured to add an X-HEADER to messages they determined were spam. Messages were processed by the Brightmail server first, then the Mail-Filters server. I verified with Mail-Filters that the presence of additional X-HEADERs (like the Brightmail one) would not effect their ability to detect spams. I think the fact that there was a large non-overlap of the systems proved this this to be true.

Faced with a corpus of 70,000 spams, finding false-positives was a challenge. It was not practical to go though the emails manually looking for false-positives. Instead, I first wrote a little program to find emails from/to people I know. I also trained a Bayesian filter on my known-good email corpus and had it score the filtered spams looking for false positives.

###