UPDATE 9/1/08
I've been trying Cloudmark again though the new IHateSpam package for the past 6 months and so far have not seen any real false positives. I'm not sure why the first trial went do badly, but clearly this product has come a log way. At this point, I do not think I could live without it.
________
6/21/2006
Overview
CloadMark is a spam identification and filtering system that relies on it's users to categorize spam emails. Each time a user marks an email as spam, a compressed version of the email (the email's "signature", "hash", or "fingerprint") is sent to CloudMark's servers and databased. Once a given signature is reported my enough reliable users, it is considered spam and any matching emails in other user's inboxes are filtered.
Spam senders often purposefully send many versions of their emails to try to foil filtering attempts. One of Cloudmark's challenges is to come up with an algorithm that is general enough to recognize spams despite these alterations. The signature algorithm must at the same time be selective enough to avoid having non-spam emails generate a signature that matches to a signature generated by a spam email.
You can read more about CloudMark on there website...
Effectiveness
A spam filter's effectiveness is a measure of how many spams it successfully filters.
To test CloudMark's effectiveness, I scanned a folder filled with known spam email that had been already been missed by Mail-Filters...
| Total Known Spam emails scanned | 11,428 |
| Spams successfully found | 7,279 |
| Spams missed | 4,149 |
So, Cloudmark is very effective considering that this was a set of spams that had already fooled the Mail-Filters filter, which itself is very effective. But effectiveness is nothing without accuracy; a filter that just deleted all incoming emails would be very effective but not desirable.
False Positives
A spam filter's false positive rate is a measure of how accurate it is at distinguishing real emails from spam.
I personally think that any spam filter that produces more than a nominal number of false-positives (that is, marks any legitimate emails as spam) is useless because for many people because the cost of loosing a single real email can easily exceed the cost of processing all of the spam that filter successfully eliminated. It is for that reason that I only consider BrightMail and Mail-Filters for use on my own mail server because both have essential zero false positive rates. (Check out my evaluation here.)
To test CloudMark's false positive rate, I scanned a folder containing known-good emails (that is, emails that I had manually marked as non-spam) so see how many of these emails CloudMark would mistakenly identify as spam.
| Total known good emails scanned | 7,586 |
| False positives that included spam content* | 7 |
| "No Excuse" false positives found | 13 |
| Total false positives | 20 |
| False positives per 1 million emails | ~2,500 |
*These were legitimate emails that consisted of a forwarded spam with some extra text like "Hey josh, can't you do anything about this spam below...?" added at the top.
Unfortunately, CloudMark's signature algorithm seems to be overly aggressive in assuming that any email that contains spam is itself a spam. Worse, it seems able to occasionally completely misread a real email. The "no excuse" emails that it filtered were 100% legitimate, personal, non-newsletter, non-spam text. Some did contain some spam-suggestive words (one had the words "south beach diet" in it), but CloudMark's value proposition against content-based filters (i.e. Bayesian) is that it uses fingerprints to analyze the whole email rather than just looking for hot words.
Analysis
Since several of the false positive matches that CloudMark generated were sent only to me, there is no chance that another CloudMark user could have mistakenly flagged them as spam. Instead, I believe that CloudMark's signature generating algorithm is not specific enough and can be fooled by spam-hot words contained in legitimate emails, leading to false positives. BrightMail and MailFilters both solve this problems by using rules rather than signatures, and the rules are hand crafted by human editors. This is a significant and difficult problem for CloudMark to solve.
Conclusion
CloudMark is a very effective spam filtering solution that compares well against traditional content-based filters. Unfortunately it is not a viable option for people who can not tolerate any false positives. If you can tolerate a small number of false positives, it is significantly more effective than rule-based systems like BrightMail and MailFilters.
Note that CloudMark is a trademark and my only relationship with them is that I downloaded their trial software.