The Evolution Spam Filter

From Computer Tyme Support Wiki

(Difference between revisions)
Jump to: navigation, search
(Foreign Languages)
(Licensing)
Line 171: Line 171:
In retaining a patent I can have some control over the IP and I can get a little something for coming up with the idea. Not looking to be greedy about it but not wanting to put myself out of business either. I'm looking to see my income model changing to licensing technology. This is the most advanced filter on the planet. It's worth licensing.
In retaining a patent I can have some control over the IP and I can get a little something for coming up with the idea. Not looking to be greedy about it but not wanting to put myself out of business either. I'm looking to see my income model changing to licensing technology. This is the most advanced filter on the planet. It's worth licensing.
 +
 +
= How to Implement this Spam Filtering Method =

Revision as of 02:56, 30 January 2016

Contents

The Evolution Filter

The last big advancement in spam filtering was done by Paul Graham - A Plan for Spam. He was the first to apply Bayesian Filtering to blocking email back in 2002. Since then not a lot has happened to make spam filtering significantly better, till now. The Evolution Filter is a new plan for spam.

I chose the name "Evolution Filter" because it's a very simple and fast recursive learner that mimics evolution in software. While most neural networks are complex AI, this one is dirt simple but extremely fast. It seems that simple and fast is the key. In 6 months it might become self aware. :)

Overview of How it Works

Have you ever looked at a list of email messages and get the feeling that you can classify 70% of them just seeing the sender name and the subject line? Have you ever wondered how it is so easy to recognize ham and spam yet computer can't seem to figure out the obvious. When you look at the list - how is it that you can tell? What's going through your mind?

The way you recognize spam and ham is that when you see a subject line that looks similar to good email and dissimilar to spam then it's good email. And if it looks similar to spam because it says stuff that good email never says, it's spam.

For example, the subject line is "let's get some dinner". We know the message is good because spammers never say that. So if the subject is something you've seen before in good email and something that spammers never say - it's good email.

In fact - that's how the Evolution filter works. If, for example, the subject has words and phrases that match good email but spammers never say then the message is good. If the subject has words and phrases that match messages that spammers have used, but never seen in good email, then it's spam.

But, you might ask, where do I get a list of words and phrases that spammers never say? It's easier than you think. What I do is create a set of every word and phrase spammers do say and test to see if it's NOT on the list. In other words, I store all the words and phrases that are said in ham, and all the words and phrases that are said in spam. If the test message matches ham and doesn't match spam, it's ham. And if it matches spam and doesn't match ham, it's spam.

So - what do I mean by words and phrases? I take the subject and I break it down.

"the quick brown fox jumps over the lazy dog"

becomes

"the" "quick" "the quick" "brown" "quick brown" "the quick brown" "fox" "brown fox" "quick brown fox" 
"the quick brown    fox" "jumps" "fox jumps" "brown fox jumps" "quick brown fox jumps" "over" "jumps over"
"fox jumps over" "brown fox jumps over" "the" "over the" "jumps over the" "fox jumps over the"
"lazy" "the lazy" "over the lazy" "jumps over the lazy" "dog" "lazy dog" "the lazy dog" "over the lazy dog"

I'm using a database called Redis. Redis lives in ram and it's very fast. It is extremely good at set comparisons. So I can create a spam corpus and a ham corpus that contain million of words and phrases and I can break the subject down into hundreds of fingerprints and compare it to both sets and see what matches and not matches to get a result. The furmula is as follows.

card(Test_message intersect Spam diff Ham) - card(Test_message intersect Ham diff Spam)

This is a simplification of the concept. In real life I'm not just testing the subject. I'm testing the name part of the from address, the attachment file names and extensions, php scripts, parts of the message body, the header structure, and behavior of the sender. All these produce scores that are combined into a result that is far more accurate than any method of sorting email ever developed.

The New Concept

The way the Evolution Spam Filter works is literally thinking outside the box. Every other filter is about matching something. One box has ham, one box has spam. Bayesian filters match the message with the two boxes to see which one is most similar. Or the filter is matching rules. Does it contain "Russian Brides"? If so - add penalty points.

the other filters are all about matching. The Evolution Filter is about NOT MATCHING. It's all about what the other side never says. While other filters match what's inside the box, the Evolution Filter matches what's outside the box. Instead of matching finite sets of known information, we are matching to the infinite set of unknown information. And we all know that an infinite set is always bigger than a finite set. How much bigger? Infinitely bigger! That's why it works so well.

Example of how NOT matching works

Let’s take 2 subject lines and see how this works.

“Meet hot Russian Brides Online!”
“I read an article about Russian Brides in a magazine”

A traditional spam filter using Bayesian or hard coded rules about “Russian Brides” might determine that only 1 out of 500 emails mentioning the phrase “Russian Brides” is a good email. Thus the second line would have points assessed against it in the classification process using these traditional methods.

Using the Evolution Filter the phrase “Russian Brides” is in both sets and therefore has no influence on the results. But the first subject matches these phrases in the Spam Only set.

“Meet hot”
“Meet hot Russian”
“Meet hot Russian Brides”
“hot Russian Brides Online!”
“Russian Brides Online!”
“Brides Online!”
“Online!”

The second subject matches these phrases on the ham only set that are never used on the spam set.

“I read an article”
“read an article”
“read an article about”
“about Russian”
“an article about”
“in a magazine”
“Brides in a”

So even though the phrase “Russian Brides” has no influence each subject hits either ham or spam many times where the same phrase was never used in the subject line in the opposite set. And the number of hits is significant enough just from these subjects to cause the fingerprints to be learned, and that’s just looking at the Subject attribute. When this is combined with testing all attributes the messages usually come out strongly on one side or the other.

In rule based systems one would not normally build a white list rule to to allocate points based on seeing the phrase “read an article about”. That’s where the Evolution Filter is different. It didn’t need to have that rule because since it is comparing to the infinite set of what is not matched on the other side, it dynamically create billions of rules automatically.

Learning by Association

For example, suppose I filter email for a machine dealer who sells “Machine A”. An email comes in from a trusted source and “Machine A” is learned as ham. And in millions of spams no one has ever mentioned “Machine A”. Then one second later someone else we filter for mentions “Machine A”. They are classified as ham based on that one match. And if there are several matches on the ham side that are not matched on the spam side then the message fingerprints can be added to the ham sets.

Because the comparison method is based on matching one set and not matching the other the learning feedback system is a lot faster and has different characteristics than a traditional Bayesian filter.

In my previous example, “Machine A” has been learned as ham and never seen in spam. Someone sends an email inquiring about “Machine A” and “Machine B”. Because “Machine B” was never ever used in a spam then “Machine B” also becomes a blessed phrase. Anyone who uses “Machine B” in their email is passed as ham. (Unless spammers start spamming about “Machine B” which would revert it to neutral.)

Once a few phrases in one email message are matched to a new email then all the fingerprints of the new email are learned as ham. And the new fingerprints that are not already in the ham sets and are not in the spam sets become effectively new rules for identifying ham. The system learns how you talk, what you are interested in, and people in your life that are interested in the same things have their email passed and learned. Then their friends interact with them and the learning continues.

Consider this example, Email Subject Only. Brackets enclose a phrase learned. Parens enclose the match phrase from previous email learned.

“Do you want to get some [lunch today]?” 
“Going to (lunch today) and [to see a movie] afterwards” 
“I want (to see a movie) about time travel, [are you interested]?”
“(Are you interested) in [getting together] after work at my place?”
“If more people were (getting together) to make a better world we would have [less poverty.]”
“[Better education] leads to (less poverty.)”

Even though the subject wanders the matches keep feeding the learner because fingerprints are learned on one side the do not match fingerprints on the other side. In reality the subjects in the above example might produce 5 to 10 fingerprints each that were never used in spam.

People who communicate by email usually have some sort of relationship and they talk about things they have in common. It’s the things that they have in common that causes not only the email to pass, but the differences in the messages to be learned as well.

Similarly on the spam side, there are only so many ways you can misspell Viagra, and the first time it catches a message with it deliberately misspelled then that spelling is learned and every spam that misspells Viagra the same way is caught. Traditional rule based system encourages spammers to misspell words so they don’t match the rule. With the Evolution classifier the misspelling is what gives them up because people who send good email never misspell, for example, Viagra as Viiaggra.

Spammers also want you to do something. it’s a business model that there are just so many scams out there and so many ways to describe these scams. So as the system learns these phrases that only spammers use then it’s easy to detect new scams based on older but similar scams.

As the recursive learning continues it separates out what is essentially 2 different cultures and languages. Things that only people who send good email talk about and things that only spammers talk about. As these sets grow the accuracy increases and less and less messages go unrecognized.

Example Data

To illustrate this concept of NOT matching I present the following lists. It would take 9 million subject rules to duplicate this. The Evolution Filter writes and scores its own rules.

Here is a list of 100,000 words and phrases used in the subject line of HAM and never seen in the subject line of SPAM.

Here is a list of 100,000 words and phrases used in the subject line of SPAM and never seen in the subject line of HAM.

Foreign Languages

One of the advantages of the Evolution Filter is that it makes its own rules. It doesn't need to know anything about the language a message is written in. As long as it has enough samples it can figure out on its own if something is spam or ham. It can be French, Spanish, German, Russian, or even Klingon spam. It can actively detect good email and spam just through the learning process.

Examples of Klingon spam:

Discount Ferengi Tooth Sharpeners
Get your deceased relative into Stovokor Now!
Orion slave girls waiting to meet you!
Are you a descendant of the House of Kahless?
Enhance your Manhood! Romulan Pharmacy Online!
Bottom Price Risa vacation, Jamaharon included!!

How Well does the Evolution Filter Work?

Our spam filtering system has always worked very well. Unlike most filtering system that focused on blocking spam and what was left over was good, we also focused on ways to positively identify good email. In this business if a few spams get through it's not a big deal. But when good email is blocked it's far worse. So we always looked for ways not just to block spam, but to look for ham. We wanted to be able to positively identify good email.

Good email was harder to identify but it can be done to a limited extent and we could classify about half of all good email as good with very little spam sneaking through. But half isn't good enough.

The strong point of the Evolution Filter is that it is very good at positively identifying good email. It is also better at identifying spam than any other method but when it comes to protecting good email, that's where it really shines. The accuracy rate is so good it's scary. This doesn't just block spam, it decimates spam. If this was widely adopted many spammers would get out of the business because it will cut their delivery down so much that it won't be profitable for them anymore.

The thing about spammers is that spammers always want you to do something. So they have to make the sales pitch in the message to convince you to do what they want you to do. They can't get away from that. This system learns all the scams and all the different ways to describe the same things and it can identify them. Spammers may be able to fake a few things but that can't fake it all. Unlike rule based systems like SpamAssassin that spammers can download and test their rules, the Evolution system create millions of rules on its own and the spammers have no idea what to match because it's a not-matching system.

So what I'm seeing is this. less spam is getting through and less good email is getting blocked. But another factor is that the confidence we have in passing good email and blocking bad email has greatly increased. We aren't passing as much simply because it has no score. We are seeing about a 95% rate of positively identifying good email as good. On the spam side we were already over 99% positively identified as spam but the scoring is even stronger now and it's picking up more of the ones we weren't sure about and classifying them.

At first I had these systems running in parallel with the idea that I would use the old system to identify problem in the new system. It started working the other way. The new system was finding problems in the old system. This has allowed me to eliminate rules that I thought were better than they really are.

The new system isn't something that replaces the old but is built on it. Since the new system is really just a simple AI, it really has no concept by itself as to what is good email and what isn't. The old system has the ability to positively identify most email as good or bad with high confidence. This high confidence email if fed directly into the learner and creates a "moral compass" as to the difference between good and evil. From there the Evolution Filter figures out the messages that are in the middle. That which hasn't been able to be classified by other methods.

The Patent

You might be wondering - why patent this? The simple answer is that if I make a huge leap in spam filtering technology that saves the world trillions of dollars, my reward shouldn't be to put myself out of business.

This filtering method is the most accurate method on the planet and the most resilient to being defeated by spammers. No one else has done this because if they had done it - everyone would be using it.

My plan is to make it free to most everyone and charge a reasonable license fee to the big providers and my competitors. But it will be highly profitable for my competitors to license it from me as the gains in business and customer satisfaction would be far less than the cost of the license.

Here are the details on my provisional patent:

Licensing

Right now this is Patent Pending. The proposed pricing is as follows:

For large email providers. Those with over 10,000 email accounts. 3 cents/eccount/year or 15 cents/email account/lifetime license.

For spam filtering providers 1/10 of 1% of your gross revenue.

And - of course - all is negotiable. If your situation is unique, we'll find a way to make a deal.

I'm planing on using the Creative Commons Model Patent License with a few restrictions.

Some people said I should just keep this method a secret and just get more business by having the best spam filter on the planet. But I feel that by releasing this method to the public that it will make trillions of dollars different in the world economy saving billions of people from being scammed or at least wasting hours deleting junk email. And I also really really hate spam. This new method is a game changer and if widely adopted can actually put spammers out of business. Their delivery rate will be so low that it won't be profitable anymore.

In retaining a patent I can have some control over the IP and I can get a little something for coming up with the idea. Not looking to be greedy about it but not wanting to put myself out of business either. I'm looking to see my income model changing to licensing technology. This is the most advanced filter on the planet. It's worth licensing.

How to Implement this Spam Filtering Method

Personal tools