Fighting Spam!

It's been nearly a decade since spammers and their enemies begun evolving competitively. As with the classic cheetah/gazelle model originally formulated by Darwin, each time one group becomes a little faster or more agile, its adversaries develop traits for outwitting and outrunning it.

In addition to wasting people's time with unwanted e-mail, spam also eats up a lot of network bandwidth. Consequently, there are many organizations, as well as individuals, who have taken it upon themselves to fight spam with a variety of techniques. But because the Internet is public, there is really little that can be done to prevent spam, just as it is impossible to prevent junk mail.

Nobody wants it or ever asks for it. No one ever eats it; it is the first item to be pushed to the side when eating the entree. Sometimes it is actually tasty, like 1% of junk mail that is really useful to some people.

The number of unsolicted commercial electronic messages received by the average American in 2001 was 571, according to Jupiter Media Metrix. By 2006, Jupiter says, that number will increase to 1,400, with more than 206 billion spam messages going out over the course of the year. While these numbers are notoriously difficult to calculate, every survey and ISP record points to dramatic increases in spam, sometimes as much as 300 percent year over year. One reliable indicator of the problem's magnitude is the size of the anti-spam effort. The range of tools available to ISPs, enterprises and consumers in the fight against spam grew considerably during the Web bubble. Simultaneously, heavyweight Web marketers and interactive ad players have been scrambling to distinguish their services from the bad guys, as well as to counteract growing calls for government controls on digital marketing.

In one of the biggest such moves, the Direct Marketing Association (DMA), through its subsidiary, the Association of Interactive Marketing (AIM), has released online commercial solicitation guidelines in an effort to promote high ethical standards among marketers. The rules require that members let e-mail recipients know how they can refuse future mailings and allow consumers to prevent the sale or rental of their addresses

Continue Reading...

What Is Spam?

You have probably seen an increase in the amount of junk mail which shows up in your email box, or on your favorite newsgroup. The activities of a small number of people are becoming a bigger problem for the Internet.

Chain letters that ask for money, whether for reports or just straight up, are illegal in the US whether they are in postal mail or e-mail. Report these frauds to your local US Postmaster. You may see e-mail coming from Nigeria or another African country, sent by someone who wants to use your bank account to transfer 20 million dollars. This is called a '419' scam and people have been killed over it.

Spam is flooding the Internet with many copies of the same message, in an attempt to force the message on people who would not otherwise choose to receive it. Most spam is commercial advertising, often for dubious products, get-rich-quick schemes, or quasi-legal services. Spam costs the sender very little to send -- most of the costs are paid for by the recipient or the carriers rather than by the sender. To the recipient, spam is easily recognizable. If you hired someone to read your mail and discard the spam, they would have little trouble doing it. How much do we have to do, short of AI, to automate this process? I think we will be able to solve the problem with fairly simple algorithms. In fact, I've found that you can filter present-day spam acceptably well using nothing more than a Bayesian combination of the spam probabilities of individual words. Using a slightly tweaked (as described below) Bayesian filter, we now miss less than 5 per 1000 spams, with 0 false positives.

One particularly nasty variant of email spam is sending spam to mailing lists (public or private email discussion forums.) Because many mailing lists limit activity to their subscribers, spammers will use automated tools to subscribe to as many mailing lists as possible, so that they can grab the lists of addresses, or use the mailing list as a direct target for their attacks.

Continue Reading...