Why do complaint rates vary across ISPs?

By George Bilbrey
President

Complaint rates are a major driver of anti-spam systems. A high complaint rate (number of "this is spam" reports relative to messages in the inbox) is known to drive bad delivery rates for commercial mailers.

One of the more interesting problems that I've run across at Return Path is trying to figure out why complaint rates for the same IP address (or domain) vary so widely across different ISPs. Here is a scatter chart of complaint rates for ~2,500 commercial email marketing IPs across two different ISPs.  In this graph 0.05 = 5% complaint rate (5 messages per 100 places in the inbox).

What is apparent is:

1. Most IPs in this group have "lowish" complaint rates (<1%)
   
2. There isn't a ton of correlation outside of that range.
   

So what might cause the variation in complaint rates? A partial list would include:

• Differences in webmail user interface. Some ISP make it easier than others to hit "this is spam." Some ISPs allow users to report all messages in the junk mail folder as spam at the same time. However, this would seem to be more of a systematic difference that would cause the average complaint rates to be different at the two ISPs. I wouldn't necessarily expect that to drive a poor correlation.
  
• Users at different ISPs have a different propensity to hit the "report as spam" button for different classes of mail.
  
• Senders of email messages behave differently at different receiving domains - there appears to be a few cases of this in this sample.
  
• Availability and use of feedback loops at different ISPs. In our analysis of the above IP addresses, this drives a large part of the variance for these IP addresses.
  

What does this mean practically for mailers and designers of anti-spam systems?

For anti-spam systems, there may be some value for gathering complaint data from variety of sources. This would allow you to catch IPs that are behaving well locally (in your network) but not behaving well outside of your network.

For commercial mailers, there are a few key "so whats":

• Sign up for feedback loops where available. These are useful for diagnosing the cause of complaint issues.
  
• Monitor complaint rates at each ISP. You can have a problem at one ISP and be doing well at another.
  

What do you think?