Mail Filtering

One of the largest issues with email today is the problem of Unsolicited Bulk Email (UBE). Also known as SPAM, such messages may also carry viruses and other forms of malware. According to some reports these messages make up the bulk of all email traffic on the Internet.

This section will cover integrating Amavisd-new, Spamassassin, and ClamAV with the Postfix Mail Transport Agent (MTA). Postfix can also check email validity by passing it through external content filters. These filters can sometimes determine if a message is spam without needing to process it with more resource intensive applications. Two common filters are dkim-filter and python-policyd-spf.

This is how the pieces fit together:

For example, if a message has a Spam score of over fifty the message could be automatically dropped from the queue without the recipient ever having to be bothered. Another, way to handle flagged messages is to deliver them to the Mail User Agent (MUA) allowing the user to deal with the message as they see fit.

Installation

See the section called “Postfix” for instructions on installing and configuring Postfix.

To install the rest of the applications enter the following from a terminal prompt:

sudo apt-get install amavisd-new
sudo apt-get install spamassassin
sudo apt-get install clamav-daemon
sudo apt-get install dkim-filter
sudo apt-get install python-policyd-spf

There are some optional packages that integrate with Spamassassin for better spam detection:

sudo apt-get install pyzor razor

Along with the main filtering applications compression utilities are needed to process some email attachements:

sudo apt-get install arj cabextract cpio lha nomarch pax rar unrar unzip unzoo zip zoo

Configuration

Now configure everything to work together and filter email.

ClamAV

The default behaviour of ClamAV will fit our needs. For more Clamav configuration options, check the configuration files in /etc/clamav.

Add the clamav user to the amavis group in order for Amavisd-new to have the appropriate access to scan files:

sudo adduser clamav amavis

Spamassassin

Spamasssassin automatically detects optional components and will use them if they are present. This means that there is no need to configure pyzor and razor.

Edit /etc/default/spamassassin to activate the Spamassassin daemon. Change ENABLED=0 to:

ENABLED=1

Now start the daemon:

sudo /etc/init.d/spamassassin start

Amavisd-new

First activate spam and antivirus detection in Amavisd-new by editing /etc/amavis/conf.d/15-content_filter_mode:

use strict;

# You can modify this file to re-enable SPAM checking through spamassassin
# and to re-enable antivirus checking.

#
# Default antivirus checking mode
# Uncomment the two lines below to enable it
#

@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);


#
# Default SPAM checking mode
# Uncomment the two lines below to enable it
#

@bypass_spam_checks_maps = (
   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);

1;  # insure a defined return

Bouncing spam can be a bad idea as the return address is often faked. Consider editing /etc/amavis/conf.d/20-debian_defaults to set $final_spam_destiny to D_DISCARD rather than D_BOUNCE, as follows:

$final_spam_destiny       = D_DISCARD;

After configuration Amavisd-new needs to be restarted:

sudo /etc/init.d/amavis restart

Postfix

For Postfix integration, enter the following from a terminal prompt:

sudo postconf -e 'content_filter = smtp-amavis:[127.0.0.1]:10024'

Next edit /etc/postfix/master.cf and add the following to the end of the file:

smtp-amavis     unix                                2       smtp
        -o smtp_data_done_timeout=1200
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes
        -o max_use=20

127.0.0.1:10025 inet    n                                   smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_delay_reject=no
        -o smtpd_client_restrictions=permit_mynetworks,reject
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o smtpd_data_restrictions=reject_unauth_pipelining
        -o smtpd_end_of_data_restrictions=
        -o mynetworks=127.0.0.0/8
        -o smtpd_error_sleep_time=0
        -o smtpd_soft_error_limit=1001
        -o smtpd_hard_error_limit=1000
        -o smtpd_client_connection_count_limit=0
        -o smtpd_client_connection_rate_limit=0
        -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

Also add the following two lines immediately below the "pickup" transport service:

         -o content_filter=
         -o receive_override_options=no_header_body_checks

This will prevent messages that are generated to report on spam from being classified as spam.

Now restart Postfix:

sudo /etc/init.d/postfix restart

Content filtering with spam and virus detection is now enabled.

Testing

First, test that the Amavisd-new SMTP is listening:

telnet localhost 10024
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 [127.0.0.1] ESMTP amavisd-new service ready
^]

In the Header of messages that go through the content filter you should see:

X-Spam-Level: 
X-Virus-Scanned: Debian amavisd-new at example.com
X-Spam-Status: No, hits=-2.3 tagged_above=-1000.0 required=5.0 tests=AWL, BAYES_00
X-Spam-Level: 
[Note]

Your output will vary, but the important thing is that there are X-Virus-Scanned and X-Spam-Status entries.

Troubleshooting

The best way to figure out why something is going wrong is to check the log files.

  • For instructions on Postfix logging see the the section called “Troubleshooting” section.

  • Amavisd-new uses Syslog to send messages to /var/log/mail.log. The amount of detail can be increased by adding the $log_level option to /etc/amavis/conf.d/50-user, and setting the value from 1 to 5.

    $log_level = 2;
    
    [Note]

    When the Amavisd-new log output is increased Spamassassin log ouput is also increased.

  • The ClamAV log level can be increased by editing /etc/clamav/clamd.conf and setting the following option:

    LogVerbose true
    

    By default ClamAV will send log messages to /var/log/clamav/clamav.log.

[Note]

After changing an applications log settings remember to restart the service for the new settings to take affect. Also, once the issue you are troubleshooting is resolved it is a good idea to change the log settings back to normal.

References

For more information on filtering mail see the following links:

Also, feel free to ask questions in the #ubuntu-server IRC channel on freenode.