EVOLUTION-MANAGER

Edit File: RELEASE_NOTES

--------------------------------------------------------------------------- October 9, 2018 amavisd-new-2.11.1 release notes - removed a trailing dot element from @INC, as a workaround for a perl vulnerability CVE-2016-1238; - amavis-services: bumping up syslog level from LOG_NOTICE to LOG_ERR for a message "PID <pid> went away", and removed redundant newlines from some log messages; - safe_decode() and safe_decode_utf8(): avoid warning messages "Use of uninitialized value in subroutine entry" in Encode::MIME::Header when the $check argument is undefined; - @sa_userconf_maps has been extended to allow loading of per-recipient (or per- policy bank, or global) SpamAssassin configuration set from LDAP. For consistency with SQL a @sa_userconf_maps entry prefixed with 'ldap:' will load SpamAssassin configuration set using the load_scoreonly_ldap() method; a patch by Atanas Karashenski; - add some Sanesecurity.Foxhole false positives to the default list @virus_name_to_spam_score_maps; - updated some comments; --------------------------------------------------------------------------- April 26, 2016 amavisd-new-2.11.0 release notes Contents: DEPRECATION NOTICE COMPATIBILITY BUG FIXES NEW FEATURES OTHER SUPERVISED PROCESS NOTES DEPRECATION NOTICE - The old DomainKey signatures (a predecessor to DKIM) has been published as a historic document RFC 4870 and obsoleted by RFC 4871 in May 2007; Support for DomainKey signatures is likely to be removed with a next version of amavisd. - It is expected that the next release of amavisd will start using some of the features made available with perl 5.10.0 (such as a defined-or operator, or a possessive quantifier in regular expressions), so consider running amavisd under perl 5.8.9 or earlier as unsupported. In practice, using such old version of perl is problematic even now, as their support for Unicode / UTF-8 is incomplete and unreliable. COMPATIBILITY There are some minor incompatibilities between versions 2.10.1 and 2.11.0: - During startup more detailed testing is performed for taint bugs of a module Encode and the function utf8::is_utf8(), which may produce warnings on old versions of perl with its old core module Encode, or may exit on detecting more sinister bugs in these modules. Note that the module Encode may be upgraded independently of perl, if desired; - with MySQL: changed character set 'utf8' to 'utf8mb4' for fields msgs.subject and msgs.from_addr, as previously some of the UTF-8 characters could not be stored in a database; - when logging to stderr a timestamp prefix to each message is only still inserted if $DEBUG is true. When $DEBUG is false each message is prefixed with a syslog log level in angle brackets, and a timestamp is omitted (for compatibility with systemd); - a perl module Digest::SHA is now a required module. It is a perl core module since perl 5.10, so it shouldn't introduce a new dependency, and it was a de-facto required module even previously, as it was needed for DKIM processing; BUG FIXES - delivery method was undefined when always_bcc was used; reported by Marieke Janssen; - avoid warnings issued by perl 5.21.7 and later: Negative repeat count does nothing at ./amavisd line 16408 and similarly in amavisd-status; - releasing from an SQL quarantine failed to provide the original envelope sender address to a released message; reported, and a fix suggested by Tom Johnson and Tobias; - remove a stale database file __db.nanny.db on a reload or restart, as it can prevent a successful start when a previous start failed for some reason; a patch by Trent Lloyd; NEW FEATURES - Polished rough corners to facilitate running amavisd as a non-daemonized supervised process, e.g. under systemd: * make it possible/easier to disable use of a pid_file; * send status notifications to systemd when a NOTIFY_SOCKET environment variable is provided; * improved logging to stderr when $do_syslog and $logfile are undefined (although logging through syslog might still be preferred, as writing to a shared pipe from multiple child processes only guarantees atomicity of writes shorter than PIPE_BUF, which is typically 512 bytes on *BSD, and 4096 bytes on Linux systems); See below for a sample amavisd.service file. - A log template macro 'report_json' can now take arguments, which can include or exclude fields (key/values) from the JSON report object. Arguments to a macro are either field names (keys) to be included in a report, or are field names to be excluded, each prefixed with an exclamation mark, to produce a report with all but excluded fields. Field names are case-sensitive. The order of fields in a serialized JSON object is unaffected by the order of field names in a filter. Unknown or non-present field names in a filter are silently ignored. Example: [:report_json|mail_id|action|content_type|queued_as|mail_from|size] or: [:report_json|!recipients|!elapsed|!os_fp|!subject|!subject_rot13] For better clarity, instead of listing field names as individual arguments to a macro, it is also possible to provide a single argument to a macro, in which field names are separated by whitespace: [:report_json|mail_id action content_type queued_as mail_from size] or: [:report_json| !message !recipients !to_addr !elapsed !os_fp !subject !subject_rot13 !user_agent !tests !tests_ham !tests_spam] As an example, a setting in a config file may look like: $log_templ = '[:report_json|mail_id action queued_as mail_from]'; If at least one field name has an exclamation mark (i.e. is to be excluded), all but excluded fields are implied, so any field names without an exclamation mark are redundant. Currently this is a simple filter where subfields of a structured object cannot be selectively filtered (e.g. elapsed.SpamCheck). For finer control on JSON content use some external JSON-processing utility. Based on a patch by Markus Benning. - Two new configuration settings are added: %smtpd_tls_server_options and %smtp_tls_client_options. These two associative arrays are passed to IO::Socket::SSL->start_SSL when establishing a server-side or a client-side TLS session with an MTA, and provide more control over a TLS session - like providing certificates and restricting ciphers. See documentation of a perl module IO::Socket::SSL for a list of all options with their descriptions and their defaults. When TLS is in use, it is recommended to stick to fresh versions of the module IO::Socket::SSL and the underlying ssl library, as it can provide a safer set of defaults (e.g. excluded SSLv2). Existing config options $smtpd_tls_cert_file and $smtpd_tls_key_file are now deprecated in favour of a more generic %smtpd_tls_server_options. Preferably set fields 'SSL_key_file' and 'SSL_cert_file' directly in %smtpd_tls_server_options instead. For compatibility with 2.10 the values of $smtpd_tls_cert_file and $smtpd_tls_key_file are fed into the associative array %smtpd_tls_server_options if fields 'SSL_key_file' and 'SSL_cert_file' are not provided (do not exist) there. Example: %smtp_tls_client_options = ( SSL_verifycn_scheme => 'smtp', SSL_version => '!SSLv2,!SSLv3', SSL_cipher_list => 'HIGH:!MD5:!DSS:!aNULL', # SSL_client_ca_file => ... , ); %smtpd_tls_server_options = ( SSL_verifycn_scheme => 'smtp', SSL_session_cache => 2, SSL_key_file => "$MYHOME/cert/amavisd-key.pem", SSL_cert_file => "$MYHOME/cert/amavisd-cert.pem", SSL_dh_file => "$MYHOME/cert/amavisd-dh.dat", # SSL_ca_file => ... , SSL_version => '!SSLv2,!SSLv3', SSL_cipher_list => 'HIGH:!MD5:!DSS:!aNULL', ); Or just to change some field and leave the rest at their default: $smtp_tls_client_options{SSL_verify_mode} = 0; # SSL_VERIFY_NONE Suggested by Marc Grooz and Patrick Ben Koetter, based on a patch by Markus Benning. - Supports receiving SMTP/LMTP connections through a HAProxy, recognizing 'PROXY protocol Version 1' data on the first line read, after a connection from HAProxy to amavisd has been established. Connection data (IP addresses and ports) received via this protocol end up replacing such data in the the Amavis::In::Connection object ($conn). Set configuration variable $haproxy_target_enabled (also a member of policy banks) to true in order to enable this protocol. - redis: allow a scoped / link-local IP address specification (avoiding current limitation in IO::Socket::IP [rt.cpan.org #89608]); - the Amavis::Unpackers::Part::digest method now holds a digest (SHA1, hex) of a decoded (base64 or quoted-printable) MIME part contents, followed by a colon and a lowercased Content-Type of the MIME part. Canonical line endings CRLF in decoded textual parts are normalized to a native newline (\n) before feeding them to a digest algorithm. These digests are passed to SpamAssassin through a 'mimepart_digests' supplementary attribute, and are available to custom hooks. As of version SpamAssassin 3.4.1, these are used as additional tokens in a Bayes plugin. Even though SpamAssassin is capable of computing the same or similar digests on its own, the advantage of computing them in amavisd is that they reflect all and completely unmodified and untruncated MIME parts of a mail message, including non-textual attachments. For debugging, search the log for "mimepart digest: ", logged at log level 5, and ".* Content-Type: .*, size:" at log level 2. Based on a suggestion by Andreas Schulze back in 2014. A configuration setting $mail_part_digest_algorithm was added, which chooses an algorithm name for generating digests of decoded MIME parts of a message. The value is an algorithm name as accepted by Digest::SHA->new(), e.g. 'sha1' or 'SHA-1' or 'SHA-256' or 'sha256', or a string 'MD5' (case-insensitive) which chooses the MD5 algorithm as implemented by a module Digest::MD5. An undefined value disables generating digests of MIME parts. The $mail_part_digest_algorithm setting is a dynamic setting, i.e. it is a member of policy banks. For compatibility with SpamAssassin the chosen algorithm should be SHA1 (which is a default), otherwise bayes tokens won't match those generated by sa-learn (which is typically used for off-line learning). Bayes auto-learning in SpamAssassin is unaffected by a mismatch of the algorithm, as it believes digests received from amavisd. - Policy bank names in a @client_ipaddr_policy setting can now accept a comma-separated list of policy names to be loaded on a match (for loading of policy banks based on an IP address of a SMTP client). Whitespace around each policy name is allowed and is stripped. Previously only a single policy bank name was allowed in each entry of @client_ipaddr_policy. This makes it consistent with loading of policy banks based on a DKIM-based setting @author_to_policy_bank_maps, and on virus checker results via the @virus_name_to_policy_bank_maps setting. - Experimental feature: IP lookups (as implemented by lookup_ip_acl() and used by @client_ipaddr_policy) can now also do DNS-based lookups, in addition to array- and hash- based lookups. Suggested by Patrick Ben Koetter and loosely based on his patch. DNS lookups follow RFC 5782 conventions (DNS Blacklists and Whitelists: DNSBL, DNSWL, collectively known as DNSxL). A DNS query of a type 'A' is performed on a reversed IP address prepended to a specified domain name (zone name). RFC 5782 suggests that only type-A resource records of a DNS reply in an address range 127.0.0.0/8 may be considered. For example, given a zone name 'rbl.example.org' and a SMTP client's IP address 198.51.100.12, a DNS type-A query for a domain name "12.100.51.198.rbl.example.org" would be sent to a specified or to a default DNS resolver or server. Similarly, an IP address 2001:db8::2:f would produce a DNS type-A query for a domain name "f.0.0.0.2.0.0.0.0. 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.rbl.example.org" . The setting @client_ipaddr_policy contains a list of pairs, each pair consisting of a lookup object (arrayref or hashref, or now also an Amavis::Lookup::DNSxL object), followed by a policy bank name (which is a string: one or more policy bank names, comma-separated). The object constructor Amavis::Lookup::DNSxL->new accepts as its arguments: a dns zone name, expected result(s) for a match, and a resolver object. Only the first argument (a DNSxL zone name) is required, the remaining two arguments are optional. A default expected result is '127.0.0.2', and a default Net::DNS::Resolver persistent object is provided implicitly if not provided by a caller (it reads a DNS resolver's IP address from /etc/resolv.conf). The "expected result(s) for a match" argument (the second argument) is compared to the address found in a DNS reply (in a 127.0.0.0/8 range). It can be: a) an integer between 0 and 255 (or a string representing such integer), which is used to match the last byte on the 127.0.0.x quad; b) a string in a dotted-quad form of an IPv4 address in a 127.0.0.0/8 range, where leading bytes may be omitted (e.g. '1.8' == '127.0.1.8'); c) a reference to an array consisting of entries in an (a) or (b) form, where a match with any of the array elements suffices for a match; d) a perl regular expression object (e.g. qr{^127\.[3-8]\.0\.\d*$} ). If an IP address in a DNS reply matches the provided "expected result" argument, the policy banks associated with that entry are loaded, and a search through a @client_ipaddr_policy list stops. As a shorthand a subroutine Amavis::Conf::q_dns_a() is provided, which is just a convenient wrapper for Amavis::Lookup::DNSxL->new(). Example: @client_ipaddr_policy = ( [qw( 0.0.0.0/8 127.0.0.0/8 [::] [::1] )] => 'MYNETS, LOCALHOST', [qw( 169.254.0.0/16 [fe80::]/10 )] => 'MYNETS, LINKLOCAL', [qw( 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 )] => 'MYNETS, PRIVATENET', \@mynetworks => 'MYNETS', q_dns_a('rbl.example.org') => 'MY-CUSTOMER-A', # 127.0.0.2 q_dns_a('rbl.example.org', 3) => 'MYNETS,MY-CUSTOMER-B', # 127.0.0.3 q_dns_a('rbl.example.org', '0.2.99') => 'MY-CUSTOMER-C', # 127.0.2.99 q_dns_a('rbl.example.org', '127.0.0.7') => 'MY-CUSTOMER-D', # 127.0.0.7 q_dns_a('rbl.example.org', qr/^127\.1\.\d+\.2\d*\z/) =>'X', # 127.1.*.2* q_dns_a('rbl.example.org', '192.0.2.0.2')=>'never matches', # not in 127/8 ); Below is an example of an amavisd.conf section with an explicitly provided Net::DNS::Resolver object, which offers finer control over its settings: use Net::DNS; my $dnsxl_res = Net::DNS::Resolver->new( config_file => '/etc/resolv.conf', port => 5333, retry => 1, persistent_udp => 1, tcp_timeout => 2, udp_timeout => 2, retrans => 1, ); $dnsxl_res or die "Module Net::DNS not available for DNSxL usage"; $dnsxl_res->udppacketsize(1220); my $myrbl = 'rbl.example.org'; @client_ipaddr_policy = ( \@mynetworks => 'MYNETS', q_dns_a($myrbl, 2, $dnsxl_res) => 'MY-CUSTOMER-A', # 127.0.0.2 q_dns_a($myrbl, [3,4,5], $dnsxl_res) => 'MY-CUSTOMER-B', # 127.0.0.{3,4,5} ); This DNS-lookup feature is considered experimental in a sense that its API may change in future versions. As it is currently implemented, each q_dns_a() entry in a @client_ipaddr_policy results in its own DNS query, which is quite inefficient with more that one or two such entries. It would make more sense to do a single DNS lookup and provide some mapping between results returned and policy bank names to be loaded. Note also that DNS lookups are performed synchronously and sequentially (one at a time, one after another), so a slowly responding DNS server combined with multisecond timeouts and retries could severely bog down the amavisd response time, easily to exceed the time a MTA or a SMTP client is willing to wait for a response. YOU HAVE BEEN WARNED! OTHER - Relax a check on a PID number found in a pid file, considering that amavisd may run as PID #1 under Docker; reported by Imre Rad. - Relax a check on $pid_file being configured or provided by a command line option -P. Amavisd can now run without checking or providing a PID file of a running master process, which is appropriate for running non-daemonized amavisd as a supervised process (e.g. under supervision suites such as systemd, s6, nosh, runit, launchd or similar). Also, specifying a command line option -P '' (i.e. giving it an empty name of a pid_file) overrides a configuration option $pid_file and is a quick way to disable usage of a pid_file. A default value of $pid_file is now only provided if a global setting $daemonize is true (which is a default, unless running with 'foreground' or 'debug' command line options). A non-daemonized amavisd leaves $pid_file undefined as a default, which facilitates running amavisd as a supervised process, e.g. $daemonize = $pid_file = $daemon_user = undef; When a pid_file is disabled and running under systemd, amavisd obtains a PID of a master process from systemd through environment variable MAINPID, which allows operations like 'amavisd reload' and 'amavisd stop' from a .service file (ExecReload and ExecStop in systemd.exec(5)). Btw, a command line argument 'foreground' is a quick way to override a configuration setting $daemonize - it sets its value to 0. To let amavisd provide and use a PID file even when not daemonized, configure a PID file explicitly, e.g.: $pid_file = "$MYHOME/amavisd.pid"; - provide sensible diagnostics when $daemon_user is undefined and starting as root; - 'sanitize_nul' function is now enabled by default (this is currently not configurable). Null octets found in a message are replaced by a pair of octets \xC0 \x80, which is a "Modified UTF-8" encoding of a NUL. This is done to avoid a mailbox server (like Cyrus) or a mail client on choking on such mail. The downside is that such sanitation can invalidate a DKIM signature - but non-encoded NUL octets are not allowed in mail anyway, so not much harm is done; - overhauled a client side of the ClamAV clamd protocol; - updated decoder for 7z archives to improve handling of encrypted content; based on a patch by Markus Benning; - recognize and handle completely encrypted zip archives by 7z (in do_7zip); a patch provided by Thomas Jarosch; - adjusted log levels of some log/debug messages; - reject a message with an 8BITMIME body type if a back-end MTA does not announce 8bit-MIMEtransport capability in its EHLO response ( 550 5.6.3 Conversion to 7BIT required but not supported ); - replaced calls to Encode::is_utf8() by utf8::is_utf8() - less buggy in old versions of perl, but requires perl 5.8.1 or later; - replaced calls to Encode::encode_utf8() by utf8::encode() - is much faster, and is less buggy in old versions of perl; - more detailed testing for taint bugs of a module Encode and in utf8::is_utf8() during startup; - decode a supposedly (or guessably) character set ISO-8859-1 as Windows-1252, which is a proper superset of ISO-8859-1 and often mistaken for ISO-8859-1; (this follows advice of HTML5); - with MySQL: changed character set 'utf8' to 'utf8mb4' for fields msgs.subject and msgs.from_addr; - in case the Net::Server receives a connection over a Unix socket (e.g. from amavisd-release) but is unable to determine a socket name, supply a dummy socket name 'UNKNOWN' so that a policy bank 'SOCK' can still be loaded; - the setting $mail_digest_algorithm is now a dynamic setting, i.e. can be configured per policy bank. The change makes it consistent with a new setting $mail_part_digest_algorithm, which is also dynamic; - updated the @av_scanners Avast entry ( http://www.avast.com/ ) in the sample config file amavisd.conf to a new version of their scanner: ['avast! Antivirus', '/bin/scan', '{}', [0], [1], qr/\t(.+)/m] Thanks to Martin Tůma from Avast for the new entry; - updated a default @$map_full_type_to_short_type_re to distinguish encrypted PGP/GnuPG files from other PGP/GnuPG containers like a detached signature, exported public key files, etc., if a newer version of a file(1) utility is in use (5.20?); - relaxed the /\bscript\b.* text executable\b/ regexp entry in the default @$map_full_type_to_short_type_re list so that a mail part such as qualified by a file(1) utility as: Python script, Non-ISO extended-ASCII text executable does not qualify as an executable; reported by Tilman Schmidt; - updated a default @$map_full_type_to_short_type_re to recognize a Microsoft Word document as type doc; thanks to Jörg Backschues; - added PhishTank.Phishing to a default @virus_name_to_spam_score_maps; - reworded some notification texts; SUPERVISED PROCESS NOTES Socket activation (running under a superserver with fd-holding) is currently not available. Note that 'amavisd reload' does fd-holding and socket passing to a new incarnation of amavisd server on its own, which means that a client (an MTA) does not see a disruption on a reload (= warm restart), unlike in case of restarting amavisd. As a reminder: a full restart is only necessary when changing the set of listening sockets in a configuration file. For all other needs (like changing other settings, updating SpamAssassin rules, upgrading the amavisd program, or perl modules, or perl itself) a reload suffices. Here is a sample file amavisd.service for use under systemd (indented for clarity). Lightly tested on Debian 8.0 (Jessie) on a Raspberry Pi. /lib/systemd/system/amavisd.service [Unit] Description=amavisd-new mail filter Before=shutdown.target After=systemd-journald-dev-log.socket network-online.target local-fs.target Wants=network-online.target Conflicts=shutdown.target [Service] Type=notify NotifyAccess=main KillMode=mixed TimeoutStartSec=1min TimeoutStopSec=3min User=amavis Group=amavis WorkingDirectory=/var/lib/amavis/tmp StandardOutput=syslog SyslogFacility=mail SyslogIdentifier=amavis ProtectSystem=full ProtectHome=yes NoNewPrivileges=yes ExecStart = /usr/sbin/amavisd-new -P '' foreground ExecReload = /usr/sbin/amavisd-new -P '' reload ExecStop = /usr/sbin/amavisd-new -P '' stop [Install] WantedBy = multi-user.target Consider the following amavisd.conf settings when running as a supervised process: $pid_file = ''; # can be overridden by a command line option -P '' $daemonize = 0; # also implied by a command line argument 'foreground' $do_syslog = 0; # or set it to 1 to log to syslog instead of stderr --------------------------------------------------------------------------- October 25, 2014 amavisd-new-2.10.1 release notes Contents: COMPATIBILITY BUG FIXES OTHER COMPATIBILITY There are no incompatibilities between versions 2.10.0 and 2.10.1. BUG FIXES - fixed a missing import of mail_addr_idn_to_ascii() and idn_to_utf8() when SQL is in use, resulting in: Undefined subroutine &Amavis::Out::SQL::Log::mail_addr_idn_to_ascii Undefined subroutine &Amavis::Out::SQL::Log::idn_to_utf8 reported by Jim Knuth; OTHER - avoid warnings issued by perl 5.21.5: Negative repeat count does nothing at ./amavisd line 17218 Missing argument in sprintf at ./amavisd line 3678 - updated documentation in 2.10.0 release notes (new macros); --------------------------------------------------------------------------- October 22, 2014 amavisd-new-2.10.0 release notes Contents: COMPATIBILITY BUG FIXES NEW FEATURES OTHER COMPATIBILITY - New requirement: perl module Net::LibIDN needs to be installed. - Uses a perl module File::LibMagic if installed, instead of spawning a file(1) utility. - Support for international email relies heavily on perl to do the right thing in its support of Unicode, so using a reasonably recent version of perl is recommended. Amavisd was tested with perl 5.18 and 5.20.1. Versions of perl older than 5.12 may cause problems with handling, encoding, and decoding of Unicode characters. It is reasonable to expect that versions 5.14 and 5.16 are fine too, but have not been tested extensively. - Default log templates and notification templates have changed in details (like in decoding of international e-mail addresses), so if locally customized templates are in use these will benefit from updating - otherwise expect some mojibake in log and notifications. - International domain names (IDN) encoded in ASCII-compatible encoding found in e-mail addresses and in Message-ID header field will be decoded to Unicode for presentation purposes (syslog, JSON structured log, notifications). This decoding does not affect a mail message itself. - Logging via syslog expects that syslogd (or equivalent) will not clobber UTF-8 octets. It may be necessary to tell syslogd to accept C1 control characters unchanged, e.g. by adding a command line option "-8" to syslogd. Failing to do so may leave logged entries (like sender and recipient address, From, Subject) in international mail garbled or poorly readable in syslog. On FreeBSD one should add: syslogd_flags="-8" to /etc/rc.conf. - Third party log parsers may need updating to accept logs with Unicode characters in UTF-8 encoding. - A SMTP response to an EHLO command will now announce SMTPUTF8 capability by default. BUG FIXES - releasing a message from an SQL quarantine was broken in version 2.9.1 due to introduction of parent_mail_id(); patches provided by Stef Simoens and Gionatan Danti; - if checking of a message was aborted prematurely (like due to a timeout or some fatal error), JSON log could receive a copy of a previous log entry; - prevent non-ASCII non-UTF-8 octets from reaching a JSON log/report (which produced an invalid JSON object and Elasticsearch complaining); - allow SMTP commands MAIL FROM and RCPT TO to accept options without values, as allowed by the RFC 5321 syntax; - in delivery status notification (DSN) the field Received-From-MTA specified 'smtp' as mta-name-type, instead of a 'dns' as prescribed in RFC 3464; - releasing from a quarantine left envelope sender address as '<>' instead of using the address found in a Return-Path header field of a quarantined message, while also logging a warning: Quarantine release $QID: missing X-Envelope-From or Return-Path reported by Pascal Volk; - avoid failure in os_fingerprint or in smtp forwarding in certain cases where the $os_fingerprint_method or $forward_method or $notify_method uses an asterisk in place of a host IP address or port number. The reported error in os_fingerprint (reported by -ben) was: os_fingerprint FAILED: Insecure dependency in socket while running with -T switch at /usr/lib/perl/5.18/IO/Socket.pm line 80 and in SMTP forwarding or notification (reported by Dennis Boone): (!)connect to *:10025 attempt #1: Insecure dependency in socket while running with -T switch at /usr/lib/perl/5.18/IO/Socket.pm line 80. - files LDAP.ldif and LDAP.schema: added a missing attribute amavisDisclaimerOptions to objectClass; reported by Quanah Gibson-Mount; NEW FEATURES - added support for Internationalized Email: * RFC 6530 - Overview and Framework for Internationalized Email * RFC 6531 - SMTP Extension for Internationalized Email (SMTPUTF8) * RFC 6532 - Internationalized Email Headers * RFC 6533 - Internationalized Delivery Status Notifications This supports UTF-8 (EAI) in SMTP/LMTP sender addresses, recipient addresses, and message header section. Feature parity with Postfix version 2.12 (support introduced in development snapshot 20140715). The SMTPUTF8 extension is supported by Gmail since 2014-08-05: http://googleblog.blogspot.com/2014/08/a-first-step-toward-more-global-email.html - added support for Internationalized Domain Names (IDN) according to IDNA (RFC 5890, RFC 5891; RFC 3490); * A-labels in ASCII-compatible encoding of domain names are converted to U-labels for presentation/logging purposed; * U-labels are converted to A-labels when feeding a mail message to an MTA which does not announce support for SMTPUTF8 extension (instead of rejecting them as invalid mail address); * For lookup purposes an international domain name is converted to ASCII-compatible encoding when used as a query key in DNS lookups and in lookups into hash, list, SQL and LDAP lookup tables (but not in regexp table lookups). These tables are expected to contain domain names in their ASCII representation (ACE). For convenience of config files subroutines idn_to_ascii() and mail_idn_to_ascii() are available, which encode a Unicode domain name to ACE (like ToASCII in RFC 3490); * Many configuration settings may have their domain names in UTF-8. These will be converted to ACE automatically where necessary (e.g. when creating a Received and Authentication-Results header fields, DKIM signatures, mail addresses in notifications, ...). These settings include: $myhostname, $localhost_name, $myauthservid, $mydomain, notification sender and recipient mail addresses ($mailfrom_notify_*, $hdrfrom_notify_*, @*_admin_maps), domain names and selectors in DKIM signing keys (in calls to dkim_key() ); - delivery notifications and admin notifications now show the following information encoded as UTF-8 (which is a default $bdy_encoding) in the plain text part of the message: IDN domain names in sender and recipient mail addresses and Message-ID are first decoded to Unicode, Subject and author display names are MIME-decoded; - 'amavisd showkeys' and 'amavisd testkeys' can now deal with IDN (international domain names): domain names in DNS zone comments end up as UTF-8, DNS labels are in ASCII (A-labels); domain names in calls to dkim_key() may be specified either as UTF-8 or in ASCII (ACE); - new macro 'mail_addr_decode' takes an e-mail address as a string of octets, where a local part may be encoded as UTF-8, and the domain part may be an international domain name (IDN) consisting either of U-labels or A-labels or NR-LDH labels. Decodes A-labels to U-labels in domain name. Returns a string of logical characters (Unicode), suitable for notification templates. If the mail address is not a valid UTF-8 string, it is interpreted as ISO-8859-1 (Latin-1). - new macro 'mail_addr_decode_octets' is like 'mail_addr_decode', except that the result is a string of octets, only valid as UTF-8 if the provided address was a valid UTF-8 (garbage-in/garbage-out); - new macro 'header_field_octets' is like 'header_field', except that a result is a string of octets in UTF-8 encoding, suitable for a log template; - new macro 'ip_proto_trace_all' expands into a list of information items from a Received header trace; each item consists of a protocol name (the WITH clause) and an IP address, optionally followed by a source port number if known; Example: ESMTP://[2001:db8::143:1]:39141 < ESMTP://2001:db8::25 < esmtps://203.0.113.172 < ESMTPSA://192.168.9.9 or: UTF8SMTP://[203.0.113.172]:51208 < UTF8SMTPSA://192.168.9.9 - new macro 'ip_proto_trace_public' is like ip_proto_trace_all, except that entries with non-public IP address are excluded from the list; 'Received' trace information in $log_verbose_templ and in notifications now include results from this macro call; - new macro 'protocol' evaluates to a protocol name by which a message was received by amavisd, according to RFC 3848 ("Transmission Types Registration") and "Mail Transmission Types" / "WITH protocol types" IANA registration http://www.iana.org/assignments/mail-parameters/mail-parameters.xhtml e.g.: SMTP, ESMTP, ESMTPA, ESMTPS, ESMTPSA, LMTP, LMTPA, LMTPS, LMTPSA, UTF8SMTP, UTF8SMTPA, UTF8SMTPS, UTF8SMTPSA, UTF8LMTP, UTF8LMTPA, UTF8LMTPS, UTF8LMTPSA, ... - new macro 'client_protocol' expands into a protocol name by which a message was received from a client by MTA; the information is passed from MTA to amavisd through XFORWARD PROTO SMTP protocol extension or through AM.PDP (milter); typical values are 'ESMTP' or 'SMTP'; - use a perl module File::LibMagic when available, instead of spawning a file(1) utility for classifying contents of mail parts. By using a direct interface to a libmagic library the startup cost of spawning an external process is avoided. Benchmarking shows that using libmagic is significantly faster especially for checking a small number of files - takes 4 ms for checking one file with libmagic vs. 27 ms with a spawned file(1); based on a patch by Markus Benning; OTHER - RFC 6533: recognize a MIME type 'message/global' as similar to 'message/rfc822', and 'message/global-headers' as similar to 'text/rfc822-headers' where appropriate (e.g. in bounce killer); - header validity check now distinguishes 'non-ASCII and invalid UTF-8' from 'non-ASCII but valid UTF-8' characters in a mail header section. By default valid UTF-8 strings in a mail header section are not treated as error even if mail is not flagged as international mail (SMTPUTF8), as these are quite common in practice. To treat non- MIME-encoded UTF-8 in a header section as error the test can be enabled by: $allowed_header_tests{'utf8'} = 1; - ORCPT attribute in SMTP 'RCPT TO' command now accepts the original recipient mail address in any of these encodings: utf-8-address, utf-8-addr-unitext, utf-8-addr-xtext, or as a legacy xtext, as required by RFC 6533; - updated do_cabextract (extraction of Microsoft cabinet .cab archives) to recognize a slightly changed output of cabextract version 1.2; patch by Thomas Jarosch; - adjusted some timeouts to leave more reserve for later stages of mail processing and forwarding; - prefer sanitizing/protecting control characters as hex code (like \x7F) instead of octal (like \177) (e.g. in logging and DSN); --------------------------------------------------------------------------- June 27, 2014 amavisd-new-2.9.1 release notes Contents: COMPATIBILITY BUG FIXES OTHER COMPATIBILITY There are no known incompatibilities between versions 2.9.0 and 2.9.1. BUG FIXES - when a $final_bad_header_destiny is set to D_BOUNCE and a mail message was a DSN or coming from a mailing list, spam checks were inappropriately skipped (logged as: "bypassing of spam checks, message will be blocked anyway", which was not true as the blocking was overruled at a later stage); reported by Bruce Pennypacker; - perl 5.20.0 may occasionally report: "Malformed UTF-8 character (unexpected end of string)" due to a bug in perl [perl #122148]. Treat this error as non-fatal and only log a warning in two such cases: when SpamAssassin is called as a spawned subprocess ($sa_spawned=1) and is returning a result to a parent process, or when constructing a structured report. OTHER - updated decoding of RAR archives to recognize a changed format in output of 'unrar' utility version 5; based on a patch by amavis17(at)iotti.biz - avoid tempfailing a message if a redis server is down, just log the error and carry on; - some minor logging changes to facilitate troubleshooting; --------------------------------------------------------------------------- May 9, 2014 amavisd-new-2.9.0 release notes Contents: COMPATIBILITY NEW FEATURES SUMMARY RELAXED LICENSE BUG FIXES NEW FEATURES OTHER WHY REDIS? COMPATIBILITY This version drops dependency on a Perl module Redis, and makes dependencies on modules Convert::TNEF and Convert::UUlib truly optional. The following change may affect third-party log parsers: To facilitate forensic log analysis and troubleshooting, log entries 'FWD from' and 'SEND from' at level 1 now carry one additional prefixed information field which is the unique internal mail_id of the message, possibly followed by a parent_mail_id in parenthesis, e.g.: (00525-02) XE9xnQYjrWyd FWD from <...> -> <...>, ... (00495-02) v1pyIOMQkUYD(CIcqao-vCDO9) SEND from <...> -> <...>, ... No other incompatibilities with a previous version 2.8.1 are expected. NEW FEATURES SUMMARY - structured log/reporting to a Redis server in JSON format; - IP address reputation (uses a Redis server); - added two minor content categories to the major ccat CC_UNCHECKED (encrypted (=1) and over-limits/mail-bomb (=2) ); - introduced a by-recipient setting %final_destiny_maps_by_ccat. RELAXED LICENSE Some utility / auxiliary programs that were previously released under a 3-clause BSD license, are now available under a more relaxed 2-clause BSD license (also known as a "Simplified BSD License" or a "FreeBSD License"). Affected programs are: amavis-mc, amavis-services, amavisd-status, amavisd-snmp-subagent-zmq, amavisd-release, amavisd-submit, p0f-analyzer.pl, amavisd-nanny, amavisd-agent, amavisd-snmp-subagent, amavisd-signer, JpegTester.pm, and TinyRedis.pm. Note that TinyRedis.pm is provided in the package as a separate file and includes a documentation section. Its copy is also included in the file amavisd, so that the separate file is not needed for Amavis operation. The separate copy is provided under a 2-clause BSD license so that it may be useful for third parties if desired. Eventually it could be moved to CPAN as an independent module. A license of the main program 'amavisd' remains unchanged GPLv2. BUG FIXES - fixed "Insecure dependency in sprintf" in Sophos SAVI av-scanner, reported by Maciej Uhlig; - fixed the interface code to virus scanners Sophie, Trophie and fpscand, where a time-out on a long-running virus scan would leave a connection to the virus scanner open and a late response from a scanner to a previous request could be interpreted as a result of the current scan; reported by David Schweikert; - fixed a bug in transforming an IPv6 alternative form IP address into a preferred form. One effect of this bug was declaring an IPv4-mapped IPv6 address as syntactically incorrect; reported by Patrick Domack; - if SQL logging was disabled a pen pals feature was non-functional even when a Redis storage back-end was available and collecting data; now pen pals is fully functional with a Redis database back-end and no SQL; - provided our own Redis client code, avoiding Redis CPAN module bugs, its slowness and non-support for IPv6. The noteworthy Redis CPAN module bug is the #38 (failing to re-select a non-zero-index database after an automatic re-connect to a server). See: https://github.com/melo/perl-redis/issues/38 https://github.com/melo/perl-redis/issues/28 - fixed a regexp in parsing wildcarded signing domain in a DKIM key declaration and in a wildcarded sender pattern of signing options (this feature is rarely used, exists for compatibility with dkim_milter); - dropped hard-coded dependency on modules Convert::TNEF and Convert::UUlib. The Convert::TNEF was made optional in amavisd-new-2.8.0, but the program still failed if the module could not be loaded at startup. Both of these modules are now loaded at run time when first used, if specified in the @decoders setting. The use of module Convert::UUlib (the do_ascii entry) is disabled in a default setting of @decoders, and the module Convert::TNEF (the do_tnef entry) is not used if an external TNEF decoder (the do_tnef_ext entry) is available, or if disabled in the @decoders list; - import a missing do_log_safe() in Amavis::LDAP::Connection to avoid a warning: _WARN: \t(in cleanup) Undefined subroutine &Amavis::LDAP::Connection::do_log_safe called at (eval 101) line 76 during global destruction; a patch by Quanah Gibson-Mount; - at startup amavis may try to find a decoder for 7z and zip extensions twice; a fix by Quanah Gibson-Mount; - fixed the amavisd-new-courier.patch which resulted in two instances of sub post_bind_hook(). Only tested for syntax. Thanks to Eray Aslan. NEW FEATURES - Structured logging/reporting in JSON format is now available through a redis server. Each processed mail message and each generated mail message (e.g. a delivery status notification) generates a structured data object (internally a perl associative array). Its fields carry information on most attributes of a mail message and its processing, similar to what is available for logging via macros. Unlike a plain text log which can be difficult to parse and inconsistent due to user configurability of the log template, the data object contains information in a structured form as key/value pairs, where each value can be a scalar or a list or an associative array. This internal data object is then serialized to a JSON format and sent to a redis server, where it is appended to a list under a key (arbitrary string) configured by $redis_logging_key setting. This list serves as a queue of log events, which may be pulled from the queue by some third party application, e.g. by a logstash utility or by some home-grown program. Redis server is quite handy for this purpose as it offers blocking requests for pulling events from a queue, which makes it easy to interface with an event processing program. The queue also allows for independent and asynchronous operation between amavisd child processes filling the queue, and a log analyzer pulling entries from the queue. The structured logging to redis is enabled when @storage_redis_dsn is configured (see below at the 'IP address reputation' section) and the setting $redis_logging_key is set to some nonempty and nonzero string, and the $redis_logging_queue_size_limit is set to some positive integer value (corresponding to a maximal number of entries allowed in a queue). Both the $redis_logging_key and $redis_logging_queue_size_limit are undefined by default, so structured logging to redis is disabled by default even if @storage_redis_dsn is configured. The string in $redis_logging_key determines the key in a redis database where the event queue (a redis list) will be maintained. Semantically it is a name of the queue. This setting is a component of policy banks, so log entries can be fed into different redis queues depending on a policy bank loaded for each mail message. To prevent a queue in the redis server from growing out of bounds, e.g. when an event-pulling program is temporarily nonfunctional or its processing is falling behind, the $redis_logging_queue_size_limit setting imposes a maximal number of events that amavisd may push into the queue, i.e. the maximal queue size. If the queue size limit is reached, new log events from amavisd are discarded as long as the queue size is at the limit. As a redis database is kept in memory, it makes sense to choose the value of $redis_logging_queue_size_limit low enough so that it does not use too much memory if the log processing program goes down, but also high enough so that short outages of the log processing program do not lose any log events. The setting $redis_logging_queue_size_limit is global (not a component of policy banks). And example setting: @storage_redis_dsn = ( { server => '[::1]:6379', db_id => 1 } ); $redis_logging_queue_size_limit = 300000; # takes about 250 MB of redis memory per 100000 log entries $redis_logging_key = 'amavis-log'; $policy_bank{'MYNETS'} = { originating => 1, redis_logging_key => 'amavis-log-myusers', # overrides global setting } The oldest event may be pulled from listed queues by the redis command: BLPOP amavis-log amavis-log-myusers 0 so from a command line this may look like: $ redis-cli -h ::1 -p 6379 -n 1 BLPOP amavis-log 0 The BLPOP redis command blocks if the queue is empty and only returns when the queue becomes nonempty, which makes it easy to use. For high event rates it may be more efficient to batch one LLEN and multiple BLPOP calls in a Lua script executed on a redis server and return events in chunks. An example of a logstash plugin configuration for pulling amavis log events from a redis server and feeding them to Elasticsearch: input { redis { type => "amavis" host => "::1" db => 1 data_type => "list" key => "amavis-log" codec => json {} } } filter { date { match => [ "time_unix", "UNIX" ] } } output { # stdout { codec => rubydebug } elasticsearch_http { host => "127.0.0.1" port => 9200 index_type => "%{type}" document_id => "%{mail_id}" codec => json {} } } As an alternative for sending log events to a redis server, it is possible to use a macro [:report_json] in a log template, which will expand to a full JSON representation of a log event. As these strings are fairly long (typically 2 kB to 3 kB), this is not a good solution when logging to syslog. It may be usable when logging to a file, but is not an efficient solution and has not been tested in production. Here is a (fake) example of a structured log report entry in JSON format, fields are loosely ordered by their semantics in this example. Not all fields are always present. When a boolean fields is missing it should be interpreted as a false. { "@timestamp" => "2014-05-06T09:29:47.048Z", "time_unix" => 1399368587.048, "time_iso_week_date" => "2014-W19-2", "partition" => "19", "type" => "amavis", "host" => "mailer.example.net", "src_ip" => "::1", "dst_ip" => "::1", "dst_port" => 10024, "log_id" => "82329-04", "mail_id" => "Jnk7NzYB8pvl", "mail_id_related" => ["men7HTERZaOF"], "client_port" => 41831, "client_ip" => "2001:db8::143:1", "ip_trace" => ["2001:db8::143:1", "192.0.2.242"], "os_fp" => "Windows XP; dist: 6; raw_mtu: 1340; ...", "originating" => true, "policy_banks" => ["PROXY-ORIGINATING", "MYNETS"], "size" => 302694, "digest_body" => "a4a7db6307c140b12f57feaf076663f8", "mail_from" => "mailing-list-1@example.com", "rcpt_to" => ["recip2@example.org", "recip1@example.net"], "rcpt_num" => 2, "message_id" => "<003701cf690d$b671b3f0$23551bd0@example.com>", "author" => ["sending-user@example.com"], "to_addr" => ["recip1@example.net"], "cc_addr" => ["recip2@example.org"], "subject" => "Fw: An example 123 - test", "subject_rot13" => "Sj: Na rknzcyr 123 - grfg", "user_agent" => "Microsoft Office Outlook 12.0", "is_bulk" => true, "is_mlist" => true, "action" => ["PASS"], "actions_performed" => "RelayedInternal RelayedOutbound", "checks_performed" => "V S H B F P", "content_type" => "Clean", "dkim_new_sig" => ["example.com"], "dsn_sent" => false, "elapsed" => { "Receiving" => 0.009, "Decoding" => 0.053, "VirusCheck" => 0.326 "SpamCheck" => 2.116, "Sending" => 0.118, "Amavis" => 0.215, "Total" => 2.672, }, "message" => "82329-04 PASS Clean <mailing-list-1@example.com> -> <recip2@example.org>,<recip1@example.net>", "queued_as" => ["3gNFyR4Mfjzc3", "3gNFyR4n6Lzc4"], "recipients" => [ { "action" => "PASS", "ccat_main" => "Clean", "queued_as" => "3gNFyR4Mfjzc3", "rcpt_is_local" => false, "rcpt_to" => "recip2@example.org", "smtp_code" => "250", "smtp_response" => "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4Mfjzc3", "spam_score" => -2.0 }, { "action" => "PASS", "ccat_main" => "Clean", "mail_id_related" => "men7HTERZaOF", "penpals_age" => 1114599, "queued_as" => "3gNFyR4n6Lzc4", "rcpt_is_local" => true, "rcpt_to" => "recip1@example.net", "smtp_code" => "250", "smtp_response" => "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4n6Lzc4", "spam_score" => -5.272 } ], "smtp_code" => ["250"], "spam_score" => -2.0, "tests" => ["ALL_TRUSTED", "AM.PENPAL", "BAYES_00", "MSGID_MULTIPLE_AT", "RP_MATCHES_RCVD"], "tests_ham" => ["AM.PENPAL","BAYES_00","ALL_TRUSTED","RP_MATCHES_RCVD"], "tests_spam" => ["MSGID_MULTIPLE_AT"], } - IP address reputation When a Redis storage back-end is enabled, besides the existing pen pals functionality, it now also offers information updating and retrieval on IP address reputation. This function is enabled by default when @storage_redis_dsn is nonempty, but can be disabled by setting $enable_ip_repu to false (to 0 or undef), per policy bank if necessary. For each mail message a list of public IP addresses (IPv4 or IPv6) is collected from its 'Received' trace header fields in a mail header section. A redis server maintains a database of each IP address encountered. For each IP address an entry carries a set of counters corresponding to the number of mail messages encountered in the past having this IP address in a trace header. These counters show: a number of spam messages, a number of ham messages, a number of banned or infected messages, and a total number of messages. Also a timestamp of the first and last encounter is kept. Each entry (a set of counters) is subject to automatic expiry, so that infrequently encountered IP addresses are eventually automatically purged from a database by a redis server itself. As a sending IP address may change its role (e.g. some machine was infected (sending spam) but now has been cleaned, or a NAT-ted address is reassigned to someone else), currently a crude way of data aging is implemented by discarding entries older than three days since created. This may be refined in the future. When a new mail message is being processed, a lookup on all its public IP addresses from a trace is done. For each IP address found in a database a spam score is computed based on a ratio of ham versus all messages, and based on a total number of messages. The largest calculated spam score of all encountered IP addresses is then contributed to a total spam score of a message. A formula for computing spam score of each IP address is currently hard-coded, is non-linear and takes into account the total number of encounters of an IP address, diluted by the ratio of ham messages versus all messages seen with this IP address. The computed score cannot be negative, i.e. the IP reputation can only contribute to spamminess of a message and cannot serve as a 'whitelisting' negative score. For the exact formula in use see query_and_update_ip_reputation() in file amavisd. A time-to-live of each IP entry is assigned dynamically: frequently encountered IP addresses are given longer expiration times (days), infrequent IP addresses are short-lived and eventually expire, typically in few hours. It is possible to exclude certain IP addresses or networks from contributing spam score by listing them in an @ip_repu_ignore_networks list, e.g.: @ip_repu_ignore_networks = qw( 192.0.2.44 192.0.2.45 198.51.100.0/24 2001:db8::1:25 ); This does not preclude a redis lookup on an IP addresses matching the list, but just takes a zero as its score and does not update counters on such address. The mechanism is appropriate for excluding site's own mailers (MSA and MX), or local (e.g. departmental) mailers, which may on occasion emit a spammy message, but should never receive a score penalty. There is no need to include private IP address networks in the list, as these are already exempt from IP reputation database. An associated list of lookup tables @ip_repu_ignore_maps (whose only default entry is the \@ip_repu_ignore_networks) offers more flexibility if needed, and is a member of policy banks. Like other self-learning mechanisms (e.g. SpamAssassin's auto-learn, AWL, TxRep), the quality of a result depends on a quality of other spam-gauging rules - the better spam/ham classification works (SpamAssassin), the more useful IP reputation becomes. For the purpose of IP reputation's spam and ham counts, a mail is considered spam if its score is at or above 5, and is considered ham when its final score is below 0.5. This is currently hard-coded (see sub save_info_final). Intermediate scores are considered unclassified. A nice feature of the mechanism is that it reacts fairly quickly to a new rush-in of unwanted messages from some IP address, either foreign, or local. For insight on the IP address reputation behaviour, search the log for ' redis: IP '. At log level 2 only spammy hits are logged, at log level 3 also the clean hits are shown. The log entry shows spam, ham, banned+infected and unclassified counts for an IP address, a percentage of unwanted (spam+banned+infected) messages out of the total count, and the associated score. Apart from starting a redis server on a loopback interface (except for changing its 'bind' setting in redis.conf, no other configuration changes are necessary, a database need not be initialized), here is an example configuration in amavisd.conf: @storage_redis_dsn = ( { server => '[::1]:6379', db_id => 1 }, { server => '127.0.0.1:6379', db_id => 1 }, ); # list your MX and MSA mailer IP addresses or networks here: @ip_repu_ignore_networks = qw( 192.0.2.44 2001:db8::/64 ); A redis server needs to support Lua scripting, which is available since version 2.6. Support for IPv6 is available since version 2.8.0 of the redis server. - Added support for decompressing LZ4 streams in mail attachments when an external utility lz4c is available and the 'file' utility recognizes such streams (probably since version file-5.17). Default settings of @decoders and $map_full_type_to_short_type_re now recognize LZ4; if these settings are replaced by a configuration file, the config file needs to be updated to include the new entry. - Added two minor content categories to the major ccat CC_UNCHECKED to allow distinguishing between reasons of decoders failure. * a minor ccat 1 now indicates that at least one mail part was encrypted or otherwise scrambled (e.g. password protected archive); * a minor ccat 2 now indicates that some of the limits for protection against mail bombs was exceeded (e.g. $MAXLEVELS, $MAXFILES, $MAX_EXPANSION_QUOTA, $MAX_EXPANSION_FACTOR). Based on a suggestion and a patch by Carsten Wolff. The additional information can be used in any of the *_maps_by_ccat settings, e.g.: $subject_tag_maps_by_ccat{CC_UNCHECKED.',1'} = [ '***UNCHECKED(Encrypted)*** ' ]; $subject_tag_maps_by_ccat{CC_UNCHECKED.',2'} = [ '***UNCHECKED(OverLimit)*** ' ]; or: $defang_by_ccat{CC_UNCHECKED.',2'} = 1; - introduced a setting %final_destiny_maps_by_ccat, which makes it possible to specify by-recipient final destiny for each contents category, e.g. use D_REJECT on spam to some users, and D_BOUNCE or D_DISCARD or D_PASS for others. Introduced mostly for completeness. As a backward compatibility measure the existing %final_destiny_by_ccat is now an alias for the new %final_destiny_maps_by_ccat; - added a setting $outbound_disclaimers_only. When set to true and disclaimers are enabled, it will only allow adding disclaimers to non-local recipients. For backward compatibility the default value is false (undef). Based on a patch by Quanah Gibson-Mount; - the $recipient_delimiter setting can now hold a multi-character string, specifying all characters that can delimit an address extension from a base e-mail address. Previously this setting was restricted to a single character (typically a '+' or a '-'). When parsing existing e-mail address any of the characters in $recipient_delimiter can delimit an address extension. When adding an address extension (through %addr_extension_maps_by_ccat), the first character in the $recipient_delimiter string is used as a delimiter. The change is now in line with a postfix 2.11 that added support for multi recipient-delimiters, and a similar feature in Dovecot. A patch contributed by Patrick Domack. - added macros report_json and rot13 (to be used in a log template): * the macro 'report_json' expands to a JSON representation of a structured log event; * the macro 'rot13' replaces a string in its argument with an obfuscated string where letters are shifted by 13 positions of an English alphabet (a popular variant of a Caesar cipher to conceal spoilers); this may serve to (poorly) hide strings such as mail Subject or an e-mail address from casual browsing of a log; OTHER - dropped dependency on a CPAN module Redis, implementing our own client-side redis protocol implementation (Amavis::TinyRedis). It is faster and smaller, and supports opening sessions with a redis server over IPv6 (or over IPv4 or over a Unix socket). The redis server supports IPv6 starting with version 2.8.0. Currently supported options in @storage_redis_dsn are: server, db_id, password, and ttl. The 'server' specifies an INET or INET6 socket (a host IP address or name and a port number) or an absolute path to a Unix socket. An IPv6 address must be enclosed in square brackets. The default value is '127.0.0.1:6379'. Match this with your redis configuration. Option 'db_id' specifies a redis database index (given to a "SELECT" redis command). Its value is a (small) integer, defaults to 0. This allows for independent databases to co-exist on the same redis server, e.g. an amavis database and a SpamAssassin Bayes database. The 'ttl' option can override a global setting $storage_redis_ttl on a per-server basis. Its value is an integer, representing a number of seconds for expiration time of pen pals records. It defaults to $storage_redis_ttl, which in turn defaults to 16 days (in seconds). This setting does not affect IP reputation records, whose expiration time is computed dynamically. Example: $storage_redis_ttl = 22*24*3600; # 22 days for pen pals records @storage_redis_dsn = ( # alternative servers, use the first which works { server => '[::1]:6379', db_id => 1 }, { server => '127.0.0.1:6379', db_id => 1, password => 'abc...' }, { server => '/tmp/redis.sock', db_id => 1, ttl => 8*24*3600 }, ); Btw, make sure to keep the setting $database_sessions_persistent at its default value (1, i.e. enabled), otherwise Redis performance will suffer somewhat. - store only essential information for pen pals operation to a Redis storage back-end to save memory on a database server; information on inbound messages is no longer stored there, i.e. only information on originating messages is kept; - more informative logging of pen pals query results when using a Redis storage back-end. The redis support code (Lua and protocol handling) was largely rewritten for efficiency since amavisd-new 2.8.1. - added LDAP attribute amavisDisclaimerOptions 1.3.6.1.4.1.15312.2.2.1.47 to LDAP.schema; contributed by Quanah Gibson-Mount; - reduced EDNS payload size from 1240 bytes to a conservative default of 1220 bytes when calling Mail::DKIM verifier; - optimization: filter for public IP addresses from a Received trace only once; - added one digit of precision in the TIMING log report to reported small elapsed times (below 5 ms); - in a milter setup (AM.PDP) the log-id wasn't unique; adding a request sequence number to it; a patch by Andreas Schulze; - avoid writing a notification to stdout about a warm reload for the benefit of a cron job; a patch by Andreas Schulze; - reduced log level on some of the less useful log messages in a milter setup; a patch by Andreas Schulze; - documentation README.sql-mysql: added "CREATE INDEX msgs_idx_mail_id..." with a note on an InnoDB requirement for a foreign key; by Jernej Porenta; WHY REDIS? A redis database was chosen initially because SpamAssassin 3.4.0 supports keeping its Bayes database in a redis server, which makes it very fast, so this makes a redis database readily available to amavisd too. Redis has some features that make it suitable for use as a pen pals database, for Bayes storage, and now for IP reputation and structured logging: - automatic expiration of entries based on key's individual time-to-live setting makes explicit database maintenance unnecessary; - accessible over INET (or Unix sockets) allows several amavisd hosts to use a common redis server, possibly running on a dedicated host; - supports Lua scripting, which makes it possible to perform multiple basic operations in one go as a single application's functional operation. It reduces multiple network round-trip times to a single network transaction, reducing network packet rate and latency; - compared to SQL storage for pen pals (and for Bayes database), the redis read speed is somewhat faster, and the write speed is MUCH faster; - as an in-memory database with optional periodic disk persistence it makes it suitable for use as a pen pals, as IP reputation and as Bayes storage: it is fast, and a potential redis server restart reloads data from the last snapshot, thus only losing the last minute or two of updates when trouble strikes, which is acceptable for these three databases. - makes it possible to eliminate SQL r/w storage if its only purpose was to provide pen pals functionality (and SpamAssassin's Bayes); Caveat: Redis server does not offer access controls or strong authentication mechanisms. For running a server on the same host as amavisd is running the solution is straightforward: just bind the redis server to a loopback interface or use a Unix socket. If a network access is desired, consider protecting the redis server by a firewall (host-local, or on a dedicated subnet). --------------------------------------------------------------------------- June 28, 2013 amavisd-new-2.8.1 release notes COMPATIBILITY - when 0MQ (a.k.a. ZeroMQ) is used between Amavis components as an internal messaging protocol, make sure to replace all 0MQ-enabled Amavis components on upgrading amavisd, as the internal protocol has changed slightly, taking advantage of 0MQ multi-part messages for better performance. Affected programs are: amavis-services, amavisd-status, amavisd-snmp-subagent-zmq, and amavisd. NOTE: The Crossroads I/O project (libxs) ceased development on July 2012, to be replaced by nanomsg eventually by the same author. The 0MQ library (libzmq) is currently (2013) the best choice, the preferred library version is 3.2.2 or later along with the ZMQ::LibZMQ3 Perl interface module and ZMQ::Constants. The older version 2 of the library, along with an older perl module ZeroMQ, should be fine too, but lacks support for IPv6. - amavisd is compatible with perl 5.18.0 and with SpamAssassin 3.4.0 BUG FIXES - fixed a bug in the SMTP client code, where the final SMTP status did not reflect a failure status of a DATA command from a back-end MTA. This caused a reception of a mail message to be confirmed but a message was then lost, as it could not be passed to a back-end MTA. The bug went unnoticed for years, as the commonly used MTAs normally reject either at the MAIL FROM, at RCPT TO, or at the data-dot stage, but not at the DATA command. Reported by Deniska-rediska; - fixed calling an external spam scanner DSPAM or Bogofilter, which failed with a message: auto-learning with spam scanner ... failed: error running program Reported by Tonio; - if a configuration file path as given through a command line option -c or as an argument to include_config_files() was not an absolute path, and that file contained an error, the do() would search the @INC list for alternative files of the same name, and reported an unrelated error (typically: No such file or directory) instead of reporting the true reason for a failure; - fixed a regular expression in amavisd.conf for an 'Avast!' AV entry to properly extract a virus name; a patch by Ralf Hildebrandt; - added LDAP errors LOCAL_ERROR and OPERATIONS_ERROR to the set of expected error conditions which lets amavisd retry the failed operation; a patch by Quanah Gibson-Mount; NEW FEATURES SUMMARY - new Redis storage for the "pen pals" feature; - improved IPv6 support; - support for p0f v3; - new macros ip_trace_all and ip_trace_public; - amavisd-status now shows a bar graph display of the number of active processes; - the timing report log entry can show CPU usage at log level 2 if a module Unix::Getrusage is available; NEW FEATURES - new Redis storage for the "pen pals" feature: instead of (or in addition to) the existing SQL storage for keeping data on past mail messages and contributing negative spam score to recent/ongoing correspondence, this data can now be kept on a Redis server. Unlike the SQL backend, to minimize memory usage the Redis backend keeps only data which are required for pen pals operation. Redis storage for pen pals can offer a small speedup compared to a well-tuned SQL server, offers automatic expiration of data based on a configured time-to-live setting, and a simpler setup (no need to manually set up an SQL schema). A drawback is that a Redis server keeps all data in memory (with optional periodic persistence on disk), which might be of concern for busy sites with a long time-to-live setting. Potential drawbacks of a Redis server are also its lack of sophisticated access controls. A redis database may be shared between hosts running amavisd. It can be accessed either locally over a Unix socket, or using an INET socket (IPv4) over a loopback interface (better security) or over a local network. Version 2.6.14 of a Redis server does not offer access over IPv6, but version 2.8.0 and later does. Required dependencies when Redis support is enabled are a perl module "Redis" ( http://search.cpan.org/dist/Redis/ ) version 1.954 (or 1.956) or later and a redis server ( http://redis.io/ ) with support for Lua scripting (i.e. version 2.6 or later). Most pen pals application-level details on queries and storage management is delegated to Lua scripts running on a Redis server. Redis module currently requires a patch to support communication with a redis server over IPv6. Expiration time of items stored in a redis database is controlled by a setting $storage_redis_ttl, which is a time-to-live time in seconds and defaults to 16 days: $storage_redis_ttl = 16*24*60*60; Redis support in amavisd is enabled by setting a list @storage_redis_dsn to a nonempty value (similar to @storage_sql_dsn for an SQL support). If @storage_redis_dsn is empty, the redis support code is not loaded and does not occupy any storage. The configuration setting @storage_redis_dsn is a list of hashrefs (a hashref is a { ... } in perl syntax), each of which specifies one Redis server that can be used: if there is more than one entry in the list, a connection to each server is attempted until one is found where connection succeeds. Each entry is an associative array of key/value options which are passed on to a new() method of a perl Redis module unmodified and unverified. Usual options are: 'server', 'sock', 'reconnect' - see documentation of a Redis module for details. All Redis module options have their default value, so it is alright to specify an empty hash, which means to connect to a default server. Apart from options which are passed to a Redis module, two additional options are interpreted by amavisd itself and are not passed on to a Redis perl module. The 'db_id' options is an optional database index, used in a SELECT redis command to choose a (sub)database to use. By default a database index is 0. The 'ttl' option overrides a global time-to-live setting as specified in $storage_redis_ttl, allowing to chose different expiration times of stored items for each server. Examples: # disables Redis (is a default) @storage_redis_dsn = (); # enables Redis, use a single default local redis server, all # defaults are supplied by a Redis perl module, database index 0 @storage_redis_dsn = ( {} ); # access a local redis server over a loopback interface on TCP port # 6379, select database index 1, try reconnecting for 20 seconds # before giving up when a redis server is down (or restarting) @storage_redis_dsn = ( { server => 'localhost:6379', reconnect => 20, db_id => 1 } ); $storage_redis_ttl = 16*24*60*60; # expiration time for data 16 days @storage_redis_dsn = ( { sock => '/tmp/redis.sock', reconnect => 20, db_id => 1 }, { server => 'localhost:6379', reconnect => 20, db_id => 1 }, { server => 'backup.example.com:6379', db_id => 1, ttl => 5*24*60*60 }, ); Some existing settings also affect Redis pen pals operation: $database_sessions_persistent, $penpals_bonus_score, $penpals_halflife, $penpals_threshold_low, $penpals_threshold_high. Starting with version 3.4.0 the module SpamAssassin can also use Redis storage for its global Bayes database. Amavisd and SpamAssassin can use the same Redis server for their databases, although it is sensible that they use separate (sub)databases by choosing a different database index (redis SELECT command) through a 'db_id' configuration option, which defaults to 0. NOTE: As more experience with Redis is gained, it is possible that a redis storage schema may change in future versions, possibly in an incompatible way. As its purpose is short-term storage, this should not be of great concern. - improved IPv6 support: p0f-analyzer.pl can now communicate with amavisd processes over an INET6 socket (or over an INET or UNIX socket as before). Extended the protocol between amavisd and p0f-analyzer.pl to allow queries on IPv6 addresses; - rewritten p0f-analyzer.pl to support a newer p0f v3 output format, while still recognizing an older p0f v2 output format; suggested by Jernej Porenta; NOTE: the p0f v3 does not provide a compact output on stdout like p0f v2 could by using an option -l, so leave out the option -l with p0f v3; - improved IPv6 support: program amavisd-snmp-subagent-zmq can now attach as an AgentX to a Net-SNMP daemon snmpd over an INET6 socket; the AgentX socket is specified by $agentx_sock_specs near the beginning of a file amavisd-snmp-subagent-zmq, e.g.: $agentx_sock_specs = 'tcp6:localhost:705'; # talk to snmpd over IPv6 $agentx_sock_specs = 'tcp:localhost:705'; # talk to snmpd over IPv4 $agentx_sock_specs = '/var/agentx/master'; # talk over a UNIX socket - improved IPv6 support: program amavisd-submit can now submit a mail message to amavisd over an INET6 socket; - a macro W can now produce a list of all virus scanners invoked, along with a list of virus names each scanner detected; suggested by Patrick Ben Koetter; - new macros ip_trace_all and ip_trace_public: ip_trace_all provides a list of IP addresses found in the 'Received from' trace of a mail header, one entry for each Received header field, including possibly invalid IP addresses and private IP addresses; Missing addresses are substituted by with a '?' (e.g. in Received header fields for local or other non-IP mail submissions). The list order corresponds to the order of 'Received' header fields as found in a mail header, top-down, i.e. first entry of the list is the topmost (the most recent) 'Received' header field, so chronologically in reverse; ip_trace_public provides a list of valid public IP addresses found in the 'Received from' trace of a mail header. Missing, invalid or private IP addresses are not included in this list, so there may be more 'Received' header fields in a mail header then entries in this list. The list order corresponds to the order of 'Received' header fields as found in a mail header, top-down, i.e. first entry of the list is the topmost (the most recent) 'Received' header field with a valid public IP address, so chronologically in reverse; suggested by Tomislav Mihaliček; - templates for administrator notifications, recipient notifications, and sender notifications now use macro 'ip_trace_all' instead of macros 'e' and 't' in order to report the full 'received' trace, not just the first hop; - macro supplementary_info recognizes new arguments: VERSION, SUBVERSION, and RULESVERSION, providing additional information from SpamAssassin correspond to equivalent SpamAssassin tags; - a new command line option -X allows controlling some exotic features, useful for example in debugging or automatic testing. The option takes one argument which is a comma-separated list of keywords. Currently the only recognized option is '-X no_conf_file_writable_check', which disables security checks on configuration files, which can be useful in automatic testing, but is dangerous to use in production. Suggested by Alexander Wirt; - a configuration setting $sa_debug may now specify a comma-separated list of SpamAssassin debug facilities, complementing a similar method of specifying these facilities through a command line option -d. If $sa_debug looks like a simple boolean (or is undefined), the traditional semantics still applies: a false prepends an 'info' to the list, while a true prepends 'info' and 'all' to the list of SpamAssassin debug facilities. Examples: $sa_debug = 0; # same as: $sa_debug = 'info'; $sa_debug = 1; # same as: $sa_debug = 'info,all'; $sa_debug = 'info,dns,async,bayes'; - pass the size of an original mail body as a 'supplementary attribute' to SpamAssassin for the benefit of a 'check_body_length' eval rule (new with SpamAssassin 3.4.0). The original mail body size may differ from the message as seen by SpamAssassin in case of truncation of large messages to mail_body_size_limit. - to the output of amavisd-status add a simple bar graph display (with an exponential-decay peak indicator) of the number of active processes; - if a module Unix::Getrusage is available, the timing report log entry (at log level 2) is enhanced: in addition to total elapsed time (wall clock) spent in processing a message, it also shows a sum of CPU user and system times spent by amavisd process and its spawned processes: old format example: size: 3815, TIMING [total 1901 ms] - ... new format example: size: 3815, TIMING [total 1901 ms, cpu 657 ms] - ... Additionally, a separate RUSAGE log entry is produced at log level 2, indicating resource usage spent by the last task. A field maxrss is a gauge (an absolute current value), all other fields are counters, so a difference between a previous and a current value is shown in the log. Each field value is a pair of numbers delimited by a plus: the first value corresponds to resource usage by the reporting amavisd child process, the second value corresponds to its spawned processes (e.g. file(1), gzip(1), etc.). Example (wrapped for clarity): size: 3815, RUSAGE minflt=10114+5223, majflt=0+0, nswap=0+0, inblock=0+0, oublock=9+0, msgsnd=819+9, msgrcv=211+3, nsignals=0+0, nvcsw=128+19, nivcsw=32+41, maxrss=164304+194012, ixrss=520+14016, idrss=66300+128392, isrss=24960+7680, utime=0.390+0.079, stime=0.079+0.108 See getrusage(2) Unix man page for details. OTHER - the 'amavisd genrsa' command will now warn if the requested DKIM signing key size is below 1024 bits, as required by RFC 6376; - on amavisd startup a check on available private DKIM signing keys (as declared by dkim_key) will now warn if a key size is below 1024 bits as required by RFC 6376, and log an information message if a key size is below a configured $dkim_minimum_key_bits size (defaults to 1024, currently 768 would still be a sensible value); - for purposes of DKIM-based whitelisting (@author_to_policy_bank_maps) and @signer_reputation_maps spam scores, valid signatures with public keys shorter than $dkim_minimum_key_bits bits (default 1024, equivalent to a lower limit as presently used by Google) are now ignored, with an informational message logged at level 1. To disable this check, set $dkim_minimum_key_bits to undef or to 0. - consider Unique local addresses (ULA) fc00::/7 non-public (RFC 4193), dropped site-local addresses fec0::/10 (deprecated by RFC 3879), adjusting the default setting of @mynetworks accordingly; - consider the "Shared Address Space" 100.64.0.0/10 non-public (RFC 6598); - adjust parsing the <zone_id> syntax of a scoped IPv6 address as per RFC 6874; - updated an AV entry for a Sophos Anti Virus: the scanning program used to be named 'sweep', now it is 'savscan'; thanks to mefiX; - updated a default value of @virus_name_to_spam_score_maps: updated entry for Doppelstern and added entries for Bofhland and PORCUPINE; - increase an arbitrary sanity limit on %smtp_reason_by_ccat strings from 100 to 450 characters (RFC 5321 allows 512 character reply lines); - relax testing file type of a configuration file, now a configuration may also be passed to amavisd through a named pipe (fifo), possibly facilitating testing or unusual deployments; - relax a requirement that a $QUARANTINEDIR directory needs to be writable: if $*_quarantine_method template settings include a subdirectory (e.g.: $spam_quarantine_method='local:W%P/spam/%m.gz'), such subdirectories must already exist and should be writable, but the top-level $QUARANTINEDIR directory need not be writable; - convert an IPv4-mapped IPv6 address into a plain IPv4 dot-quad form when found in Received header fields, in socket local or peer address, in ADDR field of an XFORWARD smtp extension command, or in an AM.PDP attribute client_address. See draft-cmetz-v6ops-v4mapped-api-harmful and draft-itojun-v6ops-v4mapped-harmful for potential caveats; - drop a support for direct queries to p0f v2, as it never worked well due to bugs in p0f v2. The p0f v3 changed the query protocol, but a query does not include port numbers (see RFC 6302), so using the p0f-analyzer.pl interface is still the only reliable approach; - use sysread() instead of read() when reading from /dev/urandom to avoid leaving entropy data in I/O buffers; also changed interface name to sub read_random_bytes(), which now reads directly into a scalar buffer, provided by an argument; - fix uniform random distribution when generating a random PIN for an attachment password (when releasing and $release_format is 'attach') (not a security issue); - added keepalive options to a call to Net::LDAP->new, recognized since Net::LDAP 0.53; a patch by Quanah Gibson-Mount; - removed option inet6 from a default LDAP setup ( $ldap_sys_default ), as Net::LDAP changed semantics in an incompatible way; presumably the Net::LDAP now does the right thing by default; suggested by Quanah Gibson-Mount; - use a low-level 0MQ interface instead of ZeroMQ / ZMQ abstractions; (i.e. ZeroMQ raw interface or ZMQ::LibZMQ3 or ZMQ::LibZMQ2); - taking advantage of 0MQ multi-part messages the number of IP packets transmitted is now radically decreased in favour of sending larger but fewer packets; - when generating 'Abuse Reporting Format (ARF) Reports' add a field Source-IP and use UTC timestamps in the Arrival-Date field, in accordance with RFC 6692; - drop (opportunistic) loading of a module Devel::SawAmpersand and testing the Devel::SawAmpersand::sawampersand(), variables $&, $` and $' are no longer slow since Perl 5.17.7, the PL_sawampersand became a constant, there is no longer any need of report it; - documentation update: remove vestiges of a field 'spam_modifies_subj' in README.sql, README.sql-mysql, this field was obsoleted in 2.7.0; thanks to Patrick Ben Koetter; --------------------------------------------------------------------------- June 30, 2012 amavisd-new-2.8.0 release notes Contents: COMPATIBILITY BUG FIXES NEW FEATURES SUMMARY NEW FEATURES - 0MQ NEW FEATURES - OTHER OTHER COMPATIBILITY - removed an old compatibility measure: default value of @banned_admin_maps was changed from: @banned_admin_maps = (\$banned_admin, \%virus_admin, \$virus_admin); to a more consistent: @banned_admin_maps = (\$banned_admin); The previous default value of @banned_admin_maps tried to maintain compatibility with versions before the setting was separated from its companion @virus_admin_maps. Now this compatibility is no longer considered necessary and contributes to some confusion, so it was dropped. See 2.4.0 and 2.2.1 release notes for previous changes to this setting. - quarantining to an mbox format file used to include a local time in an mbox separator line, which differs from RFC 4155 and common practices of using an UTC timestamp; a time zone of a timestamp in separator lines is now changed to UTC; BUG FIXES - fixed initial evaluation of dynamic (i.e. per policy bank) values of $enable_dkim_verification, $enable_dkim_signing and $bypass_decode_parts across all declared policy banks; these policy bank entries may be scalars of references to such; - finely adjust a message size for de-stuffed dots according to a size definition in RFC 1870; avoids occasional message size mismatch when using an antispam interface module SpamdClient (implementing client-side of a spamc/spamd protocol); - updated LDAP.ldif to match LDAP.schema; provided by Quanah Gibson-Mount; - updated AMAVIS-MIB.txt and amavisd-snmp-subagent: changed type of SNMP variables *MsgsSize* in the group amavisStats 7 from Counter32 to Counter64 for consistency with other *MsgsSize* variables in groups amavisStats 3 and amavisStats 9; See also the bug fixes section of 2.7.1 and 2.7.2 release notes. All fixes applied to 2.7.1 and 2.7.2 are incorporated in the 2.8.0 code. NEW FEATURES SUMMARY - For monitoring and statistics gathering purposes a new set of utilities and service processes is available based on a message passing paradigm, using a 0MQ (a.k.a. ZMQ, ZeroMQ, or Crossroads I/O) library. This replaces a functionally similar set of utilities based on a shared BerkeleyDB database, with a benefit of avoiding lock contention altogether. This can bring sigificant speedups, most pronounced on a host with many busy amavisd child processes. - Applied numerous fine-grained optimizations based on a NYTProf profiler results. Optimizations include a reduction in a number of generated Perl opcodes and similar micro-optimizations. This accounts for a large amount of small changes in the code. - Our current statistics (Q4 2011) shows that 80 % of messages are below 30.000 bytes, and 90 % of mail messages are below 100.000 bytes in size. As an optimization, messages below 100 KiB in size are now kept and processed in memory, including passing them more optimally to SpamAssassin 3.4.0. Some file activity is still there, but is much reduced. If $TEMPBASE also resides on an SSD disk (or a RAM disk), observed speedup between 2.7.2 and 2.8.0 was 3 to 8 percent on a busy host (with monitoring disabled, so as not to skew a measurement). - Use a module IO::Socket::IP if available, instead of dealing directly with low-level modules IO::Socket::INET and IO::Socket::INET6. The IO::Socket::IP is a Perl core module since Perl version 5.19.8; - choose more appropriate defaults if running on an IPv6-only host (like connecting to ::1 instead of 127.0.0.1 which may not exist); - amavisd-release now also supports connecting to amavisd over IPv6; - as a debugging aid it is now possible that a late event triggers full logging of earlier events that occurred during processing of a current mail message; - $enable_ldap setting is now dynamic, i.e. can be changed by a policy bank, which makes it possible to selectively disable LDAP lookups per policy bank; - optionally avoid persistent connections to SQL and LDAP servers; - it is now possible to disable calling an external file(1) utility but still have MIME parts decoding enabled; - added support in Amavis::SpamControl::ExtProg for an external spam scanner Bogofilter; - added locking options to @spam_scanners entries, to be used with external scanners which need but do not implement locking of their resources by themselves; - added a global configuration setting $sa_userprefs_file, which is passed on to SpamAssassin as a 'userprefs_filename' parameter at initialization; - added a subroutine iso8601_weekday(), potentially useful with partitioning; - added several new macros available to logging and notification templates; NEW FEATURES - 0MQ - added support for monitoring and auxilliary services, communicating with amavisd and among themselves through 0MQ sockets (also called ZMQ or ZeroMQ, or Crossroads I/O or XS). This method offers similar features as current services amavisd-nanny, amavisd-agent and amavisd-snmp-subagent, but use message passing paradigm instead of communicating through a shared Berkeley database. This avoids locking contention, so the gain can be significant for a busy amavisd setup with lots of child processes. New files in the package are: - amavis-mc is a master supervisor process ( master of ceremonies :), to be started at boot time as root, or as a user vscan/amavis. Currently its only function is to spawn three instances of amavis-services processes with dropped privileges, to monitor and restart them in case they fail, and to terminate them when itself if being terminated. Preferably this process should be started before amavisd and before amavisd-snmp-subagent-zmq, although things would eventually catch up even if this is not the case. This process must run on the same host as amavis-service processes. - amavis-mc_init.sh is an example FreeBSD-style startup/shutdown shell script for starting/stopping the amavis-mc process; - amavis-service implements three services, chosen by a command line argument. It should be running as user vscan/amavis (not as root!). All its instances are typically started/stopped automatically by the amavis-mc process with dropped privileges. A note for manual testing (started from a command line, not by an amavis-mc process): make sure to run amavis-service under the same UID as the amavisd is running. If 0MQ cannot write to a socket due to privilege violation, messages are silently dropped. Service processes as implemented by amavis-service must run on the same host as amavisd for two reasons: they communicate with amavisd child processes through a Unix socket, and at least some of these services need visibility of amavisd processes through signals (kill). At least the forwarding service must be running when amavisd is operational with $enable_zmq at true, otherwise amavisd child processes might eventually stall when their message queue fills up. Preferably amavis-service processes should be started before amavisd is started, although things would eventually catch up even if started late or restarted during operation. - amavisd-status is a user utility program, similar to amavisd-nanny, which connects to amavis-service 0MQ socket and displays a status of running amavisd child processes. This program communicates with amavis-service processes through an inet socket and can in principle run on a different host (in which case sockets must be bound to a publicly reachable interface, not loopback). The program can be started and stopped at any time, can run under any UID as long as it has access to a 0MQ socket $outer_sock_specs, and may run in multiple instances if necessary. - amavisd-snmp-subagent-zmq is a SNMP AgentX program, functionally equivalent to amavisd-snmp-subagent. It collects information from amavis-service processes and passes it as a MIB to an SNMP daemon. This process communicates with amavis-service processes through an inet socket and can in principle run on a different host (in which case sockets must be bound to a publicly reachable interface, not loopback). If access to the amavisMta MIB (1.3.6.1.4.1.15312.2.1.3) subtree is desired, the amavisd-snmp-subagent-zmq must run on the same host as Postfix in order to have access to its queue directories. In principle there could be more than one instance of the amavisd-snmp-subagent-zmq running at the same time, although this hardly serves any practical purpose. The old amavisd-agent utility does not currently have a 0MQ equivalent; use snmpbulkwalk with net-snmp and amavisd-snmp-subagent-zmq for similar functionality. Please see comments in amavis-service for details and configuration of sockets. To enable amavisd child processes to start sending their status and statistics information to amavis-service services, please set a configuration variable $enable_zmq to true in amavisd.conf: $enable_zmq = 1; Optionally a 0MQ socket can be changed, it defaults to: @zmq_sockets = ( "ipc://$MYHOME/amavisd-zmq.sock" ); The @zmq_sockets is a list of 0MQ sockets, so in principle amavisd processes can report their state to multiple instances of amavis-service. Both the 0MQ-based ($enable_zmq=1) and the BerkeleyDB-based ($enable_db=1) monitoring implementations can coexist: use one or the other, or both at the same time, or turn off both if monitoring is not needed. Required Perl modules are either: ZeroMQ, which interfaces with a version 2 of a libzmq library (in case of FreeBSD that would be ports net/p5-ZeroMQ and devel/zmq), or with a Crossroads I/O library libxs (now defunct); or ZMQ::LibZMQ2 and ZMQ::Constants modules with a version 2 of a libzmq library (or with a Crossroads I/O library); or ZMQ::LibZMQ3 and ZMQ::Constants with a version 3 of a libzmq library (FreeBSD ports: devel/zmq-devel). Although Crossroads I/O library is natively equivalent to a libzmq version 3 library, the ZMQ::LibZMQ3 perl module does not currently support interfacing with Crossroads I/O (libxs). NOTE: The Crossroads I/O project ceased developmenet in July 2012, to be replaced by nanomsg eventually (by the same author). The 0MQ is currenty (2012/2013) the best choice. Tested combinations of a Perl interface module with a message passing library: * with 0MQ ( http://www.zeromq.org/ ): ZeroMQ 0.21 + zeromq 2.2.0 ZMQ::LibZMQ2 1.01 + zeromq 2.2.0 ZMQ::LibZMQ3 1.00 + zeromq 3.1.0 * with Crossroads I/O ( http://www.crossroads.io/ ): ZeroMQ 0.21 + libxs 1.2.0 (zmq v2.1 compatible) ZMQ::LibZMQ2 1.01 + libxs 1.2.0 (zmq v2.1 compatible) ZMQ::LibZMQ3 1.00 + libxs 1.2.0 (native) (no, XS not supported by LibZMQ3) PERFORMANCE with 0MQ When scanning messages for spam (using SpamAssassin), a spam scan takes most of the processing time and resources, so replacing a BerkeleyDB-based monitoring with a 0MQ-based monitoring brings some speedup on a busy server, but the change is not dramatic. But as an extreme counter-example: when DKIM signing passed messages, with most other checks disabled, a speedup can be by a factor of 10. Synthetic benchmark: 7 KiB messages, 8 child processes, log level 2, CPU Intel Core i7-960 (4 cores, 8 threads), $TEMPBASE on an SSD disk, result: 19 mail messages per second with BerkeleyDB, over 200 mail messages per second with 0MQ, and still 130 msg/s with all checks (*except* spam scanning) enabled. SECURITY CONSIDERATIONS with 0MQ 0MQ libraries (zeromq or libxs) do not provide any application-level security beyond what is available with standard Unix or INET or INET6 sockets. This means that Unix-style inter-process sockets are protected by the usual file system's ownership and protection bits, and access to INET/INET6 sockets is only protected by interface binding and system firewall mechanisms. Communication between amavisd child processes and the forwarding service (amavis-services msg-forwarder) goes by default over a Unix-style socket, owned by UID vscan/amavis. Communication between utilities and service processes goes by default over an INET socket bound to a loopback interface, and as such is accessible to any process running on the same host, but is not accessible from other hosts. If access to these sockets from other hosts is desired, their binding should be changed to all or to ethernet interfaces, making them accessible to any host in the network, so host-firewall rules should be implemented if access needs to be restricted. Having said that, currently information passing through 0MQ sockets is limited to statistics and health status only, and does not affect operation of amavisd child processes, nor is any sensitive information passed around, so access to these sockets from unauthorized sources is not expected to pose a high security risk. 0MQ and IPv6 IPv6 is supported by zeromq library starting with version 3, and by libxs (any version). Because the application needs to pass information to a library about a type of sockets needed and there is no universal and backward compatible (with v2) way to do so, currently amavisd does not offer any configuration option to choose INET6 over INET on 0MQ sockets. This restriction is expected to be lifted in the next version. Currently on an IPv6-only host one can choose to use Unix-style sockets, or patch amavis programs to turn off a ZMQ_IPV4ONLY socket option (these are commented-out in present code). NEW FEATURES - OTHER - if a module IO::Socket::IP is available, amavisd will use this module to create its client-side inet or inet6 sockets, instead of using the low-level modules IO::Socket::INET and IO::Socket::INET6. This delegates some of the dirty details handling to IO::Socket::IP, such as using the getaddrinfo(3) system service to resolve host names, and dealing with dual-stack multihomed host names. If IO::Socket::IP is not available, the IO::Socket::INET or IO::Socket::INET6 are used directly instead, to preserve compatibility. Please use a fairly recent version of IO::Socket::IP, testing was done with versions 0.08 and 0.16. - added a subroutine read_cidr() which can read a Postfix style CIDR file, with a syntax interpreted according a Postfix cidr_table(5) man page. The subroutine returns a ref to an array by default (but can also produce a hash, and is able to add data to an existing array or hash). Typical use: @mynetworks_maps = ( read_cidr('/etc/postfix/mynetworks.cidr') ); @client_ipaddr_policy = map(($_,'MYNETS'), @mynetworks_maps); or: @mynetworks = @{ read_cidr('/etc/postfix/mynetworks.cidr') }; For details and more complex usage see leading comments in the read_cidr subroutine; - as a debugging aid it is now possible that a late event triggers full logging of earlier events that occurred during processing of a current mail message. This is implemented by writing all log events to a temporary file regardless of their log level and of the current $log_level setting. A later event can cause the captured temporary log to be copied to a regular log. Each child process keeps its own temporary log file open all the time, the file is rewound and truncated after each mail message processing and reused for the next capture, so its size rarely exceeds about 50 kB. Maintaining a temporary capture log is enabled by setting a configuration variable $enable_log_capture to true: $enable_log_capture = 1; Enabling a log capture costs a little bit of resources as amavisd needs to assemble and format all log messages regardless of their log level, not benefiting from early pruning of log entries not reaching the $log_level. Nevertheless the small overhead is quite acceptable when troubleshooting some rarely occurring problem and keeping $log_level permanently at the max is not acceptable due to sheer volume of debug logging. The captured log is read from a temporary file and copied to a regular log as log level 1 entries (i.e. at LOG_INFO syslog priority) if a dynamic variable $enable_log_capture_dump is true by the end of mail message processing. A chunk of captured log entries is preceded/ended by a log line: CAPTURED DEBUG LOG DUMP BEGINS CAPTURED DEBUG LOG DUMP ENDS and each such log entry has a prepended timestamp (hours, minutes, seconds with milliseconds) of a capture time. The $enable_log_capture_dump variable can be turned on directly by some debugging patch code, but is more conveniently loaded by activating a policy bank, e.g.: $policy_bank{'SLOW'} = { enable_log_capture_dump => 1, }; $policy_bank{'GOTCHA'} = { enable_log_capture_dump => 1, }; which can be loaded for example by a custom hook, e.g.: sub after_send { my($self,$conn,$msginfo) = @_; if (Time::HiRes::time - $msginfo->rx_time > 5.5) { Amavis::load_policy_bank('SLOW', $msginfo); } # or perhaps: if ($msginfo->sender =~ /some-regexp/) { Amavis::load_policy_bank('GOTCHA', $msginfo); } } Btw, the only purpose of having two different policy banks in the example is to be able to see at a glance in the log which one was activated. - the @decoders list is made a bit more flexible: the first entry in each tuple (a short type name) may be a scalar string as before, or may be a reference to a list of such names, in which case the tuple applies to all listed short types. Example: [['zip','kmz'], \&Amavis::Unpackers::do_unzip], which previously needed two entries: ['zip', \&Amavis::Unpackers::do_unzip], ['kmz', \&Amavis::Unpackers::do_unzip], - support an external decompressor lrzip for a .lrz format. Thanks to Jernej Porenta for a suggestion; - $enable_ldap setting is now dynamic, i.e. can be changed by a policy bank, which makes it possible to selectively disable LDAP lookups per policy bank. The LDAP code is loaded and a connection to an LDAP server is established if at least one policy bank has enable_ldap set to true (e.g. enable_ldap => 1 ) or a global $enable_ldap is true, but queries are disabled if currently active enable_ldap is false. Suggested by Tomislav Mihaliček; Example: $enable_ldap = 1; $policy_bank{'GUESTS'} = { enable_ldap => 0, }; or the other way around: $enable_ldap = 0; $policy_bank{'INBOUND'} = { enable_ldap => 1, }; - optionally avoid persistent connections to SQL and LDAP servers - at the expense of about 3 to 7 ms elapsed time for a reconnect. Persistent connections from mostly idling child processes consume database server resources (e.g. a TCP socket), and may become stuck when some intermediate stateful device like a firewall or a NAT decides to drop stale sessions. The behaviour is controlled by a setting $database_sessions_persistent: when true sessions remain open even after a SMTP session (from an MTA) has closed; when false sessions are closed after each SMTP session closedown. The default value is true for compatibility with earlier versions. Problem reported by Jernej Porenta; - it is now possible to disable calling an external file(1) utility but still have MIME parts decoding enabled: $file = undef; This may save some contents classification time, at the expense of losing results of a file(1) utility (i.e. short file type information) for banning checks. Disabling file(1) checks can be useful when most other checks are disabled too, e.g. in an amavisd instance whose only task is DKIM-signing, like after a mailing list manager fanout; - added Amavis::SpamControl::ExtProg support for an external spam scanner Bogofilter. An entry in @spam_scanners list for invoking the bogofilter program can be something like: @spam_scanners = ( ['Bogofilter', 'Amavis::SpamControl::ExtProg', 'bogofilter', [ qw(-e -v)], # -u mail_body_size_limit => 65000, score_factor => 1.0, ], # ['SpamAssassin', 'Amavis::SpamControl::SpamAssassin' ], ); The bogofilter interface code assigns a hard-coded score +5 to bogofilter's result status 'Spam', -5 to 'Ham' and 0 to 'Unsure'. This score is multiplied by score_factor (default 1 if not given) to produce the final spam score which is summed up with scores as contributed by other spam scanners in the @spam_scanners list. The 'X-Bogosity' header field will be inserted into forwarded message, unless prevented by a corresponding %allowed_added_header_fields entry. Based on a patch contributed by Stephen Davies. - added Amavis::SpamControl::ExtProg support for auto-learning on external spam scanners; experimental: works, but may change in future versions; *** to be documented ***; Suggested by Jernej Porenta; - added locking options to @spam_scanners entries, to be used with external scanners which do not implement database locking by themselves. Options are: 'lock_file', 'lock_type', 'classifier_lock_type' and 'learner_lock_type'. The 'lock_file' specifies a file name on which a flock(2) is acquired. A lock type can be 'shared', 'exclusive', or 'none'. The 'shared' acquires a LOCK_SH (shared read) lock, the 'exclusive' acquires a LOCK_EX (exclusive write) lock on a given file. A default lock type is 'exclusive' if the lock_type option is missing. If either a lock_file is absent or empty, or a lock type is 'none', then no locking is performed. Option 'classifier_lock_type' can override a generic 'lock_type' option when a scanner is requested to classify a message. Similarly, a 'learner_lock_type' option can override a generic 'lock_type' when a scanner is invoked for auto-learning. Example: ['CRM114', 'Amavis::SpamControl::ExtProg', 'crm', [ qw(-u /var/amavis/home/.crm114 mailreaver.crm --dontstore --report_only --stats_only --good_threshold=8 --spam_threshold=-8) ], learn_ham => [ qw(-u /var/amavis/home/.crm114 mailreaver.crm --good) ], learn_spam => [ qw(-u /var/amavis/home/.crm114 mailreaver.crm --spam) ], mail_body_size_limit => 65000, score_factor => -0.20, lock_file => '/var/amavis/crm114.lock', lock_type => 'shared', learner_lock_type => 'exclusive', ], - added a global configuration setting $sa_userprefs_file (undef by default), which is passed on to SpamAssassin as a 'userprefs_filename' parameter during its initialization. If 'userprefs_filename' parameter is nonempty, SpamAssassin tries to load a file with that name as user preferences configuration file (overriding systemwide settings), otherwise it tries to load a file '~/.spamassassin/user_prefs' if it exists. Suggested by Quanah Gibson-Mount; - added a subroutine iso8601_weekday() which takes a Unix time as an argument (seconds since 1970-01-01T00:00Z), and returns a weekday number based on local time: a number from 1 through 7, beginning with Monday and ending with Sunday, as specified in ISO 8601 (EN 28601). May be useful as a partition_tag for short-term cycling of a logging database storage (e.g. used by a pen-pals feature): $partition_tag = sub { my($msginfo)=@_; iso8601_weekday($msginfo->rx_time) }; - added a macro 'weekday', which expands to a weekday number of the current message reception time, as provided by a call to iso8601_weekday($msginfo->rx_time); - added a macro 'secret_id', which expands to a secret counterpart to mail_id, such that: b64_encode(md5(b64_decode(secret_id))) == mail_id. It is encoded in base64url (RFC 4648), e.g. laL-rCJ6MBTm (with a counterpart mail_id: XlZbJeFhn4OE). Typically used to authorize releasing from a quarantine. Suggested by Antoine Nguyen; - added a macro 'mail_id' as a synonym to a macro 'i', which is a long-term unique mail_id on this system, possibly used in log and in quarantine names, encoded in base64url (RFC 4648), e.g. XlZbJeFhn4OE (with a counterpart secret_id: laL-rCJ6MBTm); - added a macro 'log_id' as a synonym to a macro 'n', which is an internal log id (also called task id, am_id) as shown in the log and by amavisd-nanny, e.g. 58725-05-2; - added a macro 'hexenc', which encodes its string arguments as hex digits, high nybble first; - added macros 'b64enc' and 'b64urlenc', which encode their arguments as base64 strings, removing the final null padding '=' characters. The 'b64enc' encodes into a character set [A-Za-z0-9+/], while the 'b64urlenc' encodes into a character set [A-Za-z0-9-_] in accordance with RFC 4648; - added a macro 'body_digest', which expands to a digest (a hash) of a body of a mail message as computed by the algorithm chosen by a setting $mail_digest_algorithm (defaults to 'MD5', can be 'SHA-1' or 'SHA-256'). These are raw (non-encoded) bytes, not suitable for direct display. It is common to encode it with one of the macros: 'hexenc', 'b64enc', or 'b64urlenc', and possibly truncate it by a macro 'substr', e.g.: [:substr|[:b64urlenc|[:body_digest]]|0|9] The result of: [:hexenc|[:body_digest]] is the same as the result of a legacy macro call %b. - added a configuration setting $mail_digest_algorithm which chooses an algorithm name for generating a mail header digest and a mail body digest. If set to 'MD5' (case-insensitive) a module Digest::MD5 will be used, producing a MD5 digest (128 bits, 32 hex digits, 22 base64 characters). This is the fastest algorithm and is a default. Any other value is passed on to a module Digest::SHA as an argument to its method new(). The module Digest::SHA can produce digests of a SHA family: 160..512 bits, and accepts an algorithm name like 'SHA-1' or 'SHA-256' (see its documentation for details). The SHA-1 digest size is 160 bits / 40 hex / 27 base64 chars, while the SHA-256 size is 256 bits / 64 hex / 43 base64 characters. The generated digest may end up as part of a quarantine file name (%b in templates), or via macro 'b' or 'body_digest' in notification templates or a main log entry. OTHER - quarantining to a mbox format file was using mboxo rule for protecting a "From " line in a mail body, which made an original ">From " line indistinguishable from a protected From; now a mboxrd format rule is used, see http://en.wikipedia.org/wiki/Mbox - make MIME::Parser use $TEMPBASE as a temporary directory for scratch files instead of its default (which was /tmp, or failing over to a current directory, disregarding a TMPDIR environment variable). This can bring performance improvements if $TEMPBASE resides on an SSD or RAM disk and /tmp resides on a HDD; - distinguish an absence of an SMTP response from a negative SMTP response in an SMTP/LMTP client code for improved logging/debugging purposes; report delay time in case of a failure; - a default value for $inet_socket_bind now reflects the availability of socket protocol families INET (IPv4) and INET6 (IPv6): - if version of Net::Server is below 2.0: '127.0.0.1' - if both inet & inet6 are available: [ '127.0.0.1', '[::1]' ] - if only inet is available: '127.0.0.1' - if only inet6 available (IPv6-only host): '[::1]' Previously a default was always a '127.0.0.1'. - $forward_method, $notify_method and $requeue_method now default to an IPv6 address of a loopback interface ::1 instead of 127.0.0.1 when INET6 support is available and INET is unavailable (IPv6-only host); - remove an existing Authentication-Results header field only if we are capable of generating our own: keep it if $enable_dkim_verification is false or if $allowed_added_header_fields{'authentication-results'} is false; - add a field "Source-Port:" to "Abuse report format" (ARF) messages as per draft-kucherawy-marf-source-ports; - Avira SAVAPI av scanner: only log a warning instead of aborting when a QUIT command at the end of a session fails; - load all (both) applicable policy banks when %interface_policy contain both a "SOCK" entry and a Unix socket path name; and similarly when it contains both the "IPaddress:port" and a "port" entries. Previously the "SOCK" policy bank was not loaded when a socket path name entry existed in %interface_policy, and similarly a port-only -based policy bank was not loaded when a more specific "IPaddress:port" entry existed; - make use of a new SpamAssassin 3.4.0 option "skip_prng_reseeding" (description in the SpamAssassin Bug 6690); - no longer pre-load a module Mail::SpamAssassin::Plugin::SpamCop to avoid unnecessarily dragging-in modules Net::SMTP and Net::Cmd; - a spamd client code in Amavis::SpamControl::SpamdClient now obeys an option 'mail_body_size_limit' in a @spam_scanners entry and truncates a message passed to spamd (like other spam scanner interfaces do), instead of skipping a call to spamd. This interface module is mainly intended for testing spamd, or used with third-party software which uses the same spamc/spamd protocol. - modules Convert::TNEF is now made optional, instead of being required; do not load it if @decoders list is empty; - avoid a warning issued when encountering an empty ehlo-keyword in a response to an EHLO command (like on testing with a smtp-sink utility); - some fine-grained reduction in a number of generated opcodes and similar tiny optimizations; this accounts for numerous small changes in the code; - avoid some warnings issued by Test::Perl::Critic; - just in case: make sure that our SMTP responses at the incoming session are truly flushed to the socket and not stuck in a perlio I/O buffer; - updated 2.7.0 release notes, documenting that a policy bank may also be loaded based on a path name of a Unix socket receiving a connection; - updated and clarified schema and instructions in README.sql-pg based on suggested changes by Tim Howe; - fixed spelling mistakes in comments; - internal incompatible change: changed arguments and a result of a subroutine write_header; also, now it rewinds a message file by itself; --------------------------------------------------------------------------- August xx, 2012 amavisd-new-2.7.3 release notes BUG FIXES - fixed a bug in the SMTP client code, where the final SMTP status did not reflect a failure status of a DATA command from a back-end MTA. This caused a reception of a mail message to be confirmed but a message was then lost, as it could not be passed to a back-end MTA. The bug went unnoticed for years, as the commonly used MTA normally reject either at the MAIL FROM, RCPT TO or at the data-dot stage, but not at the DATA command. Reported by Deniska-rediska; - if a configuration file path as given through a command line option -c or as an argument to include_config_files() was not an absolute path, and that file contained an error, the do() would search the @INC list for alternative files of the same name, and report an unrelated error (typically: No such file or directory) instead of reporting the true reason for a failure; - fixed a regular expression in amavisd.conf for an 'Avast!' AV entry to properly extract a virus name; a patch by Ralf Hildebrandt; OTHER - updated an AV entry for a Sophos Anti Virus: the scanning program used to be named 'sweep', now it is 'savscan'; thanks to mefiX; - documentation update: remove vestiges of a field 'spam_modifies_subj' in README.sql, README.sql-mysql, this field was obsoleted in 2.7.0; thanks to Patrick Ben Koetter; --------------------------------------------------------------------------- June 30, 2012 amavisd-new-2.7.2 release notes BUG FIXES - a generated Received header field was missing the 'IPv6:' prefix in the TCP-info component of a 'by' subfield (as required by RFC 5321, section 4.1.3) when amavisd received a message over an IPv6 protocol; (btw, the TCP-info component of a 'from' subfield was correct); - changed data type of an SNMP variable LogRetries from C32 to C64 for consistency with the MIB; - updated AV entry 'AVG Anti-Virus' to consider status 403 continuation lines when searching for a virus name; suggested by Ralf Hildebrandt; OTHER - reduce a log level to 5 on a log message: Amavis::IO::RW: Error flushing on close: ... to avoid an innocent but sinister-looking warning when a pipe to a virus scanner is broken and needs to be re-established; reported by Stefan Jakobs; - updated an AV entry for 'F-Secure Linux Security' to version 9.14; options updated by Mika Ilmaranta, a patch by Tuomo Soini; - fix a Unix socket compatibility issue with Net::Server versions 2.000, 2.001 and 2.002, where a method NS_unix_path no longer exists. This method was re-introduced for compatibility reasons in 2.003. Reported by Paul MacKenzie; --------------------------------------------------------------------------- April 29, 2012 amavisd-new-2.7.1 release notes BUG FIXES - prevent rmdir() from failing with 'Invalid argument' on Solaris 10 when deleting a temporary directory: current working directory must not be within a directory which is about to be deleted; reported and diagnosed by Maciej Uhlig; - forwarding or quarantining through a 'pipe:' method failed with "Insecure dependency in exec while running with -T switch" when a sendmail command-line option -N was needed; reported by Andreas Schulze; - when multiple sockets are specified (e.g. in $forward_method) as a redundancy/failover mechanism, and SMTP session caching is enabled, a failed forwarding session does not clear a cached session, so all further attempts are stuck with the failed server, instead of picking a different server from the list; discovered by Michael Storz; - on establishing a SMTP session when multiple sockets are specified (e.g. in $forward_method) as a redundancy/failover mechanism, the random choice never picked the last socket in a list; discovered by Michael Storz; - fix defanging by mimedefang, it was failing with perl 5.10 or later due to an unhandled "Insecure dependency in sprintf" while logging the result if the $log_level was 2 or higher, or when debugging was enabled; thanks to Steve Scotter for a problem report; - fix defanging by Anomy::Sanitizer, it was failing with an error message: "mangling by anomy failed: replacement size 0, mail will pass unmodified"; - fix the 'xz' entry in a default @decoders list (in files amavisd.conf, amavisd.conf-default and amavisd); the first two variants ('xzdec' and 'xz') were glued together, so the xz decoder was only available if found under names 'unxz' or 'xzcat'; - provide a workaround for a bug [rt.cpan.org #64642] [rt.cpan.org #84879] in a perl module Encode, which gratuitously untaints a string when encoding or decoding it: https://rt.cpan.org/Public/Bug/Display.html?id=64642 https://rt.cpan.org/Ticket/Display.html?id=84879 (fixed in Encode 2.50); A module Scalar::Util is now required, which should not be a compatibility problem, as this module is a Perl core module since perl 5.8.0. - avoid the use of Encode::is_utf8 due to a bug in a perl module Encode as bundled with versions of Perl 5.8.0 to 5.8.8 (fixed in March 2007): Perl bug tracking: #32687: Encode::is_utf8 on tainted UTF8 string returns false https://rt.perl.org/rt3/Public/Bug/Display.html?id=32687 also referenced by #37170: https://rt.perl.org/rt3/Public/Bug/Display.html?id=37170 This is a re-manifestation of the same problem we had back in 2004, with a workaround provided by amavisd-new-2.2.1. Forgot that people are still using Perl 5.8 :) Reported by Peter Dieth; - fix a warning: _WARN: Invalid conversion in sprintf: "%a" - write informational messages during a stop/start/restart to stdout, instead of to stderr, avoiding unnecessary cron job messages; thanks to Cristian Seres, Sandro Janke and John Griffiths; also: https://bugzilla.redhat.com/show_bug.cgi?id=561389 - fix a syntactically incorrect 'Avira SAVAPI' av entry (missing closing bracket) in a sample configuration file amavisd.conf; - minor: get_body_digest incorrectly logged 8-bit body as 8-bit header; - no longer insist on a minimal version 2.22 of a module Digest::MD5, the 'clone' method is no longer needed since amavisd-new-2.7.0; - do not call $parser->max_parts($MAXFILES) with some old versions of MIME::Parser which did not yet provide this method; - pre-load a module File::Glob even with perl 5.8.0, otherwise autowhitelisting in SpamAssasssin may fail with "Insecure dependency"; - documentation: (files README.sql-mysql and README.sql-pg): fixed a field name "policy.unchecked_lover", previously incorrectly specified as "policy.unchecked_lovers_maps"; reported by TimH; - documentation: fixed the two SELECT examples in files README.sql-pg and README.sql-mysql, the field 'select' needs to be qualified with a table name: 'msgrcpt.content' to avoid ambiguity; reported by Gary V; - documentation bug in amavisd.conf-default: 'ESMTP' is not a valid setting for $protocol, just use 'SMTP' instead; reported by Pascal Volk; COMPATIBILITY - commented out the LHA entry in the default @decoders list and in do_executable(). The program seems to be unmaintained, was seen crashing and as such it may pose a security risk; pointed out by Thomas Jarosch; - due to popular demand, bring the 'spam-tag:' log line back to log level 2 (version 2.7.0 dropped it to log level 3) to retain compatibility with some log analyzers. Caveat: 'spam-tag' string is now entirely in lowercase. Suggested by Stefan Jakobs; OTHER - if a message is quarantined to more than one location using different quarantine methods, the SQL field msgs.quar_type indicates only the type of the last one. When archival quarantining is enabled this choice is unfortunate, as the primary quarantine type is more interesting than the permanent archival quarantine type. This is now reversed, the msgs.quar_type field now reflects the first quarantine type. Suggested by Patrick Ben Koetter. - SMTP session caching now no longer re-uses old sessions which are in use for more than a minute since their establishment; suggested by Michael Storz; - having the archive quarantine enabled should not be a sufficient reason to store information to SQL when $sql_store_info_for_all_msgs is off; Suggested by Patrick Ben Koetter. - ClamAV-clamd and ClamAV-clamd-stream av scanners: changed socket name in a sample configuration file amavisd.conf to /var/run/clamav/clamd.sock (previously the socket name was /var/run/clamav/clamd); this makes it compatible with a default socket name under several Linux distributions and under FreeBSD; suggested by Oliver Schinagl; - documentation updates; --------------------------------------------------------------------------- July 1, 2011 amavisd-new-2.7.0 release notes Contents: NEW FEATURES SUMMARY GENERAL COMPATIBILITY WITH 2.6.4 / 2.6.5 / 2.6.6 BUG FIXES SINCE 2.6.6 BUG FIXES SINCE 2.6.5 BUG FIXES SINCE 2.6.4 NEW FEATURES OPTIMIZATIONS OTHER CLEANING NEW FEATURES SUMMARY - significant improvements affecting a pre-queue content filtering setup (time limiting, warm/flying restart, ...) - requires Postfix 2.7.0 and SpamAssassin 3.3.0, or later; - new daemon amavisd-signer makes it possible to sign mail with DKIM signatures without requiring amavisd process to have access to private signing keys; - added support for the Sophos-SSSP, Avira SAVAPI and ClamAV clamd streaming protocols allows amavisd to communicate with these antivirus solutions; - allow specifying multiple (fail-over) back-end mailers for resubmission of messages from amavisd back to MTA; - support for Postfix 2.8.0 XFORWARD IDENT, passes a local message identifier (queue id) downstream to a post-queue content filter and back to Postfix; - speedup in data transfer rate on receiving large mail via SMTP/LMTP sessions by a factor of 3.9 for plain text sessions, and by a factor of 11 for encrypted (TLS) sessions; - recognize and insert header fields as prepared by SpamAssassin 3.3.0 or later through its 'add_header' configuration option; - a new setting allows a forward_method to be chosen based on a message content type and/or recipient address; this may be useful for outgoing mail routing purposes or to implement sender reputation schemes; - per-recipient (or per- policy bank) SpamAssassin configuration files or SQL configuration sets are supported (@sa_userconf_maps), and per-recipient SQL Bayes database usernames (@sa_username_maps); - new macros: client_helo, client_addr, client_port, client_addr_port, mime2utf8, rusage, ADDEDHEADERHAM, ADDEDHEADERSPAM, banned_parts_as_attr, actions_performed, new arguments to macros dkim, header_field, HEADER, YESNO and YESNOCAPS; - @listen_sockets setting offers a unified configuration of listening sockets; it may be configured directly, or the traditional way: the $inet_socket_port, $unix_socketname and $inet_socket_bind just add their entries to the @listen_sockets list; - lists of lookup tables (the @*_maps variables) can now contain explicit SQL and LDAP lookup objects as their elements, instead of (or in addition to) the implied SQL and LDAP lookups; - a new configuration variable @virus_name_to_policy_bank_maps allows loading of policy banks based on a virus name; - a new configuration variable $mail_id_size_bits allows setting the size of randomly generated mail_id and secret_id codes; - a new configuration variable $sql_store_info_for_all_msgs allows storing information on mail messages selectively just for quarantined messages; - added SNMP counters InMsgsStatus* which combine the final mail checking status with a direction of a mail flow; - optional transparent archival quarantine, retaining envelope recipient addresses on delivery to a dedicated SMTP server; GENERAL With a synergy of four solutions, using amavisd-new in a pre-queue filtering setup became a sensible / better behaved solution: - the "smtpd_proxy_options=speed_adjust" Postfix option, available since Postfix 2.7.0 (20091101), improves decoupling between SMTP clients and a content filter in a proxy setup, reducing the number of content filtering processes needed for the same mail load. With this option turned on, a Postfix SMTP server receives entire message before connecting to a before-queue content filter; - a master_deadline option and its API equivalent, available in SpamAssassin since version 3.3.0, allows for time limiting on lengthy rules checking, while still providing results when a time limit is exceeded; this makes it more suitable for time-sensitive setups like a pre-queue filtering setup; - reworked sub-task time limiting in amavisd, along with its counterpart solution in SpamAssassin, makes it better suited to a real-time nature of pre-queue filtering setups where one has no control over how long SMTP clients are willing to wait at the data-end stage; - a re-purposed command line option 'reload' now does a warm restart, keeping sockets available to an MTA client at all times, thus reducing a chance that an MTA would even notice a content filter's warm restart. Provided that required minimal versions of Postfix and SpamAssassin are available, on can try amavisd in a Postfix proxy setup. The $child_timeout setting needs to be radically reduced in this setup, matching the longest time most SMTP clients are willing to wait, and must be less than Postfix is willing to wait (smtpd_proxy_timeout), which by default is 100 s. A sensible value is somewhat less then a minute (e.g. 45 seconds). Even though RFC 5321 (section 4.5.3.2.6) recommends that clients SHOULD be willing to wait for 10 minutes at data-end stage, it is not uncommon that this recommendation is not adhered to. Note that a pre-queue filtering setup (along with its benefits) still has all its drawbacks, like the need for more filtering processes to accommodate mail arrival rate peaks (instead of averages), and much shorter and unpredictable (client-dependent) time limits. The new features of the three products only rise the thresholds where trouble starts, and make the whole setup better behaved. COMPATIBILITY WITH 2.6.4 / 2.6.5 / 2.6.6 - due to popular demand to reduce undesired and unintentional backscatter, defaults for the settings $final_spam_destiny and $final_banned_destiny were changed. Previously they both defaulted to D_BOUNCE, new defaults are: $final_virus_destiny = D_DISCARD; $final_banned_destiny = D_DISCARD; $final_spam_destiny = D_PASS; $final_bad_header_destiny = D_PASS; Please adjust to will. If you have these settings configured explicitly in a configuration file, this change of a default value does not affect you. For a pre-queue content filtering setup (smtp proxy or milter) a suitable value for undesired content is D_REJECT. For a post-queue filtering setup preferred choices are to tag-and-deliver (D_PASS), or to drop (D_DISCARD) and quarantine. It is still possible to use a D_BOUNCE setting, but please limit and monitor your backscatter. Due to a default setting of @viruses_that_fake_sender_maps the backscatter on viruses has been fully suppressed since amavisd-new-20021116 even with a D_BOUNCE. Backscatter on high-scoring spam has been controllable since amavisd-new-20030616-p8 by a family of settings @spam(_crediblefrom)_dsn_cutoff_level(_bysender)_maps, - several ancient configuration settings were removed or deactivated, see section CLEANING below; - a command line option 'reload' has been renamed to 'restart', corresponding to a shutdown followed by a normal (cold) start; while a command line option 'reload' has been re-purposed to function as a warm/flying restart. See below for details. Protection of some files may need to be examined (configurations files and DKIM private keys should be readable for group vscan/amavis and not writable by UID vscan/amavis); - a failure of all virus scanners no longer automatically tempfails the operation, but flags a message with a CC_UNCHECKED contents category (just like a failure of decoders/dearchivers), and allows the usual controls (*_destiny, *_quarantine_*) to be used to choose behaviour. The $virus_scanners_failure_is_fatal=1 reverts to previous behaviour, see below; - a default value of $hdr_encoding and $bdy_encoding has been changed from 'iso-8859-1' to 'UTF-8' which better suits reporting of banned parts; - default encoding for reading text templates from the tail of a file 'amavisd' has been changed to 'utf8', which allows replacing a default text by a non-ascii Unicode template, encoded as UTF-8; - when using SQL for logging/penpals: three fields need to be added to a table msgrcpt: msgrcpt.content, msgrcpt.rseqnum, msgrcpt.is_local, and one to a table msgs: msgs.originating . Semantics of msgrcpt.content is similar to msgs.content, but reflects individual recipient's settings (e.g. when a message is both banned and spam, a recipient with banning tests disabled will see a message as spam, while other recipient of the same message will consider it banned). The added field may also simplify queries by third party applications. The field msgrcpt.rseqnum uniquely identifies/enumerates recipients within each message, typically by assigning them sequential numbers starting with 1. The only purpose of this field is to make it possible to define a primary key for the table msgrcpt, which may be needed for some clustering/partitioning purposes. Amavisd itself does not require a primary key on this table. The field msgrcpt.is_local should be considered a boolean, its value can be: 'Y' ... yes, recipient is local, i.e. matches @local_domains_maps 'N' ... no, recipient does not match @local_domains_maps ' ' ... unknown - this is a default field value; amavisd always sets this field to either 'Y' or 'N'; The field msgs.originating should be considered a boolean, its value can be: 'Y' ... yes, message is originating from inside or from an authenticated roaming sender (the flag $originating was true); 'N' ... no, message is not submitted by our user ($originating was false); ' ' ... unknown - this is a default field value; amavisd always sets this field to either 'Y' or 'N', reflecting the $originating flag; A combination of msgs.originating and msgrcpt.is_local tells a direction a message is traveling: originating is_local N N open relay (probably misconfigured @local_domains_maps or $originating flag not set) N Y inbound message Y N outbound message Y Y internal message (inside or authenticated -> inside) Note that a direction is a per-recipient property, a multi-recipient message can be outbound for some recipients and internal for others at the same time. The following SQL directives can be used to add these new fields: ALTER TABLE msgrcpt ADD rseqnum integer DEFAULT 0 NOT NULL; ALTER TABLE msgrcpt ADD content char(1) DEFAULT ' ' NOT NULL; ALTER TABLE msgrcpt ADD is_local char(1) DEFAULT ' ' NOT NULL; ALTER TABLE msgs ADD originating char(1) DEFAULT ' ' NOT NULL; If a primary key on table msgrcpt is needed for some reason, try something like the following: *MySQL: UPDATE msgrcpt SET rseqnum=1+floor(999999999*rand()) WHERE rseqnum=0; ALTER TABLE msgrcpt ADD PRIMARY KEY (partition_tag,mail_id,rseqnum); *PostgreSQL: UPDATE msgrcpt SET rseqnum=1+floor(999999999*random()) WHERE rseqnum=0; CREATE UNIQUE INDEX msgrcpt_idx_primary ON msgrcpt (partition_tag,mail_id,rseqnum); If keeping a possibly customized copy of %sql_clause in a configuration file, entries 'ins_rcp' and 'upd_msg' will need to be updated accordingly. To facilitate transition from 2.6.6 to 2.7.0, it is possible to configure amavisd 2.7.0 to supply with SELECT and INSERT clauses a subset of parameters as used by 2.6.6. A configuration setting $sql_schema_version controls this backward compatibility. Its default value is 2.007000 . By setting it to a value below 2.007000 (such as 2.006006 or 2.006004) a subset of parameters as was used with a version 2.6.6 or 2.6.4 is selected. SQL clauses in $sql_clause{'upd_msg'} and $sql_clause{'ins_rcp'} need to be adjusted according to a chosen version of actual parameters. Below is an example of a required setting compatible with both amavisd-new 2.6.6 and 2.7.0, which lets amavisd 2.7.0 use an SQL schema of 2.6.6, which lacks the four newly added fields: our($sql_schema_version) if $myversion_id_numeric < 2.007000; $sql_schema_version = 2.006006; $sql_clause{'upd_msg'} = 'UPDATE msgs SET content=?, quar_type=?, quar_loc=?, dsn_sent=?,'. ' spam_level=?, message_id=?, from_addr=?, subject=?, client_addr=?'. ' WHERE partition_tag=? AND mail_id=?'; $sql_clause{'ins_rcp'} = 'INSERT INTO msgrcpt (partition_tag, mail_id, rid,'. ' ds, rs, bl, wl, bspam_level, smtp_resp) VALUES (?,?,?,?,?,?,?,?,?)'; Note that this is only provided to facilitate transition. Please add the new fields on an opportunity, then remove the above settings from your configuration file and restart amavisd. - SQL fields msgs.content and msgrcpt.content used to encode a content type CC_SPAMMY as 's', and CC_MTA as 't'. With default case-insensitive queries on a data type CHAR it was not possible to distinguish between lowercase 's' (= CC_SPAMMY) and uppercase 'S' (= CC_SPAM), so the CC_SPAMMY is now encoded as 'Y', and CC_MTA as 'T' (just in case). Please adjust your management tools if necessary. - please check SQL data types on fields msgs.mail_id, msgs.secret_id, msgrcpt.mail_id and quarantine.mail_id in existing databases, these must be treated case-insensitively - see details further down, please search further down for "must be treated case-insensitively"; - SQL clause $sql_clause{'sel_quar'} no longer uses a coalesce() function (introduced in amavisd-new-2.6.2) which attempted to deal with NULL quarantine.partition_tag or with undefined $partition_tag, when releasing a message from an SQL quarantine - but payed the price of not using an index. If releasing from an SQL quarantine is desired, either ensure there are no (old) records in a table 'quarantine' with a NULL partition_tag (e.g. replace such fields with a 0, and don't leave $partition_tag undefined in amavisd.conf - set it to 0 for example when partitioning is not needed), or assign a former clause to $sql_clause{'sel_quar'} in amavisd.conf : $sql_clause{'sel_quar'} = 'SELECT mail_text FROM quarantine'. ' WHERE coalesce(partition_tag,0)=coalesce(?,0) AND mail_id=?'. ' ORDER BY chunk_ind'; Thanks to Michael Scheidell and Thomas Gelf for pointing out the inefficiency. - if using Petr Rehor's amavisd-milter to call amavisd, please update it to version 1.5.0 (or later), as earlier versions did not accept a new attribute 'log_id', which is now included in a response from amavisd; - a sample configuration file amavisd.conf-sample was removed from the package - it hasn't been seriously updated for years, and it contained lots of aged or distracting information; - old helper programs amavis.c and amavis-milter.c are no longer distributed with the package, along with the entire helper-progs subdirectory. As a milter client please use the more modern 'amavisd-milter' package by Petr Rehor, available at http://sourceforge.net/projects/amavisd-milter/ - old AM.CL protocol is no longer supported; it was provided for compatibility with versions of AMaViS pre-dating amavisd-new, along with its client programs: old helper programs amavis.c and amavis-milter.c. Handling of release requests and milter requests through AM.PDP protocol remains unaffected; - a sample AM.PDP client program for mail submission to amavisd which was previously distributed as 'helper-progs/amavis.pl' has been renamed to 'amavisd-submit' and slightly modernized. It provides partial functional compatibility with a very early AMaViS client program amavis.c . It takes a message on stdin, copies it to a temporary file, passes its name to amavisd daemon using AM.PDP protocol, and based on the response adjusts its exit status value so that an invoking script or program may decide whether to deliver the mail message or not; - mail_id and secret_id are now composed of characters from a character set [ A-Z, a-z, 0-9, -, _ ] instead of [ A-Z, a-z, 0-9, +, - ] (i.e. now uses underline instead of a plus) to conform to RFC 4648 base64url specification, thus making it potentially easier to specify an id in various GUI/API interfaces without a need for quoting a plus. The change is also reflected in a choice of quarantine file names. Compatibility with releasing of old quarantined messages is retained; - relationship between mail_id and secret_id has changed and is now: mail_id = encode_base64(md5(decode_base64(secret_id))) (i.e. md5 is applied to 9 raw bytes of a secret id) while previously it was: mail_id = encode_base64(md5(secret_id)) (i.e. md5 was applied to 12 base64 characters of a secret id) Releasing from a quarantine still understands old relationship and old base64 encoding character set for compatibility, so no problems are expected even when releasing a mix of old and new quarantined messages. The change may potentially affect some third party application. - caching of virus and spam check results based on a mail body hash has been removed. It was very beneficial years ago when virus storms were common and spam was not personalized. Nowadays (2011) the feature barely pays for itself (savings are comparable to additional processing needed), and is incompatible with per-recipient spam checks (as introduced with this version), and incompatible with DKIM verification on locally originating and signed mail being returned from a mailing list. Rather than trying to fit a square peg into a round hole, the feature is now dropped. Associated configuration variables are still declared for compatibility, but have no effect: $enable_global_cache, $virus_check_negative_ttl, $virus_check_positive_ttl, $spam_check_negative_ttl, $spam_check_positive_ttl - a default value for $lock_file is now undefined instead of the former default value "$helpers_home/amavisd.lock"; an undefined value lets Net::Server choose a suitable temporary file (POSIX::tmpnam) for 'flock' serialization on socket accept(); - updated (rarely used) AV entries 'Sophos SAVI', 'Mail::ClamAV' and 'av_smtp' in an incompatible way (they now use ask_daemon interface instead of a dedicated subroutine), please update your AV entries according to the new sample file amavisd.conf; - internal: spam_level() and spam_tests() are no longer properties of a message but are now a property of each recipient, which makes possible per-recipient spam checking settings (e.g. rules, bayes username, ...); - internal: a delivery_method() is no longer a property of a message, but is now a property of each recipient, which makes per-recipient forwarding possible; - internal: a load_policy_bank() takes one additional argument $msginfo, which is passed on to any policy bank's ACTION routine if a policy bank has one; BUG FIXES SINCE 2.6.6 - take a more cautious approach on keeping evidence on an SMTP session transaction state when feeding a message back to MTA. Under certain abnormal circumstances an MTA could respond to end-of-data with a temporary failure but retain an active transaction state while amavisd would assume the transaction was closed, leading to a 'MAIL transaction in progress' failure on the next message using the same cached SMTP session. Now amavisd considers a transaction state to be unknown when there is any doubt and closes a session instead of caching it, unless the transaction is reliably known to be closed. Problem reported by Ralf Hildebrandt. BUG FIXES SINCE 2.6.5 All bug fixes that were developed during a 2.7.0 developement cycle have been backported to the 2.6 branch and released as 2.6.6. They are all documented in release notes of the 2.6.6 release. BUG FIXES SINCE 2.6.4 All bug fixes and some compatibility measures that were developed during a 2.7.0 developement cycle have been backported to the 2.6 branch and released as 2.6.5. They are all documented in release notes of the 2.6.5 release. NEW FEATURES - a command line option 'reload' has been renamed to 'restart', while keeping its semantics: to stop a currently running daemon, and then promote a process to become a new daemon. This makes a complete and independent restart with all its benefits: can start a chrooted daemon, can access config files or DKIM signing keys accessible only to root, can open sockets/ports otherwise restricted to root, can change inet and unix socket locations, their bindings and port numbers. It will also reset SNMP counters in a database, if it is enabled. A downside is that during a restart existing sockets are closed, so until new sockets are re-established an MTA client experiences connection failures, which is particularly disruptive in a pre-queue MTA setup. - a command line option 'reload' has been re-purposed to function as a warm restart: it now sends a HUP signal to a running daemon, then exits. A running daemon upon receiving a HUP signal will clone its sockets, clear their 'close-on-exec' flag, then restart itself through exec(). A reborn daemon inherits open sockets, does a normal startup (loading perl modules and config files), then reassociates inherited sockets with configured inet and Unix socket names, which is why these must not be changed in a configuration file between reloads. Sockets remain open and available to clients during the whole warm-restart period, requests are queued by kernel (queue size is configurable through $listen_queue_size, defaulting to SOMAXCONN, or there may not be any queues at all when an IP stack is using SYN cookies), so apart from a delay in connection establishment, an MTA client will not notice a restart as long as the IP stack is willing to accept new sessions (as controlled by listen queue size or SYN cookies). This makes the reloading method particularly suitable for pre-queue filter setups. A downside is that a HUP-ed daemon has already dropped root privileges during its first start, so it must restart as a nonprivileged user (typically 'vscan' or 'amavis'), which rules out its ability to chroot, and requires that configuration files, DKIM signing keys files, and perl modules must be readable by this GID or UID, otherwise a restart fails and a daemon process no longer exists. Depending on a version of perl and operating system in use, it might be necessary to specify an absolute path to amavisd on the initial start. To debug warm-restart problems it may be useful to first try a warm restart on a non-daemonized process (started manually as: amavisd foreground, or: amavisd debug), so that potential errors on stderr are visible. A sensible protection of configuration files and files with DKIM keys is to set their group ownership to vscan (amavis) and UID ownership to root, and mode to 0640 (u=rw,g=r,o=). A need for non-root accessibility of DKIM signing keys can be avoided by using a new signing service daemon included with this release (see further down: amavisd-signer). One additional feature of a warm reload is that SNMP counters in a database (visible through amavisd-agent or amavisd-snmp-subagent) are not reset to zero, unlike the restart which clears them. - on stop, restart or reload, currently busy child processes are left to complete their current task instead of being abruptly stopped. This minimizes a disruption experienced by MTA. - added a client-side and server-side support for the IDENT attribute of a Postfix XFORWARD smtp command (available since Postfix version 2.8.0). The attribute allows passing of a local message identifier (MTA queue id) downstream from a front-end MTA to a post-queue content filter and back to a back-end MTA. Amavisd makes this information available through an existing macro %Q (which was previously non-empty only in milter setups), and as such the information appears in the log when using a default amavisd log template. This information is also passed back to a re-entry MTA if it announces a support for this attribute (enabled on a back-end smtpd service with an option smtpd_authorized_xforward_hosts), so the log entries are now easier to correlate in a post-queue filtering setup: back-end MTA: postfix/smtpd[72995]: 553261D1CB0: client=localhost[::1], orig_queue_id=2F5971D1CA3, orig_client=... post-queue content filter: amavis[20341]: (20341-15) Passed CLEAN ... Queue-ID: 2F5971D1CA3, queued_as: 553261D1CB0 front-end MTA: postfix/lmtp[73130]: 2F5971D1CA3: ... relay=127.0.0.1[127.0.0.1]:10024, status=sent (250 2.0.0 from MTA(smtp:[::1]:10025): 250 2.0.0 Ok: queued as 553261D1CB0) - support Postfix 2.9 long queue IDs (enable_long_queue_ids=yes) as available since postfix-20110321 by adjusted default values of $log_short_templ and $log_verbose_templ templates; - improved support for pre-queue content filtering setups: reorganized time limiting on processing to obey more strictly a deadline time, which is the sum of $child_timeout and a timestamp at the moment of a reception of a complete message (SMTP data-end time). The deadline time is also passed to SpamAssassin, which since version 3.3.0 supports a 'master_deadline' option and can gracefully terminate its processing on a time limit, while still providing results collected so far. The setting $sa_timeout is now retired: the variable is still declared for backward compatibility, but has no effect. Instead, the time available for spam scanning is automatically determined from $child_timeout, taking into consideration the actual time left till the deadline; - $child_timeout and $smtpd_timeout settings are now dynamic, i.e. can be changed by a policy bank, which makes it possible to support (on different ports) both the pre-queue and post-queue (e.g. fallback) clients by the same amavisd daemon; - a new configuration variable $soft_bounce (also a member of policy banks) turns rejects, bounces and discards into a temporary failure when true; this is potentially useful as a short-term safety net when testing configuration changes on a low-traffic server; - added an AV entry and supporting code for Sophos-SSSP, implementing the client side of the Sophos SSSP protocol, talking to a savdid daemon (a replacement for Sophie) using its native protocol; - added an AV entry and supporting code for AVIRA SAVAPI protocol, implementing the client side of the protocol, talking to a savapi daemon; - added an AV entry for clamdscan which can serve as a useful backup scanner, connecting as client to a remote clamd; the supplied alternative config file should specify the host IP and port number where clamd is running using TCPAddr and TCPSocket options; suggested by Michael Scheidell; - added an AV entry for ClamAV clamd streaming which can serve as a main or backup scanner, connecting as client to a remote clamd. The client side implements clamd zINSTREAM command, batched in a zIDSESSION / zEND group. This approach is comparable to what is implemented in clamdscan and is somewhat less efficient than passing only a directory name to clamd, but has an advantage that the clamd daemon need not have direct access to amavisd temporary files, and may even be running on a remote host. The cost of this flexibility is additional data transfer. Suggested by Michael Scheidell. - lists of lookup tables (the @*_maps variables) can now contain explicit SQL and LDAP lookup objects as their elements, instead of (or in addition to) the implied SQL and LDAP lookups. A new configuration setting $lookup_maps_imply_sql_and_ldap controls whether the SQL and LDAP lookup objects are implicitly prepended to list in @*_maps variables (when true), or not (when false). The default value is 1 for compatibility with previous versions. Regardless of the $lookup_maps_imply_sql_and_ldap setting, the @*_maps lists of lookup tables/objects may now contain explicit lookup objects for arbitrarily named SQL fields and LDAP attributes. This provides more flexibility: the order of lookups is now configurable (previously SQL and LDAP lookup objects were prepended to lists and thus always looked up first), and the names of SQL fields or LDAP attributes can now be specified as arguments to SQL and LDAP lookup objects (previously field and attribute names were hardwired into code). Three shorthand functions are available for creating SQL lookup (query) objects: q_sql_s, q_sql_n, q_sql_b, and three for creating LDAP lookup (query) objects: q_ldap_s, q_ldap_n, q_ldap_b. The _s, _n and _b suffixes imply a data type of the expected result: a string, a numeric value, and a boolean. Due to Perl's forgiveness a string data type can in most cases be used as a number or as a boolean and may be used when data type conversion and value normalization is not necessary or when a data type is not known. Here are some examples: @spam_kill_level_maps = ( { # a hash-type lookup object 'user1@example.com' => 8, '.example.org' => 7.5, }, q_ldap_n('amavisSpamKillLevel'), # an LDAP lookup object q_sql_n('spam_kill_level'), # an SQL lookup object $sa_kill_level_deflt, # a constant-type pseudo-lookup object ); @spam_subject_tag2_maps = ( q_sql_s('subject_tag'), ); In addition to simple scalar arguments (a field or attribute name), these six lookup object-creating functions can take as their argument a listref of field or attribute names, or a hashref where hash entry values are SQL field names (or LDAP attribute names), and hash entry keys are the result data names. Lookups resulting from such lookup objects will return a hashref of key/value pairs instead of a single scalar result. This is currently only useful in the @dkim_signature_options_bysender_maps list of lookups which expects such hash results (sets of data names and their values, i.e. entire records). The listref argument is just a shorthand notation which can be used in place of a hashref when field names (or attribute names) are the same as the desired result data names. The following alternatives are equivalent: q_sql_s( { 'd' => 'd', 's' => 's', 'ttl' => 'ttl' } ) q_sql_s( { d => 'd', s => 's', ttl => 'ttl' } ) # perl shorthand q_sql_s( [ 'd', 's', 'ttl' ] ) q_sql_s( [qw(d s ttl)] ) # perl shorthand Example (artificial, not necessarily useful): @dkim_signature_options_bysender_maps = ( q_sql_s( ['d', 's', 'ttl'] ), q_ldap_s( ['d', 's', 'a'] ), q_ldap_s({ d => 'sdid', s => 'amavisSelector', a => 'amavisDkimAlg' }), { 'postmaster@example.com' => { a => 'rsa-sha1', ttl => 7*24*3600 }, '.' => { a => 'rsa-sha256', ttl => 30*24*3600 }, }, ); - a new configuration variable $sql_store_info_for_all_msgs when turned off requests storing information on mail messages selectively just for quarantined messages. At the same time turning this setting off also disables pen pals lookups. A default value is 1 (true) as before, indicating that information on all messages is to be stored into tables msgs, msgrcpt and maddr when @storage_sql_dsn is enabled, thus ensuring long-term uniqueness of mail_id and proper operation of pen pals lookups; - a new program is included with a package: amavisd-signer. It is a DKIM signing service daemon for amavisd. It uses an AM.PDP protocol lookalike to receive a request from amavisd and provides two services: choosing a signing key, and signing a message digest with a chosen DKIM private key. Amavisd uses this signing service when a $dkim_signing_service setting is defined and nonempty, and $enable_dkim_signing is true. For each mail message meeting the basic requirements for signing (originating, nonspam and not infected), the first request sent to a signing service passes some information about the message (its author, sender, recipients) and expects the service to choose and provide a suitable signing domain and selector (and optional signature options) when a signing key is available and the service considers it appropriate to sign the message. If the response does not provide a signing domain and selector, amavisd falls back to consulting its own settings (a dkim_key() set of signing keys, and @dkim_signature_options_bysender_maps). The second stage of signing occurs when a signing key has been uniquely identified during the first stage, i.e. when its signing domain and a selector have been determined. Amavisd computes a message digest according to DKIM specifications and passes it to the signing service, along with the signing domain and selector name. The signing service computes the signature and returns it as a 'b' attribute (corresponding to a 'b' tag of a DKIM signature), from which amavisd assembles the signature header field and inserts it into a message. The signing service may still choose not to sign at this stage, e.g. when a private key corresponding to the requested signing domain and selector is not available. If a signing service is not available or cannot sign, amavisd falls back to its own configured list of signing keys ( dkim_key() ) for backward compatibility. The main reason for separating the signing act from the main amavisd daemon is to make it possible to do the DKIM signing without letting amavisd have access to private keys - following the minimal 'need-to-know' security principles. For example, amavisd may be started as non-root or restarted from a jail, while the independent amavisd-signer process remains the only process with access to private signing keys (by running as root or under a separate UID or GID or with an access to a crypto device). Additional benefit is that more complex decisions on which signing key to use for which mail message can be delegated to the signing service, which can be customized (through code changes or replacing it altogether) without touching the main amavisd daemon. To let amavisd use a signing service, specify the signing service's IP address and TCP port number in amavisd.conf, e.g.: $dkim_signing_service = '127.0.0.1:20203'; matching the $inet_socket_bind and @listen_sockets settings near the beginning of the 'amavisd-service' file, then start the signing service and restart amavisd daemon. Currently all the settings for amavisd-service are contained in its file, no external configuration file or command line options are available at present. - a constant D_TEMPFAIL has been added to a set of allowed final_*_destiny values; mostly intended for completeness and testing; - a new setting @listen_sockets offers a unified configuration of listening sockets. This list may be configured directly, or the traditional way: the $inet_socket_port and $unix_socketname just add their entries (if any) to the @listen_sockets list, and $inet_socket_bind provides a default binding IP address for inet or inet6 ports. Each socket specification may either be a unix socket path (as in $unix_socketname), or an inet or inet6 socket specification (a binding IP address as in $inet_socket_bind, combined with a port number $inet_socket_port, delimited by a colon, e.g. '127.0.0.1:10024', '[::1]:10024', '10024'. When only a port number is specified without an IP address, the binding address defaults to $inet_socket_bind, which in turn (if left undefined), defaults to all interfaces. An 'unspecified' binding address '0.0.0.0' implies any socket of the inet family (IPv4), while an 'unspecified' address '::' implies any socket of an inet6 (IPv6) family. Depending on the operating system and its settings an inet6 socket may or may not be able to also accept inet connections. To be able to listen on inet6 (IPv6) sockets requires version 2.0 (or later) of the module Net::Server, or a patched Net::Server 0.99. Example: @listen_sockets = ( '10024', "$MYHOME/amavisd-proxy.sock", '0.0.0.0:10010', '127.0.0.1:10012', '10026', '9998', '[::1]:10028' ); - SMTP and LMTP client code now accepts a listref of peer socket specifications, or a single scalar specification as before. This allows for a failover in case some server is down or refuses connections. It also provides a simpleminded load balancing between next-hop (re-entry) MTA servers, as the selection from a list is random. Session caching still works, so if a recently used SMTP/LMTP session is still open, it will be reused, in which case no server randomization takes place for as long as the established session remains open. Typical configuration variables where this feature is available are: $forward_method, $notify_method, $resend_method, $release_method, and $requeue_method, but only when the specified protocol is smtp: or lmtp:, (not pipe:, local:, sql:, bsmtp:). Example: $forward_method = [ 'smtp:[::1]:10025', 'smtp:[127.0.0.1]:10025', 'smtp:*:10025' ]; $notify_method = [ 'smtp:*:*', 'smtp:192.0.2.10:10025' ]; It is assumed that the protocol specification scheme (e.g. 'smtp:') of all entries in a list is the same. Mixing different protocols in the same list of alternatives is not allowed; - when a message is being released from a quarantine as an attachment ( $release_format = 'attach' ), it is now possible to wrap this attached message into a password-protected ZIP archive to prevent accidental or automatic opening of the possibly malicious original message. The chosen password is included in the first plain text MIME part, along with an explanation / instructions for a recipient. Note that the purpose of password scrambling is only to prevent an accidental or automatic opening of an attachment. It is not intended to be a strong mechanism for keeping messages secret. There is no point in providing excessively long / strong passwords. A template for this first plain-text MIME part can be changed as before by assigning a new text to $notify_release_templ (or modifying a default template near the end of the file 'amavisd'). Three new configuration variables are added, all three are also members of policy banks: $attachment_email_name, $attachment_outer_name, and $attachment_password. Their default values are: $attachment_password = ''; # no password and no ZIP wrapping $attachment_email_name = 'msg-%m.eml'; $attachment_outer_name = 'msg-%m.zip'; The $attachment_email_name is a template for forming a name of a file, which is then inserted into a zip archive. This name will be seen as a filename containing an original mail message when a recipient unzips the archive. The supplied string may contain placeholders, the same placeholders are recognized as for filename templates used to control quarantining. For the record, here is a complete list of placeholders currently recognized: %P => $msginfo->partition_tag %b => $msginfo->body_digest %m => $msginfo->mail_id %n => $msginfo->log_id %i => iso8601 timestamp of a message reception time by amavisd %% => % The $attachment_outer_name is a template for forming a name of a ZIP archive which will be attached to a message. This name will end up in a MIME sub-header field of the attachment, and as such will be used as a filename when a recipient saves the attachment (without unzipping it). The supplied string may contain the same placeholders as above. The $attachment_password setting can be: . an empty string, in which case no ZIP wrapping will occur and no passwords are applied; also the settings $attachment_email_name and $attachment_outer_name have (currently) no effect, as the attachment is not a ZIP archive but the original message itself; this is a default setting for compatibility with earlier versions; . a fixed static string, in which case an original message is wrapped in a ZIP archive and the archive is encrypted with this fixed string password; . an undefined value ( $attachment_password = undef ), in which case a 4-digit random password (PIN) is internally generated for each quarantine release, the rest is the same as with a fixed string; . a subroutine reference, in which case the supplied subroutine is called (in a scalar context), passing it a $msginfo object as the only argument; the subroutine is expected to return a password as a string, or die if it cannot do its job; the returned value is then treated as one of the three cases above, i.e. an empty string disables zipping, an undefined value invokes internal PIN generating code, and any other value is taken as a password for encrypting the archive. Example use: $release_format = 'resend'; # choices: plain, resend (default), attach or: $release_format = 'attach'; $attachment_password = ''; # no archive, just plain attachment or: $release_format = 'attach'; $attachment_password = undef; # internally generated 4-digit random PIN or: $release_format = 'attach'; $attachment_password = 'fooBAR'; # fixed password or: $release_format = 'attach'; $attachment_password = sub { my($msginfo) = @_; my $str = qx'pwgen -N 1 -n -B -s 6'; die "pwgen failed, exit status: $?" if $?; die "pwgen returned empty result" if $str eq ''; return $str; }; As the $attachment_password is a member of policy bank, it is possible to configure amavisd to listen to release requests on two TCP ports, for example using one to release unencrypted false-positive spam messages, and the other to release possibly problematic infected messages. - updated amavisd-snmp-subagent and AMAVIS-MIB.txt by adding ten user-specifiable 64-bit counters and ten user-specifiable 32-bit gauges; Counters are placed into OID tree under 1.3.6.1.4.1.15312.2.1.1.17 and named UserCounter1..UserCounter10, whereas gauges are placed under 1.3.6.1.4.1.15312.2.1.1.18 and named UserGauge1..UserGauge10. A custom hook or a policy bank ACTION hook can be used for adjusting their values by calling snmp_count64() routine, e.g.: Amavis::Util::snmp_count64('UserCounter3', 'UserCounter9'); or: Amavis::Util::snmp_count( ['UserCounter4', 1234, 'C64'] ); - updated amavisd-snmp-subagent and AMAVIS-MIB.txt by adding the following counters, all placed under 1.3.6.1.4.1.15312.2.1.1 : .19.1 InMsgsStatusAcceptedAll same value as .2.7 .19.2 InMsgsStatusAcceptedInbound .19.3 InMsgsStatusAcceptedOutbound .19.4 InMsgsStatusAcceptedInternal .19.5 InMsgsStatusAcceptedOriginating .19.6 InMsgsStatusAcceptedOpenRelay .20.1 InMsgsStatusRelayedUntaggedAll (no equivalent) .20.2 InMsgsStatusRelayedUntaggedInbound .20.3 InMsgsStatusRelayedUntaggedOutbound .20.4 InMsgsStatusRelayedUntaggedInternal .20.5 InMsgsStatusRelayedUntaggedOriginating .20.6 InMsgsStatusRelayedUntaggedOpenRelay .21.1 InMsgsStatusRelayedTaggedAll (no equivalent) .21.2 InMsgsStatusRelayedTaggedInbound .21.3 InMsgsStatusRelayedTaggedOutbound .21.4 InMsgsStatusRelayedTaggedInternal .21.5 InMsgsStatusRelayedTaggedOriginating .21.6 InMsgsStatusRelayedTaggedOpenRelay .22.1 InMsgsStatusDiscardedAll same value as .2.9 .22.2 InMsgsStatusDiscardedInbound .22.3 InMsgsStatusDiscardedOutbound .22.4 InMsgsStatusDiscardedInternal .22.5 InMsgsStatusDiscardedOriginating .22.6 InMsgsStatusDiscardedOpenRelay .23.1 InMsgsStatusNoBounceAll same value as .2.10 .23.2 InMsgsStatusNoBounceInbound .23.3 InMsgsStatusNoBounceOutbound .23.4 InMsgsStatusNoBounceInternal .23.5 InMsgsStatusNoBounceOriginating .23.6 InMsgsStatusNoBounceOpenRelay .24.1 InMsgsStatusBouncedAll same value as .2.11 .24.2 InMsgsStatusBouncedInbound .24.3 InMsgsStatusBouncedOutbound .24.4 InMsgsStatusBouncedInternal .24.5 InMsgsStatusBouncedOriginating .24.6 InMsgsStatusBouncedOpenRelay .25.1 InMsgsStatusRejectedAll same value as .2.12 .25.2 InMsgsStatusRejectedInbound .25.3 InMsgsStatusRejectedOutbound .25.4 InMsgsStatusRejectedInternal .25.5 InMsgsStatusRejectedOriginating .25.6 InMsgsStatusRejectedOpenRelay .26.1 InMsgsStatusTempFailedAll same value as .2.13 .26.2 InMsgsStatusTempFailedInbound .26.3 InMsgsStatusTempFailedOutbound .26.4 InMsgsStatusTempFailedInternal .26.5 InMsgsStatusTempFailedOriginating .26.6 InMsgsStatusTempFailedOpenRelay For compatibility, the following counters appear in the MIB at two locations (both locations present the same value): .19.1 or .2.7 InMsgsStatusAcceptedAll (.20.1 - InMsgsStatusRelayedUntaggedAll) (.21.1 - InMsgsStatusRelayedTaggedAll) .22.1 or .2.9 InMsgsStatusDiscardedAll .23.1 or .2.10 InMsgsStatusNoBounceAll .24.1 or .2.11 InMsgsStatusBouncedAll .25.1 or .2.12 InMsgsStatusRejectedAll .26.1 or .2.13 InMsgsStatusTempFailedAll The value of a counter .2.8 InMsgsStatusRelayed is a sum of: .20.1 InMsgsStatusRelayedUntaggedAll .21.1 InMsgsStatusRelayedTaggedAll Their semantics is documented in AMAVIS-MIB.txt . Suggested by Patrick Ben Koetter. - a policy bank may now provide a custom hook as a hash key 'ACTION'. On loading a policy bank whose ACTION key has an associated value being a subroutine reference, the supplied subroutine is called when a policy bank is loaded, before its remaining keys/values are copied to the current setting. The action routine is passed two arguments: a $msginfo (a ref to an object containing all the information about a message being processed), and a policy name being loaded. Note that $msginfo may be undef if a policy bank is loaded early - before a $msginfo object is created, such as with policy banks associated with a port number or with client's IP address; Example use: $policy_bank{'TRUSTED_BOOKSHOPS'} = { bypass_spam_checks_maps => [1], spam_lovers_maps => [1], ACTION => sub { Amavis::Util::do_log(2,'Buying a book?'); Amavis::Util::snmp_count64('UserCounter2'); }, }; @author_to_policy_bank_maps = ({ 'amazon.com' => 'TRUSTED_BOOKSHOPS', 'amazon.co.uk' => 'TRUSTED_BOOKSHOPS', 'amazon.de' => 'TRUSTED_BOOKSHOPS', }); - a new configuration variable @virus_name_to_policy_bank_maps has been introduced. It allows loading of policy banks based on a virus name as reported by virus scanners. Reported names converted to spam by a @virus_name_to_spam_score_maps are no longer treated as virus names and as such are not eligible to @virus_name_to_policy_bank_maps. The @virus_name_to_policy_bank_maps is a list of lookup tables. A lookup key is each virus name as reported by any virus scanner. A result of a lookup is expected to be a string containing a comma-separated list of policy bank names. Nonexistent policy banks are ignored. Duplicate names are merged into a single name. The most suitable lookup mechanisms are a regexp lookup and a hash lookup, as these are able to provide an arbitrary user-specifiable result (unlike a list-based (ACL) lookup, which can only provide a boolean value). Suggested by Patrick Ben Koetter. Example use: @virus_name_to_policy_bank_maps = ( new_RE( # a regexp lookup [ qr'^(W32/MyDoom|W32/Netsky|Mal/BredoZp)' => 'VIRUS,MASS_VIRUS' ], [ qr'\bEICAR\b'i => 'EICAR_TEST' ], ), 'VIRUS', # constant (pseudo)lookup, catchall for any other virus name ); $policy_bank{'VIRUS'} = { ACTION => sub { Amavis::Util::snmp_count('UserCounter1') }, }; $policy_bank{'EICAR_TEST'} = { log_templ => $log_short_templ . ', EICAR test message, not to worry', final_destiny_by_ccat => { CC_VIRUS() => D_BOUNCE }, }; $policy_bank{'MASS_VIRUS'} = { # mute everything using a big hammer final_destiny_by_ccat => { CC_VIRUS() => D_DISCARD }, warnsender_by_ccat => { REPLACE => 1 }, warnrecip_maps_by_ccat => { REPLACE => 1 }, quarantine_method_by_ccat => { REPLACE => 1 }, admin_maps_by_ccat => { REPLACE => 1 }, newvirus_admin_maps => [], log_templ => 'MASS VIRUS DROPPED, ' . $log_templ, ACTION => sub { Amavis::Util::snmp_count('UserCounter2') }, }; - a policy bank may now be loaded based on a path name of a Unix socket receiving a connection. Example use: @listen_sockets = ( "$helpers_home/amavisd.sock1", "$helpers_home/amavisd.sock2", "$helpers_home/amavisd.sock3", ); $interface_policy{"$helpers_home/amavisd.sock1"} = 'UX-S1'; $interface_policy{"$helpers_home/amavisd.sock2"} = 'UX-S2'; $interface_policy{"$helpers_home/amavisd.sock3"} = 'UX-S3'; $policy_bank{'UX-S1'} = { ... }; $policy_bank{'UX-S2'} = { ... }; $policy_bank{'UX-S3'} = { ... }; - transparent archival quarantine is a special case of archive quarantining which retains all recipient addresses unmodified in an envelope of a message directed to a quarantine. It makes sense when $archive_quarantine_method specifies protocols 'smtp:' or 'lmtp:' or 'bsmtp:' and a dedicated server is used which guarantees these quarantined messages will *not* be delivered to recipients in the envelope. Transparent archiving is used when $archive_quarantine_to (actually the @archive_quarantine_to_maps) results in a reserved string '%a' for all recipients. Think of the '%a' as a placeholder in a replacement string, being substituted by a full original recipient address. There may be other substitution placeholders available in the future, equivalent to placeholders %l, %d, etc. in SQL query templates. Example: $archive_quarantine_method = 'smtp:127.0.0.1:7777'; $archive_quarantine_to = '%a'; @archive_quarantine_to_maps = (\$archive_quarantine_to); or: $archive_quarantine_method = ['smtp:[::1]:7777', 'smtp:127.0.0.1:7777']; @archive_quarantine_to_maps = ( { '.example.com' => '%a', '.example.net' => '%a', '.example.org' => 'quarantine@example.org', '.' => undef, } ); The envelope sender address of messages sent to an archival quarantine is still controlled by the $mailfrom_to_quarantine setting as before. When this value is undef (which is a default) the envelope sender address remains unchanged - is the same as in a received message. Any other value replaces the original sender address, so an empty string implies a null return path. When delivering quarantine messages to a dedicated SMTP server it must be ensured that the receiving server will not bounce or reject quarantine messages or deliver them to recipients specified in the SMTP envelope! If a receiving SMTP server announces a DSN capability in its response to EHLO, amavisd will add option NOTIFY=NEVER with each recipient to prevent potential backscatter. This is an additional safeguard to prevent potential backscatter, therefore it is recommended that the receiving quarantine server implements and announces the DSN capability. Specifying an empty string for the $mailfrom_to_quarantine achieves the same effect (a null return path implies NOTIFY=NEVER) thus preventing backscatter, but loses original sender address in the envelope. Suggested by Patrick Ben Koetter. - as a convenience, two pre-defined logging templates are provided: $log_short_templ and $log_verbose_templ. The former is the same as was a default $log_templ in previous versions, the later is quite verbose and provides most of the interesting information about a message. An initial value of $log_templ is taken from the $log_short_templ. To change $log_templ, either assign a new template directly as before, or, as a shorthand if $log_verbose_templ is appropriate, just assign it, e.g.: $log_templ = $log_verbose_templ; - added a configuration variable @debug_recipient_maps. Similarly to @debug_sender_maps, a debug level logging is temporarily turned on for the duration of processing of this message when a recipient address matches a list of lookup tables @debug_recipient_maps; suggested by Patrick Ben Koetter; - internal: a delivery_method() is now a property of a recipient instead of being a property of a message as a whole. This makes per-recipient forwarding method selection possible. When recipients of a multi-recipient message specify different forwarding methods, a message is forwarded in multiple transactions, one for each unique delivery_method() setting; i.e. recipients are clustered into sets with the same delivery_method setting and a message for each subset of recipients if forwarded as one transaction; - finer custom control over the forwarding method is available through a before_send() custom hook which may override the $r->delivery_method(...) for all or just some of the recipients with whatever forwarding method specification is suitable - for example a next-hop server's IP address or its port number can be chosen based on spam score or based on a sender domain or some other characteristics of a message or of a recipient; - a new configuration variable @forward_method_maps (along with making a delivery_method() a property of a recipient instead of being a property of a message) makes per-recipient forwarding method selection possible; suggested by Ralf Hildebrandt; Example: @forward_method_maps = ({ # use lowercase keys with hash-type lookups! 'user@example.com' => 'smtp:[::1]:10025', '.sub1.example.com' => 'smtp:[::1]:10026', '.sub2.example.com' => 'smtp:[::1]:10027', '.example.net' => 'smtp:[127.0.0.1]:10025', '.example.org' => [ 'smtp:[192.0.2.9]:125', 'smtp:[2001:db8::f]:125' ], '.' => $forward_method, }); - a new configuration variable %forward_method_maps_by_ccat allows the forward_method_maps to depend on content type and allows per-recipient specification of a forward method, such as specifying a next hop MTA's IP address and port number. This offers new possibilities to control mail routing for purposes like implementing sender reputation schemes which dynamically choose an SMTP source IP address (typically of outgoing mail) based on the contents of a mail message or based on recipient's e-mail address or domain. This needs to be complemented by a suitable configuration of an MTA, such as Postfix 2.7.0 or later. The default is to use the $forward_method setting, ensuring compatibility. There is no need to specify entries for content types which are not being forwarded (often: CC_VIRUS, CC_BANNED, CC_SPAM). Example use: $forward_method = 'smtp:[127.0.0.1]:10025'; %forward_method_maps_by_ccat = ( CC_BADH.',3', [ 'smtp:*:10027' ], CC_BADH.',4', [ 'smtp:*:10027' ], CC_BADH.',5', [ 'smtp:*:10027' ], CC_BADH.',6', [ 'smtp:*:10027' ], CC_BADH.',8', [ 'smtp:*:10027' ], CC_SPAMMY, [ 'smtp:[192.0.2.22]:10025' ], CC_CATCHALL, sub { ca('forward_method_maps') }, ); - added a global configuration setting $allow_preserving_evidence, defaults to true. Turning it off disables preserving temporary files (as evidence) in case of trouble, which is potentially useful for unattended and unmonitored operation. The setting has no influence on preserving evidence in case of @debug_sender_maps or @debug_recipient_maps triggering, which always preserves evidence; - an entry for CC_UNCHECKED was added to %admin_maps_by_ccat, defaulting to @virus_admin_maps. Hence administrator notifications are also sent for messages which cannot be decoded (e.g. are encrypted or contain a mangled archive) if virus administrator notifications are enabled. To turn off sending administrator notifications for unchecked contents: delete $admin_maps_by_ccat{&CC_UNCHECKED}; - to avoid a need to directly manipulate the *_by_ccat settings, a set of individual configuration variables associated with CC_UNCHECKED contents category was added, with their default values compatible with earlier versions of amavisd: $final_unchecked_destiny = D_PASS; $unchecked_quarantine_method = undef; $unchecked_quarantine_to = 'unchecked-quarantine'; @unchecked_quarantine_to_maps = (\$unchecked_quarantine_to); LDAP attribute: amavisUncheckedQuarantineTo SQL field: unchecked_quarantine_to - a failure of all virus scanners no longer automatically tempfails the operation, but flags a message with a CC_UNCHECKED contents category (just like a failure of decoders/dearchivers), and allows the usual controls (*_destiny, *_quarantine_*) to be used to choose behaviour; for example: $final_unchecked_destiny = D_TEMPFAIL; $unchecked_quarantine_method = 'local:unchecked/%m.gz'; To revert to a previous behaviour where a failure of all virus scanners resulted in a temporary failure, set the $virus_scanners_failure_is_fatal to true, e.g.: $virus_scanners_failure_is_fatal = 1; The setting $virus_scanners_failure_is_fatal is a member of policy banks. - support decompression of a .xz file format and legacy .lzma file formats through XZ Utils ( http://tukaani.org/xz/ ) if an entry with a decoding program is found in the @decoders list; it defaults to finding a program 'xz' or 'xzdec' in the $path ; - added two new functions: iso8601_year_and_week() and iso8601_yearweek() to accompany the existing function iso8601_week(); they all provide a week-of-the-year number (ISO 8601 / EN 28601, 1..53, in local time zone) given a Unix timestamp (seconds since 1970-01-01T00:00Z) as an argument, optionally together with the corresponding year number. The result is an integer or a pair of integers as follows: $w = iso8601_week($unix_time); # e.g. 49 $yw = iso8601_yearweek($unix_time); # e.g. 201049 ($y, $w) = iso8601_year_and_week($unix_time); # e.g. (2010,49) Semantics is equivalent to PostgreSQL extract(week from ...), and to MySQL week(date,3). These functions can be useful for assigning to a $partition_tag (in amavisd.conf), e.g.: $partition_tag = sub { my($msginfo)=@_; iso8601_week($msginfo->rx_time) }; or: $partition_tag = sub { my($msginfo)=@_; iso8601_yearweek($msginfo->rx_time) }; or based on a day of a week for short-term cycling (Mo=1, Tu=2,... Su=7): $partition_tag = sub { my($msginfo)=@_; ((localtime($msginfo->rx_time))[6]+6)%7+1 }; (a note from a future: starting with 2.8.0 the following is equivalent: $partition_tag = sub { my($msginfo)=@_; iso8601_weekday($msginfo->rx_time) }; ) Suggested by Michael Scheidell. - the two placeholders %k and %a in templates for SQL lookup clauses $sql_clause{'sel_policy'} (i.e. $sql_select_policy) and $sql_clause{'sel_wblist'} (i.e. $sql_select_white_black_list) were augmented by four new placeholders: %l, %u, %e, and %d, potentially facilitating forming of more complex SQL queries; suggested by Marco Fretz. The following replacements are made: %a -> exact/unmodified e-mail address (same as the first entry in %k) %l -> full unmodified localpart (all up to, but not including the '@') %u -> lowercased username (a localpart without extension) %e -> lowercased address extension (including a delimiter), if any %d -> lowercased domain (without '@') %k -> a list of lookup strings, as before (see below) For example, given an e-mail address: User+Foo@Sub.Example.COM the placeholders would be substituted by: %a User+Foo@Sub.Example.COM %l User+Foo %u user %e +foo %d sub.example.com %k User+Foo@sub.exAMPLE.COM user+foo@sub.example.com user@sub.example.com user+foo user @sub.example.com @.sub.example.com @.example.com @.com @. - per-recipient (or per- policy bank, or global) SpamAssassin configuration files or SQL configuration sets are now supported (the @sa_userconf_maps setting, a policy.sa_userconf SQL field). A multi-recipient message whose recipients map to different configuration sets will be checked by calling SpamAssassin multiple times, once for each unique SpamAssassin configuration set. A configuration set is either a filename, or a set of SQL records obtained from SpamAssassin's user_scores_dsn SQL database by calling its method load_scoreonly_sql(). A lookup on a list of lookup tables @sa_userconf_maps may return undef or an empty string implying no user preferences file, or may provide a file name (absolute path, or relative to $MYHOME) of a SpamAssassin's 'user preferences' configuration file, or may start with a string 'sql:' which implies loading user preferences from a user_scores_dsn SQL database (as declared in a SpamAssassin's configuration file) for a username provided by a lookup on @sa_username_maps (see further down). SpamAssassin will be requested to load a user preferences configuration through its read_scoreonly_config() or load_scoreonly_sql() method, which otherwise (in spamd) serves to load user's .spamassassin/user_prefs file or SQL preferences when switching users. See SpamAssassin documentation file sql/README for SQL details. SpamAssassin's SQL database is only consulted if user_scores_dsn is declared in a SpamAssassin configuration file, and the @sa_userconf_maps returns a string starting with 'sql:' (case insensitive, the rest of the string is currently ignored). If a username as provided by a lookup on @sa_username_maps equals the username under which amavisd was started, SpamAssassin's SQL preferences for that username will not be loaded - it is assumed that preferences for a default username are empty, i.e. that it uses a default SpamAssassin configuration. Each time that currently loaded configuration needs to be replaced by another or restored to a systemwide default, an initial SpamAssassin configuration is restored through SpamAssassin's copy_config() method. Note that saving an original SpamAssassin configuration, loading a user configuration, and restoring to the original configuration does not come cheap: it can take 200 ms for a load and restore, and 370 ms for the initial saving of the configuration (saving is only done once per child process, and only if needed). Saved configuration can occupy additional 2 MB of virtual memory, so use the feature sparingly. No penalty occurs until a child process does its first loading of a user configuration, so rarely activated or inactive policy banks or per-recipient setting using this feature do not cause any additional processing or occupy additional memory. According to SpamAssassin documentation, a user preferences file or SQL preferences can include scoring options, scores, whitelists and blacklists, etc. If 'allow_user_rules' is enabled (local.cf), then user preferences file can also include rule definitions and privileged settings - but not administrator settings. The feature is only available since SpamAssassin 3.3.0. Example: @sa_userconf_maps = ( { 'user1@example.com' => '/etc/mail/spamassassin/special_user_config', '.example.org' => 'sql:', } ); Based on a suggestion by Alexander Wirt and initially based on his patch; - added a global configuration setting $sa_num_instances with a default value of 1, which is the only sensible setting for sites not using per-recipient SpamAssassin configuration switching (as described in the previous section). The $sa_num_instances controls the number of Mail::SpamAssassin objects (instances) created by a parent amavisd process during a startup. Each SpamAssassin instance does its own initialization (loading of rules and configuration settings) during a program startup and occupies a sizable portion of virtual memory (like 7 MB on a 64-bit platform with SA 3.4 rules). When switching SpamAssassin configurations (@sa_userconf_maps), and given more than one instance of the Mail::SpamAssassin object, amavisd has a choice of picking an instance which may already have loaded a selected user configuration file previously, and thus save some time by not having to store and reload SpamAssassin state again. This may be beneficial for example when a sizable portion of users use a default SpamAssassin configuration, while other users need a per-user or per-domain preferences settings; Note that as of SpamAssassin 3.3.2 some features (like compiled rules) are global and not a property of a SpamAssassin instance object. The problem is tracked in the SpamAssassin project as Bug 6236. Until this is resolved please consider the feature experimental. - per-recipient (or per- policy bank) SpamAssassin SQL database usernames are supported (setting @sa_username_maps, a policy.sa_username SQL field). This makes it possible to implement per-user or per-user-group or per-domain Bayes databases when SpamAssassin is configured to keep its Bayes database on an SQL server. It also makes it possible to load per-recipient SpamAssassin preferences (configurations) from an SQL database (as described in a previous section). Switching between Bayes usernames is cheap compared to switching between SpamAssassin configuration files. A multi-recipient message whose recipients map to different usernames will be checked by SpamAssassin multiple times, once for each unique username; Example: @sa_username_maps = ( { 'user1@example.com' => 'user1', 'user2@example.com' => 'user2', '.example.com' => 'user_ex', } ); - passes a value of $originating flag to SpamAssassin through its suppl_attrib argument in a $spamassassin->parse call; it is expected that this information would be treated by SpamAssassin 3.4.0 similarly to msa_networks; - a new configuration variable $mail_id_size_bits allows setting the size of randomly generated mail_id and secret_id codes which are used to identify a message on releasing it from a quarantine, and are used as a key when logging to SQL (pen pals) or storing to quarantine. The variable specifies a length of mail_id in bits, and must be an integral multiple of 24 (i.e. must be divisible by 6 and by 8). The mail_id is represented externally as a base64url-encoded string of $mail_id_size_bits / 6 characters, and internally as a string of $mail_id_size_bits / 8 octets. The default value is 72 bits, as in previous versions. Sensible values are 48, 72 and 96 bits. See entry "introduce a concept of 'mail_id'" in amavisd-new-2.3.0 release notes for probability analysis of collisions. The default size should fit all practical current needs. The size of SQL fields msgs.mail_id, msgs.secret_id, msgrcpt.mail_id and quarantine.mail_id may need increasing to accommodate $mail_id_size_bits/6 characters if using a non-default value of $mail_id_size_bits. See also the next entry regarding a type for these fields. - SQL fields msgs.mail_id, msgs.secret_id, msgrcpt.mail_id and quarantine.mail_id must be treated case-insensitively. A suitable data type for these fields in PostgreSQL is bytea, and varbinary in MySQL (of size 12 or 16 characters). In order not to lose entropy in mail_id, and not to increase a probability of collisions, please check existing database schema and adjust as necessary, either a data type, or chose a case-sensitive collation setting. See README.sql-pg and README.sql-mysql for an ALTER command to change data type of these fields. - added optional SQL and LDAP lookups for @spam_tag3_level_maps (sql field: policy.spam_tag3_level, ldap attribute: amavisSpamTag3Level), and for @spam_subject_tag3_maps (sql field: policy.spam_subject_tag3, ldap attribute: amavisSpamSubjectTag3); suggested by Thomas Johnson; - added the following LDAP settings to $default_ldap: sslversion, clientcert, clientkey, cafile, capath, verify, sasl, sasl_mech, sasl_auth_id, localaddr, scheme, inet6; allows bind authentication with a certificate or SASL and allows connections to an LDAP server over IPv6; based on a patch by Christian Roessner; - added macros ADDEDHEADERHAM and ADDEDHEADERSPAM, which expand to newly generated header fields which SpamAssassin prepared for insertion into a header section, in case the message is eventually declared to be nonspam or spam respectively; this information is available from SpamAssassin since version 3.3.0; - macro 'dkim' now recognizes two additional keywords: 'selector', 'sig_sd', and 'newsig_sd', see README.customize for details; in particular, the 'newsig_sd' allows to add information on newly applied signature to a main log entry, and is now included in a default log template; - added a macro 'mime2utf8' which takes a string as its first argument, and an optional truncation length as the second. The string is decoded as a MIME-Header string (understands Q or B character set encodings like =?iso-8859-2?Q?...?=, =?koi8-r?B?...?=) and is converted to UTF-8, optionally truncated to the specified size at clean UTF-8 boundaries, and returned as a result. Suggested by Bastian. The macro is useful in a logging template or in notification templates to decode Subject or From header fields, e.g.: [? [:header_field|Subject]||,\ Subject: [:dquote|[:mime2utf8|[:header_field|Subject]|100]]]# - added a macro 'client_helo', which provides a client-supplied EHLO/HELO domain name of the original SMTP session. The information is obtained through an XFORWARD extension to an SMTP protocol as provided by Postfix, or through a 'helo_name' attribute in an AM.PDP request. Add something like: , helo=[:client_helo]# to the log template if it needs to be logged. Suggested by Xueron Nee; - added a macro 'client_addr' which is a synonym for macro 'a'; the information is obtained through an XFORWARD extension to an SMTP protocol as provided by Postfix, or through a 'client_address' attribute in an AM.PDP request; if neither of these are available, the client's IP address is parsed from a topmost Received header field; - added a macro 'client_port', yields a TCP source port number of an original SMTP session; the information is obtained through an XFORWARD extension to an SMTP protocol as provided by Postfix, or through a 'client_port' attribute in an AM.PDP request. See RFC 6302; - added a macro 'client_addr_port' which combines a client's IP address and a TCP source port number (if available) of an original SMTP session; it is similar to: \[[:client_addr]\]:[:client_port] or when a port number is not available: \[[:client_addr]\] This macro is now included in a default main log template, so the TCP source port number is logged along its IP address. This information is useful in reporting abuse (e.g. client behind a NAT), troubleshooting, forensics and law enforcement. If this information is not desired, one may assign a customized template to the $log_templ configuration variable. See RFC 6302: Logging Recommendations for Internet-Facing Servers. Suggested by Rok Potočnik. - added a macro 'banned_parts_as_attr' and an associated per-recipient attribute banned_parts_as_attr(); it provides the same information as a macro 'banned_parts' and its associated attribute banned_parts(), but uses a different syntax, possibly facilitating parsing and reporting names or types or location of banned parts. The following example illustrates the difference. A result is a single string in both cases, wrapped here for clarity, and shows a path in a message tree of a banned leaf node: a macro 'banned_parts' can yield: multipart/mixed | application/octet-stream,.rar,Setup1.1.rar | .exe,.exe-ms,setup.exe while a macro 'banned_parts_as_attr' yields: P=p003,L=1,M=multipart/mixed | P=p002,L=1/2,M=application/octet-stream,T=rar,N=Setup1.1.rar | P=p007,L=1/2/4,T=exe,T=exe-ms,N=setup.exe for the same banned part in a message. The single-character attribute names are unchanged from previous versions. For documentation, here is a legend: P: part's base name, i.e. a file name in a ./parts/ temporary directory L: part's location (path) in a mail tree (branch enumeration, top-down) M: MIME type as declared in MIME header fields of a message T: short part's content type according to a file(1) utility and mapped through @map_full_type_to_short_type_maps N: declared part names (none, one or more), as declared in MIME header fields or in an archive (tar, zip, ...) A: part's attributes as follows: U=undecodable, C=crypted, D=directory, S=special(device), L=link - macro 'header_field' and its alias 'HEADER' now has an optional third parameter (index), which chooses the header field in case of multiple header fields of the same name; the default (-1) is to return the last (bottommost) occurrence, as before; see README.customize for details; - added a macro 'actions_performed', which expands into a comma-separated list of words: Accepted, Relayed(Untagged), RelayedTagged, Discarded, Rejected, Bounced, NoBounce or TempFailed, followed by a mail flow direction word: Inbound, Internal, Outbound or OpenRelay. For brevity the 'RelayedUntagged' status appears in this list as 'Relayed'. Additionally, the list may include words Quarantined and Archived. For multirecipient messages it is possible that the list includes more than one combination. The purpose of this macro is to augment the bare-bones 'Passed CLEAN' or 'Blocked SPAM' in the main log entry. For this purpose the default log template now includes this macro call. If the additional information is not desired in the log, please assign a customized template to the $log_templ configuration variable. Some examples of the new log entries: Passed CLEAN {RelayedOutbound}, ... Passed CLEAN {RelayedInbound}, ... Passed CLEAN {RelayedInternal,RelayedOutbound}, ... Passed SPAMMY {RelayedTaggedInbound}, ... Blocked SPAM {RejectedInbound,Quarantined}, ... Blocked INFECTED (Mal/BredoZp-B) {DiscardedInbound,Quarantined}, ... Semantics of entries in the 'actions_performed' list corresponds to the newly added SNMP variables 1.3.6.1.4.1.15312.2.1.1.19 - .26 (with the exception that 'RelayedUntagged' counter is abbreviated in this macro as 'Relayed'). Please see their detailed description in a file AMAVIS-MIB.txt . - added a macro 'rusage', which expands to a resource usage entry as provided by a system service getrusage(2); the argument to the macro should be one of the field names in the structure rusage (see getrusage(2) man page), e.g. ru_utime, ru_stime, ru_maxrss, ru_ixrss, ru_idrss, ru_isrss, ru_minflt, ru_majflt, ru_nswap, ru_inblock, ru_oublock, ru_msgsnd, ru_msgrcv, ru_nsignals, ru_nvcsw, ru_nivcsw; the information is only provided if an optional perl module Unix::Getrusage is available; - SpamAssassin-compatible macros 'YESNO' and 'YESNOCAPS' now optionally accept two string arguments, replacing the default strings 'Yes' and 'No' in the result; - settings $enable_dkim_verification and $enable_dkim_signing are now dynamic, i.e. became members of policy banks, thus facilitating selectively enabling or disabling these features on a policy bank basis; - internal: added a message property object dkim_signwith_sd() which allows custom hooks to provide the DKIM signing code with a selector and a domain name preferred for choosing a signing key. If this information is not available the signing code will consult an external signing service if provided ($dkim_signing_service), or else use the built-in algorithm for choosing a signing key. A custom hook may provide the information as follows: $msginfo->dkim_signwith_sd( ['some_selector', 'some_domain'] ); After a successful signing, the dkim_signwith_sd will contain a pair [selector,domain] which was actually chosen for signing; - recognize and insert header fields as prepared by SpamAssassin 3.3.0 or later through its 'add_header' configuration option; some of the standard X-Spam-* header fields are still overruled by equivalent ones generated by amavisd itself, primarily to provide true per-recipient handling; header field names must still be listed in the associative array %allowed_added_header_fields in order to be inserted; overrides are configurable through %prefer_our_added_header_fields, for example: $prefer_our_added_header_fields{lc('X-Spam-Status')} = 0; - added an attribute 'log_id' to server responses in an AM.PDP protocol, allowing the client to match its request with the amavisd daemon logging; - added LDAP attributes: amavisAddrExtensionVirus, amavisAddrExtensionSpam, amavisAddrExtensionBanned, and amavisAddrExtensionBadHeader for consistency with SQL; suggested by Stefan Palme; - added LDAP attributes: amavisSpamTag3Level, amavisSpamSubjectTag3, amavisUncheckedQuarantineTo, amavisCleanQuarantineTo, amavisUncheckedLover, amavisForwardMethod, amavisSaUserConf and amavisSaUserName for consistency with SQL; - added LDAP attribute amavisDisclaimerOptions, along with its corresponding SQL field 'disclaimer_options'. It finds its way to the list of lookup tables @disclaimer_options_bysender_maps, so the replacement of the _OPTIONS_ placeholder in @altermime_args_disclaimer could be made dynamic; suggested by Quanah Gibson-Mount; - for consistencly, added LDAP attribute amavisUncheckedLover, along with its corresponding SQL field 'unchecked_lover' and a statical list of lookup tables @unchecked_lovers_maps, which appears in the %lovers_maps_by_ccat. Previously the CC_UNCHECKED entry of the %lovers_maps_by_ccat was (ab)used, and shared the @virus_lovers_maps value. Suggested by Patrick Ben Koetter; - added a variable $myprogram_name, which defaults to a program name (perl variable $0), but may be modified in a configuration file typically depending on a value of $instance_name. It is used to dynamically change a process name in $0, which shows up in a ps(1) and top(1) output on most Unix systems. Along with $syslog_ident, it offers a handy way to distinguish amavisd instances. - if running other spam scanners besides SpamAssassin through a @spam_scanners mechanism (such as DSPAM or CRM114), make header fields produced by them visible to SpamAssassin too, so that its rules can benefit from additional information. Note that in order for SpamAssassin to be able to see such header fields from other scanners, such scanners must be listed in the @spam_scanners list *before* the 'SpamAssassin' entry. Suggested by Marco. OPTIMIZATIONS - rewritten a code section on receiving SMTP/LMTP DATA, replacing perl line-by-line reads & processing by reading & processing 32 kB chunks of data at a time; as a result, data transfer rate has been increased by a factor of about 3.9 for plain text session, and by a factor of 11 for encrypted (TLS) session. Measured data rates of an SMTP DATA transfer between Postfix and amavisd on a loopback interface: No TLS (no session encryption): . amavisd receiving, old code: 8.3 MiB/s . amavisd receiving, new code: 32.3 MiB/s . amavisd sending: 18 MiB/s With TLS (encrypted session, AES256-SHA): . amavisd receiving, old code: 1.0 MiB/s . amavisd receiving, new code: 11.2 MiB/s . amavisd sending: 4.3 MiB/s - save about 6 MB of virtual memory per amavisd child process by properly deleting some larger data items from variables, known not to be reused; thanks to the insight of Perl Monks (ikegami) in: http://www.perlmonks.org/?node_id=803515 - speed up lookup_ip_acl lookups on larger lists of CIDR network addresses (like @mynetworks and @inet_acl list) by using a radix trie (Patricia Trie) representation; the patricia trie is used when a module Net::Patricia is available and a list contains more than 20 elements; minimal required version of Net::Patricia is 1.015; - avoid entering and exiting a block in most map() and grep() calls saves on opcodes, achieving a small reduction of code size and a tiny speedup; - 'use constant' for CC_* and D_* constants allows perl to inline them; OTHER - provide a workaround for a Perl 5.8.9 bug #62502, where O_WRONLY, O_APPEND and other Fcntl constants can become tainted; this is an application of the same workaround as already applied in 2.6.3, but covers two additional code sections; the bug could manifest itself as a taint problem during opening a pipe to an external mail submission problem. This only affects perl 5.8.9; the 5.8.8 and 5.10.0 are fine. Tracked down and a patch provided by Petr Rehor; - RFC 5617 now defines an "Author Domain Signature" as a valid signature in which the domain name of the DKIM signing entity, i.e., the d= tag in the DKIM-Signature header field, is the same as the domain name in the Author Address. The change came with draft-ietf-dkim-ssp-10; previously the "Author Domain Signature" was based on an 'i' tag (identity). This change is now followed by amavisd-new 2.7.0 in macro 'dkim', in evaluation of the @author_to_policy_bank_maps list, in internal attribute dkim_author_sig(), and reflected in logging. Similarly, the evaluation of @signer_reputation_maps is now based on a signing domain ('d' tag), instead of the 'i' tag as previously. The change only affects signatures where the domain name of a signing identity is a subdomain of a signing domain (not identical to the signing domain), which is rare in current practices. - updated generating and parsing of Authentication-Results headear field according to RFC 5451 and RFC 6008 - previously it followed a draft-kucherawy-sender-auth-header. This header field is now also inserted for new DKIM signatures as just-generated and inserted to a passed internal-to-internal message when it is eligible for signing; suggested by Florian Effenberger. - added a setting $myauthservid, also a member of policy banks, which controls the "authserv-id" token in the Authentication-Results header field, according to RFC 5451. Its default value is $myhostname as before. Its value must comply with the RFC 5451 syntax ("dot-atom"). Having a separate control may facilitate setups where a message is processed by amavisd more than once, e.g. for DKIM signing of a mailing list fanout messages, where the second pass should not remove an Authentication-Results header field from a first pass. - updated ARF notifications to RFC 5965 (An Extensible Format for Email Feedback Reports); the $report_format = 'arf' implementation was based on ARF draft, now it complies with RFC 5965; - tightened some sanity limits on DKIM verification to better handle mail messages with a huge number of signatures; problem reported by Tuomo Soini; - amavisd.conf: added file types ini, lib, ocx, sys, vxd to the commented-out long list of file types in $banned_filename_re, along with a commented-out list of type names for consideration: asd, asf, asx, url, vcs, wmd, wmz; - updated default @virus_name_to_spam_score_maps with new or changed entries: [ qr'^(Heuristics\.)?Phishing\.' => 0.1 ], [ qr'^Doppelstern\.(Scam4|Phishing)' => 0.1 ], [ qr'^ScamNailer\.Phish\.' => 0.1 ], [ qr'^HTML/Bankish' => 0.1 ], thank to Giampaolo Tomassoni for Heuristics.Phishing and HTML/Bankish; - when inserting a subject tag into a Subject header field, remove existing copies of the same string first to avoid subjects like "***UNCHECKED*** Fwd: ***UNCHECKED*** Re: foo bar" ; based on a patch by Thomas Arendsen Hein; - p0f-analyzer.pl: convert an 'IPv4-mapped IPv6 addresses in alternative form' to an IPv4 address, otherwise the p0f-analyzer.pl would ignore such queries, as the p0f daemon did not handle IPv6 until version 3. The 'IPv4-mapped IPv6 addresses' is returned for an IPv4 connection when TCP/IP stack is configured to allow inet6 sockets to accept inet sessions; problem reported by Vytautas Kasparavicius; - suppress generating a non-delivery notification if a SpamAssassin test DKIM_ADSP_DISCARD is hit, honouring RFC 5617; - amavisd.conf: commented-out calls to do_ascii to match defaults in the amavisd program; the uulib code (as invoked by Convert::UUlib) has a history of stability problems, seems it is causing more grief compared to the benefits it brings; - new AV entry for 'Avira for UNIX 3.x', thanks to g0rbi, Thomas Mueller, Steffen Ille, Klaus Fuerstberger and Andreas Schulze; - add three more exception cases to mercifully ignore an EBADF I/O error due to a Perl bug on line-by-line reading; - entries 'SpamAssassin' and 'SpamdClient' in the @spam_scanners list now recognize options 'mail_body_size_limit' and 'score_factor', to match their behaviour with 'DSPAM' and 'CRM114' entries; - dropped a logging level (from -1 to 2) on a warning message: INFO: dot-stuffing error (only one leading dot): ... as Postfix in a pre-queue proxy filtering setup does not do any dot-stuffing sanitation, so garbage in the DATA section as received by a Postfix smtpd service comes unchanged to a proxy filter; reported by Ralf Heidenreich, confirmed by Victor Duchovni and Wietse Venema; - use module File::Temp to create a temporary working directory, instead of using a home-brewed code; as a result, these directory names are now a bit longer; - avoid slurping the whole directory contents into memory when recursively tidying, removing, or checking a temporary directory, when purging old database files on a restart, and when preparing a list of files to be scanned; - collect a couple of random bytes from /dev/urandom (if available) at a start of the main process and at each child process birth (when our entropy pool is rather depleted), then stir our entropy pool and perl's srand() to prevent File::Temp from working with the same pseudorandom sequence in each child process; - reworked fetching random bits from entropy pool and deriving mail id from secret id (after re-reading RFC 4086); much less of the private entropy accumulator is now exposed to observers; added a new function fetch_entropy_bytes(), dropped a function fetch_entropy(); - a macro %S no longer corresponds to sender_contact, which was a relict from times of early viruses; for compatibility with existing templates it is now equivalent to %s, but should no longer be used and might be retired or re-purposed with the next version; default notification templates were adjusted accordingly - please adjust your customized templates if using them; - drop dependency on Digest::SHA1; - README.chroot: document that sa-update needs to update rules in the jail and refresh the text somewhat; thanks to Francois Rolland; CLEANING - retired often misused settings $warnvirussender and $warnspamsender (but kept marginally useful $warnbannedsender, $warnbadhsender, and their parent %warnsender_by_ccat). To bounce or reject viruses and spam use D_REJECT and D_BOUNCE settings for corresponding $final_*_destiny. It is no longer supported to both deliver (D_PASS) a virus or spam message while also sending a notification to sender. Both retired variables are still declared for compatibility with old config files, but their value is ignored. An attempt to set their value to a non-default value produces a warning. - retired a setting $syslog_priority, it was not particularly useful since the introduction of dynamic syslog priorities with amavisd-new-2.0 . The new behaviour is equivalent to a previous $syslog_priority='debug'; The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning. - retired a setting $SYSLOG_LEVEL, it was obsoleted by amavisd-new-2.4.0; please use the setting $syslog_facility instead, defaulting to $syslog_facility='mail' . The variable $SYSLOG_LEVEL is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning. - renamed $DO_SYSLOG to $do_syslog, and $LOGFILE to $logfile; old names are kept as aliases for compatibility; - retired a setting $relayhost_is_client, it became obsolete with amavisd-new-2.0. Please use a '*' in place of a host IP address and port number when amavisd should pass a checked mail message back to the same host from which the request came, e.g.: $forward_method = 'smtp:*:*'; The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning. - retired a setting $sa_timeout, the variable is still declared for backward compatibility, but has no effect. Instead, the time available for spam scanning is automatically determined from $child_timeout, taking into consideration the actual time left till the deadline; An attempt to set its value to a non-default value produces a warning. - retired a setting $sa_spam_report_header, it was obsoleted in amavisd-new-2.4.3 with the introduction of %allowed_added_header_fields. To enable insertion of X-Spam-Report header field, please use instead: $allowed_added_header_fields{lc('X-Spam-Report')} = 1; The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning. - retired settings $sa_spam_modifies_subj and @spam_modifies_subj_maps. Disabling insertion of spam tag into a Subject header field can be achieved by turning off the corresponding entries in %subject_tag_maps_by_ccat: undef $subject_tag_maps_by_ccat{CC_SPAM()}; undef $subject_tag_maps_by_ccat{CC_SPAMMY.',1'}; undef $subject_tag_maps_by_ccat{CC_SPAMMY()}; undef $subject_tag_maps_by_ccat{CC_CLEAN.',1'}; or by emptying corresponding lists of lookup tables, e.g.: @spam_subject_tag_maps = (); @spam_subject_tag2_maps = (); @spam_subject_tag3_maps = (); or individually (by-recipient) by specifying suitable lookup tables in @spam_subject_tag_maps / @spam_subject_tag2_maps / @spam_subject_tag3_maps, either statically, or through SQL or LDAP lookups; Both settings are still declared for compatibility with old config files, but their value is ignored. An attempt to set the value of a variable $sa_spam_modifies_subj to a non-default value produces a warning. - retired a setting $insert_received_line, it was obsoleted in amavisd-new-2.4.3 with the introduction of %allowed_added_header_fields. To disable insertion of a Received header field, please use instead: $allowed_added_header_fields{lc('Received')} = 0; The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning. - retired a setting $notify_xmailer_header, the X-Mailer header field is not inserted into notifications, as was a default. The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning. - retired a setting $sa_auto_whitelist, it became obsolete with amavisd-new-2.1.0 and SpamAssassin 3.0.0 (released in 2004) by a 'use_auto_whitelist 1' option in local.cf . The variable is still declared for compatibility with old config files, but its value is ignored. An attempt to set its value to a non-default value produces a warning. - retired a deprecated macro 'x-mailer', use macros 'header_field' or 'useragent' instead; - removed a constant CC_TEMPFAIL, it was retired with amavisd-new-2.5.0; - renamed a configuration variable $sql_partition_tag to $partition_tag in order to reflect its more general usage outside of SQL; the old name $sql_partition_tag is retained for compatibility and is an alias for $partition_tag; - dropped compatibility of $final_*_destiny settings with old numerical values (used by versions of amavisd older than amavisd-new-20030314); - internal: retire Amavis::In::Message::client_addr_mynets, no longer in use; - internal: retire Amavis::In::Message::sender_contact, no longer in use; - internal: retire Amavis::In::Message::PerRecip::infected, no longer in use; - internal: drop a redundant argument $conn from the following subroutines: make_received_header_field, check_header_validity, defanged_mime_entity, msg_from_quarantine, check_amcl_policy, postfix_policy, add_forwarding_header_edits_common, add_forwarding_header_edits_per_recip, prepare_modified_mail, do_notify_and_quarantine, do_quarantine, save_info_preliminary, save_info_final, mail_dispatch, dispatch_from_quarantine, virus_scan, spam_scan, white_black_list, Amavis::SpamControl::{ExtProg,SpamdClient,SpamAssassin}::check As a compatibility measure, the do_quarantine() may still be called with or without the first argument $conn, its value is now ignored if present. Although the $conn argument is also redundant in calls to custom hooks (as this information is available through $msginfo->conn_obj), these calls are left unchanged for compatibility with existing custom hooks. --------------------------------------------------------------------------- May 18, 2011 amavisd-new-2.6.6 release notes This version is strictly a maintenance release, it incorporates bug fixes backported from 2.7.0-pre* series and/or posted as patches to the mailing list. BUG FIXES - amavisd-release was not sending a 'mail_file' attribute when a quarantined message was a non-compressed file in a single-level directory quarantine (not SQL-based), causing a release failure; reported by Jarno Huuskonen; - quarantining to SQL was sporadically failing, reporting some unrelated random error (like 'not available' or 'OpenSSL error: header too long'); reported by Tonio; - avoid a warning "_WARN: Use of uninitialized value in string eq at ... line 275." when an SQL-based white/black-listing is used; reported by Tonio; - wrap the sql clause SET NAMES 'utf8' so that only a warning at a log level 2 is issued if an SQL server does not understand the command (SQLite, old versions of MySQL) instead of aborting; reported by Roland Holzner; --------------------------------------------------------------------------- April 7, 2011 amavisd-new-2.6.5 release notes This version is strictly a maintenance release, it incorporates bug fixes backported from 2.7.0-pre* series and/or posted as patches to the mailing list. BUG FIXES - when a back-end MTA rejected a message, amavisd would send a non-delivery status notification, but also propagate the reject status back, which is wrong, only one or the other response would be appropriate. A fix also allows choosing either a D_REJECT, D_BOUNCE or D_DISCARD response for such a case, configurable through %final_destiny_by_ccat at a CC_MTA entry, defaulting to D_REJECT; reported by Peer Heinlein; - checking header section syntax could take excessive amounts of time in some degenerate cases of a very long header section, now fixed; - do not bypass spam checking of a bounce message when its referenced domain in Message-ID is non-local but pen pals are disabled; reported by Stefan; - removed some of the guesswork in bounce killer to prevent false positives in certain cases of forwarding a mail message as an attachment, at the expense of passing through some undesired but nonstandard bounces; (also, deal with non-delivery notifications from yahoogroups.com, and fixed one particular case of a false-positive in bounce killer (mixed/multipart with an attached full message, sent through a mailing list); - fixed a 'Zoo archive' entry in the $map_full_type_to_short_type_re list; - fixed a test for $myhostname being a FQDN to allow IDN domains (with a dash); - fixed a REPLACE hack (feature introduced in 2.6.2) on loading a policy bank; - fixed choosing the module IO::Socket::INET in ask_daemon_internal() to avoid versions of IO::Socket::INET6 older than 2.55 (2.56?) failing with "Address family not supported by protocol family" when an IPv4 address with a port number is specified for connections to a virus scanner; based on a patch by Phil Pearl (Lobbes); - do_unzip: avoid testing a version of Compress::Raw::Zlib, the module may not be loaded at all and the test would fail, resulting in inoperative zip unpacking; reported by Tuomo Soini; - when logging or quarantining to SQL, execute a clause: SET NAMES 'utf8' after connecting to a database to ensure the decoded Subject and From header fields are correctly interpreted by an SQL server as UTF-8 encoded strings. It seems the module DBD::mysql does not observe a MySQL setting for 'character_set_client' and needs an explicit SET NAMES. The problem did not affect PostgreSQL. Reported by Zhang Huangbin; - avoid LDAP lookups aborting the scan when a %d placeholder is used in a $default_ldap{base} setting and the resulting base does not exist in an LDAP schema; reported by Zhang Huangbin; - the amavisd-new 2.6.3 relaxed semantics of a number of hard links on a directory in TempDir::prepare(_dir), but left out an equivalent change necessary in TempDir::check, which is now fixed; the change only affects certain file system (like the one used on Mac OS X); - treat an empty PID file or a junk one-liner file the same as a nonexistent PID file; previously an empty PID file (e.g. after an unclean shutdown) would prevent amavisd from starting; problem reported by Michael Scheidell; - changed amavisd-release to only provide a 'quar_type' attribute in its request when it is reasonably sure of its appropriate value, otherwise leave the decision to the amavisd daemon; this solves releasing from a file-based quarantine when compression is not used and all files are at the top directory; reported by Voytek Eymont; - provide a workaround for a [perl #62048] bug affecting versions of perl older than (approx) 5.12.3, when a banning check if using rules in $banned_namepath_re and a lookup_re() could abort with an: Unwarranted "Malformed UTF-8 character" on certain tainted mail part names (with a valid UTF-8 representation); reported by Jakob Curdes; - provide a workaround for logging to syslog using an old version of Unix::Syslog which didn't prepare and keep its own copy of the 'ident' argument on a call to openlog(3); thanks to Bill Landry; OTHER - ensure compatibility with a new version 5.500 of MIME-Tools, which changed the way mime attributes content-disposition.filename and content-type.name are decoded, now properly respecting their declared encodings (character set). As a result, the declared (recommended) file names of MIME parts are now represented as native Perl character strings (Unicode), and as such may also end up in reported names of banned parts. Regular expressions in @banned_filename_maps, $banned_filename_re and $banned_namepath_re may also see these strings as native Perl characters, along with their MIME-encoded form. The change also affects interpretation of names with earlier versions of MIME-Tools, making them behave more like the 5.500. - amavisd.conf: exclude names starting with 'cid:' from matching the double extensions banning rule, avoiding false positives; - a small update to a default @virus_name_to_spam_score_maps; - the 'originating' flag is now passed on to SpamAssassin through its %suppl_attrib argument - potentially useful with current trunk version of SpamAssassin (treats originating mail submission as a MSA submission), and ignored by older versions; - some documentation updates; the RELEASE_NOTES file is now encoded as UTF-8, instead of ISO-8859-1; --------------------------------------------------------------------------- June 25, 2009 amavisd-new-2.6.4 release notes NOTE: When upgrading Perl to version 5.10 or planning to do so, please do not forget to add a missing /m flag to regular expressions in your existing AV entries (if you haven't already done so with a 2.6.3 upgrade), as suggested in an example file amavisd.conf in a package. Perl 5.8 does not mind missing /m flags, but with perl 5.10 the results from a virus scanner may no longer be properly recognized. See the BUG FIXES section in 2.6.3 release notes. COMPATIBILITY WITH 2.6.3 The output of amavisd-agent and contents of a database snmp.db has changed according to the now published MIB. Several new SNMP counters were added, a few retired, and some renamed. If you are parsing the output of amavisd-agent or accessing snmp.db directly, please review AMAVIS-MIB.txt, and perhaps switch to using the new amavisd-snmp-subagent. BUG FIXES - amavisd failed to start when spam scanning was disabled either by @bypass_spam_checks_maps=(1) or by @spam_scanners=(), giving: Can't locate object method "new" via package "Amavis::SpamControl" As a workaround one could use a @spam_scanners=(undef) to disable spam scanning; reported by Steve; - several decoders failed to propagate "Exceeded storage quota" exception, so the protection of AV scanners against mail bombs was ineffective; reported by Jorgen Lundman; - milter usage (AM.PDP): verbatim header edits inserted a header body of "1" instead of the correct string, for example: "Authentication-Results: 1"; - updated AV entry for BitDefender's bdscan to recognize tabs around a colon in its output; contributed by Steve; - fix parsing of a combined result from DSPAM (option --classify), as earlier versions of DSPAM did not include a signature with a combined result line; problem reported by Marijan Vidmar; NEW FEATURES SUMMARY - provide a true SNMP agent and a MIB, facilitating monitoring the health of a content filtering system, its performance and mail characteristics; - a new AV interface to SMTP-based antivirus scanners; - allow customizing SMTP-status response reason text for blocked messages; - prevent inserting fake copies of certain important mail header fields without breaking a DKIM signature; NEW FEATURES - newly supplied with the package is a program amavisd-snmp-subagent, acting as an SNMP AgentX, exporting amavisd statistical counters database (snmp.db) as well as a child process status database (nanny.db) to a SNMP daemon supporting the AgentX protocol (RFC 2741), such a NET-SNMP. It is similar to combined existing utility programs amavisd-agent and amavisd-nanny, but instead of writing results as text to stdout, it exports data to an SNMP server running on a host (same or remote), making them available to SNMP clients (such a Cacti or mrtg) for monitoring or alerting purposes. The amavisd program does not have any additional requirements, but to run amavisd-snmp-subagent the following Perl modules are required: NetSNMP::OID, NetSNMP::ASN, NetSNMP::agent, NetSNMP::default_store. All of these come with a Net-SNMP package (previously known as "ucd-snmp"), home at http://net-snmp.sourceforge.net/, FreeBSD ports: net-mgmt/net-snmp. Also, an snmpd daemon must be running on a host. It can be an snmpd from a Net-SNMP package or some other SNMP server supporting AgentX protocol. When using snmpd from Net-SNMP, just add the following to its snmpd.conf: master agentx agentXSocket tcp:127.0.0.1:705 so that amavisd-snmp-subagent will be allowed to connect to it. The setup was tested with Net-SNMP versions 5.3.2.3, 5.4.2.1 and 5.2.0. If you experience wild numbers served in Counter64 variables on a 64-bit platform, the following patch (at the server side) solves the problem: http://www.mail-archive.com/ net-snmp-users@lists.sourceforge.net/msg19502.html The patch seems to already be incorporated into version 5.3.3 of Net-SNMP, and into 5.5 (but not in 5.4.2, nor in 5.2.0). A MIB module (SNMP Management information base) is provided in a file AMAVIS-MIB.txt. It is not necessary to make it available to an SNMP server, and not even necessary for SNMP clients, but making it available to clients allows them to display data with names of variables, not just their OIDs. A query example with no MIB modules: snmpbulkwalk -v2c -c xxx host.example.com .1.3.6.1.4.1.15312.2.1 A query example when a file AMAVIS-MIB.txt is in a subdirectory ./mibs/ : snmpbulkwalk -m+AMAVIS-MIB -M-mibs -OQ -v2c -c xxx host.example.com amavis The amavisd-snmp-subagent can be started at any time, either before or after amavisd, and either before or after snmpd. It can also be restarted at any time. Also, amavisd can be restarted without having to restart amavisd-snmp-subagent, as it will automatically notice a database change and connect to a new database. Similarly, an snmpd daemon can be restarted at any time and amavisd-snmp-subagent will reconnect to it if necessary. A natural starting order is: snmpd first, then amavisd and then amavisd-snmp-subagent. Restarting amavisd will reset its counters. An SNMP client typically interprets a decremented value of a counter variable as a wraparound, which results in a large spike when graphing data. There are two common solutions to the problem: a reasonable upper limit can be provided to a client, so that a spike will be treated as invalid data and ignored, or else a AMAVIS-MIB::sysUpTime variable can be monitored, and if its value is smaller than on a previous reading, this indicates that counters were reset (i.e. amavisd was restarted) and values of counters should not be treated as wrapped on maxint. Consult your SNMP client documentation. The amavisd-snmp-subagent should have access to databases snmp,db and nanny.db in a $db_home directory (environment variable AMAVISD_DB_HOME, defaults to /var/amavis/db) and have rights to connect to an snmpd daemon. It is safe to run it as root, although perhaps not necessary. For testing purposes start amavisd-snmp-subagent from a command line using a command line option -f to let it stay in foreground, and optionally increase debug level, e.g: amavisd-snmp-subagent -f -d 5 If everything goes well, start it without -f and let it daemonize. Supplying a filename with an option -P tells a daemonized agent to write its PID to that file, and remove the file on shutdown (on receiving a signal TERM or INT): amavisd-snmp-subagent -P /var/run/amavisd-snmp-subagent.pid Some suggested sets of OIDs making up interesting diagrams (e.g. for displaying by Cacti): counters: * inMsgsStatusRelayed, inMsgsStatusDiscarded, inMsgsStatusNoBounce, inMsgsStatusBounced, inMsgsStatusRejected * inMsgs, inMsgsOriginating * inMsgsSize, inMsgsSizeOriginating * inMsgsSize, outMsgsSizeSubmitQuar, outMsgsSizeRelay * inMsgs, outMsgsRelay, outMsgsSubmitQuar, outMsgsSubmitDsn, outMsgsSubmitNotif * contentCleanMsgs, contentCleanMsgsOriginating * inMsgs, contentSpamMsgs, contentBannedMsgs, contentVirusMsgs * contentSpamMsgsOriginating, contentBannedMsgsOriginating, contentVirusMsgsOriginating * timeElapsedTotal, timeElapsedDecoding, timeElapsedVirusCheck, timeElapsedSpamCheck * procGone gauges: * procBusy, procAll * procBusy, procBusyTransfer, procBusyDecode, procBusyVirus, procBusySpam * procBusy0, procBusy1s, procBusy2s, procBusy4s, procBusy8s * procBusy15s, procBusy30s, procBusy1m, procBusy2m, procBusy4m * mtaQueueEntriesIncoming, mtaQueueEntriesActive, mtaQueueEntriesDeferred Note that even frequent or extensive SNMP queries do not burden amavisd processes. The amavisd-snmp-subagent process keeps a cache of current variable values. It queries one or the other berkeley database as needed, i.e. when cached data is stale and there was an actual SNMP query for a variable in one or the other database. When a berkeley database needs to be accessed, all its data is fetched in one quick sweep by using a database cursor with a read lock, so that data is consistent. No more than one database sweep in 4 seconds is performed, and less often when queries are less frequent and preferably batched in groups. If some time has passed since the last SNMP query (more than 4 seconds currently), resulting values are always fresh as collected from a database at the time of an SNMP query. There is one additional experimental feature - experimental in a sense that it may change or be dropped in future versions. If running Postfix on the same host as amavisd-snmp-subagent, a count of files (mail messages) in each of the Postfix queue directories is provided as Gauge32 variables in the MIB under .1.3.6.1.4.1.15312.2.1.3, i.e. under amavisMta subtree. The following SNMP variables are available: mtaQueueEntriesMaildrop, mtaQueueEntriesIncoming, mtaQueueEntriesActive, mtaQueueEntriesDeferred. Although semantically outside the scope of amavisd, it provides a quick insight into health of an MTA, and indirectly into health of amavisd. Data is made available only if a command 'postconf -h queue_directory' is successful at amavisd-snmp-subagent startup time and provides a sensible result. Like with the other two real databases, MTA directories are only scanned if and when actually queried by an SNMP client (again, subject to caching). As a safety measure for times when MTA queue grows huge, there is a time limit for scanning each directory subtree (currently 5 seconds, which is about how much a typical SNMP client is willing to wait for a response). Also, a long scan time automatically increases cache validity time (time-to-live) of that measurement. - a new experimental interface to SMTP-based antivirus scanners is provided; an @av_scanners entry may look like the following: ['av_smtp', \&ask_av_smtp, ['{}', 'smtp:[127.0.0.1]:5525', 'dummy@localhost'], qr/^2/, qr/^5/, qr/FOUND:\s*(.*?)\s*$/m ], The ask_av_smtp mechanism connects to a virus scanner using the specified protocol (typically SMTP or LMTP) on a given IP address and a port number, considering the virus scanner as an ordinary MTA. The full original message is then fed to the scanner (currently ignoring the "{}" argument), using the original envelope sender address and a given address as a single recipient (defaults to 'dummy@localhost'). It is expected that a virus scanner will accept a clean message (2xx) and reject an infected message (status 5xx). An SMTP response is parsed as usual for any output from a virus scanner, typically considering a response starting with 2 as clean, a response starting by 5 as infected, and anything else as a scanner failure. The SMTP-based virus scanner should be configured not to deliver a message. This may be achieved by feeding its SMTP output to a dummy SMTP listener, such as smtp-sink as supplied by a Postfix package. It is not a particularly efficient interfacing mechanism, but some virus scanners do not provide a choice. Prompted by Kevin M. Myer; - a new configuration variable %smtp_reason_by_ccat allows customizing SMTP-status response reason text. The reason strings are subject to macro expansion, so built-in macros are available (README.customize). Multi-line texts are allowed and produce a valid multi-line SMTP response, but use it sparingly, as some nonstandard mailers may not like it. Currently %smtp_reason_by_ccat is only consulted for blocked messages, a passed clean message still uses a hardwired reason text; suggested by Ralf Hildebrandt, based on a patch by Noah Baker; - support inclusion of null header field names in an 'h' tag of a DKIM signature generated by amavisd for specified header field names, thus preventing third parties from prepending additional occurrences of these header fields without breaking a signature; useful for example for protecting a recipient's mail reader or a filter from being tricked by supplying a duplicate From or Subject header fields; the protection is requested by specifying a value larger than 1 in %signed_header_fields, e.g.: $signed_header_fields{'from'} = 2; $signed_header_fields{'subject'} = 2; $signed_header_fields{'message-id'} = 2; $signed_header_fields{'content-type'} = 2; Please restrict values used in %signed_header_fields to 0, 1, or 2, consider other values reserved for future use. By default the following header fields are protected from duplicates by a DKIM signature generated by amavisd: From, Date, Subject, Content-Type. To revert to a classical behaviour, set their value in %signed_header_fields to 1, e.g.: $signed_header_fields{lc($_)} = 1 for qw(From Date Subject Content-Type); - add a config variable @spam_notifyadmin_cutoff_level_maps, which allows suppressing of spam administrator notifications when spam score exceeds a level resulting from a lookup into this list of lookup tables; suggested by Rudy Gevaert; - new configuration variables: $snmp_contact, $snmp_location, empty strings by default; these end up in a MIB as exported by the new amavis SNMP agent; OTHER - failure of a file(1) utility is now only logged (at syslog level LOG_ERR) and no longer treated as a fatal error; suggested by Matija Grabnar; - require a minimal version 2.017 of Compress::Raw::Zlib when unpacking a zip archive (by Archive::Zip) to avoid an archive (e.g. in an infected mail) from causing amavisd process to hang; thanks to Alexander 'Leo' Bergolth for troubleshooting the memory allocation issue in a zip library; - amavisd-nanny: write notes about lost processes to STDERR instead of to STDOUT, making it easier to use it non-interactively, e.g. from cron; suppress printing of "exited" when not interrupted; based on a patch provided by Thomas Gelf; - amavisd-agent: suppress printing of "exited" when not interrupted; - amavisd-agent: updated according to changes in MIB; - reduce a log level of a virus scanner failure from -2 to -1, unless all virus scanners have failed; - added to @virus_name_to_spam_score_maps : /^Structured\.(SSN|CreditCardNumber)\b/ /^Sanesecurity.TestSig_/ /^Email\.Spammail\b/ /^winnow\.(phish|spam)\./ /^INetMsg\.SpamDomain/ - internal: added user_policy_id and user_policy_id to recipient data in object Amavis::In::Message::PerRecip, facilitating storing users.policy_id into msgrcpt.sql_policy_id; suggested by Stefan Palme; - amavisd.conf-sample: updated comment explaining the use of %a and %k expansions in $sql_select_policy; thanks to Max-Julian Pogner; - documentation: change all "author signature" to "author domain signature" according to draft-ietf-dkim-ssp-10; --------------------------------------------------------------------------- April 22, 2009 amavisd-new-2.6.3 release notes NOTE: When upgrading Perl to version 5.10 or planning to do so, please do not forget to add a missing /m flag to regular expressions in your existing AV entries, as suggested in an example file amavisd.conf in a package. Perl 5.8 does not mind missing /m flags, but with perl 5.10 the results from a virus scanner may no longer be properly recognized. See the BUG FIXES section below. COMPATIBILITY WITH 2.6.2 - support for DSPAM has been removed from Amavis::SpamControl::SpamAssassin module, merging DSPAM scores into SpamAssassin and DSPAM autolearning is no longer available. Nevertheless, it is now possible to use DSPAM instead of SpamAssassin, or by adding results from each. See description below for @spam_scanners; - please see a note below about having to explitly re-assign the @client_ipaddr_policy list if the @mynetworks_maps (not the @mynetworks) is changed in amavisd.conf; - there are no other known incompatibilities with 2.6.2; BUG FIXES - when logging to SQL (pen pals), the msgs.message_id field always received a value '1' instead of a Message-Id, thus making pen pals less effective (only matching on sender/recipient pairs worked, not on message threads) and letting some bounces bypass a bounce killer; bug was introduced with version 2.6.2; reported by Michael Scheidell; - timer was not reset after a persistent failure to connect to a daemonized virus scanner, so a subsequent call to a backup scanner only had 10 seconds available before it was aborted, which was often too short for a command line backup scanner like clamscan; reported by Bill Landry; - if a virus scanner interface did not find a name of a virus in the output of a virus scanner (despite noticing infection), the infection was ignored; reported by Thomas Mueller; - added missing /m flags to regular expressions in AV entries (a bug is revealed with Perl 5.10.0; previous versions of Perl happened to work, unintentionally accepting a /m flag if added late during a regexp evaluation); reported by Rafael; - $banned_namepath_re setting only worked globally, but was not usable in policy banks; reported by Danny Richter; - do_uncompress: signal run_command_copy() errors, instead of returning a status, thus allowing decompose_part() to detect 'Exceeded storage quota' or 'Maximum number of files exceeded', and flag mail as CC_UNCHECKED; - if $mailfrom_notify_admin was not specified in a configuration file but defaulted to an e-mail address in $hdrfrom_notify_admin, the following was reported (due to missing angle brackets) on an attempt to submit a notification: (!)SEND via SMTP: virusalert@example.com -> <virusalert@example.com>... 501 5.1.7 Bad sender address syntax (!)FAILED to notify admin: 501 5.1.7 Failed, id=40690-23, from MTA([::1]:10027): 501 5.1.7 Bad sender address syntax Notification was not sent, the rest of the processing was unaffected; reported by Peter Pechnik, Thomas Mueller, and Stefan Förster; - fetch_modules: only suppress the "Can't locate ... in @INC" diagnostics if exactly the requested module is missing, but do show the error if some subordinate module is missing and preventing the requested module to be loaded; - do_unrar: recognize an information line with a '<->'; - fixed a syntax error in LDAP.ldif; by Quanah Gibson-Mount - fixed a bug in SpamdClient; reported by Filip Valder NEW FEATURES - added a configuration variable @client_ipaddr_policy, which maps smtp client's IP address lookup lists to a policy bank name. This allows for loading a policy bank based on a client IP address, and generalizes a formerly hard-wired mapping of @mynetworks_maps into 'MYNETS'. The list is traversed in order, the first matching networks list stops the search and its associated policy name is used. Suggested by Jo Rhett. The default setting retains backward compatibility: @client_ipaddr_policy = map { $_ => 'MYNETS' } @mynetworks_maps; but please keep in mind that this assignment is made during startup before evaluating a config file, so if amavisd.conf changes the @mynetworks_maps list, the assignment to @client_ipaddr_policy needs to be re-evaluated to retain a desired default. This is not necessary when @mynetworks_maps is left untouched but only its component @mynetworks is changed. Example: @client_ipaddr_policy = ( [qw( 0.0.0.0/8 [::] 127.0.0.0/8 [::1] )] => 'LOCALHOST', [qw( !172.16.1.0/24 172.16.0.0/12 192.168.0.0/16 )] => 'PRIVATENETS', [qw( 192.0.2.0/25 192.0.2.129 192.0.2.130 )] => 'PARTNER', \@some_other_networks => 'OTHER', \@mynetworks => 'MYNETS', ); - large messages beyond $sa_mail_body_size_limit are now partially passed to SpamAssassin and other spam scanners for checking: a copy passed to a spam scanner is truncated near or slightly past the indicated limit. Large messages are no longer given an almost free passage through spam checks. Note that message truncation can invalidate a DKIM or DK signature. If using (non-default) SpamAssassin rules to assign score points to mail with no valid signatures from authors which are expected to always provide a valid signature, the message truncation can cause false positives on these rules. As a workaround, to a truncated message passed to spam scanners, amavisd inserts a header field: X-Amavis-MessageSize: mmmmm, TRUNCATED to nnnnn which can be captured by SpamAssassin rules, e.g.: header __TRUNCATED X-Amavis-MessageSize =~ m{\A[^\n]*TRUNCATED}m and used in rules like NOTVALID_EBAY to prevent them from triggering. Starting with version 3.3.0 of SpamAssassin, its DKIM plugin understands the issue and receives undamaged DKIM signature objects directly from amavisd, so the above workaround is not needed. Also, a hit on a __TRUNCATED rule is automatically generated (explicit header rule is not necessary), just in case it might be useful for some purpose. - supports passing an extra argument suppl_attrib to $spamassassin->parse, as recognized by SpamAssassin 3.3.0, passing a set of DKIM signature objects to a SpamAssassin's plugin DKIM, which saves having to do the same signature verification operation again within a plugin, and provides uncrippled signatures to SpamAssassin even when a large message is truncated by amavisd and only partially submitted to spam analysis; - add global variables $sa_configpath and $sa_siteconfigpath (undef by default), which are passed to SpamAssassin as options 'rules_filename' and 'site_rules_filename' during its initialization call; this makes it easier to run multiple instances of amavisd, each with a different SpamAssassin configuration, using the same amavisd configurations file by taking advantage of option -i; suggested by Noah Baker; - report process resource usage at log level 2 by calling getrusage(1) if a perl module Unix::Getrusage is available; - a configuration variable @spam_scanners is added, along with a module Amavis::SpamControl::ExtProg (which is only loaded if needed). This is similar in concept to @av_scanners list, and allows using amavisd with different spam scanners, not just with SpamAssassin. The default setting is backward compatible: @spam_scanners = ( ['SpamAssassin', 'Amavis::SpamControl::SpamAssassin'], ); The first element of each tuple is a scanner name, the second is a module name to be invoked, it must implement a method new(). Remaining arguments are passed to a module as arguments in a call to its new(). The exact syntax and semantics of these arguments is module-specific and may change in future versions as more experience is gained. Currently supported spam scanners are: - SpamAssassin: backward compatible, uses the module Mail::SpamAssassin directly as before; - SpamdClient: a client to spamd, equivalent to a spamc usage; the main reason for existence of this module is to allow amavisd to serve as a test client for exercising spamd; not envisaged for production use; - CRM114: spawns an external program 'crm'. A well trained crm114 system gives good results (even with a global database). An alternative is to use a CRM114 plugin to SpamAssassin, with a benefit of autolearning and combining its results with other rules, but at some processing cost; - DSPAM: spawns an external program 'dspam'; Spam score and test results from all spam scanners are added together, currently it makes most sense to only have one of these entries enabled at a time. A possible (artificial, not particularly useful) configuration with multiple entries is illustrated by the following setting: @spam_scanners = ( ['SpamAssassin', 'Amavis::SpamControl::SpamAssassin' ], ['SpamdClient', 'Amavis::SpamControl::SpamdClient' ], ['CRM114', 'Amavis::SpamControl::ExtProg', 'crm', [ qw(-u /var/amavis/home/.crm114 mailreaver.crm --dontstore --report_only --stats_only --good_threshold=8 --spam_threshold=-8) ], mail_body_size_limit => 64000, score_factor => -0.20, ], ['DSPAM', 'Amavis::SpamControl::ExtProg', $dspam, [ qw(--stdout --classify --deliver=innocent,spam --mode=tum --tokenizer=chained,noise --user), $daemon_user ], # use option --feature instead of --tokenizer with dspam < 3.8.0 mail_body_size_limit => 64000, score_factor => 1, ], ); A module Amavis::SpamControl::ExtProg implements an interface to external spawned programs. These are expected to receive a mail message on their stdin, and produce a result on their stdout (and errors on stderr). The result typically consists of some header fields the spawned spam scanner wishes to report to a caller, but can also be a complete rewritten header section or a complete rewritten mail message. The ExtProg module just collects the information it needs from the output of a scanner and discards the rest (i.e. an external scanner can not rewrite a message), so to avoid unnecessary processing, it is best to configure an external scanner to only return what is needed. Currently some post-processing of CRM114 and DSPAM results is hard-wired into the ExtProg module. Collected header fields are typically inserted into a passed message, subject to a list of allowed header fields in %allowed_added_header_fields. Some important header fields are also added to a quarantined message, but a different mechanism is involved ($msginfo->supplementary_info, the same mechanism as used in obtaining tags from SpamAssassin). The module Amavis::SpamControl::ExtProg expects two required parameters: a path to a program to spawn, and a ref to a list of command line arguments. Following these two arguments there may be options in a form of key/value pairs. Unrecognized options are ignored. Currently the only two options are: mail_body_size_limit ... the ExtProg module only feeds up to about this number of bytes (or slightly more) of a message to a spam scanner; if unspecified or undefined a default limit is $sa_mail_body_size_limit, and if that is undefined, an entire message is passed regardless of its size; score_factor ... a floating point number by which a score produced by a spam scanner is multiplied to yield a final score (with a SpamAssassin semantics, values near or below 0 are ham, values near or above 5 are spam). Note that crm114 uses opposite sign semantics, so a score_factor for this scanner should be negative. The dspam scanner produces hard-wired score -1 (innocent) or 10 (spam), which is then multiplied by score_factor to yield a final score. OTHER - supports a SpamAssassin plugin CRM114, thanks to Jules M, and to Martin Schütte for his CRM114 plugin for SpamAssassin; - updated AV virus scanner entry for ESET Software ESETS Command Line Interface to version 3.0, commenting out entries for old versions 2.7 and 2.71.12; thanks to Hugo Slabbert; - provide a workaround for a Perl 5.8.9 bug #62502, where O_WRONLY, O_APPEND and other Fcntl constants can become tainted; the bug could manifest itself as a taint problem during file-based quarantining, during MIME decoding, decoding archives and decompressing mail parts, and possibly elsewhere. The bug is triggered by a legitimate code in Archive::Zip and affects subsequent operations in amavisd for the lifetime of the child process. This only affects perl 5.8.9; the 5.8.8 and 5.10.0 are fine. - provide a workaround for a Perl I/O bug, where a bounce killer could abort with: "inspect_a_bounce_message failed: Error reading mail header section: Bad file descriptor at /usr/local/sbin/amavisd line 11486" - uncommented the qr'^MAIL$' in @keep_decoded_original_maps (amavisd.conf and amavisd.conf-sample); seems it is becoming increasingly more important for virus scanners to also see the complete undecoded message; suggested by Michael Scheidell and others; - added qr'^Safebrowsing\.' to the @virus_name_to_spam_score_maps list, suggested by Michael Scheidell; - no longer unconditionally pre-loads the following SpamAssassin modules, as not all sites are using SQL-based bayes and AWL database: Mail::SpamAssassin::BayesStore::SQL Mail::SpamAssassin::BayesStore::MySQL Mail::SpamAssassin::BayesStore::PgSQL Mail::SpamAssassin::SQLBasedAddrList These modules will be dynamically loaded if needed by SpamAssassin after daemonization (which will be logged as "extra modules loaded after daemonizing/chrooting:"). If running chrooted, these modules may not be available in a jail, so their loading can be forced by including their names in the @additional_perl_modules list (in amavisd.conf), either as absolute file paths or as module names, e.g.: @additional_perl_modules = qw( Mail::SpamAssassin::BayesStore::SQL Mail::SpamAssassin::BayesStore::MySQL Mail::SpamAssassin::SQLBasedAddrList ); - when inserting a warning into a defanged mail body, crop very long diagnostics 'WARNING: bad headers - ...' to some sane size; - when all virus scanners failed, let a 451 smtp response message only say 'ALL VIRUS SCANNERS FAILED' or 'NO VIRUS SCANNERS AVAILABLE', but leave out a detailed failure reason for each failing scanner, along with line numbers; suggested by David Schweikert; - reduce log level of a test in TempDir::prepare for a number of links left on a directory after purging it, seems like it does not play well with a file system on Mac OS X, producing an occasional warning: TempDir::prepare: directory /var/amavis/tmp/amavis-2009... has 2 subdirectories reported by Matthias Schmidt; - do not add a per-recipient contents category CC_UNCHECKED to recipients which have virus checking bypassed, and subsequently do not insert $undecipherable_subject_tag into their Subject; suggested by Jorgen Lundman; - more precise bypassing of spam checks: bypass spam checks if it is already known that a message will be blocked due to banned contents; do not bypass spam checks for virus lovers, even if a message is infected; - when checking if a valid DKIM signature is an author domain signature, treat address localpart as case sensitive, according to ADSP and RFC 5321; - testkeys command: improved reporting of signing failures; - removed X-Virus-Scanned from a list of DKIM-signed header fields; - turn on the 'require_rules' option when initializing SpamAssassin to match behaviour with spamd and a spamassassin command; - make sure a temporary directory name supplied through AM.PDP protocol is really a directory before deleting it; - files_to_scan: make sure a file encountered during directory traversal is not a symlink before changing its protection (chmod); - do_ascii: chmod a file only if it is a regular file (not a symlink) and is not readable; - do_7zip: remove the -p option with its dummy password, looks like it has been removed and now produces an 'Incorrect command line' error; - log (at log level 2) a list of loaded SpamAssassin plugins during startup; suggested by Giuseppe Ghibò; - convert_keysfile: do not print @dkim_signature_options_bysender_maps assignment when a list of options is empty and thus redundant; - convert_keysfile: make '*' in the first field equivalent to '*@*'; - internal: when deciding whether to skip spam scanning and pen pals checks test for is_in_contents_category(CC_VIRUS) instead of @virusnames; - internal: removed unused method infected() from package Amavis::In::Message::PerRecip; - internal: store_mgr() now signals errors instead of returning a status; update callers of store_mgr() accordingly; - amavisd.conf-default: document that a default value of $bounce_killer_score is 0; pointed out by Michael Scheidell; --------------------------------------------------------------------------- December 15, 2008 amavisd-new-2.6.2 release notes MAIN NEW FEATURES SUMMARY - bounce killer: improved detection of nonstandard bounces; - bounces to be killed no longer waste SpamAssassin time; - tool to convert dkim-filter keysfile into amavisd configuration; - compatibility with SpamAssassin 3.3 (CVS head) regained; - rewritten and expanded documentation section on DKIM signing and verification in amavisd-new-docs.html; COMPATIBILITY WITH 2.6.1 - apart from small differences in logging and notifications, the version 2.6.2 is compatible with 2.6.1, with its configuration file and its environment; - virus scanner entries were updated (as described below, most notably by adding a regexp flag m), so be sure to update existing configuration file; updated virus scanner entries can be used with 2.6.1 too; - the %sql_clause default has changed in detail (see below), if its value is overridden in a configuration file the setting may need updating; BUG FIXES - when feeding a message by SMTP back to MTA and MTA rejects a recipient as invalid and an smtp connection cache is enabled, the SMTP protocol can get out of step, rejecting the next message in the same connection with a "503 5.5.1 Error: nested MAIL command"; this only affects (hopefully) rare sites where recipient validation is performed after content filtering instead of before content filtering; reported by Richard Smits; - logging routines reporting warnings failed to include a diagnostics message in a log, instead only a dry '_WARN:' or '_DIE:' with no explanation was logged; a bug was introduced in 2.6.1; reported by Mike Cappella; - amavisd-release: add a 'partition_tag' attribute to a release request if a specified quarantine name ends up in a partition tag string in square brackets; this feature was announced in 2.6.1 release notes, but never made it into a distribution; - amavisd-report failed on reading a message from SQL quarantine: dispatch_from_quarantine failed: read: sql select failed, DBD::Pg::st fetchrow_arrayref failed: no statement executing reported by Achraf Tangui; - while evaluating compiled regular expressions (qr), perl 5.10.0 ignores flag m when present in the final expression but not in the qr itself, causing messages containing multiple viruses not to report any virus names (mail is still considered infected, but list of names is empty). Changed regular expressions in virus entries by appending a /m flag to regular expressions in the 6th element of each entry. According to Perl maintainers this was a bug in 5.8.x and earlier, and the behaviour of perl 5.10.0 is now according to specs; reported by Martin Huber; - envelope sender address for administrator- and recipient notifications ($mailfrom_notify_admin, $mailfrom_notify_spamadmin, $mailfrom_notify_recip, %mailfrom_notify_admin_by_ccat, %mailfrom_notify_recip_by_ccat) was not expanded when their value is left unspecified in a configuration file and defaults to parsing of $hdrfrom_notify_* settings. This leads to MTA rejecting a notification from 'postmaster@${myhostname}' by a '501 5.1.7 Bad sender address syntax'. Reported by Aleksey Chudov, Jonas Jacobsson, Durk Strooisma, and Adam; - remove unintentionally hard-coded SSL certificate and key file locations stored in variables $smtpd_tls_key_file and $smtpd_tls_cert_file, they are now configurable through a configuration file as intended; - a macro 'rfc2822_sender' now returns a Sender address in a quoted form, just like its cousin 'rfc2822_from'; - when stopping or restarting amavisd, check a PID file for being stale _before_ testing whether a process exists, not the other way around; previously an unlucky starting amavisd process could hit a: Can't send SIG 0 to process [nnnn]: Operation not permitted which prevented its startup when a stale PID was reused by an unrelated process; reported by Zhang Huangbin; - error reporting improvement: localize variables $@ and $! in all DESTROY methods, thus preventing these variables from being clobbered behind the scenes (e.g. by calling eval or system routines from DESTROY), which could cause a surprising empty (or unrelated) error message being reported by surrounding eval blocks; - avoid problematic perl constructs open('|-') and open('-|') which fail to catch certain fork errors, or waits indefinitely when resources are tight; just explicitly create a pipe and call fork in subroutines run_command, run_command_consumer and in run_as_subprocess. The change possibly also solves some mystery cases where amavisd would appear to hang when resources are tight (running out of swap space or near a maxprocesses limit) instead of reporting a fork failure. Problem with fork failing without giving a reason for failure reported by Uwe Kiewel; - amavisd.conf-default: definition of %sql_clause default was out of date; reported by Roland; - releasing a non-existent message from an SQL quarantine produced an inappropriate error message about a subsequent failure, instead of reporting a missing record; reported by Rick (rn). Also let SQL treat a NULL in mail_text.partition_tag as 0 by using coalesce() - changed a $sql_clause{'sel_quar'} from: SELECT mail_text FROM quarantine WHERE partition_tag=? AND mail_id=? ORDER BY chunk_ind into: SELECT mail_text FROM quarantine WHERE coalesce(partition_tag,0)=coalesce(?,0) AND mail_id=? ORDER BY chunk_ind to facilitate transition from not having a partition_tag defined (resulting in NULL partition_tag fields in SQL) into using it as a numeric value (e.g. a week-of-the-year number); - modified AV entry for a grisoft.com virus scanner by adding a regexp flag /m to let ^ match at any line beginning of a possibly multi-line response from a virus scanner; problem reported by John Beranek; - recognize any 'ERROR:' result from a file(1) utility - not just an 'ERROR: Corrupted', and do not treat its exit status 1 as fatal, but just log a warning; - protect logging from being recursively re-entered when an error occurs during writing of a log entry; NEW FEATURES - bounce killer: improved parsing of nonstandard bounce messages (from qmail, spamarrest.com and similar) yields more effective protection against third-party bounces, including those without a Message-ID. An analysis of 1000 previously passed bounces showed that 2/3 of those are now recognized and blocked, bringing a bounce killer rate to 94 % of all received bounces (with about 4 % of passed unverifiable bounces not carrying an original mail header, and a tiny trickle of true bounces), while still ensuring that bounces (in response to our genuine outbound mail) and message disposition notifications (MDN, RFC 3798) are still received reliably. As a reminder: bounce killer is enabled by setting $bounce_killer_score to a large value, e.g. 100. This value is added to a final spam score if a message analysis determines this is a bounce to a third-party message, i.e. a backscatter. Spam score of genuine bounces is not affected. If a $bounce_killer_score value is above 20 and we know for certain the bounce will be killed, SpamAssassin scanning is bypassed, saving substantial resources when under a backscatter storm. A pre-requisite for proper operation of a bounce killer is a working SQL logging database (pen pals), or that outbound DSN messages have a Message-ID with a fully qualified domain name matching the @local_domains_maps list of lookup tables. Parts decoding must also not be disabled ($bypass_decode_parts=0), which is a default. Conditions are easily met when all mail from local users is submitted through a domain's official mailer, which goes hand in hand with the requirement for DKIM signing and for other similar anti-spoofing techniques (SPF, whitelisting by IP address in Received trace, ...). The $bounce_killer_score should not be enabled when not all outgoing mail can be identified either by a local domain name in Message-ID or by being registered in pen pals SQL database, otherwise genuine bounces and returning MDN messages will be considered spam. - to facilitate transition of DKIM signing from dkim-milter to amavisd-new, a new command-line tool is available (the extra utility code is not loaded during normal operation), taking a file name as its argument, e.g.: # amavisd convert_keysfile /var/db/dkim/keysfile.txt and writing to stdout a set of lines that may be directly included into amavisd.conf configurations file, matching semantics of a dkim-filter keys file. It can be useful during transition, or for those who prefer to specify signing keys and sender-to-key mappings as a file in a syntax compatible with options -K -k of dkim-filter, and can live with limitations of such syntax. See dkim-filter(8) man page for details on the syntax. The produced output consists of signing key declarations (calls to a procedure dkim_key), where each call normally corresponds to exactly one DNS resource record publishing a corresponding DKIM public key. When necessary output also produces an assignment to a list of lookup tables @dkim_signature_options_bysender_maps, which supplies non-default mappings of sender domains to signing keys, e.g. when third-party signatures are desired. From the dkim-filter man page: The keyfile should contain a set of lines of the form sender-pattern:signing-domain:keypath where sender-pattern is a pattern to match against message senders (with a special character "*" interpreted as "zero or more characters"), signing-domain is the domain to announce as the signing domain when generating signatures (or a '*', implying author's domain), and keypath is a path to the PEM-formatted private key to be used for signing messages which match the sender-pattern. The selector used in the signature will be the filename portion of keypath. A line starting with "/" is interpreted as a root directory for keys, meaning the keypath values after that line in the file are taken relative to that path. If a file referenced by keypath cannot be opened, the filter will try again by appending ".pem" and then ".private". '#'-delimited comments and blank lines are ignored. - DKIM verification now logs a note (at log level 2) when a signature timestamp is in future; - allow expiration time (tag x) to be requested with DKIM signing, it is now supported since Mail::DKIM 0.29; - when determining which DKIM-signing key should be applied or which disclaimer options to apply, consider also addresses in all Resent-Sender header fields. The search order is: From, followed by Resent-From and Resent-Sender address pairs traversed top-down by resent blocks, followed by Sender, and by envelope sender; - amavisd-report no longer lets amavisd strip header fields found in a quarantined message which were previously inserted by amavisd; most of X-Spam-* and X-Amavisd-* header fields are now retained in a reported message; suggested by Achraf Tangui; - support IPv6 when connecting over an INET socket to virus scanners; - support SMTP, LMTP and TCP_LOOKUP protocols also over Unix sockets; - added an LDAP attribute amavisArchiveQuarantineTo to code and to LDAP.schema; a patch was provided by Anand Palaniswamy (back in October 2006); prompted by Quanah Gibson-Mount, both of zimbra.com. - new file in the package: LDAP.ldif, same schema as in LDAP.schema, but in ldif format; contributed by Quanah Gibson-Mount and independently also by Michael Hall; - @remove_existing_spam_headers_maps is now a per-recipient list of lookup tables, so pre-existing X-Spam* header fields may be selectively removed according to preferences of individual recipients or sub-domains, e.g.: @remove_existing_spam_headers_maps = ({ 'user@example.com' => 0, 'user@office.example.net' => 1, '.office.example.net' => 0, '.' => 1, # all the rest }); - added a macro b64encode, which could be used in log template to facilitate log parsing, perhaps by using the following in a $log_templ: [? [:header_field|Subject]||, \ Subject: [:b64encode|[:header_field|Subject|100]]]# Suggested by Rajkumar S; - added a macro HEADER as a synonym for a macro header_field for compatibility with SpamAssassin; - added a configuration variable $logline_maxlen (default value is 980, lower bound is 50), allowing user to customize syslog line wrapping threshold; based on a patch by Charles A. Scheidecker; - when loading a policy bank, most entries from an associative array (hash) being loaded entirely replace entries of the same key in the currently active policy bank, but entries which are references to a hash are normally merged with existing hashes, replacing only specified key/values but leaving remaining (non-existent in a new hash) key/values pairs unchanged. In some rare cases it would be more desirable to entirely replace existing hashes, which so far was not possible. For this purpose a hack was introduced: if a hash in a policy bank being loaded contains a key name 'REPLACE' (uppercase) and its value is true, this hash replaces a current hash, instead of being merged with it. For example, a policy bank 'DITCH' when loaded replaces a hash %final_destiny_by_ccat entirely, leaving only a key CC_CATCHALL there: $policy_bank{'DITCH'} = { final_destiny_by_ccat => { REPLACE=>1, CC_CATCHALL() => D_DISCARD }, }; Without specifying a REPLACE=>1 remaining key/value pairs in a hash %final_destiny_by_ccat would stay unchanged and only the CC_CATCHALL key/value pair would be replaced by a new setting (which may not be desirable): $policy_bank{'DITCH'} = { final_destiny_by_ccat => { REPLACE=>0, CC_CATCHALL() => D_DISCARD }, }; or equivalently: $policy_bank{'DITCH'} = { final_destiny_by_ccat => { CC_CATCHALL() => D_DISCARD }, }; - rewritten and expanded documentation section on DKIM signing and verification in amavisd-new-docs.html; OTHER - tested with perl 5.10.0; - package Amavis::IO::FileHandle now supports a method READ (i.e. invoked by a perl functions sysread through a tied hash), which is needed by SpamAssassin revisions since 2008-09-25 (3.3), bringing a little speedup to transferring a message from amavisd to SpamAssassin, and avoiding a Perl I/O bug (perl bug 39060; SA: bug 5985) on some installations; - updated @virus_name_to_spam_score_maps to recognize new malware name formats used by some popular third-party ClamAV signatures (Sanesecurity, MSRBL, MBL); thanks to Mike Cappella, Gary V, Wijatmoko U. Prayitno, Steve Basford, Luca Gibelli, Bill Landry, Henrik K; - keep only one (unique) copy of each malware/spam name when infection is downgraded to spam through @virus_name_to_spam_score_maps or when reported as a virus; - macro F now only shows the first (if any) banned leaf part name, preceded by comments from a banning rule regexp (if any), instead of a list of multiple banned parts each with its full MIME/archive path. Note that Perl syntax for a comment within a regexp is: (?# ... ) For example, given the following rule... $banned_filename_re = new_RE( qr'^\.(exe-ms|dll)$(?# rule #9)', # banned file(1) types ); ... a macro expansion of macros banning_rule_key, banning_rule_comment, banning_rule_rhs, banned_parts and F will be: banning_rule_key: (?-xism:^\\.(exe-ms|dll)$(?# rule #9)) banning_rule_comment: rule #9 banning_rule_rhs: 1 banned_parts: multipart/mixed | application/octet-stream,.exe,.exe-ms,videos.exe F: rule #9:application/octet-stream,.exe,.exe-ms,videos.exe Likewise an SMTP response (with D_REJECT) would match a macro F as before and would yield: 554 5.7.0 Reject, id=42721-01 - BANNED: rule #9:application/octet\ -stream,.exe,.exe-ms,videos.exe A default administrator and recipient notification (still using a macro F in their template), a main log entry, as well as a DSN and a rejection message, will now be shorter and hopefully less confusing to an end user. A full list of banned part paths (as previously produced by a macro F) is now available under a new name as a macro 'banned_parts' and can be used in custom templates to retain previous behaviour if desired. Suggested by Andreas Schulze and Peer Heinlein; - remove a 'LIMIT 1' from default $sql_clause{'sel_penpals_msgid'} and from $sql_clause{'sel_penpals'} clauses, it is redundant and it happens to make a MySQL 5.1 optimizer choose a slow plan; investigated by Michael Scheidell; - changed a default $sql_select_policy from: SELECT *, users.id FROM users LEFT JOIN policy ON ... into a: SELECT users.*, policy.*, users.id FROM users LEFT JOIN policy ON ... MySQL and PostgreSQL are happy with a 'SELECT *, users.id', but Oracle wants 'SELECT users.*, policy.*, users.id', which is also acceptable to MySQL and PostgreSQL and shouldn't make any difference; problem reported and a solution provided by Chris Bryant; - optimize storage of DKIM signing keys when multiple calls to dkim_key() (from amavisd.conf) specify the same file to be associated with different keys - now only one copy of a private key is kept in memory; - sanitize (strip) bare CR characters in mail before DKIM-signing a message and when forwarding it over SMTP (or LMTP) protocol which prohibits CR characters outside of CRLF pairs. Previously a DKIM signature generated by amavisd on messages with embedded bare CR characters broke when passed back through Postfix (following a principle of garbage-in, garbage-out). This was mainly an issue when a message was incorrectly sanitized or a disclaimer added by an external program such as altermime, which (due to a bug in versions 0.3.10 and older) could inappropriately introduce CR characters into a message. Reported by Patrick Wong; - retain original rfc822 quoting of envelope sender address when forwarding mail, instead of using a sanitized version (de-quoted & re-quoted); - insert autolearn=... information field into an X-Spam-Status header field, similar to how SpamAssassin does it; suggested by Jonathan Skanes; - an SQL field msgs.spam_level now receives a sum of SA score plus a minimum of internally generated score boosts across all message recipients, to facilitate coarse assessment by third party utilities without having to look into msgrcpt records; previously it only reflected a SA score (but field msgrcpt.bspam_level remain unchanged, storing a sum for each individual recipient as before); - insert "AM:BOOST=boost_scores_list" into a list of triggered spam tests to make visible the internally generated per-recipient spam score boosts (like from: pen pals, soft white/black-listing, bounce killer) in the log and in the X-Spam-Status header field. The 'tests' list in X-Spam-Status or in the log (macro %T) can now look like: tests=[AM:BOOST=+1.3+0.51-1.1, BAYES_99=3.6, ...] Multiple summands appear in multi-recipient messages where boost scores differ between recipients (the list is squashed, only unique values are shown). Apparent mismatch in score addition pointed out by John Beranek. - when quarantining and generating notification for administrator and recipients, and the per-recipient contents category differs from a per-message summarized contents category, use the per-recipient contents category for lookups into settings; a desirable side effect is that recipients which are bypassing some tests (like tests for virus or spam) no longer receive a recipient notification for cases they are not interested in; undesired behaviour pointed out by Erin D. Hughes; - drop log level of 'smtp resp to NOOP' and related messages from 2 to 3 to reduce log clutter; reported by kfx; - log a warning during startup when DKIM verification is not enabled (when $enable_dkim_verification is at its default value of undef). To quench down the warning and keep DKIM verification disabled, set the $enable_dkim_verification explicitly to 0; - when mail with banned part is to be passed but defanged, provide a more informative warning in the text part: WARNING: banning rules detected suspect part(s), do not open unless you know what you are doing suggested by Gerald Macinenti; - amavisd-agent: Content*Msgs* now take as a 100% reference the InMsgs$2 counter instead of Content$1Msgs; - when exec in a forked process fails, call POSIX::_exit with exit status 6 (SIGABRT) instead of 8 which has different meanings on different OS; - ensure a BDB cursor is unlocked in put_initial_snmp_data() even in case of errors or signals during writing of the initial sys* set of SNMP variables; - provide a generous but firm 4 MB sanity limit on a header section size to avoid excessive storage requirements while parsing and storing a runaway header section; the limit also protects DKIM signature verifier on huge headers; exceeded limit does not affect other mail checks and forwarding, only access to individual header fields beyond the limit is crippled and DKIM signatures would most likely be invalidated; - do not pre-load module Mail::SPF::Query with versions of SpamAssassin 3.2.0 or later, it has been replaced by Mail::SPF; - internal: modify mail_to_local_mailbox and do_quarantine to better deal with suppressing multiple quarantining to the same mailbox, e.g. when two quarantining methods are active but point to the same file; - internal: brush up I/O modules for consistency: open method should implicitly close a previously open file, print method should print all its arguments, read methods now support reading to a buffer at an offset; - internal: change most calls to lookup() into calls to a newer lookup2() for added flexibility, adding option 'Label' to some calls to facilitate debugging; - internal: passing options to lookup2 (and to other subordinate lookup methods) as a hash instead of a hashref; - internal: renamed Amavis::In::Message::PerRecip methods: banned_keys -> banning_rule_key, banned_rhs -> banning_rule_rhs, added: banning_rule_comment and banning_reason_short; - internal: replace subroutine unique with two: unique_list and unique_ref; - internal: remove dependency on a module IO::Wrap; - internal: many rather cosmetic changes for consistency, updated comments; - README.sql-mysql: add 'ALTER table' suggestions to change CHAR to BINARY and VARCHAR to VARBINARY data types; suggested by Peter Huetmannsberger: --------------------------------------------------------------------------- June 29, 2008 amavisd-new-2.6.1 release notes BUG FIXES - avoid a bounce-killer's false positive when a message is multipart/mixed with an attached message/rfc822 (looking like a qmail or a MSN bounce) and having attached a message with a foreign Message-ID - by restricting the check to messages with an empty sender address or a 'postmaster' or 'MAILER-DAEMON' author address; - privileges were dropped too early when chrooting, causing chroot to fail (a workaround was to specify a jail directory through a command line option -R); reported by Helmut Schneider; - fix unwarranted 'run_av error: Exceeded allowed time' error when using a virus scanned Mail::ClamAV; reported by Chaminda Indrajith; - fix a bug in helper-progs/amavis-milter.c where atoi could be reading from a non-null terminated string which could result in wrong milter return status, or even cause a read-access violation; reported by Shin-ichi Nagamura; - dsn_cutoff_level was ignored if SpamAssassin was not invoked (e.g. on large messages) even if recip_score_boost was nonzero, causing a DSN not to be suppressed for internally generated large score values; reported by Bernd Probst; - add back the 'Ok, id=..., from MTA(...):' prefix to an MTA status responses on forwarded mail when generating own SMTP status response (it was lost in code transition from 2.5.4 to 2.6.0); reported by Thomas Gelf; - replaced '-ErrFile=>*STDOUT' with '-ErrFile=>\*STDOUT' in a call to BerkeleyDB::Env::new in amavisd-nanny and amavisd-agent; seems it was failing in some setups (even though it was in accordance with a BerkeleyDB module documentation); reported by Leo Baltus; - README.sql-mysql: fixed an SQL data type mismatch between maddr.id (used as a foreign key) and msgs.sid & msgrcpt.rid; they all should be of the same type, either integer unsigned or bigint unsigned; a schema as published in README.sql-mysql could not be built because of a conflict in a data type; reported by Leonardo Rodrigues Magalhães and Zhang Huangbin; NEW FEATURES - recognize an additional place-holder %P in a template used to build a file name in file-based quarantining, for example: $spam_quarantine_method = 'local:Week%P/spam/%m.gz'; A %P is replaced by a current partition tag, which makes it easier to better organize a file-based quarantine by including a partition tag (e.g. an ISO week number) in a file name or a file path. For the record, here is a complete list of place-holders currently recognized in filename templates: %P => $msginfo->partition_tag %b => $msginfo->body_digest %m => $msginfo->mail_id %n => $msginfo->log_id %i => iso8601 timestamp of a message reception time by amavisd %% => % The following example organizes spam quarantine into weekly subdirectories: cd /var/virusmails mkdir -p W01/spam W02/spam ... W53/spam (weeks 01..53) chown -R vscan:vscan W01 W02 ... W53 (weeks 01..53) amavisd.conf: $spam_quarantine_method = 'local:W%P/spam/%m.gz'; $sql_partition_tag = sub { my($msginfo)=@_; sprintf("%02d",iso8601_week($msginfo->rx_time)) }; - add a macro %P as a synonym for a macro 'partition_tag', mainly for completeness with the added place-holder %P in a file name template; OTHER - disabled a do_ascii decoder in the default @decoders list: # ['asc', \&Amavis::Unpackers::do_ascii], # ['uue', \&Amavis::Unpackers::do_ascii], # ['hqx', \&Amavis::Unpackers::do_ascii], # ['ync', \&Amavis::Unpackers::do_ascii], The do_ascii is invoking a module Convert::UUlib which in turn calls a troublesome library uulib, which has a history of security problems and on occasion misinterprets a text file as some encoded text, causing false positives (e.g. making it look like an executable); a recent false positive on base64-decoding reported by Jeffrey Arbuckle; a recent DoS (looping in uulib) reported by Thomas Ritterbach; - added a rule into $map_full_type_to_short_type_re to cope with another example of misclassification by a file(1) utility, where a plain text file is considered a DOS executable: [qr/^DOS executable \(COM\)/ => 'asc'], # misclassified? An example was provided by Leonardo Rodrigues Magalhães; - until the issue is better understood, revert the use of 'my_require' and go back to the standard but less informative 'require'; some people were reporting problems with my_require (loading of some Perl modules can fail, apparently depending on a current directory where amavisd is started from); reports by Tuomo Soini, Max Matslofva, Bill Landry; - use the $myproduct_name value in generated Received header field instead of a hard-wired 'amavisd-new'; suggested by Thomas Gelf; - added missing required header fields to some test mail messages in a directory test-messages to quench down a complaint about a bad header; - changed SQL default clauses in %sql_clause (upd_msg, sel_quar, sel_penpals) to always join tables using both the partition_tag and the mail_id fields, previously just the mail_id field was used in a join. The change has no particular effect (and is not really necessary) on existing 2.6.0 databases where a primary key is mail_id (it is just a redundant extra condition), but saves a day when a primary key is a composite: (partition_tag,mail_id), which may be a requirement of an SQL partitioning mechanism. Thanks to Thomas Gelf for his testing of MySQL partitioning, reporting deficiency in amavisd SQL schema (primary keys) which did not meet MySQL requirements for partitioning, and suggestions; - an AM.PDP release request can specify an additional optional attribute: partition_tag=xx where a requester can supply a partition_tag value of a message to be released. This helps to uniquely identify a message in case where an SQL database did not enforce a mail_id field to be unique (as may be necessary with some partitioning schemes). If a partition_tag information is readily available to a requester, it is advised that the attribute is included in a request even if mail_id is known to be unique. This may expedite a search and provide a double check to a validity of a request. For backward compatibility amavisd performs a query on msgs.mail_id for a partition_tag value if it is missing form a request, the query uses an SQL clause in a new entry $sql_clause{'sel_msg'}. If exactly one record matches, then everything is fine, and releasing may proceed. If multiple records with the same mail_id exist the release request is aborted with a message asking user to supply a disambiguating partition_tag=xx attribute; - a quarantine id for an SQL-quarantined message as logged in a main log entry is changed from: quarantine: aX3C4f6btXgX to: quarantine: aX3C4f6btXgX[25] i.e. a partition_tag in brackets is appended to mail_id. Correspondingly the amavisd-release is also changed to be able to parse 'aX3C4f6btXgX[25]', splitting it into mail_id and partition_tag, and providing each as a separate attribute in an AM.PDP release request; - README.sql-mysql: changed SQL datatype VARCHAR into VARBINARY for data fields mail_id, secret_id and quar_loc, and CHAR into BINARY for msgs.content and msgs.quar_type to preserve case sensitivity on string comparison operators; suggested by Thomas Gelf; The same change should eventually be done on README.sql-pg too, but as PostgreSQL is more picky than MySQL on matching a field data type to a supplied data value, the change of a data type would need to be reflected in SQL calls in amavisd. This will have to wait until some future version of amavisd-new, having to undergo more testing than I have available before the 2.6.1 release. Background information on UNIQUE constraint in table SQL msgs Amavisd does not know and need not be aware of what is a primary key or what are UNIQUE constraints in SQL table msgs. When generating a mail_id for a message being processed, amavisd tries to INSERT a record with a randomly generated mail_id into table msgs (using SQL clause in $sql_clause{'ins_msg'}). If the operation fails, another mail_id is generated and attempt repeated, until it eventually succeeds. Thus it depends entirely on SQL's decision whether a particular record is allowed or would break some UNIQUE constraint. So, by only changing a declaration on table msgs (PRIMARY KEY or adding a CONSTRAINT), it changes what keys amavisd will be allowed to insert and what kind of duplicates would be allowed. Classically the msgs.mail_id is a PRIMARY KEY and as such it is unique. This was a requirement for versions of amavisd up to and including 2.6.0. Starting with 2.6.1 the JOINs have been tightened to include a partition_tag besides mail_id in a relation, which makes it possible to loosen a unique requirement on msgs.mail_id and only require a pair (partition_tag,mail_id) to be unique. In other words, this way the mail_id is only needed to be unique within each partition tag value. This change allows a partitioning scheme to meet requirements on MySQL partitioning. For non-partitioned databases the change shouldn't make any difference, and one is free to choose between having mail_id unique across the entire table or just within each partition_tag value. Changing a primary key to (partition_tag,mail_id) brings consequences to quarantining, in particular to releasing from an SQL quarantine, where it no longer suffices to specify mail_id=xxx in AM.PDP request, but may be necessary to specify also a partition_tag=xx to distinguish between SQL-quarantined messages which happen to have the same mail_id. --------------------------------------------------------------------------- April 23, 2008 amavisd-new-2.6.0 release notes MAIN NEW FEATURES SUMMARY - integrated DKIM signing and verification; see sections A QUICK START TO DKIM VERIFICATION and A QUICK START TO DKIM SIGNING by the end of this release note; - loading of policy banks based on valid DKIM-signed author's address can be used for reliable whitelisting, for bypassing banned checks, etc. - bounce killer feature: uses a pen pals SQL lookup to check inbound DSN; - SQL logging and quarantining tables have a new field 'partition_tag'; - captures SpamAssassin logging, more flexibility specifying SA log areas; - collects and logs SpamAssassin timing breakdown report (requires SA 3.3); - releasing from a quarantine can push a released message to an attachment; - new experimental code for abuse reporting using formats: ARF/attach/plain; - TLS support on the SMTP client and server side; - connection caching by an SMTP client; - amavisd-nanny and amavisd-agent now re-open a database on amavisd restarts; - amavisd-nanny and amavisd-agent new command line option: -c count; - updated p0f-analyzer.pl to support source port number in queries; - amavisd can send queries either to p0f-analyzer.pl or directly to p0f; COMPATIBILITY WITH 2.5.4 - when using SQL for logging (e.g. for a pen pals feature) or for quarantining, SQL tables tables maddr, msgs, msgrcpt and quarantine need to be extended by a new field 'partition_tag'; see below for details; - when SQL logging (pen pals) or SQL lookups are used, one can choose a binary or a character data type for fields users.email, mailaddr.email, and maddr.email; now may be a good opportunity to change a data type to binary (string of bytes); see below for details; - when using SQL for logging, a default for $sql_clause{'upd_msg'} has changed, so if a configuration file replaces this SQL clause by a non-default setting, it needs to be updated; - perl module Mail::DKIM is now required when DKIM verification or signing is enabled or when spam checking by SpamAssassin is used and a DKIM plugin is enabled; a required version of this module is 0.31 (or later); - because privileges are now dropped sooner, pid and lock files as generated by Net::Server can no longer be located in a directory which is not writable by UID under which amavisd is running (e.g. /var/run). A location of these files is controlled by $pid_file and $lock_file settings, and by default are placed in $MYHOME, which still satisfies the new requirement; - white and blacklisting now takes into account both the SMTP envelope sender address, as well as the author address from a header section (address(es) in a 'From:' header field). Note that whitelisting based only on a sender-specified address is mostly useless nowadays. For a reliable whitelisting see @author_to_policy_bank_maps below, as well as a set of whitelisting possibilities in SpamAssassin (based on DKIM, SPF, or on Received header fields); - if using custom hooks, some of the internal functions have changed, in particular the semantics of a method orig_header_fields - use new functions get_header_field() or get_header_field_body() instead; see updated sample code amavisd-custom.conf, and see entries labeled 'internal' below; - a configuration variable $append_header_fields_to_bottom is now obsolete; the variable is still declared for compatibility with old configuration files, but its value is ignored: new header fields are always prepended, i.e. added to the top of a header section; - semantics of a command line option 'debug-sa' has changed due to a merge of SpamAssassin logging with a mainstream amavisd logging mechanism. A command 'amavisd debug-sa' is now equivalent to 'amavisd -d all' with an implied redirection of all logging to stderr. Previously it only rerouted SpamAssassin logging to stderr but did not affect normal amavisd logging, which still followed the usual $DO_SYSLOG and $LOGFILE settings. Also, a SpamAssassin log level 'info' is now turned on by default (as was previously achievable by a command line option '-d info'), and shows merged with a normal amavisd logging at level 1 or higher. The following table shows mapping of SpamAssassin log levels to amavisd log levels, and for completeness also shows mapping of amavisd log levels to syslog priorities (which has not changed since previous version): SA amavisd syslog ----- ------- ----------- -3 LOG_CRIT -2 LOG_ERR error -1 LOG_WARNING warn 0 LOG_NOTICE info 1 LOG_INFO 2 LOG_INFO dbg 3 LOG_DEBUG 4 LOG_DEBUG 5 LOG_DEBUG - an additional requirement for loading a policy bank 'MYUSERS' is that 'originating' flag must be on, which typically means that mail must be coming from internal networks or from authenticated roaming users to be able to load a policy bank 'MYUSERS'; BUG FIXES - run_av: limit the number of filenames given as arguments to a command line scanner to stay within a safe (POSIX) program argument space limit, run a command line scanner multiple times if necessary. This required a larger change in the program (run_av, ask_av) which is why the fix was listed for a long time on a TODO list and not implemented so far. The problem affected command line virus scanners which are unable to traverse a directory by themselves and need a list of filenames as arguments (such as KasperskyLab's aveclient and kavscanner, MkS_Vir mks, and CyberSoft VFind). Actual problem reported by Danny Richter; NEW FEATURES - DKIM signing and verification - see sections below: A QUICK START TO DKIM VERIFICATION and A QUICK START TO DKIM SIGNING. Not to forget upgrading Mail::DKIM to 0.31 (or later) and adding the following to amavisd.conf; $enable_dkim_verification = 1; $enable_dkim_signing = 1; - SQL tables tables maddr, msgs, msgrcpt and quarantine are extended by a new field 'partition_tag'. When amavisd creates new records in these tables, a current value of a configuration variable $sql_partition_tag (or its value from policy banks) is written into 'partition_tag' fields. An undefined value translates to 0. The 'partition_tag' field is usually declared in a schema as an integer, but in principle could be any data type, such as a string. A value of 'partition_tag' field may be used to speed up purging of old records by using partitioned tables (MySQL 5.1 +, PostgreSQL 8.1 +). A sensible value is a week number of a year, or some other slowly changing value, allowing to quickly drop old table partitions without wasting time on deleting individual records. Records in all tables carrying the 'partition_tag' field are self-contained within each value of a field. In other words, foreign keys never reference a record in a subordinate table with a value of a 'partition_tag' field different from the referencing record. Consequently, mail addresses in table maddr are also self-contained within a partition tag, implying that the same mail address may appear in more than one maddr partition (using different 'id's), and that tables msgs and msgrcpt are guaranteed to reference a maddr.id within their own partition tag. Too fine a granularity of partition tags (e.g. changing a value daily) wastes space in table maddr by storing multiple copies of the same mail address. The $sql_partition_tag may be a scalar (usually an integer or a string), or a reference to a subroutine, which will be called with an object of type Amavis::In::Message as argument (giving access to information about a message being processed), and its result will be used as a partition tag value. Possible/typical usage (in amavisd.conf): $sql_partition_tag = sub { my($msginfo)=@_; iso8601_week($msginfo->rx_time) }; yields an ISO 8601 (EN 28601) week number (1..53) corresponding to a mail reception timestamp in a local time zone. This week number definition is equivalent to PostgreSQL extract(week from ...), and MySQL week(date,3). Another possible use of 'partition_tag' field is to let a policy bank set its specific value (a fixed value or a subroutine) for $sql_partition_tag. This would allow for example labeling of SQL records for mail originating from inside with a different partition_tag value, compared to entries for incoming mail, and consequently let them be stored in a separate partition if desired. Amavisd process itself does not use the 'partition_tag' field for its own purposes, all records regardless of their 'partition_tag' value are available for example to pen pals lookups, as before. The field is provided only as a convenience to SQL database maintenance, and can be ignored by smaller sites where current practice of database maintenance is fast enough. If SQL partitioning is not in use (or not intended to be used in a near future), it is more economical to use a fixed value (such as 0, which is a default) for the $sql_partition_tag. Using week numbers as partition tags adds about 50 % to the number of records in table maddr, the exact number depends on retention period and a ratio of regular vs. infrequent mail addresses observed. To convert tables of an existing database, please use ALTER command. Here is a conversion example (MySQL or PostgreSQL, probably others): ALTER TABLE maddr ADD partition_tag integer DEFAULT 0; ALTER TABLE msgs ADD partition_tag integer DEFAULT 0; ALTER TABLE msgrcpt ADD partition_tag integer DEFAULT 0; ALTER TABLE quarantine ADD partition_tag integer DEFAULT 0; As the maddr.email is no longer guaranteed to be unique, but a pair of (maddr.partition_tag, maddr.email) is unique, the constraint and an associated index needs to be changed: => PostgreSQL: ALTER TABLE maddr DROP CONSTRAINT maddr_email_key, ADD CONSTRAINT maddr_email_key UNIQUE (partition_tag,email); => MySQL: ALTER TABLE maddr DROP KEY email, ADD UNIQUE KEY part_email (partition_tag,email); Should a need arise to revert to amavisd-new-2.5.4 while keeping the new partition_tag field, the 'SELECT id FROM maddr ...' may become slow due to dropped index on a field maddr.email, which is replaced by an index on a pair (maddr.partition_tag, maddr.email). The following change to amavisd 2.5.4 solves the problem: @@ -901,2 +901,2 @@ 'sel_adr' => - 'SELECT id FROM maddr WHERE email=?', + 'SELECT id FROM maddr WHERE partition_tag=0 AND email=?', The use of partitioned tables to speed up purging of old records was suggested by Robert Pelletier. - when SQL logging (pen pals) or SQL lookups are used, one can choose a binary or a character data type for fields users.email, mailaddr.email, and maddr.email; now may be a good opportunity to change a data type to binary (string of arbitrary bytes, no character set associated). Background: values of these fields come from SMTP envelope or from a mail header section of processed mail. Even though RFC 2821 and RFC 2822 restrict these addresses to 7-bit ASCII, there is nothing preventing a malicious or misguided sender from supplying any 8-bit byte values. If SQL fields are declared as VARCHAR or CHAR, a character set is associated with data and its rules apply, e.g. control characters may not be permitted, or UTF-8 byte sequences are validated, or a restriction to codes below 128 apply. Depending on strictness of an SQL server on validating data, a violation of character set rules may lead to aborting an SQL operation and failing of mail processing. Even though new standards for e-mail addresses are being negotiated allowing for UTF-8 encoding, an actual e-mail address may still supply arbitrary bytes, which may violate UTF-8 byte sequence rules. A new configuration variable $sql_allow_8bit_address now controls how amavisd passes e-mail addresses to SQL. If a value is true, then it is expected that SQL tables will accept strings of arbitrary bytes for these fields, without associating a character set with data. No data sanitation is done by amavisd. An appropriate SQL data type is 'VARBINARY' or with PostgreSQL a 'BYTEA'. If a value of $sql_allow_8bit_address is false (which is a default for compatibility) then amavisd performs sanitation before passing data to SQL: control characters and characters with codes above 127 are converted to '?', which brings strings within ASCII character set restrictions. A suitable SQL data type is VARCHAR or CHAR. Note that some information is lost in this case. The following clauses can convert pre-2.6.0 tables into the now preferred and more universal form: MySQL: ALTER TABLE users CHANGE email email varbinary(255); ALTER TABLE mailaddr CHANGE email email varbinary(255); ALTER TABLE maddr CHANGE email email varbinary(255); PostgreSQL: ALTER TABLE users ALTER email TYPE bytea USING decode(email,'escape'); ALTER TABLE mailaddr ALTER email TYPE bytea USING decode(email,'escape'); ALTER TABLE maddr ALTER email TYPE bytea USING decode(email,'escape'); If a binary data type is chosen for these three fields, the setting $sql_allow_8bit_address MUST be set to true to let the amavisd program use the appropriate data type in SQL commands, otherwise PostgreSQL will complain with: 'types bytea and character varying cannot be matched' when amavisd tries to execute SQL commands. MySQL is more forgiving and does not complain about a data type mismatch, so one may get away with a mismatch, although it is appropriate to eventually make it right. If a change of a data type of these fields is chosen while using some third-party management interface to SQL data set (e.g. MailZu), make sure the management interface supports the changed data type. This is primarily a concern with PostgreSQL which is more strict in requiring a match between field data types in tables and data in SQL clauses. The need for a change was pointed out by Xavier Romero, reporting that PostgreSQL SQL lookups with pre-2.6.0 versions of amavisd can fail when 8-bit data appears in SMTP envelope addresses: lookup_sql: sql exec: err=7, 22021, DBD::Pg::st execute failed: ERROR: invalid byte sequence for encoding "UTF8" - bounce killer feature: uses a pen pals SQL lookup to check inbound DSN, attempting to match it with a previous outbound message. If a Message-ID found in an attachment of the inbound DSN matches a Message-ID of a message previously sent from our system by a current recipient of the DSN, the DSN message is spared, otherwise it receives $bounce_killer_score spam score points (0 by default, i.e. disabled) and can be blocked as spam (although technically it is just a misdirected bounce, not spam). A received delivery status notifications is parsed looking for attached header section of an original message in an attempt to find a Message-ID. A standard DSN structure (RFC 3462, RFC 3464) is recognized, as well as a few nonstandard but common formats. Other automatic reports and bounces with unknown structure and no attached header section are ignored for this purpose (are subject to other regular checks). Unfortunately there are still many nonstandard mailers around (12+ years after DSN format standardization) and many ad-hoc filtering solutions which do not supply the essential information. A message is unaffected by this check and the DSN message is considered a genuine bounce if a Message-ID can be found in an SQL log database matching a previous message sent by a local user (which is now a recipient of a DSN), either using a normal pen pals lookup (no extra SQL operations are necessary), or if a domain part of the Message-ID is one of local domains. On the other hand, if the attached DSN header does look like a complete original header but it does not meet the above criteria, then it is assumed that the message is a backscatter to a faked address belonging to our local domains, and $bounce_killer_score spam score points are added, so the message can be treated as spam (subject to spam kill level and other spam settings). The only user-configurable setting is $bounce_killer_score (also member of policy banks), its default value is 0. To activate the bounce killer feature, set the $bounce_killer_score to a positive number, e.g. 100. A bounce killer score does not contribute to SpamAssassin's auto-learning. A pre-requisite for proper operation of a bounce killer is a working SQL logging database (pen pals), or at least that all outbound DSN messages have a Message-ID with a domain matching the @local_domains_maps list of lookup tables. The condition is easily met when all mail from local users is submitted through a domain's official mailer, which goes hand in hand with the requirement for DKIM signing and for other similar antispoofing techniques (SPF, whitelisting by IP address in Received trace, ...). A couple of SNMP-like counters are added to facilitate assessing effectiveness of the feature (e.g. viewed by amavisd-agent utility): InMsgsBounce 21310 333/h 9.9 % (InMsgs) InMsgsBounceKilled 19967 312/h 93.7 % (InMsgsBounce) InMsgsBounceRescuedByDomain 7 0/h 0.0 % (InMsgsBounce) InMsgsBounceRescuedByOriginating 242 4/h 1.1 % (InMsgsBounce) InMsgsBounceRescuedByPenPals 67 1/h 0.3 % (InMsgsBounce) InMsgsBounceUnverifiable 1027 16/h 4.8 % (InMsgsBounce) More information on operations can be obtained from a log, search for: inspect_dsn: bounce killed bounce rescued by penpals bounce rescued by domain bounce unverifiable The feature was suggested by Scott F. Crosby. See also http://www.postfix.org/BACKSCATTER_README.html, http://wiki.apache.org/spamassassin/VBounceRuleset and a SpamAssassin man page Mail::SpamAssassin::Plugin::VBounce for additional ideas on fighting joe-jobbed backscatter mail. - a new configuration variable @author_to_policy_bank_maps (also a member of policy banks) is a list of lookup tables (typically only a hash-type lookup table is used), which maps author addresses(es) (each address in a 'From:' header field - typically only one) to one or more policy bank names (a comma-separated list of names). A match can only occur if a valid DKIM author domain signature or a valid DKIM third-party signature is found, so in as much as one can trust the signing domain, loading of arbitrary policy banks can be safe, offering a flexibility of whitelisting against spam (absolute or just contributing score points), bypassing of checks (banned, virus, bad-header), using less restrictive banned rules for certain senders, by-sender routing, turning quarantining/archiving on/off, and other tricks offered by the existing policy bank loading mechanisms. When a message has a valid DKIM (or DomainKeys) author domain signature (i.e. when a 'From:' address matches a signing identity according to DKIM (RFC 4871) or DomainKeys (RFC 4870) rules), a lookup key is an unchanged author address and the usual lookup rules apply (README.lookups - hash lookups). When a valid third-party signature is found, a lookup key (author address) is extended by a '/@' and a lowercased signing domain, as shown in the example below. The semantics is very similar to a whitelist_from_dkim feature in SpamAssassin, but is more flexible as is allows any dynamic amavisd setting to be changed depending on author address, not just skipping of spam checks. A few examples of a SpamAssassin's whitelist_from_dkim (as in local.cf) along with equivalent amavisd @author_to_policy_bank_maps entries follow. To whitelist any From address with a domain example.com when a message has a valid author domain signature (i.e. a signature by the same domain): SA: whitelist_from_dkim *@example.com am: 'example.com' => 'WHITELIST', which is equivalent to a lengthy but redundant: SA: whitelist_from_dkim *@example.com example.com am: 'example.com/@example.com' => 'WHITELIST', Similar to above, but applies to subdomains of example.com carrying a valid author domain signature (i.e. signature BY THE SAME SUBDOMAIN): SA: whitelist_from_dkim *@*.example.com am: '.example.com' => 'WHITELIST', Note that in amavisd hash lookups a '.example.com' implies a parent domain 'example.com' too, while in SpamAssassin and in Postfix maps a parent domain needs its own entry if desired. To whitelist From addresses from subdomains of example.com which carry a valid third-party signature of its parent domain: SA: whitelist_from_dkim *@*.example.com example.com am: '.example.com/@example.com' => 'WHITELIST', To whitelist any From address as long as a message has a valid DKIM or DomainKeys signature by example.com, i.e. a third-party signature. Typical for mailing lists or discussion groups which sign postings. SA: whitelist_from_dkim *@* example.com am: './@example.com' => 'WHITELIST', Here is a complete example that can be included in amavisd.conf: @author_to_policy_bank_maps = ( { # 'friends.example.net' => 'WHITELIST,NOBANNEDCHECK', # 'user1@cust.example.net' => 'WHITELIST,NOBANNEDCHECK', '.ebay.com' => 'WHITELIST', '.ebay.co.uk' => 'WHITELIST', 'members.ebay.co.uk/@ebay.co.uk' => 'WHITELIST', 'ebay.at' => 'WHITELIST', 'ebay.ca' => 'WHITELIST', 'ebay.fr' => 'WHITELIST', 'ebay.de' => 'WHITELIST', 'members.ebay.de/@ebay.de' => 'WHITELIST', '.paypal.co.uk' => 'WHITELIST', '.paypal.com' => 'WHITELIST', # author domain signatures './@paypal.com' => 'WHITELIST', # 3rd-party sign. by paypal.com 'alert.bankofamerica.com' => 'WHITELIST', 'ealerts.bankofamerica.com'=> 'WHITELIST', 'amazon.com' => 'WHITELIST', 'amazon.de' => 'WHITELIST', 'amazon.co.uk' => 'WHITELIST', 'cisco.com' => 'WHITELIST', '.cnn.com' => 'WHITELIST', 'skype.net' => 'WHITELIST', 'welcome.skype.com' => 'WHITELIST', 'cc.yahoo-inc.com' => 'WHITELIST', 'cc.yahoo-inc.com/@yahoo-inc.com' => 'WHITELIST', '.linkedin.com' => 'MILD_WHITELIST', 'google.com' => 'MILD_WHITELIST', 'googlemail.com' => 'MILD_WHITELIST', './@googlegroups.com' => 'MILD_WHITELIST', './@yahoogroups.com' => 'MILD_WHITELIST', './@yahoogroups.co.uk' => 'MILD_WHITELIST', './@yahoogroupes.fr' => 'MILD_WHITELIST', 'yousendit.com' => 'MILD_WHITELIST', 'meetup.com' => 'MILD_WHITELIST', 'dailyhoroscope@astrology.com' => 'MILD_WHITELIST', } ); $policy_bank{'MILD_WHITELIST'} = { score_sender_maps => [ { '.' => [-1.8] } ], }; $policy_bank{'WHITELIST'} = { bypass_spam_checks_maps => [1], spam_lovers_maps => [1], }; $policy_bank{'NOVIRUSCHECK'} = { bypass_decode_parts => 1, bypass_virus_checks_maps => [1], virus_lovers_maps => [1], }; $policy_bank{'NOBANNEDCHECK'} = { bypass_banned_checks_maps => [1], banned_files_lovers_maps => [1], }; - smtp client connection caching is a new feature which allows smtp client code in amavisd to keep an SMTP session to MTA open after forwarding a message or a notification, so that a next mail message that needs to be sent by this child process can avoid re-establishing a session and the initial greeting/EHLO (and TLS) handshake. A current value of a global settings $smtp_connection_cache_enable controls whether a session will be retained after forwarding a message or not. Its default initial value is true. A global setting $smtp_connection_cache_on_demand controls whether amavisd is allowed to dynamically change the $smtp_connection_cache_enable setting according to its estimate of the message frequency. The heuristics is currently very simple: if time interval between a previous task completion by this child process and the arrival of a current message is 5 seconds or less, the $smtp_connection_cache_enable is turned on (which will affect the next message); if the interval is 15 seconds or more, it is turned off. The default value of the $smtp_connection_cache_on_demand is true, thus enabling the adaptive behaviour. On a busy server the connection caching can save some processing time. Savings are substantial if client-side TLS is enabled, otherwise just a few milliseconds are saved. On an idle server the feature may unnecessarily keep sessions to MTA open (until MTA times them out), so one can disable the feature by setting both controls to false (to 0 or undef). To monitor the connection caching effectiveness, some SNMP-like counters were added, so amavisd-agent may display something like: OutConnNew 2764 319/h 98.2 % (OutMsgs) OutConnQuit 2521 291/h 89.5 % (OutMsgs) OutConnReuseFail 7 1/h 0.2 % (OutMsgs) OutConnReuseRecent 21 2/h 0.7 % (OutMsgs) OutConnReuseRefreshed 31 4/h 1.1 % (OutMsgs) OutConnTransact 2816 325/h 100.0 % (OutMsgs) - client-side TLS support is added, i.e. on forwarding a passed mail back to MTA. Currently only encryption is supported, no client certificates are offered. A $tls_security_level_out is a per-policy-bank setting which controls client-side TLS, its value is either undefined (default), or a string: undef ... client-side TLS is disabled (a default setting); 'may' ... TLS is used if MTA offers a STARTTLS capability (RFC 3207), otherwise a plain text SMTP session is established; 'encrypt' TLS is used if MTA offers a STARTTLS capability, otherwise amavisd refuses to forward a message. The client-side TLS imposes some performance penalty on passing a message back to MTA, although it is still reasonably fast: a benchmark indicates a drop in transfer rate by about a factor of 2, from 22 MB/s (no TLS) to 9 MB/s (with TLS). The smtp client connection caching (see previous item) should preferably be left enabled (permanently or opportunistically), as TLS negotiation adds significantly to the initial SMTP handshake time. - server-side TLS support is added, i.e. on accepting mail from MTA. Encryption is supported, server (i.e. amavisd) offers its certificate, but client certificates are not verified. A $tls_security_level_in is a per-policy-bank setting which controls server-side TLS, its value is either undefined (default), or a string: undef ... server-side TLS is disabled, STARTTLS capability is not offered; 'may' ... STARTTLS capability is offered by amavisd, but client is not required to enter TLS, plain text sessions are permitted; 'encrypt' STARTTLS capability is offered and enforced by amavisd, any SMTP command other than STARTTLS, NOOP, EHLO or QUIT is rejected. If $tls_security_level_in is enabled (any value other than undef or 'none'), amavisd offers a certificate to a connecting client requesting TLS, so a path to a certificate and to its private key must be provided through two global settings: $smtpd_tls_cert_file and $smtpd_tls_key_file, e.g.: $smtpd_tls_cert_file = "$MYHOME/cert/amavisd-cert.pem"; $smtpd_tls_key_file = "$MYHOME/cert/amavisd-key.pem"; The private key should be guarded as secret (not world-readable). A self-signed certificate is acceptable by most mailers. Server-side TLS imposes a significant performance penalty on accepting a message from MTA. A benchmark indicates a drop in transfer rate by a factor of 10, from about 10 MB/s (no TLS) to 1 MB/s (using TLS), so it should only be enabled with a good reason or for experimentation. - enhanced a subroutine delivery_status_notification (along with dispatch_from_quarantine and msg_from_quarantine) to produce a message in one of several formats (derived from a message being processed, or from a quarantined message). Its new arguments can be strings as follows: $request_type: dsn, release, requeue, report $msg_format: dsn, arf, attach, plain, resend $feedback_type: abuse, fraud, miscategorized, not-spam, opt-out, opt-out-list, virus, other (according to ARF draft) Per-policy settings $report_format and $release_format control the format of a generated message. Their value can be one of the following strings, although not all combinations make sense: 'arf' .... an abuse report is generated according to draft-shafranovich-feedback-report-04: "An Extensible Format for Email Feedback Reports"; a plain-text part contains text from a template; 'attach'.. generates a report message as plain text according to a template, with an original message attached; 'plain'... generates a simple (flat) mail with an only MIME part containing a text from a template, followed inline by original message (some service providers can't handle abuse reports with attachments, e.g. Yahoo!); 'resend'.. original message is forwarded unchanged, except for an addition of header fields Resent-From, Resent-Sender, Resent-To, Resent-Date and Resent-Message-ID; 'dsn' .... (for internal use) a delivery status notification is generated according to rfc3462, rfc3464 and rfc3461; When a request_type is 'release' or 'requeue', the format of a generated message is governed by a per-policy setting $release_format according to the table above. Only the 'attach', 'plain' and 'resend' values are useful. A default setting is: $release_format = 'resend'; # with alternatives: attach, plain, resend A plain-text part (if used) is taken from a $notify_release_templ template and a sending address is obtained from %hdrfrom_notify_release_by_ccat. When a request_type is 'report', the format of a generated message is governed by a per-policy setting $report_format according to the table above. Only the following settings are useful: arf, attach, plain, resend. A default setting is: $report_format = 'arf'; # alternatives: arf, attach, plain, resend A plain-text part (if used) is obtained from a $notify_report_templ template, and a sending address from %hdrfrom_notify_report_by_ccat. It is possible to automatically generate abuse reports from custom hooks by calling delivery_status_notification() and mail_dispatch(). Extreme care must be taken to only produce legitimate abuse reports (about true fraud and true spam), sent only to parties that are truly responsible for a message being reported. Non-repudiation is a key factor here - trust only header fields covered by a valid DKIM signature, or generated by your own MX MTA (such as an IP address of the last hop), and only report messages received from a network which officially belongs to the party (according to whois). Rate-limiting should be used, and abuse reports on the same abuser should only be sent once in a time interval of several hours. A SQL database can be used to maintain a list of recently reported abusers, thus preventing excessive reports. - introduced a variation of a message release from a quarantine, allowing a releaser to send an abuse report based on a quarantined message. It is implemented by: * enhancing a subroutine delivery_status_notification as described in the previous item; * extending AM.PDP protocol with a 'request=report' attribute which can be used in place of a 'request=release', * enhancing the 'amavisd-release' utility program to allow sending an attribute 'request=release' or 'request=requeue' or 'request=report' based on its program name. By making a soft or hard link named 'amavisd-report' linking to 'amavisd-release', the utility will send a 'request=report' in place of the usual 'request=release', e.g.: # ln -s amavisd-release amavisd-report # ln -s amavisd-release amavisd-requeue $ amavisd-report spam/k/kg2P0rP9Lpu3.gz '' abuse@example.com - releasing from a quarantine can push a released message to an attachment (Content-Type: message/rfc822), with a configurable template for a header section and the plain-text part; select by: $release_format='attach'; suggested by Patrick Ben Koetter; - detect and save a new attribute SOURCE from an XFORWARD smtp command; the value is also accepted as AM.PDP protocol attribute 'client_source'. Possible values are: 'LOCAL', 'REMOTE', or '[UNAVAILABLE]', the information corresponds to 'local_header_rewrite_clients' postfix setting and is not supposed to be used for security decisions according to Postfix documentation (which makes it less interesting for our purposes); - added client and server support for a PORT attribute of an XFORWARD command, allowing MTA to pass a TCP port number of a remote client to a content filter (and back if necessary); the PORT attribute is made available with Postfix version 2.5 (20071004); a source port number is also accepted as an AM.PDP protocol attribute 'client_port'; - updated p0f-analyzer.pl now supports a source port number information in queries while preserving backward compatibility with previous versions of amavisd-new. Version 2.6.0 of amavisd requires a new version of p0f-analyzer.pl (supplied in the 2.6.0 distribution) if operating system fingerprinting is enabled. A source port number information in a query allows p0f-analyzer.pl to locate a matching entry in its cache faster and also more accurately when multiple connections are present from clients behind NAT using the same IP address. The source port number is made available to a content filter since Postfix version 2.5 (20071004); - besides the ability to send queries to p0f-analyzer.pl, amavisd now also supports sending queries directly to a p0f program over a Unix socket using a p0f query protocol. There is a bug in p0f-2.0.8 (and probably in earlier versions) which makes it send back incorrect results at times, i.e. results belonging to some other unrelated session, so a patch to p0f-2.0.8 MUST be applied in order to use a direct querying mechanism - author has been notified. The patch is supplied: p0f-patch. There are currently no advantages (and some disadvantages) in choosing direct queries to p0f, compared to sending queries to p0f-analyzer.pl, so this new method is not currently recommended. Disadvantages are: * p0f uses a linear search over its list of recent sessions (at least as of version 2.0.8), whereas p0f-analyzer.pl uses a fast hash lookup; * p0f keeps a relatively small list of recent sessions which is limited by the number of slots (size can be specified on a command line, but is limited by a linear search time), whereas p0f-analyzer.pl expires old entries according to time since entered and is thus independent of a current mail rate; * a direct p0f query protocol uses packed binary data and its on-the-wire representation may depend on a compiler used, so it may be incompatible with queries sent by amavisd, whereas the p0f-analyzer.pl queries and replies use a more environment-independent textual representation. To let amavisd sent queries directly to p0f, specify a p0f socket path: $os_fingerprint_method = 'p0f:/var/amavis/home/p0f.sock'; and specify an IP address and a port number on which MTA is listening: $os_fingerprint_dst_ip_and_port = '[192.0.2.3]:25'; because p0f requires this information in a query and the information is not made available to a content filter via XFORWARD command (the p0f-analyzer.pl does not need this information). To send queries to p0f-analyzer.pl (traditional and recommended), use: $os_fingerprint_method = 'p0f:127.0.0.1:2345'; as before. The $os_fingerprint_dst_ip_and_port in this case is not needed and is ignored. - usually a sending address in spam messages is faked and it is desirable to suppress most if not all bounces by keeping $sa_dsn_cutoff_level low, but sometimes it may be possible to be more certain of the validity of a sending address, and when such mail is considered spam, it may still be desirable to send a non-delivery notification, knowing that a notification will most likely be addressed to a genuine sender. Two new settings are provided for this purpose: @spam_crediblefrom_dsn_cutoff_level_bysender_maps and @spam_crediblefrom_dsn_cutoff_level_maps (with their default being $sa_crediblefrom_dsn_cutoff_level), complementing the existing @spam_dsn_cutoff_level_bysender_maps and @spam_dsn_cutoff_level_maps. It is expected that $sa_crediblefrom_dsn_cutoff_level would be set somewhat higher than $sa_dsn_cutoff_level, allowing for more bounces to be generated for spam from likely-to-be-genuine senders (possibly false positives). The choice between taking a cutoff value from one or the other pair of settings depends on an attribute $msginfo->sender_credible - when it is true (e.g. some nonempty string) the *spam_crediblefrom_* settings will be used instead of the baseline @spam_dsn_cutoff_level_*maps. An initial value of a sender_credible attribute as provided by amavisd is true if either the 'originating' flag is true (e.g. mail from inside), or if dkim_envsender_sig attribute is true, e.g. a domain of a valid DKIM signature matches envelope sender address, otherwise it is false. A user-provided custom hook code is free to change the value of sender_credible attribute. An exact value does not matter (it is only interpreted as a boolean), but serves for logging purposes. Heuristics may be based on some tests provided by SpamAssassin, on DKIM signatures, on p0f results, on policy banks, etc. Here is one complete example of a custom hook, which turns on the sender_credible attribute based on some criteria. Note that some of the referenced SpamAssassin tests may not yet be available in the last officially released version of SpamAssassin. added to amavisd.conf: include_config_files('/etc/amavisd-custom.conf'); /etc/amavisd-custom.conf : package Amavis::Custom; use strict; sub new { my($class,$conn,$msginfo) = @_; bless {}, $class } sub after_send { my($self,$conn,$msginfo) = @_; if ($msginfo->sender ne '') { my(@cred); local($1); my($tests) = $msginfo->supplementary_info('TESTS'); $tests = '' if !defined($tests) || $tests eq 'none'; push(@cred,'orig') if $msginfo->originating; push(@cred,$1) if $tests =~ /\b(RCVD_IN_DNSWL_HI)\b/; push(@cred,$1) if $tests =~ /\b(RCVD_IN_DNSWL_MED)\b/; push(@cred,$1) if $tests =~ /\b(RP_MATCHES_RCVD)\b/; my($os_fingerprint) = $msginfo->client_os_fingerprint; if ($os_fingerprint !~ /^Windows XP(?![^(]*\b2000 SP)/) { push(@cred,'dkim') if $msginfo->dkim_envsender_sig; push(@cred,$1) if $tests =~ /\b(SPF_PASS)\b/; } $msginfo->sender_credible(join(",",@cred)) if @cred; } } 1; # insure a defined return - a new setting $reputation_factor (also a member of policy banks) with a value between 0 and 1 (default 0.2), controlling an amount of 'bending' of a calculated spam score towards a fixed score assigned to a signer identity (i.e. its 'reputation') through @signer_reputation_maps; the formula is: adjusted_spam_score = f*reputation + (1-f)*spam_score; which has the same semantics as auto_whitelist_factor in SpamAssassin AWL; - a new setting @signer_reputation_maps (also a member of policy banks) may contain a list of lookup tables (typically just one hash lookup table), mapping a signing identity to a score, which is typically a long term average spam score of all messages signed by this signing identity. Based on a lookup result and a formula given above ($reputation_factor), the resulting value (positive or negative) is added to the spam score. Here is an example setting: @signer_reputation_maps = ( { 'ebay.fr' => -10.95, 'ebay.ca' => -9.57, 'ebay.co.uk' => -8.59, 'ebay.com' => -8.03, 'ebay.at' => -3.59, 'ebay.de' => -3.38, 'reply3.ebay.com' => -4.57, 'reply.ebay.com' => -3.20, 'paypal.com' => -6.66, 'intl.paypal.com' => -3.70, 'email.paypal.co.uk' => -0.67, 'alert.bankofamerica.com' => 1.35, 'ucsd.edu' => -7.89, 'izb.knu.ac.kr' => -7.51, 'ijs.si' => -4.44, 'tu-graz.ac.at' => -5.44, 'tugraz.at' => -4.03, 'aitech.ac.jp' => -3.03, 'univie.ac.at' => -2.99, 'uni-bremen.de' => -2.80, 'uu.se' => -2.54, 'univ-tours.fr' => -2.06, 'phys.huji.ac.il' => -2.01, 'cern.ch' => -1.69, 'prime.gushi.org' => -9.85, 'dostech.ca' => -9.31, 'resistor.net' => -9.18, 'kitterman.com' => -9.05, 'schetterer.org' => -9.04, 'hege.li' => -8.96, 'fouter.net' => -8.84, 'inetmsg.com' => -8.81, 'charite.de' => -8.81, 'porcupine.org' => -8.75, 'secnap.net' => -7.99, 'netoyen.net' => -7.58, 'state-of-mind.de' => -6.93, 'gmurray.org.uk' => -5.39, 'mtcc.com' => -4.93, 'messiah.edu' => -4.91, 'consulintel.es' => -2.16, 'delphij.net' => 1.06, 'channing-bete.com' => -9.11, 'megan.vbhcs.org' => -8.64, 'scent-team.com' => -8.32, 'suedfactoring.de' => -8.15, 'sendmail.net' => -5.14, 'cisco.com' => -4.95, 'hermes-softlab.com' => -3.77, 'altn.com' => -2.16, 'amazon.com' => 0.09, 'eurescom.eu' => -0.63, 'skype.net' => -1.50, 'welcome.skype.com' => -0.34, 'newsdesk.world-nuclear-news.org'=> -0.74, 'youtube.com' => 1.11, 'email.innocentive.com' => 1.98, 'update.hallmark.com' => 2.26, 'newsletters.trendmicro.com'=> 3.02, 'mail.communications.sun.com'=> 3.73, 'alerts.hp.com' => 0.51, 'email.greenpeace.org' => 2.04, 'avaaz.org' => 3.51, 'mail.cnn.com' => 4.12, 'm-w.com' => 4.81, 'medcompare.com' => 2.46, 'biocompare.com' => 3.48, 'dentalcompare.com' => 4.58, 'news.biomedcentral.com' => 2.89, 'yahoo.com' => -0.53, 'yahoo.se' => -1.48, 'yahoo.de' => -1.33, 'yahoo.co.uk' => 0.85, 'yahoo.ca' => 1.13, 'yahoo.no' => 1.22, 'yahoo.es' => 2.04, 'yahoo.ie' => 4.48, 'yahoo.fr' => 6.35, 'yahoo.it' => 7.09, 'yahoo.dk' => 7.10, 'yahoo.co.in' => 5.34, 'yahoo.com.cn' => 4.34, 'yahoo.com.ar' => 5.48, 'yahoo.co.jp' => 9.25, 'yahoo.com.au' => 9.30, 'yahoo.com.hk' => 13.97, 'yahoo.cn' => 21.94, 'yahoo-inc.com' => -1.57, 'yahoogroups.com' => -1.76, # 'yahoogroups.co.uk' => 0.10, # 'yahoogroupes.fr' => 5.99, 'yahoogroups.de' => 19.96, 'yahoogrupos.com.br' => 17.41, 'gmail.com' => -3.52, 'googlegroups.com' => -3.10, 'googlemail.com' => 3.42, 'google.com' => 1.47, 'prodigy.net' => -8.91, 'amis.net' => -2.38, 'earthlink.net' => -2.27, 'btinternet.com' => -1.96, 'pacbell.net' => -0.96, 'rogers.com' => 2.05, 'ipost.com' => 2.40, 'incertum.net' => -9.80, 'yousendit.com' => -4.70, 'news.yousendit.com' => 3.38, 'abv.bg' => -4.57, 'uclouvain.be' => -4.48, 'birthdayalarm.com' => -4.32, 'geni.com' => -3.50, 'mail120.subscribermail.com'=> -2.58, 'mail6.subscribermail.com' => -1.97, 'spock.com' => -11.67, 'meetup.com' => -1.94, '123greetings.com' => -1.92, 'rocketmail.com' => -1.91, 'arcamax.com' => -0.81, 'skynet.be' => -0.10, 'news.virtualtourist.com' => 3.58, 'bighip.com' => 10.53, 'investorsinsight.com' => 12.19, 'lspromos2007online.com' => 12.41, 'postmaster-direct.com' => 12.71, 'smsacfriends.com' => 12.88, 'specialtyofficial.com' => 13.03, 'usafisnews.org' => 13.82, 'freelotto.com' => 15.59, 'partnershopping.net' => 19.46, 'muffinlocate.com' => 19.49, 'stallrust.com' => 19.71, 'sugargrowth.com' => 19.75, 'washusual.com' => 20.01, 'poursmock.com' => 20.02, 'overseeinvest.com' => 20.19, 'waveholer.com' => 20.27, 'headattomic.com' => 20.32, 'moneysuffer.com' => 20.84, 'ibnyes.com' => 20.95, 'honorfamous.com' => 20.96, 'trapdull.com' => 20.96, 'solvehigh.com' => 21.57, 'tracerope.com' => 21.68, 'yesalumni.com' => 21.71, 'internetpromotional.net' => 23.16, 'domaingln11track.com' => 23.69, 'news-central-99.com' => 24.00, 'sports-bobble-heads.com' => 24.69, 'dans-fishing-adventure.com'=> 26.39, 'myhottdeals.net' => 26.65, 'nitroda.com' => 28.75, 'spaninns.com' => 29.08, 'pinaycamgirls.net' => 29.24, 'aspsbulletins.com' => 13.06, 'latestsbulletins.com' => 13.35, 'bulletinshops.com' => 13.40, 'academicsbulletins.com' => 13.49, 'tigersbulletins.com' => 14.13, 'nakedsbulletins.com' => 14.17, 'paintedsbulletins.com' => 14.33, 'opensbulletins.com' => 14.58, 'domainsbulletins.com' => 14.61, 'seniorssbulletins.com' => 14.72, 'dodgesbulletins.com' => 15.23, 'mindsbulletins.com' => 15.96, 'researchsbulletins.com' => 15.98, 'salessbulletins.com' => 16.12, 'netsbulletins.com' => 16.83, 'virtualsbulletins.com' => 17.14, 'petbulletins.com' => 17.30, 'citysbulletins.com' => 18.46, 'wearesurveys.net' => 23.36, 'takeoursurveys.net' => 24.50, 'thankyousurveys.net' => 24.62, 'greetingssurveys.info' => 24.63, 'allaboutsurveys.net' => 25.71, 'internetssurvey.info' => 25.98, 'theinternetsurveys.net' => 26.14, 'netsurveysnet.net' => 26.85, 'surveydept.info' => 23.39, 'survysample.info' => 25.82, 'gosurrvey.info' => 26.02, 'surrvytime.info' => 26.41, 'thedailyinfo.info' => 21.96, 'thenetinfo.info' => 23.27, 'nettinfo.info' => 27.41, 'theinfoguide.info' => 29.91, 'alloursamples.info' => 22.71, 'officeofferonline.info' => 24.77, 'cashsuggestion.info' => 26.84, 'insaneofferdeal.info' => 26.87, 'rewardcenterrs.info' => 30.04, 'gifttsgroup.info' => 31.48, 'ourgiftworld.info' => 32.46, 'giftwinninngs.info' => 34.11, 'mygiftwinninngs.net' => 31.12, 'gifttsgroup.net' => 33.05, 'giftwinninngs.net' => 34.95, }); - drop privileges sooner if possible - right after reading config file and before forking; - allow inserting X-Quarantine-ID header field into passed (and quarantined) mail for local recipients only; remote recipients should not be made aware that we may have a copy in a quarantine; reported by Robert Fitzpatrick; - check for multiple occurrences of RFC 2045 and RFC 2183 MIME header fields (in addition to checks on RFC 2822 header fields): MIME-Version Content-Type Content-Transfer-Encoding Content-ID Content-Description Content-Disposition and the RFC 3834 header field: Auto-Submitted - capture SpamAssassin logging and integrate it into the usual amavisd log; suggested by Jeff Moss; - when parsing SpamAssassin log areas/facilities, recognize negations (a name prefixed by 'no') and remove duplicates, last entry wins, e.g. amavisd -d rules,noall,norules,dcc,norules,rules debug-sa is equivalent to: amavisd -d noall,dcc,rules debug-sa - a SpamAssassin debug level 'info' is now implicitly prepend to a list of SpamAssassin facilities; it may be overridden by explicitly negating it, e.g. 'amavisd -d noinfo'; - when a command line argument 'debug-sa' is present, or if $sa_debug is true, a SpamAssassin debug level 'all' is prepend to a list of SpamAssassin facilities, which would bring SpamAssassin log level to 'dbg'; it may be overridden by negating it, e.g. 'amavisd -d noall,plugins,rules debug-sa'; - include milliseconds in a log timestamp when logging is directed to stderr; - create a custom hook object sooner, so that loading a policy bank from a custom hook becomes useful (but not soon enough to influence partition_tag); - added a custom hook after_send, which may suppress DSN, send reports, quarantine, etc; - fetch additional information (tag) 'TIMING' from SpamAssassin, making it available through macro 'supplementary_info' (if a version of SpamAssassin in use provides it - available since 3.3.0); - a SpamAssassin TIMING-SA report (timing breakdown by sections) is now collected and logged at log level 2 when available; the information is available since version 3.3.0 of SpamAssassin (currently in development and available through SVN); - add a global configuration variable $listen_queue_size (undef by default) which is passed as an option 'listen' to Net::Server, which in turn passes it on to listen(2) as a 'backlog' parameter. The Net::Server provides a default value of SOMAXCONN in the absence of a valid integer in $listen_queue_size (e.g. 128, but on Solaris it defaults to 5, which is too small for some purposes). Suggested by David Schweikert. A workaround for a small SOMAXCONN default on Solaris is provided by Net::Server 0.98 (?). - in the absence of an smtp client's IP address (normally received by XFORWARD smtp command from Postfix, or in the 'client_address' attribute of AM.PDP), parse the topmost one or two Received header fields and use the first valid IP address found there; based on a suggestion by Richard Bishop; - new macros (useful in notifications and $log_template): - 'week_iso8601' returns an ISO 8601 week number (between 1 and 53); - 'partition_tag' returns a current value of a $sql_partition_tag variable; - 'dkim' reports various DKIM verification results; - 'report_format' gives one of the: dsn, arf, attach, plain, resend - 'feedback_type' is expected to give one of the ARF (draft) strings: abuse, fraud, miscategorized, not-spam, opt-out, opt-out-list, virus, other - 'rfc2822_from' an e-mail address from a From header field; - 'rfc2822_sender' an e-mail address from a Sender header field, or empty; - 'tls_in' returns TLS ciphers in use by an SMTP session if mail came to amavisd through a TLS-encrypted session, otherwise empty - 'limit' takes two arguments: a string size limit and some string, returning the string from the second argument unchanged if its size is below the limit, or cropped to the size limit, with '[...]' appended; For details see README.customize. - new configuration setting $allow_fixing_long_header_lines, also member of policy banks, defaults to true - provides control over truncation of header section lines longer than 998 characters as limited by RFC 2822. The $allow_fixing_improper_header must also be true for fixing to take place. Previously it was only possible to turn off all header fixes, but not specifically just the long header truncation; - strip X-Spam-* headers and other prepended header fields when releasing a quarantined message; - turn on message part attributes 'C' (crypted) and 'U' (undecipherable) if a MIME Content-Type of that body part is <anything>/encrypted; (note: if $defang_undecipherable is turned on, this would push a received PGP/GPG-encrypted MIME top-part into an attachment, just as with other password-protected archives); - fetch additional information (tags) DCCB and DCCR from a SpamAssassin DCC plugin, making them available through a macro 'supplementary_info'; - amavisd-nanny and amavisd-agent now reopen a database if/when the underlying database file is re-created (i.e. when its inode changes), as is the case when amavisd restarts; based on a patch by Rob Foehl; - amavisd-nanny and amavisd-agent got a new command line option: -c count, which can restrict the number of iterations for display; usability deficiency pointed out by John Evans; - amavisd-nanny and amavisd-agent will use a value from an environment variable AMAVISD_DB_HOME (if it exists) as a database home directory, otherwise fall back to a built-in default '/var/amavis/db'; suggested by Leo Baltus; OTHER - added an AV entry for a new version of a ESET File Security (for Linux and FreeBSD) command line scanner (ESET Software ESETS Command Line Interface v 2.71.12); the nod32cli utility has been replaced by esets_cli; update provided by Willi Gruber; - keep time of a message reception (or creation) in rx_time as a floating point value as provided by Time::HiRes::time in order to avoid discrepancy in SNMP-like counters showing elapsed times; suggested by David Schweikert; - store additional attributes in per-message and per-recipient objects, reducing a need for repeated lookups; - to save time the: tag, tag2, tag3 and kill levels are not looked up when a recipient is bypassing spam checks; - no longer reverse-resolve an IP address obtained from a Received header field on infected mail: it does not work for IPv6, it can be stuck for prolonged periods waiting for a response from a non-responsive DNS server, and is only used for logging/notifications (macros %e and %o) on infected mail, so it is not too bad to drop it; - obey the always_bcc_by_ccat even when releasing from a quarantine; thanks to Tomas Horacek; - for consistency with other quarantine methods: store recipient address (e.g. a quarantine mailbox e-mail address) as a quarantine location even if a quarantine method is 'smtp:' or 'lmtp:'; - a quarantined message now receives one additional header field: an X-Envelope-To-Blocked. An X-Envelope-To still holds a complete list of envelope recipients, but the X-Envelope-To-Blocked only lists its subset (in the same order), where only those recipients are listed which did not receive a message (e.g. being blocked by virus/spam/ banning... rules). This facilitates a release of a multi-recipient message from a quarantine in case where some recipients had a message delivered (e.g. spam lovers) and some had it blocked; - a release request now takes its default recipients list from a header field X-Envelope-To-Blocked, no longer from X-Envelope-To. This avoids releasing a message to recipients which have already received it in the first place, e.g. spam lovers. For backward compatibility, if X-Envelope-To-Blocked header field is not found in a quarantined message, the recipients list defaults to X-Envelope-To as before. A release request can still provide its explicit list of recipients to override a default, like before. Loosely based on suggestions by Christer, by Paolo Schiro and others; - when mail is received from a helper program such as a milter, update a true mail size ($msginfo->msg_size) according to size definition in RFC 1870, adjusting for CRLF line terminators; - updated SMTP enhanced status response codes to an AUTH command according to RFC 4954 (SMTP Service Extension for Authentication); - quote_rfc2821_local: within qcontent all non-qtext characters should be represented as quoted-pairs (RFC 2822); in addition to " and \ (which were already handled), this includes NUL, CR, LF, HT, SP and 8-bit characters; - macros T, C and B now bring e-mail addresses in quoted form and in angle brackets, notification templates no longer add angle brackets around %T and %C; - when defanging mail body no longer insert our own Sender header field on a pretense that it helps with DKIM resigning - according to ADSP (ex SSP) the DKIM/ADSP does not care for the Sender header field (unlike a historical DomainKeys); - always provide X-Amavis-PolicyBank header field in a copy of a mail as submitted to SpamAssassin, even if a policy bank path is empty - this allows for simpler SpamAssassin rules to avoid being tricked by a presence of such header field inserted by third parties; - add missing installs of amavisd-nanny and amavisd-agent utilities to amavisd-new.spec; suggested by Eddy Beliveau; - internal: original header section is no longer kept in two copies (in orig_header and in orig_header_fields). Instead, orig_header now holds an array of header *fields* (previously: array of header *lines*), and orig_header_fields now stores a by- header-field-name hash of indices into orig_header (previously it held copies of header field bodies). To facilitate access to individual header fields and allow for top-down and bottom-up search for a n-th occurrence of a header field, two new access methods are provided: get_header_field and get_header_field_body; these are optimized for a quick access to the *last* header field (previously the *first* one was kept easily accessible); - internal: when specific header fields are looked up in an original header section (such as From, To, Subject, Message-ID, ...) now the *last* (the most to the bottom) occurrence of a header field (instead of the first) is used for better compatibility with DKIM which searches for header fields bottom-up; - internal: when a message is received, a current setting of a boolean attribute $originating (global or from current policy banks) is now copied to an Amavis::In::Message object and becomes a property of a message; this allows for other message objects (like notifications, quarantine) to have their own individual setting of this attribute, for example notifications are always flagged as originating and as such are eligible for DKIM signing; - internal: provide an Amavis::MIME::Body::OnOpenFh package which acts as a MIME::Body -type object (read-only) with an underlying representation in an existing (permanently open) temporary mail file, avoiding a need to open it by file name on a separate file handle. It is useful (simpler, faster) when defanging (pushing original mail to an attachment), or when generating notifications or reports which contain an original mail; - internal: new subroutines init_child and rundown_child in the interface module Amavis::SpamControl::SpamAssassin, which call SpamAssassin plugin methods "spamd_child_init" and "spamd_child_post_connection_close"; this is required for correct operation of some SpamAssassin plugins such as the Mail::SpamAssassin::Plugin::DBI; thanks to Michael Scheidell for pointing out that plugin; - internal: renamed a subroutine string_to_mime_entity into build_mime_entity and generalize its functionality to make it useful for generating reports; - internal: generalized subroutine delivery_status_notification which can now also prepare message in a format of a feedback report (ARF) or just as a plain included (inline) original mail; - internal: a subroutine passed to edit_header() may return undef, in which case a header field will be deleted instead of being replaced/edited; - internal: sub lookup_sql incompatible change to the order of arguments; - internal: a small speedup in receiving mail contents over SMTP or LMTP; - internal: abandon package Amavis::Lock, do the four calls to flock directly; - internal: move subroutines dealing with processes from module Amavis::Util into a new module Amavis::ProcControl; - internal: reduce buffer sizes in some copying loops from 64kB back to 32kB, larger buffers do not perform any better; - documentation: amavisd.conf-default: add a list of legacy dynamic configuration variables which can be used in policy banks; thanks to Gary V; - terminology: use a term 'header section' (in comments, log entries and templates) instead of a 'header' to go along with the 2822upd draft; A QUICK START TO DKIM VERIFICATION Starting with 2.6.0, verification of DKIM signatures (and historical DomainKeys signatures) is provided directly by amavisd (not only by a SpamAssassin plugin DKIM). A required version of a perl module Mail::DKIM is 0.31 or later. Signature verification is sufficiently fast so there is no need for concern about extra processing load. To turn on DKIM (and historical DomainKeys) signature verification, please add the following line to amavisd.conf (if not already there): $enable_dkim_verification = 1; Benefits - Whitelisting of banned checks or spam on messages carrying valid DKIM or DomainKeys signatures from trustworthy signers is possible through the @author_to_policy_bank_maps list of lookup tables. The mechanism uses loading of policy banks based on author's e-mail address (addresses in a 'From:' header field) and a signing domain, so a full flexibility of per-policy-bank settings is available. See description of a new configuration variable @author_to_policy_bank_maps earlier in this release notes. - To each message passed to local recipients (matching the @local_domains_maps) amavisd inserts a header field Authentication-Results (according to draft-kucherawy-sender-auth-header, now RFC 5451) for each signature found in a message, reporting a corresponding verification result. These header fields can reliably tell a recipient or his MUA what domains claimed responsibility for a message, or can be used for troubleshooting DKIM signing, verification and tracking of mail transformations. - Can adjust spam score based on some metrics on a signing domain's reputation for valid signatures found in a message. A useful reputation metric is an average long term spam score for past messages signed by a domain, which can currently be provided manually by @signer_reputation_maps in a configuration file (see example earlier in this release notes). A spam score is shifted towards this reputation score by a configurable factor $reputation_factor (value between 0 and 1, default is 0.2) using a formula: adjusted_spam_score = f * reputation + (1-f) * spam_score Semantics of a $reputation_factor is equivalent to auto_whitelist_factor in a SpamAssassin's AWL plugin, which shifts spam score towards a long term spam score average of a sender. - Notifications and bounces show a "(dkim:AUTHOR)" next to a From address, and a "(dkim:SENDER)" next to a Sender address if these header fields were signed and their domain corresponds to a signer's domain identity. - A valid DKIM or DomainKeys signature turns on a 'sender_credible' attribute which serves to choose one of the two DSN cutoff levels, so that delivery status notifications can be restricted to or preferred for likely-to-be-valid sending addresses, and bounces to possibly fake addresses can be minimized. More information on the 'sender_credible' attribute can be found earlier in this release notes. Currently the ADSP (Author Domain Signing Practices, formerly SSP) draft is not implemented, neither by amavisd, nor by SpamAssassin's plugin DKIM. Work on a draft is still in progress, and until it settles and comes into wider use one needs to resort to SpamAssassin rules to block fake mail with no valid signature from domains which are known to be signing all their mail, such as PayPal, eBay, alert.bankofamerica.com, and others. In essence, the ADSP information (usually inferred, or actually published (quite rare today)) from such domains needs to be encoded into SpamAssassin rules. (A note from the future: ADSP is now RFC 5617, SpamAssassin's DKIM plugin does implement ADSP starting with version 3.3.0.) A QUICK START TO DKIM SIGNING 1. Generate one or more keys to be used for signing and enable signing code by adding the following line to amavisd.conf (if not already there): $enable_dkim_signing = 1; # loads DKIM signing code Signing keys must be made available to amavisd, each private key in a separate file in PEM format. Customarily such keys would be generated and kept in a dedicated directory such as /var/db/dkim or /var/lib/dkim, preferably owned by root. Private keys can be generated by a 'openssl genrsa' command (see RFC 4871 Appendix C), or by an amavisd equivalent. Commonly one key per signing domain or one key per signing host is used, but other choices are possible. If such keys were already prepared for some other DKIM-signing solution, they can be reused by amavisd. RFC 6376 warns against using RSA keys shorter than 1024 bits and some recipients may choose to ignore short keys. # amavisd genrsa /var/db/dkim/a.key.pem # amavisd genrsa /var/db/dkim/b.key.pem 1536 # amavisd genrsa /var/db/dkim/sel-example-com.key.pem # amavisd genrsa /var/db/dkim/g-guest-ex-com.key.pem # amavisd genrsa /var/db/dkim/notif-mail.key.pem 768 Amavisd already ensures the generated files are only readable by owner, but a manual procedure may require explicitly setting file permissions. Private keys must be protected from unauthorized access, only the signing software such as amavisd should have read access. On a fresh program start the amavisd daemon loads these files before dropping privileges, so if amavisd is started as root these key files are readable regardless of their file access permission. Note however that starting with version 2.7.0 amavisd is also capable of a warm reload, where the daemon restarts with existing UID, so it does not have elevated privileges to read files with private keys. Therefore it is recommended that the file is owned by a group under which amavisd runs and UID root, and the file access permission allows read access for a group, e.g.: -rw-r----- 1 root vscan ... /var/db/dkim/xxx.key.pem 2. Add commands to amavisd.conf to load private keys, associate them with signing domains and selectors, and describe constraints (tags) to be published with public keys. For example: # Load all available private keys and supply their public key RR constraints. # Arguments are a domain, a selector, a key (a file name of a private key in # PEM format), followed by optional attributes/constraints (tags, represented # here as Perl hash key/value pairs) which are allowed by RFC 4871 in a public # key resource record (v, g, h, k, n, s, t), of which only g, h, k, s and t # are considered to be constraints limiting the choice of a signing key. # A command 'amavisd showkeys' can be used for displaying corresponding # public keys in a format directly suitable for inclusion into DNS zone files. # # signing domain selector private key options # ------------- -------- ---------------------- ---------- dkim_key('example.org', 'abc', '/var/db/dkim/a.key.pem'); dkim_key('example.org', 'yyy', '/var/db/dkim/b.key.pem', t=>'s'); dkim_key('example.org', 'zzz', '/var/db/dkim/b.key.pem', h=>'sha256'); dkim_key('example.com', 'sel-2008', '/var/db/dkim/sel-example-com.key.pem', t=>'s:y', g=>'*', k=>'rsa', h=>'sha256:sha1', s=>'email', n=>'testing; 1, 2'); dkim_key('guest.example.com', 'g', '/var/db/dkim/g-guest-ex-com.key.pem'); dkim_key('mail.example.com', 'notif', '/var/db/dkim/notif-mail.key.pem'); A selector paired with a domain name uniquely identifies a key, both for a signer as well as for a recipient. There may be multiple keys for each domain as long as each one has its own selector. A selector along with a domain name will be used by a receiving mailer in assembling a DNS query (selector._domainkey.signingdomain) to fetch a public key from a signing domain's DNS server when verifying signature validity. A selector paired with a domain name will also be used by a signing amavisd when choosing a key applicable to signing, meeting constraints on its public key (tags, RFC 4871 section 3.6) as given by optional arguments. Optional arguments serve as site documentation, may help amavisd choose between multiple choices (ruling out keys with incompatible tags), and supply additional information for step 3. For a list of options (tags) see RFC 4871 section 3.6. Amavisd does not check the syntax of tag values, except for performing qp-section encoding of a tag 'n'. Note the Perl syntax of key/value pairs, e.g. t => 's:y' will end up as "t=s:y", and n => 'testing; 1, 2' will end up encoded as "n=testing=3B 1, 2". 3. Prepare and publish public keys. Public keys can be extracted from generated key files (which contain both a private and a public key). To publish public keys they need to be edited into a format suitable for inclusion in a DNS server's zone file for each signing domain, either by following a procedure in RFC 4871 Appendix C, or if step 2 was completed, by asking amavisd to do so: # amavisd showkeys or more selectively, e.g.: # amavisd showkeys .org example.com This step is not needed if public keys were already prepared and published earlier for some other DKIM-signing solution. 4. Edit zone files in master DNS server(s) for each signing domain, adding the just prepared TXT resource records, not forgetting to bump up the serial number in a SOA record. Optionally add a TXT record with ADSP information (formerly SSP) if a default Author Domain Signing Practices is not appropriate. Then reload zone(s) or restart DNS server(s). 5. Test published public keys. Similar to 'showkeys', a 'testkeys' command walks through available signing keys (as declared by calls to dkim_key), generates test messages each signed with one key, and validates them by fetching a corresponding public key from a DNS server. # amavisd testkeys or more selectively, e.g.: # amavisd testkeys .org example.com (btw, if testkeys fails and you believe your DNS is correctly serving your DKIM public keys, you may need to upgrade Perl module Mail-DKIM to version 0.33 or at least 0.32_6) 6. Restart amavisd, watch the log at log level 2, searching for " dkim: ". Note that signing could be started (amavisd reload) right after completing step 2, but mail recipients would not be able to verify validity of signatures until public keys are made available by a signing domain through its DNS. Recipients are supposed to treat mail with signatures which fail verification exactly the same as mail with no signatures, so there is usually no harm done with a premature start of signing, but there is no benefit either. 7. Optional: to override default values for signature tags, one may specify by-sender signature tags through @dkim_signature_options_bysender_maps, e.g.: # @dkim_signature_options_bysender_maps maps author/sender addresses or # domains to signature tags/requirements; possible signature tags according # to RFC 4871 are: (v), a, (b), (bh), c, d, (h), i, l, q, s, (t), x, z; # of which the following are determined implicitly: v, b, bh, h, t # (tag h is controlled by %signed_header_fields); currently ignored tags # are l and z; instead of an absolute expiration time (tag x) one may use # a pseudo tag 'ttl' to specify a relative expiration time in seconds, which # is converted to an absolute expiration time prior to signing: x = t + ttl; # a built-in default is provided for each tag if no better match is found # @dkim_signature_options_bysender_maps = ( { 'postmaster@mail.example.com' => { a => 'rsa-sha1', ttl => 7*24*3600 }, 'spam-reporter@example.com' => { a => 'rsa-sha1', ttl => 7*24*3600 }, 'mail.example.com' => { a => 'rsa-sha1', ttl => 10*24*3600 }, # explicit 'd' forces a third-party signature on foreign (hosted) domains 'guest.example' => { d => 'guest.example.com' }, '.example.com' => { d => 'example.com' }, # catchall defaults '.' => { a => 'rsa-sha256', c => 'relaxed/simple', ttl => 30*24*3600 }, # 'd' defaults to a domain of an author/sender address, # 's' defaults to whatever selector is offered by a matching key } ); The result of a by-sender lookup into @dkim_signature_options_bysender_maps is a hash (a set) of DKIM signing requirements (tags), i.e. canonicalization method, hashing algorithm, domain, identity, selector and expiration time. All matching entries can participate in the result: for each tag individually the first setting (the most specific) is chosen from all matching entries. Resulting tags are then used to choose the most appropriate signing key from a set of keys as declared by calls to dkim_key. Main selection criterium is a match on tags d (domain) and s (selector), but other signature requirements must also meet the constraints of a public key (e.g. subdomain matching flag, granularity, hashing algorithm, key type). If a lookup does not find a signing key which meets requirements, no signing takes place. Also, only mail with 'originating' flag is eligible for signing. A lookup is based on either the From header field, the Sender header field, or on a mail_from address from the envelope, whichever yields a useful result first. Note that neither the Sender header field nor a mail_from address has any special meaning in the standard (RFC 4871). This results either in an author domain signature (i.e. a first-party signature, when based on a From header field), or in a third-party signature (when signing domain does not match the From, regardless of whether it is based on Sender header field or a mail_from or forced through d tag). An associative array %signed_header_fields controls which header fields are to be signed. By default it contains a standard (RFC 4871) set of header field names, augmented by some additional header field names considered appropriate at the time of a release (RFC 4021, RFC 3834). In addition a 'Sender' header field is excluded because it is frequently replaced by a mailing list, and as the RFC 2821 mandates there can only be one such header field the original one is dropped, invalidating a signature. Also the 'To' and 'Cc' are excluded from a default set because sendmail mailers are known to gratuitously reformat the list, invalidating a signature. A value in key/value pairs of %signed_header_fields is currently interpreted as a boolean, but stick to values 0 and 1 for now to allow for future enhancements. The default set of header fields to be signed can be controlled by setting %signed_header_fields elements to true (to sign) or to false (not to sign). Keys must be in lowercase, e.g.: $signed_header_fields{'received'} = 0; # turn off signing of Received $signed_header_fields{'sender'} = 1; # turn on signing of Sender $signed_header_fields{'to'} = 1; # turn on signing of To $signed_header_fields{'cc'} = 1; # turn on signing of Cc --------------------------------------------------------------------------- March 12, 2008 amavisd-new-2.5.4 release notes BUG FIXES - simplify regular expressions in parse_quoted_rfc2821() to avoid perl crashing on a long degenerated e-mail address; reported by Sébastien Aveline; - further simplify (split in two) regular expressions in parse_address_list() to avoid perl crashing on long degenerated e-mail addresses in From, To, and Cc header fields, also reported and sample provided by Tomi Lukkarinen; - incorrect parsing of header fields could let a header field to be ignored when preparing notification templates, or when adding a spam tag to a Subject header field, causing a second Subject header field to be inserted; reported by Mike Cisar; - untaint a policy bank name when it comes from an AM.PDP protocol request; symptom was a failure to insert a pen pals SQL record in a milter setup; reported by Peter Huetmannsberger; - smtp client code inappropriately concluded there is no progress being made when forwarding a message back to MTA, and exited a rw_loop when sysread returned status EAGAIN despite a 'select' setting an input-ready flag; the problem was detected on Solaris, although it could be more general; thanks to Aleksandr for a detailed problem report; - limit the number of filenames given as arguments to a file(1) utility to stay within a safe program argument space limit, run file(1) multiple times if necessary; - change the sprintf format for conversion of 64-bit SNMP-like counter values into a string (replaced %020d by %020.0f) to properly convert large values (beyond 32 bits) into strings on versions of Perl which are not compiled with support for 64-bit integers (Solaris?); reported by David Schweikert; OTHER - invoke unrar and rar without option 'av-', which is no longer valid since version of unrar 3.7.5, and was previously apparently just ignored; the rar documentation states that option 'av' is only available with registered versions, so it appears the 'av-' is redundant with rar as well. Unrar aborting with 'unknown option' error was reported by Piotr Meyer; - a new AV entry for fpscan 6.x, (F-PROT Antivirus for UNIX command-line scanner) based on information from Erik Slooff, Bruno Friedmann, and Steve); - a new AV entry for fpscand 6.x (F-PROT Antivirus for Linux/BSD/Solaris), based on initial research by Alexander Wirt, Henrik K, and F-Prot documentation, thanks also to Haukur Valgeirsson of F-Prot International; - ask_daemon_internal now keeps persistent connection to an F-PROT fpscand daemon, in addition to Sophie and Trophie daemons; - new AV entry 'bdscan' for a new version of BitDefender Antivirus Scanner for Linux and FreeBSD; thanks to Gary V; - updated README.postfix, thanks to Chris Pepper and Patrick Ben Koetter; - updated amavisd-new-docs.html; --------------------------------------------------------------------------- December 12, 2007 amavisd-new-2.5.3 release notes BUG FIXES - fix parsing an SMTP status response from MTA when releasing from a quarantine, when an MTA response did not include an enhanced status code (RFC 3463) (such as with old versions of Postfix); a parsing failure resulted in attribute "setreply=450 4.5.0 Unexpected:..." in an AM.PDP protocol response, even though a release was successful; reported by Ron Miller, John M. Kupski, investigated by Tony Caduto and Jeremy Fowler; - change parsing of addresses in From, To, and Cc header fields, avoiding complex Perl regular expressions which could crash a process on certain degenerate cases of these header fields; thanks for detailed problem reports to Carsten Lührs and Attila Nagy; - completely rewritten parsing of Received header field to work around a Perl regular expression problem which could crash a process on certain degenerate cases of mail header fields; problem reported by Thomas Gelf; - harden to some extent regular expressions in parse_message_id to cope better with degenerate cases of header fields carrying message-id; - sanitize 8-bit characters in In-Reply-To and References header fields before using them in Pen Pals SQL lookups to avoid UTF-8 errors like: penpals_check FAILED: sql exec: err=7, 22021, DBD::Pg::st execute failed: ERROR: invalid byte sequence for encoding "UTF8": 0xd864 - when turning an infection report into a spam report, avoid adding newly discovered virus names (i.e. fraud names) to a cached list if these names are already listed; previously the list would just grow on each passage through a cache, leading to unsightly long lists of spam tests in a report; based on a patch by Henrik Krohns; - fix diagnostics when an invalid command line argument is given; OTHER - reduce log clutter when certain Perl modules are loaded late, i.e. after chrooting and daemonizing, but still before a fork; now only issue one log entry by a parent process: "extra modules loaded after daemonizing: "; - slightly relax mail address syntax in subroutine split_address; - fetch additional information (tags) from SpamAssassin: TESTS, ASN, ASNCIDR, DKIMDOMAIN and DKIMIDENTITY, making them available through a macro 'supplementary_info' (if a version of SpamAssassin in use provides them); - updated DKIM section in amavisd-new-docs.html, removing the historical DomainKeys milter from examples; - declared a dummy subroutine dkim_key() and new dummy configuration variables @dkim_signature_options_bysender_maps, %signed_header_fields, $reputation_factor, @signer_reputation_maps and $sql_partition_tag, members of policy banks, in preparation for 2.6.0 - declared now for improved downgrade compatibility of 2.6.0 configuration files, if need arises. --------------------------------------------------------------------------- June 27, 2007 amavisd-new-2.5.2 release notes BUG FIXES - in a milter setup log_id was left undefined, which resulted in log lines without id, and an SQL constraint violation "Column 'am_id' cannot be null" when logging to SQL was enabled. The bug was introduced in 2.5.1; problem reported by Martin Svensson; - suppress a second quarantining attempt if the message also needs to be archived to the same location (same sql key or same local filename); reported by Wazir Shpoon; - adjust $socketname in amavisd-release to match its default counterpart in amavisd (i.e. /var/amavis/amavisd.sock); reported by Stanley Appel; NEW FEATURES - add snmp-like counters for PenPalsSavedFromKill, PenPalsSavedFromTag3 and PenPalsSavedFromTag2, which correspond to the number of messages since a program (re)start in which spam level would have exceeded a corresponding level had there not been for (negative) score points contributed by pen pals lookups. Note that for any message only one of the three counters could increment, the one corresponding to the highest level crossed. To find more information about rescued mail messages, search the log for a string 'PenPalsSavedFrom' (available at log level 2 or higher). Practical value: mail saved by pen pals from being blocked often indicate false positives by SpamAssassin; examining rules which contributed significantly to the score may indicate which rules need adjustment; - when preparing an SQL SELECT clause in lookup_sql, provide an additional placeholder %a in a clause template, which is much like the existing %k, but evaluates to an exact mail address (i.e. the same as the first entry in the %k list), which makes it suitable for SQL pattern matching; suggested by Daniel Duerr; - macro supplementary_info can supply information on two additional SpamAssassin tags: AUTOLEARNSCORE and LANGUAGES if corresponding plugins are enabled in SpamAssassin; see README.customize for the complete list; - provide two new subroutines available for calling from config files: include_config_files() and include_optional_config_files(), each take a list of filenames as arguments, and reads & evaluates them just like normal configuration files specified on a command line (option -c or a default amavisd.conf). This provides a simplified and uniform mechanism for 'including' additional configuration files, which formerly could be invoked through a perl do() function. The only difference between include_config_files and include_optional_config_files is that the former aborts if some specified file does not exist, while the later silently ignores specified but missing files. Both/each subroutine may be called multiple times, recursion is allowed (but some sanity limit to recursion is provided); based on a suggestion by Gary V. Example line in amavisd.conf: include_config_files('/etc/amavisd-custom.conf'); OTHER - provide a workaround for a crashing altermime by removing its leftover temporary file which would otherwise cause a temporary failure: TempDir::check: Unexpected file problem reported by Dennis A. Kanevsky; - add a mapping to 'doc' for a result 'Microsoft Installer' from a file(1) utility; it seems like versions 4.20 and 4.21 of file(1) (possibly earlier versions too) misclassify MS Word, Excel, and PowerPoint documents as 'Microsoft Installer'; problem investigated and a workaround suggested by Noel Jones, Mike Cappella and Michael Scheidell; - add a mapping to 'asc' for a result 'COM executable for DOS' from a file(1) utility; it seems like later versions of file(1) can misclassify a text in a GB2312 character set as a COM file; reported by Daniel J McDonald; - updated AV entry for ESET NOD32 Linux Mail Server again - command line interface (nod32cli): added a status 3 (e.g. corrupted archive) back to the list of clean statuses; the 3 was removed in 2.5.1 as the entry was substituted with the one from a NOD32 documentation; reported by Tamás Gregorics; - updated AV entry for 'F-Secure Antivirus for Linux servers' to cope with version 5.5 and later; a new entry provided by Peter Bieringer; - when a command line option -g requests changing of group ID, do so by calling POSIX::setgid, after also attempting to assign to perl variables $( and $), which may not work correctly on systems where group ID can be negative (like group 'nobody' being -2 on Mac OS X); follows a SpamAssassin problem report 3994, investigated by Sidney Markowitz; - when an AUTH command parameter (RFC 2554, now RFC 4954) is supplied on a MAIL FROM SMTP command but AUTH support has not been previously offered (like when authentication is disabled by an empty @auth_mech_avail), no longer treat the situation as a fatal error: 503 5.7.4 Error: authentication disabled but mercifully ignore the parameter and just log an informational message. This is a deviation from RFC 2554, but makes it friendlier for those insisting on running amavisd as a Postfix pre-queue smtp proxy; suggested by Alexander 'Leo' Bergolth; - adjust the list of pre-loaded perl modules required by SpamAssassin; - internal: pass a mail message to SpamAssassin as a GLOB instead of an array reference, saving one in-memory copy of a message during a SA call; - internal: make it slightly easier to switch message digest from MD5 to a Digest::SHA family by turning a hard-wired key length into a parameter (admittedly it is still ugly, requiring a change in three places for switching); also pave a transition from Digest::SHA1 to Digest::SHA; - documentation: updated files README.postfix and README.postfix.html now include a section 'Advanced Postfix and amavisd-new configuration' explaining a multiple cleanup service architecture; thanks to Patrick Ben Koetter; retired file: README.postfix.old - documentation: updated README.sql-pg to include a faster alternative to purging an SQL logging database: the alternative 'DELETE FROM maddr' on PostgreSQL runs faster by a factor of 1.5 to 2 from the one previously suggested; - suggestion: when using SpamAssassin plugin Rule2XSBody (available in more recent versions of SA), adding an entry like: Mail::SpamAssassin::CompiledRegexps::body_0 to the @additional_perl_modules list allows preloading of compiled rules. Adding the following two lines to amavisd.conf adds the directory name containing modules with compiled rules to Perl modules search path and allows Perl to find the listed module(s): my($sa_instdir) = '/var/lib/spamassassin/compiled/3.002001'; unshift(@INC, $sa_instdir, $sa_instdir.'/auto'); --------------------------------------------------------------------------- May 31, 2007 amavisd-new-2.5.1 release notes COMPATIBILITY WITH 2.5.0 - setting $bypass_decode_parts to true now also disables MIME decoding (see below); SECURITY - provides checking the number of archive members against $MAXFILES quota even when just listing an archive directory, providing some additional protection (besides a time limit) against runaway dearchivers (such as a recent Zoo archiver DoS); - please use the most recent versions of file(1) utility (currently 4.21) and recent versions of external dearchivers/decoders to avoid known security vulnerabilities in them; NEW FEATURES - introduced a variation of a message release from a quarantine, allowing a releaser to choose between forwarding a message to the back-end MTA port as usual (avoiding re-checking of a message), or to send it to MTA on its incoming port (normally 25) and let the message be rescanned, which might be useful after adjusting spam rules or antivirus database. It is implemented by: * adding a configuration variable $requeue_method (also a member of policy banks), with a default value: 'smtp:[127.0.0.1]:25' * extending the AM.PDP protocol with a 'request=requeue' attribute which can be used in place of a 'request=release', * enhancing the 'amavisd-release' utility program to choose between sending 'request=release' and 'request=requeue' based on its program name, i.e. by making a soft or hard link to amavisd-release (or its copy) named 'amavisd-requeue', the utility will send a 'request=requeue' in place of the usual 'request=release', e.g.: # ln -s amavisd-release amavisd-requeue $ amavisd-requeue spam/k/kg2P0rP9Lpu3.gz * enhancing amavisd daemon to choose between forwarding a released message either to $release_method or to $requeue_method destination based on a 'request' attribute value in an AM.PDP request; - new AV entry: ArcaVir for Linux and Unix, see below for links; - a new macro 'supplementary_info' gives access to some additional information provided by content scanners, such as a provided by SpamAssassin API routine get_tag. The macro takes two arguments, the first is a tag name (a name of some attribute which is expected to provide an associated value), the second argument is a sprintf format string and is optional, if missing a %s is assumed. Currently the only available attributes are AUTOLEARN, SC, SCRULE, SCTYPE, and RELAYCOUNTRY. These are nonempty only when an associated SpamAssassin plugin or function is enabled. BUG FIXES - fixed quarantining to an SQL database of messages with a null envelope sender address (broken in 2.5.0, causing such messages to tempfail); reported by Markus Edholm, Vahur Jõesalu and Michael Scheidell; - fixed parsing of certain broken 'From' header fields, which would result in a temporary failure and the following logged error: check_init2 FAILED: parse_address_list PANIC1 53 at /usr/local/sbin/amavisd line 3292 reported by Michael Scheidell; - avoid encoding nonprintable characters in X-Envelope-From and X-Envelope-To header fields in a quarantined message even if envelope mail addresses contain such invalid characters, so that a quarantine release is possible; (RFC 2047 allows encoding of a 'phrase' in From, To, and similar headers, as well as in comments, but not in the address specification); - avoid unnecessarily RFC 2047 -encoding of 8-bit characters in those lines of inserted X-Spam-Report (and similar) multiline header fields which only contain ASCII characters; also avoid encoding of newlines; reported by Anant Nitya; - sanitize 8-bit characters in SpamAssassin report before inserting it into an X-Spam-Report header field; - properly recognize PostgreSQL error code 'S8006' and reconnect to a disconnected server right away; thanks to Brian Wong; - call $mail_obj->finish after a SA call to allow for garbage collection and removal of SA temporary files; see: http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5444 - avoid nonstandard SMTP status code 254 on discarded malware; on discarding turn status 554 into a 250 instead; violation of a SHOULD in RFC 2822 pointed out by Alexander 'Leo' Bergolth; - an informational log message was reported inappropriately: INFO: truncated ... header line(s) longer than 998 characters it didn't reflect reality, it was always reported together with the: INFO: unfolded 1 illegal all-whitespace continuation lines - when an SMTP option BODY=8BITMIME (RFC 1652) is not given on mail reception, avoid turning it on while forwarding, even if mail body contains 8-bit characters; following a garbage-in-garbage-out principle, this doesn't break anything that isn't already broken, but might prevent later conversion to 7-bit quoted-printable MIME by some downstream MTA, invalidating signatures (DKIM, S/MIME, PGP, ...) - at a risk that some overzealous firewall might block a mail transfer; - fixed a couple of documentation typos/bugs in README.customize, thanks to Mike Cappella; OTHER - modified code for checking each eval {} status: it turns out that eval is able to capture certain error conditions (e.g. certain I/O errors) but without setting the $@ variable, leaving it empty; use new idiom throughout for proper error handling and more informative reporting, showing errno in such cases; P.S.: actually what happens is the $@ gets cleared by some DESTROY method which uses eval, so the outer eval does notice an exception but is unable to capture en error message; - setting $bypass_decode_parts to true now also disables MIME decoding, not just decoders/dearchivers listed in a @decoders list, and also implicitly retains full original message for virus checking, equivalent to having a regular expression /^MAIL$/ in a @keep_decoded_original_maps list; prompted by Bill Landry; - new AV entry: ArcaVir for Linux and Unix, see: http://www.arcabit.pl/ http://www.arcabit.com/download_product.html?product=ArcaVirLinux2007 http://www.arcabit.com/products_arcavir_for_unix_2006.html the entry was kindly provided by Michal Seremak; - updated AV entry for ESET NOD32 Linux Mail Server - command line interface (nod32cli), version 2.7, thanks to Simon; - updated AV entry for Sophos sweep, adding options -mime and -oe ; - avoid repeatedly reporting the same set of modules by a log entry 'extra modules loaded:', only report it on changes to the list; repeated reports could be misinterpreted that modules were loaded with each mail task, where actually missing modules were only loaded once within each child process; - avoid reporting 'BOUNCE' in an SMTP response text when a bounce (i.e. a nondelivery status notification) was actually suppressed, such as is usually the case with infected mail or when spam score exceeds spam_dsn_cutoff_level. Previously the SMTP response text only reflected the setting of a final_*_destiny, which could mislead mail administrators into believing that excessive unconditional backscatter was being generated. The new text looks like: 250 2.5.0 Ok, id=67685-15, DISCARD(bounce.suppressed) instead of previous: 250 2.5.0 Ok, id=67685-15, BOUNCE A general note worth reiterating: to reduce backscatter pollution (sending of bounces to innocent sender addresses), please either: * set $final_virus_destiny and $final_spam_destiny to D_DISCARD or to D_PASS (_not_ to D_REJECT or D_BOUNCE), or: * carefully configure virus and spam bounce suppression by: . configuring @viruses_that_fake_sender_maps correctly (the default is fine, it suppresses all bounces to infected mail), this way one may safely set $final_virus_destiny to D_BOUNCE, it is equivalent to D_DISCARD for all infected mail containing malware matching the @viruses_that_fake_sender_maps; . and: configuring @spam_dsn_cutoff_level_maps and @spam_dsn_cutoff_level_bysender_maps, keeping levels just slightly over a kill level, have a well maintained SpamAssassin with network tests enabled and updated rules - then one may set $final_spam_destiny to D_BOUNCE, which will produce bounces for mail with spam score between kill level and cutoff level, and suppress bounces above a suppress level; some domains may still consider such practice abusive, so do not take this choice lightly; . to monitor bounces generated by amavisd, one may assign some dedicated monitoring e-mail address to $dsn_bcc, which will then receive a copy of all delivery status notifications sent out by amavisd; - dspam options changed with version 3.8.0, replacing option --feature with --tokenizer; reported by Jim Knuth; - modified syslog writing to check errno after calling Unix::Syslog::syslog, and to informatively attempt to log status when unsuccessful; an unsuccessful status is just informational, as syslog(3) routine does its own retries and leaves an unsuccessful status of a previous attempt in errno even if a subsequent logging attempt did succeed; unfortunately the system routine syslog(3) returns no value according to documentation (and according to its source code), so its completion status can not be tested; a problem of a loss of logging on a syslogd restart on OS X was reported by Paul Walker, but unfortunately can not be solved on the application side; - uncomment some debugging printouts in p0f-analyzer.pl and land them under control of a $debug variable; --------------------------------------------------------------------------- April 23, 2007 amavisd-new-2.5.0 release notes COMPATIBILITY WITH 2.4.5 The 2.5.0 is upwards compatible with 2.4.* versions, except for the following: Default notification and logging templates are enhanced to take advantage of new macros and new concepts, so it is prudent to update templates if defaults are overridden, e.g. $log_templ, $notify_*_admin_templ, ... A client-side AUTH (rfc2554: SMTP Service Extension for Authentication) is currently not available. The last version with this feature working is amavisd-new-2.5.0-pre4. The feature may be restored in a future version if sufficient interest is demonstrated. A workaround for a qmail bug (which complains when CR and LF are split across a TCP segment boundary) is no longer available, as the program has no control over IP packet splitting done by the TCP/IP stack. When pen pals feature is in use, it is worth creating an index on a msgs.message_id field. NEW FEATURES AT A GLANCE - new concept: blocking contents category; - true per-recipient defanging/sanitation of a mail body (previously a true per-recipient handling was available for mail header edits, but not for mail body modifications); - added interface code to invoke Anomy Sanitizer or the 'altermime' program allows defanging or adding disclaimers by external utilities on a per-recipient basis; - rewritten SMTP client code: get rid of the troublesome module Net::SMTP; new code now supports pipelining, client-side LMTP, IPv6, Unix sockets, more reliable error detection and handling, passes on ENVID parameter unchanged, is bare-CR-clean, tidier code (no more workarounds for rough corners in Net::SMTP), fewer context switches (handshake handovers) due to pipelining if pipelining is offered by MTA (which usually is); - makes available pedantically parsed addresses from a mail header: From, Sender, To, Cc. Addresses from mail header may be needed for deciding on inserting disclaimers, signing mail (DKIM), custom hooks (like 'vacation'-type applications), and other future applications. Get rid of inexact parsing by module Mail::Address, provide own parser; - phishing fraud as returned by ClamAV is now treated as spam, no longer as a virus; - compatible with SpamAssassin 3.2.0; - enhancements to amavisd-nanny: shows more detailed states of processes; - enhancements to amavisd-agent: shows average processing times per message; - extended AM.PDP protocol with an attribute 'policy_bank' which may be used in a client's request to require loading additional policy banks; - add support for 7-Zip archives if external utility 7z is available; - custom hooks allow custom code to be called at few strategic places; - the pen pals feature can now also match replies which reference previous outgoing mail by its Message-Id (taking into account References or In-Reply-To header field); - new key 'originating' in policy banks generalizes a MYNETS policy bank; - a documentation rewrite for setting up amavisd-new with Postfix by Patrick Ben Koetter (one of the two authors of The Book of Postfix). Previous documentation has been renamed to README.postfix.old and will be removed in the next version; the new documentation is README.postfix.html, and its automatically converted plain text version is README.postfix. A big thanks to Patrick for his efforts! BUG FIXES - if a sender is both white- and black-listed at the same time, then inserted X-Spam-* header fields were inconsistent, e.g. X-Spam-Level, X-Spam-Flag and X-Spam-Status reflected a whitelisted status (no asterisks, not a spam), while X-Spam-Score showed 64 points; now whitelisting prevails in all X-Spam-* header fields; - relax argument parsing in amavisd-release to allow releasing of quarantine id containing a body hash in a name (%b in template); reported by Ron Rademaker; - skip an SQL-logging database operation if an associated clause in %sql_clause is disabled, e.g. set to undef or ''; this allows for example to selectively disable SQL logging based on a policy bank; thanks to Riaan Kok; - let LHA decoder (do_lha) recognize also other listing formats, e.g. MS-DOS, symlinks, not just plain Unix archives; problem reported by Ryuhei Funatsu; OTHER - catch and log uncaught '__DIE__' and '__WARN__' Perl pseudo-signals which would otherwise go to stderr and not be noticed with a daemonized process; patch by Alexander 'Leo' Bergolth; - insert 'X-Spam-Flag: NO' if spam level is above tag level but below tag2 level, previously it was not inserted at all (it is still redundant, but some may appreciate an explicit statement nevertheless); - drop support for Archive::Tar; main drawback of this module is: it either loads an entire tar into memory (horrors!), or when using extract_archive() it does not relativize absolute paths (which makes it possible to store members in any directory writable by uid), and does not provide a way to capture contents of members with the same name. Use pax program instead! - abandon the use of libnet (modules Net::SMTP and Net::Cmd), replaced by own code to implement client-side SMTP and LMTP protocol support, with a full support for pipelining and IPv6. Some of my issues with Net::SMTP go back to year 2002, but the one that broke camel's back is a 3+ months status quo in not fixing a serious misfeature introduced in 1.20, which mangles 8-bit characters in mail, causing a series of support questions, and even affecting some larger service providers without them realizing there is a problem. Here are some relevant bug reports: http://rt.cpan.org/Public/Bug/Display.html?id=24835 http://rt.cpan.org/Public/Bug/Display.html?id=2608 http://rt.cpan.org/Public/Bug/Display.html?id=2607 http://rt.cpan.org/Public/Bug/Display.html?id=14875 http://rt.cpan.org/Public/Bug/Display.html?id=9394 P.S. libnet-1.21 eventually fixed the UTF8 encoding problem and added support for ENVID and AUTH options; other issues are still open (e.g. split code/text in smtp status, no pipelining support, no LMTP); - represent score in X-Spam-Status header field as a single numeric field instead of the previous explicit sum of a SA score and a score boost, which some MUAs don't know how to interpret and label mail spaminess incorrectly; - recipient notifications (in their default template) now keep the original To and Cc header fields, instead of placing a recipient envelope address into a To header field; suggested by Jorgen Lundman; - no longer removes an X-Amavis-Alert header field (as inserted by some foreign MTA), it does not hurt to keep it; - kavscanner AV entry (Kaspersky Antivirus): added a new entry to the search list of paths to a binary, kavscanner changed its default installation location again; provided by Gary V; - added a rule into a $map_full_type_to_short_type_re list, mapping a file(1) type 'RIFF...animated cursor' into '.ani'; a patch by Eric; - example rules (commented-out) to block animated cursors (ANI) and icons are added to configuration files amavisd.conf and amavisd.conf-sample; - suppress bounces not only for Precedence: (bulk|list|junk) or null return path (as before), but also for mail with a 2822.From header field matching one of: *-request | *-owner | *-relay | *-bounces | owner-* | postmaster | mailer-daemon | mailer | uucp, and for mail containing a header field List-Id (RFC 2919). Note that this new additional rule practically never triggers in practice, true bounces and common mailing list traffic is already covered by previous checks; - make available a list of detected virus names in an Amavis::In::Message object (virusnames), suggested by Tom Sommer; - split SQL documentation into three files: README.sql general SQL considerations and some examples README.sql-mysql MySQL-specific notes and schema README.sql-pg PostgreSQL-specific notes and schema (also SQLite) - README.sql-pg now adds CHECK (x >= 0) for fields that are supposed to contain unsigned integers; suggested by Hanne Moa; - repurpose/rename a contents category CC_TEMPFAIL to CC_MTA; it is turned on when MTA rejects or tempfails a mail when amavisd attempts to pass it back after checking it; (this should not normally happen, MTA should be doing most of its checks (e.g. recipient validation) before passing mail to a content filter, not after); NEW FEATURES - custom hooks allow custom code to be called at few strategic places: * during child process initialization: allows initialization of custom code, including establishing an additional SQL session or similar; * after built-in checks: allows custom code to inspect and/or modify results of checks; * before sending, allows for additional quarantining and for sending additional notifications; * at the end of message processing: allows inspecting results of checks and status of mail forwarding (e.g. for statistics or logging purposes). Mail processing sequence: child process initialization *custom hook: new() loop for each mail: receive mail mail checking and collecting results *custom hook: checks() - may inspect or modify checking results deciding mail fate (lookup on *_lovers, thresholds, ...) quarantining sending notifications (to admin and recip) *custom hook: before_send() - may send additional notifications, additional quarantining, may modify mail forwarding (unless blocked) sending delivery status notification (if needed) issue main log entry, manage statistics (timing, counters, nanny) *custom hook: mail_done() - may inspect results endloop after $max_requests or earlier If Amavis::Custom::new returns undef then no further custom calls are made. Otherwise, this method is supposed to return an object, thus enabling other custom hooks, which receive this value as their first argument. See amavisd-custom.conf for an example, and the example invocation of amavisd-custom.conf at the end of file amavisd.conf-sample; thanks to Kasscie (Yohanna Monsalvez); - formerly pen pals could only match replies to previous outgoing mail where envelope sender and recipient addresses are exactly reversed. Now, in addition to this, pen pals can also match replies which reference previous outgoing mail by its 'Message-ID' (taking into account the 'References' or 'In-Reply-To' header fields), even if the envelope sender address of the reply is null or does not match a recipient address of a previous outgoing mail. This covers for incoming replies to mailing list postings, incoming message disposition notifications (MDN, RFC 3798) and incoming replies from alias or role addresses. A query on a message-id is fast compared to matching on recipient id, and if it succeeds, the later one is skipped. Based on a suggestion and a patch by Alexander 'Leo' Bergolth; The %sql_clause now contains one additional SQL SELECT clause under 'sel_penpals_msgid' key, which is used in place of 'sel_penpals' when a list of references in a reply is nonempty. It is worth creating an index on a msgs.message_id field to speed up SQL lookups: CREATE INDEX msgs_idx_mess_id ON msgs (message_id); Note that a spammer can gain a (small) advantage by including a reference to a recent outgoing message in his message. With private correspondence this information is hard to come by efficiently in a timely manner, and can effectively be exploited only if a generated Message-IDs of outgoing mail is very easy to guess (e.g. generated by a very poor random number generator). With postings to public mailing lists this information is more readily available. Setting $sql_clause{'sel_penpals_msgid'} to undef or to empty disables matching on message-id, if it turns out the mechanism is being actively exploited. - added penpals snmp-like counters, which are displayed by amavisd-agent like the following: PenPalsAttempts 10222 438/h 18.3 % (InMsgsRecipsInboundOrInt) PenPalsHits 2957 127/h 48.9 % (ContentCleanMsgsInboundOrInt) PenPalsHitsMid 88 4/h 3.0 % (PenPalsHits) PenPalsHitsMidNullRPath 26 1/h 0.9 % (PenPalsHits) PenPalsHitsMidRid 917 39/h 31.0 % (PenPalsHits) PenPalsHitsRid 1926 82/h 65.1 % (PenPalsHits) Some comments on the above figures: - PenPalsAttempts shows that SQL lookups are performed only on a fraction of all incoming and internal mail (and not at all on outgoing mail) because a lookup is not performed on a high score spam (score above $penpals_threshold_high, which was 8 in this example, typically the same as kill level); - PenPalsHits shows that more than half of incoming clean messages can be associated with previous outgoing mail and can benefit from pen pals soft-whitelisting, such message is either a reply directly referencing a previous mail, or a new conversation between a sender/recipient pair which had previous conversation on a different topic; - PenPalsHitsMidRid shows that 1/3 of incoming matching replies match a previous outgoing mail by both the Message-ID as well as the exact recipient address; these are true replies to previous outgoing messages; - PenPalsHitsRid shows that 2/3 of incoming matching replies match a a previous outgoing mail by recipient address, but not by a Message-ID; these messages usually correspond to new topics among previous correspondents; - PenPalsHitsMid match only in reference to a previous Message-ID, but not by sender/recipient addresses; these are usually mailing list replies to a previous posting by a local user; - PenPalsHitsMidNullRPath are messages with null return path which are matching some previous outgoing message by Message-ID; these are usually incoming message disposition notifications (MDN, RFC 3798), or certain bounces which specify In-Reply-To or References in their header; - configuration variable %defang_by_ccat is renamed to %defang_maps_by_ccat and may now contain a list of by-recipient lookup tables (or a boolean as before for compatibility); this allows defanging/mangling to be selected on per-recipient basis; compatibility is retained by making the old variable %defang_by_ccat an alias for %defang_maps_by_ccat; - provided interface code to allow mangling/defanging/sanitation to be performed by an external utility, either by directly calling a Perl module Anomy Sanitizer (within the same process, avoiding startup cost), or by invoking a program 'altermime' (or by internal defanging code as before). Mail body defanging is only allowed for local recipients (those matching @local_domains_maps), i.e. for inbound and internal-to-internal mail. If there is more than one mangling code option available, the result of a %defang_maps_by_ccat can choose between them by returning one of the following strings, the selection can depend on mail content type and on by-recipient lookups if needed: 'anomy' chooses Anomy Sanitizer (if $enable_anomy_sanitizer is true); 'altermime' chooses a program whose path is $altermime (if found); 'attach' chooses the traditional amavisd-new defanging method which pushes an original mail message to an attachment; 'null' for testing purposes - doesn't modify mail body, but pretends it does (in logging and mail header); other non-empty and non-zero value automatically choose one of the above options depending on what is available; at least the 'attach' is always available; an empty, zero or undef value disables mail body modifications; Controls: $enable_anomy_sanitizer, @anomy_sanitizer_args, and: $altermime, @altermime_args_defang; Typical use: # with altermime: $altermime = '/usr/local/bin/altermime'; @altermime_args_defang = qw(--verbose --removeall); # with Anomy Sanitizer: $enable_anomy_sanitizer = 1; @anomy_sanitizer_args = qw( /usr/local/etc/sanitizer.cfg ); $defang_spam = 1; # old style, applies the first available mangler # to all spam-loving local recipients # unnecessarily complicated example of selective choices: $defang_maps_by_ccat{+CC_BANNED} = [ 'altermime', # use altermime for everybody (a 'constant' lookup table) ]; $defang_maps_by_ccat{+CC_SPAM} = [ { # a per-recipient hash lookup table 'user@example.com' => 1, # old style, auto-selects a mangler 'user-a@example.com' => 'anomy', 'user-m@example.com' => 'altermime', 'user-t@example.com' => 'attach', '.example.net' => 0, # no mangling }, $defang_spam, # fallback to old style setting if no match above ]; - a special case of mangling is adding a disclaimer, by invoking an external program 'altermime' (if available and enabled). This differs from mangling inbound mail in two details: * uses a separately configurable list of arguments to altermime: @altermime_args_disclaimer; and * it applies only to mail submitted from internal networks or roaming users (as recognized through a policy bank which sets: allow_disclaimers => 1), and where any of the following addresses matches local domains: author (2822.From) or sender (2822.Sender) or return path (2821.mail_from); Typically the $allow_disclaimers should be set by a policy bank which also sets the $originating flag. In addition to strings that may be returned by %defang_maps_by_ccat as described above, there are two more, only taken into account when $allow_disclaimers is true: 'disclaimer' invokes $altermime program for outgoing mail with arguments as given in @altermime_args_disclaimer; 'nulldisclaimer' for testing purposes - doesn't modify mail body, but pretends it does (in logging and mail header); Typical use: $altermime = '/usr/local/bin/altermime'; @altermime_args_disclaimer = qw(--verbose --disclaimer=/etc/altermime-disclaimer.txt); $defang_maps_by_ccat{+CC_CATCHALL} = [ 'disclaimer' ]; @mynetworks = qw( ... ); $policy_bank{'MYNETS'} = { # mail originating from our networks originating => 1, allow_disclaimers => 1, } For the moment there is one limitation: there can only be one mangler in effect at a time, it is not currently possible to both defang and to append a disclaimer on the same message: for internal-to-internal mail inserting a disclaimer takes precedence. To make it possible to provide different disclaimer texts when hosting multiple domains, there is an experimental additional configuration variable available: the @disclaimer_options_bysender_maps. It is a list of lookup tables, looked up by a sender address. The sender address is chosen from the following list, first match wins: * 'Sender:' header field, if its domain matches @local_domains_maps; * 'From:' header field, if its domain matches @local_domains_maps; * envelope sender address, if its domain matches @local_domains_maps; We already know that at least one of the above will match, otherwise adding disclaimers would be skipped at an earlier stage. The result of lookups should be one simple string, which replaces a string '_OPTION_' anywhere in @altermime_args_disclaimer elements. Typical use: @altermime_args_disclaimer = qw(--disclaimer=/etc/_OPTION_.txt); @disclaimer_options_bysender_maps = ( { 'host1.example.com' => 'altermime-disclaimer-host1', 'boss@example.net' => 'altermime-disclaimer-boss', '.example.net' => 'altermime-disclaimer-net', '.' => 'altermime-disclaimer-default' }, ); It is currently not possible to disable adding disclaimers through @disclaimer_options_bysender_maps results. This needs to be improved. The exact interpretation of the @disclaimer_options_bysender_maps lookup result may change in the future (which is why I call it 'experimental'). Note that disclaimers are pretty much useless legally. If you can help it at all, please avoid the pollution. See: http://www.goldmark.org/jeff/stupid-disclaimers/ - as mentioned above, the new SMTP/LMTP client code now supports an LMTP protocol too. This allows amavisd-new to act as an LMTP-to-LMTP content filter, possibly being inserted between MTA and an LMTP-based mail delivery agent such as Cyrus (if checking of outgoing mail is not needed). LMTP is selected when the first field of a $*_method (such as $forward_method, $notify_method, $resend_method, $release_method, $*_quarantine_method) is a 'lmtp:'. Possible uses: $forward_method = 'lmtp:/var/imap/socket/lmtp'; # over a Unix socket or: $forward_method = 'lmtp:[127.0.0.1]:24'; # over IPv4 or: $forward_method = 'lmtp:[::1]:24'; # over IPv6 If a Postfix 'lmtp' service is used to feed amavisd (instead of the more usual content filter feed through a service named 'amavisfeed' or 'smtp-amavis'), make sure not to forget to limit the number of concurrent feeds to amavisd (e.g. lmtp_destination_concurrency_limit=15) to a value same (or less) than $max_servers, or limit the maxproc field in master.cf such as: 'lmtp unix - - n - 15 lmtp' . Note that invoking amavisd as an LMTP delivery agent has a disadvantage that outgoing mail is not being checked, so infected internal hosts are able to pollute the world. Also the pen pals feature is no longer useful, as it requires the information on previous outgoing mail to be present in an SQL database. - a new command line option -i, it takes one argument which can be any string (an instance/personality name), which is then made available to amavisd.conf in a variable $instance_name (intended to be read-only); code in amavisd does not assign any semantics to this argument and does not use it for any purpose, it is purely intended for administrator's use in amavisd.conf if desired; this simple mechanism may facilitate running multiple instances of amavisd using a single configuration file, or to choose at startup time between amavisd personalities using the same config file; A possible usage is to start a test instance of amavisd while a production amavisd is still running, and letting a test instance listen on its dedicated TCP port number. Each server instance needs its own pid and lock files, its own TCP port number or socket name, and its own $db_home (nanny, cache, agent) unless bdb usage is disabled. A working directory may be shared or kept separate. An example to put by the end of amavisd.conf: if ($instance_name eq 'test') { $log_level = 5; $sa_debug = 1; $max_servers = 1; $TEMPBASE = "$MYHOME/tmp-am2"; $ENV{TMPDIR} = $TEMPBASE; $pid_file = "$MYHOME/home/amavisd2.pid"; $lock_file = "$MYHOME/home/amavisd2.lock"; $enable_db = 0; $inet_socket_port = [8888]; # listen on port 8888 } Start a test instance: # amavisd -i test debug and submit a test message to it, e.g.: $ mini_sendmail -fpostmaster@example.net \ -s127.0.0.1 -p8888 test@example.net <0.msg - policy banks now contain a new key 'originating', which generalizes a previously hard-wired policy bank MYNETS. It is a boolean variable, turned on automatically in the currently loaded policy bank when a smtp client's IP address matches @mynetworks_maps, to retain full compatibility with existing setups. When a new policy bank is loaded over a current one, the new policy bank may also modify the 'originating' key - a typical use is to turn it on by a policy bank activated by mail submission from authenticated roaming users (SASL/AUTH), so that such users are treated as locals (originating mail) even though their IP address does not match a @mynetworks_maps list of lookup tables. The current value of variable 'originating' is now the only control to some macros or decisions, which were previously controlled implicitly by a @mynetworks_maps match. These are: * macro %l (letter ell) now directly corresponds to the current value of the 'originating' variable (returning a '1' or an empty string); * some statistics counters differentiate between 'Inbound' and 'Internal' mail directly based on the value of the 'originating' variable (applies to mail with local recipients, otherwise it is 'Outbound'); * pen pals is skipped for senders claiming to be from a local domain, but 'originating' is false (i.e. unauthorized foreigners faking your domain); * only mail with 'originating' flag will be eligible for DKIM signing (starting with version 2.6.0); * there may be other uses in the future, so it is prudent to keep @mynetworks_maps and @local_domains_maps configured correctly, and (when appropriate) turn on the 'originating' flag for mail that is supposed to be treated as originating from internal or authorized roaming users; Example use: $interface_policy{'10026'} = 'ORIG'; $policy_bank{'ORIG'} = { # mail originating from our users originating => 1, # declare that mail was submitted by our smtp client allow_disclaimers => 1, # enables disclaimer insertion if available virus_admin_maps => ["virusalert\@$mydomain"], spam_admin_maps => ["virusalert\@$mydomain"], warnbadhsender => 1, # forward to an smtpd service providing DKIM signing service forward_method => 'smtp:[127.0.0.1]:10027', # force MTA conversion to 7-bit (e.g. before DKIM signing) smtpd_discard_ehlo_keywords => ['8BITMIME'], bypass_banned_checks_maps => [1], # allow sending any file names & types terminate_dsn_on_notify_success => 0, # don't remove NOTIFY=SUCCESS opt. }; - make it possible for a virus scanner to derate an infection report to a spam report, contributing to spam score and to spam report/status. A new configuration variable @virus_name_to_spam_score_maps (also member of policy banks) can turn a reported virus name into a spam score. Its default setting is: @virus_name_to_spam_score_maps = (new_RE( [ qr'^(Email|HTML)\.(Phishing|Spam|Scam[a-z0-9]?)\.'i => 0.1 ], [ qr'^(Email|Html)\.Malware\.Sanesecurity\.' => undef ], [ qr'^(Email|Html)(\.[^., ]*)*\.Sanesecurity\.' => 0.1 ], # [ qr'^(Email|Html)\.(Hdr|Img|ImgO|Bou|Stk|Loan|Lott|Cred|Job|Dipl|Doc) # (\.[^., ]*)* \.Sanesecurity\.'x => 0.1 ], [ qr'^(MSRBL-Images/|MSRBL-SPAM\.)' => 0.1 ], )); and can be replaced in amavisd.conf. To disable the feature assign an empty list to the configuration variable: @virus_name_to_spam_score_maps = (); When a virus scanner returns names of viruses, and all provided names are matched by the @virus_name_to_spam_score_maps, and no other virus scanner has anything more sinister to report, then a message is _not_ flagged as a virus, but a corresponding spam score is contributed to other spam results as returned by a normal spam scan by SA. All the usual spam rules are then followed. Phishing fraud as indicated by ClamAV is now by default treated as spam, and no longer as a virus. The log can now show entries like: amavis[26733]: (26733-03-2) Turning AV infection into a spam report: score=0.1, AV:HTML.Phishing.Auction-289=0.1 amavis[26733]: (26733-03-2) adding SA score 38.628 to existing 0.1 from an earlier spam check amavis[26733]: (26733-03-2) Blocked SPAM, ... Hits: 34.728, ... Tests: [AV:HTML.Phishing.Auction-289=0.1, ... L_AV_Phish=14, ...] The information is also available to SA rules in a form of a synthetic header field X-Amavis-AV-Status which will be seen by SA only (not inserted into passed or quarantined mail). One has a choice to adjust scores either in the @virus_name_to_spam_score_maps in amavisd.conf, or by providing rules to match on the provided header field. Doing it by SA rules has an advantage of letting other SA rules contribute their score points, possibly preventing a false positive of a ClamAV rule, or pushing score even higher for a clean bounce suppression. It also allows auto-whitelisting in SpamAssassin to account for these score points. In addition, it makes more sense when checks are cached and result reused later for some other message with the same contents in body. Here is one example of such SA rules (some long lines are wrapped, these should be unwrapped before placing them into local.cf): header L_AV_Phish X-Amavis-AV-Status =~ m{\b(Email|HTML)\.Phishing\.}im header L_AV_SS_Phish X-Amavis-AV-Status =~ m{\b(Email|Html)\.Phishing(\.[^., ]*)*\.Sanesecurity\.}m header L_AV_SS_Scam X-Amavis-AV-Status =~ m{\b(Email|Html)\.(Scam[A-Za-z0-9]?)(\.[^., ]*)*\.Sanesecurity\.}m header L_AV_SS_Spam X-Amavis-AV-Status =~ m{\b(Email|Html)\.(Spam|Bou|Stk|Loan|Lott|Cred|Job|Dipl|Doc) (\.[^., ]*)*\.Sanesecurity\.}m header L_AV_SS_Hdr X-Amavis-AV-Status =~ m{\b(Email|Html)\.Hdr(\.[^., ]*)*\.Sanesecurity\.}m header L_AV_SS_Img X-Amavis-AV-Status =~ m{\b(Email|Html)\.(Img|ImgO)(\.[^., ]*)*\.Sanesecurity\.}m header L_AV_MSRBL_Img X-Amavis-AV-Status =~ m{\bMSRBL-Images/}m header L_AV_MSRBL_Spam X-Amavis-AV-Status =~ m{\bMSRBL-SPAM\.}m header L_AV_Scam X-Amavis-AV-Status =~ m{\bScamNailer\.}i score L_AV_Phish 14 score L_AV_SS_Phish -3 score L_AV_SS_Scam 8 score L_AV_SS_Spam 8 score L_AV_SS_Hdr 6 score L_AV_SS_Img 3.5 score L_AV_MSRBL_Img 3.5 score L_AV_MSRBL_Spam 6 score L_AV_Scam 8 - added a new concept of a 'blocking contents category', which in most cases corresponds to a familiar 'main contents category' (the highest ranking category of contents pertaining to a message, e.g. virus, blocked, spam, spammy, bad header ...). The difference between the two arises when recipients are declared to be 'lovers' of some higher-ranking contents, or when a higher ranking contents category has its *_destiny set to D_PASS. For example: a message contains a banned part, but is also spam and may even have a bad header. Its contents categories are (simplified): CC_BANNED, CC_SPAM and CC_BADH, in this order. The main contents category of a message is CC_BANNED, which usually is also a reason for blocking a message, yielding a blocking ccat to also be CC_BANNED. But if some recipient is banned_files_lover (or if $final_banned_destiny is set to D_PASS), then the main ccat remains to be CC_BANNED, but the blocking ccat is CC_SPAM, i.e. the next in the list which is responsible for actually blocking the mail. If recipient would also be a spam lover, the blocking ccat might be CC_BADH (if $final_bad_header_destiny were not D_PASS); If a message is not being blocked, the 'blocking contents category' (i.e. a blocking_ccat attribute of a per-message or a per-recipient object) remains empty (undefined). For convenience some internal routines and some new macros fall back to showing the main contents category in this case. Almost all processing decisions, DSN, notification assembling, quarantining, logging etc. is now based on 'blocking contents category' when a message is being blocked, and on 'main contents category' (as before) when a message is not being blocked. There is a new macro 'ccat' which is useful in notification and logging templates, which can query the blocking contents category, as well as a main contents category. It provides access to information that was formerly available through macros ccat_maj, ccat_min, ccat_name, plus access to additional information. Macros ccat_maj, ccat_min and ccat_name are still available, but their use is deprecated, as their functionality has been incorporated into the new macro 'ccat'. Macro 'ccat' takes two optional fixed-string arguments, which are interpreted case-insensitively. In their absence it expands to a string "(maj,min)" which shows a major and a minor contents category number of a blocking ccat for a blocked message, and of a main contents category for a passed message. The first argument specifies which attribute of a ccat is to be provided, the second argument specifies whether a main or a blocking contents category is to be consulted: The first argument may be any of the following strings: name ... provide a human-readable name of a ccat (%ccat_display_names) major ... provide a number: a major contents category, values correspond to CC_* constants in the program minor ... provide a number: a minor contents category, often a 0 <empty>... empty argument (also a default) results in a string "(maj,min)" is_blocking ... '1' if blocking_ccat is true (message is being blocked), or an empty string when a message is being passed; is_nonblocking .. the opposite: '1' if blocking_ccat false, '' otherwise is_blocked_by_nonmain .. '1' if blocking_ccat is true _and_ is different from a main contents category; The second argument may be any of the following strings: main ... provide information on main contents category when asked for name/major/minor/<empty> blocking.. provide information on blocking contents category if it exists, otherwise it falls back to providing info on main ccat; this is also a default in the absence of this argument; For illustration, instead of a former call [:ccat_maj] use [:ccat|major] , instead of [:ccat_min] use [:ccat|minor], and instead of [:ccat_name] please use [:ccat|name] . For more examples please consult the default templates, glued to the end of file 'amavisd'. - when amavisd-nanny is given an invalid command-line argument it now shows 'Usage: ...' as well as a legend for process states; - amavisd-nanny enhanced and new process-state instrumentation added to amavisd daemon; previously only busy/idle states of child processes were shown in amavisd-nanny output, now a more detailed process state can be shown by setting a new verbosity control configuration variable $nanny_details_level to a higher than a default value of 1, e.g. to 2; The following characters in amavisd-nanny bars represent amavisd child process states as follows, in the shown order of events: A accepted a connection b begin with a protocol for accepting a request m 'MAIL FROM' smtp command started a new transaction in the same session d transferring data from MTA to amavisd = content checking just started D decoding of mail parts V virus scanning S spam scanning P pen pals database lookup and updates r preparing results Q quarantining and preparing/sending notifications F forwarding mail to MTA . content checking just finished A nanny bdb database has changed in an incompatible way, so older versions of amavisd-nanny would complain about contents of a new database. Backward compatibility is retained: new version of amavisd-nanny is able to deal with a database from older versions of amavisd-new. There is no need for conversion, a new database is created on each amavisd restart. Note that a history of process states is _not_ maintained in a nanny database, but only in a running amavisd-nanny, which is why a just-started amavisd-nanny can not show previous states of processes from time before amavisd-nanny was started - a '=' is shown instead. A display eventually catches up and all newly-entered states are shown correctly. - snmp-like database can now also store 64-bit counters data type, amavisd-agent utility modified accordingly; - amavisd-agent utility and amavisd daemon enhanced to provide and to display cumulative elapsed time by sections; currently only some of the more important sections have been instrumented, e.g.: TimeElapsedReceiving 10631 s 0.608 s/msg (InMsgs) TimeElapsedDecoding 970 s 0.056 s/msg (InMsgs) TimeElapsedVirusCheck 629 s 0.036 s/msg (InMsgs) TimeElapsedSpamCheck 75866 s 4.341 s/msg (InMsgs) TimeElapsedPenPals 2150 s 0.123 s/msg (InMsgs) TimeElapsedSending 2231 s 0.128 s/msg (InMsgs) TimeElapsedTotal 94709 s 5.419 s/msg (InMsgs) Don't be surprised if the total elapsed time exceeds amavisd uptime, 10 processes progressing slowly for 5 seconds each will accumulate 50 seconds of reported elapsed time. The average seconds-per-message figure as reported in the last column makes more sense; - amavisd-agent utility and amavisd daemon enhanced to provide and to display cumulative mail sizes in bytes, and additional message counters based on outbound/inbound/internal mail direction: message counts: InMsgs all mail received by amavisd (as in previous versions); InMsgsOutbound at least one recipient NOT in @local_domains_map; InMsgsInternal at least one recipient in @local_domains_map, and client IP address (submitter) in @mynetworks_maps; InMsgsInbound at least one recipient in @local_domains_map, and client IP address NOT in @mynetworks_maps; message sizes: InMsgsSize total mail size in bytes as received by amavisd; InMsgsSizeOutbound as above, but with at least one recipient NOT in @local_domains_map; InMsgsSizeInternal at least one recipient in @local_domains_map, and client IP address in @mynetworks_maps; InMsgsSizeInbound at least one recipient in @local_domains_map, and client IP address NOT in @mynetworks_maps; Note that a mail with multiple recipients can be both internal and outbound. P.S. in later versions a hard-wired testing against a @mynetworks_maps list is replaced by testing value of a boolean variable $originating. Example output: InMsgsSize 4332MB 109MB/h 100.0 % (InMsgsSize) InMsgsSizeInbound 3102MB 78MB/h 71.6 % (InMsgsSize) InMsgsSizeInternal 554MB 14MB/h 12.8 % (InMsgsSize) InMsgsSizeOutbound 816MB 21MB/h 18.8 % (InMsgsSize) - new configuration variables $always_bcc and %always_bcc_by_ccat, also members of policy banks, allow adding one extra envelope recipient to each message, either regardless of contents ($always_bcc), or selectively based on contents category. For example: $always_bcc = 'archiver+clean@example.com'; or selectively based on contents category: $always_bcc_by_ccat{+CC_CLEAN} = 'archiver+clean@example.com'; $always_bcc_by_ccat{+CC_VIRUS} = 'archiver+virus@example.com'; or as a member of policy banks: $policy_bank{'MYNETS'} = { always_bcc_by_ccat => { CC_BADH, 'archiver@example.com', CC_CLEAN, 'archiver@example.com', CC_CATCHALL, undef, }, }; - amavisd-nanny and amavisd-agent utilities now recognize an optional command-line option: -w <wait> , where the specified value is time in seconds between re-displays. The default interval is 2 seconds for amavisd-nanny, and 10 seconds for amavisd-agent as before. The specified interval time may be fractional; - macro 'useragent' can accept an optional argument: a string 'name' or 'body', restricting the information to be returned as follows: macro 'useragent' returns 'User-Agent: ...' or 'X-Mailer: ...' header field from a message (whichever is present, or empty); an optional argument specifies whether: an entire field is to be returned (empty or unrecognized argument), or just a field name (argument: 'name'), e.g. 'X-Mailer'; or just a field body (argument 'body'), e.g. 'Thunderbird_1.5.0.9'; - interfacing to Mail::ClamAV (a perl module to a clamav library) now performs processing in a subprocess to prevent bugs in external library from bringing down amavisd process, and to prevent virtual memory of an amavisd child process from expanding uncontrollably - at the expense of additional 20..30 ms for a fork; - extended AM.PDP protocol with an attribute 'policy_bank' which may be used in a client's request to require loading additional policy banks, e.g.: policy_bank=TLS,ORIGINATING,MYNETS Its value is a comma-separated list of policy bank names. Names of nonexistent banks are silently ignored, so are leading and trailing spaces and TABs around each name. The order of policy bank loading generally follows the order in which information about a message were obtained: - interface- or socket-based policy banks (when MTA connects to amavisd); - MYNETS (when client's IP address becomes known); - the list of policy bank names as specified in a 'policy_bank' attribute of AM.PDP protocol, comma-separated; - MYUSERS (when sender's e-mail address becomes known); - added a field 'Final-Log-ID' to a DSN report (RFC 3464), which will provide information on log_id and mail_id, e.g. '77790-10-3/uez9wtcVNTO5' in a standard way, much like the 'Our internal reference code for your message is: ...' in a DSN plain text part; - added mapping from 'RIFF...animated cursor' to ['movie','ani'] in $map_full_type_to_short_type_re, to facilitate blocking animated cursors (Microsoft Windows ANI header stack buffer overflow is being actively exploited); by Henrik Krohns; - add support for 7-Zip archives if external utility program 7z is available (under names 7zr, 7za or 7z); suggested by Bob Marcan; see: http://www.7-zip.org/ - add configurable global settings $min_servers, $min_spare_servers, and $max_spare_servers (all undefined by default, see Net::Server::PreFork documentation for their semantics), pass them to Net::Server at startup time (complementing the usual $max_servers setting) and allow to choose between Net::Server personalities Net::Server::PreForkSimple and Net::Server::PreFork - if $min_servers is defined the PreFork is chosen, otherwise the more usual PreForkSimple. The feature is mostly intended for use of amavisd as a pre-queue content filter, which is unsupported anyway. For normal post-queue use the PreForkSimple already does a good job. Based on a patch by Alexander 'Leo' Bergolth; - internal: incompatibly changed order and indirection level of arguments to routines dealing with contents categories, and some of their names; - new macro 'join', behaves like a Perl join function: the first argument is a separator string, remaining arguments are strings to be concatenated, with a separator string inserted at every concatenation point; - macro 'dquote' (as used in a default log template to protect Subject) previously escaped double quotes with \, but missed to escape \ itself, making log parsing tricky; also, as logging layer escapes \ by itself, the result was ugly and inconsistently parsable; new behaviour is to protect a double quote within a string by doubling it, so a [dquote|one"oops"two] now yields "one""oops""two", instead of "one\"oops\"two", which when logged showed as "one\\"oops\\"two"; - convenience: do not drop privileges early despite a command line option -u when an option -R is also specified with a non-empty (and non-slash) value, otherwise the requested chroot operation is not possible (root privileges are required for chrooting); - version 2.4.3 introduced some substitutions of subject tag template strings: SCORE, REQD, YESNO and YESNOCAPS; this list is now extended with few more, to facilitate cross-host troubleshooting; the full list now consists of: _SCORE_ spam score (hits), same as macro %c _REQD_ tag2_level _YESNO_ score above tag2_level? 'Yes' or 'No' _YESNOCAPS_ same, but yields: 'YES' or 'NO' _HOSTNAME_ fqdn of this host ($myhostname), same as macro %h _DATE_ rfc2822 timestamp of mail entering this amavisd, as macro %d _U_ iso8601 UTC timestamp of mail entering this amavisd, as %U _LOGID_ log id (am_id) as shown in the log, e.g. 58725-05-2, as %n _MAILID_ mail_id as used in quarantine names, e.g. jaUETfyBMJHG, as %i See also README.customize for explanation of macros. --------------------------------------------------------------------------- January 30, 2007 amavisd-new-2.4.5 release notes SECURITY - Recommended version of Convert::UUlib is 1.08 or higher to avoid processing of uninitialized data containing 'random' garbage. Note that a security hole in uulib which comes with Convert::UUlib 1.04 and older is now (as of 2006-12-05) known to be exploitable: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2005-1349 credits to Jean-Sébastien Guay-Leroux; - p0f-analyzer.pl will no longer reply to queries coming from low-numbered UDP ports below 1024 or from nfsd port 2049, and will ignore queries with nonce longer than 1024 character or containing characters outside of \040-\177 (octal) range to limit its usefulness as a potential reflector for an attacker from internal networks. INCOMPATIBLE CHANGE WITH 2.4.4 - p0f-analyzer.pl now only binds to a loopback interface by default, instead of to all interfaces; change $bind_addr in p0f-analyzer.pl to '0.0.0.0' if p0f-analyzer.pl is running on a different host from amavisd or from other querying clients; suggested by Shaun T. Erickson and Mario Liehr; BUG FIXES - let p0f-analyzer.pl exit when a pipe on stdin is closed (e.g. when p0f is killed or crashes), instead of entering a tight loop; reported by Justin Piszcz and Henrik Krohns; - hard-blacklisting no longer skips quarantining when $spam_quarantine_cutoff_level is undefined (or is an empty string); - restart timer after Sophie times out; previously the next attempt would run with no time limit; reported by Nick Leverton and Nicklas Bondesson; - fix error reporting in open_on_specific_fd when POSIX::dup2 fails; thanks to Chris (decoder); - fix signal handling in read_snmp_variables() and register_proc(), a signal could previously get lost (not re-signaled) if it occurred within these subroutines; - fixed get_body_digest which incorrectly determined 7- or 8-bitness of mail header and body, setting body_type incorrectly (with only cosmetic ill-effects); - fixed AM.PDP code to always provide an smtp-quoted form in angle brackets in 'delrcpt' and 'addrcpt' attributes of a response, i.e. in the same form as was received in 'sender' and 'recipient' attributes. The attribute value syntax is specified in RFC 2821 as 'Reverse-Path' (i.e. smtp-quoted form, enclosed in <>); previously enclosing angle brackets were missing in a server reply; - documentation - amavisd.conf-default incorrectly stated that a default value for $prepend_header_fields_hdridx is 1; actually the default is 0 as correctly indicated in release notes; reported by Jo Rhett; OTHER - qmail interfacing notice: MTA timeout for waiting on results from amavisd should be longer than $child_timeout (8 minutes by default) with some margin, setting MTA timeout to 15 or 20 minutes is usual. With qmail however the QMQP code in qmail has hard-coded timeouts set, 10 seconds for connect and 60 seconds for read/write. If amavisd processing takes longer than 60 seconds, the MTA drops connection and retries later, yet amavisd continues processing and eventually delivers a mail (with each MTA retry), causing repeated deliveries of the same message. The following patch by Eric Huss on the www.qmail.org page: http://www.ehuss.org/qmail/qmqpc-timeout.tar.gz should be applied to qmail when interfacing it to a post-queue content filter. Problem researched by Nicklas Bondesson; - better timeout handling in interface code to daemonized virus scanners like clamd, Sophie, Trophie: allow short time (10 s) for connect and for sending a request, then allow normal (long) time to collect results; keep evidence of the initial deadline on retries; - prefer '7bit' as Content-Transfer-Encoding when attaching original message or its headers (message/rfc822 or text/rfc822-headers) to DSN or to a defanged mail, and only specify '8bit' when necessary; - remove protecting the $ and @ characters in second argument of a regexp selector macro, it is unnecessary and confusing; - macros %m, %r and header_field now return parsed and sanitized message IDs in header fields Message-ID, Resent-Message-ID. In-Reply-To, or References, void of CFWS (comments and FWS as specified by RFC 2822), through the use of new subroutine parse_message_id(); - when logging to SQL, the field msgs.message_id now contain just a message id, without CFWS and other garbage that might appear in a Message-ID header field; this facilitates queries, and pen pals matching of IDs in In-Reply-To or References header fields of a reply to an original Message-ID; - updated $map_full_type_to_short_type_re to avoid mapping file(1) result 'MS-DOS executable (built-in)' to types 'exe-ms' and 'exe'; the file(1) utility generously declares any text file starting with LZ to be a 'MS-DOS executable (built-in)'; thanks to Noel Jones, Jakob Curdes and Clifton Royston for troubleshooting; - add X-Spam-* header fields to quarantined mail if spam score is at or above tag_level. Previously message needed to be recognized as spammy or spam (tag2 or kill level) in order to receive spam header fields in quarantined copy. This also makes it more consistent with adding such header fields to passed mail; suggested by Michael Gaskins; - add X-Amavis-OS-Fingerprint header field to quarantined mail; - header field X-Spam-Score in a passed or quarantined mail now reflects score boost even when SA score is unknown (e.g. when SA was not called), and reflects white and blacklisting by pushing score to 0 or 64, to make it consistent with a bar size in X-Spam-Level header field; - resignal "timed out" after (almost) every eval {} which has no subsequent call to prolong_timer() to ensure we do not continue running with disabled timer. Exceptions are DESTROY and END handlers, and code which handles timer in some other way (e.g. by keeping evidence of a deadline); - for the purpose of looking up client IP address in @mynetworks_maps, treat unknown/unavailable IP address as 0.0.0.0; this allows treating directly submitted mail on the MTA host (not submitted through SMTP) as coming from IP address 0.0.0.0 (i.e. "This" Network - according to RFC 1700); Note that this is indistinguishable from other reasons when IP address is not made available to amavisd, e.g. when smtp_send_xforward_command option in Postfix smtp service is not enabled, which is why the default setting of @mynetworks does not include a 0.0.0.0/8 network to prevent unintentionally loading a MYNETS policy bank. One should add 0.0.0.0/8 to a @mynetworks list only when XFORWARD is known to work and if some software on the MTA host is submitting its mail to MTA directly, e.g. through a sendmail mail submission command (or its lookalike), and MYNETS policy bank loading is needed for proper processing of such mail (e.g. DKIM signing or adding disclaimers in later versions of amavisd); - report a more informative message when a file(1) utility fails to produce useful results: joins exit status with a parsing report into one message; thanks to Andres, whose file(1) utility was crashing with SEGV; - consistency: rearrange implicitly adding $X_HEADER_TAG to a hash %allowed_added_header_fields so that it is possible to turn off insertion of $X_HEADER_TAG header field by turning off associated key in %allowed_added_header_fields even when $X_HEADER_TAG is explicitly defined; - let %allowed_added_header_fields also control insertion of header fields into quarantined message; - amavisd-nanny now displays a title line indicating the semantics of columns; - Courier patch: ensure the information is stored to newly introduced recip_addr_smtp and sender_smtp object attributes, which are needed to preserve pristine address forms for DSN and ORCPT use and for logging; a patch by Martin Orr; - qmqpqq (qmail): ensure the information is stored to newly introduced recip_addr_smtp and sender_smtp object attributes; - qmail patch now activates line-by-line sending to qmail to avoid qmail bug ('bare LF' reported when CR and LF are separated by a TCP packet boundary); - tighten a regexp on matching a p0f fingerprint for Windows XP to avoid matching 'Windows XP SP1+, 2000 SP3'; suggested by Michael Scheidell; - updated AV entry for CentralCommand Vexira (vascan): removed hard-coded option '--vdb'; by Brian Wong; - internal: move code dealing with a SA call to a dedicated subroutine call_spamassassin; - internal: provide new routines to collect scalar and structured results from a subprocess (collect_results, collect_results_structured) and take advantage of them in decoding, in AV and in dspam interface routines, unifying code and providing results size sanity limit and consistent killing of runaway external programs; - experimental: taking advantage of the above, make it possible to run SA in a spawned process, requested by setting a new config variable $sa_spawned to true (it is off by default); benefits are that a mainstream child process can not be brought down by potential processing problems in SA or its external modules, and timeouts are handled cleanly by a calling process; downside is an increase of process count (worst case: doubled), with corresponding increase in memory footprint, plus about 20 .. 30 ms of additional processing time for each call to SA; - added a tuning tip on buffer sizes to README.sql for MySQL with InnoDB, by Wayne Smith; - updated URL of Sophie AV scanner; --------------------------------------------------------------------------- November 20, 2006 amavisd-new-2.4.4 release notes COMPATIBILITY WITH 2.4.3 - PostgreSQL quarantining: data type of field quarantine.mail_text should be 'bytea' (instead of 'text') to allow storing arbitrary octets without associating them with a character set. See below for a conversion of an existing database. Similarly with MySQL the data type should be 'blob'. - Note: in a sendmail milter setup with Petr Rehor's helper program amavis-milter, one should set: $prepend_header_fields_hdridx = 1; when dk or dkim signing milters are used in the same setup. See below for details. BUG FIXES: - do_ascii: fix a bug where timer was not restored after decoding of a textual mail part, so a timeout for subsequent decoding operations on the same message was limited to 10 seconds (and to 30 seconds for a call to SpamAssassin), regardless of $child_timeout setting; - don't call PerlIO::get_layers with Perl 5.8.0, the function was introduced with 5.8.1; reported by Joel Nimety; - avoid deep recursion in evaluating a regular expression in header checks which caused very slow testing for presence of a all-whitespace lines in folded header fields for degenerate cases of header; the inefficient expression was introduced with amavisd-new-2.4.0; reported and a sample provided by Kai Risku; - when spam above kill level is to be passed and spam defanging is enabled, SA summary was inserted twice (once for mail contents category being CC_SPAMMY and once for CC_SPAM), fixed. Reported by Gary V and MHahnen; - when logging directly to a file, do create a log file if it does not already exist; (bug introduced with 2.4.3) - make sure a quota limit is untainted when it is given as a command line parameter to external TNEF decoder; reported by MK; - updated Courier patch to loosen up socket protection and allow group write access to the socket; reported by Bill Taroli; - SQL logging: cleanly chop an UTF-8 octet sequence according to RFC 3629 (avoid truncating character octet sequence tail) when Subject, From or Message-Id header field is longer than 255 characters; - PostgreSQL: when storing mail text to a quarantine use pg_type=PG_BYTEA attribute on a field 'quarantine.mail_text'; previously the following error could be reported: 451 4.5.0 Storing to sql db as mail_id ... failed: writing mail text to SQL failed: Error closing, flush: sql inserting text failed, sql exec: err=7, 22P02, DBD::Pg::st execute failed: ERROR: invalid input syntax for type bytea - updated documentation in README.sql to suggest using data type 'bytea' instead of inappropriate data type 'text' for a field quarantine.mail_text To convert an existing table (when quarantining to SQL) please use: ALTER TABLE quarantine ALTER mail_text TYPE bytea USING decode(replace(mail_text,'\\','\\\\'),'escape'); If conversion of data type for 'quarantine.mail_text' is not done, the following error will be reported when storing a message to an SQL quarantine is attempted: TROUBLE in check_mail: quar+notif FAILED: temporarily unable to quarantine: 451 4.5.0 Storing to sql db as mail_id ... failed: writing mail text to SQL failed: Error closing, flush: sql inserting text failed, sql exec: err=7, DBD::Pg::st execute failed: ERROR: column "mail_text" is of type text but expression is of type bytea HINT: You will need to rewrite or cast the expression If converting quarantine table is not desirable or possible in a short term, it is possible to continue use existing SQL quarantine table without conversion by specifying the following in amavisd.conf: $sql_clause{'ins_quar'} = "INSERT INTO quarantine (mail_id, chunk_ind, mail_text)". " VALUES (?,?,encode(?,'escape'))"; $sql_clause{'sel_quar'} = "SELECT decode(mail_text,'escape') FROM quarantine". " WHERE mail_id=? ORDER BY chunk_ind"; This will allow PostgreSQL to convert data types on-the-fly, converting octets (any byte) into escaped text, and vice versa when releasing from a quarantine; Problem reported by Justin Hillyard, correct data type suggested by Nikola Milutinovic; - MySQL: updated documentation in README.sql to suggest using data type 'blob' instead of inappropriate data type 'text' for a field quarantine.mail_text. To convert an existing table please use: ALTER TABLE quarantine CHANGE mail_text mail_text blob; Seems like MySQL does not complain on incompatibility between provided data type and a data type of a field in table, but there are reports that MySQL may silently truncate data which it finds violating character set constraints, so conversion to 'blob' is highly recommended. Truncation of quarantined message at an 8-bit character reported by Lubor Kolar. OTHER CHANGES: - limit recursion in MIME::Parser to $MAXFILES to prevent MIME parser from fully traversing degenerate cases of broken MIME messages which can take excessive amount of time and memory; reported and a sample provided by Joshua Goodall, solution suggested by David F. Skoll, and requires a parser method max_parts(), available in MIME::Parser 5.417 or later; - check for already running daemon at startup time, preventing a user mistake of trying to start another instance of the daemon without stopping the currently running process; suggested by Jo Rhett; - keep sender and recipient addresses in original unparsed form (in addition to an internal form) to be able to always provide exact original address in delivery status notifications, in ORCPT, and when appending extensions in a milter setup (AM.PDP), which requires exact matching to the original form (without stripping route and without fixing poorly SMTP-quoted address forms); - new configuration variable %allowed_header_tests, also member of policy banks, allows for selectively disabling some of the header checks, e.g. checks for non-encoded 8-bit characters. The %allowed_header_tests hash contains all available header test names as its keys by default (with a value of true); removing a key, or setting its value to false, disables a test, e.g.: $allowed_header_tests{'8bit'} = 0; $allowed_header_tests{'missing'} = 0; Currently available keys (i.e. test names) are: other mime 8bit control empty long syntax missing multiple each corresponding to its own minor contents category of CC_BADH; ccat test min name description --- ------- ----------- 0 other (catchall for everything else, normally not used) 1 mime Bad MIME (sub)headers or bad MIME structure 2 8bit Invalid non-encoded 8-bit characters in header 3 control Invalid control characters in header (CR or NUL) 4 empty Folded header field made up entirely of whitespace 5 long Header line longer than RFC 2822 limit of 998 characters 6 syntax Header field syntax error 7 missing Missing required header field 8 multiple Duplicate or multiple occurrence of a header field legend: ccat min: minor contents category under a major category CC_BADH, available in templates as a macro ccat_min; test name: corresponding test name - a key in %allowed_header_tests; descr.: description of a header test or MIME subheaders/structure test; - timing report has a couple of new entries to facilitate troubleshooting: header checks section, separate entry for header and body digests, check_mail initialization, entries 'SMTP greeting' and 'SMTP response'; - when exec in a forked process fails, call POSIX::_exit with exist status 8 (ENOEXEC) instead of the more common 1 to make the failure more obvious; (a note from the future: status 6 (SIGABRT) used since 2.6.2); - initialize logging earlier so that do_log may be called earlier during program startup; also log attempts to stop and to reload, including unsuccessful ones; - avoid logging by a forked process before exec, when there is a chance the log file descriptor is in a range 0..2; - sub run_command and run_command_consumer: distinguish between undefined and empty values of argument $stderr_to, undef now prevents reopening of file descriptor 2, making it possible for the caller to keep it attached to the current stderr; this is useful when run_command is called by the master process before logging has been configured; - SQL: explicitly call DBI::bind_param to be able to specify data types of values passed in @args to Amavis::Out::SQL::Connection::execute; - bump up buffer size from 16 kB to 64 kB in some cases of copying data from/to a pipe, mostly to reduce the amount of logging; - av scanner update: 'FRISK F-Prot Antivirus' entry modified to recognize name of a 'security risk' result, thanks to Michael Renner; - in a commented-out code providing a qmail CF/LF bug workaround, replaced $smtp_handle->datasend by $smtp_data_fh->print, which is more efficient in a line-by-line writing mode needed by qmail; thanks to Ronald Vazquez; - in a (banning) check for double extensions allow for whitespace around the second filename extension (files amavisd.conf and amavisd.conf-sample); based on a sample provided by Patrick T. Tsang; - setting $max_requests to 0 disables the limit, process will not be replaced based on the number of requests it has completed (but may still be replaced for other reasons); primarily intended for testing; - bump up a default value for $max_requests from 10 to 20 to match the suggested/example value in amavisd.conf-sample; - AM.PDP/milter setup: new configuration setting $prepend_header_fields_hdridx, also a member of policy banks, with a default value of 0. It is used as an argument hdridx in an AM.PDP attribute 'insheader' which in a milter setup is passed on as an argument hdridx to a smfi_insheader call. The value of $prepend_header_fields_hdridx only affects AM.PDP protocol and only if $append_header_fields_to_bottom is false (it is false by default). If more than one milter is used, all milters should be inserting their header fields at the same index (all prepending or appending, avoiding insertion in the middle of a header), otherwise the resulting order of header fields in a modified header becomes surprising, and in combination with signing milters like DKIM or DK the signature verification will most likely fail. The default value of 0 is normal and useful in combination with other content-checking milters. Signing milters like dkim-milter and dk-milter insert their header at index 1 (just below the new Received header fields), and when amavisd-new with Petr Rehor's helper program amavis-milter is used as a milter along with dkim-milter or dk-milter, the value of $prepend_header_fields_hdridx MUST BE SET TO 1, otherwise the generated signature will fail verification at the receiving site! Discussion: when sendmail calls its milters, its Received header field is not yet created and passed on to milters, yet it is already counted as one header field for the purpose of smfi_insheader hdridx interpretation. When a milter wants to prepend its header field(s), specifying hdridx of 0 does prepend its header fields above the yet-to-be-inserted Received header field as expected, and specifying 1 inserts its header field(s) just below the yet-to-be-inserted Received header field. If some milters in a chain specify 0 and others a 1 it affects the final order of inserted header fields in unexpected ways. It would be natural to always prepend fields with an index 0, but for signing milters like dk-milter this is not acceptable, as it would be expected to include a not-yet-available Received header field in its signature. For this reason signing milters like dkim-milter and dk-milter insert their header fields (signature) at index 1, and if amavisd-milter wants to coexist in such a setup, it must also insert its header fields at index 1. The conclusion: when amavisd (with its helper program) is used in a milter setup along with other milters, it should use the same hdridx value as other milters, which in case of signing dkim-milter and dk-milter is 1. If there are no such milters, either a 1 or a 0 would do, although a value of 0 produces a more natural order of header fields, matching that of a post-queue content filtering setup. See threads: http://archives.neohapsis.com/archives/postfix/2006-10/1777.html http://archives.neohapsis.com/archives/postfix/2006-11/0467.html --------------------------------------------------------------------------- September 30, 2006 amavisd-new-2.4.3 release notes For new instructions on setting up DKIM and DomainKeys with Postfix and amavisd-new please see: http://www.ijs.si/software/amavisd/amavisd-new-docs.html#dkim COMPATIBILITY WITH 2.4.2 - in a sendmail milter setup using AM.PDP protocol (e.g. by Petr Rehor's amavisd-milter), inserted header fields are now prepended to mail header by a new AM.PDP protocol attribute 'insheader', so an upgrade of the milter helper is needed to support the change (e.g. amavisd-milter 1.1.3), otherwise header field insertions will be ignored; - due to enhanced header checks (checking for missing required fields and checking for multiple occurrences of header fields which are allowed to occur only once), new cases of invalid mail may pop up in bad headers category, e.g. mail posted by MUA like Eudora; - a string %i in quarantine filename templates (such as $spam_quarantine_method) now uses a 'T' as a date/time separator, e.g. 20060814T173846 (conforming to iso8601), instead of a former '-'; - when loading a policy bank, its entries of type hash are now merged with existing hash, a key at a time; previously a newly loaded hash replaced a previous one entirely; BUG FIXES AND WORKAROUNDS - fixed a bug (introduced with amavisd-new-2.4.0): when receiving mail from MTA through an LMTP protocol (not SMTP) and with D_BOUNCE as a final*destiny setting, a suppressed non-delivery notification (e.g. spam above cutoff_level) did not turn LMTP status into a success, so an undesired bounce was generated by MTA in a post-queue filtering setup, contributing to excessive bounce backscatter; reported by Michael Scheidell, thanks to Gary V for analysis; - bug fix to amavisd-release: a regexp needs to be relaxed to allow quarantine names like Y/spam-Y5y7A3J5r2Ax.gz, reported by Rob Chanter; - fix a bug in LDAP lookups which could lead to an infinite loop while expanding %m in the filter; reported by Petr Vokac; - add "LOCAL_STATE_DIR => '/var/lib'" to the SA object initialization for versions of SA 3.1.4 or older, so that SpamAssassin would see additional rules provided by sa-update and placed to its default location; the SA 3.1.5 provides its own default so this becomes unnecessary; - bug fix: don't reject mail when mail size restriction is in force, the limit is exceeded, and $final_destiny_by_ccat{+CC_OVERSIZED} is not D_REJECT; - treat blacklisting as high spam score when considering suppressing quarantining (@spam_quarantine_cutoff_level_maps) or suppressing sending a DSN (@spam_dsn_cutoff_level_maps); - calling do_quarantine() multiple times on the same message would accumulate header edits from each invocation, fixed; (such situation can only happen with a modified program); - when defanging mail or releasing mail from a quarantine, with a goal of not breaking DKIM Sender Signing Practices (SSP) and DomainKeys policy, do not copy existing Sender header field to a new header, and insert our own Sender field (configurable by %hdrfrom_notify_recip_by_ccat); - explicitly set PerlIO layer to ":bytes" on a temporary file handle for email.txt (just in case); based on a problem report by Alexander Schäfer; - in a string produced by a macro %c remove a decimal dot if score happens to be an integer; - reduce $sa_mail_body_size_limit from 512 kB to 400 kB in amavisd.conf and amavisd.conf-sample for the time being, while the SA folks work on http://issues.apache.org/SpamAssassin/show_bug.cgi?id=5041 (MS Outlook Express seems to be chopping long mail in approx 500 kB chunks); - another workaround for Perl taint bug: IO::Handle::_open_mode_string taints the $1 when mode string to IO::File::open is '+<', use O_RDWR instead; thanks to Ryan Frantz; - abort if a specified syslog facility name is unknown, instead of switching to LOG_DAEMON as before; - change the code which selects defanging so that defanging is triggered if any applicable contents category of a message chooses defanging; counterintuitive behaviour reported by Tapani Tarvainen; - fix example in amavisd.conf-sample to use +CC_SPAM instead of CC_SPAM as a key to a hash, e.g. $final_destiny_by_ccat{+CC_SPAM}, otherwise Perl would implicitly turn CC_SPAM into a string when used in such a context. Note that any Perl expression syntax would do, as long as the argument does not look like a plain variable which receives implicit quoting; possibilities include $xx{&CC_SPAM}, $xx{+CC_SPAM}, $xx{CC_SPAM()}, $xx{(CC_SPAM)} and similar; a more obvious &CC_SPAM is avoided because it prevents subroutine call inlining optimization in Perl; Note that the '+' trick does not prevent implicit quoting with the '=>' operator, so this is wrong: %h = (+CC_SPAM => 1); use '&' or a '()', e.g. %h = ( CC_SPAM() => 1 ); or %h = ( &CC_SPAM => 1 ); or avoid the '=>' operator and use a comma: %h = ( CC_SPAM, 1 ); - qmail: update amavisd-new-qmqpqq.patch to be compatible with Net::Server version 0.91 or later; thanks to mr from DBA Lab S.p.A.; - AM.PDP protocol: change the order of attributes returned in an reply: delete and edit header fields before adding new header fields; problem of deleting just-inserted header fields in a sendmail milter setup reported by Petr Rehor; - AM.PDP protocol change - with version 2 of the protocol the following changes to the protocol were made: * "version_server=2" is provided in a server response as the first attribute, older versions did not provide such attribute (assumed version on the server side was 1); * delheader and chgheader now stand in a response before insheader and addheader, assuming that milter MTA will execute these in the same order; * new attribute: "insheader=hdridx hdr_head hdr_body" (where hdridx as used by amavisd will always be 0 for now), making it possible to prepend header fields in a sendmail milter setup (instead of appending them, breaking compatibility with DomainKeys); problem noted by Adam Gibson and Petr Rehor; * new attribute: "quarantine=reason" to place message on hold or to a quarantine maintained by MTA, and supply a reason text (e.g. client may call smfi_quarantine milter routine); For future use - it is currently (2.4.3 or earlier) never used. OTHER CHANGES AND SMALL FEATURES: - turn UTF8 warnings into fatal errors by: use warnings FATAL=>'utf8'; - reduced default $spam_check_negative_ttl to 10 minutes (from 30 minutes) - when defanging, enforce 998 characters line length limitation imposed by RFC 2822 by truncating long lines and appending a "..."; - incompatible change: a string %i in quarantine filename templates (such as $spam_quarantine_method) now uses a 'T' as a date/time separator, e.g. 20060814T173846 (conforming to iso8601), instead of a former '-'; - enhance header checks with checking for missing required fields (Date, From) and with checking for multiple occurrences of header fields which are allowed to occur only once: Date, From, Sender, Reply-To, To, Cc, Bcc, Message-ID, Subject, In-Reply-To, References. Added minor content types to major category CC_BADH: 7=missing, 8=multiple header field; - use enhanced status code 5.7.0 instead of 5.7.1 for blocking spam and viruses, and use 5.6.0 for blocking mail with invalid header; - macro SCORE now returns a single number instead of an explicit sum of SA score and boosts (contributed by soft-w/b-listing or pen-pals). When a single number is preferred in the log (Hits: ...), use this macro instead of %c in a log template: , Hits: [:SCORE]# based on a problem report by Ed Lucero; - updated Panda pavcl AV entry to match the new version called 'Panda CommandLineSecure 9 for Linux', thanks to Andrzej Kukula; - updated AV scanner entries for ESET NOD32 for Linux Mail servers, ESET NOD32 for Linux File servers, F-Secure Antivirus for Linux servers and clamscan, thanks to Anders Norrbring: - updated AV entry for Kaspersky AV version 5.5, thanks to Harrie Overdijk, Anders Norrbring and Gary V; - 'reload' and 'stop' command line options now report a process id of the previous daemon that was killed; - allow command-line option -d to specify a list of SA debug areas (also called 'facilities'); some useful examples: amavisd -d plugin,dkim,dk,spf debug-sa amavisd -d auto-whitelist,bayes,learn debug-sa amavisd -d dcc,razor2,pyzor,util debug-sa amavisd -d dns,uri,uridnsbl debug-sa amavisd -d received-header,metadata debug-sa amavisd -d rules,check debug-sa - substitute string _SCORE_ in spam_subject tag templates with actual score, _REQD_ with tag2 level, and _YESNO_ with 'Yes' (_YESNOCAPS_ with 'YES') if score is above tag2 level, and 'No' (or 'NO') otherwise; this is a quick-fix measure for often demanded feature; currently the mechanism is a simple string substitution and not a true macro expansion, so other macros are not currently available - more refined solution is expected for some future release; - reshuffled the order of rules (with minor adjustments) in example files amavisd.conf and amavisd.conf-sample to make it easier to permit certain files within archives; - introduce new variable @spam_dsn_cutoff_level_bysender_maps (also member of policy banks), complementing an existing by-recipient list of lookup tables @spam_dsn_cutoff_level_maps. The new variable serves to make it possible to trim down spam bounces to domains sending their own bounces with non-null return path (envelope sender address) and without DSN NOTIFY=NEVER option, but also to frequently abused domains, or to those sending marginal spam. When spam level exceeds either the @spam_dsn_cutoff_level_bysender_maps or the @spam_dsn_cutoff_level_maps level, (non)delivery status notification is suppressed even with $final_spam_destiny set to D_BOUNCE; - introduce new variables $resend_method and $release_method (also members of policy banks), both are undefined by default. If defined and nonempty, $resend_method overrides forward_method on forwarding a defanged mail, and $release_method overrides notify_method on releasing a message from quarantine. The $resend_method might be useful when a modified mail requires local DKIM or DomainKeys re-signing; - added global configuration variables $sql_lookups_no_at_means_domain and $ldap_lookups_no_at_means_domain, both false by default. They control whether a database mail address field with no '@' character represents a local username, or a domain name. By default (value false) it indicates a username in SQL and LDAP lookups (but represents a domain in hash and acl lookups), so domain names in SQL and LDAP should be specified as '@domain'. Setting these to true will cause 'xxx' to be interpreted as a domain name, just like in hash or acl lookups, which may facilitate interoperability with databases from other applications; - added global configuration variable $sql_quarantine_chunksize_max, which determines a maximum size (in bytes) for data written to a field 'quarantine.mail_text' when quarantining to SQL. Must not exceed size allowed for a data type on a given SQL server (e.g. maximum size for data type 'blob' in MySQL is 65535 bytes). It also determines a buffer size in amavisd. Too large a value may exceed process virtual memory limits or just waste memory, too small a value splits large mail into too many chunks, which may be less efficient to process; defaults to 16384; - added configuration variables @archive_quarantine_to_maps and $archive_quarantine_method, allowing for archival quarantine of all mail (configurable by recipient and by policy banks) regardless of its contents category. This archive is independent from other quarantining, i.e. if spam quarantining and archival quarantining are both enabled, two copies will be stored to quarantine. When quarantining for archive one has two choices: archive_quarantine would store all mail addressed to recipient, whereas enabling clean quarantine as in: $quarantine_method_by_ccat{+CC_CLEAN} = 'local:clean-%m'; $quarantine_to_maps_by_ccat{+CC_CLEAN} = 'clean-quarantine'; would quarantine only clean mail, no spam, no viruses, no banned, no badh. Note that logging to SQL has only one field to store quarantine location, so in case of multiple quarantine locations only the first is remembered. The usual logging however reports all quarantine locations with the main log entry. - added a global configuration variable @additional_perl_modules, which is a list of additional Perl module names or absolute file names that should be compiled/executed (by calling 'require') at a program startup time by a master parent process, before chroot-ing and before changing UID takes place. Its purpose is to pre-load additional non-standard SpamAssassin plugins and similar modules that a standard SpamAssassin initialization would miss, causing them to be loaded later by each child process, which is inefficient and may not work in a chrooted process. Example: @additional_perl_modules = qw( /usr/local/etc/mail/spamassassin/FuzzyOcr.pm /usr/local/etc/mail/spamassassin/ImageInfo.pm /usr/local/etc/mail/spamassassin/WebRedirect.pm String::Approx Net::HTTP Net::HTTP::Methods URI URI::http URI::_generic URI::_query URI::_server HTTP::Date HTTP::Headers HTTP::Message HTML::HeadParser HTTP::Request HTTP::Response HTTP::Status LWP LWP::Protocol LWP::Protocol::http LWP::UserAgent LWP::MemberMixin LWP::Debug ); Make sure these files are owned by root and not writable by unprivileged users such as amavis! - added a global config variable $enforce_smtpd_message_size_limit_64kb_min, true by default; when true an rfc2822 requirement that a limit on mail size must not be below 64 kB is enforced, so that any specified limit below 64 kB is treated as 64 kB; setting this variable to false disables this check, so mail size restrictions below 64 kB can be used and are effective; - added a by-contents-category setting %subject_tag_maps_by_ccat, unifying former separate settings @spam_subject_tag_maps, @spam_subject_tag2_maps, @spam_subject_tag3_maps and $undecipherable_subject_tag, and making it possible to specify subject tags (strings to be inserted into Subject:) for other categories, such as viruses, banned, and bad headers. Note that now only one such tag is inserted - previously if passed mail was both spam and undecodable two tags were inserted; - when spam level is at or above tag_level, turn on contents category CC_CLEAN with a minor category 1, making it easier to configure actions and settings (like subject_tag) through %*_by_ccat variables; - treat empty string in tag_level the same as undef, i.e. lets X-Spam-* header fields always be inserted for local recipients; - new configuration variable: $allow_fixing_improper_header (also a member of policy banks) is a more-general big brother of an older configuration variable $allow_fixing_improper_header_folding. It controls fixing of a mail header in passed and released mail; it currently controls truncating of header lines longer than 998 characters, and is a pre-condition for $allow_fixing_improper_header_folding, controlling removal of all-whitespace continuation lines. The $allow_fixing_improper_header defaults to true for backward compatibility. Fixing header may protect poorly written mail readers, but may break DomainKeys/DKIM validation of messages with illegal header if verification is done after content filtering, so if this is of concern, one has a choice of turning it off; - added macro sprintf, which invokes Perl's sprints with the usual arguments semantics, e.g. [:sprintf|%%s %%.1f %%%%|text|[:SCORE]] based on suggestion (but swapped arguments) from Joel Nimety; - added macros min and max, which returns minimal or maximal value from their arguments, ignoring empty string arguments, e.g. [:min|100|[:SCORE]] - extend a set of saved header fields in a hash $msginfo->orig_header_fields, which is accessible through a macro 'header_field'; currently the set of kept header fields consists of: From, To, Cc, Sender, Subject, Received, Message-Id, Resent-Message-Id, Precedence, User-Agent, X-Mailer, DKIM-Signature, DomainKey-Signature, Authentication-Results; - added macro useragent, which returns a 'User-Agent: ...' or 'X-Mailer: ...' header field (whichever is present); note that this is an entire field, including a header field head, unlike macros header_field and x-mailer; - added macros dquote and uquote to facilitate sanitation of header fields in logging: dquote encloses its argument in double quotes and replaces existing double quotes by \" (suitable to sanitize Subject header field); uquote replaces one or more consecutive space or tab characters by '_', but does not protect existing underlines, which makes it a lossy transformation (suitable for From or To header fields); provisional - exact interpretation may change; - updated notification templates to make use of new macros; - insert X-Amavis-OS-Fingerprint header field (if available) into a passed message to local recipients (not just to a message copy submitted to SpamAssassin for checking); suggested by Jeff Noxon; - insert X-Amavis-PenPals header field (if information is available) into a passed message, showing time interval (age) since the last message sent in the opposite direction, i.e. from the current recipient to the sender of the current message; - insert X-Amavis-PolicyBank header field into a message that is passed to SpamAssassin for a check (but not to a passed message as seen by recipient). A header field body is a slash-separated list of all policy banks loaded, e.g.: X-Amavis-PolicyBank: AM.PDP-SOCK/MYNETS/MYUSERS (but more usually just: MYNETS). If no policy banks are loaded, the header field will not be inserted. This information may be used by SA rules to add score points based on policy bank, or to countermeasure or conditionalize other rules, for example: header L_MYNETS X-Amavis-PolicyBank =~ m{(^|/)MYNETS(/|$)}m # apply score directly: score L_MYNETS -0.8 # use rule to countermeasure other rules: meta L_MYNETS_UNDISC_RECIPS UNDISC_RECIPS && L_MYNETS score L_MYNETS_UNDISC_RECIPS -0.841 score UNDISC_RECIPS 0.841 # use rule to conditionalize other rules: meta L_OTHERS_UNDISC_RECIPS UNDISC_RECIPS && !L_MYNETS score L_OTHERS_UNDISC_RECIPS 0.841 score UNDISC_RECIPS 0.001 - a new hash variable %allowed_added_header_fields (also member of policy banks) is consulted for each header field insertion, and if the result is false the header field is not inserted into passed mail. Can be used to suppress inserting header fields such as X-Virus-Scanned, X-Spam-Report, X-Spam-Level, X-Amavis-PenPals, X-Amavis-OS-Fingerprint, X-Amavis-Modified, Received, etc. Only applies to passed mail, not to mail that is being written into a quarantine or to a copy submitted to SA for checking. Keys (header field names) must be in lowercase. Example use - disables insertion of certain header fields: $allowed_added_header_fields{lc('X-Amavis-OS-Fingerprint')} = 0; $allowed_added_header_fields{lc('X-Amavis-PenPals')} = 0; $allowed_added_header_fields{lc('X-Spam-Report')} = 0; $allowed_added_header_fields{lc('X-Spam-Status')} = 0; $allowed_added_header_fields{lc('X-Virus-Scanned')} = 0; Note that turning off 'X-Spam-Report' through %allowed_added_header_fields is equivalent to having $sa_spam_report_header at false, turning off 'Received' is equivalent to having $insert_received_line at false, and the last line in the above example is equivalent to setting $X_HEADER_TAG to undef or $X_HEADER_LINE to undef. For compatibility the $X_HEADER_TAG is treated somewhat specially: if explicitly set to a nonstandard value, it is implicitly added to the %allowed_added_header_fields in the base policy bank. No automatism is provided for $X_HEADER_LINE in other policy banks. Example use in a policy bank: $policy_bank{'ALT'} = { allowed_added_header_fields => { lc('X-Amavis-PenPals') => 0, # turn it off lc('X-Amavis-OS-Fingerprint') => 0, # turn it off }, }; - when loading a policy bank, its entries of type hash are now merged with existing hash, a key at a time; previously a newly loaded hash replaced a previous one entirely; this is relevant for policy bank entries such as %allowed_added_header_fields and %*_by_ccat, making it more natural and specifying just keys that need to be changed in a new policy bank instead of entire hashes; - kill external tnef decoder if running for too long; - abort Convert::UUlib::LoadFile or other Convert::UUlib processing in do_ascii() if running for too long; problem case provided by Martin Grimm; - add Net::Server hooks post_configure_hook() and post_bind_hook(), making it easier to affect protection of Unix sockets created by Net::Server (like allowing write access for group) by uncommenting a call to umask in post_configure_hook(); suggested by Mike Gaskins; - internal: wrap most top-level initializations in BEGIN blocks, so that compiled initialization code will be discarded after it did its job, saving about 100 kB of process memory footprint; - update and add several comments - either to clarify code, or to fix typos; - document that a method 'smtp:' can be used for quarantining to a dedicated mailbox, much like the older way of using 'local:anything' with a fully qualified email address in quarantine_to, see: http://www.ijs.si/software/amavisd/amavisd-new-docs.html#quarantine traditionally: $notify_method = 'smtp:[127.0.0.1]:10025'; $quarantine_method_by_ccat{+CC_SPAM} = 'local:%m'; $quarantine_to_maps_by_ccat{+CC_SPAM} = ['quar@example.com']; now more obvious and preferred: $quarantine_method_by_ccat{+CC_SPAM} = 'smtp:[127.0.0.1]:10025'; $quarantine_to_maps_by_ccat{+CC_SPAM} = ['quar@example.com']; - make @viruses_that_fake_sender_maps a member of policy banks to make it possible to bounce certain types of viruses if originating from inside; - added configuration variable @smtpd_discard_ehlo_keywords (also member of policy banks), which is a case-insensitive list of EHLO keywords (AUTH, DSN, 8BITMIME, PIPELINING, SIZE, etc.) that the SMTP/LMTP server will not send in an EHLO response to a remote SMTP client. It is equivalent to a Postfix configuration variable of the same name. Practical use example - DKIM/DomainKeys signing of locally-originating mail after it has passed a content filter: # Configure MTA to send to port 10026 mail originating from our users # (from mydomains or authenticated roaming users, it will be returned # to port 10027 after checking), and to send to port 10024 all the rest # (incoming mail), which will be returned to port 10025 after checking: $forward_method = 'smtp:[127.0.0.1]:10025'; # MTA with non-signing service $notify_method = 'smtp:[127.0.0.1]:10027'; # MTA with DKIM signing service $inet_socket_port = [10024,10026]; # listen on two ports $interface_policy{'10026'} = 'ORIGINATING'; # switch policy bank on 10026 $policy_bank{'ORIGINATING'} = { # mail originating from our users # force MTA to convert mail to 7-bit before DKIM signing # to avoid later conversions which could destroy signature: smtpd_discard_ehlo_keywords => ['8BITMIME'], # forward to an smtpd service providing DKIM/DomainKeys signing service: forward_method => 'smtp:[127.0.0.1]:10027', # other special treatment of locally originating mail, e.g.: spam_admin_maps => ["virusalert\@$mydomain"], # warn of spam from us banned_filename_maps => ['ALT-RULES'], # more relaxed rules... }; master.cf: # mail return from a content filter (non-signing) 10025 inet n - n - - smtpd -o content_filter= ... # mail from our users returning from a filter (DKIM or DK signing service) 10027 inet n - n - - smtpd -o content_filter= ... -o milter_macro_daemon_name=ORIGINATING -o smtpd_milters=inet:127.0.0.1:4444 Note that the same effect (making Postfix convert outgoing mail to 7-bits before DKIM signing) could be achieved by a Postfix setting smtp_discard_ehlo_keywords=8bitmime on an smtp service feeding mail to be signed to amavisd, but this would require setting up two such services, one with the option and one without. - README.postfix: suggest options "-o local_header_rewrite_clients=" and "-o smtpd_milters=" on smtpd at port 10025; thanks to Noel Jones; - README.sendmail-dual: add FEATURE(`nocanonify') on MTA-TX, and FEATURE(`nocanonify',`canonify_hosts') on MTA-RX; thanks to Ricardo Stella; --------------------------------------------------------------------------- June 27, 2006 amavisd-new-2.4.2 release notes SUMMARY OF CHANGES: - new feature: "pen pals soft-whitelisting" lowers spam score of received replies to a message previously sent by a local user to this address; - new feature: added command line options to override certain configuration settings from a config file, see below; - documentation bug fixes, especially on the use of SQL data type TIMESTAMP; - zoo decoder interface routine can now use utility unzoo(1) or zoo(1); COMPATIBILITY WITH 2.4.1: There are no incompatible changes since 2.4.1, but please notice below the fixes to SQL and to LDAP documentation, which may affect you. BUG FIXES AND CHANGES since 2.4.1: - LDAP.schema: add missing LDAP attribute amavisSpamQuarantineCutoffLevel to the list of allowed attributes in objectclass amavisAccount; pointed out by Paolo Cravero; - README.sql PostgreSQL notes: fixed incorrect advice in README.sql which suggested to declare a field msgs.time_iso as TIMESTAMP WITHOUT TIME ZONE instead of the correct TIMESTAMP WITH TIME ZONE. Previous instructions were also contradictory to suggested data type on ALTER TABLE msgs ALTER time_iso. Using inappropriate WITHOUT TIME ZONE when comparing time_iso to now() (which is aware of a time zone) offsets results by a current time zone offset, which unexpectedly includes too many or too few records in maintenance purging operation. A workaround if WITHOUT TIME ZONE continues to be used is to: SET TIME ZONE 'UTC'; before purging. If you were misled by previous documentation choosing WITHOUT TIME ZONE for time_iso, and decided now to convert it to WITH TIME ZONE, the following clause can convert time_iso to proper universal time by manually providing appropriate time offset: ALTER TABLE msgs ALTER COLUMN time_iso TYPE TIMESTAMP WITH TIME ZONE USING time_iso [-+] INTERVAL '[offset]'; Thanks to Brian Wong for a problem description and advice. - README.sql MySQL notes: fixed incorrect advice in README.sql which suggested to declare a field msgs.time_iso as TIMESTAMP instead of the correct TIMESTAMP NOT NULL DEFAULT 0. The "DEFAULT 0" is mandatory to prevent MySQL from overwriting mail reception timestamp with current local time when other fields are updated at the end of processing of a message. Also not to be forgotten: $timestamp_fmt_mysql *MUST* be set to 1 in amavisd.conf with MySQL when msgs.time_iso data type is TIMESTAMP... ! - README.sql MySQL notes: if using field msgs.time_iso to select records for purging (instead of msgs.time_num), and its data type is TIMESTAMP... (as opposed to CHAR...), one should use function utc_timestamp() in place of now() in the DELETE clause to make it work correctly regardless of time zone. Alternatively, now() can continue to be used, provided that SQL client time zone is set to UTC in the purging SQL script: SET time_zone='+00:00'; thanks to Gary V for investigation; - README_FILES/README.sql: added short for-the-impatient sections: * BRIEF MySQL EXAMPLE of a log/report/quarantine database housekeeping * BRIEF MySQL EQUIVALENT EXAMPLE based on time_iso if its type is TIMESTAMPS * BRIEF PostgreSQL EXAMPLE of a log/report/quarantine database housekeeping - a message with only a header, without empty separator line and with no body, lost the last line of a header on forwarding or writing to quarantine; observed by Elias Oltmanns, reported through Debian bug tracking; - header validity checks inappropriately reported 'header field syntax error' as a 'header field too long' (BadHdrLong) instead of BadHdrSyntax; - ensure that notification would not be sent if notification template is empty, solving the following problem: when recipient notifications for bad headers is enabled, and a message is spam with bad headers, recipient would receive an empty notification message (because message contents category is spam and recipient notification template for spam is empty); reported by Alex; - changed SMTP status code 550 to 554 when rejecting mail contents, the 550 is not envisioned in RFC 2821 as a valid reply code to a "." after data transfer; thanks to Victor Duchovni; - fixed case mismatch when storing e-mail address to SQL table maddr, which wasted one unnecessary failed attempt on INSERT; - ignore $timestamp_fmt_mysql if SQL database driver (DBD) is not 'mysql'; - perl taint workaround in lookup_sql() where SQL select clause could become tainted; problem reported by Christer Borang; - fixed amavisd.conf-default which stated incorrect default values of keys 'ins_rcp' and 'ins_quar' in %sql_clause; reported by Glenn Sieb; - limit reported boost score to three decimal places; long fractions observed by Gary V; - Postfix since version 20060610 uses xtext-encoded (rfc3461) strings in XCLIENT and XFORWARD attribute values, previous versions used plain text with neutered special characters - amavisd-new now xtext-decodes value if it looks xtext encoded, and encodes it on sending; the change could affect exotic host names (e.g. with a plus in host name) from broken mailers or DNS; thanks to Ralf Hildebrandt for pointing out the recent change in Postfix; - improve regular expressions in the $map_full_type_to_short_type_re list to cope better with different versions of the file(1) utility regarding recognition of various MS executables; based on a problem report by Misha; - use stricter suggested regular expression in amavisd.conf for matching CLSID (Class ID extension); previous expression was loose and too easily matched file names with braces in the name; suggested by Martin Schuster through Debian bug tracking; - zoo decoder interface routine (do_zoo) can now use utility unzoo(1) or the traditional zoo(1); the unzoo(1) recognizes some additional parameters which makes it more resilient (but still not watertight) against some attempts to hide archive contents or to extract members to unexpected locations, but unfortunately does not recognize all zoo compression schemes ("error, LZD not yet implemented"), and the relative modes "-j ./" or "-j X" do not protect against all malicious cases - so it is a mixed blessing. The way amavisd calls zoo(1) (piping members to stdout, which can be slow) avoids some of the security problems with zoo (writing to arbitrary directories), which were probably the main reason for ClamAV project deciding to switch to unzoo(1); - zoo sucks, unzoo (v4.4) sucks more: considered, but decided against changing zoo entry in @decoders to ['unzoo','zoo'] in amavisd.conf, as was suggested by Gábor Kövesdán. It would not necessarily be an improvement (see previous item, misses extracting members from my test cases), so feel free to choose between the two poor choices, I still prefer zoo(1), partly also because it covers cases which clamd decoding misses; - kill external zoo or unzoo decoder if running for too long; - internal: saving recipient addresses to SQL table maddr is now done earlier to make information available to pen pals code; - explicitly test if SQL 'prepare' silently fails to return a statement handle, just in case; - adjusted list of pre-loaded SA modules to cater for SA 3.1.3; NEW FEATURES: - new feature: added command line options which override some configuration settings from a config file (an option to override pid_file suggested by Paul Murphy and Gábor Kövesdán): -d log_level ... overrides $log_level -m max_servers ... overrides $max_servers -L lock_file ... overrides $lock_file (Net::Server serialization) -P pid_file ... overrides $pid_file -H home_dir ... overrides $MYHOME directory -Q quarantine_dir ... overrides $QUARANTINEDIR directory, empty disables -T tempbase_dir ... overrides $TEMPBASE directory -S helpers_home_dir ... overrides $helpers_home directory (SA workplace) -D db_home_dir ... overrides $db_home, empty arg turns off $enable_db -R chroot_dir ... overrides $daemon_chroot_dir, empty avoids chroot -p listen_port_or_socket ... overrides $inet_socket_port as well as $unix_socketname, argument may be a decimal TCP port number, or an absolute path name of a Unix socket; may be specified multiple times: daemon can listen on multiple inet sockets and/or multiple Unix sockets; example: -p 10024 -p 9998 -p /var/amavis/amavisd.sock -V ... shows version and exits -h ... shows version and command line options, then exits For completeness, here are remaining options, unchanged from previous versions: -u user ... overrides $daemon_user -g group ... overrides $daemon_group -c config_file ... config file name, may be specified multiple times - new feature: "pen pals soft-whitelisting" lowers spam score of received replies (or followup correspondence) to a message previously sent by a local user to this address; Pre-requisites: * both the outgoing and the incoming mail must pass through amavisd (although outgoing mail may have checks disabled or made more permissive if desired); * SQL logging must be enabled (@storage_sql_dsn) and records should be kept for at least several days (some statistics (2006-11 update): 90% of replied mail (or followups) is sent within 2 weeks since previous correspondence, 40% within 24 hours, 20% within 3 hours, 10% within 30 minutes, 5% within 12 minutes); * @mynetworks and @local_domains_maps must reflect reality, allowing amavisd to distinguish between outgoing, incoming and internal-to-internal mail; * the information about client IP address must be available to amavisd, i.e. Postfix XFORWARD protocol extension must be enabled, or AM.PDP+milter; * configuration variable $penpals_bonus_score must be set to a positive value (such as 1.0, increase to perhaps 5 or 8 after seeing that it works), zero disables the feature and is a default; * $sql_clause{'sel_penpals'} must contain a SELECT clause (which by default it does, unless overridden by an old assignment to %sql_clause in amavisd.conf); * sender/recipient address pair must exactly match recipient/sender pair of previous correspondence (except for allowed case-changes in domain part), which means that care must be taken when canonical and/or virtual mapping is performed by MTA (such as mapping between internal and external address forms) - if external address forms of local addresses are to be seen by a content filter then canonical mapping (int->ext) must be done *before* filtering and virtual mapping (ext->int) *after*; alternatively, if internal address forms are to be seen by a content filter, then canonical mapping should be done after filtering, and virtual mapping before; see README.postfix, section "TO DO 'VIRTUAL ALIAS' MAPPING AND OTHER POSTFIX CLEANUP PROCESSING BEFORE OR AFTER CONTENT FILTERING?" (P.S. later renamed to 'Advanced Postfix and amavisd-new configuration'); How it works: * SQL logging stores records about all mail messages processed by amavisd, their sender, recipients, delivery status, mail contents type (no changes there, this feature was introduced with amavisd-new-2.3.0); for the purpose of pen pals scheme only records with local-domain senders matter; * when a message is received, an SQL lookup against an SQL logging database is performed, looking for previous messages sent in reverse direction, i.e. from a local user (which is now a recipient of the current mail) to the address that is now the sender of the message being processed; A SELECT clause in $sql_clause{'sel_penpals'} is used, which by default only considers records of previous messages that were actually delivered (not rejected, discarded or bounced), and were not infected. SQL lookup returns a timestamp of the most recent such message (if any), the difference (in seconds) between the current time and the timestamp is an 'age' as used in the following formula; * an exponential decay formula calculates score points to be deducted from the SA score: weight = 1 / 2^(age/penpals_halflife) score_boost = -penpals_bonus_score * weight i.e. penpals_bonus_score is multiplied by 1, 1/2, 1/4, 1/8, 1/16, ... at age 0, 1*halflife, 2*halflife, 3*halflife, 4*halflife ... weight is a continuous function of age (actually, in steps of one second); * main configuration variables, members of policy banks: $penpals_bonus_score ... a maximal (positive) score value by which spam score is lowered when sender is known to have previously received mail from our local user from this mail system. Zero or undef disables pen pals lookups, and is a default. $penpals_halflife ... exponential decay time constant in seconds, defaults to 7 days; pen pal bonus is halved for each halflife period since the last mail sent by a local user to the current message's sender; * auxiliary configuration variables, global settings: $penpals_threshold_low ... SA score below which pen pals lookups are not performed to save time, defaults to 1.0; undef lets the threshold be ignored (useful for testing and statistics gathering); $penpals_threshold_high ... when (SA_score - $penpals_bonus_score > $penpals_threshold_high) pen pals lookup will not be performed to save time, as it could not influence blocking of spam even at maximal penpals bonus (age=0); usual choice for value would be kill level or tag2 level, or other reasonably high value; undef lets the threshold be ignored and is a default (useful for testing and statistics gathering); Caveats / notes / exceptions with "pen pals soft-whitelisting": * pen pals soft-whitelisting aids incoming mail, and internal-to-internal mail, but has no effect on outgoing mail; * if SQL logging was not used so far and you are considering enabling it for a busy site, you would appreciate PostgreSQL 8.1 compared to MySQL, as purging old records seems to be *much* faster than in MySQL 4.1, which could lock down mail processing for an hour or more during a weekly (or daily) purge, as opposed to minutes or seconds; * infected messages are exempted from pen pals checks; * mail with (unadjusted) SA score below $penpals_threshold_low (1 by default) is exempted from pen pals check to save time and lighten the load on SQL; similarly for high score spam which would not have a chance of being 'saved' even by a maximal pen pals bonus score; * non-delivery notifications have null sender address, so can not match previous correspondence and can not receive a pen pal bonus; * unauthenticated sender address matching local domains but coming from outside is not trusted and is exempted from pen pals checks; * messages from a local user to self are exempted from pen pals check; * outgoing messages (i.e. to non-local recipients) are exempted from pen pals checks to save some time and simplify reasoning (which reverse mail transaction to trust?); assuming that local users rarely send spammy mail, outgoing mail would rarely need help from pen pals checks; * messages received from mailing list typically use ML bounce or admin address (possibly VERPed) as the sending address, so they would not be considered replies to postings to a mailing list from a local user (to be addressed in future version 2.5.0); * underlying assumption is that a local-domains sender address in mail coming from inside can be trusted not to be faked; if this is not the case, an internal user cooperating with a spammer can widen spam tolerance for another internal user (but it probably does not pay off, too much trouble for too little effect); * if a spammer knows or can guess that a local user is frequently sending mail to some address (e.g. a mailing list unprotected by DKIM or SPF), he can gain few bonus score points by using such sending address in his spam; * there may be multiple MTA+amavisd servers, but all must use the same logging SQL database; * forwarding is compatible with the pen pals scheme; * a forwarding scheme like SRS (with SPF), where envelope sender address is replaced by a forwarding mailbox address is counterproductive; for example: a local user may also have an external mailbox at some remote provider with poor spam protection; forwarding from the remote to a local mailbox is set up and a forwarding MTA misguidedly substitutes original sender address with a mailbox address; spam reaching remote mailbox is forwarded to a local site with a sender address rewritten, making it look like it is coming directly from a user's remote mailbox, and inappropriately benefiting from pen pals bonus of user's previous correspondence with his remote mailbox; Testing: * set $penpals_bonus_score initially to a low value such as 1 to avoid surprises; * set $penpals_threshold_low and $penpals_threshold_high to undef to perform pen pals lookups regardless of the score; * at log level 2 (or higher) search the log for a string "penpals: " (only shows on incoming mail sent by a non-local sender); the log also shows mail_id of the referenced message (previous communication), and Subject header fields of previous and current message; Based on a feature request by Aaron P. Martinez, thanks to Gary V for suggestions and prompting and to Michael Scheidell and Richard Bishop for feedback. --------------------------------------------------------------------------- May 8, 2006 amavisd-new-2.4.1 release notes INCOMPATIBLE CHANGE WITH 2.4.0: - notification templates incompatibility with 2.4.0 (but not with versions 2.3.3 or older): major contents category numbers are renumbered due to a newly inserted category CC_SPAMMY; it affects the use of macro ccat_maj in templates (one field added), and only affect users which provide non-default templates based on 2.4.0 templates; older templates (2.3.3 or earlier) are unaffected as they do not use macro ccat_maj; OTHER CHANGES: - revert a change introduced with 2.4.0, which was adding address extensions at CC_SPAM, i.e. when score exceeds kill level. Previously (2.3.3) address extensions were inserted at tag2 level. Implemented by a new mechanism: a new major contents category CC_SPAMMY is inserted just below the CC_SPAM, where CC_SPAMMY is controlled by tag2_level and CC_SPAM continues to be selected at kill_level. Also spam defanging (if enabled) is now activated at CC_SPAMMY and no longer at CC_SPAM (which was on a TODO list for some time); undesired change in 2.4.0 reported and changes tested by Mario Liehr; - fixed old nuisance bug (probably present since 2.3.0) when an external decoder program for self-extracting archives (rar/unrar, lha, arj/unarj) is defined but the program does not exist, which resulted in logged non-fatal errors like: run_command: failed to exec SCALAR(0x8598550) lq ... run_command: failed to exec REF(0x85985c8) v -c- -p- -av- -idcdp -- ... run_command: failed to exec ARRAY(0x89e5f0c) l ... No such file or directory reported by Martin Baertl, Maurizio Marini, boka, and Donald Teed, investigated by Gary V; - bug fix in a Courier setup: add a missing reset of per-recipient data to prevent previous message check affecting the next one performed by the same process; fix by Martin Orr, reported by Bowie Bailey; - the amavisd-new-courier.patch now requires Net::Server version 0.90 or later (preferably 0.93 or later); to use older version of Net::Server please apply the older amavisd-new-courier-old.patch and follow README.courier-old - both will go away with next version of amavisd-new; - updated amavisd-new-qmqpqq.patch patch (qmail interface) to work with the current code, by Martin Solciansky, testing by Nicklas Bondesson; - fix error handling when a problem occurs during temporary directory cleanup; - when defanging mail make a 'Subject' header field be editable by header edits, so that Subject tags like ***UNCHECKED*** can still apply; - modify unquote_rfc2821_local so that it appends an '@' as a domain name only if localpart contains '@', so that read_array() can still be used to read a list of networks in CIDR notation; a change in 2.4.0 to properly handle addresses like "aaa@bbb" (local part in quotes, no domain) made read_array unsuitable for reading list of networks; pointed out by Petr Vokac; - add another round of local($1) declarations as a workaround for already familiar Perl taint bugs, popping up again on some Perl installations; reported by Jaap Struyk; reported symptoms were: Insecure dependency in chown while running with -T switch at /usr/lib/perl5/site_perl/5.8.7/Net/Server.pm line 488 Insecure dependency in eval while running with -T switch at /usr/lib/perl5/site_perl/5.8.7/Mail/SpamAssassin/PluginHandler.pm line 91 - added config variables: @spam_subject_tag3_maps, @spam_tag3_level_maps (and $sa_tag3_level_deflt), which makes it possible to split contents category CC_SPAMMY into two sublevels (minor categories) and give each its own Subject tag text; the "CC_SPAMMY,0" contents category still corresponds to tag2 level, and "CC_SPAMMY,1" contents category corresponds to tag3 level (if defined). Only static maps are available (also members of policy banks), but no corresponding SQL and LDAP attributes are provided. Example: @spam_tag2_level_maps = (5.5); @spam_tag3_level_maps = (12); @spam_subject_tag2_maps = ('***LIKELY*SPAM*** '); @spam_subject_tag3_maps = ('***BLATANT*SPAM*** '); based on suggestion from Benedict White; - add LDAP attributes: amavisSpamSubjectTag, amavisSpamSubjectTag2, amavisSpamDsnCutoffLevel, amavisSpamQuarantineCutoffLevel to match equivalent SQL lookup fields; missing amavisSpamQuarantineCutoffLevel noticed by Paolo Cravero; - presence of LDAP attributes is now tested with 'defined', no longer as Perl booleans; - mail_via_bsmtp: storing mail in BSMTP format now saves DSN information, as permitted by RFC 2442; - apply the concept of separate timers $child_timeout and $smtpd_timeout as used in an SMTP session to AM.PDP and AM.CL protocols; - apply the concept of separate timers $child_timeout and $smtpd_timeout as used in an SMTP session to Courier patch; by Martin Orr; - new macros: remote_mta, smtp_response, remote_mta_smtp_response and score_boost available to log templates and notification templates; - enhanced regexp selector macro [~string|regexp|then|else], which can now capture parenthesized regexp subexpressions and make them available as %1, %2, ... %9 to 'then' and 'else' replacements; a copy of the first argument (a string) is available to replacements as %0; - extend the semantics of the regexp selector macro, which can now take more than one pair of regexp+then arguments, catering for a nested 'if then elseif then elseif then else' structure: [~string|regexp1|then1|regexp2|then2|...|regexpN|thenN|else] - enhanced iterator macro, which can now take a long macro name as its first argument, and imply a %x as iterator name; - make use of the new macro remote_mta_smtp_response and add it to a default $log_templ, so that a Postfix queue-id of a forwarded message shows up like 'queued_as: DCF2A17B9E4' in the main log entry, facilitating search for a related log entry in an MTA log. In case of a mail split, all the MTA responses would now be shown, e.g.: queued_as: F3DBD17B847/F3DBD17B847/F3DBD17B847 (customizable by the use of macros in $log_templ); - sophos_savi_internal (SAVI module): don't include errno ($!) in the error message, it may be misleading; reported by Matthias Ivers; - internal - programming style: use more predictable $b=1 instead of $b++ where variable $b is supposed to be a boolean and not a counter; --------------------------------------------------------------------------- April 3, 2006 amavisd-new-2.4.0 release notes The most important changes since 2.3.3 at a glance: Delivery status notifications (DSN) are now supported, both as an SMTP protocol extension and in notifications. Header fields like X-Amavis and X-Spam are now prepended to mail header for DomainKeys compatibility. Configuration variables can be chosen based on mail contents category, which is now represented explicitly. A built-in macro expander is enhanced, providing new macros and call types. Added support for passive operating system fingerprinting with the use of p0f, supplying collected information as a header field to SpamAssassin. Provide compatibility with Net::Server 0.91 and later. INCOMPATIBLE CHANGES SINCE 2.3.3: - incompatible change when logging or quarantining to SQL: added field 'quar_loc' to table 'msgs' to facilitate quarantine release, and added FOREIGN KEY constraint for data consistency and simplified purging; see below for a simple database modification; - inserted header fields like X-Amavis-* and X-Spam-* are now _prepended_ to mail header instead of being appended, and occupy position just above the inserted Received header field; this pairing with Received makes it easier to identify which MTA/content filter inserted them, makes it consistent with position of Resent-* header fields as required by RFC 2822, and avoids the possibility of breaking DomainKeys, DKIM, and similar mail signing schemes. SpamAssassin implemented the same change with 3.1.0. To achieve former behaviour, specify: $append_header_fields_to_bottom=1; - trailing whitespace is no longer trimmed by default from SQL fields, from LDAP attribute values and from associative array righthand-sides (hash values) as read by read_hash(); see below if trimming is really still needed; - SMTP server side: no longer allow e-mail address without enclosing angle brackets in MAIL FROM and RCPT TO smtp commands; such syntax is illegal according to RFC 2821 and RFC 821, no compliant MTA is using it, so the change should not effect anyone (except perhaps sloppy testers); - changed defaults for banned & bad header administrator address to: $banned_admin = undef; $bad_header_admin = undef; @banned_admin_maps = (\$banned_admin, \%virus_admin, \$virus_admin); @bad_header_admin_maps = (\$bad_header_admin); In other words, if $banned_admin is left at a default value (undefined), banned admin falls back to %virus_admin or $virus_admin (a note from the future: no longer true since 2.8.0, see its release notes). If $bad_header_admin is left at a default value (undefined), bad header admin has no default, admin notifications for bad headers are not sent; MAJOR NEW ENHANCEMENTS: - support for DSN (RFC 3461) in the SMTP protocol (parameters NOTIFY and ORCPT in ESMTP RCPT commands, parameters RET and ENVID in ESMTP MAIL command), with corresponding updates to Delivery Status Notifications (RFC 3462, RFC 3464); (about a missing support for option ORCPT in Net::SMTP please see http://rt.cpan.org/Public/Bug/Display.html?id=18456 ); - represent mail contents category more explicitly internally, and provide new configuration variables: %final_destiny_by_ccat %lovers_maps_by_ccat %defang_by_ccat %quarantine_method_by_ccat %quarantine_to_maps_by_ccat %notify_admin_templ_by_ccat %notify_recips_templ_by_ccat %notify_sender_templ_by_ccat %warnsender_by_ccat %hdrfrom_notify_admin_by_ccat %mailfrom_notify_admin_by_ccat %hdrfrom_notify_recip_by_ccat %mailfrom_notify_recip_by_ccat %hdrfrom_notify_sender_by_ccat %admin_maps_by_ccat %dsn_bcc_by_ccat %warnrecip_maps_by_ccat %addr_extension_maps_by_ccat gradually phasing out separate configuration variables for each category; the change is fully backward compatible, existing variables are referenced through default values of the new variables, and no longer used directly; The chain of lookups adhere to the following evaluation sequence for settings with an associated *_by_ccat mechanism: * policy bank chooses a *_by_ccat associative array (by TCP port or by client's IP address (MYNETS)); * the most relevant contents type of the message chooses an entry in a _by_ccat associative array; the entry can be a final settings value, or a ref to an array of by-recipient lookup tables (*_maps); mostly for compatibility reasons an entry can also be a ref to CODE, which allows for delayed evaluation through legacy *_maps settings (which may again be members of policy banks); * the chosen list of lookup tables is queried based on recipient address, producing a final setting; Note that currently only settings which are applicable _after_ the mail contents type has already been determined, have their associated _by_ccat associative array. Settings like @bypass_spam_checks_maps which need to be evaluated _before_ mail contents is assessed, do not have their associated _by_ccat variable; - added ability to explicitly kill externally running decoder process or a command-line virus scanner process if running for too long; - enhanced built-in macro expander now allows long macro names (previously limited to one character), neutral and active macro calls, dynamically defining macros, new regexp matching built-in macro, more robust and explicit bookkeeping of quoting levels, as well as speedups achieved by pre-tokenization; details in see README.customize; - improved wrapping of inserted header fields, fields in DSN, and in generated text sections of the new notification templates; - improved text of notification templates, taking advantage of new macros; - compatible with Net::Server 0.90, 0.91, 0.92 and 0.93 by providing workarounds; thanks to Paul Seamons, the author of Net::Server, for his cooperation, the 0.93 solves problems introduced by a change in 0.91 (but it remains incompatible with version of amavisd-new 2.3.3 and older); - experimental support for passive operating system fingerprinting with the use of externally running utility p0f, supplying collected information as a header field to SpamAssassin, making possible to add rules to score SMTP client hosts based on educated guess about their operating system type and IP distance; see below for details; - make variable $myhostname a dynamic variable, member of policy banks, likewise for syslog parameters facility, priority and ident; details below; - added config options to enable quarantining (archiving) of clean mail; - lots of cleanups and generalizations in the code; SECURITY: - fix insufficient sender address sanitation when storing quarantined or forwarded files as BSMTP files _and_ having a %s in the corresponding *_method template; potential security vulnerability (with limited scope) in versions of amavisd-new 2.3.1, 2.3.2 and 2.3.3 discovered by Thomas Jarosch; - recognize result "ms-windows metafile" (or "ms-windows metafont") from a file(1) utility and provide short type 'wmf' for it; added two example rules to amavisd.conf (and amavisd.conf-sample) to block files containing Windows Metafiles, based on US-CERT Alert TA05-362A; OTHER CHANGES: - incompatible change when logging or quarantining to SQL is enabled (as mentioned above, here is a more detailed description of the change): * added column 'quar_loc' to table 'msgs' to store quarantine file name (the same string as in macro %q, normally seen in the main log entry); based on input from Andrew A. Neuschwander, Brian Wong and Craig Herring; * add constraint FOREIGN KEY ... ON DELETE CASCADE to keep database consistent (free of orphaned records) and simplify maintenance deletions and possibly speed them up; suggested by Brian Wong; The following clause must be executed for upgrading pre-2.4.0 amavisd-new SQL schema to the 2.4.0 schema: ALTER TABLE msgs ADD quar_loc varchar(255) DEFAULT ''; The following clause should preferably be executed to take advantage of the ON DELETE CASCADE: ALTER TABLE msgrcpt ADD FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE; ALTER TABLE quarantine ADD FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE; See updated suggested set of DELETE clauses at the end of README.sql. The following clause can optionally be used to create an index on field msgs.time_num to speed up deletions in MySQL; CREATE INDEX msgs_idx_time_num ON msgs (time_num); or if purging is based on field msgs.time_iso instead of msgs.time_num: CREATE INDEX msgs_idx_time_iso ON msgs (time_iso); (compatibility note with pre-releases of 2.4.0: there were added fields msgrcpt.time_num and quarantine.time_num in pre-release versions of 2.4.0, which are now dropped in favor of FOREIGN KEY constraint; these fields are no longer set by the program and should not be relied-on when purging records, they may be removed from tables); - solve compatibility issues brought up by changes in file descriptors usage as introduced with Net::Server version 0.91; thanks to Ralph Seichter, Matt Jackson, Jim Knuth and Paul Seamons (the author of Net::Server) for help; - bug fix with LDAP lookups: if an LDAP connection to the server drops (i.e. after being idle for some time) amavisd is unable to reconnect; a symptom in the log is: 'do_search: failed again'; a fix by Petr Vokac, and later independently fixed by Matteo Brancaleoni and Mike Hall; problem also reported by Paolo Cravero; - bug fix with LDAP amavisBannedRuleNames lookups, failing to looking up a set of banned rules names and referencing them in the users lookup table, like in other lookup tables. The LDAP lookup was returning an array reference for the 'amavisBannedRuleNames' attribute since it was a list (multivalued) and the reference wasn't being dereferenced down the line. The fix is to make the attribute single valued, the value can be a comma-separated list of names. This brings it in-line with SQL lookups which also uses a single field of comma-separated names. The included LDAP.schema and documentation files are fixed accordingly. Also moved the LDAP stuff out of README.lookups into its own README.ldap and updated it accordingly for the banned rules stuff. Fixed by Michael Hall; problem reported by Jérôme Schell, Aury Fink Filho and Brian Wong; thanks also to Jack Stewart and Willi Gruber; - bug fix: properly disconnect SMTP session with 421 response if it times out; watchdog timer needs to be nudged during DATA transfer as well; pointed out by Victor Duchovni; - introduced new configuration variable $smtpd_timeout (default 8*60 seconds) which controls the amount of time we are willing to wait for slow/idle client during incoming SMTP session before disconnecting a session. Previously the $child_timeout was covering complete elapsed time, both our processing and waiting for client, now the $child_timeout only still limits our processing, and $smtpd_timeout only limits waiting time. With Postfix after-queue setup the $smtpd_timeout should be higher than Postfix setting max_idle (default 100s). Some other setups (like a pre-queue setup) may demand substantially higher $smtpd_timeout values; inflexibility pointed out by Martin Schmitt; - added ability to kill externally running decoder process or a command-line virus scanner process if running for too long; currently implemented for all command-line virus scanners and for more common and/or more troublesome external decoders: do_unrar, do_unarj, do_uncompress, do_pax_cpio, do_lha, and partly for do_arc, do_zoo; allowed time is calculated as 2/3 of the remaining time (initially at $child_timeout), but at least 10 seconds; - use the same timeout calculation as above for calls to SA, taking $sa_timeout instead if that value is bigger than the calculated time, thus making $sa_timeout pretty much redundant; - let do_pax_cpio recognize (and ignore) a single character in place of a date in more exotic cases of a pax listing; reported by Ralf Hildebrandt; - standards compliance: recognize (and discard) source route in mail address as required by rfc2821; - no longer bother to convert addresses like <""@yahoo.com> to <@yahoo.com>, both forms are invalid anyway, and recent versions of Postfix treat them the same. It is probably a good idea to set strict_rfc821_envelopes=yes in main.cf to reject such non-replyable sender addresses straight away, otherwise we end up processing such mail with inability to bounce it when needed, effectively losing it; - make address with '@' in the localpart but without a domain (such as <"aaa@bbb"> ) distinguishable from <aaa@bbb> by appending an empty domain ('@' only) to the internal (unquoted) address form; also, we used to strip off empty domain on rfc2821-quoting, but this leads Postfix to interpret an address with an '@' in the local part like <"hhh@example.net"> as <hhh@example.net> (subject to the 'resolve_dequoted_address' Postfix setting), which is not what the sender requested (perhaps unintentionally) so we no longer do that. Both measures together, along with the new address parsing code, solve the inconsistency problem reported by Les Ault; - fix string_to_mime_entity() to properly split header from body even in some corner cases (empty header or empty body); as a bonus a tiny speedup in template message splitting is gained; - mail header of a 'defanged' message should not contain broken original headers (with illegal characters or whitespace lines); now sanitize such header fields; reported by Ivers Matthias; - do not fix illegal all-whitespace continuation header lines when writing to quarantine (or when submitting notifications) in order to preserve the original bad header; only fix the header when such message is forwarded or released from a quarantine; masking problem brought up by Michael Scheidell; - when quarantining in a Unix-style mbox format, replace null return path in a delimiting 'From ' line with a string 'MAILER-DAEMON', like Postfix and sendmail local delivery agents do, otherwise some mbox-reading clients do not recognize the line as a message delimiter; - when quarantining to a mbox file, ">"-escape all /^From / lines, not just the ones following a blank line; this is more universal and does not break on more sloppy mail readers (thunderbird, kmail, mutt and pine); MUAs like elm and mail(1) (the later usually comes with the OS) are more robust, treating as a message delimiter only /^From / lines following a blank line, these did not mind the more compact approach used by amavisd-new so far; - new config variable $syslog_ident makes it possible to configure syslog ident string, its value defaults to 'amavis'; suggested by Andrzej Kukula; - instead of the old config variable $SYSLOG_LEVEL (default value 'mail.debug') there are now two config variables $syslog_facility and $syslog_priority, defaulting for compatibility to the before-the-dot and after-the-dot substrings of the variable $SYSLOG_LEVEL. The variable $SYSLOG_LEVEL still exists, can still be used, but is considered obsolete; - make variables $syslog_ident, $syslog_facility and $syslog_priority dynamic variables, members of policy banks. This makes it possible for each policy bank to use its own specific syslog settings, for example to log to a different file (by using a different syslog facility like 'LOCAL3' for mail originating from inside (policy 'MYNETS')), or to change syslog ident to 'outgoing-amavis' for certain policy bank, or to rise syslog priority for releases from a quarantine. Note that switching syslog_ident or syslog_facility is done only when necessary, and involves closing and reopening syslog connection, which involves some (quite small) cost for each change. Dynamically changing syslog_priority however is for free; - make variable $myhostname a dynamic variable, member of policy banks. This makes it possible for each policy bank to exhibit a different identity in notifications, in inserted header fields and in certain log entries. A quick overview of where myhostname may appear: From, Resent-From, Resent-Sender, Message-ID, Resent-Message-ID, X-Amavis-Modified, Reporting-MTA, macro %h, SQL logging in field msgs.host, log identity, and can be incorporated in variables $smtpd_greeting_banner, $smtpd_quit_banner and $hdrfrom_notify_* by specifying a substring ${myhostname}, which will be replaced by a current value of $myhostname just before use; - new configuration variables for finer control on propagation of DSN options: $propagate_dsn_if_possible and $terminate_dsn_on_notify_success (both part of policy banks). One or the other may be used to hide internal mail routing from outsiders if desired. Although Postfix can be configured to selectively announce or not announce DSN smtp extension based on client address (e.g. announcing it for internal clients but not for the rest of the world) letting DSN options reach a content filter may be desirable because a content filter can provide a more informative delivery status notification, and perhaps more importantly, it can suppress sending a DSN when it suspects the sender address is faked (viruses, high score spam). This means the front-end Postfix smtpd service should not be terminating DSN chain, but unfortunately the second instance of smtpd service at port 10025 can no longer distinguish between internal and external clients, because it only sees an IP address of a content filter. One solution is to turn on the $propagate_dsn_if_possible within policy bank MYNETS, and turn it off globally, e.g.: $propagate_dsn_if_possible = 0; $policy_bank{'MYNETS'} = { propagate_dsn_if_possible => 1, ... }; Turning off $propagate_dsn_if_possible is exactly equivalent to the case where MTA on the return port (10025) does not announce support for DSN extension to the SMTP protocol. The only difference is that the amavisd-new setting can be controlled more selectively by a policy bank. Turning on $terminate_dsn_on_notify_success is similar, but more refined. It tells amavisd not to pass NOTIFY=SUCCESS option when submitting checked mail back to MTA, which lets amavisd behave as if talking to a non-DSN compliant server, so that DSN success notification will be generated by amavisd itself (unless suppressed for other reasons). This is similar to $propagate_dsn_if_possible=0, the difference is that other DSN options (if any) *will* be passed to MTA, so options like NOTIFY=NEVER or RET, ENVID or ORCPT will not be lost. Such behaviour is not strictly by the book (RFC 3461), but is still in its spirit :) Here is the most useful setting: $terminate_dsn_on_notify_success = 1; $policy_bank{'MYNETS'} = { terminate_dsn_on_notify_success => 0 }; or if you prefer this syntax, changing only one key in an existing p.bank: $policy_bank{'MYNETS'}{terminate_dsn_on_notify_success} = 0; - new configuration variable $dsn_bcc (also part of policy banks) and a corresponding %dsn_bcc_by_ccat (by-contents-category settings) allows to specify an additional e-mail address to receive a copy (blind-CC) of a delivery status notification (DSN) that is being sent to the envelope sender address; it allows administrator to monitor what DSN messages (and how many) are being sent out by amavisd-new; Possible uses: $dsn_bcc = 'admin@example.com'; # applies to all content categories or more selectively (overrides $dsn_bcc): %dsn_bcc_by_ccat = ( CC_VIRUS, 'admin+infected@example.com', CC_BANNED, 'admin@example.com', CC_BADH, 'admin@example.com', CC_SPAM, undef, ); or perhaps: %dsn_bcc_by_ccat = ( CC_SPAM, undef, CC_CATCHALL, 'admin@example.com', ); Note the use of comma as a delimiter (instead of '=>'), as _values_ of CC_* constants must be used as hash keys, not constant _names_ as strings; (the Perl '=>' operator implicitly quotes its left operand); - passive operating-system fingerprinting (p0f) support lets SA gain information about SMTP client's operating system and estimated IP distance, and can reduce the number of bounces: * preconditions are: $os_fingerprint_method must be configured, the p0f-analyzer.pl process must be running, and amavisd must be receiving client IP address information from MTA, which in a Postfix case means the XFORWARD protocol extension to SMTP must be enabled in the Postfix service feeding mail to amavisd, e.g. "-o smtp_send_xforward_command=yes", or in a sendmail/milter setup the more sophisticated AM.PDP protocol must be used; * find and install the p0f utility: http://lcamtuf.coredump.cx/p0f.shtml or in FreeBSD ports collection as 'net-mgmt/p0f'; * start a p0f process on the same host where MTA (MX) is running, making it listen only to incoming TCP sessions (to reduce its workload) to the IP address and TCP port (25) where MTA is accepting incoming mail from outside (it doesn't hurt to let it see other traffic too, it just isn't needed); after testing p0f alone and seeing that it works, you may start it up, feeding its output to program p0f-analyzer.pl that comes with amavisd-new package, e.g.: p0f -l 'tcp dst port 25' 2>&1 | p0f-analyzer.pl 2345 & on multi-homed boxes one may need to specify interface and IP address where MTA is listening, the filter syntax is the same as in tcpdump, e.g.: p0f -l -i em1 'dst host 192.0.2.66 and tcp dst port 25' 2>&1 \ | p0f-analyzer.pl 2345 & * the program p0f-analyzer.pl reads p0f reports on stdin, keeps a cache for a limited time (10 minutes, configurable) of data about incoming TCP sessions organized by remote IP address, and listens on UDP port 2345 (specified as its command line argument) for queries; only queries from allowed IP addresses are accepted and responded to, other queries are silently ignored - configure @inet_acl accordingly, defaults to 127.0.0.1; * adding the following line to amavisd.conf, matching the chosen port number to the one specified on the command line to the p0f-analyzer.pl: $os_fingerprint_method = 'p0f:127.0.0.1:2345'; makes amavisd send queries to p0f-analyzer.pl (on the supplied IP address and UDP port number) to collect information about remote SMTP client's OS; collected response is then supplied as a header field when SpamAssassin is invoked; query/response is very quick and imposes no burden on amavisd process nor does its extend its processing time. The $os_fingerprint_method setting is also a member of policy banks to make it more flexible to disable fingerprinting for mail from site's own SMTP clients, e.g: $policy_bank{'MYNETS'}{os_fingerprint_method} = undef; * one may now add scoring rules to SA local.cf file, e.g.: describe __L_P0F_EXISTS A header field X-Amavis-OS-Fingerprint does exist header __L_P0F_EXISTS exists:X-Amavis-OS-Fingerprint describe L_P0F_WXP Remote system is truly a Windows XP, not Windows 2000 header L_P0F_WXP X-Amavis-OS-Fingerprint =~ /\AWindows XP(?![^(]*\b2000 SP)/m score L_P0F_WXP 2.3 describe L_P0F_W Remote system is some Windows variant, except Win. XP header L_P0F_W X-Amavis-OS-Fingerprint =~ /\AWindows(?! XP)/m score L_P0F_W 1.3 describe L_P0F_UNKN P0f was unable to determine remote OS type header L_P0F_UNKN X-Amavis-OS-Fingerprint =~ /\AUNKNOWN/m score L_P0F_UNKN 0.8 describe L_P0F_Unix Remote system is running Unix, not Linux header L_P0F_Unix X-Amavis-OS-Fingerprint =~ /\A((Free|Open|Net)BSD|Solaris|HP-UX|Tru64|AIX)/m score L_P0F_Unix -1.0 describe L_P0F_Linux Remote system is running Linux header L_P0F_Linux X-Amavis-OS-Fingerprint =~ /\ALinux/m score L_P0F_Linux -0.1 It is also possible to add score based on estimated IP distance, for example to slightly favorize nearer hosts (this is probably good for Europe or academic/university networks, and possibly less useful elsewhere): header L_P0F_D1234 X-Amavis-OS-Fingerprint =~ /\bdistance [1-4](?![0-9])/m header L_P0F_D5 X-Amavis-OS-Fingerprint =~ /\bdistance 5(?![0-9])/m header L_P0F_D6 X-Amavis-OS-Fingerprint =~ /\bdistance 6(?![0-9])/m header L_P0F_D7 X-Amavis-OS-Fingerprint =~ /\bdistance 7(?![0-9])/m header L_P0F_D8 X-Amavis-OS-Fingerprint =~ /\bdistance 8(?![0-9])/m header L_P0F_D9 X-Amavis-OS-Fingerprint =~ /\bdistance 9(?![0-9])/m header L_P0F_D10 X-Amavis-OS-Fingerprint =~ /\bdistance 10(?![0-9])/m header L_P0F_D11 X-Amavis-OS-Fingerprint =~ /\bdistance 11(?![0-9])/m score L_P0F_D1234 -0.5 score L_P0F_D5 -0.5 score L_P0F_D6 -0.5 score L_P0F_D7 -0.5 score L_P0F_D8 -0.5 score L_P0F_D9 -0.4 score L_P0F_D10 -0.3 score L_P0F_D11 -0.3 It can be useful to tame false positives from some other rules, e.g. # tame a Botnet plugin, reducing its false positives score BOTNET 0.1 meta BOTNET_W !DKIM_VALID && (L_P0F_WXP || L_P0F_W) && BOTNET score BOTNET_W 2.8 meta BOTNET_WU !DKIM_VALID && L_P0F_UNKN && BOTNET score BOTNET_WU 2.0 meta BOTNET_OTHER !BOTNET_W && !BOTNET_WU && BOTNET score BOTNET_OTHER 0.5 * make sure the @mynetworks is configured correctly, otherwise you will be inappropriately penalizing mail from internal hosts running Windows! Other methods to turn off fingerprinting for our own SMTP client hosts is to put $os_fingerprint_method in policy banks, and/or to specify more selective packet filter on the p0f command line; * based on statistics, less than 0.7 % of mail coming from external Windows XP -based hosts is ham, yet 20 % of all spam is coming from external Windows XP hosts; amavisd-new suppresses bounces to external Windows XP hosts, reducing bounce pollution. The amavisd-agent utility now provides some additional statistics based on p0f information. Some statistics collected from our logs in February 2006: p0f OS guess ham : spam ----------------------------- Windows-XP 0.7 % : 99.3 % Windows-2000 5.8 % : 94.2 % UNKNOWN 16.5 % : 83.5 % Linux 58.8 % : 41.2 % Unix 80.3 % : 19.7 % (Unix+Linux 66.5 % : 33.5 %) (ham: mail with score below 3, spam: score above 6) - new configuration variable $allow_fixing_improper_header_folding (also a member of policy banks) controls fixing improperly folded header fields made up entirely of whitespace (prohibited by RFC 2822) by removing all-whitespace continuation lines; defaults to true for backward compatibility; fixing such header fields is desirable and can protect poorly written mail readers, but may break DomainKeys/DKIM validation of messages with illegal header, so if this is of concern, one has a choice of turning it off (if local MUAs can't be fixed); - make config variable $insert_received_line a member of policy banks; - removed mail header (macro %H) from the default template of the plain text part of the virus or banned sender notifications; these headers are available in the third MIME part of the DSN, so having them twice was redundant; - new macros: date_unix_utc, date_iso8601_utc, date_iso8601_local, date_rfc2822_local, x-mailer, header_field, ccat_name, ccat_maj, ccat_min, wrap, lc, uc, substr, index, len, incr, decr and a couple of SpamAssassin lookalike macros - see README.customize for details; - new macro ccat_min expands to a minor contents category; this makes it possible for a notification template to distinguish (for example) between cases of bad message header: 1=bad MIME, 2=8-bit char, 3=NUL/CR, 4=empty, 5=long, 6=syntax error; See also README.customize for the list of currently available macros; see %ccat_display_names for minor ccat numbers currently in use, feel free to add new ones; - edited text of default notification templates to make them tidier and more informative; also make a good use of new macros; older templates are still compatible with this version of amavisd-new; - new global variable $timestamp_fmt_mysql, defaults to false; specific to MySQL, when logging to SQL is enabled and field msgs.time_iso is declared as TIMESTAMP instead of char(16); setting $timestamp_fmt_mysql=1 changes the timestamp format written to field msgs.time_iso to avoid ISO 8601 standard delimiter 'T' and trailing timezone 'Z', which MySQL does not like. Don't turn on $timestamp_fmt_mysql when using PostgreSQL! Should not turn on $timestamp_fmt_mysql with MySQL unless msgs.time_iso is of type TIMESTAMP. - new config variable $trim_trailing_space_in_lookup_result_fields controls trimming of trailing whitespace from SQL fields, from LDAP attribute values and associative array righthand-sides (hash values) as read by read_hash(); disabled by default; turn it on for compatibility with pre-2.4.0 versions. Note that trailing spaces may still be trimmed by SQL itself (contrary to SQL-99 specification). Trailing spaces in MySQL version up to and including 4.1 are removed from values when stored in a VARCHAR column; this also means that spaces are absent from retrieved values. Starting with MySQL V5.1, trailing spaces are retained when values are stored and retrieved. Investigated by Gary V; - treat exit status 2 as a warning when returned by bzip2, gzip and other decompressors handled by do_uncompress; problem with decoding of corrupted bzip2 file reported by Kim Leandersson; - when determining file short type, match "Microsoft Cabinet file" result from a file(1) command case-insensitively; reported and patch provided by ap at zip com au via Debian bug tracking system; also recognize "Microsoft Cabinet archive data" as .cab; - do_unzip: set attribute 'U' (undecodable) if zip archive fails to be decoded; based on a patch by Oliver Geisen; -- well, on a second thought, perhaps not, this line is now commented out; it flagged too many bounces containing chopped-off ZIP attachments as ***UNCHECKED***; - added a minimalistic decoder interface routine to call a command line unpacker from stuffit.com to decode Macintosh StuffIt archives. Not tested extensively, program source is not available for inspection, use at YOUR OWN RISK (and the risk is non-negligible!). If using non-default assignment to @decoders, the following entry can be added to the list of decoders: ['sit',\&do_unstuff,'unstuff'], thanks to Oliver Geisen for the suggestion; - keep X-Spam-Level bar empty if sender is whitelisted; - untaint recip_score_boost when writing a log report entry to SQL; 'Insecure dependency in parameter 7' could have been reported when SQL-based score_sender lookup table is used; reported by Jim Knuth; - treat undefined spam level as 0 for the purpose of comparing it to tag/tag2/kill levels, e.g. when spam scanning is skipped due to large message size; this now allows score_sender_maps to push non-checked messages over a tag2/kill limit by its score boost if desired; - header fields X-Spam-Status, X-Spam-Score, X-Spam-Level and X-Spam-Flag in a quarantined message now take into account also the score_sender boost and white- and blacklisting (using 'any' and 'max' to summarize in case of multiple recipients); - in passed and quarantined mail a header field X-Spam-Status now shows score as an explicit sum of SA score and a by-recipient score_sender boost (when the boost is nonzero); the X-Spam-Score header field still shows a sum of both as a single number so as not to confuse MUA filters which may operate on that header field; - insert X-Quarantine-ID header field into passed mail if a passed message was also quarantined (e.g. *_lover or final_*_destiny=D_PASS), suggested by Pavel Urban; - when folding is needed for long new or edited header fields, use TAB instead of a space on fold points; also: take into account the apparent display size of expanded TABs instead of the actual character count, to make header look prettier; use of TABs suggested by Debian community; - improved wrapping of inserted header fields and in generated text sections of notification templates; - automatic wrapping of long header lines from notification templates; - standards compliance: wrap Diagnostic-Code field in message/delivery-status section of delivery status notifications according to rfc3461 section 9.2; - added config options to enable quarantining (archiving) of clean mail; defaults settings are: $clean_quarantine_method = undef; $clean_quarantine_to = 'clean-quarantine'; quarantining clean messages is disabled by default; to enable: $clean_quarantine_method = 'local:clean-%m'; - reports "Blocked TEMPFAIL" instead of "Blocked CLEAN" in case of a temporary 4xx failure; - in generated MIME parts (notifications, defanging) replace suggested file names like 'message.txt', 'dsn_status.txt' and 'header.txt' with names without an extension, i.e. 'message', 'dsn_status' and 'header'; Reportedly Outlook Express 6.0 (but not Outlook) determines how to present a MIME part based on its name extension, instead of based on MIME type. Depriving it of name extensions makes it obey a MIME type. This solution has been successfully tested with Outlook (5 & 6), Outlook Express (2k, 2k3), Thunderbird, JavaMail and Squirrelmail. Investigation and suggestion by Ivers Matthias; - clamscan AV entry: change test for status 1 by a test for /:.*\sFOUND$/ to prevent system errors like failed load (e.g.: 'ld-elf.so.1: Shared object "libgmp.so.6" not found', which also produce exit status 1) from causing all mail to be treated as infected; suggested by Tomasz Kojm in response to my problem report; - add three AV entries for avast products (FreeBSD and Linux): 'avast! Antivirus daemon', 'Client/Server Version' and the command-line avastcmd; kindly provided by Frantisek Mensik, ALWIL Software; later refined based on testing and feedback from Bill Landry; - updated regexp in McAfee uvscan entry to cope with spaces in virus name; reported by, and output samples provided by Andreas Schulze; - updated bdc AV entry: newer versions of BitDefender don't use option --all any longer; thanks to Max Matslofva, Andreas, Gary V and Bill Landry; - updated NOD32 AV entry based on NOD32 documentation and advise from Willi Gruber about adding status code 3 to the list of success values; - better handle failed decoder attempts when checking an executable file for self-extracting archive (SFX), avoid decoder 'dry runs' where possible; - require minimal version 0.32 of Net::LDAP, a subroutine Net::LDAP::Util::escape_filter_value() is needed; reported by Harry Hoffman; - allow to specify option 'deref' in calls to Net::LDAP->search to control dereferencing of aliases to locate the base object for the search; the default remains 'find' as before. Use a key 'deref' in the default_ldap hash to specify a different value. The values are those documented in the Net::LDAP manpage in the search function under the deref section; (btw, dereferencing in LDAP is supported in both Postfix and Courier); a patch provided by John Allman; - added a timing report entry fwd-data-cmd for time it takes MTA to respond with a status to a DATA command; - read_hash: trim whitespace off the right-hand side data only if followed by a comment (#), otherwise keep the rhs as it is; - include a currently chosen dataset name (dsn) in the log entry when quarantining to SQL, to be able to retrieve a quarantined message from the correct SQL server; suggested by Cami Sardinha; - dwell in $tempdir (chdir to) instead of in $TEMPBASE most of the time; - edit_header() may now be called multiple times for the same header field to provide iterative header edits - provided mostly for completeness; - Courier code (the patch) now supports D_BOUNCE and D_DISCARD message destinies and adding and deleting recipients (due to differing per-recipient configuration or addr_extension_*_maps); it also supports DSN (RFC 3461); by Martin Orr; - prepend (!!) or (!) to log messages at levels below 0 to facilitate log parsing and make critical messages stand out; based on a patch by Henrique de Moraes Holschuh (Debian port maintainer); - silence logging of AM.PDP commands delivery_care_of, tempdir_removed_by and tempdir when using amavisd-milter; a patch by Petr Rehor; - silence Perl warnings about uninitialized values; - optimization: about 15% speedup in macro expansion due to pre-tokenization of template strings at startup (affects preparing main log entry and each notification message); - modified old amavis.c client program to return status 0 (success) even when LDA command-line arguments are not specified (i.e. when amavisd daemon is in charge of delivery), and amavisd daemon returns status 99 (=discard); simplified code in amavisd/check_amcl_policy() to take advantage of it; - internal/coding: a much needed and appreciated hefty patch from Martin Orr introduces class Amavisd::TempDir and collects there existing code from various places dealing with maintaining a temporary directory and files within; it also makes possible for the Courier interface code to use this module and avoid duplicating code; - internal/coding: the Amavis::In::Courier package has been tidied up and split into multiple methods, of a hopefully manageable length. Several new features in amavisd-new core code, which required changes to Amavis::In::Courier to take advantage of, have also been dealt with (in particular MYNETS and MYUSERS policy banks and SNMP counters); by Martin Orr; - require minimal version 1.43 of DBI, working last_insert_id is needed; (actually last_insert_id is no longer needed, but the requirement stays :) - internal/coding: when logging to SQL avoid a need for last_insert_id() by doing SELECT after INSERT when adding a new e-mail address to table maddr; this also avoids a rollback/retry when more than one process tries to insert the same new address into a database; - internal/coding: merge subroutines do_spam and do_virus into a single do_notify_and_quarantine, and use the same code to prepare spam-describing headers as in subroutine add_forwarding_header_edits_per_recip; - internal/coding: new $msginfo data object: contents_category, holding a reference to a sorted (descending order) array of entries, each one corresponding to one contents category under which a message can be classified, e.g. it can be both a CC_VIRUS and a CC_BANNED. Supporting subroutines are: add_contents_category, main_contents_category, is_in_contents_category, setting_by_contents_category; thanks for suggestions and feedback to Gérald Macinenti; The contents_category list is a sorted list of strings, each of the form "major" or "major,minor", where major and minor are numbers, representing major and minor category type. Sort order is descending by numeric values, major first, and subordered by a minor value. When an entry "major,minor" is added, an entry "major" is added automatically (minor implied to be 0). A string "major" means the same as "major,0". See CC_* constants for major category types. Minor category types semantics is specific to each major category, higher number represent more important finding than a lower number; - internal/coding: merge sections in sub check_mail dealing with different content types one-at-a-time into a single section 'decide_mail_destiny', taking advantage of the new information on contents category, improving consistency and simplifying code; - internal/coding: new $msginfo data objects: spam_level, spam_status, spam_report, autolearn_status, avoiding ugly global variables; - internal/coding: separate SpamAssassin-specific code from general anti-spam code - new module Amavis::SpamControl::SpamAssassin; based on suggestion and patch by Felix Schwarz; - internal/coding: use File::Spec::catfile to splice full file name from its components (in mail_to_local_mailbox, more needed); a patch by Felix Schwarz. Actually, on a second thought, comment that out and revert to previous code: there are so many other similar cases which were not generalized, that it makes no sense to generalize (through File::Spec) one percent of them (and drag-in yet another Perl module), and leave the rest hard-coded; - collect the most commonly needed header fields into an associative array $msginfo->orig_header_fields, removing the need to call ensure_mime_entity() in certain cases, saving on mime decoding when it is not really needed; - internal/coding: do_log now takes optional arguments, and if they are present, the message text is treated as a format string to snprintf. Take advantage of this in most calls to do_log. The message (format) argument should not be tainted (not enforced), but arguments may be. In the absence of additional arguments, do_log behaves as before; - updated README.postfix to explicitly override (just in case) two newer Postfix options: smtpd_data_restrictions and smtpd_end_of_data_restrictions, thanks to Noel Jones for the suggestion; - documentation: updated README.sendmail-dual - added custom rules to reject unknown users outright; provided by Matej Vela, thanks to Simone Marx; added a reference to the 'milter-ahead' project info; thanks to Adam Gibson; - documentation: fixed README.exim_v4 (don't let messages with null return path get through unchecked), by Igor D'Astolfo; - documentation: updated README.customize, describing new features of a built-in macro expander, and describing new macros; - documentation: updated README.sql, describing new SQL log purging recommendations, improved PostgreSQL instructions, and the (optional) use of data type TIMESTAMP in field msgs.time_iso; --------------------------------------------------------------------------- August 22, 2005 amavisd-new-2.3.3 release notes Version 2.3.3 is a maintenance release over 2.3.2. Besides fixing known problems and providing some optimizations, no new features were added. If using SpamAssassin older than 3.1, an upgrade of either SA to 3.1, or an upgrade of amavisd-new to 2.3.3 is recommended. - privacy: add a safety fuse / workaround around calls to SA to detect SA's failure (in SA versions before 3.1) to catch a failed exec() in a forked process, which could produce runaway process clones. See SA bug report #4370. An incident of a mail copy being delivered to unrelated recipient reported by Joel Nimety; - privacy: turn warning into a fatal error when a quarantine ID of a message requested for a quarantine release does not match the requested mail_id; - security: require minimal version 1.35 of Compress::Zlib to avoid vulnerability in the zlib compression library; - the dsn_cutoff_level should have been ignored if undefined according to documentation, but was not, causing DSN to be suppressed regardless of spam level; discovered by Gary V; - ensure the banned check is not performed if all recipients agree it is not needed, even in presence of $banned_namepath_re; undesired behaviour (not strictly incorrect) reported by Joel Nimety; - missing import of lookup_ip_acl in module Amavis::In::AMCL caused failure in sendmail milter setup when using the new AM.PDP protocol; reported by Mic And; - document and explicitly define handling of syntactically invalid IP address in lookup_ip_acl: it matches a zero-length-mask net, a constant lookup table, or a hash entry with an undef key, but no other entries in IP lookup tables; syntactically invalid IP addresses are now logged; - fix parsing of IPv6 address in $notify_method and $forward_method in case of dynamic destination override (the use of '*' in method fields); - check during startup that $myhostname is a fully qualified domain name (or 'localhost', if you must), and abort if it isn't, otherwise a non-FQDN can end up in places where RFC 2822 does not allow it; if uname(3) does not provide a FQDN, then an assignment to $myhostname must be done explicitly in amavisd.conf; - when quarantining to a single file in mbox format the 'From ...' line needs an English date, regardless of current locale; fixed by globally setting locale LC_TIME to "C"; - pass on the parameter BODY=8BITMIME on MAIL FROM when submitting to MTA when original message reception indicated it is needed (RFC 1652). Note that mail forwarding may now fail if the feeding MTA requests BODY=8BITMIME SMTP service extension (or just passes data with msb set), but the MTA on the output side does not allow the use of the BODY parameter in SMTP. In case of Postfix this may only happen when receiving service on port 10025 is misconfigured and does not announce ESMTP capability and support for the SMTP service extension 8BITMIME; - RFC 2554 requires auth_param to be xtext-encoded addr-spec (no angle brackets) or "<>", not the xtext-encoded addr-spec enclosed in angle brackets (when specifying submitter during authentication); fixed; - apply some sanity limit on collected bad-header samples to ensure that a grossly broken mail does not unnecessarily fill up memory; - when sending recipient warnings for viruses, banned files, or bad headers, recipient address must not be rfc2822-quoted twice; fixed; - fix interpretation of $defang_all to really imply all; previously it only affected clean messages; - in quarantined mail the reported spam score in X-Spam-Status header field now includes maximum of all by-recipient score boosts (less surprising when soft-whitelisting through @score_sender_maps is in use); suggested by Mike Cappella and Gary V; - when a policy delegation protocol attribute "request" is not "AM.PDP" (perhaps it is a Postfix policy delegation request) don't attempt to find and open a mail file; - do_ascii and do_unarj: set environment variable TMPDIR on a command line temporary directory option to "$tempdir/parts" instead of $TEMPBASE to minimize possible pollution of top level directory; - don't abort even if amavisd.conf returns undef as a final value, as long as there are no errors reading or interpreting it; - if during 'amavisd stop' or 'amavisd reload' the old running daemon does not go away for one minute after sending it a SIGTERM, use a bigger hammer and send it a SIGKILL; suggested by Sven Riedel; - extend LDAP lookups to allow multiple search attributes (multiple occurrences of %m in a query); a patch by Michael Hall (and a similar one by Matthias Bandemer); - LDAP lookup on an empty envelope address (e.g. a null return path) adds another lookup key "<>", as it is difficult if not impossible to have LDAP attributes with empty string as a value; by Michael Hall; - LDAP.schema: drop "MUST ( mail )" from objectclass 'amavisAccount'; suggested by Michael Hall; - updated comments and documentation, most notably the README.chroot; - contributed file Macintosh.tar.gz updated by Dale Walsh; COMPATIBILITY - replaced 'hits=' with 'score=' in inserted X-Spam-Status header field (and in some internal log entries) for compatibility with a changed default in SpamAssassin 3.1; - insert X-Spam-Score header field for compatibility with SA (previously insertion of this header field was commented-out because the information is redundant, as the score already appears in X-Spam-Status); OPTIMIZATION - speed up sending a mail header of full defanged (rewritten) mail over SMTP back to MTA by a factor of 4 by buffering header fields into large chunks to avoid bottleneck in Net::Cmd::datasend, which has lots of overhead for line-by-line writes. Previously slow writes mostly affected mail messages with extreme header lengths (such as results of a broken mail loop), or when delivering defanged messages, particularly at sites with large MTA mail size limits, sometimes to a point of exceeding timeout limits; reported by Dominik Weber and Ralf Hildebrandt; - move subroutine lookup_ip_acl() and associated ip_to_vec() into its own dedicated new package Amavis::Lookup::IP; provide a constructor to pre-parse IP lookup tables to speed up IP lookups in lookup_ip_acl; prepare pre-parsed commonly used IP lookup tables (@mynetworks_maps, @publicnetworks_maps, @inet_acl); - optimized reading loop in SMTP DATA state, receiving data is now about 35% faster when mail size limit is not enforced (which is a default); no speedup (nor slowdowns) when mail size limit _is_ enforced; - cache results of evaluated macros during a single call to expand(), as macro calls often come in pairs, like: [?%e||\[%e\] ] or [? %#T ||, Tests: [%T|,]]; together with the above optimization in pre-parsed IP lookups it shaves off 25% of time in preparing main log entry; - set locale LC_TIME to "C" globally, avoid changing and restoring locale for every log write and when generating RFC2822 timestamps; - added an optimization note in README.sql about indexes and about SELECT count(*) in MySQL with InnoDB; investigation by Paolo Cravero; --------------------------------------------------------------------------- June 29, 2005 amavisd-new-2.3.2 release notes INCOMPATIBILITY with 2.3.1 and earlier versions: If running amavisd daemon in chroot please note: Each child process now opens its own syslog connection or a file descriptor to a log file, and no longer inherits a connection from its parent. When running in chroot jail and logging to syslog, the syslog client routines need syslogd socket to be present in the chroot subtree to be able to establish a connection with syslogd, otherwise logging output may be lost. Additional syslogd sockets (to be made available in the jail) may be requested from the syslogd daemon, see its documentation. This requirement is equivalent to the requirement of chrooted Postfix services (see Postfix documentation file BASIC_CONFIGURATION_README). BUG FIXES since 2.3.1: - do not enforce $MAXFILES limit during top-level MIME decoding to avoid tempfailing mail; MIME parts are still counted, so a limit exceeded may still be reported during subsequent decoding, but this is handled more gracefully and does not cause preserved temporary directories to be left behind; reported by Marcin Lemanski; suggested by Stephane Lentz and Robert LeBlanc (noted in the 2.0 release notes); - use recv() instead of read() to get results from daemonized virus scanners in an attempt to avoid a bogus Perl I/O status on some Linux installations (reported by Sander Steffann); we now get a meaningful status codes like ECONNRESET instead of a bogus EBADF (Bad file descriptor); - ignore status ECONNRESET when reading results of a daemonized virus scanner from a socket, specific to some Linux versions; thanks to Sander Steffann for the initial report and extensive help in debugging the Perl problem; - run_av and other similar code sections: replace line-by-line reads by block-by-block reads wherever possible to avoid inappropriate status report EBADF (Bad file descriptor) caused by Perl I/O bug when last line is not terminated by a newline. The problem was affecting reading response from some command line virus checkers; reported by Sander Steffann; Perl bug tracking: #39060: readline of a not NL-terminated last line results in Bad file descriptor; - ignore status EAGAIN when reading results on a pipe from a forked process; the status EAGAIN seems to be an artifact of Perl I/O on some installations; reported by several people to cause problems on FreeBSD with Perl 5.8.7 (but Perl 5.8.6 is fine); thanks to Bart Matterne for testing and feedback; - allow one level of indirection when collecting %needed_protocols; global setting $protocol='COURIER' did not work, a workaround was needed with previous version, e.g.: $policy_bank{'QMQPqq'}={protocol=>'QMQPqq'}; reported by Nicklas Bondesson and Martin Orr; - fix a bug (introduced with 2.3.0) in Courier and QMQPqq setups, where global information about processed message wasn't always reset and could leak into processing of a subsequent message; reported by Nicklas Bondesson; - SQL: fix arguments in calls to last_insert_id(), failing under PostgreSQL (MySQL didn't mind); pointed out by Henrik Krohns; - if module SAVI is loaded, insist it is version 0.30 or later; incompatibility with earlier versions reported by Andrzej Kukula; - make use of the new Net::Server 0.88 hook run_n_children_hook() to reload SAVI database; removes a need to apply SAVI patch to Net::Server; the Net::Server hook was suggested by Paul B. Henson and others, and incorporated into Net::Server 0.88 by Paul Seamons; - reopen log file or syslog connection in each child process to make it use its own file descriptor; also minimizes transients when syslogd is restarted and its socket re-created, as reported by Les Ault. When running in chroot please make sure a syslogd socket is also available in the chroot jail, see README.chroot for syslogd options (and BASIC_CONFIGURATION_README in Postfix documentation for the Postfix equivalent); - close log file or syslog in forked process before exec, just to play nicely; - do_lha: fix extracting archive member filename in case of broken archive or empty name (avoid interpreting creation date as a file name); do not increment OpsDecByLha counter for empty archives, which are most likely not lha archives at all; - obey $final_bad_header_destiny D_DISCARD or D_REJECT even for messages with bad headers from mailing lists or with a null envelope sender (DSN); previously such messages were passed; undesired behaviour reported by Cami Sardinha. Such messages are still let through with $final_bad_header_destiny set to D_BOUNCE, as otherwise they will be lost because a bounce is suppressed for null sender messages and for mail from mailing list. This behaviour is retained for backward compatibility, but may need to be reconsidered. - fix regexp for extracting am_id from amavis-milter helper program requests; - if fork/exec fails, try to commit suicide in forked process with POSIX::_exit(1) first, before trying kill('KILL',$$) as a last resort; - updated $log_templ example in amavisd.conf-sample to match the default; pointed out by Gary V; - further reduce a couple of more frequent Perl warnings about the use of uninitialized values in expressions; - pre-load additional Perl modules required by SA 3.1 plugins; - require minimal versions of modules: Time::HiRes 1.49, Archive::Zip 1.14; - replaced nonexistent variable @sa_spam_modifies_subj_maps by @spam_modifies_subj_maps in commented-out example in amavisd.conf-sample; noticed by Joachim Schoenberg; LDAP CHANGES by Michael Hall: All the LDAP changes are transparent to the user. - rewritten some of the code similar to the restructuring of the SQL code in version amavisd-new-2.3.0. A new package Amavisd::LDAP::Connection was added which is an LDAP connection object, and the old connection-related code in Amavis::Lookup::LDAP has been moved to the new package. Amavisd-new will now try to reconnect (once) while processing a message, similar to SQL; - added the ability to specify a '%d' (domain) token in the LDAP base DN; based on idea from Alexander Wittig; - updated default LDAP port based on whether SSL/TLS is being used or not; based on idea from Timo Veith; - updated the search code to query for multiple records and return the results sorted in 'make_query_keys' order versus doing a query for each key. As a result performance is enhanced, and the tweaks 'ldap_get_all', and 'use_query_keys' (recently added) are no longer applicable or needed and have been removed; - improved LDAP error reporting and misc changes to multivalued attributes; - documentation changes (amavisd.conf-default, README.lookups); MINOR IMPROVEMENTS: - macro %c (commonly used in a log template) reports spam score no longer as a single number, but as an explicit sum of a SA score and a by-sender boost score (from @score_sender_maps) when boost score is nonzero; suggested by Ed Walker; - enhancement to amavisd-release: if its only command line argument is '-', then read arguments from stdin, one release request per line, ignoring empty lines; input lines have the same format as command line arguments, i.e.: mail_file mail_file secret_id mail_file secret_id alt_recip1 alt_recip2 ... - better handle cases where a persistent temporary file email.txt as prepared by the SMTP server module gets replaced as a result of some user program modification (e.g. when invoking altermime); problems reported by Dinesh Shah and Leonardo Rodrigues; --------------------------------------------------------------------------- May 9, 2005 amavisd-new-2.3.1 release notes INCOMPATIBILITY with 2.3.0 and older versions: - command line option 'foreground' no longer automatically redirects logging to STDERR; to request logging to STDERR turn off the $DO_SYSLOG and $LOGFILE; the improved flexibility suggested by Matthias Andree and Ralf Hildebrandt; BUG FIXES since 2.3.0: - don't test errno when line-by-line reading loop is exited before eof, it was inappropriate and Perl on some versions of Linux does not like it, possibly reporting "Error reading mail header: Bad file descriptor"; Besides fixing the loops that needed the fix, modified also all remaining reading loops for consistency. Reported by Ralf Hildebrandt; - don't call $per_msg_status->get_autolearn_status with SA older than 3.0; reported by Ian Abbott; - pre-load some additional SpamAssassin modules, needed by SA 3.1 (CVS); - reading from dspam process used wrong variable, producing empty lines for SA checking; reported by Chris Lewicki; - SAVI-Perl: libsavi option for turning on mime parsing is 'Mime', not 'MIME'; libsavi is case-sensitive since version 3.93.0 and was rejecting incorrect option name; thanks to Paul B. Henson; - fixed disabling of SQL wblist ($sql_select_white_black_list=undef); bug reported by Tom Sommer; - do_tnef: extract data from attribute 'Attachment' in addition to data from a more common attribute 'AttachData'; example data provided by Goetz Rieger; - avoid some more frequent Perl warnings on the use of uninitialized variables; - add prototypes for decoding subroutines; prototype mismatch warnings reported by Michael Muenz; - fixed prototype for add_forwarding_header_edits_common(), thanks to Ian Abbott; - replace a simple-minded logic for loading input protocol modules by a slightly more sophisticated code which takes into consideration field 'protocol' in all defined policy banks; reported by Brian Wong; - when replacing existing address extensions don't treat the whole localpart as an extension if the address happens to start with a $recipient_delimiter; pointed out by Kaj J. Niemi; MINOR IMPROVEMENTS: - unfold obsolete-syntax folded header fields composed entirely of white space (RFC 2822 section 4.2); suggested by Ian Abbott and others. Note that such unfolding breaks DomainKeys/DKIM "simple" canonicalization algorithm (but is transparent to "nofws") if the affected header is included in the signature; - do_tnef_ext: add support for decoding TNEF containers by an external program 'tnef' if available; selectable by an entry in the @decoders list; - mail_via_bsmtp enhancement: substitute %s in quarantine filename template by a defanged sender name; based on a patch by Thomas Jarosch; - lookup_ip_acl enhancement: when a supplied lookup table is an associative array (a hash) and the looked-up address is an IPv4 address, allow simple classful subnet specification by repeatedly truncating the trailing byte from the looked up IPv4 address until a match is found or until further truncation is not possible. Note that this does not apply to IPv6 addresses. For more flexible CIDR subnet specifications please use lookup arrays. - provide a routine read_array, which is modelled after read_hash, but stores lines read from a file into an array lookup table, instead of a hash. --------------------------------------------------------------------------- April 24, 2005 amavisd-new-2.3.0 release notes QUICK OVERVIEW: Provides more flexible configuration of decoders. Allows recipients to have individual banning rules. Assigns a long-term unique id to each message, reducing clashes and facilitating retrieval of information. The daemon can store information to an SQL database for logging, reporting and quarantine retrieval, optionally storing entire message to an SQL database. File-based quarantine can disperse files to 62 subdirectories. Provides a quarantine release mechanism. Reconnects to SQL if connection is broken. Can skip quarantining high-score spam. Compatibility with IPv6-enabled Postfix is improved. SECURITY: - require minimal version 1.05 of Convert::UUlib to avoid a security problem in the underlying uulib: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2005-1349 which is now known to be exploitable (2006-12-05), credits to Jean-Sébastien Guay-Leroux; INCOMPATIBILITY with 2.2.1 and older versions: - structure of @banned_filename_maps config variable has changed in incompatible way to allow per-recipient banned rules: it is now a two level map indexed by recipient address (similar to the structure of @score_sender_maps). See further down for details. The change will not affect existing installations which either: * leave @banned_filename_maps at its default value and use the traditional $banned_filename_re configuration variable to specify banned rules, which most installations do and remains the most commonly used method; * or, assign to @banned_filename_maps an empty list to disable it; * or, use the alternative mechanism $banned_namepath_re and disable $banned_filename_re or @banned_filename_maps. - macro %i (used in log and other templates) now always shows mail_id (see below) which is often but not necessarily also the name of a quarantined file (like before, available as a macro %q); previously the %i reflected the actual file name, which was longer/different than now; - default values of $virus_quarantine_method, $spam_quarantine_method, $banned_files_quarantine_method and $bad_header_quarantine_method now specify shorter file names based on the newly-introduced mail_id: virus-%m, spam-%m.gz, banned-%m and badh-%m respectively; - a config variable $file now defaults to a string 'file' instead of being undefined; this makes it unnecessary to be explicitly assigned to in the configuration file; - SQL fields virus_lover, bypass_virus_checks and bypass_spam_checks in table 'policy' are now optional, and if missing their value is treated as undef (same as if the field is present but is NULL) - instead of being interpreted as false. This makes it consistent with the interpretation of other missing fields. The change is unlikely to affect existing installations, because these fields were considered non-optional in previous versions. BUG FIXES or missing functionality: - avoid modifying directory which is being read by readdir, it can cause premature termination of the directory traversal; reported by Dale Walsh; - minimize deep recursion in regexp evaluation while parsing some degenerate cases of Received header subfields via/with/id/for, which could result in a process crash; - turned loops in banned checks inside out to make 'allow' rules useful and let them behave as one may expect: when checking mail parts against lookup tables in @banned_filename_maps the evaluation order of checking part's attributes against rules has changed: on each rule in a list, all attributes of a part are now checked together; previously all rules were evaluated (inner loop) for each attribute (outer loop), which made 'allow' rules hardly ever useful; - ignore dsn_cutoff_level if undefined, instead of treating it as 0; - fix generating the positive delivery notifications (requested by $warn*sender settings): the DSN was missing entirely in case of Courier or sendmail-milter setups and the chosen template was not the most appropriate; Courier problem reported by Sander Holthaus; - fix the text in notification templates 'neutral' and 'virus/banned sender' to claim non-delivery when the message was truly not delivered, and to claim delivery for delivered messages; - when per-recipient subject tag strings are different for each user, the mail forwarding clustering algorithm should have split the forwarded mail into separate deliveries but did not do so, resulting in all recipients of a multi-recipient mail to get the same string inserted into Subject; reported by Michael Goth; - at last: when mail is received through LMTP protocol, gracefully handle a temporary failure 4xx reply from MTA to a RCPT TO command and pass it back to an LMTP client for tempfailed recipients only, instead of returning 450 for _all_ recipients (needed the sending routine to be aware of the receiving side capabilities, which was previously not available); - stricter and more consistent error checking and better error reporting on Perl read, sysread and getline operations; - use O_CREAT|O_EXCL when creating files that are not supposed to pre-exist, to be able to detect a potential race condition and other programming errors; - fixed reporting of virus names for av scanner Perl-SAVI; - with Sophos Perl-SAVI module, avoid setting 'MaxRecursionDepth' if $MAXLEVELS is undefined or zero, matching its semantics to other uses in the program; debugging and a fix by Paul B. Henson; - fixed sloppy regexps when parsing SMTP commands; - fixed a typo in README.lookups leading to confusion between fields spam_subject_tag and spam_subject_tag2, making one to believe there is only one such field; pointed out by Max Clark and others; - handle special case: Postfix hates ""@domain but does not mind @domain; NEW FEATURES: - new config variable @decoders (with its policy banks counterpart) makes it possible to enable/disable decoding of individual file content types from the configuration file, and to adjust the external decoders paths and search order, all in one place. This list now makes the following config variables obsolete: $arc, $gzip, $bzip2, $lzop, $uncompress, $unfreeze, $lha, $unarj, $unrar, $zoo, $pax, $cpio, $ar, $rpm2cpio, $cabextract, $ripole; although they are still observed for compatibility reasons if the @decoders list is left at its default value; - banned filename/filetype rules can now be specified on a per-recipient basis Structure of @banned_filename_maps config variable has changed in incompatible way to allow per-recipient banned rules: it is now a two level map, similar to the structure of @score_sender_maps. Lookup keys used at the first level are recipient addresses, results from this lookup can be either a ref to a list of second-level lookup tables, or a string which is interpreted as a comma-separated (or whitespace-separated) list of _names_ of second-level lookup tables. These names are mapped to actual second-level lookup tables through an associative array %banned_rules. The indirection by names through %banned_rules is primarily intended for SQL and LDAP lookups, which can not return complex data structures and Perl code (and should not do so for security reasons), but names may just as well be used by static lookups. The resulting list of second-level lookup tables (which in most cases is a list containing a single element $banned_filename_re, which makes it compatible with existing setups) is looked up by using keys such as part file name and part file type, exactly like in previous versions. The previous default assignment: @banned_filename_maps = ( \$banned_filename_re ); is now incorrect, it can be rewritten (if explicit assignment is desired) as: @banned_filename_maps = ( {'.' => [$banned_filename_re]} ); or (by giving a name to a lookup table): @banned_filename_maps = ( {'.' => 'MYRULES'} ); %banned_rules = ('MYRULES' => $banned_filename_re); which is equivalent to the default setting of both variables: @banned_filename_maps = ( {'.' => 'DEFAULT'} ); # proper two-level struct. @banned_filename_maps = ( 'DEFAULT' ); # same as previous, but shorter %banned_rules = ('DEFAULT' => $banned_filename_re); The SQL table 'policy' may now contain an optional field 'banned_rulenames', which is a comma-separated (or whitespace-separated) list of _names_ of second-level lookup tables, with semantics as described above for static lookups. The configuration variable @banned_filename_maps is a member of policy banks as before. The associative array %banned_rules is global and is not a member of policy banks. The alternative 'banned file' mechanism $banned_namepath_re hasn't changed and is still not merged into @banned_filename_maps, which means it can not be used when per-recipient banned rules are needed. Perhaps in the next version... - introduce a concept of 'mail_id', which is similar to the am_id as reported in the log and elsewhere (e.g. 92598-11-5), but has much stronger long-term uniqueness property and can be used for the purpose of uniquely identifying a quarantined mail, or for other uses. The mail_id is a 12-character string consisting of characters [A-Za-z0-9+-] (like base64, except for a '/' being substituted by a '+') [a note from the future: since amavisd-new-2.7.0 the character set is [A-Za-z0-9-_] and follows rfc4648 base64url, derived from base64 by substituting '+' -> '-' and '/' -> '_'], guaranteed to start and to end with an alphanumeric character (i.e. not with '+' or '-' or '_'). It is derived by a cryptographically strong method (MD5), cumulatively collecting entropy during the life of child processes, folding-in entropy from processed mail and other inexpensively accessible sources, collected when an opportunity arises (e.g. file system file-IDs, SA results etc), without placing a burden on system sources of randomness (see RFC 4086). Note that MD5 has been demonstrated to have some weaknesses, but we are not talking about cryptographic attacks here, but rather about spreading message identity codes which have no inherent intention of causing collisions. The mail_id carries 71.9 bits of information (subject to quality of sources of entropy). For a high-end system that keeps a year's worth of mail messages in evidence (e.g. in quarantine) and receives 10e6 messages each day (20..200 TB of yearly mail contents), the probability of a mail_id collision happening during one year (while gradually displacing an entire collection with a new set of IDs) is n^2/m = 0.3 % (10e6 * 365)^2 / (62 * (64^(12-2)) * 62) = (10e6 * 365)^2 / 2^71.9 = 0.003 Eventual clash is still possible and needs to be detected by testing each generated code against the set of message IDs currently kept in a database. On a smallish system receiving 10.000 mail messages daily an 8-character mail_id would suffice, but savings are not worth the trouble of providing a configuration flexibility. Paired with a mail_id there is its companion secret_id generated for each message, such that mail_id can be derived from secret_id and pairing checked, but not the other way around. The secret_id can serve as a 'ticket', granting user a right to release a quarantined message addressed to him. - SQL: can store information about every processed mail to SQL; the information is similar to level 0 log entries, but more detailed; an SQL database can be used as a basis for searching for a particular mail, for preparing reports and to facilitate quarantine management (searching and releasing). Enabled by configuring the @storage_sql_dsn list which contains information about an SQL server and dataset name, just like the @lookup_sql_dsn does for the SQL lookups. If @storage_sql_dsn is the same as the @lookup_sql_dsn, a single connection to SQL database will be used, otherwise separate and independent connections are established, possibly to different SQL servers. Loosely based on ideas from Maia Mailguard by Robert LeBlanc and a patch by Brian Wong. Thanks to Brian Wong for testing and valuable feedback. See README.lookups for an SQL schema. See new file amavisd-sql-maintain (incorporated into README.sql in later versions) for ideas on database housekeeping (expiring old entries). - SQL: can quarantine to an SQL database; selected by setting config variables $*_quarantine_method to 'sql:' The @storage_sql_dsn list of dataset names is used to choose SQL server and dataset name, and must be nonempty when $*_quarantine_method is 'sql:'; When $*_quarantine_method is set to 'sql:' the SQL logging must be enabled as well; - SQL: clean the mess of needing SQL lookup objects to be aware of each other, by separating SQL connection information (Amavis::Out::SQL::Connection object) from objects holding SQL statement handles. Statement handles are invalidated on reconnect, and are dynamically 'prepared' as needed. - SQL: thanks to a reorganization of SQL objects an automatic reconnect to an SQL server is done without temporary failing a processed message; - SQL: new configuration variable (an associative array) %sql_clause, also part of policy maps, allows SQL clauses to be switched with policy banks. The components of %sql_clause under keys 'sel_policy' and 'sel_wblist' perform the duty of legacy configuration variables $sql_select_policy and $sql_select_white_black_list. Compatibility with older configuration files is maintained when %sql_clause is left at its default value; - can add one layer of 62 subdirectories to the quarantine directory for more efficient file system use by uniformly distributing quarantined mail; enabled by setting a new config variable $quarantine_subdir_levels to 1; - choosing mail_id as a quarantine file name now greatly reduces the likelihood of two quarantined messages processes by the same child process shortly one after another from attempting to get quarantined under the same filename, leading to a temporary failure ("File...already exists, refuse to overwrite") and leaving behind a preserved temporary directory; reported by Martin Svensson; - release from quarantine functionality is now a built-in feature; a message release can be requested via enhanced AM.PDP protocol. There is a new utility program 'amavisd-release', which currently mostly serves to demonstrate how to request releasing of a quarantined file. Currently the supported quarantine types are: plain file, gzipped plain file with a name ending in .gz, and an SQL-based quarantine. Currently not supported is a release from a BSMTP-encoded plain file or from a mbox (Unix-style) mailbox quarantine file. Example use: $ amavisd-release spam/V/V5htXBh0y0Hr.gz H2huh4wfrSyC or providing a replacement list of recipients: $ amavisd-release spam/V/V5htXBh0y0Hr.gz H2huh4wfrSyC user@example.com The first argument is a (relative) quarantine file name, as reported in the log. It must include a 12-character mail_id which is automatically recognized. The second argument is a secret_id, which can be fetched from an SQL database if @storage_sql_dsn is enabled (see README.sql), for example by the command: $ mysql amavis -e 'SELECT secret_id FROM msgs WHERE mail_id="V5htXBh0y0Hr"' or (preferably) by some other more advanced utility program. Current simple-minded heuristics in the amavisd-release program is to assume a message is stored in an SQL database when the file name (first argument) consists only of a 12-character mail_id. Please adjust the program if this assumption is not true, e.g. when $virus_quarantine_method='local:%m' instead of a default $virus_quarantine_method='local:virus-%m'; If secret_id is not available, administrator may choose to skip checking of secret_id in the amavisd daemon by setting a configuration variable $auth_required_release to false (it defaults to true), and supplying an empty secret_id or not supplying it at all in the AM.PDP release request. The variable $auth_required_release is also part of policy banks. If the release client program specifies a nonempty secret_id in the request, the secret_id will be validated and a request will fail if not valid, regardless of the setting of $auth_required_release. Release requests contend for the same amavisd child processes as regular mail processing. This may cause delays in responses to release requests, especially when Postfix feeds mail to amavisd-new via LMTP which is more persistent in keeping connections open than the Postfix SMTP client service. To ensure one child process is always available for processing extra requests such as release requests, one may choose to set $max_servers larger (by one) than MTA's maxproc setting. To enable recognition and processing of AM.PDP protocol requests in amavisd daemon, a dedicated Unix socket or a TCP port needs to be opened, for example by the following assignments in amavisd.conf: $unix_socketname = "$MYHOME/amavisd.sock"; $interface_policy{'SOCK'} = 'AM.PDP'; $policy_bank{'AM.PDP'} = { protocol => 'AM.PDP' }; or similarly for connections through a dedicated TCP port 9998, and restricts it to accepting IP connections from local IP address only: $inet_socket_port = [10024,9998]; $interface_policy{'9998'} = 'AM.PDP'; $policy_bank{'AM.PDP'} = { protocol => 'AM.PDP', # Amavis policy delegation protocol inet_acl => [qw( 127.0.0.1 )], # log_level => 3, }; - new configuration variable @spam_quarantine_cutoff_level_maps turns off quarantining if spam score is at or above this level; suggested by Gary Verchick, MrC and others; - more informative logging and SMTP status generation in smtp client code; - IPv6: allow optional brackets around IPv6 address in lookup tables and configuration variables, e.g. [FE80::]/10 is treated the same as FE80::/10 Allow (and ignore) link-local scoped address in extended numeric IPv6 syntax (interface specification) when parsing link-level IPv6 addresses, e.g. fe80::1%lo0; - IPv6: adjust the default @mynetworks to include link-local and site-local address ranges [FE80::]/10 and [FEC0::]/10, and add (optional) brackets around [::1]; (although the use of site-local address ranges seems to be deprecated nowadays); - IPv6: tested sending mail via slightly modified Net::SMTP through an IPv6 socket to an IPv6-enabled Postfix; updated README.postfix; To experiment, replace the: @ISA = qw(Net::Cmd IO::Socket::INET); with: use IO::Socket::INET6; @ISA = qw(Net::Cmd IO::Socket::INET6); in Net/SMTP.pm, to make use of the: $notify_method='smtp:[::1]:10025'; Don't forget to add [::1] to mynetworks at MTA smtpd service on port 10025; - bring earlier the initialization of %local_delivery_aliases and %builtins so that the config file can override the defaults; - add SA autolearn status to the logged spam status (log line: "SPAM, ..."), as well as full TESTSCORES info to the list of SA tests (including score points for each SA test); contributed by John Sivak; - new small wrapper module Amavis::IO::Zlib around Compress::Zlib allows for reading back compressed quarantine files for a mail release, and allows for writing compressed quarantine files without having to fork a gzip process. This makes the utility program gzip(1) optional, which may be appreciated when running in a chroot environment; - modified do_gunzip to use the new Amavis::IO::Zlib module; - added LDAP lookups for the following attributes: amavisVirusAdmin, amavisNewVirusAdmin, amavisSpamAdmin, amavisBannedAdmin, and amavisBadHeaderAdmin; by Michael Hall; added attribute amavisBannedRuleNames by Brian Wong and Michael Hall; - a policy bank name 'MYUSERS' now has a special semantics: this policy bank gets loaded whenever the sender matches @local_domains_maps. This only makes sense if local sender addresses can be trusted -- for example by requiring authentication before letting users send with their local address; the feature requested and a patch provided by Steffen Hansen; (a note from future: starting with 2.6.0 an additional requirement for loading a policy bank 'MYUSERS' is that 'originating' flag is on, which typically means that mail is coming from internal networks or authenticated roaming users); - add cumulative percents to the TIMING report to make it easier to locate large contributors to elapsed time; - updated interface code to Sophos Perl-SAVI module, taking advantage of its new ability to reload virus data: amavisd-new will initialize the SAVI object in the parent, which will be inherited by the children. The children will detect whenever the virus data is stale and automatically exit (reducing the number of messages processed with out-of-date protection), and the parent will reload the virus data before spawning new children; update provided by Paul B. Henson. To have this feature fully functional a small patch to Net::Server is needed: http://www.csupomona.edu/~henson/www/projects/SAVI-Perl/dist/ Net-Server-0.87-parent_fork_hook-1.diff.gz (patch is no longer necessary since Net::Server 0.88 and amavisd-new 2.3.2) - provide a routine dump_hash for debugging purposes: given a reference to an associative array (a Perl hash) it writes its contents to a log. Note: if called within amavisd.conf the log is still directed to STDERR. For example: @local_domains_maps = ( read_hash("$MYHOME/local_domains") ); dump_hash($local_domains_maps[0]); OTHER CHANGES: - new documentation file README.sql; - tightened up a sample regular expression in amavisd.conf for catching Class ID (CLSID) extensions in file names; - restrict the 'double extension' banning rule to require at least one letter in the next-to-last dot-delimited field; this allows filenames such as prog.33.22.01.exe not to be blocked by this rule; - change hash lookups code to stop a hash search when a matching key exists, even if the matching result (corresponding hash value) is undef; this reverts the change made in a release 2.1.0 to a previous behaviour which is consistent with regexp lookups (but leaves SQL lookups to be different, continuing the scan to more general entries on a NULL field value; SQL lookup is specific because all attributes live in the same record); inconsistency reported by Gary Verchick; - as a workaround for some versions of unrar (sparc64?) which are unable to create a subdirectory parts/rar by themselves, do_unrar now prepares the subdirectory explicitly; suggested by Andrzej Kukula; - provide a new CentralCommand Vexira 'vascan' av entry (distinct from the 'Antivirus', which remains H+BEDV AntiVir -based) to work with the new Vexira scanner, thanks to Brian Wong and Norman C Rice; - Vexira vascan: added status codes 3 (password protected) and 9 (unsupported format), recognize "sequence found"; thanks to Brian Wong; - F-Prot Antivirus: enhance the pattern to capture virus names; - renamed AV entry "H+BEDV AntiVir" to "Avira AntiVir", reflecting the company name change; thanks to Patrick Ben Koetter; - change the default $sql_select_policy and $sql_select_white_black_list to use the LEFT JOIN ... ON instead of WHERE for the 'join' relation; should be functionally equivalent, but makes the join operation explicit; - changes to LDAP Schema to make it import and play nicely with Novell NDS, by Michael Tracey, SONOPRESS USA, LLC April 07 2005 (changes are included in LDAP.schema but commented out by Mark) - remove special handling in unmangle_sender() for ancient viruses Magistr and Badtrans, leave decisions on suppressing DSN entirely to @viruses_that_fake_sender_maps; - rise a limit on the number of logged matching SA tests from 10 to 50 (still some arbitrary sanity limit); based on observation of Bojan Zdrnja; - add a minimal version requirement 2.22 for Digest::MD5, we need the new 'clone' method; reported by Thomas Jarosch; - a command line option 'debug-sa' now sets variable $sa_debug to a string '1,all' instead of 1, in anticipation of the next version (3.1) of SA which changed interpretation of its debug option; the '1,all' seems to be compatible both with the old and the new interpretation, despite producing a warning in pre-3.1 versions of SA; - documentation note: Macintosh.tar.gz installation instructions for Macintosh are not recent, they apply to Mac OS X 10.2.0-10.3.9 --------------------------------------------------------------------------- December 22, 2004 amavisd-new-2.2.1 release notes SECURITY: - add support for the pax(1) archive decoder, which can handle tar/cpio/pax archives (including legacy format variants). Due to limitations in cpio (and in Archive::Tar), for security reasons it is preferred to decode such archives with pax and no longer with cpio; please add a line: $pax = 'pax'; to amavisd.conf and verify that the program pax is installed on the system (and in the jail if running in chroot); problem reported by Ron Ogle; - perform additional tests at startup time on proper protection of the configuration file; - add file name extensions wmf, emf and grp to the example list of banned extension, according to recent Microsoft security bulletins; suggested by Stephane Lentz; - introduces 'clean but inconclusive' av scanner result to avoid a specialized or quick partial av scanner like jpeg checker to claim mail is clean when all other general purpose av scanners fail (see below); INCOMPATIBILITY: - removed some legacy $*_ldap variables, as they are no longer needed; These variables were still declared but ignored in 2.2.0 for compatibility with older amavisd.conf files. Such variables need to be removed from the amavisd.conf if they are still present there from older versions, otherwise Perl will complain with 'Global symbol ... requires explicit package name"; OTHER FIXES: - files_to_scan and decompose_mail are now able to remove unexpected directories which may have been left behind by some failed decoding and were causing temporary failures and mail delivery retries; error recovery problem after failed unarj reported by Ralf Hildebrandt, and a related problem with tar, reported by Ron Ogle; - error recovery code in files_to_scan and rmdir_recursively now tries to change protection on directories and files, and retry if the first attempt to access them fails because of denied permission; - pre-load some additional Perl modules needed by SA when running in chroot; - add module Net::LDAP::Search to a list of pre-fetched modules; omission pointed out by Paul Jacobson; - when quarantining is disabled by keeping $QUARANTINEDIR undefined, the log entry and administrator notification message inappropriately suggested that mail was quarantined, which in fact (appropriately) it was not. Setting $QUARANTINEDIR='' did work as expected. Reported by Sascha Lucas; - avoid the use of Encode::is_utf8 due to a Perl bug (still present in 5.8.8, Encode::is_utf8 on tainted utf8 character string produces false); Perl bug tracking: #32687: Encode::is_utf8 on tainted UTF8 string returns false; - modify safe_encode() to guarantee the result is a string of octets, not a string of UTF-8 characters; it saves some unnecessary work in further processing and keeps MIME::Entity from UTF swamp when running in chroot; problem pointed out by Branko F. Gracnar; - avoid braindead Perl default where an empty regexp implies the last successfully matched regexp, which (if not being very careful) brings in some completely unrelated last-executed regular expression; - change kill 'TERM' into kill 'KILL' when a forked process within run_command and run_command_consumer gets into deep trouble, to avoid exit handlers being invoked in the subprocess (which could lead to two processes trying to clean the same set of temporary files); - in an old sendmail setup using the amavis(.c) helper program without LDA arguments, avoid inappropriate warning: "WARN: no recips left (forgot to set $forward_method=undef using milter?) and return status 0 instead of 99 when message is to be blocked, as the helper program amavis(.c) does not recognize status 99 in this situation and inappropriately passed it on to sendmail; reported by The Mindflayer; - the @bypass_header_checks_maps is now able to also bypass the bad header checks as provided by MIME::Parser; inconsitency reported by CRivera; - avoid some Perl warning messages; thanks to Bill Landry; CHANGES AND MINOR NEW FEATURES: - add configuration variable @newvirus_admin_maps (and $newvirus_admin, along with corresponding SQL field 'newvirus_admin') which works like the existing @virus_admin_maps (and $virus_admin), except that it sends virus administrator notification to specified e-mail address only for newly encountered viruses which have not yet been encountered since the amavisd startup. It makes use of by-virusname counters in the SNMP counters database. If more than one child process starts working on infected message containing a not-yet-accounted-for virus, there might be more than one 'first time' notification, this is not a malfunction. Both the @newvirus_admin_maps and the @virus_admin_maps may be enabled, each (possibly both) would receive their notifications as appropriate. A useful setting is to globally enable only the new virus notifications, and additionally enable _all_ administrator notifications for internally originating mail only (by the use of policy banks); - provide separate configuration variables @banned_admin_maps and @bad_header_admin_maps, along with corresponding SQL fields 'banned_admin' and 'bad_header_admin'; their function was previously covered by @virus_admin_maps, which now only still controls administrator notifications in case of viruses; - introduces 'clean but inconclusive' av scanner result to avoid a specialized or quick partial av scanner like jpeg checker to claim mail is clean when all other general purpose av scanners fail: in av scanner entries (lists @av_scanners and @av_scanners_backup) give an extended meaning to undefined fourth argument (the 'match for clean' list or regexp). The interpretation of the fourth argument is now: 4. an array ref of av scanner exit status values, or a regexp (to be matched against scanner output), indicating NO VIRUSES found; a special case is a value undef, which does not claim file to be clean (i.e. it never matches, similar to []), but suppresses a failure warning; to be used when the result is inconclusive (useful for specialized and quick partial scanners such as jpeg checker); Also modified example jpeg checker entry in amavisd.conf accordingly. - NOD32 av scanner: changed @av_scanners entry to match the new version of the scanner; thanks to Nejc Skoberne; - added @av_scanners entry for File::Scan; - when preparing an SQL SELECT clause for white/blacklisting lookup, take into account a relative position of ? and %k in the $sql_select_white_black_list template to improve flexibility of specifying the clause; suggested by Matt Petteys; - reduce the log level of some more common and harmless log messages; - macro %p and the log entry now reports full policy bank path, not just the last loaded policy bank name; - added LDAP attributes amavisWarnVirusRecip, amavisWarnBannedRecip, and amavisWarnBadHeaderRecip; by Joel Nimety and Michael Hall; - renamed LDAP attribute name amavisSpamModifiesSubject to amavisSpamModifiesSubj in order to match the documented LDAP schema; noticed by Kees Bos, patch by Michael Hall; - add support for ripOLE decoder, which attempts to extract embedded documents from MS OLE documents (MS Office) (http://www.pldaniels.com/ripole/, by Paul L Daniels)); ripOLE is still experimental/alpha code; To make amavisd-new find the installed program 'ripole', add the: $ripole = 'ripole'; to the amavisd.conf. Suggested by David Wilson and Noel Jones; - allow multiple occurrences of command line option: -c config_file and execute the provided configuration files one after the other; based on a subset of functionality provided as a patch by Davor Ocelic; - a slight improvement in classifying mpeg and some other multimedia files (in the default $map_full_type_to_short_type_re); - several minor code cleanups; - add a recommendation by Daniel J McDonald to a documentation file INSTALL: If different UID is preferred for an AV scanner, a solution for ClamAV is to add user clamav to the amavis group, and then add AllowSupplementaryGroups to clamd.conf; - enclosed a simple demonstrational Perl program amavis.pl, which is functionally much like the amavis.c helper program, but talks the new AM.PDP protocol with the amavisd daemon. See README.protocol for the description of AM.PDP protocol. To be placed in amavisd.conf: $protocol='AM.PDP'; $unix_socketname='/var/amavis/amavisd.sock'; Usage: amavis.pl sender recip1 recip2 ... < message.txt - documentation updates; --------------------------------------------------------------------------- November 2, 2004 amavisd-new-2.2.0 release notes SECURITY: - modified MIME entity traversal to include MIME container parts (e.g. multipart/*) as pseudo parts. Such pseudo-parts do not carry any body contents but do show up (with their MIME content type only) in the tree structure as seen by banned names checks. This makes it possible to specify more complex banned rules based on the placement of leaf nodes within or outside of mime multipart containers. This also re-enables the possibility to check such MIME wrappers for banned MIME Content-Types (most notably for the message/partial and message/external-body), which was lost with the change of internal representation of mail parts in version 2.0 (amavisd-new-20040701); - preserve original zip archive for virus scanners to see, if the archive contains any zero-length members (Archive::Zip module chooses not to extract members with declared zero size, even if the size does not correspond to the actual size); based on a patch by Dirk Datzert; - add tests to mime_traverse to verify that the files produced by MIME::Parser really do exist and are readable; (and sent a patch adding I/O checks for MIME::Tools to its maintainer; please use MIME::Tools 5.414 or later from CPAN to avoid possibility of full /tmp partition causing infected or spam mail to pass through); - changed recommendation in INSTALL to choose the latest version of MIME-Tools from CPAN - the 5.415 at the time of this writing; - do_unrar: recognize encrypted entire archives (not just their individual members), and flag mail as undecipherable; - recognize file(1) report/^MS Windows\b.*\bDLL\b/ as short type 'dll'; add 'dll' to example patterns in amavisd.conf and amavisd.conf-sample; add name extension '.cpl' to the list of basic banned names; INCOMPATIBILITY: - incompatible change: the default value for $recipient_delimiter is now undef and no longer '+'; adding address extensions must now be explicitly enabled; - minor change in the default X-Virus-Scanned: header field, see below; OTHER CHANGES SINCE 2.1.2: - rewritten LDAP modules, by Michael Hall; - improvements in handling of double errors (errors that occur while handling consequences of some previous error); be permissive on failures in DESTROY methods; prevent some cases for child process not being able to sign off from the nanny database; - enforce $max_requests more strictly, dropping SMTP session after task count is exceeded by one; this is in response to the new smtp session caching behaviour in Postfix, which is now much more persistent in keeping the session open on a busy mailer; although dropping session at the server side is discouraged by RFC 2821, this change was recommended on the postfix-users mailing list; - added a site-wide mail size limit $smtpd_message_size_limit, and a per-recipient mail size limit lookups @message_size_limit_maps along with SQL and LDAP fields, making it possible to reject mail based on its size. The list of lookup tables maps recipients to mail size in bytes, e.g.: @message_size_limit_maps = ({'user1@example.com' => 20_000_000, 'user2@example.com' => 15*1024*1024, 'user3@example.com' => 0, # uses global limit '.example.com' => 10*1024*1024 }); $smtpd_message_size_limit = 20*1024*1024; # global limit if nonzero A value of 0 or undef disables the check and is a default. A per-recipient limit is bound on the high side by the $smtpd_message_size_limit, and on the low side by 64kB, which is a minimal allowed size limit imposed by RFC 2821. This limiting really belongs to MTA and is only partially supported here (no admin notifications, no quarantine, no final_*_destiny configurability). It is mostly provided here to be able to place some sanity limit on runaway or malicious clients, or if someone insists on using amavisd-new in a pre-queue filtering setup; suggested by Tuomo Soini; - internal: add new object $msginfo->mail_text_fn to hold the file name of the original mail, decouple the location of file email.txt from the temporary directory, which was implied until now. This is presently needed for the Courier interface. Add optional attribute 'mail_file' to the AM.PDP protocol, see README.protocol; - in banned parts descriptor strings which are used in banned name checking, provide a 'T=empty' short type for empty mail parts, including for empty MIME parts (instead of omitting T=... altogether). This can be used in banned rules to test for empty parts, generally or restricted to empty MIME parts; suggested by Ricardo Stella and Stephane Lentz; - a banned lookup result (which is interpreted as boolean for most purposes: zero or empty for false, anything else for true) may give a result 'DISCARD' (which is true as well), which will disable DSN if it turns out the mail was blocked by such banned rule (akin to viruses_that_fake_sender and spam_dsn_cutoff_level). Here is an example rule in $banned_namepath_re: [ qr'(?# BLOCK EMPTY MIME PART APPLICATION/OCTET-STREAM ) ^ (.*\t)? M=application/octet-stream \t(.*\t)* T=empty (\t.*)? $'xmi => 'DISCARD' ], Suggested by Ricardo Stella and Stephane Lentz; - fix 'Insecure dependency in unlink' in sub files_to_scan which could happen when some decoder left non-regular files in the directory; reported by Andrzej Kukula; - bug fix: only insert LDAP and SQL lookups objects into lists of maps at the first task of a child process, later tasks should not insert duplicates; - new subroutine do_ar and new configuration variable $ar to handle standard Unix archives, including Debian binary packages; - recognize a Unix archive and give it a short type .a; - recognize a Unix relocatable binary and give it a short type .o; - do not penalize SMTP status "554 5.1.1 Error (DATA): no valid recipients", the situation arises regularly when pipelining is in effect and all recipients happen to be rejected; - protect spaces and newlines when logging broken Message-ID and Resent-Message-ID header fields (macros %m and %r) to facilitate log parsing; protect newlines in logged Subject header field (macro %j); parsing difficulty pointed out by Chris Lee; The present solution uses =XX (hex) encoding and is a quick and dirty fix. A cleaner solution to avoid double sanitation of special characters is needed, but would involve a deeper reorganization; - updated example list of "banned extensions - long" in amavisd.conf and amavisd.conf-sample; - change the default lock file name from "$MYHOME/amavisd-$$.lock" to "$MYHOME/amavisd.lock", to avoid inventing a new name at every restart and leaving old files behind; pointed out by Dale Walsh and Martin Orr; - updated av entry for nvcc (Norman Virus Control v5 / Linux) to include statuses 10,11, and 2,14 to the status lists according to documentation; password protected or corrupted archive (status 11) was not recognized as non-infected status; thanks to Michael Ramke of Norman Data Defense Systems GmbH; - updated DrWebD entry to recognize and ignore flag DERR_SKIPPED; - support Mail::ClamAV 0.12 and 0.13 or later, which is incompatible with 0.11 due to change of constant names in the underlying ClamAV library; - added 'check-jpeg' example entry to the @av_scanners list and provide the associated module JpegTester.pm; it offers a fully-fledged check for jpeg comment field buffer overflow attempts; should serve mainly as an example for adding similar quick responses to new threats; - added 'check-jpeg-simple' example entry to the @av_scanners list (only in amavisd.conf-sample); it offers a quick check (and not very exact one: checks only the first 32kB) for jpeg comment field buffer overflow attempts; should serve mainly as an example for adding similar quick responses to new threats; - relax too restrictive sanity check on temporary directory name when accepting requests from a helper program or via AM.PDP protocol (e.g. with sendmail milter setup); reported by Babu Kanagala; - relax allowed set of characters when receiving XFORWARD attribute values, it turns out that characters like '=' and '+' are allowed; - when using "bsmtp:" delivery method suppress X-Envelope-From and X-Envelope-To header fields, as the addresses are already available in the envelope; - when using the "bsmtp:" quarantine method the *_quarantine_to was completely ignored, which made it impossible to turn off quarantining selectively for certain users by specifying an empty or undef value. Since 2.2.0 an empty *_quarantine_to turns off quarantine for a recipient regardless of the quarantine method. A nonempty string in *_quarantine_to (the exact value is still ignored) must now be used even with "bsmtp:" to enable quarantining. Inconsistency discovered by Sean Doherty; - suppress leading $QUARANTINEDIR string from the value of macro %q, thus hiding the absolute file path from notifications; - add configuration variable $local_client_bind_address (and equivalent policy bank key), to allow for explicitly binding local socket address to a specific interface in SMTP client; suggested by Wouter de Jong; - keep whatever (if any) file results from gunzip and family (do_uncompress) even if the decompressor's exit status is nonzero; reason: gzip returns status 2 when decompressing file with trailing garbage; reported by Tobias Reckhard; - collect declared original file name from gzipped (and friends) files if reported by file(1), making them available to banned name checking; - avoid unnecessarily checking white/black lists if spam checks will not be used (e.g. infected mail); - use qquote_rfc2821_local to properly quote e-mail addresses in the most visible log entries; - if there is more than a single (or less then one) occurrence of %k in the SQL SELECT clause template, multiplicate the set of query keys accordingly, making possible more complicated custom SELECT clauses; - don't forget to load amavis policy delegation protocol support code if AM.PDP is explicitly requested in $protocol, even if not listening on a Unix socket; - add 'queue_id' attribute to the AM.PDP protocol; equivalent to a change in the Courier support code by Martin Orr; - include the declared (faked) sender address in the virus recipient notification template, in addition to the originator IP address; - add macro %Q and method Amavis::In::Message::queue_id, holding a MTA queue ID if available (in Courier and milter/AM.PDP setup); by Michael Musikhin (through Martin Orr); - add macro %y to show elapsed processing time; suggested by Ed Walker; - sanitize newlines and spaces (and some other characters) when moving syntactically invalid Message-ID and Resent-Message-ID to macros %m and %r for logging purposes; resulting wrapped main log entry reported by CRivera; - bring up syslog priority to LOG_NOTICE when debug or debug_oneshot is in effect; - make a product name, version ID and version date available as separate variables to avoid the need to parse $myversion for the purpose of customizing e.g. the setting of $X_HEADER_LINE; based on suggestion by Dale Walsh; the re-introduction of a date ($myversion_date) also suggested by Stephane Lentz. Added variables: $myproduct_name, $myversion_id, $myversion_id_numeric, and $myversion_date, which serve to assemble the $myversion. Modified the default templates of $smtpd_greeting_banner and $smtpd_quit_banner to take advantage of the new variables. Changed $X_HEADER_LINE default from "by amavisd-new at $mydomain" to "$myproduct_name at $mydomain" and added an example of a $X_HEADER_LINE setting with version number included to the amavisd.conf-sample; - added SQL fields 'virus_admin' and 'spam_admin' to lookup lists @virus_admin_maps and @spam_admin_maps; patch by Robert Collier; - add a log message 'SPAM-KILL, ...' (at log level 3) for not-passed mail, to complement the 'SPAM-TAG' log message for passed mail; - add Mail::SpamAssassin::Plugin::SPF to a list of modules that SA fails to load at init time; - prevent sending the same SMTP response more than once, if the first attempt failed due to disconnected SMTP session; - fix a double-@ formatting buglet in the log message "adding address extension _spam to user@@domain", reported by Vincenzo; - add kill('TERM',$$) to the arsenal of attempts to get rid of unwanted forked process; - includes rpm spec file, including the init script, contributed by Marius Andreiana, based on previous work by Dag Wieers; - document the localization template directory contents (in file amavisd.conf-sample) when read_l10n_templates is used; thanks to Joël Brogniart; - includes file Macintosh.tar.gz, which contains auto-startup scripts and installation instructions for Mac OS X, contributed by Dale Walsh of the Dale Enterprise L.L.C. --------------------------------------------------------------------------- September 6, 2004 amavisd-new-2.1.2 release notes - fixed (hard)black- and white-listing on static lookup tables which failed to match any sender; reported by Derck Floor; - use $hdrfrom_notify_recip address in the From: field for recipient notifications, instead of $hdrfrom_notify_admin; inconsistency pointed out by Ekkehard Burkon; - the 'neutral' sender notification template was joining the Subject and the Message-ID header fields into one longer Subject when it was reporting a nondelivery other than the 'invalid characters in header'. Likewise the first body line of this same DSN was eaten up: "This nondelivery report was generated by the amavisd-new program" (the problem was introduced in amavisd-new-20030616 and never reported); - in amavisd-agent, amavisd-nanny, amavisd: extend the signal and error handling in code sections holding bdb locks from just ignoring the SIGINT, to controlled catching and re-signaling several signals and error conditions; problem reported by Tom Mulder; - suppress duplicate names from the list of virus names in macro %V; by Gregor Weiss; - fine-adjusted log level of some log messages; - discard leading and trailing whitespace from the macro %t (Received trace); - extend the search for IP in the Received trace from 4 to a maximum of 6 entries; - ignore private IPv6 addresses (RFC 3513: link-local, site-local, multicast) when searching through Received trace for the origin of mail; - place mail header field X-Envelope-From in front of the X-Envelope-To in quarantined mail; also changed case of X-Quarantine-id into X-Quarantine-Id for consistency with other header fields; - provide new macro %e which evaluates to our best guess of the originator IP address collected from the Received trace, complementing similar macros %t, %a and %g; suggested by Gregor Weiss; - add the result of macro %e to the default 0-level log entry; - provide new macros %u and %U to evaluate to a timestamp of the message reception similar to an existing macro %d (RFC 2822 local date-time); the (%u) as Unix time (seconds since 1970-01-01T00:00Z as a decimal integer, suggested by Gregor Weiss), and (%U) as ISO 8601 (EN 28601) UTC date-time; - avoid some empty lines in default DSN templates and fix some inconsistencies in their formatting; - internal: collect existing common code for time formatting as new subroutines iso8601_timestamp and iso8601_utc_timestamp; collect existing common code to find IP address in the Received trace as a new subroutine best_try_originator_ip; - bump up the version number in $myversion - the 2.1.1 still presented itself as 2.1.0; - add a note about a data structure difference between @score_sender_maps and $per_recip_blacklist_sender_lookup_tables (amavisd.conf-sample, amavisd) --------------------------------------------------------------------------- August 24, 2004 amavisd-new-2.1.1 (amavisd-new-20040824) release notes - unconditionally initialize @banned_filename_maps to (\$banned_filename_re), otherwise $banned_filename_re is ignored by default (unless amavisd.conf explicitly assigns to @banned_filename_maps); a patch by Thomas Jarosch; - fixed inappropriate log entry in SQL whitelisting: wbl: (SQL) recip <xxxx@yyyy.com> whitelisted sender <...>, unexpected wb field value: "1"; reported by Carlos Horowicz; - added missing import of &ca to the amavisd-new-courier.patch; by Martin Orr; - produce a warning when there is an unknown field in the policy bank to be loaded; - with delivery method 'bsmtp:' prepend a directory $QUARANTINEDIR to the file path if not explicitly specified, to behave like the 'local:' delivery method, making it possible to hide full path from the X-Quarantine-Id and notifications; a patch by Thomas Jarosch; - pre-load SA 3.0.0 module Mail::SpamAssassin::Plugin::Hashcash to make it available in the chroot jail; - pre-load modules Mail::SpamAssassin::SpamCopURI and URI::* for SA older than 3.0.0; - enhancement to amavisd-nanny: when terminating a process and SIGTERM produces no result for some time, try SIGKILL; contributed by Philip Engdahl; --------------------------------------------------------------------------- August 15, 2004 amavisd-new-2.1.0 (amavisd-new-20040815) release notes The 2.1.0 release is mostly a maintenance release over 2.0, with only a handful of smaller features added. Based on a manual code audit the number of smaller internal code changes is rather extensive, some changes dealing with long-standing known deficiencies, minor bugs, documentation problems and typos. Only a few fixes are for new bugs introduced in 2.0. The files amavisd.conf, amavisd.conf-default and amavisd.conf-sample have been extensively reworked, with the hope to suit better the new installations, while possibly causing some head-scratching for existing installations when looking at a diff. The file amavisd.conf is the one that should serve as a sound base for the initial config file, while keeping an eye on the list of all variables and their defaults in amavisd.conf-default. The amavisd.conf-sample is being phased out of active maintenance, and should serve mostly as a set of examples and the source of documentation until better documentation is available. Two nice features are available: - the use of BerkeleyDB and libdb is now optional; controlled by variables $enable_db and $enable_global_cache; - a new program 'amavisd-nanny', with the accompanying instrumentation in amavisd, displays the general health of all amavisd child processes, reports crashed ones and attempts to kill long overdue processes; It is still experimental and minimalistic, problem reporting is currently only to stdout. Other changes: - incompatible change since 2.0: the use of BerkeleyDB is now off by default; The use of BerkeleyDB and libdb is made optional, controlled by variables $enable_db and $enable_global_cache, both false by default. $enable_db: enables the use of BerkeleyDB/libdb (for SNMP counters database and nanny, and optionally for cache); $enable_global_cache: enables the use of libdb-based cache when $enable_db is also true; If either the $enable_global_cache or $enable_db are false, cache of mail body MD5 digests is kept in child-local memory as in pre-2.0 versions, and is therefore local and short-lived, with lower expected cache hit rate; - incompatible change: DSPAM 3.0 or better is needed (if $dspam is enabled), no longer works with 2.x; - incompatible change: changed name of the (hardly ever needed) configuration variable auth_supported_out to auth_required_out, to better reflect its semantics (should be true if MTA server to which amavisd is sending notifications and forwarding mail requires authentication (AUTH smtp command)); - a new small program included: amavisd-nanny is a program to show the status and keep an eye on the health of amavisd child processes (experimental); - fixed a bug in lookup_acl where a "." did not act as a catchall; thanks to JP; - fixed a problem in SQL lookup which could return undef even when not all the matched records had NULL in the field; - fixed compatibility with old 'amavis' helper program ('delivery_care_of' defaulted to 'client', instead of depending on the presence of ldaargs), reported by Charlie Schluting and Christer; - fixed long standing problem in do_ascii, which could return without calling Convert::UUlib::CleanUp, occasionally spilling state into subsequent mail checks within the same process; - fixed macros %D, %O and %N when log_recip_templ is being expanded; a patch by Ed Walker; - fixed recognition of separators in a nested call during macro expansion; - pre-load missing modules Net::LDAP, Net::LDAP::Schema, Net::LDAP::Search, and Net::LDAP::RootDSE; suggested by Paul Jacobson; - fix locale-related bug in rfc2822 date generation, where we were restoring the saved LC_TIME value to LC_CTYPE (!); a patch by Henrique de Moraes Holschuh / Debian support team; - protect from signals while bdb cursor holds a lock; - new subroutine inherit_header_edits() and slight code restructuring makes possible for spam_scan() and other code before the final delivery to start submitting common header edits into $msginfo->header_edits, avoiding the need for passing them through global variables; - now a loglevel-based automatic syslog priority assignment can no longer lower a message syslog priority below the syslog priority specified in the $SYSLOG_LEVEL, it can only increase it; the violation of the least-surprise principle pointed out by Andy Dills; - a small optimization in logging: a new subroutine ll allows to save time in preparing complex log entries when we know their log level exceeds the current log level and won't be logged; - in default macro templates $log_templ and $log_recip_templ: * placed 'spam' condition before 'bad header' for consistency with program behaviour; * added reporting of tag/tag2/kill levels in $log_recip_templ (experimental: macro names may change in future versions); * rewritten templates using negation (i.e. [?x|1] ) to avoid unsightly selector nesting; - MIME decoding is now allowed to exceed the decoding quota, avoiding the problem when a small quota settings might not allow even a plain mail through; - override LC_TIME to "C" on every log message, to work around issues with Unix::Syslog, which would log stuff with the date stamp localized, which syslog would dislike and add its own, and the resulting mess is not recognized by amavis log processors; a patch by Henrique de Moraes Holschuh / Debian support; - changed dspam command line options to work with dspam 3.0 (no longer with 2.x), a patch by Ron Ohmer, Nabil Sefrioui, and Reech; - dspam header fields are now inserted into passed mail if all recipients are local; - supply better defaults for $hdrfrom_notify_sender, $hdrfrom_notify_recip, $hdrfrom_notify_admin and $hdrfrom_notify_spamadmin, similar to defaults from amavisd-new-20030616-p10; - when parsing output from the 'AVG grisoft' virus scanner, don't include CR in virus name; reported by Vernon A. Fort; - new file 'amavisd-new-qmqpqq.patch' provided by Martin Solciansky, (similar to fixes by Christopher Odenbach) making it work again with the current version of amavisd-new; - use lstat instead of stat, and test for soft links wherever appropriate; - remember inode and device number when creating temporary directory and temporary file, and test for change before removing them; - enhanced security: certain tainted values are allowed to enter deeper into program, untainting them only where and when really necessary; - avoid a taint problem in Mail::ClamAV; - added AV entry for CAI eTrust Antivirus; by Stephane Lentz (requires a suid shell wrapper around inocmd32); - added status 9 to the set of infected statuses for the drweb command line scanner (DrWeb Antivirus); - use our subroutine q_encode to Q-encode header fields from the notification templates, instead of MIME::Words::encode_mimeword (a similar fix in 2.0 applied only to encoding of modified headers in passing mail); - add attribute 'x-spam-type=original' to the Content-Type header field (the SpamAssassin's code to recognize an original email) when defanging spam, facilitating reporting spam via SA to other spam fighting tools; a patch by Brian May; - add a note to amavisd.conf that $sa_auto_whitelist has no effect on SA since 3.0.0 - SA has now a configuration file option 'use_auto_whitelist'; - turn off timer in post_process_request_hook() to avoid periodically recreating child processes on an idle machine; - added @mynetworks_maps and enhanced lookup_ip_acl() to take a list of lookup tables: currently members can be an array ref (as before), or a hash ref (new) or a plain constant (new); - generalized @debug_sender_acl into @debug_sender_maps along the lines of other lookup tables, and make it part of policy banks; - added @warnvirusrecip_maps, @warnbannedrecip_maps, @warnbadhrecip_maps; - added @spam_subject_tag_maps and @spam_subject_tag2_maps, to allow per-recipient spam tags string; suggested by Ed Walker; Note, there is an inconsistency in names of legacy variables and the new @*_maps, in an attempt to rectify an unfortunate choice of name for seldomly used variable $sa_spam_subject_tag1: @spam_subject_tag_maps = (\$sa_spam_subject_tag1); # exotic @spam_subject_tag2_maps = (\$sa_spam_subject_tag); # in common use Note also that corresponding SQL fields are 'spam_subject_tag' and 'spam_subject_tag2'; usually only the 'spam_subject_tag2' would be used; - added configuration variable $auth_reauthenticate_forwarded, which directs amavisd to apply its own credentials ($amavis_auth_user and $amavis_auth_pass) to unauthenticated forwarded (passed) mail, besides using them for submitting original messages (notifications and quarantine). This is similar to how mailing list managers are allowed to work (rfc2554). Note that the Perl module Net::SMTP in its current version is unable to specify the 'submitter' in its 'MAIL FROM' command, this should be rectified; --------------------------------------------------------------------------- July 1, 2004 amavisd-new-20040701 / amavisd-new-2.0 release notes MAJOR NEW FEATURES (since amavisd-new-20030616-p10): - security improvements: no shell required in chroot jail, checks performed to see if dropping privileges was successful, can drop privileges before reading config file; - the 'amavisd reload' command reimplemented, it now works even if running chroot-ed; - new feature: policy banks hold sets of configuration variables that may be switched with another predefined set based on incoming port number or original SMTP client IP address, avoiding the need to run more than one instance of amavisd daemon; - new feature: @score_sender_maps is a soft variant of black- and whitelisting; - extended semantics of SQL field wblist.wb for soft black/white-listing; - redesigned mail structure representation allows better control over 'banned' names and types; - MIME defanging can wrap the entire original mail in a MIME container; - more flexible control on lookups: configurable list of lookup tables observes the specified order of tables and permits arbitrary number of tables of any available type; - level-0 logging either by-message or by-recipient; - syslog priorities are now dynamically derived from the log level; - constantly updates a small database of SNMP-style counters, providing real-time measurements for status monitoring and statistics reporting, reducing the need for analyzing a log file; includes a sample/demo program 'amavisd-agent'; - new 'policy delegation protocol' between helper program and the daemon can pass more information to the daemon and allows the daemon to pass more instructions back to MTA (useful for sendmail milter setup); - persistent cache of recent virus and spam checks, common to all child processes, improves the cache hit rate; - support for IPv6 address formats; - provisional/experimental support for DSPAM spam checker; - support for ClamAV virus scanner via Perl module Mail::ClamAV; - cleaned amavisd.conf : amavisd.conf configuration file with the essentials amavisd.conf-default lists all configuration variables with their defaults amavisd.conf-sample traditional-style commented amavisd.conf with examples ---- INCOMPATIBLE CHANGES since amavisd-new-20030616 (any patch level) - requires Perl module BerkeleyDB with libdb version 3.1 or later (tested with db 4.1); This requirement will be made optional at a later date (in amavisd-new-2.1.0); - a directory at $db_home (default /var/amavis/db) must be manually created to store cache and snmp DB files. It should be writable by user running amavisd. The db files within are removed and re-created at each restart to avoid having to deal with db recovery (but need not be, as far as the program logic is concerned); - sending signal HUP in order to restart amavisd no longer works (previously it only worked in non-chrooted environment and relied on guessing amavisd absolute path); please use 'amavisd reload', or 'amavisd stop' and restart; If the HUP method is really still needed, please replace the line commandline => [], # disable by: commandline => ['/usr/local/sbin/amavisd','-c',$config_file], in file amavisd, adjusting the path if necessary. - due to changed names of temporary files, the old 'DrWebD' av entry needs to be replaced with the current one; - changed $final_virus_destiny default from D_BOUNCE to D_DISCARD - changed $final_spam_destiny default from D_REJECT to D_BOUNCE - changed defaults for variables $virus_quarantine_to, $banned_quarantine_to, $bad_header_quarantine_to and $spam_quarantine_to from undef (no quarantine) to values 'virus-quarantine', 'banned-quarantine', 'bad-header-quarantine' and 'spam-quarantine' respectively. Set them to undef or '' (empty string) to disable quarantine; - add address extension at tag2 level, not at kill level as before; suggested by Jacob Elder and others; - because of the reorganization of lookup tables, a new way of quickly disabling virus or spam checks in amavisd.conf is used. Instead of: # @bypass_virus_checks_acl= qw( . ); # uncomment to DISABLE anti-virus code # @bypass_spam_checks_acl = qw( . ); # uncomment to DISABLE anti-spam code the new recipe is: # @bypass_virus_checks_maps = (1); # uncomment to DISABLE anti-virus code # @bypass_spam_checks_maps = (1); # uncomment to DISABLE anti-spam code - @virus_admin_maps and @spam_admin_maps now take as lookup keys recipient addresses, not sender address as before. The new semantics was often requested, the old semantics was not useful because modern viruses and spam fake sender address, so the choice was made to incompatibly change semantics and use the same config variable names, rather than introduce new names and leave behind useless variables; An implication is that with multiple-recipient mail it is now possible to have more than one admin notification generated when recipients have different admin addresses associated. Still, each distinct admin address receives only one admin notification; - removed old compatibility variable $mailto. Use $virus_admin and $spam_admin variables instead, as suggested in amavisd-new-20021116 release notes, or the more recent inventions @virus_admin_maps and @spam_admin_maps, or their equivalents in policy banks; - removed support for old configuration variable $mailfrom. Use variables $mailfrom_notify_admin, $mailfrom_notify_sender, $mailfrom_notify_recip, and $mailfrom_notify_spamadmin as introduced in amavisd-new-20020630 for the purpose; - removed ancient variable @local_domains, use @local_domains_acl instead, or the more general @local_domains_maps - removed old compatibility code which allowed for a couple of traditional variables to treat value "no" as false. The use of "no" for false has been deprecated since amavisd-new-20021116. - revoke an old compatibility measure where a missing (undef) tag2 level would fall back to the kill level value; - lookup_hash incompatible change (but hardly anyone will notice): a key presence in the Perl hash but with undefined hash value used to be interpreted as true, but now it is treated as undef, causing search to continue with remaining lookup tables (if any). This is more general and more in spirit with other lookup mechanisms; - changed a default to initialize SAVI-Perl every time a child process is started, no longer at master process startup time only. This is to avoid the need to restart amavisd every time the Sophos IDE database is changed. One can revert to the previous behaviour by uncommenting a call to Amavis::AV::sophos_savi_init in subroutine fetch_modules_extra; SECURITY - no longer invoke shell to call gzip for compressing quarantined files or to call the sendmail command for submitting messages; the most important consequence is that a shell is no longer needed in a chroot environment and should preferably be removed; - not to forget what was introduced in -p10: inserted a security check for a missing Net::Server patch, and abort if vulnerable; - new command line options '-u user' and '-g group' are available. These are pretty much equivalent to doing a su(1) to the specified user first (in which case the use of these options is redundant). By doing 'su' or by specifying a command-line option '-u username' one can prevent a potential security risk on misconfigured sites where amavisd.conf is writable by UID running amavisd (e.g. not owned by root). If a (non-root) username or UID is specified, privileges are now dropped _before_ opening and evaluating a configuration file. The consequence is that the configuration variables $daemon_user and $daemon_group (in amavisd.conf) can not have an after-effect (a warning is issued if different). If -u is not specified, the behaviour is as before, i.e. the config file is read and evaluated under the current UID (as root unless 'su' was done), and the values of $daemon_user and $daemon_group from the config file are passed to Net::Server, which changes UID during its startup after chroot-ing (if requested). If chroot is desired, the -u must not be used: the root privilege is required to do chroot, and the config file must be read _before_ doing chroot. A case of Catch-22. Be doubly careful of who can modify the configuration file. Another consequence of specifying -u is that any external files (e.g. templates, lookup hashes) as possibly read from amavisd.conf, are now accessed as unprivileged user and no longer as root. The same goes for opening the log file when not logging via syslog. MAJOR NEW FEATURES - policy banks hold sets of configuration variables controlling most of per-message settings, including: static lookup tables, IP interface access rules, forwarding address, log level, templates, administrator addresses, spam trigger levels, quarantine rules, lists of anti-virus scanner entries (or just a subset), banned names rules, defang settings, etc. The whole set of these settings may be replaced with another predefined set based on incoming port number, making it possible for one amavisd daemon to cope with more diverse needs of served user communities which could so far only be implemented by running more than one instance of the amavisd daemon, each with its own configuration file; This mechanism brings new potentials for the future: in principle policy banks could be swapped not only based on port number or SMTP client IP address, but on any characteristics pertaining to a mail message as a whole (not specific to each of its recipients), or to characteristics of a connection from a mailer (e.g. the interface address or protocol); Until a better mechanism is available, a policy bank named 'MYNETS' has special semantics: this policy bank gets loaded whenever MTA supplies a SMTP client IP address (Postfix XFORWARD extension or a new AM.PDP protocol) and that address matches the @mynetworks list. A hash %$interface_policy is a current mechanism of assigning a policy bank to an incoming TCP port number (port must be in the list @$inet_socket_port). Whenever the connection from MTA is received, first a built-in policy bank with an empty name - the $policy_bank{''} gets loaded, which bringings in all the global/legacy settings. Then it is overlaid by whatever configuration settings are in the bank named in the $interface_policy{$port} if any, and finally the bank 'MYNETS' is overlaid if it exists and the SMTP client IP address is known (by XFORWARD SMTP extension command from MTA) and it matches @mynetworks. See amavisd.conf-sample for examples. When a new policy bank is overlaid over an existing set of configuration variables, variables not present in the new policy bank retain their value. The built-in policy bank (with empty name) is predefined, and includes references to most other variables (the dynamic config variables), which are accessed only indirectly through the currently installed policy bank. Overlaying a policy bank with another policy bank may bring in references to entirely different variables, possibly unnamed. Configuration variables are referenced from a built-in policy bank (which is implemented as a perl hash, i.e. an associative array) by keys of the same name, e.g. { log_level => \$log_level, inet_acl => \@inet_acl, ...}. For scalars one level of indirection is allowed, e.g. a policy bank { log_level => \$log_level }; $log_level=2; is equivalent to { log_level => $log_level } or { log_level => 2 }, but in the first example with an indirect reference, the $log_level may be assigned to even _after_ the policy bank has already been formed. A word of caution: the syntax of entries within a policy bank hash is slightly different from assignments to configuration variables. This is because entries within policy bank are not assignments, but key=>value pairs as in any Perl hash. And these pairs are delimited by commas, unlike statements, which are delimited by semicolons. Value is separated from its key by '=>' (or by a comma), whereas the assignment operator is '='. Keys of a policy bank are without leading $ or @ or %, unlike variable names. Values of a hash can only be scalars (e.g. strings or references). Compare: - value of a policy bank is a reference to a Perl hash, e.g.: { log_level => 3, forward_method => 'smtp:[127.0.0.1]:10025', spam_admin_maps => ["spamalert\@$mydomain"], } - normal assignments look like: $log_level = 3; $forward_method = 'smtp:[127.0.0.1]:10025'; @spam_admin_maps = ("spamalert\@$mydomain"); And a final note: Perl can detect and report typing mistakes in variable names, but mistyped key is just some unused hash entry lurking in a hash, never used and never reported as mistyped/useless. - @score_sender_maps is a soft variant of black- and white-listing, contributing positive or negative score points based on sender e-mail address. Btw, the 'score' in the name '@score_sender_maps' is meant as a verb, recipient is scoring a sender (= to grade, to determine the merit of); Whitelisting is becoming deprecated because it is often and easily abused (but blacklisting can still be useful); # ENVELOPE SENDER SOFT-WHITELISTING / SOFT-BLACKLISTING # Instead of hard black- or whitelisting, a softer approach is to add # score points (penalties) to the SA score for mail from certain senders. # Positive points lean towards blacklisting, negative towards whitelisting. # This is much like adding SA rules or using its white/blacklisting, except # that here only envelope sender addresses are considered (not addresses # in a mail header), and that score points can be assigned per-recipient # (or globally), and the assigned penalties are customarily much lower # than the default SA white/blacklisting score. # # The table structure is similar to $per_recip_blacklist_sender_lookup_tables # i.e. the first level key is recipient, pointing to by-sender lookup tables. # The essential difference is that scores from _all_ matching by-recipient # lookups (not just the first that matches) are summed to give the final # score boost. That means that both the site and domain administrators, # as well as the recipient can have a say on the final score. - the MIME defanging (defang: to make harmless or less powerful) wraps the entire original mail in a MIME container of type 'Content-type: multipart/mixed', where the first part is a text/plain with a short explanation, and the second part is a complete original mail, enclosed in a 'Content-type: message/rfc822' MIME part. Defanging is only done when enabled (selectively by malware type) and the malware is allowed to pass (*_lovers or *_destiny=D_PASS). The feature is global, i.e. not available on a per-recipient basis. Conventional mail header fields are retained, and header fields Resent-{From,Date,Message-ID} are added. A header field X-Amavis-Modified is inserted to indicate that the mail body has been modified. Note that defanging changes mail body and makes subsequent DomainKeys/DKIM verifications to fail. The DomainKey-Signature header field is not retained in defanged mail (but is retained in the attached original mail). It is an experimental feature (disabled by default, except for mail bombs), and not very efficient for large mail. MIME defanging is unconditionally done for mail bombs, i.e. when the X-Amavis-Hold header field is added. The text in the first MIME part describes the reason, e.g.: WARNING, possible mail bomb, NOT CHECKED FOR VIRUSES: Exceeded storage quota 29089500 bytes by do_unzip; ... When MIME defanging is enabled for passed spam, the first MIME part contains the full SpamAssassin report. The MIME defanging feature is not available in the sendmail milter setup. - new parameter: @mynetworks It is an IP access list which determines if the original SMTP client IP address belongs to our internal networks. It is much like the Postfix parameter 'mynetworks' in semantics and similar in syntax, and its value should normally match its Postfix counterpart. It affects the value of a macro %l (=sender-is-local), and the loading of policy 'MYNETS' (if present). Note that '-o smtp_send_xforward_command=yes' (or its lmtp counterpart) must be enabled in the Postfix service that feeds amavisd, otherwise client IP address is not available to amavisd-new and new features based on @mynetworks do not work (the %l macro works as before and bases its decision on sender e-mail address matching local_domains); LOGGING - syslog priorities are now dynamically derived from the log level of each log message (the first argument of sub do_log). The priority as specified in the $SYSLOG_LEVEL configuration variable is ignored in 2.0 (no longer in 2.1), but the 'facility' is not ignored. This makes it possible to influence the log verbosity by syslog.conf settings. Here is an example of a useful syslogd.conf setting (some levels may be left out; assumes the $log_level is high enough, e.g. 2, to produce any low-priority log messages at all): mail.err /var/log/messages mail.notice /var/log/amavisd.log mail.info /var/log/amavisd-info.log mail.debug /var/log/amavisd-debug.log (On linux do not forget to prefix filenames in syslog.conf with a '-' to disable sync, which has much impact on syslog performance!) At the moment the mapping of message log levels to syslog priorities is hardwired: level <= -3: LOG_CRIT level <= -2: LOG_ERR level <= -1: LOG_WARNING level <= 0: LOG_NOTICE level <= 1: LOG_INFO (in version 2.0) level <= 2: LOG_INFO (since 2.1) else: LOG_DEBUG Not to be confused with the $log_level configuration variable setting, which still works as before, suppressing generation of all log messages with log levels above $log_level. For efficiency reasons one should not specify unnecessarily high $log_level and then discard low syslog priority messages in the syslogd. - added $log_recip_templ variable, which is similar to $log_templ, but gets called for every recipient (the $log_templ is evaluated only once per message). Normally one or the other log template should be disabled by assigning undef or an empty string to the corresponding variable; - the default $log_templ no longer shows quarantine ID if quarantining is disabled; - added a macro %. (a dot), which might be useful in the $log_recip_templ; Its value is empty when $log_templ is expanded, and is a recipient counter (starting by 1) when $log_recip_templ template is expanded. Based on this macro one can provide a single template for both the $log_templ and the $log_recip_templ if needed, or perhaps let the log entry for the first recipient be more verbose that the rest; - added a macro %T which expands to a list of triggered SA tests, but only when $log_templ and $log_recip_templ are expanded. In notifications the %T is still a list of To: addresses. An overlaid semantics, but we are running out of letters and a macro expander rewrite would be needed; - new macros %k, %1, %2, %O, please see README.customize LOOKUPS - new configuration variables make it more flexible to specify arbitrary list of lookup tables. Legacy configuration variables are still available and are referenced from the default values of @*_maps lists. If these lists are redefined, legacy variables are not used. The new variables (lists) are: @local_domains_maps @bypass_virus_checks_maps @bypass_spam_checks_maps @bypass_banned_checks_maps @bypass_header_checks_maps @virus_lovers_maps @spam_lovers_maps @banned_files_lovers_maps @bad_header_lovers_maps @virus_admin_maps @spam_admin_maps @virus_quarantine_to_maps @banned_quarantine_to_maps @bad_header_quarantine_to_maps @spam_quarantine_to_maps @spam_quarantine_bysender_to_maps @banned_filename_maps @spam_tag_level_maps @spam_tag2_level_maps @spam_kill_level_maps @spam_dsn_cutoff_level_maps @spam_modifies_subj_maps @whitelist_sender_maps @blacklist_sender_maps @score_sender_maps @addr_extension_virus_maps @addr_extension_spam_maps @addr_extension_banned_maps @addr_extension_bad_header_maps @keep_decoded_original_maps @map_full_type_to_short_type_maps @viruses_that_fake_sender_maps - for more informative logging of lookup operations, a new object type Amavis::Lookup::Label can be inserted to lists of lookup tables for the purpose of labeling the main purpose of the list; - all lookup* subroutines can now return matching key when called in a list context; - lookups can now return a list of all matching entries (not just the first match); used for the new soft- white/blacklists (@score_sender_maps); - sub lookup() now allows for one level of list elements dereferencing, which makes possible the construction of the argument list and later still be able to modify its members (e.g. creation of regexp lookup table objects in the configuration file); It facilitates transition from old hard-wired lists of lookup tables to new @*_maps list variables which permits specifying an arbitrary number of lookup tables and to specify their search order; - simplify and unify calls to lookup() by collecting arguments (references to lookup tables) in lists, e.g. @local_domains_maps, @virus_lovers_maps, @virus_admin_maps. These array variables default to lists of legacy variables, which are now never directly used by the program. Either the individual legacy variables may be assigned to, or the entire list replaced, in which case the legacy variables no longer have any effect. - lookup_acl: respect $localpart_is_case_sensitive setting; - lookup_hash and lookup_sql: rewritten lookup_hash and factored out the common code from lookup_hash and lookup_sql to make_query_keys(); - lookup_hash bug fix: avoid splitting address literal as if it were a domain name; (a bug with key '.' not being tried for address literals fixed thanks to Uwe S. Fuerst); SQL, LDAP LOOKUPS - extended semantics of SQL field wblist.wb, which can hold a score value boost, which is interpreted as soft black/white-listing (the same semantics as the value in @score_sender_maps); - recognize SQL server error 2013/"Lost connection to" and treat it the same as 2006/"MySQL server has gone away"; by Max Kalika; - full domain stripping: @.sub.example.com @.example.com @.com @. - lookup_hash and lookup_sql: limit the list of subdomain search keys to 10 levels as a sanity measure; e.g. for address user@14.13.12.11.10.9.8.7.6.5.4.3.2.com the subdomains keys search list starts at .9.8.7.6.5.4.3.2.com; (domain names are limited by standard to 127 levels); - prepare SELECT statements on demand; - recognize all-zero and all-null boolean fields as false; - recognize new (optional) fields in the table 'policy': spam_dsn_cutoff_level virus_quarantine_to, banned_quarantine_to, bad_header_quarantine_to addr_extension_virus, addr_extension_spam, addr_extension_banned, addr_extension_bad_header (the addition of virus_quarantine_to was suggested independently by Harald Kapper and by Dipl.Ing. Martin Boeck); - consider the SQL user id a string (no longer required to be numeric); thanks to Max Kalika / Gentoo support; - LDAP white/black list support by Jacques Supcik (similar to contribution from Scott Dier and Eric Dorland, which I forgot about, sorry); - added amavisSpamQuarantineTo to the LDAP schema; new version of LDAP schema; by Jacques Supcik: In the previous schema, the tag levels have been defined as integer. This is too restrictive, and have now been changed to strings (there is no float type in LDAP); - added LDAP attributes for completeness: amavisBadHeaderLover, amavisBypassBannedChecks, amavisBypassHeaderChecks, amavisVirusQuarantineTo, amavisBannedQuarantineTo, amavisBadHeaderQuarantineTo; by Jérôme Schell; DECODING / DEARCHIVING - provide optional ability to retain complete email message in its un-decoded form (alongside its decoded parts) for a virus scanner to see (enabled if $keep_decoded_original_re matched string 'MAIL'); suggested by Tomasz Papszun; (partly backported to amavisd-new-20030616-p8); - rewrite code that generates new file names (Amavis::Unpackers::Part->new), and rewrite code dealing with banned names. Keep information about each part organized as a tree, matching the descendence of each part, new package/object Amavis::Unpackers::Part to collect such information; - make Amavis::Unpackers an optional module: the interface code to external decoder/unpackers/dearchivers does not get compiled and does not consume virtual memory if $bypass_decode_parts is true; (previously it just didn't get called, but was sitting in memory nevertheless); - decode RPM archives by converting them to cpio, if rpm2cpio and cpio are available; - do_tnef: extract $tnef->message if it exists, not just $tnef->attachments; - support extracting MS cabinet files (CAB) by calling cabextract, if enabled and found. Beware: Lars Hecking warns that cabextract 0.6 is quite buggy and the author has been notified. VIRUS SCANNING - support for ClamAV virus scanner via Perl module Mail::ClamAV, based on code by Roberto Pereira da Rosa; - don't call virus scanners if there are no files in the directory to be scanned (e.g. mail with an empty body); some virus scanners don't like to be given an empty directory (e.g. Symantec newer savsecls); reported by Marco Bicca; - rewritten/unified/generalized subroutines ask_daemon and sophos_savi based on the new subroutine ask_av; - scan parts directory for file names exactly once regardless of the number of virus scanners and their arguments (containing '{}/*' or not); - supply full original mail to virus scanners in case of MIME parse errors (in addition to any possibly decoded parts); - when collecting file names to be virus scanned, prepare a hash which maps base file names to Amavis::Unpackers::Part objects, and make it available to virus scanner interface routines, which may benefit from the additional information about the file to be scanned. In particular, the new interface to Mail::ClamAV now turns on the option CL_MAIL, and the interface to SAVI turns on the option MIME, when entire mail is passed to AV scanner for checking. This enables ClamAV and SAVI to attempt MIME decoding the file by itself. TODO: The same option (--mime) would need to be specified when calling 'clamscan' and supplying a non-decoded mail for checking; pointed out by Riccardo Ghiglianovich and Michael Boelen; OTHER EXTERNALLY VISIBLE CHANGES - The 'amavisd reload' command is now implemented differently: old: signals SIGHUP to a running amavisd process and exits immediately; the running amavisd process (under control of Net::Server) when it receives a SIGHUP starts its own copy with same arguments and exits; new: signals SIGTERM to a running amavisd process, waits for it to finish, then continues (same as 'amavisd start') to become a new daemon; The new method works even when chrooted, and is more reliable when the existing process is slow to terminate, as it actively waits for the previous daemon to finish before proceeding to promote itself to become a new daemon. - a simple demo program 'amavisd-agent' is provided, allowing for continuous inspection of SNMP counters; a path to the /var/amavis/db is hardwired in the program, modify it if necessary. - server-side support for optional Postfix SMTP/LMTP command XFORWARD: information about the original SMTP client IP, its DNS name, HELO name and protocol used is now made available to the amavisd program for logging and other purposes. The same information can also be obtained from the 'Amavis policy delegation protocol (AM.PDP)' if the helper program supports it (useful for sendmail milter setup); - client-side support for optional Postfix SMTP command XFORWARD: if MTA announces in its SMTP EHLO response that it supports XFORWARD, amavisd will provide additional information about the original SMTP client if the information is available (either from XFORWARD on the receiving side, or by the 'Amavis policy delegation protocol'; - server side support for the new amavis helper protocol AM.PDP which allows for header modifications, removal of recipient addresses (e.g. non- spam lovers) or rewrite of recipient addresses (e.g. adding address extensions), and specification of full SMTP response; - modified search logic for matching mail parts against $banned_filename_re; The old search order for names did not result in what one might expect when pattern list elements with a value of false were used in $banned_filename_re. Namely, all three components were searched independently (Content-Type, declared name, and file(1) type) and a logical or was used. Because searches for each mail part were independent, it was not possible to specify for example that anything within a zip would be allowed. If any of these searches returned true, mail was blocked. To make this useful, a complete rewrite of mail unpacking was needed and all information be made available in one place after the unpacking is over, so that checking for banned names can be done all at once. The search order is now much the same as used in rsync and its server, see man rsync, section 'EXCLUDE PATTERNS'. The new comments in amavisd.conf-sample explain the new situation. - replaced $relayhost_is_client by a more flexible specification: To make it possible for several hosts to share one content checking daemon, the IP address and/or the port number in $forward_method and $notify_method may be specified as an asterisk. An asterisk in the colon-separated second field (host) will be replaced by the SMTP client peer address (i.e. the MTA host). An asterisk in the third field (tcp port) will be replaced by the incoming SMTP/LMTP session port number plus one. This obsoletes the previously used less flexible configuration parameter $relayhost_is_client. An example: $forward_method = 'smtp:*:*'; $notify_method = 'smtp:[127.0.0.1]:10025'; The same functionality can also by achieved by using a bigger hammer, the policy banks. These may completely replace the global settings for $forward_method and $notify_method, based on incoming port number; - turn address extension variables (the so called "plus addressing") into recipient-based lookup tables, including the SQL lookups. For example: @addr_extension_virus_maps = ('virus'); # defaults to empty @addr_extension_spam_maps = ('spam'); # defaults to empty @addr_extension_banned_maps = ('banned'); # defaults to empty @addr_extension_bad_header_maps = ('badh'); # defaults to empty or perhaps: @addr_extension_virus_maps = ( {'sub.example.com'=>'infected', '.example.com'=>'malware'}, 'virus' ); suggested by Gentoo modification, Jacques Supcik, and others; - log and report hits and tag/tag2/kill levels rounded to three decimal places (trailing zeroes trimmed), no longer rounded to one decimal place; - added @spam_dsn_cutoff_level_maps, making it possible to specify different DSN cutoff levels for different recipient domains or users. In multi-recipient messages where recipients can specify different values, the maximum value is used for deciding whether DSN should be suppressed; suggested by Ales Casar; - configuration variable $gets_addr_in_quoted_form is no longer used; knowledge about address form (quoted or not) is now implicit in the receiving protocol; - if tag level turns out to be undef, it will not be shown in X-Spam-* header fields, and will be interpreted as having a value lower than any spam score when deciding whether to insert X-Spam-* header fields or not; - added macros %a and %g: * %a original SMTP session client IP address (empty if unknown) * %g original SMTP session client DNS name (empty if unknown) (like macros %I and %M that were once proposed by Dibo in his 2002-07 patch) This information may be available from Postfix when XFORWARD protocol extension to SMTP is enabled, and can be made available by helper program (e.g. from sendmail milter) when the new AM.PDP protocol is used; - added macro %p, expanding to a current policy bank name (or empty if a built-in policy bank is still in place); - added macro %r, expanding to the contents of the first Resent-Message-ID header field, or empty if no such field exists. Include reporting the Resent-Message-ID in the log and in the sender notification; suggested by Oliver Gorwits; - new configuration variable $addr_extension_bad_header for completeness; - added $bad_header_quarantine_to, @bad_header_quarantine_to_maps, $bad_header_quarantine_method, and $warnbadhrecip for completeness; suggested by Robin Lynn Frank; - MIME::Parser errors now contribute to bad-header checks, so that the header checking is now conceptually extended to MIME sub headers (Postfix similarly considers MIME subheaders part of mail headers); MIME::Parser 6.1xx or later is recommended. - allow $*_quarantine_method to be undef as a quick way of disabling some quarantine (it also can be disabled as previously, by using method 'local:' and following its rules); - persistent cache of recent virus and spam checks, global to all child processes, can improve the cache hit rate. Uses BerkeleyDB database (hash and queue) and its interlocking mechanisms (Berkeley DB Concurrent Data Store) for the purpose. The V3.1 or better is required, V4 is recommended. - include version information in the 'Usage' text; - rewritten lookup_ip_acl() and added ip_to_vec() to allow for IPv6 address syntax as specified in rfc3513 to be used in IP lookups; - @inet_acl now defaults to ('127.0.0.1', '::1'), i.e. it adds the IPv6 loopback address to the list; - new configuration variable $sa_spam_level_char (defaults to '*') allows specifying another character for X-Spam-Level bar. Empty or undefined value disables inserting the X-Spam-Level header field; - added configuration variable $sa_spam_report_header to enable/disable inserting the X-Spam-Report header; patch by Craig Sanders; - added $banned_quarantine_to configuration setting to make possible the quarantining of banned mail to a different place from viruses; - don't insert virus-, banned- and bad headers- related headers for passed mail to recipients with corresponding bypass_*_checks, making them believe the mail was not spam-checked (as they are not expecting such headers anyway); This was already done in version amavisd-new-20030616-p6 but only for spam-related headers; - for choosing address extensions use the same criteria as for adding header fields, e.g. pretend to not know the result of a certain test (virus, spam, ...) when recipient chooses to bypass such test, even if the result of the test is known; - added variable $sa_spam_subject_tag1 (undef by default). If $sa_spam_subject_tag will not be inserted (at tag2 level), and $sa_spam_subject_tag1 is nonempty, this string (e.g. '***possible SPAM*** ') will be inserted into the Subject header field for spam levels above tag level; suggested by Immo Goltz; - added separate configuration variables $banned_files_quarantine_method and $bad_header_quarantine_method. Quarantining of banned files and bad headers were previously controlled by $virus_quarantine_method; - rewritten read_hash, it is now possible to specify key value (right-hand side) for each key. If value is not specified, a '1' is assumed as before; - SMTP server support for rfc2554 authentications (PLAIN and LOGIN only) allows client authentication to be relayed to the MTA when message is forwarded. Might be useful if amavisd-new is used in a Postfix SMTP proxy setup, but is not needed for other setups. Disabled by default, see variables $auth_required_inp and @auth_mech_avail. - SMTP client support for rfc2554 authentications (any authentication method as supported by Net::SMTP and Authen::SASL Perl modules). Authentication of forwarded mail (PLAIN and LOGIN only) is carried over from the incoming mail, authentication to be used when submitting notifications is controlled by configuration variables $amavis_auth_user and $amavis_auth_pass. Disabled by default, see variable $auth_supported_out (later renamed to $auth_required_out). - when passing envelope sender address to SpamAssassin, supply it as a rfc2822-standard header field Return-Path, and no longer as X-Envelope-From (the change came with a pre-release amavisd-new-20040301). - provisional/experimental support for DSPAM spam checker (pre 3.0): if configuration variable $dspam is nonempty and represents a path to a 'dspam' program, a message is passed to dspam and its inserted headers of the form X-DSPAM-* are axtracted and then made available for SpamAssassin rules to check and score if desired. All messages are currently presented to dspam as the same user, affecting how its database is being built. False negatives and false positives (based on SA assessment) are fed back into DSPAM as a simple form of auto-learning. Works reasonably well, but do not expect miracles. See subroutine spam_scan. Here is how DSPAM can be installed to be able to be used by amavisd-new: dspam 2.x: ./configure --enable-alternative-bayesian \ --with-userdir=/var/amavis/dspam --enable-signature-headers \ --without-local-delivery-agent --without-quarantine-agent dspam 3.0.0: ./configure \ --with-dspam-home=/var/amavis/dspam --enable-signature-headers \ --without-delivery-agent --without-quarantine-agent make install chmod u-s,a+rx /usr/local/bin/dspam chown vscan:vscan /var/amavis/dspam User 'vscan' may need to be added to DSPAM trusted.users file. The following can be inserted into the SA config file (local.cf) to make it recognize and incorporate DSPAM's assessment: header DSPAM_SPAM X-DSPAM-Result =~ /^Spam$/m describe DSPAM_SPAM DSPAM claims it is spam score DSPAM_SPAM 0.5 header DSPAM_HAM X-DSPAM-Result =~ /^Innocent$/m describe DSPAM_HAM DSPAM claims it is ham score DSPAM_HAM -0.1 Eventually DSPAM support should be removed from amavisd-new, as soon as SA will be able to call it on its own. INTERNAL CHANGES - reformatted the whole program, reducing indentation from 4 to 2 and replacing TABs with spaces (with some dubious help from perltidy, plus lots of manual adjustments); - completely rewritten code to handle both the old and the new amavis helper protocol, as well as Postfix 'TCP client/server table lookup protocol' as specified in the Postfix documentation: tcp_table(5); (process_policy_request, prepare_policy_query, check_amcl_policy) - type_short may now be a list of short types, not necessarily just a single value. Typical use is to classify a MS executable as both an 'exe' and as 'exe-ms', which makes more specific banned rules possible without unnecessary complication in regexps; - parts now carry attributes, which can be inspected for banned checks; current attributes are U for undecodable, and C for (en)crypted; - opened another can of Perl worms (taint bugs): turn on Perl pragma "use re 'taint'" in all modules, and selectively turn it off where needed. It replaces cumbersome manual preservation of taintedness when regexp saved ranges are used without intention to untaint. Because of Perl bugs, strategically placed local($1,$2,...) are needed, otherwise previous taint flag in $1, $2, ... can be brought on to new variables, which can all of a sudden become tainted out of nowhere; - catch and report throws (die) in pre_loop_hook() to properly report problems during initialization; - introduced subroutine exit_status_str and unify reporting of subprocess status; - enhanced sub best_try_originator to ignore IP addresses from private, local and dedicated IANA networks (rfc3330) and look for the first public address in the 'Received' path; - examine first four (chronologically) Received header fields (instead of first two) when looking for an originator IP address, and ignore those with private IP addresses; - moved code dealing with body digest cache to a new package Amavis::Cache to facilitate transition to shared or persistent cache; - new explicit cache expiration time controls (time to live in seconds): $virus_check_negative_ttl $virus_check_positive_ttl $spam_check_negative_ttl $spam_check_positive_ttl $cache_entry_ttl - discard cache db ($db_home) and recreate it during restart; - more informative changes of child process $0, which may show in the ps(1) output; - store tempdir of a current message to the Amavis::In::Message object; - gather some statistics about idle time; - reorder and adjust mapping from file(1) results to file type classes; - optimization: instead of invoking file(1) utility program for each part to be analyzed, now call it once for each round of currently available parts, giving it the list of all available parts as arguments; - shorten the names of parts from part-..... to p..., to be able to stash more files names into a command line, e.g. when calling file(1) or external virus checker which can not deal with directories; - use regexp lookup table mechanism (table $map_full_type_to_short_type) to match long types (output of file(1) utility) to short types (.exe, .jpg, .doc, ...). The default table can be replaced by a customized table in amavisd.conf; - replace $(?!\n) with \z in regular expressions throughout; replace most of the remaining $ with \z in regular expressions. The regular expression primitive \z is available since perl 5.005. - TODO: disregard $MAXFILES during initial MIME unpacking; reported by Stephane Lentz and Robert LeBlanc (done in 2.3.2); Some un-edited notes on the new banned rules mechanism: (wrapped log lines, and replaced \\ by \ for clarity: | Feb 24 19:07:29 hauptpostamt amavis[29847]: (29847-04-5) p.path BANNED: | | "P=p002,M=application/octet-stream,T=zip,N=document.zip | | P=p003,T=exe,T=exe-ms,N=document.htm .scr", part p003 is of type (file(1)) MS executable, with suggested name "document.htm .scr" (lots of spaces in the name) its parent resides on temp file p002 (i.e. p003 was extracted from it), which is of type (T) zip archive, with suggested (MIME) name (N) "document.zip", and has a MIME type (M) "application/octet-stream". such a component p003 lying within such p002 is considered banned by the following regexp rule (one rule within the $banned_namepath_re list): | matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^./\t\n]* \. | (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)" which says that any component at any level must not have a name (N) matching a pattern: any number of characters, a dot, any number of non-dot and non-slash characters a dot, and ending with: exe or vbs ... (basically: double extension ending with listed extensions) The complications such as using [^\t\n]* instead of .* are there to keep regexp contained within fields and ancestors/descendents. There is one detail to remember when comparing logged p.path log entries and the actual matching rules: - for the sake of readability the logged entry has \n (newlines) converted into ' | '. The \n is a separator between components in the tree from the root (the mail itself, hidden) to the leaf component which can not be further expanded (i.e. not an archive) - for the sake of readability the logged entry has \t (a tab) converted into comma, separating information fields such as P=... M=.. T=.. M=.. So the above logged string: P=p002,M=appl...,T=zip,N=document.zip | P=p003,T=exe,T=exe-ms,N=document.htm.scr is actually a single string: P=p002\tM=appl...\tT=zip\tN=document.zip\nP=p003\tT=exe\tT=exe-ms\tN=document.htm.scr and a Perl regexp is applied directly to it. The raw string is rather unsightly, but the \n and \t were chosen to minimize clash with valid characters within file names. If a \n or \t is present in a name of the components, such character is converted into a space to avoid clashing with separators. | Feb 24 19:11:58 hauptpostamt amavis[31505]: (31505-01-5) p.path BANNED: | "P=p002,M=application/octet-stream,T=zip,N=paypal.zip | | P=p003,T=exe,T=exe-ms,N=paypal.scr", a MS executable named "paypal.scr" within a zip archive "paypal.zip" | matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. | (exe|vbs|pif|scr|bat|com) (\t.*)? $)" block component at any level with a name (N) terminating by dot followed by any of the listed extensions. | Feb 24 19:18:25 hauptpostamt amavis[32159]: (32159-01-2) p.path BANNED: | "P=p002,M=application/octet-stream,T=zip,N=text.zip | | P=p003,T=exe,T=exe-ms,N=text.txt .exe", a MS executable named "text.txt .exe" (with lots of spaces in the name) within a zip archive named "text.zip" | matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^.\t\n]* \.\n | (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)" blocked by the double-extension rule. | Feb 24 19:30:15 hauptpostamt amavis[1690]: (01690-02-8) p.path BANNED: | "P=p002,M=application/octet-stream,T=zip,N=jokes.zip | | P=p003,T=exe,T=exe-ms,N=jokes.doc .exe", | matching_key="(?mix-s:^ (.*\t)? N= [^\t\n]* \. [^.\t\n]* \.\n | (exe|vbs|pif|scr|bat|cmd|com|dll) (\t.*)? $)" same thing --------------------------- --------------------------------------------------------------------------- June 29, 2004 Patch: amavisd-new-20030616-p10 - insert a security check to test for a missing Net::Server patch, and abort if vulnerable; - provide and use our own subroutine q_encode to do the Q-encoding when editing an existing invalid header field with non-encoded 8-bit characters, e.g. when inserting ***SPAM*** or ***UNCHECKED*** into Subject. The MIME::Words::encode_mimeword() does not encode spaces and does not limit encoded words to 75 characters, which violates the RFC 2047 and breaks mail readers; reported by Sebastian Hagedorn and Gregor Hoffleit; - fixed a bug in inserting the tag_level header field, which was missing if sender was blacklisted and tag_level was greater or equal to 0; thanks to Joerg Thaler; - amavis-milter.c * log envelope sender address at the same log level (DBG_INFO) as recipient addresses; * remove the log message "(mlfi_eom) header already present", it was inappropriate, the call to smfi_chgheader succeeds even if no such header was already present; * relax permissions on created directory and temporary file to allow group read access (needed if virus scanner runs under a different user id but within the same group); by Adam C. Migus; * add queue id (when available) to most log messages; - do not preserve evidence just because a message gets an X-Amavis-Hold header field; - when logging directly to a file and started as root, change UID of a log file to $daemon_user to avoid restart problems; based on a patch by Carsten Hoeger and Gregor Weiss; - added configuration variable $first_infected_stops_scan to stop anti-virus scanning when the first scanner detects a virus; the default is false, all scanners in a group are called (as usual); - in the AV entry for clamscan added the option '--tempdir=$TEMPBASE'; - in the AV entry for 'Norman Virus Control v5 / Linux' changed the command name 'nvccmd' into 'nvcc'; correction by Michael Ramke of the Norman Data Defense Systems GmbH; - insert debug reports into sub ip_addr_to_name to help recognize DNS problems; - revoke deleting an existing 'X-Scanned-By' header field, which was introduced in -p8; removing it gets in a way when more than one content filter is being chained; - load a missing SpamAssassin v3.0 module, needed when running in chroot; - contributed (for now in the form of an optional patch): support for the incoming qmqpqq protocol over a TCP socket - to be used with qmail. Apply the provided patch 'amavisd-new-qmqpqq.patch', it updates files amavisd.conf and amavisd in the current directory; by Martin Solciansky; - documentation updates; --------------------------------------------------------------------------- April 2, 2004 Patch: amavisd-new-20030616-p9 - avoid choking on undefined $banned_filename_re; thanks to Ales Casar, Sebastian Hagedorn and E. Falk; - if Subject mail header field got $undecipherable_subject_tag inserted, it would also receive the spam tag $sa_spam_subject_tag unconditionally; fixed, thanks to Francis Stevens; - updates to the @av_scanners list in amavisd.conf: * DrWebD now works with a new DrWeb daemon 4.31, thanks to information provided by Krzysztof Cegielski, DrWeb Polska; * updated BitDefender bdc, thanks to Alfredo Milani Comparetti; * clamscan: use documented option --no-summary instead of the undocumented/old(?) option --disable-summary; by Georgy Salnikov; * updated Kaspersky aveclient and F-Secure fsav, thanks to Tomi Hakala; * recognize that KasperskyLab antivirus in demo mode turns on the bit 0x10 in status; avp and avpdc now use the same two sets of status codes; avp: statuses 3 and 6 moved to infected (to match avpdc); statuses 2 and 5 considered infected; all suggested by Georgy Salnikov; - when original undecoded mail is to be kept for virus scanners (requested by patterns /^MAIL$/ or /^MAIL-UNDECIPHERABLE$/ in $banned_filename_re), the preserved file is now named parts/part-00000 instead of parts/email.txt to preserve the size of the name, upon which some virus scanners depend (e.g. DrWebD); A note to Courier users: due to the way a file name in Courier is created and passed to amavisd-new, it is currently not possible to use the triggers /^MAIL$/ and /^MAIL-UNDECIPHERABLE$/ in $keep_decoded_original_re, or a failure to create a hard link occurs; Thanks to Bowie Bailey for helping to troubleshoot the problem; (this limitation is lifted in amavisd-new-2.2.0) - remove option -c when calling gzip, bzip2, compress, lzop and unfreeze (as has long been done in the development version); the option -c is not needed when no file argument is present, and some implementations of gzip and compress may choke on it; - make loading of Perl module Carp::Heavy optional, versions of Carp that came with Perl 5.005 did not have it; reported by Jefferson Pizzolatti; - load module Mail::SpamAssassin::BayesStore::DBM, required by SA 3.0 running in chroot jail; - look for program 'gcpio' ( $cpio = ['gcpio','cpio'] ), as on OpenBSD the plain cpio does not support required options (the --no-absolute-filenames is essential), but GNU cpio does; thanks to Manfred Gloiber; - relax parsing of file(1) output to allow tab as well as space to follow a file name; the file(1) on Solaris uses tab instead of space; suggested by Glen Harris; - make LHA understand self-extracting archives (SFX); patch by Georgy Salnikov: Although lha unpacks any non-SFX archive independently if its extension, it unpacks the SFX lha archives only if they have the extension .exe. Now the file is symlinked to $part.exe for checking by lha: this will now work for SFX, and will still work for non-SFX. Also, do_lha will return 0 if $exec and the part cannot be de-archived, the same as in do_unrar. The executable file formats are checked for being zip/rar/lha SFX archives. However, if the SFX archive is not a zip archive, do_unrar always returns success, so that if the archive is also not a rar, it will be never checked by lha. Now do_unrar returns 0 if $exec and the corresponding part can not be un-RARed; - do_executable and do_unarj: added checking for SFX arj; by Georgy Salnikov: (commented out the call to do_unarj in do_executable until more experience is gained on how well unarj survives certain mail contents; Mark); - do_unarj: let arj/unarj work on file named part*.arj; - when calling IO::File::open() use '+>' instead of 'w+' to avoid Perl taint bug ($mode turns tainted) (bug still present in 5.8.2) triggered by expression in IO::Handle::_open_mode_string(); - attack the Perl 5.8.0/5.8.1/5.8.3 taint bug (once variables $1,$2,etc get tainted they start spreading taintedness to other variables): * insert local($1,$2,$3,...) in blocks of code which call external modules which trigger the bug (Mail::SpamAssasin, MIME::Parser, ...) * insert local($1,$2,$3,...) in blocks of code which depend on these variables to be clean, and which demonstrated through bug reports and experience with various version of Perl that these variables were not always taint-clean; The last taint incident triggered by SA 3.0.0 (svn) reported by Luc de Louw; - recognize status EX_NOUSER when forwarding via pipe to sendmail (old setup); previously it was treated as a temporary failure; patch by The Mindflayer; - turn error message 'error reading from client socket' back into 'client broke the connection without a QUIT' for consistency with P7; - when %virus_admin lookup table is used, prefer $msginfo->sender_source (unmangled sender domain) over $msginfo->sender; suggested by Pawel Golaszewski; - nicety: in virus recipient notifications now supply the To: header field with the true recipient address instead of "undisclosed-recipients:;" in case of single-recipient mail; suggested by several people; - amavisd would insert differently capitalized header fields (either X-AMaViS-Alert or X-Amavis-Alert), depending on the reason being reported; now use the X-Amavis-Alert throughout; reported by Carsten Hoeger; - fetch Perl auto-loaded modules auto::POSIX::setgid and auto::POSIX::setuid if they exist; they are needed by older versions of Perl when running in chroot. A manual change was needed until now (documented in README.chroot), which should no longer be needed; a FreeBSD problem report 64636. - helper program amavis-milter.c (used in the sendmail milter setup): * amavis-milter.c only included the client IP and client host name on the first mail transaction of a multiple-transaction SMTP session, but not in subsequent transactions; pointed out by Stephane Lentz. The solution was once already provided back in July 2002 by Radoslav Dibarbora - Dibo, and forgotten; credits where credits are due: http://marc.theaimsgroup.com/?l=amavis-tech&m=102569946910555 * make a clear distinction between message data and connection data, which required code reshuffling and revealed previous unclean solutions; * add error checking and reporting to mkdir/rmdir/open/unlink/write system calls; previously an error could pass by unnoticed or just caused a tempfail without an explanation; * change final milter status ACCEPT into CONTINUE to allow further milters in the milter chain to examine the mail; * code cleanup; * regenerated helper-progs/configure with Autoconf 2.57 to make it capable of detecting mkdtemp and still be able to find sendmail libmilter files; thanks to Sebastian Hagedorn for a problem report, for testing, and for revealing a bug in the pre-released version; * use different syslog priority for different internal message log levels; by Sebastian Hagedorn; * adjust log levels of messages, set default verbosity 1 (DBG_WARN); - documentation updates, including the updated comments in amavisd.conf regarding whitelisted senders, to reflect the change indicated in the amavisd-new-20030616-p4 release notes; --------------------------------------------------------------------------- March 9, 2004 Patch: amavisd-new-20030616-p8 - be compatible with SpamAssassin version 2.70 and 3.0 as well as 2.6x; SA changed its API, replacing Mail::SpamAssassin::NoMailAudit with $spamassassin_obj->parse and belonging objects; - as a stop-gap solution to the W32/Bagle-{F,...} detection problem (password-protected zip archives), three new measures are available: * ability to present the full non-decoded original message to virus scanners was partly back-ported from the development version. Enabled by adding qr'^MAIL$' or qr'^MAIL-UNDECIPHERABLE$' to the list in $keep_decoded_original_re, as illustrated in amavisd.conf; similar to a patch by Ted Cabeen. The following keys are used for a lookup into $keep_decoded_original_re: 'MAIL' always provide a full original message to virus scanners (besides its successfully decoded components); 'MAIL-UNDECIPHERABLE' same as for 'MAIL', but only if it contains undecipherable components such as password protected archive members, unsupported compression methods or encrypted parts (e.g. with PGP). Don't put too much trust into this, as some more exotic file formats may not be understood and not flagged as undecipherable; * a key 'UNDECIPHERABLE' is matched against $banned_filename_re when mail contains any undecipherable components, and if lookup returns true, mail will be banned. For example: $banned_filename_re = new_RE( qr'^UNDECIPHERABLE$', # contains any undecipherable components ... ); * a string can be prepended to Subject (for local recipients only) if mail could not be decoded or checked entirely, e.g. due to password-protected archives or non-decodable mail bombs: $undecipherable_subject_tag = '***UNCHECKED*** '; # undef disables it NOTE: this solution is a quick-fix response to popular demand. Although the same or similar functionality will probably remain in future versions, the syntax and exact semantics may be refined. - bring do_unarj in line with the rest of de-archivers: provide the same degree of mail bomb protection that was available in do_unrar; retain original archive for inspection by a virus scanner if it contains any members that can not be extracted (e.g. password protected members or unsupported compression schemes). (The .arj size checking deficiency was mentioned in the AMaViS Security Announcement (ASA) 2004-01-19); - don't call a virus scanner if there are no files (no mail parts) to scan; it caused problems with certain scanners like aveclient, which expect a list of file names as arguments and complain if the list is empty; reported by Daniel Luttermann; - a much needed feature: can specify $sa_dsn_cutoff_level in amavisd.conf to suppress sending a DSN (delivery status notification) when spam level is above this value, effectively turning D_BOUNCE into D_DISCARD for this message; undef disables the feature and is a default. A good first approximation value is 10, or with some risk go down to 8. The parameter has no effect if DSNs are already disabled (e.g. when $final_spam_destiny is D_DISCARD or D_REJECT); - do_unrar: double check the archive size (against summary line as well); - derive the value of macro %l from sender_source instead of from a declared sender address, so as to not rise false alarms when sender address is known to be faked; patch by Joerg Friedrich; - reduce the number of retries to connect to a daemonized virus scanner from 3 to 2, so that a fallback to backup scanners occurs sooner; - updated $banned_filename_re example in amavisd.conf, modify it to will; - updates to the @av_scanners list in amavisd.conf: * KasperskyLab AVP - aveclient: *IMPORTANT* updated entry for the exit status only reflects the result of the last file scanned, we must use regular expression to detect viruses; fix provided by Andreas Triller; * Trend Micro FileScanner - vscan: added option -za, otherwise some broken archive may sneak-in a virus; suggested by Stephane Lentz; * Sophos sweep: added options -cab -tnef --no-reset-atime; * Dr.Web command line scanner: updated options, recognize exit status when using evaluation license; * ClamAV clamd: change socket location to a more usual value, adjust to will; - amavisd.conf: added new virus names to the $viruses_that_fake_sender_re list, and uncommented the [qr/.*/=>1] line, so that by default any unknown virus will be treated as a sender-faking virus; adjust to will; - do not send bad header notifications in response to messages from mailing lists; - added header check for folded header field lines made up entirely of whitespace (a 'header space gap' violation to rfc2822); a check is only made when other header checks (bad character in header) are enabled; - distinguish an empty string from undef in $mailfrom_to_quarantine, making it possible to specify a null return path when quarantining to a mailbox; - helper-progs/amavis-milter.c (sendmail) enhancement based on work by Stephane Lentz: * the name of a temporary directory is derived from the sendmail queue ID, making it easier to match sendmail and amavisd-new log entries; the queue ID is also a part of the quarantine file name; * a phantom 'Received:' header field is prepended on the temporary file to preserve the information on the original SMTP client IP address, host name and queue ID. This trace header field does not propagate to recipients (is not inserted into the original message), but is available in the quarantined messages and is visible to SpamAssassin. Adding a milter macro {b} on ENVFROM is advised to preserve the MTA timestamp in the log, although not mandatory (falls back to current time): define(`confMILTER_MACROS_ENVFROM', confMILTER_MACROS_ENVFROM``, {b}'')dnl If the new amavis-milter.c gives you trouble, switch the soft link to the previous version in the same directory. Keeping macro {b} does not hurt. - catch possible errors in pre_loop_hook and report them properly; - better check for I/O errors on SMTP input socket; - add the following modules to the list of pre-fetched modules: Carp::Heavy, IO::Handle, IO::Socket::UNIX, IO::Socket::INET. The absence of Carp::Heavy could mute error report or a backtrace when running chroot-ed; the rest are for completeness only; - added a macro %z which expands to the original mail size (in bytes); it could be useful in the $log_templ; thanks to Nick Leverton; - make SpamAssassin timeout value configurable: variable $sa_timeout; by Henrique M. Holschuh (Debian); - 'Received:' header field cosmetics: use 'unknown' in case the HELO argument was empty, and put-in a 'unix socket' if message was received from a helper program; by Henrique M. Holschuh (Debian); - remove more pre-existing X-Spam* header fields from other scanners: X-Spam-Tests, X-Scanned-By; by Henrique M. Holschuh (Debian); - helper-progs/amavis.c updates (old sendmail setup): report errors, log to syslog, change default dir to something obvious for bug reporting; by Henrique M. Holschuh (Debian); - updated documentation; --------------------------------------------------------------------------- January 5, 2004 Patch: amavisd-new-20030616-p7 - do_unzip and do_unrar: retain an archive if any of its components is password protected or encrypted (plus unpack what can be unpacked, as before). This gives virus checkers a chance to examine the original unpacked archive as a whole (e.g. scanning for variants of W32/Mimail), matching it in non-decoded form against virus patterns even if containing password-protected components. As a consequence, some virus scanners may now log their complaint when encountering such protected archives which previously didn't reach them. Such log entries should be considered informational only; - add module Net::DNS::RR::AAAA to the list of Perl modules to be fetched before chroot takes place; thanks to Per olof Ljungmark; - preload Perl modules DBD::*, based on @lookup_sql_dsn; required when using SQL lookups from a chroot jail; - updated example dsn in @lookup_sql_dsn (file amavisd.conf) to use the new DBD::mysql syntax, and to show how to force accessing SQL server via inet socket which makes it easier to use from a chroot jail; - disregard cached spam results for mail with small body. The most pronounced undesirable effect was in mail with an empty body where spam score was derived from header only; - change the default value for local_domains_sql lookup for the catchall key '@.' under conditions: when using SQL lookups and user record with key '@.' is present in the database and a field 'local' is not present. Previously it surprisingly defaulted to true, now it falls back to static lookup table defaults, the same as if the record '@.' were not present in the table; - fixed ugly text formatting in recipient notifications template; reported by Florian Effenberger; - fixed parsing of 'Received:' header field for some unusual cases; - updated amavisd-new-courier.patch to apply cleanly against -P6 (now P7); by Martin Orr; - added av_scanners entry for 'AVG Anti-Virus', kindly provided by Grisoft s.r.o. from Czech Republic, http://www.grisoft.com/ - make Dr.Web Daemon av entry work with evaluation or regular license; thanks to Andrew I Baznikin; - added av_scanners entry for new AVP client (aveclient) that is shipped whith avp 5.x.x.x. Moved kavscanner to the @av_scanners_backup list (it is presumably slow and less reliable than aveclient), updated other AVP/Kaspersky entries; thanks to Nabil Sefrioui; - added status codes 10 and 15 to the list of ok statuses for kavscanner (10=Password-protected archives, 15=Corrupted files); - add an example dummy virus scanner 'all-clean' to the @av_scanners_backup list which always succeeds, always returning false (= status clean). Uncomment it if desired to avoid mail requeue when all other scanners fail, and to just pass the mail unchecked; - disregard rfc2821 recommendation that 552 smtp response code should be treated as 452. It is unnecessary in amavisd-new setups, and it is wrong because 552 has other meanings assigned to it besides "too many recipients"; - allow IP address in $forward_method and $notify_method to be bracketed, which is needed for IPv6 addresses containing colons. Both formats are now allowed: 'smtp:127.0.0.1:10025' and 'smtp:[127.0.0.1]:10025', the later is now preferred; - change IP address bracketing in log entries 'SEND via SMTP' and 'FWD via SMTP' from [127.0.0.1:10025] into [127.0.0.1]:10025 to match the syntax in the connect log entry and to facilitate parsing of IPv6 addresses from the log; - update received_line() to generate valid Received header field even for IPv6 client addresses; - modified helper program amavis.c to allow it to be run non-root and to set temp file mode for group accessibility. Change its default log level to DBG_WARN; - fix an incorrect SELECT example in README.lookups; thanks to Nabil Sefrioui; - documentation updates; --------------------------------------------------------------------------- November 10, 2003 Patch: amavisd-new-20030616-p6 - change SQL lookup code to better handle SQL database server restarts. After an SQL server restart amavisd-new would previously TEMPFAIL (4xx) all messages until amavisd child process would run down, then resume normal operation with the new child birth. Now the SQL server reconnect is done when the next mail arrives, so only one mail with each amavisd child process TEMPFAILs during SQL server restart, and normal operation resumes faster; - in flatten_and_tidy_dir and in do_ascii: sanitize protection of files and directories which may otherwise be made inaccessible to virus scanners; based on patch by Henrique de Moraes Holschuh (Debian amavisd-new support); problem reported by Aspa and by Tomasz Papszun; - fix a potential security problem: don't let rmdir_recursively and rmdir_flat follow symbolic links; this might be exploited by attempting to delete some foreign file using privileges of the amavisd process (which should not be root); - use cpio option -d when unpacking cpio archives; - don't insert spam-related headers for passed mail to recipients with bypass_spam_checks, making them believe the mail was not spam-checked (as they are not expecting such headers anyway); - text added to the banned mail sender notification template (at the end of file amavisd), explaining to the sender what happened and how to avoid the restriction; edit to will; - amavis-milter.c program in the helper-progs subdirectory is now based on the most recent version 1.1.2.3.2.40 from the AMaViS CVS (maintained by Lars Hecking), but hacked a bit to make its options mostly compatible with the previous version. Start it with option -h to get current usage text. See helper-progs/README if using sendmail milter setup; Please revert to the old one if the .40 gives you trouble; both version are included in the helper-progs subdirectory. - fix parsing of unrar info lines; - consistency with other virus/banned logic: don't send recipient notification (reporting banned name) if mail contains both a virus and a banned name, but $warnvirusrecip is false; reported by Nathan G. Grennan and Urška (Brinar) Mlakar; - check for possible I/O errors when reading from SMTP socket, and distinguish error condition from normal TCP session teardown; - do not redirect stderr to /dev/null when calling file(1). This way the diagnostics from file(1) will now at least be visible in the debug session; - determine_file_types: make 'bzip compressed' pattern match older bzip format (v1) as well; thanks to Davaeron; - sub run_command: explicitly close STDERR before reopening it; this way the reporting of possible problems in each operation would be separate, and it seems to avoid a rare problem (open STDERR '/dev/null' failing) reported by Sam Hart; - don't attempt to do lookups in regexp table $viruses_that_fake_sender_re if it is undefined; reported by Chris Paul; - remove existing X-Spam-Score along with other X-Spam* header fields if spam scanning is enabled; - do not skip inserting 'X-Spam-Status: No, hits=- ... WHITELISTED' for whitelisted senders which caused SA to be bypassed - artificially assume score is -10 for the purpose of comparing it to tag_level; reported by Mike Vanecek; - explicitly qualify wblist.rid field in $sql_select_white_black_list just in case someone has a field 'rid' in the other table; - fix showing the value of LC_TYPE environment variable in the log, show LC_CTYPE as well; - SAVI-perl: remove MIME option from the default set of options. Even with recent versions of Sophos SAVI there are cases where the library goes into a spin while trying to decode broken MIME message (same applies to Sophie - one may want to change its configuration); - updated vfind entry in @av_scanners to work with the new version, changing '--vexit {}' to '--vexit {}/*'; thanks to Lowell Filak; - updated kavdaemon entry in amavisd.conf; thanks to Michael Hall and Daniel Melanchthon; - added option '-packed' to 'FRISK F-Prot Daemon' entry; thanks to Manfred Gloiber; - NAI uvscan entry: commented pre/post actions in the entry to show how to remove environment variable LD_PRELOAD after finishing, suggested by David Tilley; added option --mime, suggested by Max and Kevin W. Gagel; added note on how to treat password-protected files as viruses, by Seth Parker; - added 'Dr.Web Daemon' entry which talks directly to Dr.Web daemon over its Unix socket, speeding up a single-file check more than 200 times; provided by Andrew I Baznikin; - another entry for Symantec AntiVirus Scan Engine provided by Guido R. Rolon A.; I'm not sure which is which, check your documentation; - added 'dumaru', 'parite', 'gibe' and some other virus names to $viruses_that_fake_sender_re; - add examples to $viruses_that_fake_sender_re to show how to make a default result true, and only list exceptions; - placed a comment in amavisd.conf pointing out the proper syntax for $hdrfrom_notify_* variables; thanks to Wouter de Jong; - bump up the size of $sa_mail_body_size_limit to 150 kB in amavisd.conf - documentation updates; fixed typos and spelling mistakes in the documentation files and in amavisd.conf comments; - new documentation file README.protocol, specifying the new (and the old) protocol between helper programs and amavisd daemon, to be made available with the next major release. The description of the current (traditional) protocol was contributed by Stephane Lentz. --------------------------------------------------------------------------- August 25, 2003 Patch: amavisd-new-20030616-p5 - fix 'Modification of non-creatable array value attempted' bug when no 'Received' header field was present in an infected mail; reported by Paul Miner; - caching of SQL lookups on white/blacklist was based on sender address only, instead of sender _and_ recipient. This could lead to white-/blacklisting of one recipient to affect other recipients of the same message; reported and debugged by Paul Gamble; - added LDFLAGS to helper-progs/Makefile.in. It is needed at least on NetBSD. Patch by Julian C. Dunn (the NetBSD package maintainer for amavisd-new); - more obvious logging of HOLD reason in sendmail/milter setup (reported by Pascal Martinez); - $MAXLEVELS zero or undef should disable the limit according to docs, but was not honoured; reported by Rob Hutton; - if notifications delivery encounters a temporary failure (4xx), propagate this status to the final result instead of only logging a warning; - amavisd.conf: * VirusBuster entry changed to match newer version of the product; information from Marcus Schopen; * another entry in amavisd.conf for KasperskyLab kavscanner (v4.5?), contributed by Simone Marx; --------------------------------------------------------------------------- August 12, 2003 Patch: amavisd-new-20030616-p4 - revert to using alarm() instead of Time::HiRes::alarm(). It is nonstandard to mix the two, and is causing problems on some operating systems (e.g. Solaris); thanks to Geoff Gibbs; - rise log level for log entries on intentional mail drops in case a DSN (a bounce) should be sent, but will not be; such as on rejected bounces, viruses with forged names, and spam from mailing lists. The new log entries now say: '... Not sending DSN ...' and provide more information on the reason for dropping DSN; - if sender is whitelisted, don't insert 'X-Spam-Flag: YES' header field, don't append spam address extension, and don't quarantine. This makes it less surprising, although previous behaviour was according to documentation. NOTE: the documentation still describes former behaviour, this needs to be fixed; - report deaths of command line scanners and some external programs distinctly from normal exits with nonzero exit status; - fix replacing * or {}/* in the pattern with actual file names, causing MkS_Vir (mks32), VFind, Dr.Web Daemon, and KasperskyLab aveclient to receive its file arguments glued together; based on patch by Rafael J. Wysocki; - update 'Panda Antivirus for Linux' entry to work with new (and older) versions of pavcl; updated entry kindly provided by Panda Software; - stop the timer after SMTP transaction is over to better behave in persistent SMTP/LMTP sessions; start the timer at the beginning of a SMTP transaction, in addition to restarting it when DATA mode is entered; - sub mail_via_smtp_single: properly report SMTP response code when all recipients are rejected by MTA, instead of logging a 'mail_via_smtp:' without a value. The problem was commonly seen with the W32/Mimail-A virus which fakes an often invalid local sender address and gets rejected by Postfix outright; reported by Turgut Kalfaoglu; - use SMTP response code 554 (instead of 550) for rejecting syntactically invalid header (according to rfc6409 (ex rfc4409, rfc2476)); - add am_id to SMTP response code generated by one_response_for_all() to make it easier for MTA log to be correlated with amavisd-new log; some cosmetic improvements in the generated SMTP response text; - added 'Return-Path:' in notifications to make it more obvious to see envelope address from reports; - the 'Message-ID:' in neutral DSN notifications template was inadvertently pushed into the DSN body; - report undefined spam score in X-Spam-Status header field as 'hits=-' instead of 'hits=0.0' which can be misleading; - indicate blacklisting in X-Spam-Status header field of quarantined messages; - add X-Envelope-From header field to quarantined messages; - added virus names: tanatos, lentin, bridex (alternative names for bugbear, yaha and braid) to the $viruses_that_fake_sender_re; thanks to Harrie Overdijk; - set environment variables LINES and COLUMNS to sensible defaults to avoid some external program get puzzled about the terminal settings (e.g. older versions of pavcl from Panda Software); - another attempt at fixing the Subject header field duplication. The patch amavisd-new-20030314-p2 fixed the case of entirely missing Subject header field, but did not fix the case of Subject header field present but with an empty text. Reported by Steven Cobb and Francois Rolland; - rewritten 'Received' header fields parsing to better cope with valid, as well as with more common cases of broken syntax; used when trying to report originator IP address for believed-to-be-faked senders; - more permissive parsing of SMTP addresses and options on MAIL FROM and RCPT TO commands; - 'neutral' (=space) field in SQL black/whitelists now terminates the lookup search, avoiding fallback to static black/whitelists. It enables recipient to explicitly express its neutral stance towards the sender, overruling the site default; - $sa_mail_body_size_limit now takes into account some portion of the mail header size so that huge mail headers that can cause slow SA calls are avoided (such degenerate cases were reported by Ralf Hildebrandt); - taint fix in read_l10n_templates (as used by the Debian distribution), patch by Henrique de Moraes Holschuh; - don't send recipient notifications to recipients that have bypass_virus_checks/bypass_banned_checks; suggested by Joe Breu; - replace /bin/false with a more usual /usr/bin/false as a last resort exit; - fetch modules 'Net::Ping' and 'bytes', which seem to be needed in certain chrooted setups; - change log level from 0 to 1 for the log entry 'BAD HEADER from'; reported by Thomas Lamy via Debian bug reports; same for 'unrar: all %d members are encrypted'; - fix typo in variable name: $spamassasin_obj -> $spamassassin_obj - explicitly set pipes and sockets to binmode, as is a default since Perl 5.8.1 - documentation updates: * new file LDAP.schema, by Jacques Supcik, PhD * updated README.chroot to tell that /dev/urandom is needed in chroot jail (otherwise creation of MIME notifications fails); thanks to Lynn Duerksen and Jimmy Porter; * updated README.sendmail-dual, thanks to Robert LeBlanc and Stephane Lentz; * added URLs of external programs to INSTALL; * small updates to other doc files; --------------------------------------------------------------------------- June 28, 2003 Patch: amavisd-new-20030616-p3 - avoid 'savemail: cannot save rejected email anywhere' sendmail panic when feeding mail via LMTP and using D_BOUNCE settings in dual-sendmail setup. --------------------------------------------------------------------------- June 27, 2003 Patch: amavisd-new-20030616-p2 - when running chrooted, fix pre-loading of modules needed by SpamAssassin and Razor agents, which SpamAssassin forgets to pre-load by itself; reported by Neil Camara; - updated KasperskyLab AVPDaemonClient entry to protect against carriage returns in collected virus names; thanks to Harrie Overdijk; - updated 'Symantec CommandLineScanner' entry to look also for the new scanner name 'savsecls'; thanks to Guido Rolon; - documented extended uses of read_hash() in the release notes; - updated README.chroot; --------------------------------------------------------------------------- June 24, 2003 Patch: amavisd-new-20030616-p1 - bug fix: allow stderr to be joined with stdout (>&1), instead of creating a file with a name "&1" when calling command-line scanners or doing LHA decoding (it got broken in 20030616); noticed by Henrik Larsson; - bug fix: the X-Spam-Level header field would carry 64 asterisks instead of none when spam is configured to pass (mail tagging only) and when SA was not called (e.g. sender is whitelisted, ...) and tag_level or tag2_level is set to 0 or less; reported by Ralf Hildebrandt; - bug fix: untaint e-mail addresses when forwarding via pipe; reported by Sam Tilders; - modify code to match documentation: $relayhost_is_client should influence $notify_method too, not just $forward_method (adding an extra argument to mail_dispatch() was necessary to make this possible); reported by Zhu Yicun; - SQL failure modes consistency: when initial connect to SQL failed, previous versions of amavisd-new would fall back to static defaults, disabling SQL lookups. Now initial connect failure is fatal, just like if connect failure occurred during operation - mail flow just stops during SQL servers outage, and is resumed when servers become reachable again; - commented-out adding of 'X-Spam-Report' header field to retain behaviour from amavisd-new-20030314 - I find this header too long and intrusive for regular use. Uncomment it if adding X-Spam-Report is desired; - support lzop (.lzo) and freeze (.F) compressors if available; - use external program cpio (if available) to efficiently and safely handle the following archive formats: cpio binary, HPUX binary cpio, cpio crc, old ASCII cpio, new ASCII cpio, and HPUX old ASCII cpio, as well as POSIX.1 tar (also GNU tar) and old tar if allowed (see below) without reading whole archive members into memory; - if your cpio(1L) can read tar format (as is common on FreeBSD and Linux), it is recommended to uncomment the following line (in file 'amavisd'): # /^\.tar$/ && defined $cpio && return do_cpio($part,$tempdir); which will cause tar archives to be safely and efficiently decoded by cpio(1L) instead of Archive::Tar, which loads the whole archive into memory; - use separate file(1) 'classifications' ".uue" and ".hqx" (instead of ".asc") for uuencoded and binhex formats; - merge 'H+BEDV AntiVir' and 'CentralCommand Vexira Antivirus' entries into a single entry - they are basically the same product (the Vexira entry didn't work for new version of Vexira); thanks to Vivek Khera; - do_unrar: some switches to rar/unrar are not recognized by older versions of rar/unrar; issue a more informative log when this happens, and retry without newer switches for compatibility; thanks to Geoff Gibbs; - crop backtrace on SA timeouts at 980 characters; - preload Perl modules Net::DNS::* if spam scanning is enabled; they could not be loaded if running in a chroot jail, so SpamAssassin did not use Razor; - documentation: collected documentation pieces pertaining to LDAP into README.lookups; fix SQL example table data in README.lookups; ( the patched MIME-tools by David F. Skoll is recommended over 5.411, as it better handles broken/bad MIME syntax: http://www.mimedefang.org/ -> Download section ) --------------------------------------------------------------------------- June 16, 2003 amavisd-new-20030616 release notes This is mostly a maintenance release. Only a handful of new features were added that were small or easy enough to provide. - revised the incoming SMTP session abort procedure to properly shut down the remote SMTP client in case of unexpected trouble (e.g. disk full) during SMTP data reception; - subroutine mail_via_smtp_single() overhaul to produce cleaner diagnostics and to properly abort outgoing SMTP session in case timeout occurs in SMTP 'DATA' mode (avoids recipients repeatedly receiving partial message in successive delivery attempts when the receiving MTA responds very slowly); problem reported by Scott Vintinner; - fixed the generation of LMTP response code when D_BOUNCE was chosen, but DSN was not actually sent because sender was thought to be faked or mail came in from a mailing list. Since DSN was not really sent, the final response remained at 5xx (causing bounce by MTA) instead of being converted to 2xx as D_BOUNCE would suggests; reported by Peter Bates; - security: avoid inadvertently untainting values (e.g. mail addresses) at several places throughout the program; - work around a Perl 5.8.0 taint bug where global variable $1 could become tainted, making further untainting attempts unsuccessful; - work around another Perl 5.8.0 taint bug where Encode::encode fills up all available memory when given a tainted string with a non-encodable character. New subroutine safe_encode() provides a wrapper around the Encode::encode; - treat 'Maximum number of files exceeded' the same as 'Exceeded storage quota', i.e. inserting the X-Amavis-Hold header field; noted by Christopher Odenbach; - call exec with explicitly specified program path, preventing exec from even considering calling a shell; - catch and better report the Perl's failure to fork on open; - avoid inappropriately reporting 'Illegal seek' in run_av (and elsewhere) even though nothing went wrong; reported by William Yodlowsky; - new subroutine flatten_and_tidy_dir() recursively descends into a directory containing potentially unsafe files with unpredictable names, soft links, etc., rename each regular nonempty file giving it a generated name, and discard all the rest. - do_unrar and do_unarj now let the archiver itself recursively unpack the archive and then use flatten_and_tidy_dir() to tidy up the result. With previous method it was not possible to extract archive members with names containing non-ASCII characters from rar archives due to a bug in unrar and rar programs. Problem reported by Pan Bambaryla. Protection from mail bombs in do_unrar is retained. - do_unrar: use program rar or unrar, whichever is available. Both rar and unrar recognize the same options to extract. (amavisd.conf: $unrar=['rar','unrar'] ). Under some circumstances unrar falls in a loop while rar extracts the archive correctly. - do_unrar: don't bail out on exit status 1 (warning) from unrar after collecting file names from the archive. Be more careful when extracting file names, such as archive members with names starting with '-' or spaces; Disable showing archive comment and file comments which can easily break member-name parsing. Avoid extracting files in subdirectories twice: once when directory itself is listed, and the second time when each file in a directory is listed. Licence note: unrar is free, rar is not ( see http://www.rarlab.com/ for fresh unrar sources ). - do_unarj: prefer more versatile arj over unarj (amavisd.conf: $unarj=['arj','unarj'] ). Both programs support subcommand 'e' that we use; suggested by Guillem Jover (via Debian support); ARJ since 2.78/3.10 is Open Source (GPL license). When arj is available (recognizing several new archive types, archive versions and options), put these options to good use, recovering several archive members that are unaccessible by the demo program unarj, such as old member versions, memebers with equal names, etc. Disable showing comments. - add X-Virus-Scanned header only if a mail was actually scanned for viruses (i.e. av scan not bypassed for this set of recipients); pointed out by Phil Regnauld; - updated entry for 'KasperskyLab AvpTeamDream' virus scanner; by Daniel Melanchthon; - truncate instead of round the spam level when producing the bar in the X-Spam-Level header field for compatibility with SA. Clip the level bar at 64 characters instead of 60 (more would not be allowed by RFC 2822, although SA used to crop at 100); - subroutine hdr() incorrectly fixed an illegal header field body which did not have a space after a line fold; noticed by Virna Gupta; (the buglet did not show unless one modified the program); - replace several remaining regular expressions /...$/ with /...$(?!\n)/ as a matter of principle (as was done in amavisd-new-20030314-p2 for the more urgent cases). The use of simple $ was almost always subtly semantically wrong, but fortunately in most cases without consequences; - specify open mode explicitly in several 'open's, and protect special characters in file names from being 'too cleverly' interpreted by Perl open (still not all cases done); - SpamAssassin changes umask to 0077 - restore our mask after the SA call; - be a bit more careful regarding platform-independency, distinguishing between use of \012 and \015 against \r and \n (note the \n on Mac is CR); - change log level of several less important av scanner log messages from 2 to 3; - small documentation updates in amavisd.conf; - updated README.lookups: the new substring capture and reference mechanism in regexp lookups explained. The SQL example modified to work under PostgreSQL, thanks to Phil Regnauld; - updated documentation: README.sendmail-dual, README.postfix, README.chroot, new file: README.exim_v4_app2 - by Louis Erickson; NEW FEATURES - provide a command line option 'debug-sa', which is equivalent (but more convenient) then setting $sa_debug to true and starting as '# amavisd foreground'; - list version numbers of the more important Perl modules at startup; - lookup_re(): RE pattern now allows for capturing of parenthesized substrings, which can then be referenced from the result string using the $1, $2, ... notation, as with the Perl m// operator. The number after the $ may be a multi-digit number. To avoid possible ambiguity the ${n} or $(n) form may be used. Substring numbering starts with 1. Nonexistent references evaluate to empty strings. If any substitution is done, the result inherits the taintedness of the key. Keep in mind that the $ character needs to be backslash-quoted in qq() strings (but not in q() strings). Example: $virus_quarantine_to = new_RE( [ qr'^(.*)@example\.com$'i => 'virus-${1}@example.com' ], [ qr'^(.*)(@[^@]*)?$'i => 'virus-${1}${2}' ] ); - read_hash() now returns the hashref (first argument) also as a return value, making it easier to create new hashes (instead of only adding values to existing ones). Also the first argument (hashref) is now optional, and if missing a new hash is created. The following three cases are now equivalent: a) $sa_tag2_level_deflt = {}; read_hash($sa_tag2_level_deflt, '/var/amavis/tag2_levels.dat'); b) $sa_tag2_level_deflt = read_hash({}, '/var/amavis/tag2_levels.dat'); c) $sa_tag2_level_deflt = read_hash('/var/amavis/tag2_levels.dat'); - extend the semantics of the configuration variable $inet_socket_port, which can now either be a scalar as before (i.e. a single port number), or a ref to a list of port numbers, e.g: $inet_socket_port=[10024,10026,10028]; # accept SMTP on all these TCP port When $relayhost_is_client is true, the semantics of $forward_method has changed slightly: instead of taking the port number from $forward_method, it is now calculated as being one higher than the port number on which the incoming SMTP connection came in (one from the $inet_socket_port list). This allows for multiple MTA pairs on the same host to share the same amavisd daemon without having to use multiple loopback interfaces. Useful for example if incoming and outgoing mail is handled by separate mailers on the same host. - add field 'spam_quarantine_to' to SQL table 'policy' - a patch provided by Vivek Khera. For compatibility a missing field defaults to undef, causing lookup search to proceed with the next lookup map; - add fields: spam_lover, banned_files_lover, bad_header_lover, bypass_banned_checks and bypass_header_checks to SQL table 'policy' for completeness; For compatibility a missing field defaults to undef, causing lookup search to proceed with the next lookup map; - add a 'SPAM-TAG, ...' log entry at log level 2, which is produced when inserting spam-related header fields for each cluster of recipients with same settings. It complements the 'SPAM, ...' log entry, which is triggered at kill level. Feel free to adjust their log levels if you think one or the other is redundant; - insert a 'X-Spam-Report' header field at tag2 level when spam is passed. Feel free to comment out the ...append_header('X-Spam-Report'... line if this is undesired. - fold-in the 'bad_headers.patch' to the main code (available in the distribution since amavisd-new-20030314-p1), but renamed *bad_headers_lovers* to *bad_header_lovers* and added $final_bad_header_destiny and *bypass_header_checks*. To refresh memory: Enables checking headers of each mail for invalid (non-encoded) 8-bit characters, and produces a bounce (non-delivery status notification, or an SMTP REJECT if desired) with the full explanation of the problem, with offending header fields trimmed, sanitized and included in the text. New setting $final_bad_header_destiny, defaults to D_PASS for backward compatibility, but a value D_BOUNCE is suggested. Similarly to other *lovers* settings, a hash and an ACL lookup %bad_header_lovers and \@bad_header_lovers_acl are available, and a setting $warnbannedsender. Certain recipients may be exempt from the checking (*bad_header_lovers*). Similarly mail from mailing lists (Precedence: list or bulk), and mail with null reverse-path mail (e.g. bounces) is passed, even if violating the RFC 2822 header syntax. A log entry is produced nevertheless. Postfix users: this is similar to the Postfix strict_7bit_headers=yes functionality, but produces a much more informative problem report (non-delivery notification) to the sender. One difference is that amavisd-new header check takes into account only the RFC 2822 header, not MIME headers in the mail body. It is reasonably efficient to use amavisd-new as a header checker only, without any anti-virus or anti-spam checks, if desired. - added configuration settings $warnbannedsender and $warnbannedrecip, separating this function from $warnvirussender and $warnvirusrecip, which previously applied to both virus and banned files; - new macro %c which evaluates to spam level/hits (mnemonic: sCore) as provided by SpamAssassin; useful in $log_templ; - new configuration settings *bypass_banned_checks* and *bypass_header_checks*; - new parameter $spam_quarantine_bysender_to (in contrast to the usual $spam_quarantine_to) makes possible collecting quarantined spam by sender names or domains, instead of (or in addition to) the more usual by-recipient map; - new configuration setting $remove_existing_spam_headers (defaults to true); - bring-in the localization (l10n) support contribution from Debian Linux, by Henrique de Moraes Holschuh; see README.l10n, and comments at the call to read_l10n_templates() in file amavisd.conf; - supply X-Envelope-From and X-Envelope-To header fields containing envelope addresses to the messaged passed to SpamAssassin for checking; - there is now an official OID: 1.3.6.1.4.1.15312.2.2 under which LDAP schema for amavisd-new LDAP lookups is to be defined (iso.org.dod.internet.private.enterprise. 'ijs.amavisd-new.ldap' ); --------------------------------------------------------------------------- May 6, 2003 Patch: amavisd-new-20030314-p2 - blacklisted sender only caused spam headers to be inserted, but did not cause rejecting the message and other 'evasive actions' due to over-optimization; reported by Lawrence Farr, Robin Elfrink and Eric Vollmer; - fix improper evaluation of bypass_spam_checks* in multi-recipient mail where some recipients match 'bypass_spam_checks*' and others do not. Spam-check was bypassed if _any_ (instead of _all_) recipient matched 'bypass_spam_checks*'; thanks to Joseph W. Breu; (it got broken in amavisd-new-20030314, the amavisd-new-20021227 was ok); - adding address extensions was not done for *_lovers, only for *_destiny=D_PASS; reported by Steve Khoo. The "deal_with_spam" section needed to be overhauled for this reason. For consistency with the documentation and with the change in adding address extensions, quarantining is now done if mail is considered spam, regardless of it being delivered (e.g. to spam lovers), or not; - if only spam checking is performed, but no virus scanning nor banned file names checking is enabled, and the spam is detected but configured to pass, and the Subject is configured to be edited, the Subject header field was duplicated because the existing Subject header field was not detected. Solution: make MIME entity available even if virus and banned files scanning is not performed. Header information is needed by add_forwarding_header_edits_per_recip() when deciding whether to add or replace the Subject header field in case spam is passed. (workaround for previous versions: don't leave $banned_filename_re undefined. If not needed, set it to an empty list: new_RE() ); - when quarantining to a mbox file, excape 'From ' with '>' only if it follows an empty line, as sendmail mail.local man page prescribes; - added parameter $notify_xmailer_header to control the body of the header field X-Mailer as placed in the notification messages (e.g. bounces) generated by amavisd-new; by default this header is no longer inserted, as it was felt it revealed too much information. Choose from: $notify_xmailer_header = undef; # no X-Mailer inserted (default) $notify_xmailer_header = ''; # X-Mailer: MIME-tools x.xxx (Entity x.xxx) $notify_xmailer_header = 'your-text-here'; # X-Mailer: your-text-here - replace regular expressions /...$/ with /...$(?!\n)/ or with string equality operator in some places (many more to come in the next major release); the use of simple $ was almost always subtly semantically wrong, but in most cases without consequences; - enable timeouts after SMTP connection has been established, to be able to handle cases where MTA is very slow in accepting checked mail, which could result in multiple deliveries if MTA on the input side times out; - debugging aids: call stack backtrace is logged if Mail::SpamAssassin::NoMailAudit::check exceeds allowed time; - LDAP enhancement: allow "hostname" to be a string or a reference to an array, in which case each entry will be tried in order until a connection is made; by Jacques Supcik; - X-Spam-Status now says WHITELISTED if sender was whitelisted; contributed by Joseph W. Breu; - if mail rejection was due to sender blacklisting, say so in the SMTP response; thanks to Robin Elfrink; - updated file bad_headers.patch to apply cleanly against this version. The bad_headers.patch will be folded-in with the next major release, please try it out, comments welcome. For patch description see release notes for amavisd-new-20030314-p1; - specify option 'ExactAddresses=>1' in a call to Net::SMTP::new() in anticipation for the next release of Net::SMTP; - amavisd.conf: updated documentation / comments; - amavisd.conf: removed option '-ni' in the DrWeb virus scanner entry; thanks to Mike Boev and the FreeBSD community ( http://www.freebsd.org/cgi/query-pr.cgi?pr=50893 ) - amavisd.conf: added entry for 'MkS_Vir daemon', thanks to Dariusz Grzegorski; - amavisd.conf: updated entry for 'F-Secure Antivirus' to cope with fsav 4.50 as well as with 4.1x; solution by Juhan Tamsalu; - documentation: updated README.chroot; new README.sendmail describing dual-MTA sendmail setup, small updates to other doc files; March 21, 2003 Patch: amavisd-new-20030314-p1 - fix passing of remote MTA reject code back to the input side (DSN was not sent if MTA rejected the message); - fix broken logic which allowed a virus to pass if a recipient was a 'banned file lover' and the same message also contained a 'banned file'; (or the other way around: pass a banned file if containing a virus and sent to virus lovers); thanks to Ortwin Gentz, I'm glad somebody is looking over my shoulder; - remove a log entry: one_response_for_all: SHOULDN'T HAPPEN The test was inappropriate, but harmless; - enhanced sub run_av() to fix a problem with MkS_Vir virus scanner, where a '*' in the template command was not expanded to actual file names; reported by Pan Bambaryla; - trim virus scanner output (in sub run_av) to a manageable size of 900 characters so as not to clutter log entries and notifications too much; provoked by the Panda pavcl; - move the check for 'Precedence:' mail header field from sub delivery_status_notification() to its caller for flexibility; - replaced the text in notification templates: The message WAS delivered to: into: The message WILL BE delivered to: to reflect the truth. The message may still be rejected by the MTA when forwarding takes place. - small documentations edits/updates; - cosmetic: trim trailing whitespace in the program; - cosmetic: make shorthand subroutine nf() locally scoped; by Jacques Supcik; - make available an experimental patch (file: bad_headers.patch, not applied by default), which, when applied (with patch(1) utility) to amavisd-new, makes it check headers of each mail for invalid (non-encoded) 8-bit characters, and produces a bounce (non-delivery status notification, or an SMTP REJECT if desired) with the full explanation of the problem, with offending header fields trimmed, sanitized and included in the text. Certain recipients may be exempt from the checking (*bad_header_lovers*). Similarly mail from mailing lists (Precedence: list or bulk), and mail with null reverse-path mail (e.g. bounces) is passed, even if violating the RFC 2822 header syntax. A log entry is produced nevertheless. Postfix users: this is similar to the Postfix strict_7bit_headers=yes functionality, but produces a much more informative problem report (non-delivery notification) to the sender. One difference is that amavisd-new header check takes into account only the RFC 2822 header, not MIME headers in the mail body. It is reasonably efficient to use amavisd-new as a header checker only, without any anti-virus or anti-spam checks, if desired. The patch is experimental in the sense that it may change in future versions, but is fully functional when applied; - Jacques Supcik cleaned up and generalized his LDAP lookups support code (which is off by default). In his words: The major change is in the definition of ldap lookups. In the config file, you have to enable ldap first: | $enable_ldap = 1; Then you can define defaults for your ldap queries: | $default_ldap = { | hostname => 'localhost', tls => 0, | base => 'ou=hosting,dc=example,dc=com', scope => 'sub', | query_filter => '(&(objectClass=amavisAccount)(mail=%m))' | }; And then the lookups themselves: | $virus_lovers_ldap = {res_attr => 'amavisVirusLover'}; | $banned_files_lovers_ldap = {res_attr => 'amavisBannedFilesLover'}; | ... With this method, you can define every parameter individually. You could have a different ldap server for each lookup! Like that, the configuration is closer to the one in postfix or courier. The hashes are converted into lookups objects in the Amavis::Lookup::LDAP class method. --------------------------------------------------------------------------- March 14, 2003 amavisd-new-20030314 release notes MAIN NEW FEATURES AT A GLANCE - per-user white- and blacklisting, including an SQL lookup mechanism; refined semantics for white/blacklists; - easier to run in chroot jail, see README.chroot; - can specify two sets of virus scanners: the main list and a backup list, which gets consulted only if all scanners from the main list fail; - split REJECT into separate destiny settings: REJECT and BOUNCE, giving you a choice of who is responsible for sending a non-delivery notifications to the sender; see comments in amavisd.conf; - edited configure file amavisd.conf, providing more informative comments, cleaner examples and simplified common settings; - made it work better with the Unicode-aware Perl 5.8.x (not against it) and distinguish between character data / byte data / octet data (binary), provide UTF-8 (Unicode) support for notification messages, while avoiding some possible nasty surprises in the UTF-8 locale (which is enabled by default in Red Hat 8.0); - SQL SELECT clause settable from the config file; - can store quarantined or forwarded messages as BSMTP files (RFC 2442); Changes since amavisd-new-20021227 patch level 2 BUG FIXES - fixed caching problem when bypass_*_checks are used selectively: when only one of the virus/spam tests were performed but the other bypassed due to a per-recipient setting, the result of a non-performed test was inserted into cache as clean. This could cause tests to be bypassed for subsequent message with the same body but different recipients with different bypass* settings, but only within a lifetime of a child process (10 messages by default). Noticed by Jürgen Louis Fluk, thanks! - changed caching of SQL lookups to match the documented behaviour, i.e. SQL lookup result is cached (for the benefit of field lookups) for a single message only, and no longer for the child lifetime. It is possible to revert to the previous behaviour by commenting-out one line (look for $sql->clear_cache). Pointed out by Paul Miner; - see also further down about the field 'banned_files_lover' in the SQL table 'policy', and the UTF-8 (Unicode) workarounds. SECURITY - running amavisd-new in chroot jail is now easier to set up due to added configuration variable $MYHOME and its simplification effects on other settings in the config file. Relevant settings are better commented; - avoid the need to have a shell in chroot jail (use of the new subroutine run_command() instead of Perl functions qx() and system() ); - new file README.chroot with guidelines on how to run amavisd-new in a chroot jail, to prevent possible vulnerabilities from affecting the rest of your system; - added a security notice (web page and other places) that the Unix utility file(1) 3.39 and older contains an exploitable security vulnerability, which can cause system access with access rights of the user running amavisd-new daemon (not root, unless you violated the recommendations). Please upgrade file(1) to 3.41 or newer: ftp://ftp.astron.com/pub/file/file-3.41.tar.gz - new section (Security considerations) on the web page; CLEANUPS AND CHANGES IN amavisd.conf - improve explanation texts (comments) in amavisd.conf, edit the file to improve legibility and structure, change some examples. The daemon is thought to be compatible with amavisd.conf from previous version, but it may be worthwhile to switch to the new file to avoid asking FAQ; - introduce a configuration variable $MYHOME, which is never used directly by amavisd daemon, but is a handy way of setting default values for other configurable variables in the config file. Also makes default settings more chroot-friendly. - introduce a configuration variable $mydomain, which is never used directly by amavisd daemon, but is a handy way of setting default values for other configurable variables when there is only a single domain (with subdomains) behind the mailer. This avoids the need to edit half-a-dozen configuration variables in simple setups; - moved debugging-related settings to a separate section (Section VIII - Debugging); - make SQL SELECT clause settable from the config file; - renamed variable @local_domains to @local_domains_acl for consistency; The old name became a synonym for @local_domains_acl, so compatibility with existing config files is retained; - comment out the example: read_hash(\%whitelist_sender, ...); it slipped into the amavisd-new-20021227 released code from my tests; - comment out the setting $relayhost_is_client=1; It is only useful for multi-access daemon setup, and potentially confusing for others; VIRUS SCANNERS - introduced a list of backup virus scanners: @av_scanners_backup, complementing the normal list of virus scanners. If no virus scanners from the @av_scanners list produce 'clean' nor 'infected' status (e.g. they all fail to run or the list is empty), then all scanners in the @av_scanners_backup list are tried. When there are both daemonized and command-line scanners available, it is customary to place slower command-line scanners in the @av_scanners_backup list; - removed the Mac and SafeMacDfHandling options from SAVI-Perl, as they are no longer supported by Sophos library since 2003-03; set new option NamespaceSupport to SOPHOS_DOS_AND_MAC_FILES; thanks to Paul B. Henson; - changed a default to initialize SAVI-Perl only at startup time, and no longer every time the child process is started. This is because starting with SAVI 3.0 (March 2003) the initialization is slower. NOTE: it is now necessary to restart amavisd master process after SAVI database is updated, to make it notice new virus descriptions. - in 'MkS_Vir for Linux' entry replaced options -e -c with -s, to make mks work, suggested by Pan Bambaryla; - added status 8 ('corrupted') to the list of clean statuses for 'KasperskyLab AVPDaemonClient', as is already the case for some other virus scanners. See amavisd-new web page (Security considerations) if the idea bothers you; - RAV command line scanner no longer supported off-the-shelf, as interfacing to it seems to be violating their license terms. Btw, if RAV command line scanner does not work as user amavis, check permissions and ownerships of /usr/local/rav8/* or reinstall RAV as the user that will run amavisd; thanks to Oyku Gencay; - RAV does not like '</dev/null' (segfault) (reported by Johnny Ljunggren), most others don't care, but possibly Panda needs it. Tried to remove it, but later put is back, as I don't like external command to have access to the amavisd socket over which SMTP or AM.CL session is running; - H+BEDV AntiVir: changed option -allfiles to --allfiles; thanks to Dr Patrick Atlas, confirmed in the H+BEDV documentation; - comment out the default entry for Sophos sweep, as the name 'sweep' clashes with the Debian sweep package (audio samples editor). If you uncomment the entry, make sure the correct 'sweep' is found in the specified path; thanks to Guido Guenther and Brian May; - added av definition for BitDefender (based on amavis-ng code by Hilko Bengen); thanks to Stephane Lentz for testing and suggesting command line options; - added av definition for Ikarus AntiVirus for Linux, based on amavisd-0.1 code by Lars Hecking; PERL 5.8 UNICODE AND INTERNATIONALIZATION (I18N) - make program aware of the type of data coming-in from external files, pipes and sockets - distinguish between binary data, byte data, and character data / plain text. This is necessary in order to be able to survive in the UTF-8 locale-aware Perl 5.8.0 environment (e.g. to avoid 'Malformed UTF-8 character' and 'Wide character in print' warnings/errors, among others). - check character ranges in regular expressions throughout the program, and fix/rewrite them to avoid assumption that all characters are in the range 0..255, which is no longer true! Some nasty surprises could have been lurking in that corner. Note the Perl 5.8.0 terminology (see perluniintro and Encode man page): * character: a character in the range 0..(2**32-1) (or more) (what Perl's strings are made of); * byte: a character in the range 0..255 (a special case of a Perl character); * octet: 8 bits of data, with ordinal values 0..255 (term for bytes passed to or from a non-Perl context, e.g. a disk file) - modified sanitize_str() and hdr() to cope with UTF-8 characters; - modified string_to_mime_entity() and hdr() to encode RFC2047-permitted headers according to $hdr_encoding, and notification body according to $bdy_encoding settings (both default to iso-88591-1, aka Latin1). Notification template texts are supposed to be stored internally as native or UTF-8 Perl strings, which is handled (mostly) transparently by Perl 5.8.x; - added option $hdr_encoding_qb to be able to explicitly specify MIME encoding ("Q" or "B") for $hdr_encoding MIME charset; - rfc2822_timestamp: produce correct date format regardless of LC_TIME locale setting; - read_text() subroutine (which may be used in amavisd.conf to read notification text templates) may be given a second argument, specifying character encoding used in the external file. If Perl is Unicode-aware (5.8.0), this argument is given to binmode to influence the reading of the file and its conversion into internal Perl string representation. When notification text is generated from the internal Perl string representations (UTF-8), the following settings influence the character encoding (charset in MIME terminology): $hdr_encoding = 'iso-8859-1'; # ... header field bodies The header is first encoded into $hdr_encoding, which is then encoded into 7-bit ASCII as required by RFC 2821 and defined by RFC 2047. The specified encoding (charset) will appear in the RFC 2047 Q-encoded strings in header fields where this is allowed, i.e. in Subject, Comments, and X-* header fields. In other header fields where RFC 2047 encoding is not allowed, encoding into plain ASCII is used, where invalid characters are treated as Perl Encode module does it, i.e. by converting them to '?'. $bdy_encoding = 'iso-8859-1'; # ... notification body text Notification body text is encoded from the internal Perl representation according to $bdy_encoding. The encoding (charset) is also inserted in MIME header as Content-Type.charset. OTHER - make it work with Net::Server version 0.85, which changed the handling of the CHLD signal, this is breaking qx and system calls used by previous versions of amavisd-new; - with Net::Server 0.85 you may see a message: Net::Server: Couldn't POSIX::setuid to ... [] If the process UID remained 0 (root), the program will terminate, otherwise just consider the message harmless (or complain to the author of Net::Server); - temporarily removed utility amavisdconf, it is not yet updated to cope with the new configuration variables; - determine_file_types: 'script text executable' is now considered .txt, not .exe; thanks to Stephane Lentz and Michael Dengler; - report correct system error if Convert::UUlib::decode fails (e.g. permission denied); thanks to Nick Murtagh for figuring out that UUDecode() in uunconc.c tries to decode into a temporary file in /root/tmp/whatever; - avoid the need for a Unix utility file(1) to support option -b; suggested by Clifton Royston considering OpenBSD; - new macro %b for customized notifications and log: it evaluates to the message digest of mail body (MD5, hex). Used as a caching key, can serve to identify messages in quarantine with same body text. - provide a new BSMTP output method that can be used as $forward_method, $notify_method, $virus_quarantine_method or $spam_quarantine_method. It is simple in a sense that it generates one file for each message, so it is not terribly efficient. The parameter after the colon is used as a file name prototype, where %i in the prototype name is replaced by current time, %n by amavis internal message id, and %b by message digest of the body (MD5,hex). For example: $forward_method = "bsmtp:$TEMPBASE/out-%i-%n.bsmtp"; $spam_quarantine_method = "bsmtp:$QUARANTINEDIR/spam-%b-%i-%n.bsmtp"; The bsmtp file has ".tmp" appended to a name during writing, and is renamed to the final name when it is done, to avoid being seen by possible fetcher prematurely. It is experimental in a sense that configuration and implementation details may change in the future, including possible changes in the old 'local:' method; - besides the default (implied) quarantine method 'local:', it is now possible to specify 'bsmtp:...' as a quarantine method, and have different settings for virus and spam quarantines. New settings: $virus_quarantine_method and $spam_quarantine_method, e.g.: $virus_quarantine_method = "bsmtp:$QUARANTINEDIR/virus-%i-%n.bsmtp"; $spam_quarantine_method = "bsmtp:$QUARANTINEDIR/spam-%b-%i-%n.bsmtp"; - if ***SPAM*** is to be inserted into Subject header field, but the original mail does not contain the Subject field, insert one; suggested by Christopher Odenbach; - per-recipient SQL field and static flag (spam_modifies_subj) for modifying Subject: header field when spam is passed; - updated README.lookups, describing new SQL fields and tables; - split kill_level setting into two independent settings: tag2_level, and kill_level. A quick reference: tag_level controls adding the X-Spam-Status and X-Spam-Level headers, tag2_level controls adding 'X-Spam-Flag: YES', and editing Subject, kill_level controls 'evasive actions' (reject, quarantine); The default is tag2_level=kill_level, which retains compatibility. - work around a bug in Perl 5.8.0 in UTF-8 environment, which could produce error "Bad RFC822 field name 'C0c0\x{0}\x{0}T-Type'" when building notifications (the Perl bug is triggered in Mail::Header::_tag_case by statement $tag =~ s/\b([a-z]+)/\L\u$1/gio; reported on several Red Hat 8.0 hosts, seemingly only on Intel platforms). Many thanks to Chris Kloiber and Steve Sloan for making available their test machines for debugging. - replace all calls to qx by run_command, to avoid 'Malformed UTF-8 character' when reading from external utilities that do not honour locale settings, e.g. Unix file(1) utility, which may return non-printable characters with codes above 127. This also ensures better error reporting. run_command() moved into Util module, part of its code moved into a new subroutine run_command_copy(). - improved diagnostics when certain external programs fail to run, such as when fork or exec fails; - new LDAP lookups (experimental), by Jacques Supcik, PhD, IP-Plus Internet Services - Swisscom Enterprise Solutions Ltd, Genfergasse 14, 3050 Bern, Switzerland (http://www.ip-plus.net/) Many thanks for the contribution! - changed enhanced status code 5.7.0 to 5.7.1 when message is considered bad (virus, banned file, spam); - when sending message via SMTP, don't go into DATA part if any RCPT TO returned 4xx code, since without implementing proper queueing we wouldn't be able to handle the situation unless message came in via LMTP; - updated README.postfix to describe: * the now recommended Postfix 2.0 setup with pre-cleanup service, instead of the (post-)cleanup2; * how to specify a backup content filter(s) in case the primary fails; * the Postfix setting if you want X-Amavis-Hold header field to cause message to be placed on hold by Postfix, e.g. when amavisd-new encounters a mail bomb which can not be checked for viruses. - unconditionally delete old headers X-Spam-Status, X-Spam-Level, X-Spam-Flag and X-Spam-Report, when anti-spam code is enabled. Previously deletion was done only when new headers were to be inserted. - replace sloppy file accessibility tests (Perl functions -e, -f, -d, -r, ...) with more reliable method, which also produces more informative diagnostics; - set $SIG{CHLD}='DEFAULT' in Net::Server hooks to make qx work in Net::Server::Single environment (useful for debugging), as well as with the version of 0.85 of Net::Server; - adjust the log level in do_ascii() (reported by Pan Bambaryla), and in few other places; - fix $log_templ - an unconditional comma was causing the log entry to be always issued, regardless of mail content; reported by Stephane Lentz; - enhance reject/discard/pass destinies with reject/bounce/discard/pass, making it more flexible to specify what action to perform on detection of bad content; (thanks again to Chris Hastie for earlier discussion on the topic); see explanations in amavisd.conf; - added hash lookup table %local_domains for those that have a great number of hosted domains and prefer to read a list from a separate file; - a special shorthand provision when SQL lookups are used: when a match for recipient address or domain is found in SQL tables (regardless of field values), the recipient is considered local, regardless of static @local_comains_acl or %local_domains lookup tables. This simplifies life when a large number of dynamically changing domains is hosted. To overrule this behaviour, add an explicit boolean field 'local' to table 'users' (missing field defaults to true, meaning record presence implies locality). - changed SQL table 'policy' field name 'banned_files_lover' to a name 'banned_files_lovers' (added an 's') for consistency with variables %banned_files_lovers and @banned_files_lovers_acl, and updated README.lookups; - SQL field treated as boolean now recognize '\0' character as false (not to be confused with NULL SQL field, which is still mapped to undef); - when generating Received: header field, move receiving port number into the second comment field - some software were too picky about the syntax of the first comment field ('TCP-info' in rfc2821); - consistency: apply late 'string sanitation'; meaning that strings such as virus names, banned attachment names, e-mail addresses, etc, are sanitized at the point where the string is leaving amavisd-new, not at the point where it enters is. This allows the quotation rules to depend on the output properties (log entry, notification text, SMTP response text, ...) / (ASCII, iso-8859-1, UTF-8, binary, ...); - delegate responsibility to call sanitize_str to write_log, instead of having a caller do it; - wrap long syslog entries at 980 characters (less than 1023 minus prefix); thanks to Ted Cabeen; - skip spam-related actions if $spam_level is undefined (e.g. when anti-spam code is not enabled) so that missing $sa_kill_level_deflt setting in amavisd.conf won't cause a surprise. Thanks to Marcio Merlone. - new version of amavis-milter.c based on amavis-milter.c 1.1.2.3.2.36 (from amavisd-0.1); - use the /m regexp modifier by default when parsing av-scanner output for clean and infected regular expressions, making it easier to specify patterns, and to make it like the expression which collects virus names; thanks to Daniel Hunziker for reporting a problem with Symantec CommandLine Scanner; - refined the semantics of sender whitelisting/blacklisting, add per-recipient sender while/blacklisting mechanisms: ENVELOPE SENDER WHITELISTING / BLACKLISTING - GLOBAL (RECIPIENT-INDEPENDENT) WHITELISTING: use envelope sender lookups to ENSURE DELIVERY from whitelisted senders even if the message is recognized as spam. Effectively, for the specified senders, message recipients temporarily become 'spam_lovers', with further processing being the same as otherwise specified for spam lovers. It does not turn off inserting spam-related headers, if they are enabled. BLACKLISTING: messages from specified senders are DECLARED SPAM. Effectively, for messages from blacklisted senders, spam level is artificially pushed high, and the normal spam processing applies, resulting in 'X-Spam-Flag: YES', high 'X-Spam-Level' bar and other usual reactions to spam, including possible rejection. If the message nevertheless still passes (e.g. for spam loving recipients), it is tagged as BLACKLISTED in the 'X-Spam-Status' header field, but the reported spam value and set of tests in this report header field (if available from SpamAssassin, which may not have been called) is not adjusted. A sender may be both white- and blacklisted at the same time, settings are independent. For example, being both white- and blacklisted, message is delivered to recipients, but is tagged as spam. If ALL recipients of the message either white- or blacklist the sender, spam scanning (calling the SpamAssassin) is bypassed, saving on time. ENVELOPE SENDER WHITELISTING / BLACKLISTING - PER-RECIPIENT The same semantics as for global white/blacklisting applies, but this time each recipient (or its domain, or subdomain, ...) can be given an individual lookup table for matching senders. The per-recipient lookups override the global lookups, which serve as a fallback default. One can specify a two-level lookup table: the key for the outer table is recipient, and the result should be an inner lookup table (hash or ACL or RE), where the key used will be the sender. Per-recipient white/blacklisting is also available as an SQL lookup. See configuration variable $sql_select_white_black_list and README.lookups . INCOMPATIBILITIES WITH amavisd-new-20021227 - the previous configuration file amavisd.conf from version amavisd-new-20021227 is usable with amavisd-new-20030314. Nevertheless it may be a good idea to adopt the new amavisd.conf, as it describes new options, is cleaner and better commented; - reordered hash lookups from: user+foo@sub.example.com, user@sub.example.com, sub.example.com, .sub.example.com, .example.com, .com, . , user+foo@, user@ into: user+foo@sub.example.com, user@sub.example.com, user+foo@, user@, sub.example.com, .sub.example.com, .example.com, .com, . to make user@ still useful even when catchall ( . ) is used, and to bring it closer to the recommended SQL order; Not likely to affect existing setups, as previously the combination of user@ and catchall was not useful; - SQL lookups: * incompatible change (avoiding unexpected behaviour): when the requested SQL field name does not exist in the SQL table, return a sensible default value for boolean fields (according to field name and its type), instead of 1 as previously. Logs a warning when no default is available. Thanks to Ortwin Gentz for noticing the problem; * new fields spam_tag2_level, spam_modifies_subj and local, with useful defaults if absent, so it is not necessary to immediately update an existing database; * changed SQL table 'policy' field name 'banned_files_lover' to a name 'banned_files_lovers' (added an 's') for consistency with variables %banned_files_lovers and @banned_files_lovers_acl; --------------------------------------------------------------------------- amavisd-new-20021227-p2 (2003-01-10): - sender notification were not sent for messages containing banned filenames but no virus; fixed (thanks to Philip Ross); - added virus name 'Sobig' to the default $viruses_that_fake_sender_re (amavisd.conf); - extended default $banned_filename_re to match also a longer (4 character) middle file extension (e.g. Movie_0074.mpeg.pif) (amavisd.conf); - add explicit path to the program vscan in the 'Trend Micro FileScanner' entry (amavisd.conf), thanks to Matthew Hobbs; - improvements in the setup of helper programs (files configure.in, Makefile.in, config.h, amavis.c and amavis-milter.c), by Radu Greab; - add Sophos error code 527 ('Not supported in this SAVI implementation') to the list of (mostly)harmless codes in the SAVI-Perl interface routine (thanks to Paul B. Henson and Tim Winders for discussion); - initialize SAVI-Perl module (if available) in each new child, instead of only once during daemon startup time. This avoids the need to reload amavisd for every change in the Sophos IDE database - at the expense of additional SAVI-Perl startup time every $max_requests messages; --------------------------------------------------------------------------- December 30, 2002 amavisd-new-20021227-p1 (2002-12-30): fix the problem when using multiple virus scanners, where the last scanner doesn't find a virus, but some other scanner does, and amavisd-new (incorrectly) does not report a virus. Thanks to Okke Timm, Henrik Larsson and Eduardas Paulavicius; --------------------------------------------------------------------------- December 27, 2002 amavisd-new-20021227 release notes amavisd-new-20021227 is primarily a maintenance release, consolidating amavisd-new-20021116 with its mandatory patches, updating documentation, but also brings a couple of niceties. Changes since amavisd-new-20021116 patch level 5 - The new configure/makefile in the helper_progs subdirectory is contributed by Stephane Lentz, based on MIMEDefang configure.in. It is needed to build helper programs for sendmail (milter and nonmilter) setups (but not needed for Postfix and Exim v4 setups, or any dual-MTA setup). Similar to the prior work done by Ramiro Morales - thanks to both! - Updated documentation (README, INSTALL, RELEASE_NOTES, README.lookups, README.customize, README.postfix). - new subdirectory test-messages, containing sample mail messages to test virus and spam detection - see README there - For daemonized antivirus scanners (only) the semantics of few parameters in the @av_scanners list has been incompatibly changed and enhanced. The same internal mechanism now handles both Unix sockets and inet socket. More than one socket may now be specified, to be tried one after the other until one is successfully connected to (useful for 'FRISK F-Prot Daemon'). Also the usual regexp pattern parameters in the @av_scanners are now observed by Sophie and Trophie interfaces as well, bringing them more in line with other scanners. - The (?m) regexp flag in @av_scanners patterns is now turned-on by default, so it is no longer necessary to specify it in @av_scanners, and has been removed to shorten the expressions. - The SAVI-Perl interface routine now turns on most Sophos engine options, enabling recognition of some more file formats. - All active antivirus scanners are now called, the scan no longer stops when the first one finds a virus. This is on the average a very small price to pay (only payed in virus storms), but makes possible to see which virus scanners found the virus and which did not, and to get their timing in the logs. The list of virus scanners that noticed the virus is now included in the notification message to the virus administrator. A new macro %W is available for this purpose. - More sensible logic when checking more than one file for those daemonized av scanners that need to be handed individual files instead of directory name. - The 'Clam Antivirus-clamd' daemon is now supported, both via Unix socket, as well as via inet socket (but Unix socket should be used, due to lack of self-protection in this respect by clamd). clamd is a fast and open-source anti-virus scanner ( http://clamav.elektrapro.com/ ), and I recommend it to be used in combination with other commercial virus scanners as the last resort scanner, to avoid bouncing mail in case primary scanner(s) stumble across some unhandled situation, like corrupted or encrypted mail. - New utility program 'amavisdconf' to check the values of configurable variables set by /etc/amavisd.conf or defaulted (modeled after Postfix utility 'postconf', but -e option is not yet supported). Usage: amavisdconf [-d | -n] [-c conf-file] [variable] Options: (none) show the value as will be used by amavisd; -d show default value as in the absence of the config file; -n only show variable if its value is different from the default; -c conf-file ... use the specified file instead of /etc/amavisd.conf If a variable is specified show only that variable, otherwise show all configurable variables. (Note: the leading $, @ or % must be included with the variable. Make sure to protect $ from shell evaluation, e.g. by enclosing variable name in single quotes, or prepending a backslash: $ amavisdconf -d '$forward_method' $ amavisdconf -n This is the first attempt at such a utility, some details would need to be polished (e.g. how to report code reference), but it is hoped to be usable already in its present form. - Two new subroutines (read_hash and read_text) are now available for use from amavisd.conf file, with a primary intent to move larger texts out of the amavisd.conf file: * read_hash allows for lookup hashes (e.g. white/blacklist, static %spam_lovers table, ...) to be read from a file during startup; It takes quoted form of addresses, optionally enclosed in angle brackets, and converts them into unquoted (raw) form, as needed for the hash keys. Format: one address per line, everything from '#' to end of line is a comment, leading and trailing whitespace removed, empty (or comment-only) lines are ignored. * read_text allows for reading more complex strings (e.g. multi-line, or including special characters such as backslash) from a file. Useful to override the default template text for notification messages. See example usage in amavisd.conf. - Three synonym (alias) subroutine names are available for use in amavisd.conf to shorten the expressions: new_RE same as Amavis::Lookup::RE->new ask_daemon same as Amavis::AV::ask_daemon sophos_savi same as Amavis::AV::sophos_savi amavisd.conf file is changed to use the short names, but full subroutine names are still valid. - Provide protection mechanism for SMTP/LMTP server code to limit the number of recipients in a mail transaction to some sane value ($smtpd_recipient_limit, default 1000). Even if the number is exceeded, mail will still be normally delivered in more than one transaction (provided your MTA is rfc2821 compliant). - SMTP client code now correctly handles the situation where remote SMTP server returns '452 Too many recipients'. When this happens the mail forwarding or submission is split into several SMTP transactions, and as long as the remote SMTP server is willing to accept at least one recipient per transaction, mail delivery is assured. - Auto white-listing (AWL) by SpamAssassin may be turned on by setting $sa_auto_whitelist to true (contributed by Hamish Marson). This only works as one global database, the per-recipient AWL is presently not available. - New configuration variables $hdrfrom_notify_sender, $hdrfrom_notify_admin, and $hdrfrom_notify_spamadmin, to remove the overloading (double semantics) from variables $mailfrom_notify_*. Now $mailfrom_notify_* control ONLY envelope addresses, while $hdrfrom_notify_* only controls From header in notifications. The default values for $hdrfrom_notify_* are chosen to preserve compatibility. - The spam headers in quarantined message now reflect the lowest tag and kill levels when more than one recipient is given, instead of reporting the levels for the first specified recipient. - Tested with SpamAssassin 2.50 and with its new Bayesian classification (used as a global database by amavisd-new). Given two sets of spam and non-spam (=ham) messages, one can build a database by using new SpamAssassin utilities sa-learn-spam and sa-learn-nonspam as user amavis, and the database built this way will be used by amavisd-new/SpamAssassin. - Initialize SpamAssassin as $daemon_user (no longer as root), as a safety measure, and to ensure the files it creates are still accessible after the UID change. Incompatible changes to 20021116: - Different parameters in @av_scanners list for daemonized antivirus scanners (as described above). - lookup_hash: an undefined hash value was previously treated as a special case, resulting in value true. This is no longer the case, which brings it in line with other lookup methods. It is not likely anyone will be affected, as this practice was only documented but not encouraged by any example. --------------------------------------------------------------------------- Here is a brief description of patches that needed to be applied to amavisd-new-20021116, and were available from the amavisd-new web page: patch5 (2002-12-09): - fix a simple test to decide if the anti-virus and anti-spam code is needed; - use the /m regexp modifier by default when parsing av-scanner output for virus names - seems like (?m) in qr is ignored (have we found a Perl bug?); - fix DSN to report original smtp response in case reporting MTA is remote (thanks to Chris Hastie for his keen eye in understanding DSN); - fix reporting quarantine mailbox name(s) in case of per-recip quarantine or quarantine forwarding to MTA; - macro %q (quarantine id) is now a list of quarantine mailbox names or addresses (may have more than one entry in case of per-user quarantines); - macro %v (av-scanner output) is now a list of scanner output lines, to facilitate formatting of multi-line av-scanner output; - fix spam admin notifications, which used the virus template instead of the spam template; - fix problem of undefined entity causing spam notification failure in case other content checking (virus/banned) is not enabled; - make SQL lookups (DBI access) optional - no DBI code is loaded if @lookup_sql_dsn of empty; - support SAVI-Perl module 0.10 or later (direct access to the Sophos library engine) ( http://www.csupomona.edu/~henson/www/projects/SAVI-Perl/ ); - add macro %j that evaluates to message 'Subject:' header field; - fix minor SMTP protocol inconsistency when multiple MAIL FROM commands are issued by the client; - more informative timing report when more than one av-scanner is used; - make sendmail initial submission happy when null reverse-path is specified (instead of the correct empty argument, which works but makes sendmail issue a warning, we now use <> in arguments to specify null paths), (thanks to Sebastian Hagedorn and Dale Perkel); - replace log entry "spam from=<%s>, to=%s" with "SPAM, <%s> -> %s ...". - dropped (redundant) historical %bypass_checks @bypass_checks_acl (use bypass_virus_checks and %bypass_spam_checks instead); patch4 (2002-11-21): In amavisd: - SQL lookup fix; - finished per-recipient header edits and split-forwarding, where one multi-recipient message may be forwarded as several messages if different headers need to be inserted for different recipients (not with milter); - the check for banned file contents now looks for both the short and the original full type classification as provide by 'file' utility (thanks to Wolfgang Lumpp for the idea); - #!/usr/local/bin/perl => #!/usr/bin/perl - cosmetics In amavisd.conf: - suggested group name change $daemon_group 'sweep'=>'amavis'; so that non-Sophos users are not puzzled where 'sweep' group comes from; - added path and fixed args for KasperskyLab AVPDaemonClient (thanks to Mike Hall); - added path for Trend Micro FileScanner vscan (thanks to Eduardas Paulavicius); - added 'use strict;' and '1;' at the end; prevent read_config from failing if the config file does not return true (thanks to Vivek Khera); - cosmetics, fixed typos in comments; patch2 & patch3 (2002-11-18, 2002-11-19): - fixes a problem with 'recip_done' object method (causing TEMPFAIL) when $warnvirusrecip is enabled (thanks to Chris Hastie, Dale Perkel, and Didi Rieder); - pass null return path to sendmail in such a way to make it happy (thanks to Sebastian Hagedorn and Dale Perkel); - added a safety precaution to prevent somebody shooting himself in the foot by specifying an empty $forward_method while using the SMTP in/out setup (e.g. Postfix), which could send mail down the bit bucket; patch1 (2002-11-17): - fixes a problem in milter setup where per-recipient response codes needed to be handled differently (the problem was causing a TEMPFAIL) (thanks to Stephane Lentz and Didi Rieder for the initial milter testing and problem reports); --------------------------------------------------------------------------- November 16, 2002 amavisd-new-20021116 release notes - provide a mechanism to load only required code sections (anti-virus scanning, anti-spam scanning, SMTP/LMTP server module, traditional amavis client & milter server module), resulting in reduced memory usage and less installation dependencies; - introduce the per-recipient status handling and make possible some per-recipient functionality that was not possible before, such as handling the per-recipient spam thresholds. This required a major rewrite and cleanup of some sections, hopefully providing cleaner mechanism for possible future advances in this area. Consider for example a previously incorrectly handled situation where there is more than one recipient, and some (but not all) of them get a reject from the (outgoing) relay MTA - it is not possible for a single SMTP response or status code to describe the situation; - generate proper (non-)delivery status notifications (DSN), compliant with rfc1892 (now rfc3462) and rfc1894 (now rfc3464). This form supersedes the simple virus/spam sender notifications, but also covers the area of genuine delivery failures, such as selective recipient rejects by the outgoing relay MTA, which could previously lead to multiple deliveries; NOTE: to avoid sender getting two non-delivery messages (one from MTA, and another from amavisd), do not set $warnvirussender and $final_virus_destiny=-1 (REJECT) at the same time (and equivalently for spam settings); - enhanced mail system status codes (rfc1893 (now rfc3463), and rfc2034) are now included with all SMTP responses and DSN notifications; - added checking for banned MIME types and names. If any mail part matches, the whole mail is rejected, much like the way viruses are handled. A list in object $banned_filename_re can be defined to provide a list of Perl regular expressions to be matched against each part's: * Content-Type value (both declared and effective mime-type), including the possible security risk content types message/partial and message/external-body, as specified by rfc2046; * declared (recommended) file names as specified by MIME subfields Content-Disposition.filename and Content-Type.name, both in their raw (encoded) form and in rfc2047-decoded form if applicable; * file content type as guessed by 'file' utility and classified into short type names such as .asc, .txt, .html, .doc, .jpg, .pdf, .zip, .exe, ... - see subroutine determine_file_types(). This step is done only if $bypass_decode_parts is not set. NOTE: by default the $final_banned_destiny is set to 1 (pass), so detected banned file names only cause a header line to be added, quarantining, and added address extension - but the mail is delivered nevertheless. This default is set so that we can get initial experience without being too obtrusive. Change as you prefer. - besides SMTP/ESMTP protocol, the server now also accepts LMTP protocol (rfc2033). This now explains why I had to implement PIPELINING, ENHANCEDSTATUSCODES and 8BITMIME SMTP extensions, as these are required by rfc2033. One advantage of using LMTP to feed content filter is that LMTP uses per-recipient status response, as opposed to 'one-size-fits-all' SMTP status response, which require that the SMTP client (e.g. content filter) is capable of generating DSN. Another advantage is specific to the Postfix setup, as Postfix LMTP client supports multiple transactions per session, saving on connection teardown/reconnect for every message being checked. To enable LMTP feed from Postfix, add 'max_use=10' to main.cf, and replace last argument 'smtp' with 'lmtp' in the master.cf line: 'smtp-amavis unix - - n - 2 lmtp'. No changes are needed in amavisd-new, both protocols can coexist, the distinction is based on HELO/EHLO vs. LHLO command. Btw, older versions of Postfix lmtp client inappropriately lowercased the envelope addresses. This is fixed in Postfix Snapshot 1.1.11-20021015, and in the regular Postfix release 1.2 (when available). Lowercasing can be a problem for addresses where local part is case-sensitive, although such setups are rare. - improved per-recipient SMTP response code handling when sending mail via SMTP. Previously one rejected recipient (by MTA) in a multi-recipient message caused the whole message to me rejected. This is now correctly handled with the new ability to send DSN; - added command line option '-c config-file' so one can override the default location of the configuration file (/etc/amavisd.conf); - explicitly set PATH and HOME environment variables (settings: $path and $helpers_home) - added another form of lookups: Perl regular expression matching. See README.lookups for details. Corresponding new variables in amavisd.conf are: $virus_lovers_re, $spam_lovers_re, $bypass_virus_checks_re, $bypass_spam_checks_re, $local_domains_re, as well as $banned_filename_re, $viruses_that_fake_sender_re, $keep_decoded_original_re, $whitelist_sender_re, $blacklist_sender_re. - besides whitelist_sender* lookup tables, there is now also a blacklist_sender* set of tables, which causes mail to be declared spam and to skip remaining spam checks. See examples in amavisd.conf. - provide a configurable lookup table $keep_decoded_original_re of file types, for cases where unpacker is not very trustworthy. The lookup key is what 'file' utility returned. If the part contents match the lookup table, we keep both the original and the unpacked file for virus scanner to check; - provide a configurable list of regular expressions $viruses_that_fake_sender_re, which recognizes viruses that usually fake envelope sender address. Don't send sender notification if a match is found. For syntax of the new regexp lookup tables see README.lookups; - specifying per-recipient quarantine address or location is now possible by setting $virus_quarantine_to and $spam_quarantine_to be a ref to a hash lookup table. Thanks to Vivek Khera for the idea; - no 'configure' and 'make' for the daemon; all configuration is done via amavisd.conf configuration file at the daemon startup time; Also the ./mta, ./av and ./notify configuration subdirectories are now gone; - provided a sample init shell script amavisd_init.sh (edit to will, and move it to /etc/init.d/amavisd if you want). Based on the script from Wil Cooley; - one-shot debugging mechanism: if envelope sender matches @debug_sender_acl lookup table, turn debugging fully up just for this one message and cause temp file and directories not to be cleaned for this message. This facilitates debugging a particular problem even in the presence of regular traffic; - cleaner debug log entries for multi-transaction SMTP/LMTP protocol sessions; - cleaner log entries - avoid misleading and incomplete text when quarantine is disabled; thanks to Michael Leone; - avoid using (nonstandard) field width in the %e format specifier when calling strftime, which lead to bad date syntax on some systems; - local time zone offset is now automatically computed, no more '-0000' in rfc2822 dates; - put Message-ID field value in angle brackets as required by rfc2822 when generating notifications; - add a In-Reply-To field to notifications when original Message-ID is known; - don't bounce a virus (or spam) back to a mailing lists even if $final_virus_destiny (or $final_spam_destiny) is set to REJECT; a patch by Brian May from the Debian support crew, thanks; - because some external module may play games with STDIN and STDOUT (like SpamAssassin seems to do when local_tests_only=>0) run the input protocol directly on the Net::Server's socket, not on STDIN and STDOUT, which are aliases to the socket; - a patch to amavis-milter.c by Didi Rieder to support REJECTing mail, by instructing sendmail to return non-delivery notification to sender; - a modified amavis.c helper program to make possible to invoke local delivery agent from it, for those still using such a setup; - a new macro %l is available for use in notification messages (via 'expand'); it evaluates to true (1) if the sender matches @local_domain, and returns empty otherwise; by default it is now used in creating a 'Subject:' line, inserting word 'LOCAL' before the sender name when appropriate; - new macros %D and %N expand to lists of recipients that got the mail delivered (%D), or not-delivered (%N). Union of both sets gives %R, i.e. a list of all recipients as specified in the envelope; - dropped macro %a - improved parsing of e-mail addresses according to rfc2821 (full address literals syntax, etc.); - proper line folding for generated rfc2822 header fields; - proper SMTP response wrapping (for very long responses such as the ones that include a trouble report) according to rfc2821; - allow for obsolete rfc822 syntax of permitting whitespace before colon in header field name; - added optional spam-sender nondelivery notifications, based on patch from Lazslo E. Miranda (lazslo@dcc.ufmg.br) and Fernando F. Morais (frota@cecom.ufmg.br); - dynamically change process name (Perl variable $0) to reflect the process state; suggested by Chip Paswater. Not all operating systems make this process state visible by ps(1); - determine location of external programs (or their absence) at startup time; in amavisd.conf one may specify absolute path or just rely on PATH. This mechanism is also used to determine absolute path of the daemon itself, making reload (after HUP) more predictable; - explicitly specify lock file for serialization to be used by the Net::Server::PreForkSimple module, instead of relying on default provided by POSIX::tmpnam. The default approach has a possibility for a minor security problem, because the lockfile is created with open()..., so it will follow symlinks. Observed, and a patch provided, by Jarno Huuskonen; - new defense against mail bombs: for the cumulative total of all decoded mail parts we set max storage size. The formula is: quota = max($MIN_EXPANSION_QUOTA, $mail_size*$MIN_EXPANSION_FACTOR, min($MAX_EXPANSION_QUOTA, $mail_size*$MAX_EXPANSION_FACTOR)) In plain words (later condition overrules previous ones): allow MAX_EXPANSION_FACTOR times initial mail size, but not more than MAX_EXPANSION_QUOTA, but not less than MIN_EXPANSION_FACTOR times initial mail size, but never less than MIN_EXPANSION_QUOTA - if the permitted quota is exceeded (or the defense triggered by other similar safeguards), the virus scanning is skipped to protect the virus scanner from tripping over the mail bomb, a header field is inserted: X-Amavis-Hold: ... reason ... and a log entry 'Placing on HOLD: reason' at level 0 is produced. Also the temporary directory is preserved. The 'X-Amavis-Hold: ...' header field can be used by your MTA to put the message 'on hold' (freeze). If MTA is not set up to catch messages with this tag (the default), they are passed normally to recipients. This is likely to be the best action under the circumstances. - tested with razor-agents-2.20 and SpamAssassin-2.43 (Razor2 is now called by SpamAssassin, and no longer directly by amavisd-new). New configuration variable $helpers_home, which defaults to $TEMPBASE. Thanks for hints by Chris Hastie and John Stewart; - to avoid CPU loop in SpamAssassin-2.43 and earlier, my patch needs to applied to SA - see amavisd-new web page; - to avoid taint problem in Razor 2.20 (if SpamAssassin-2.43 is configured to call it), my patch needs to be applied to it - see amavisd-new web page; - chroot available (but not well tested): $daemon_chroot_dir = '/var/amavis' - provide a fail-over mechanism for SQL database connect - given a list of SQL servers/databases, pick the first that is available. Thanks to Ken McKittrick for making available the patch, and to Ben Ransford <ben@terakeet.net> for writing it. - remove existing virus-related and spam-related headers (some of them optionally) if we'll be providing our own; suggested by Borut Mrak; - avoid 'insecure dependency' in lookup_sql when calling DBI::execute (thanks to ric* at mpc.com.br) - fixed macro %H to provide original header lines, not the ones stored internally by MIME::Entity, which may have been modified; (thanks to Chris Hastie for noticing the problem); - no longer inserts X-Razor-id header field; INCOMPATIBLE CHANGES: - when specifying boolean values to variables in the amavisd.conf file, please specify 1 (or old style "yes") for true, and 0 or "" or undef for false. The old style "no" yields true for Perl, and is only still supported (converted to 0) for some traditional variables for compatibility with amavisd(-snapshot); - removed variables $sendmail_wrapper*, and changed the syntax for specifying $forward_method/$notify_method, with the intention to do all mail sending settings at one place. See amavisd.conf for examples; - variable $mailto is now deprecated (but still works as a fallback default for compatibility with previous version). Use $virus_admin and $spam_admin lookup tables instead, they also offer a mechanism to specify per-domain administrator address; - variable $warnadmin is no longer used. Use $virus_admin and $spam_admin lookup tables instead. Not specifying administrator address (e.g. leaving $virus_admin, $spam_admin and $mailto undefined) turns off admin warnings. - dropped variable $LOGDIR, the variable $LOGFILE now specifies the full path; - renamed: $warnsender -> $warnvirussender, $warnrecip -> $warnvirusrecip - dropped variables $enable_relay, $sendmail_cf_orig, $QMAILDIR; they were never used in amavisd-new; - dropped $MAX_ARCHIVE_NESTING, which is replaced by new storage limitations: $MIN_EXPANSION_QUOTA, $MIN_EXPANSION_FACTOR $MAX_EXPANSION_QUOTA, $MAX_EXPANSION_FACTOR - SQL database is now specified differently (@lookup_sql_dsn) MTA-SPECIFIC: sendmail/milter - $notify_method now specifies deferred delivery mode ('-odd') by default, when submitting notifications to sendmail. This is to avoid calling milter immediately during submission, which in turn calls amavisd-new, possibly leading to a deadlock situation when the number of amavisd-new child processes is small. Seems like this change is needed since Sendmail 8.12 or so. The following recommendation is from mimedefang-filter man page: | You MUST run a client-submission queue processor if you use | Sendmail 8.12. We recommend executing this command as part of the | Sendmail startup sequence: | sendmail -Ac -qp1m DECODERS-UNPACKERS: - rewritten decode_parts() to allow for retaining source text if the unpacker sw is considered unreliable. This more cleanly resolves the problem reported on the amavis-user mailing list on 2002-06-06: "Amavisd passing through VBS/VBSWG.gen@MM" A nice side-product is that a directory search is avoided for each nested unpacking step; - refined do_unzip to control and limit the size of decompressed members (among others it defends against the 42.zip-type bombs); - rewritten external decompressors interface to the gzip/bzip2/compress family. Instead of using a 'system' call, they are now called through fh_copy, making possible to control and limit the size of the decompressed contents on the fly, avoiding denial-of-service attacks. Affected: b(un)zip2, g(un)zip, (un)compress; - fh_copy now uses IO::Handle object to assure the forked process gets reclaimed even in case of aborted contents extraction; stdin gets redirected to /dev/null or to a specified input file for the exec'd process; - Convert::UUlib is called again (the amavisd-new-20020630 removed its usage due to problems with improper decoding). This time the originals are kept, so that virus checker sees both the original and the attempted-decoded part; - completely rewritten do_ascii and its usage of Convert::UUlib to fix: * coding error (bitwise op treated as logical op and improperly negated): ... if (!$uu->state || !FILE_OK || -z $newpart); * it never reset the state, so if a successfully decoded ASCII file contained another ASCII file, each decoding level would decode all previously decoded parts again, plus add new ones at this level; recursion would not stop until the hard limit, resulting in TEMPFAIL; * it never checked nor reported errors that should have been detected (I/O errors, out of memory, trouble accessing or creating files); * more informative log entry; - save MIME preambles and epilogues (if nonempty) as extra (pseudo)parts to be scanned. This also mitigates the problem of syntactically-incorrect MIME mail as produced by some user agents, which (rightfully) gets treated as one long preamble by MIME-Tools, and previously went by unchecked; - supports unpacking arc archives using 'nomarch' (by Russell Marks, http://rus.members.beeb.net/nomarch.html), (thanks to David D. Kilzer for the initial code). Using 'nomatch' fixes a nasty habit of arc which gratuitously appends a form-feed at the end of file when using the 'p' (pipe) option, which might mislead a virus scanner. Besides, 'nomarch' is GPL licensed; - fixed a 'broken pipe' problem when calling unrar, thanks to Ricardo Campos Passanezi and Rainer Link; VIRUS SCANNERS: - rewritten interfacing for most command-line virus scanners. All settings for them is now done in amavisd.conf. New ones may be added without having to modify the daemon source. More complex scanners (e.g. daemonized scanners) still need to have a corresponding interface routine in the daemon; - rewritten Sophie and Trophie interfaces to be more resilient to Sophie/Trophie daemon restarts during virus database reloads, avoiding an unnecessary retry (TEMPFAIL) - thanks to Cor Bosman for the suggestion and code, and to Dale Perkel for testing the Trophie interface; - make sophos sweep tolerant to encrypted attachments: if all files are password protected, then the scanner failure is ignored and the message is allowed to pass. Based on patch by Radu Greab; - updated nai uvscan interface to recognize the result 'Found trojan or variant Exploit-CodeBase !!!', a patch by Anton Berezin; - Clam Antivirus supported; - fixed a problem in 'avp' scanner interface ($TEMPBASE not imported), thanks to Joshua E Warchol; - avpdc (KasperskyLab AVPDaemonClient): recognize additional exit codes, based on patch from Christian Hammers; - Panda new regexps, no TERM vt100 setting (thanks to Benjamin Zwittnig); - Trend vscan exit code seems to be the number of infected parts. Updated the test to reflect that, based on observation from Stephane Lentz; - MksVir scanner interface returns error code 2 if viruses are removed (if --clean option is passed) - a patch by Robert Litwiniec; ---------------------------------------------------------------------------- June 30, 2002 amavisd-new-20020630 release notes Since it seems like several people are adapting amavisd-new in details to their requirements, and certain improvement requests have much in common, I'm making available the 20020630 release, shortly before leaving on vacation. As this means my support will be absent for the coming few weeks, and this release is perhaps by few days premature, please consider it primarily a development and new features release. It is available at the usual location at: http://www.ijs.si/software/amavisd/ (or ask Google about 'amavisd-new') Having said that, it is still a fully functional and tested version, and it is running in production at a couple of sites now. Also it is my version of choice in view of dependability, having it running at our site while I'm away. If you get into trouble, you can still go back to amavisd-new-20020517 with which it is fully upwards compatible. The main changes and features since amavisd-new-20020517 are: - the code is thoroughly rearranged, interfaces cleaned, separated into namespaces (packages), several sections generalized (e.g. lookups, appending/editing header lines). The AV scanner and unpacking sections are still mostly the same and compatible with amavisd, so whatever improvements and new AV scanner support becomes available for amavisd, applies almost without a change to amavisd-new; - SMTP on the input side (used with Postfix and Exim) now talks ESMTP (rfc2821) and not just rfc821, including some SMTP extensions: command pipelining (rfc2920), message size declarations (rfc1870), and 8bit-MIME transport (rfc1652). The main reason for this was the change in recent Postfix versions which can now do MIME transformations to support 7bit transports (implied by SMTP). To ensure the transparent 8bit path and avoid message transformation by MTA, amavisd-new needs to declare it does present an 8bit-clean path. A side benefit is a little speedup in passing chunks of mailing list addresses due to pipelining support; - split certain previous amavisd.conf settings (variables) into several variables or lookup tables: * sender address for notifications: $mailfrom -> $mailfrom_notify_admin, $mailfrom_notify_sender, $mailfrom_notify_recip, $mailfrom_notify_spamadmin (these may also be empty to specify null reverse path <>, which is most useful for sender notification); * administrator address for notifications: $mailto -> $virus_admin, $spam_admin (per-sender lookups) * %bypass_checks -> %bypass_virus_checks, %bypass_spam_checks - supports SQL database lookups via Perl module DBI (interface to popular database types). Some examples are provided, if you need other SQL lookups just modify the code by analogy. SQL lookups are most useful for per-user settings of virus_lovers, bypass_virus_checks, bypass_spam_checks, spam thresholds etc. when the user base is large and subgroups can not be identified through their (sub)domains. Another use is for dynamically changing settings without having to restart amavisd-new; - can optionally insert 'Received:' header if acting as a mail relay (not with milter); does loop detection as required by rfc2821 section 6.2; - notification messages now contain 'Date:' and 'Message-id:' headers; - quarantined viruses contain X-AMaViS-Alert header line with names of detected viruses; - quarantined spam contains X-Spam-Status and X-Spam-Level header lines; - optionally send spam admin notifications, which include the full SpamAssassin spam report and message header; - when started as root, changes UID and GID to $daemon_user, $daemon_group; - to facilitate startup scripts and debugging, supports few simple command line parameters: amavisd ... standard run: changes uid/gid and daemonizes amavisd start ... same thing amavisd debug ... starts with full debug level, stays attached amavisd reload ... finds amavisd master process and sends it a HUP amavisd stop ... finds amavisd master process and sends it a TERM - some more (minor) configurable options: $daemonize, $pid_file, $replace_existing_extension, $localpart_is_case_sensitive - no longer calls Convert::UUlib for uuencoded, xxencoded, and binhex attachments. The first two are handled by MIME::Parser, the remaining are likely to be handled by anti-virus scanners, especially when some virus would use such encoding. The use of Convert::UUlib is dropped because it was causing recent problems with garbling virus so that it could no longer be recognized by AV scanners, and because the underlying library does not seem very dependable. See the thread 'Amavisd passing through VBS/VBSWG.gen@MM' from the beginning of June 2002 in amavis-user mailing list archives; - quarantine files now include internal amavis id in the file name, instead of the process number; previous naming scheme could stumble across a name contention on a busy system; - only a recommendation: logging via syslog is now preferred to direct logging to a file. It serializes the logging, and avoids locking/unlocking and reopening a log file by amavisd* for every log entry. The syslog daemon does it more efficiently and reliably. NOTE for Linux users: make sure you prefix the file name in syslogd.conf with a '-' tag to disable fsync after every write; this is most necessary for heavy logging such as from MTA and/or amavisd-new. The downside is that you may lose the last few log entries in case of machine crash. See man page of your syslogd for details. - changed examples according to rfc2606 and recent complaints on the postfix-users list. CAVEATS: - header rewriting is only available in SMTP-in/SMTP-out setup, i.e. with Postfix and Exim, but not with sendmail milter setup; - the SQL lookups are a very recent addition and not so well tested as the rest of the program. Also since SQL lookups are supposed to introduce per-user settings (e.g. spam thresholds), the code still does not accommodate it, and spam thresholds of the first recipient in a message affects the whole message; - as Razor2 is still pretty much unstable, I left the spam_scan() routine much as it was in the May version. Contributions welcome; - as I ran out of my time for this release, I didn't prepare a separate version without SpamAssassin and Razor 1.20 support, so you will need to strip it out if you do not need it. Similarly for the required DBI Perl module. I'm very sorry; - the time stamp used in the 'Date:' header in notifications, and in an optional 'Received:' header, does not contain true time zone offset, but -0000 instead (standard meaning for unknown). I did not want to include a fat Perl package for handling time zones. For now just edit the subroutine rfc2822_timestamp() if you want to change that; - amavisd.conf settings $daemon_user and $daemon_group are not automatically set by ./configure options. Please set them manually. I would like to thank many people on the amavis-user list, on the postfix-users list, and in private conversations, who contributed valuable ideas and improvements, and offered much appreciated encouragements. --------------------------------------------------------------------------- May 17, 2002 Available at: http://www.ijs.si/software/amavisd/ amavisd-new-20020517 is primarily a response to popular demand for Mail::SpamAssassin support. If amavisd-new-20020424 meets your needs, there is no urgent need to upgrade. There are no incompatible changes between these two versions, except some new (optional) amavisd.conf variables, so you may keep old amavisd.conf file if you wish. A new file README.exim is now provided, so that Exim 4.x is now a supported and tested configuration, besides the usual Postfix and sendmail/milter MTAs. Also included is a brand new qmail amavis client by Lars Hecking (untested, please try it), plus his updated version of amavis.c (no longer needed in recommended Postfix and Exim configurations, and Sendmail milter and qmail configurations use a different client). Several files are unchanged from the base amavisd CVS release 2002-05-13 and do not reflect the amavisd-new state: FAQ, HINTS, INSTALL, BUGS, NEWS, TODO, ChangeLog, tests/, doc/. Please start with this file README.amavisd-new-RELNOTES and follow it. More recent instructions and last-minute changes are available from the web page. Changes since amavisd-new-20020424: - supports Mail::SpamAssassin and Vipul's Razor (1.20 required) for spam checking (but not for modifying mail body - only add headers and/or address extension, or reject/discard/quarantine spam). NOTE: spam checks are off in the default amavisd setup. Copy file ./amavis/amavisd.in.all to ./amavis/amavisd.in before running ./configure and make, to use the spam-check-enabled version! Amavisd calls Mail::SpamAssassin directly, avoiding the need to set up spamc/spamd or to chain filters. This is more efficient, one daemon less to worry about, although maybe less flexible for some taste. At the moment the SA per-user database is not used. Feel free to experiment with it and let me know what you came up with. The usual SA config files are observed, but remote tests are disabled by default (Razor is handled directly, RBL lookups can/should be performed early by MTA which knows what IP address mail came from, and that information is (mostly) lost afterwards). If you feel otherwise, change the hard-wired settings in the call: $spamassasin_obj = Mail::SpamAssassin->new( { dont_copy_prefs => 1, local_tests_only => 1 } ) (also the Razor score contribution is hard-wired, modify: '$razor_spam_found ? 3 : 0' to will). Amavisd-new handles Razor checks directly to be able to exercise more control over it than would otherwise be available through SA: timing, signatures are needed for insertion into header, skips one-liner body checks which are common Razor false-positives. To avoid SA calling Razor again, either keep the default setting 'local_tests_only=>1', or set 'score RAZOR_CHECK 0' in the SA configuration. Added are 'whitelist_sender' hash and ACL lookups (see README.lookups), which approves spam from specified SMTP originator addresses - SpamAssassin can only check and whitelist rfc822 headers, not the envelope addresses, and I see no way of passing envelope addresses to it - which is a pity, as important information is lost. SpamAssassin checks are computationally quite expensive compared to other amavisd activities, and the time needed for SA check goes up significantly with the message size. SA check are skipped (but not the Razor check) if mail size exceeds 64k. My analysis shows that presently less than 1% of spam exceeds 64k characters, and this is probably well below the false-negative SA rate, so it is not worth wasting time to check large mail. remaining changes: - new file README.exim (thanks to Jochen Erwied, Patrice Fournier and Igor D'Astolfo); - updated README.postfix to describe how to avoid running header_checks, body_checks and dns_lookups in Postfix twice; - put back file README.customize, which was omitted from amavisd-new-20020224 by mistake (but available with previous version and on the web); - added introductory paragraph to README, the rest still needs to be updated; - added missing last-step check for '.' in hash lookups to match examples and make it more useful; - added timeouts to certain tasks to make it better suitable for unattended operation (less, but still subject to certain DoS attacks, similar to the official amavisd). The main reason for adding timeouts is that we don't want spam checking to slow down amavisd operation too much: if it takes too long, just skip it (assume not spam) and move on; - change unmangle_sender() to believe sender address for Klez viruses; this is sometimes wrong, but seems like people prefer to sacrifice few false accusations in favour of some warranted sender notifications. Feel free to hack this routine at will (and publish good ideas), it is intended to be modified; - use Perl module Errno instead of errno.ph to avoid using broken file errno.ph on some popular platforms; - make possible to have per-sender-domain administrator e-mail address for admin notifications (hash lookup %mailto, sub warn_admin() ) - relax temporary file/dir protection to allow them to be readable by the group. This makes possible for anti-virus (daemon) checker to run under a different user (but in the same group). Doing so should be safer as it makes impossible for virus-checking daemon to clobber files. It is only supported in the all-SMTP configuration. To use it with traditional amavis clients (e.g. milter), you will have to modify their sources to change umask and mods for file/dir they create; - avoid changing sender address <> to <""> under certain circumstances (not strictly wrong, but still a bad idea); - replaced header 'X-Razor-Warning:' with SA-compatible 'X-Spam-Status:' ; the 'X-Razor-id:' is still provided to facilitate user spam reporting; - added '--' between options and argument when calling $sendmail_wrapper to be triply sure we avoid problems with some mailers (note that $sendmail_wrapper is only still needed in the sendmail setup, Postfix and Exim do not need it in the recommended configuration); - tidy the %local_delivery_aliases mechanism for local delivery / quarantine; - allow MIME::Parser to decode uuencoded parts, if it feels it can (and should) do it; - write warnings from MIME::Parser to the log (at log level 1 or higher), instead of discarding them. The package is available at the usual location at: http://www.ijs.si/software/amavisd/ where also the most up-to-date version of FAQ and certain other fresh documentation files live. --------------------------------------------------------------------------- April 24, 2002 amavisd-new-20020424 is primarily a maintenance release to summarize one week's worth of experience with amavisd-new-20020418 and to implement some good ideas from the amavis-user mailing list. It also brings one or two new features. It is available at: http://www.ijs.si/software/amavisd/ or more specifically, at: http://www.ijs.si/software/amavisd/amavisd-new-20020424.tar.gz Changes since amavisd-new-20020418: - removed Perl 'my' declaration from configurable variables which prevented them from being changed in amavisd.conf (thanks to Sebastian Hagedorn and Wouter de Jong for reporting) - introduced child timeouts to prevent bad amavis client from monopolizing a child forever (thanks to Sebastian Hagedorn for reporting the problem) - supported and documented (in README.postfix) configuration where multiple remote or local SMTP-in/SMTP-out MTAs (e.g. Postfix) can use the same amavisd server, by making it deliver checked mail back to the same IP address it came from (see variable $relayhost_is_client in amavisd.conf) (thanks to Wouter de Jong for the splendid idea which fitted naturally into the overall scheme) - in certain log messages include the SMTP-in and SMTP-out MTA IP address - new access list checking for IP addresses - used to limit SMTP access to authorized MTAs only: @inet_acl; access control is now enabled by default - slightly more sophisticated hash-based access list lookups, modeled after Postfix map lookups. The sequence now goes through the following steps: - hash lookup for user+foo@do.ma.in - hash lookup for user@do.ma.in (only if $recipient_delimiter is nonempty) - hash lookup for do.ma.in - hash lookup for .do.ma.in - hash lookup for .ma.in - hash lookup for .in - hash lookup user+foo@ - hash lookup user@ (only if $recipient_delimiter is nonempty) - updated README.lookups - cleaner quarantine code; new variable $mailfrom_quarantine allows to choose either the original envelope sender, or admin-specified fixed sender address; include a special per-user quarantine example (look for 'trouble-user-quarantine' in the amavisd) - fixed problem with localized system error messages - use numeric errno instead of strings in Sophie and Trophie clients (thanks to Igor D'Astolfo for reporting the problem with Italianized version of Linux and for a good suggestion) - successfully tested with Exim (thanks to Igor D'Astolfo for testing and for reporting the <<doubly-quoted>> sender (but not recipient!) problem) - wrong variable used in the spam section ($final_virus_destiny instead of $final_spam_destiny) (thanks to Wayne Smith for reporting and for testing Vipul's Razor 'plugin' patch) - changed pattern match in the ./av/oav to support new version of OpenAntiVirus ScannerDaemon (thanks to Rainer Link) - changed 'configure' to make it recognize Sophie 1.33rc1 (thanks to Igor D'Astolfo and Lars Hecking) - include acinclude.m4/acx_pthread.m4 macro (thanks to Rainer Link) - updated README.postfix: includes instructions on how to avoid body_checks and header_checks for reinserted mail (contributed by Wayne Smith, works nicely, requires Postfix version 1.1.7-20020331 or later) - clarified comments in amavisd.conf - optionally keep existing X-Virus-Scanned: header lines, or remove them before adding our own header line - see $remove_existing_x_scanned_headers (requested by Darryl Harvey) - avoid historical (misleading) parameter name $localhost_ip; use $relayhost and $relayhost_port instead, but take old variables into account for backward compatibility with existing amavisd.conf files - Here is an overall picture (sequence of events) of how pieces fit together: bypass_checks? ==> PASS no viruses? ==> PASS log virus if $log_templ is nonempty quarantine if $virus_quarantine_to is nonempty notify admin if $warnadmin notify sender if $warnsender notify recips if $warnrecip final_destiny==pass? ==> PASS virus_lovers? ==> PASS DISCARD or REJECT (depending on final_*_destiny) --------------------------------------------------------------------------- April 18, 2002 This it to announce the second release of amavisd-new-20020418, available at: http://www.ijs.si/software/amavisd-new-20020418.tar.gz It is a version of amavisd (a daemonized AMaViS, which is an interface between MTA and virus scanners), based on amavisd CVS from today (20020418) (same configuration, amavis clients in C, instructions, AV client code), while also being a successor of the initial release of amavisd-new-20020329, which is a performance-enhanced pre-forked Net::Server -based amavisd with SMTP-in/SMTP-out capability, written in Perl. (I tried to make this release based on amavisd-snapshot-20020300, but failed, as that version still has $errval semantics bug (the bitwise-operations problem was discussed on the AMaViS-user list some time ago). This was fixed in the CVS version and most AV clients were changed then. As the CVS version contains AV client code for new AV scanners, this makes it incompatible with config stuff from amavisd-snapshot-20020300. If you have problems with ./configure or make, these problems would be common to both versions. Also the unpackers and decompressors code is mostly the same for all recent amavisd versions, so any problems in this arena (like DoS mishandling) are most likely to be common for both the official amavisd and the amavisd-new version. Compared to the first release of amavisd-new-20020329, the second release brings further significant performance improvements especially in the SMTP-in/SMTP-out configuration, e.g. with Postfix, but also brings some interesting new features and new configuration possibilities. No important bug fixes were needed, so upgrade at your leisure if you are running my initial version. The Postfix users would perhaps want to rush a bit though, to put new performance improvements into use. The summary of changes since the initial version: - significant SMTP-in speedups (25% with fast AV scanner), file reuse - pass reject reason to MTA on the input side - more informative MTA log entries in the SMTP-in/SMTP-out setup - amavis internal id (am_id) in log entries and passed to MTA in SMTP response - ISP features: specify subgroups of users who want to receive viruses - address extensions: e.g. user@domain -> user+virus@domain if virus detected - can specify final_virus_destiny: reject, discard, pass - quarantine new options: save to individual file, save to mailbox, pass to MTA - new headers in quarantined viruses preserve envelope addresses and quarantine id (similar to the suggestion from Furio Ercolessi - see code) - detailed timing breakdown report for each passed message - anti-spam hooks and examples, example patch to integrate Vipul's Razor client - body cache now always enabled - heavy speedup for mailing list bursts - Sophie 1.33-ready - rewritten Trophie client, based on new Sophie client code - rewritten README.postfix, describing new setup possibilities - new file README.lookups (to be used with virus_lovers and bypass_checks) - new file README.customize (same as in the initial version) - new file README.performance (unfinished) - new amavisd.conf options, documented there - code heavily commented, cleaned, generalized again - does not accuse innocent users of sending viruses if we are suspicious of sender address (see FAQ below) (same as in the initial version) CAVEATS: - no test mode; - only Postfix and sendmail milter are fully supported and tested; to integrate with other mailers one would need some understanding of their operation to set up properly; contributions are welcome; (P.S. note: this is probably a non-issue, related to the file system problem: All amavisd versions seem to share one still unresolved problem, probably with amavis-milter.c client, its use of libmilter, or perhaps even in the libmilter code itself - see recent thread on the AMaViS-user list (subject: Leftover email.txt files with amavisd 'standard' also). The problem is most pronounced in burst of heavy traffic. ) FAQ: - Net::Server 0.82 triggers a Perl 5.005 bug (the problem is obvious: you get syntax errors). Either upgrade to Net::Server 0.83, or upgrade your Perl - 5.6.x should be ok. - if you intend to play with customized notifications, it is wise to remove the '-t' option from $sendmail_wrapper_args in amavisd.conf. That way you are free to screw up notification mail headers any way you want, and the message would still be delivered to the correct recipients. Removing '-t' is now the recommended setting, but both variants should work. If you use SMTP-out method for notifications (e.g. Postfix recommended and default setting), this does not affect you. - if you see virus notifications claiming the virus originator is <?> or <?@some.domain> and sender notifications are not sent, this is not a bug, but a feature - see comments at the subroutine unmangle_sender(). The original idea comes from Furio Ercolessi: as some viruses tend to use forged or corrupted sender or 'From:' addresses, we try to determine the true virus sender, and if we can not do that, we avoid accusing innocent users of sending viruses. - if you kill or HUP amavisd, temporary directories may be left undeleted; this is normal and mails are not lost; - if amavisd does not restart after receiving HUP, a possible reason may be that amavisd can not be found in the path as set in the $ENV{PATH} variable (near the beginning of amavisd program). Another reason may be a syntax error if you changed the amavisd.conf file. Try to start it manually: $ su vscan -c amavisd If that does not make you wiser, set $DEBUG = "yes" and retry. This is also the recommended first-time start method. - after changing $inet_socket_bind in amavisd.conf, you must stop amavisd and start it anew. The HUP method causes amavisd to stumble over its feet. Below is my announcement notice for the initial version of amavisd-new, just slightly edited to remove some mistakes. --------------------------------------------------------------------------- Date: Sat, 30 Mar 2002 04:13:25 +0100 From: Mark Martinec <Mark.Martinec@ijs.si> Subject: [AMaViS-user] ANNOUNCE: new amavisd - leaner and meaner To: amavis-user@lists.sourceforge.net Message-id: <01KFYI1DJ2O200AMKT@CATHY.IJS.SI> I would like to announce an updated version of amavisd, based on recent CVS code of amavisd (which is not far from the February amavisd snapshot). It is a result of my three weeks work on the code, caused by our needs for: - better reliability; - higher throughput (less overhead); - versatility (e.g. separating amavis and MTA hosts, load sharing), Available at: http://www.ijs.si/software/amavisd-new-20020329.tar.gz This is now finally a version which I can recommend to friends :) TODO: better DoS handling in unpackers. Main features - in brief: - pre-forked reusable children - saving on process creations; - persistent connections to certain AV scanners, e.g. Sophie, saving on forks; - both SMTP and pipe (sendmail wrapper) interfaces independently available and configurable on all three sides: input, output, notifications; - sendmail Milter interface supported and tested; Postfix supported and thoroughly tested (Exim untested; classical sendmail untested, no qmail); - customizable notification messages; - compatibility with existing configurations; - cleanups, generalizations, speedups, fixes, better code documentation; - HUP signal causes restart with new configuration; - ISP feature: certain recipients may be allowed to receive viruses (with alert header line added (not with milter), notifications are still generated); - anti-spam hooks, caching-ready, Vipul's Razor interface in a form of a patch included - should make integration with SpamAssassin easier. Install: Unpack the tar over the checked-out CVS version of amavisd branch (or February snapshot). It overlays some files, the rest is unchanged. For milter interface make sure you use amavis-milter.c from CVS, as the snapshot version contains a bug which can cause message loss if amavisd dies. Details: - child and socket handling is now delegated to Perl module Net::Server, which gives us pre-forked children which are able to do more than one mail-check during their lifetime, saving on process creations and giving better response time; - Net::Server controls number of children, does signal handling, takes care of dead children, handles listening on multiple sockets (both Unix and TCP/UDP), delegating tasks and synchronizing 'accept's; - as a consequence, certain AV-scanner interfaces (most notably Sophie, and soon to follow Trophie) can keep persistent connections to the AV checking daemon, saving on AV scanner process creations (forks on accept), and socket setups/teardowns; - on the input side: both SMTP (TCP) and traditional amavis client protocol (Unix socket) are now accepted - even both at the same time (by default) to ease transition to SMTP interface; - new input-side SMTP interface (SMTP server) is easier to set up (no need to worry about file ownerships, UIDs and GIDs), and more versatile (e.g. SMTP responses carry more information that sysexits.h-based status codes, SMTP can talk to remote host); It is based on Perl native I/O and it is quite fast - no OO overhead, _not_ SMTP::Server-based (which I do not consider production quality); - a nice by-product of SMTP input interface is the extra information available in MTA logs, e.g. Postfix log: postfix/smtp[7656]: DA7B147FA6: to=<user@domain>, relay=localhost[127.0.0.1], delay=5, status=sent (250 Ok, discarded - VIRUS: EICAR-AV-Test) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - notifications messages can be delivered either via SMTP or as traditionally via pipe to sendmail wrapper / MTA pickup. This means that in the Postfix setup for example, one needs to worry only about one output interface (SMTP or pipe). This also saves unnecessary work of checking the just-generated notification messages for viruses; - the output pipe interface is now more careful with status checking and forking; - output SMTP interface error handling more compact; - sender notifications can be sent to more than one contact address (not used at the moment, but the mechanism is available); - sender address unmangling patch incorporated (but I'm willing to take it out if considered inappropriate for the base distribution); - see file ./amavis/mta/postfix_init for comments describing how to select SMTP- or pipe-based output interface (for re-injection and notifications); - customizable notification messages and log entry text (see README.customize); - clean notion of when mail addresses are in their quoted and when in unquoted form (RFC2821). All internal handling uses unquoted form, addresses get quoted as required by the output interface, and quoting gets stripped away as required by each input interface; - besides traditional choices where a mail can be forwarded/accepted, discarded, or temporary failure/retry indicated, there is now a fourth choice: REJECT. Depending on the MTA this requires a message rejection to be done by the input side MTA itself. This comes handy in cases when amavisd accepts a non-infected mail, but outgoing MTA does not want to take it back for final delivery, e.g. in case of some policy violation. Traditionally amavisd would indicate temporary failure on its input side, causing the message to be retried and re-scanned over and over again, without having a chance of ever being accepted; - all existing virus scanners are still there (one little change in each: replaced a call to do_virus with return 1); Sophie client rewritten to take advantage of persistent connections; - decoding sections are mostly unchanged. This area needs more work in the future; - includes exit status codes from sysexits.ph instead or having them hard-wired; - anti-spam code easier to integrate into amavisd due to some code rearrangements; a patch to integrate Vipul's Razor is included as an example, but it may be better to tie amavisd with SpamAssassin. Anti-spam code will NOT be integrated into amavisd, but the least we can do is making it easier for people to add their own code; - cache-ready (example in the included Razor patch). This works by calculating a message digest (hash, signature) of the message body and keeping it in storage for a short while, e.g. for the lifetime of a child process (10 consecutive requests by default). If another message with the same body content arrives in the near future we can skip a virus check. This comes handy where mailing list traffic is frequent, especially if we have to deal with poorly done mailing list managers or heavy spam traffic; To support this concept the functions of MIME decoding and unpacking of archives are now separated. - MIME parsing is now supplied with our own Filer subclass. This was necessary to avoid MIME parser complaints when it tried to reconstruct file name extensions from file names in strange character sets. Its work was completely unnecessary and harmful, since we want to supply our own file names and do not care for file name extensions. - as a consequence of our own Filer subclass, we now avoid the first (or the only) directory traversal (reading) in the first decoding pass; Not much, but every little bit counts. - when using sendmail wrapper it is no longer necessary to supply the sendmail -t option. If -t is not specified, sendmail wrapper receives addresses via command arguments (exec, no shell), which is more reliable than having to parse mail headers - which are now more error-prone due to user-customizable notifications. The -t option is still supported though, but not recommended. - do_unzip no longer complains with multi-line backtrace when it dies; just a single message is issued, like with other decoders; - HUP signal causes restart and re-reading of config file; - many new comments, code unifications, supplied some missing error checks, code generalizations; MTA support: - thoroughly tested with Postfix in all combinations of input, output, and notifications interfaces; in production use; - big thanks to Sebastian Hagedorn who helped to test the milter interface on his Solaris! - Exim and traditional sendmail interface untested, but should work without much work. - Qmail interface is still missing, like in current CVS amavisd. I believe the amavis client for qmail can be written as a few-dozen line Perl program. Anybody want to try? A quick cookbook on how to set up Postfix / amavisd interface using SMTP on both amavisd input and output side (including notifications). Amavisd by default now accepts both Unix socket and SMTP on the loopback interface, so the transition is easier: first install new amavisd, then at some other time change Postfix configuration (if desired) to: master.cf: # MTA -> amavisd smtp-amavisd unix - - n - 2 smtp # amavisd -> MTA localhost:10025 inet n - n - - smtpd -o content_filter= main.cf: # choose transport to amavisd content_filter = smtp-amavisd:localhost:10024 [see new file README.postfix for details] The amavisd can now easily be located on a different host than MTA, also Postfix load-balancing transport methods can be used (e.g. multiple MX records). Although amavisd now talks SMTP and incorporates some rudimentary defenses against malicious SMTP clients, do not expose its SMTP server directly to the world - always front-end it with MTA. By default it binds to the local interface only, but other access restrictions are also available. Happy amavising! Experiences and comments are most welcome.