EVOLUTION-MANAGER

Edit File: README.sql-mysql

USING SQL FOR LOOKUPS, LOG/REPORTING AND QUARANTINE =================================================== This text only describes SQL specifics for a MySQL database and provides a schema. For general aspects of lookups, please see README.lookups. For general SQL notes and further examples please see README.sql. For PostgreSQL-specific notes and schema please see README.sql-pg (which in most respects also applies to a SQLite database). SERIAL can be used instead of INT UNSIGNED NOT NULL AUTO_INCREMENT with databases which do not recognize AUTO_INCREMENT; The attribute SERIAL was introduced with MySQL 4.1.0, but it implicitly creates an additional UNIQUE index, which is redundant. Instead of declaring a time_iso field in table msgs as a string: time_iso char(16) NOT NULL, one may want to declare is as: time_iso TIMESTAMP NOT NULL DEFAULT 0, in which case $timestamp_fmt_mysql *MUST* be set to 1 in amavisd.conf to avoid MySQL inability to accept ISO 8601 timestamps with zone Z and ISO date/time delimiter T; failing to set $timestamp_fmt_mysql makes MySQL store zero time on INSERT and write current local time on UPDATE if auto-update is allowed, which is different from the intended mail timestamp (localtime vs. UTC, off by seconds) Field quarantine.mail_text should be of data type 'blob' and not 'text' as suggested in earlier documentation; this is to prevent it from being unjustifiably associated with a character set, and to be able to store any byte value; to convert existing field from type 'text' to type 'blob' the following clause may be used: ALTER TABLE quarantine CHANGE mail_text mail_text blob; Although MySQL is not particularly picky in checking and enforcing data types, it is appropriate to declare fields users.email, mailaddr.email, and maddr.email as byte strings with no associated character set (which is what these fields are, according to RFC 2821) instead of CHAR or VARCHAR. The following clauses convert pre-2.6.0 tables into the now recommended form: ALTER TABLE users CHANGE email email varbinary(255); ALTER TABLE mailaddr CHANGE email email varbinary(255); ALTER TABLE maddr CHANGE email email varbinary(255); If VARBINARY data type is chosen for these three fields, the setting $sql_allow_8bit_address should be set to true to let the amavisd program use the appropriate data type in SQL commands: $sql_allow_8bit_address = 1; # maddr.email: VARCHAR (0), VARBINARY/BYTEA (1) Similarly, semantics of some other fields is strings of octets too, with no associated character set (which is a characteristic of char and varchar data types), so the following alternations to pre-2.6.2 schema may be beneficial to prevent SQL server from checking validity of octets data against non-applicable arbitrary character set constraints: ALTER table msgs CHANGE mail_id mail_id varbinary(16); ALTER table msgs CHANGE secret_id secret_id varbinary(16); ALTER table msgs CHANGE quar_loc quar_loc varbinary(255); ALTER table msgrcpt CHANGE mail_id mail_id varbinary(16); ALTER table quarantine CHANGE mail_id mail_id varbinary(16); and for good measure: ALTER table msgrcpt CHANGE rid rid bigint unsigned; ALTER table msgs CHANGE sid sid bigint unsigned; Starting with amavisd-new-2.7.0, three fields need to be added to table msgrcpt and one to table msgs: ALTER TABLE msgrcpt ADD rseqnum integer DEFAULT 0 NOT NULL; ALTER TABLE msgrcpt ADD content char(1) DEFAULT ' ' NOT NULL; ALTER TABLE msgrcpt ADD is_local char(1) DEFAULT ' ' NOT NULL; ALTER TABLE msgs ADD originating char(1) DEFAULT ' ' NOT NULL; If you need to create a primary key on table msgrcpt for some reason (clustering?), try something like: UPDATE msgrcpt SET rseqnum=1+floor(999999999*rand()) WHERE rseqnum=0; ALTER TABLE msgrcpt ADD PRIMARY KEY (partition_tag,mail_id,rseqnum); -- local users CREATE TABLE users ( id int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY, -- unique id priority integer NOT NULL DEFAULT '7', -- sort field, 0 is low prior. policy_id integer unsigned NOT NULL DEFAULT '1', -- JOINs with policy.id email varbinary(255) NOT NULL UNIQUE, fullname varchar(255) DEFAULT NULL -- not used by amavisd-new -- local char(1) -- Y/N (optional field, see note further down) ); -- any e-mail address (non- rfc2822-quoted), external or local, -- used as senders in wblist CREATE TABLE mailaddr ( id int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY, priority integer NOT NULL DEFAULT '7', -- 0 is low priority email varbinary(255) NOT NULL UNIQUE ); -- per-recipient whitelist and/or blacklist, -- puts sender and recipient in relation wb (white or blacklisted sender) CREATE TABLE wblist ( rid integer unsigned NOT NULL, -- recipient: users.id sid integer unsigned NOT NULL, -- sender: mailaddr.id wb varchar(10) NOT NULL, -- W or Y / B or N / space=neutral / score PRIMARY KEY (rid,sid) ); CREATE TABLE policy ( id int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY, -- 'id' this is the _only_ required field policy_name varchar(32), -- not used by amavisd-new, a comment virus_lover char(1) default NULL, -- Y/N spam_lover char(1) default NULL, -- Y/N unchecked_lover char(1) default NULL, -- Y/N banned_files_lover char(1) default NULL, -- Y/N bad_header_lover char(1) default NULL, -- Y/N bypass_virus_checks char(1) default NULL, -- Y/N bypass_spam_checks char(1) default NULL, -- Y/N bypass_banned_checks char(1) default NULL, -- Y/N bypass_header_checks char(1) default NULL, -- Y/N virus_quarantine_to varchar(64) default NULL, spam_quarantine_to varchar(64) default NULL, banned_quarantine_to varchar(64) default NULL, unchecked_quarantine_to varchar(64) default NULL, bad_header_quarantine_to varchar(64) default NULL, clean_quarantine_to varchar(64) default NULL, archive_quarantine_to varchar(64) default NULL, spam_tag_level float default NULL, -- higher score inserts spam info headers spam_tag2_level float default NULL, -- inserts 'declared spam' header fields spam_tag3_level float default NULL, -- inserts 'blatant spam' header fields spam_kill_level float default NULL, -- higher score triggers evasive actions -- e.g. reject/drop, quarantine, ... -- (subject to final_spam_destiny setting) spam_dsn_cutoff_level float default NULL, spam_quarantine_cutoff_level float default NULL, addr_extension_virus varchar(64) default NULL, addr_extension_spam varchar(64) default NULL, addr_extension_banned varchar(64) default NULL, addr_extension_bad_header varchar(64) default NULL, warnvirusrecip char(1) default NULL, -- Y/N warnbannedrecip char(1) default NULL, -- Y/N warnbadhrecip char(1) default NULL, -- Y/N newvirus_admin varchar(64) default NULL, virus_admin varchar(64) default NULL, banned_admin varchar(64) default NULL, bad_header_admin varchar(64) default NULL, spam_admin varchar(64) default NULL, spam_subject_tag varchar(64) default NULL, spam_subject_tag2 varchar(64) default NULL, spam_subject_tag3 varchar(64) default NULL, message_size_limit integer default NULL, -- max size in bytes, 0 disable banned_rulenames varchar(64) default NULL, -- comma-separated list of ... -- names mapped through %banned_rules to actual banned_filename tables disclaimer_options varchar(64) default NULL, forward_method varchar(64) default NULL, sa_userconf varchar(64) default NULL, sa_username varchar(64) default NULL ); -- R/W part of the dataset (optional) -- May reside in the same or in a separate database as lookups database; -- REQUIRES SUPPORT FOR TRANSACTIONS; specified in @storage_sql_dsn -- -- MySQL note ( http://dev.mysql.com/doc/mysql/en/storage-engines.html ): -- ENGINE is the preferred term, but cannot be used before MySQL 4.0.18. -- TYPE is available beginning with MySQL 3.23.0, the first version of -- MySQL for which multiple storage engines were available. If you omit -- the ENGINE or TYPE option, the default storage engine is used. -- By default this is MyISAM. -- -- Please create additional indexes on keys when needed, or drop suggested -- ones as appropriate to optimize queries needed by a management application. -- See your database documentation for further optimization hints. With MySQL -- see Chapter 15 of the reference manual. For example the chapter 15.17 says: -- InnoDB does not keep an internal count of rows in a table. To process a -- SELECT COUNT(*) FROM T statement, InnoDB must scan an index of the table, -- which takes some time if the index is not entirely in the buffer pool. -- -- Wayne Smith adds: When using MySQL with InnoDB one might want to -- increase buffer size for both pool and log, and might also want -- to change flush settings for a little better performance. Example: -- innodb_buffer_pool_size = 384M -- innodb_log_buffer_size = 8M -- innodb_flush_log_at_trx_commit = 0 -- The big performance increase is the first two, the third just helps with -- lowering disk activity. Consider also adjusting the key_buffer_size. -- provide unique id for each e-mail address, avoids storing copies CREATE TABLE maddr ( partition_tag integer DEFAULT 0, -- see $partition_tag id bigint unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY, email varbinary(255) NOT NULL, -- full mail address domain varchar(255) NOT NULL, -- only domain part of the email address -- with subdomain fields in reverse CONSTRAINT part_email UNIQUE (partition_tag,email) ) ENGINE=InnoDB; -- information pertaining to each processed message as a whole; -- NOTE: records with NULL msgs.content should be ignored by utilities, -- as such records correspond to messages just being processes, or were lost -- NOTE: instead of a character field time_iso, one might prefer: -- time_iso TIMESTAMP NOT NULL DEFAULT 0, -- but the following MUST then be set in amavisd.conf: $timestamp_fmt_mysql=1 CREATE TABLE msgs ( partition_tag integer DEFAULT 0, -- see $partition_tag mail_id varbinary(16) NOT NULL, -- long-term unique mail id, dflt 12 ch secret_id varbinary(16) DEFAULT '', -- authorizes release of mail_id, 12 ch am_id varchar(20) NOT NULL, -- id used in the log time_num integer unsigned NOT NULL, -- rx_time: seconds since Unix epoch time_iso char(16) NOT NULL, -- rx_time: ISO8601 UTC ascii time sid bigint unsigned NOT NULL, -- sender: maddr.id policy varchar(255) DEFAULT '', -- policy bank path (like macro %p) client_addr varchar(255) DEFAULT '', -- SMTP client IP address (IPv4 or v6) size integer unsigned NOT NULL, -- message size in bytes originating char(1) DEFAULT ' ' NOT NULL, -- sender from inside or auth'd content char(1), -- content type: V/B/U/S/Y/M/H/O/T/C -- virus/banned/unchecked/spam(kill)/spammy(tag2)/ -- /bad-mime/bad-header/oversized/mta-err/clean -- is NULL on partially processed mail -- (prior to 2.7.0 the CC_SPAMMY was logged as 's', now 'Y' is used; -- to avoid a need for case-insenstivity in queries) quar_type char(1), -- quarantined as: ' '/F/Z/B/Q/M/L -- none/file/zipfile/bsmtp/sql/ -- /mailbox(smtp)/mailbox(lmtp) quar_loc varbinary(255) DEFAULT '', -- quarantine location (e.g. file) dsn_sent char(1), -- was DSN sent? Y/N/q (q=quenched) spam_level float, -- SA spam level (no boosts) message_id varchar(255) DEFAULT '', -- mail Message-ID header field from_addr varchar(255) CHARACTER SET utf8mb4 COLLATE utf8_bin DEFAULT '', -- mail From header field, UTF8 subject varchar(255) CHARACTER SET utf8mb4 COLLATE utf8_bin DEFAULT '', -- mail Subject header field, UTF8 host varchar(255) NOT NULL, -- hostname where amavisd is running PRIMARY KEY (partition_tag,mail_id) -- FOREIGN KEY (sid) REFERENCES maddr(id) ON DELETE RESTRICT ) ENGINE=InnoDB; CREATE INDEX msgs_idx_sid ON msgs (sid); CREATE INDEX msgs_idx_mess_id ON msgs (message_id); -- useful with pen pals CREATE INDEX msgs_idx_time_num ON msgs (time_num); -- alternatively when purging based on time_iso (instead of msgs_idx_time_num): -- CREATE INDEX msgs_idx_time_iso ON msgs (time_iso); -- When using FOREIGN KEY contraints, InnoDB requires index on a field -- (an the field must be the first field in the index). Hence create it: -- CREATE INDEX msgs_idx_mail_id ON msgs (mail_id); -- per-recipient information related to each processed message; -- NOTE: records in msgrcpt without corresponding msgs.mail_id record are -- orphaned and should be ignored and eventually deleted by external utilities CREATE TABLE msgrcpt ( partition_tag integer DEFAULT 0, -- see $partition_tag mail_id varbinary(16) NOT NULL, -- (must allow duplicates) rseqnum integer DEFAULT 0 NOT NULL, -- recip's enumeration within msg rid bigint unsigned NOT NULL, -- recipient: maddr.id (dupl. allowed) is_local char(1) DEFAULT ' ' NOT NULL, -- recip is: Y=local, N=foreign content char(1) DEFAULT ' ' NOT NULL, -- content type V/B/U/S/Y/M/H/O/T/C ds char(1) NOT NULL, -- delivery status: P/R/B/D/T -- pass/reject/bounce/discard/tempfail rs char(1) NOT NULL, -- release status: initialized to ' ' bl char(1) DEFAULT ' ', -- sender blacklisted by this recip wl char(1) DEFAULT ' ', -- sender whitelisted by this recip bspam_level float, -- per-recipient (total) spam level smtp_resp varchar(255) DEFAULT '', -- SMTP response given to MTA PRIMARY KEY (partition_tag,mail_id,rseqnum) -- FOREIGN KEY (rid) REFERENCES maddr(id) ON DELETE RESTRICT, -- FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE ) ENGINE=InnoDB; CREATE INDEX msgrcpt_idx_mail_id ON msgrcpt (mail_id); CREATE INDEX msgrcpt_idx_rid ON msgrcpt (rid); -- mail quarantine in SQL, enabled by $*_quarantine_method='sql:' -- NOTE: records in quarantine without corresponding msgs.mail_id record are -- orphaned and should be ignored and eventually deleted by external utilities CREATE TABLE quarantine ( partition_tag integer DEFAULT 0, -- see $partition_tag mail_id varbinary(16) NOT NULL, -- long-term unique mail id chunk_ind integer unsigned NOT NULL, -- chunk number, starting with 1 mail_text blob NOT NULL, -- store mail as chunks of octets PRIMARY KEY (partition_tag,mail_id,chunk_ind) -- FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE ) ENGINE=InnoDB; -- field msgrcpt.rs is primarily intended for use by quarantine management -- software; the value assigned by amavisd is a space; -- a short _preliminary_ list of possible values: -- 'V' => viewed (marked as read) -- 'R' => released (delivered) to this recipient -- 'p' => pending (a status given to messages when the admin received the -- request but not yet released; targeted to banned parts) -- 'D' => marked for deletion; a cleanup script may delete it -- ===================== -- Example data follows: -- ===================== INSERT INTO users VALUES ( 1, 9, 5, 'user1+foo@y.example.com','Name1 Surname1', 'Y'); INSERT INTO users VALUES ( 2, 7, 5, 'user1@y.example.com', 'Name1 Surname1', 'Y'); INSERT INTO users VALUES ( 3, 7, 2, 'user2@y.example.com', 'Name2 Surname2', 'Y'); INSERT INTO users VALUES ( 4, 7, 7, 'user3@z.example.com', 'Name3 Surname3', 'Y'); INSERT INTO users VALUES ( 5, 7, 7, 'user4@example.com', 'Name4 Surname4', 'Y'); INSERT INTO users VALUES ( 6, 7, 1, 'user5@example.com', 'Name5 Surname5', 'Y'); INSERT INTO users VALUES ( 7, 5, 0, '@sub1.example.com', NULL, 'Y'); INSERT INTO users VALUES ( 8, 5, 7, '@sub2.example.com', NULL, 'Y'); INSERT INTO users VALUES ( 9, 5, 5, '@example.com', NULL, 'Y'); INSERT INTO users VALUES (10, 3, 8, 'userA', 'NameA SurnameA anywhere', 'Y'); INSERT INTO users VALUES (11, 3, 9, 'userB', 'NameB SurnameB', 'Y'); INSERT INTO users VALUES (12, 3,10, 'userC', 'NameC SurnameC', 'Y'); INSERT INTO users VALUES (13, 3,11, 'userD', 'NameD SurnameD', 'Y'); INSERT INTO users VALUES (14, 3, 0, '@sub1.example.net', NULL, 'Y'); INSERT INTO users VALUES (15, 3, 7, '@sub2.example.net', NULL, 'Y'); INSERT INTO users VALUES (16, 3, 5, '@example.net', NULL, 'Y'); INSERT INTO users VALUES (17, 7, 5, 'u1@example.org', 'u1', 'Y'); INSERT INTO users VALUES (18, 7, 6, 'u2@example.org', 'u2', 'Y'); INSERT INTO users VALUES (19, 7, 3, 'u3@example.org', 'u3', 'Y'); -- INSERT INTO users VALUES (20, 0, 5, '@.', NULL, 'N'); -- catchall INSERT INTO policy (id, policy_name, virus_lover, spam_lover, banned_files_lover, bad_header_lover, bypass_virus_checks, bypass_spam_checks, bypass_banned_checks, bypass_header_checks, spam_modifies_subj, spam_tag_level, spam_tag2_level, spam_kill_level) VALUES (1, 'Non-paying', 'N','N','N','N', 'Y','Y','Y','N', 'Y', 3.0, 7, 10), (2, 'Uncensored', 'Y','Y','Y','Y', 'N','N','N','N', 'N', 3.0, 999, 999), (3, 'Wants all spam','N','Y','N','N', 'N','N','N','N', 'Y', 3.0, 999, 999), (4, 'Wants viruses', 'Y','N','Y','Y', 'N','N','N','N', 'Y', 3.0, 6.9, 6.9), (5, 'Normal', 'N','N','N','N', 'N','N','N','N', 'Y', 3.0, 6.9, 6.9), (6, 'Trigger happy', 'N','N','N','N', 'N','N','N','N', 'Y', 3.0, 5, 5), (7, 'Permissive', 'N','N','N','Y', 'N','N','N','N', 'Y', 3.0, 10, 20), (8, '6.5/7.8', 'N','N','N','N', 'N','N','N','N', 'N', 3.0, 6.5, 7.8), (9, 'userB', 'N','N','N','Y', 'N','N','N','N', 'Y', 3.0, 6.3, 6.3), (10,'userC', 'N','N','N','N', 'N','N','N','N', 'N', 3.0, 6.0, 6.0), (11,'userD', 'Y','N','Y','Y', 'N','N','N','N', 'N', 3.0, 7, 7); -- sender envelope addresses needed for white/blacklisting INSERT INTO mailaddr VALUES (1, 5, '@example.com'); INSERT INTO mailaddr VALUES (2, 9, 'owner-postfix-users@postfix.org'); INSERT INTO mailaddr VALUES (3, 9, 'amavis-user-admin@lists.sourceforge.net'); INSERT INTO mailaddr VALUES (4, 9, 'makemoney@example.com'); INSERT INTO mailaddr VALUES (5, 5, '@example.net'); INSERT INTO mailaddr VALUES (6, 9, 'spamassassin-talk-admin@lists.sourceforge.net'); INSERT INTO mailaddr VALUES (7, 9, 'spambayes-bounces@python.org'); -- whitelist for user 14, i.e. default for recipients in domain sub1.example.net INSERT INTO wblist VALUES (14, 1, 'W'); INSERT INTO wblist VALUES (14, 3, 'W'); -- whitelist and blacklist for user 17, i.e. u1@example.org INSERT INTO wblist VALUES (17, 2, 'W'); INSERT INTO wblist VALUES (17, 3, 'W'); INSERT INTO wblist VALUES (17, 6, 'W'); INSERT INTO wblist VALUES (17, 7, 'W'); INSERT INTO wblist VALUES (17, 5, 'B'); INSERT INTO wblist VALUES (17, 4, 'B'); -- $sql_select_policy setting in amavisd.conf tells amavisd -- how to fetch per-recipient policy settings. -- See comments there. Example: -- -- SELECT *,users.id FROM users,policy -- WHERE (users.policy_id=policy.id) AND (users.email IN (%k)) -- ORDER BY users.priority DESC; -- -- $sql_select_white_black_list in amavisd.conf tells amavisd -- how to check sender in per-recipient whitelist/blacklist. -- See comments there. Example: -- -- SELECT wb FROM wblist,mailaddr -- WHERE (wblist.rid=?) AND (wblist.sid=mailaddr.id) AND (mailaddr.email IN (%k)) -- ORDER BY mailaddr.priority DESC; -- NOTE: the SELECT, INSERT and UPDATE clauses as used by the amavisd-new -- program are configurable through %sql_clause; see amavisd.conf-default Some examples of a query: -- mail from last two minutes: SELECT UNIX_TIMESTAMP()-msgs.time_num AS age, SUBSTRING(policy,1,2) as pb, msgrcpt.content AS c, dsn_sent as dsn, ds, bspam_level AS level, size, SUBSTRING(sender.email,1,18) AS s, SUBSTRING(recip.email,1,18) AS r, SUBSTRING(msgs.subject,1,10) AS subj FROM msgs LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id LEFT JOIN maddr AS sender ON msgs.sid=sender.id LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id WHERE msgrcpt.content IS NOT NULL AND UNIX_TIMESTAMP()-msgs.time_num < 120 ORDER BY msgs.time_num DESC; -- clean messages ordered by count, grouped by domain: SELECT count(*) as cnt, avg(bspam_level), sender.domain FROM msgs LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id LEFT JOIN maddr AS sender ON msgs.sid=sender.id LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id WHERE msgrcpt.content='C' GROUP BY sender.domain ORDER BY cnt DESC LIMIT 50; -- top spamy domains with >10 messages, sorted by spam average, -- grouped by domain: SELECT count(*) as cnt, avg(bspam_level) as spam_avg, sender.domain FROM msgs LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id LEFT JOIN maddr AS sender ON msgs.sid=sender.id LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id WHERE bspam_level IS NOT NULL GROUP BY sender.domain HAVING count(*) > 10 ORDER BY spam_avg DESC LIMIT 50; -- sender domains with >100 messages, sorted on sender.domain: SELECT count(*) as cnt, avg(bspam_level) as spam_avg, sender.domain FROM msgs LEFT JOIN msgrcpt ON msgs.mail_id=msgrcpt.mail_id LEFT JOIN maddr AS sender ON msgs.sid=sender.id LEFT JOIN maddr AS recip ON msgrcpt.rid=recip.id GROUP BY sender.domain HAVING count(*) > 100 ORDER BY sender.domain DESC LIMIT 100; Upgrading from pre 2.4.0 amavisd-new SQL schema to the 2.4.0 schema requires adding column 'quar_loc' to table msgs, and creating FOREIGN KEY constraint to facilitate deletion of expired records. The following clauses should be executed for upgrading pre-2.4.0 amavisd-new SQL schema to the 2.4.0 schema. Creating a FOREIGN KEY ... ON DELETE CASCADE constraint may (or may not) facilitate deletion of expired records. -- mandatory change: ALTER TABLE msgs ADD quar_loc varchar(255) DEFAULT ''; -- optional, avoids need to purge tables msgrcpt and quarantine explicitly: ALTER TABLE quarantine ADD FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE; ALTER TABLE msgrcpt ADD FOREIGN KEY (mail_id) REFERENCES msgs(mail_id) ON DELETE CASCADE; -- the following two ALTERs are not essential; if data type of maddr.id is -- incompatible with msgs.sid and msgs.rid (e.g. BIGINT vs. INT) and MySQL -- complains, don't bother to apply the constraint: ALTER TABLE msgs ADD FOREIGN KEY (sid) REFERENCES maddr(id) ON DELETE RESTRICT; ALTER TABLE msgrcpt ADD FOREIGN KEY (rid) REFERENCES maddr(id) ON DELETE RESTRICT; EXAMPLE of a log/report/quarantine database housekeeping ======================================================== Using a changing partition_tag, perhaps by using an ISO 8601 week number (value 1 to 53) as a partition_tag: $partition_tag = sub { my($msginfo)=@_; sprintf("%02d",iso8601_week($msginfo->rx_time)) }; allows for probably the fastest method of purging old records, e.g.: DELETE FROM msgs WHERE partition_tag >= 13 AND partition_tag <= 23; DELETE FROM msgrcpt WHERE partition_tag >= 13 AND partition_tag <= 23; DELETE FROM quarantine WHERE partition_tag >= 13 AND partition_tag <= 23; DELETE FROM maddr WHERE partition_tag >= 13 AND partition_tag <= 23; Note: using native SQL table partitioning as offered by MySQL, the above may be even faster by dropping entire partitions. Not documented here. Alternatively, purge records from table msgs by their creation time: DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP() - 21*24*3600; DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP() - 3600 AND content IS NULL; Optionally certain content types may be given shorter lifetime: DELETE FROM msgs WHERE time_num < UNIX_TIMESTAMP() - 7*24*3600 AND (content='V' OR (content='S' AND spam_level > 20)); ( or equivalently, if a data type of msgs.time_iso is TIMESTAMPS and in amavisd.conf the $timestamp_fmt_mysql is set to true: DELETE FROM msgs WHERE time_iso < UTC_TIMESTAMP() - INTERVAL 21 day; DELETE FROM msgs WHERE time_iso < UTC_TIMESTAMP() - INTERVAL 1 hour AND content IS NULL; DELETE FROM msgs WHERE time_iso < UTC_TIMESTAMP() - INTERVAL 7 day AND (content='V' OR (content='S' AND spam_level > 20)); ) Then delete unreferenced records from tables msgrcpt and quarantine, unless they were already automatically deleted while purging the msgs table and FOREIGN KEY ... ON DELETE CASCADE is in place: DELETE FROM msgrcpt WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=msgrcpt.mail_id); DELETE FROM quarantine WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=quarantine.mail_id); ( or equivalently: DELETE msgrcpt FROM msgrcpt LEFT JOIN msgs USING(mail_id) WHERE msgs.mail_id IS NULL; DELETE quarantine FROM quarantine LEFT JOIN msgs USING(mail_id) WHERE msgs.mail_id IS NULL; ) Finally delete unreferenced records from table maddr: DELETE FROM maddr WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE sid=id) AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id); SOME FURTHER THOUGHTS on a log/report/quarantine database housekeeping ====================================================================== Discarding indexes makes deletion faster; if we expect a large proportion of records to be deleted it may be quicker to discard index, do deletions, and re-create index; for daily maintenance this does not pay off DROP INDEX msgs_idx_sid ON msgs; DROP INDEX msgrcpt_idx_rid ON msgrcpt; DROP INDEX msgrcpt_idx_mail_id ON msgrcpt; Re-create indexes (if they were removed in the first step): CREATE INDEX msgs_idx_sid ON msgs (sid); CREATE INDEX msgrcpt_idx_rid ON msgrcpt (rid); CREATE INDEX msgrcpt_idx_mail_id ON msgrcpt (mail_id); Optionally, optimize tables once in a while: OPTIMIZE TABLE msgs, msgrcpt, quarantine, maddr;