How to make Exim run a lot Faster
From Computer Tyme Support Wiki
(→Overview) |
|||
| Line 16: | Line 16: | ||
driver = dnslookup | driver = dnslookup | ||
transport = outscan_smtp | transport = outscan_smtp | ||
| - | |||
fallback_hosts = retry.junkemailfilter.com | fallback_hosts = retry.junkemailfilter.com | ||
| - | |||
no_more | no_more | ||
Revision as of 04:18, 17 May 2010
Contents |
Spool to RAM Disk
In some comparisons Exim is rated as running slower than other email systems. This is due in part because of how Exim writes its undelivered email to disk (which fulfils the RFC requirement to commit messages to stable storage before acknowledging receipt). However, Exim can run over 10x faster if you use a ran disk to store your undelivered email. Using a ram disk makes Exim many times faster with few drawbacks.
Introduction
I, Marc Perkel, run a spam filtering service Junk Email Filter which receives email, cleans out the spam, and forwards the good email to the recipient server. This is a high volume operation and lots of email has to be processed and filtered fast, with as few computers as possible. We use Exim because it's the best. I can't imagine running a spam filtering service without it. But speed is always an issue and here's what we did to get the speed.
The biggest performance bottleneck in Exim is the message queues. Keeping the message queues in ram make Exim run really fast. But you have to do a lot more than just create a ram disk since ram is limited and it loses what is stored if the server is shut down. So beside just creating a ram disk there are a number of other things you have to deal with to prevent email loss and overflowing your memory.
Overview
To get really fast performance you'll need to use several servers. The main Exim server will run Exim and very little else. Email comes in, processed very fast, and forwarded to a destination server. Exim has a feature where you can specify a "fallback_host". So if the delivery fails to the intended recipient it will try a different computer. So a second computer us set up that I call the retry computer. I user the follows routing example:
process_and_forward_outscan: driver = dnslookup transport = outscan_smtp fallback_hosts = retry.junkemailfilter.com no_more
With this configuration the mail Exim server makes one and only one try to deliver the email. If that try fails the message is transferred to the retry server. The retry server does NOT use a ram dis for the queues and it will retry delivery for several days until the message is either delivered or times out. This takes the load off of the main Exim server and reduces the amount of messages in the queue to a minimum keeping the main server fast while at the same time preserving the integrity of the delivery process by writing undelivered email to a real disk.
Preserving the RAM disk through a Reboot
Sometimes a server has to be shut down for servicing and you don't want to lose the email in the ram queues. Here's a simple script that I install as a service that backs up the queue to hard disk and restores it upon reboot. (exim-save)
#
# exim-save This shell script takes care of starting and stopping exim
#
# chkconfig: 2345 79 31
# description: Exim Save Data
# Source function library.
. /etc/init.d/functions
[ -f /usr/sbin/exim ] || exit 0
start() {
touch /var/lock/subsys/exim-save
if [ -d /var/spool/exim-backup ]
then
cp -a /var/spool/exim-backup/* /var/spool/exim/
rm -Rf /var/spool/exim-backup
fi
}
stop() {
rm -f /var/lock/subsys/exim-save
if ! [ -d /var/spool/exim-backup ]
then
rm -Rf /var/spool/exim/db
rm -Rf /var/spool/exim/scan
rm /var/spool/exim/*.pid
cp -a /var/spool/exim /var/spool/exim-backup
rm -Rf /var/spool/exim/input
fi
}
restart() {
stop
start
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo $"Usage: $0 {start|stop|restart}"
exit 1
esac
exit $RETVAL
I install it in the /etc/init.d directory as a service and activate it by running:
chkconfig exim-save on
Thus when you shut down the server it saves the queue and restores it upon reboot.
Drawbacks
Using this method has two significant drawbacks. First, if the server crashes or loses power or is shut down in some other unclean manner you will lose the email in the queue. So if you use this method you will have to accept that some email will be at risk.
The second issue is that if something happens due to a misconfiguration, the retry server goes offline, or some other unusual circumstance you haven't though of, email could accumulate in the queues and fill up memory creating a "disk full" situation. The first defense to this is to have a lot of ram. Memory is cheap these days.
If your ram drive fills Exim will respond to incoming email with a 4xx error which should tell incoming servers to try a higher numbered MX record. (Qmail servers generally get this wrong but every other email server is OK.) So in addition to a retry server you want a backup MX server. That way if the main server is down or full the backup server (not using a RAM disk) will pick up the load.
Other Servers
For a high performance system you want to split up the load and use other servers for other operations. Here we outlined using mail Exim server for the bulk processing of email at high speed and using a retry server to assist in the delivery if the first attempt fails. We also specified a backup MX server to get your email if there is a problem with the main server. (BTW, there are free backup MX services out there if you want to save on servers.) But there are other things you can do,
If you are running Spam Assassin you would want to run that on a separate dedicated server or in some cases several separate dedicated servers. I also have a dedicated DNS server just to do lookups and cache the results. It is also where I run my blacklists. I would also recommend having a separate server where incoming email is stored, after spam filtering, that runs your IMAP, POP, and Webmail services.
