Minimal Mail Setup

This article will be applicable to those who are more or less familiar with Linux and/or other Unix-like environments. Here, by saying familiar, I mean that, user has enough experience and knowledge to operate with process, file systems, connections, private keys etc. It is definitely not for an everyday average user, thus I intentionally will stay away from saying "... here is my config use it like ...". Instead, I will be explaining why some piece in the picture are needed and providing snippets on here, how it can be achieved. One, who decides to implement similar setup, should be capable to collect pieces of the puzzle into whole. If you are not one of such people, just stay away from this for your own safety, at least you may screw/loose all your own emails, at worst you may harm others. Information provided here, for learning purposes only. OK, enough precautions, let's dive…

Since I first started interacting with computer and computer systems, I had chance to deal with numerous mail servers, mail clients and various tools around them, both in Unix-like and Dos-like (including Windows) environments. Some of them are Sendmail (on very first versions of Linux), Procmail, Postfix, MDaemon (Windows-based server), Lotus Domino, MS Exchange of various versions, The Bat (Windows-based popular clients), Firefox Thunderbird, MS Outlooks, numerous cloud based mail service providers etc. List will go on and on. Unfortunately, in the beginning I was spending more time in Dos/Windows environments, rather than Unix-like environments for various reasons. However, last 6-7 years at least, I do spend almost all of my time on Linux and other Unix-like environments. This could become possible only after I left Windows-based mail client, for both my personal and business life. Before that transition, I was bouncing between happiness of "oh man, my mail setup (server/client) works nice" and "here is your money, Cloud, just take this pain away". Horrible moments…

With time, I realised that such struggle is caused by lack of understanding how mail is working and what we really want from it. While there was no lack of understanding how mail is working, formulating the requirements and expectations was a problem for me.

In this article I will be explaining what requirements I formulated for myself, and how I achieved to meet them.

High level goals for my mail setup I set as following:

• Must be minimal, i.e. use as less programs, modules, protocols as possible, re-use anything useful without introducing more complexity
• Must be maintainable, i.e. I should not spend tons of time to configure, backup and support my systems
• Must be local, i.e. single source of truth for my mails should be one local machine of my choice, backed up by me personally
• Should be secure as much as possible according to my understanding
• Should be easy to use as final user

For the one reading this, I should mention that making my mails accessible 24/7, from all devices and from all places is NOT among my goals. I don't want to spend all my free time staring on the screen of my phone, tablet or what's so ever device I may use. New mail in my mail box, does not signify that, I have to drop everything I do at the moment and start reading it, thinking on it, and/or taking actions on it. It just says that, here is new information arrived from the world. I want to be free to choose my time, which I will spend on processing this information. Thus, I deliberately choose not know what mails are arriving into my mail box, while I'm sleeping, eating, reading or doing some other thing.

Suppose, you are on vacation, and you received worst mail in your life, on which you won't be able to take action on straight away. Basically, you will loose peace of mind, and your vacation will be ruined. Normally we do not receive super urgent mail requiring immediate actions from people we don't know. In case of emergency, people we know will find a way to may contact.

Here is small picture, which illustrates the big picture, that we should keep in mind all the time.

digraph big_picture {
  rankdir = LR;

  bgcolor = transparent;

  internet -> MTA [label = "SMTP"];
  internet -> MDA [dir = back, label = "SMTP"];

  MTA -> store;

  store -> MUA [dir = back];
  MDA -> MUA [dir = back, label = "SMTP"];

  store [shape = cylinder];
  internet [shape = plaintext];
  MUA [shape = circle];

  MTA [shape = rect];
  MDA [shape = rect];
}

Figure above, illustrates very basic, functions and protocol level connections between them. Surprise, surprise, there is no POP3 or IMAP as you may see, just because they are totally optional if one decides to simplify things.

Considering all above, my minimal mail setup looks like following:

digraph minimal_mail_setup {

  bgcolor = transparent;

  subgraph cluster_virtual_server {
    label = "Virtual Server";

    MTA_MDA -> transient_store [label = "local\nfile\naccess"];
  }

  subgraph cluster_local_machine {
    label = "Local Machine";
    labelloc = b;

    MTA_MDA -> sendmail [dir = back, label = "SMTP over SSH"];
    sendmail -> MUA [dir = back, label = "execute\ncommand"];
    permanent_store -> MUA [dir = back, label = "local\nfile\naccess"];
  }

  internet -> MTA_MDA [label = "SMTP", dir = both];

  transient_store -> permanent_store [label = "rsync over SSH"];

  transient_store [shape = cylinder, label = "transient\nmaildir\n/var/mail"];
  permanent_store [shape = cylinder, label = "permanent\nmaildir\n~/.private/mail"];
  internet [shape = plaintext];
  MUA [shape = circle];
  MTA_MDA [shape = rect, label = "exim server\nMTA/MDA"];
  sendmail [shape = rect, label = "exim mua-wrapper\nsendmail\ncommand"];
}

When starting from scratch, first thing to do, is to make sure that messages at least flowing. exim has quite complex configuration syntax, but basically it drills down do three things, which are router, transport and acl. When message arrives over SMTP to exim, using routers it decides which transport should be used to deliver the message.

Considering our goals, here is the sample router, that we may use to select transport in exim:

begin routers

  # SKIPPED

  local:
    driver = accept
    domains = +local_domains
    require_files = /var/mail/$domain/$local_part/
    transport = localdir

Very simple and easy to understand, basically saying that, we accept mail messages for local addresses, domain part of mail address in the list of local_domains and /var/mail/$domain/$local_part/ directory exists. For some address like <me@staging.muradm.net>, this means that, router named local will check existence of /var/mail/staging.muradm.net/me/ directory. Does this work? For exim version before 4.94 it could, but for 4.94 it fails, while it works at the stage of router computation, it fails later in transport. exim on version 4.94 introduced tainted data thing, where variables filled with data from parsing the incoming mail message cannot be used directly to form destination of this message. To solve this we can use dsearch lookup mechanism (note that, your exim should have compiled support for it, i.e. LOOKUP_DSEARCH=yes should be set in Local/Makefile). Here is improved working snipped:

begin routers

  # SKIPPED

  local:
    driver = accept

    # we will serve mail addresses if there is a directory
    # named after it domain part under /var/mail directory
    domains = dsearch,filter=subdir;/var/mail

    # we will accept an optional suffix after local part
    # so that me/some-topic@my-domain.com is a valid mail address
    local_part_suffix = /*
    local_part_suffix_optional

    # our local part of mail address is valid only if
    # there is a directory named after it under directory
    # named after domain part
    local_parts = \
      ${lookup {$local_part} dsearch,filter=subdir {/var/mail/$domain_data}}

    # we will accept this message if and only if there is
    # a destination directory for it
    require_files = /var/mail/$domain_data/$local_part_data/$local_part_suffix_v/

    # once destination directory is validated for existence
    # we are preparing untainted version of destination for
    # use in transport and save it as address_data
    address_data = ${if eq{$local_part_suffix_v}{} \
                        {${lookup {$local_part_data} \
                                  dsearch,filter=subdir,ret=full \
                                  {/var/mail/$domain_data}}} \
                        {${lookup {$local_part_suffix_v} \
                                  dsearch,filter=subdir,ret=full \
                                  {/var/mail/$domain_data/$local_part_data}}} \
                   }

    # we will deliver it using transport named localdir
    transport = localdir

While it looks very crowded, due to configuration syntax, it is very simple in action, and also avoids use of global list maintenance like local_domains. Once I introduced dsearch lookup, I decided to add support for delivering into sub-directory based on optional suffix, which is very useful feature. Again, mail will not be accepted for any arbitrary directory, but only for the one is already created.

Now, as per exim, we need transport named localdir, where we are going to store received mail messages. As noted before, my choice was to store messages in maildir format. By default, maildir support in exim packaged for my Linux distribution, just like dsearch support was missing. To enable it, exim should be compiled with SUPPORT_MAILDIR=yes set in Local/Makefile. Here is transport, which delivers mail messages to maildir formatted directory.

begin transports

  # SKIPPED

  localdir:
    driver = appendfile
    directory = $address_data
    maildir_format = true

appendfile transport driver, stores messages locally, while maildir_format = true specifies that it should be locally in maildir format. Our destination directory is already computed by our router and provided in variable named $address_data.

Finally, exim requires us to have ACLs (Access Control Lists). The simplest form to make our basic flow working, can be as following:

# we will use diffrerent rules for incoming and outgoing
# messages based on the port value of connection established
acl_smtp_rcpt = ${if ={25}{$interface_port} \
                     {acl_check_rcpt} \
                     {acl_check_rcpt_submit} \
                }

begin acl

  # rules for incoming messages
  acl_check_rcpt:

    # again our address domain part should exist as directory
    deny sender_domains = dsearch,filter=subdir;/var/mail

    # sender's domain part should not exist as directory
    deny domains = ! dsearch,filter=subdir;/var/mail

    require verify = sender
    require verify = recipient

    accept

  # rules for outgoing messages
  acl_check_rcpt_submit:

    # domain part to whom we are sending should not exist
    deny sender_domains = ! dsearch,filter=subdir;/var/mail

    # our domain part should exist as directory
    deny sender_domains = dsearch,filter=subdir;/var/mail

    require verify = sender
    require verify = recipient

    # if one decides to add aditional level of protection
    # by submittion own mail over ssh, accepting can be
    # restricted to from localhost only
    # accept hosts = 127.0.0.1/8
    accept

With this, we are covering our required message flows. At this point we can test and validate that exim accepts only mail address combinations as we outlined before in Message Flow Cases.

Now that, our mails are flowing, we can observe new mail messages are being written to /var/mail/staging.muradm.net/me/. Let's look at what is inside:

# find /var/mail/staging.muradm.net/me/
/var/mail/staging.muradm.net/me/
/var/mail/staging.muradm.net/me/tmp
/var/mail/staging.muradm.net/me/new
/var/mail/staging.muradm.net/me/new/1631303258.H136013P19212.localhost
/var/mail/staging.muradm.net/me/new/1631303130.H958762P19203.localhost
/var/mail/staging.muradm.net/me/cur
#

We see three sub-directories, and two messages in sub-directory new. This is how maildir is working. Under each target directory, three directories are being created by exim:

• tmp - directory where exim writes message when delivering
• new - once write operation is complete, thus message is fully written to disk, it is being moved here, this is need in order to prevent readers access messages which are in the process of being written
• cur - tools, after successful presentation of message to the user, are moving it under this directory

But, this is on remote virtual server, how do we access them? Idea is very simple, if we use same maildir format to store messages on our local machine, the directory layout will be similar to the one on the server. According to maildir specification, file names of every message is generated in unique manner. So what is left is to move message from server to local machine. Which can be achieved with simple rsync over SSH:

rsync -avz \
  --remove-source-files \
  --exclude=*temp* \
  our-mail-server:/var/mail/staging.muradm.net/me/ \
  ~/.private/mail/me/

Provided that, you are accessing remote mail server over SSH using public key authentication, it should just work. And our "mail checking" or "mail retreival" procedure is becomes as simple as periodically run this command either manually, or using some periodic job scheduler. The following should be noted:

• --remove-source-files - will remove the files copied from the server
• --exclude=*temp* - will exclude files having "temp" in their name, we don't wont to receive partial messages which are in the progress of being written by exim

This way, we are keeping our remote footprint as minimal as possible. In the event of disaster on remote server, we might loose at most only messages we didn't move to local machine.

One should note that, file names generated according to maildir format, supposed to be unique. This will protect us from overwriting any message being moved to local machine.

When all messages are stored locally, we are free to arrange our backup strategy the way we want. All responsibility to manage our messages now lays on our shoulders.

As explained previously, we will be using DKIM for both authentication of users, and mail authenticity verification. Here again, exim comes to be very handy program. It has special mode where it is able to operate as command line tool to submit messages via SMTP, called mua_wrapper. This makes our life very much easier, since exim supports DKIM out of the box.

Since we are already familiar with exim configuration, figuring out way to use it on client side is not very hard task. Basically, we need one router and one transport without any ACLs in our configuration. router should pick the transport and transport will be deliver the message provided via standard input.

begin routers

  remote_delivery:

    driver = manualroute

    # we already validate sender/recipient on our mail server
    # just delegate this function to it, all messages to be
    # submitted there
    route_list = * mail.staging.muradm.net::587

    # if one uses suffix or prefix, it should be known to
    # exim client as well for proper $sender_address_local_part
    local_part_suffix = /*
    local_part_suffix_optional

    # using "from" address, we are picking which private key
    # to use for creating DKIM-Signature header
    # again in order to avoid tainted data, we are proving
    # its existence using dsearch
    address_data = ${lookup {$sender_address_local_part@$sender_address_domain-dkim.pem} \
                            dsearch,filter=file,ret=full \
                            {/home/muradm/.private/mail}}

    # finally pick transport
    transport = remote_forward

exim signs message in transport, when it delivers message to receiving mail server. In router, we will be leveraging from similar technique to pick the right primary key and store its absolute path in address_data. Here I'm using a number of mail addresses, for each I'm using separate private key for DKIM-Signature. But logic in picking private key can be different.

Once we have all necessary input, we can add transport:

begin transports

  remote_forward:
    driver = smtp
    dkim_selector = $sender_address_local_part
    dkim_domain = $sender_address_domain
    dkim_private_key = $address_data
    dkim_hash = sha256
    dkim_canon = relaxed
    dkim_strict = true

Very simple and straight forward. This way, exim will sign every message and add DKIM-Signature header for every message before submitting to our mail server.

In addition to that, one should be careful with exim_user and exim_group configurations when running exim as command line client.

Now let's go back to the mail server configuration. By default exim validates DKIM-Signature out of the box, but it does not decide whether to accept or reject messages with absent or invalid DKIM-Signature. Since we are using DKIM-Signature for authentication purposes also, we need to add extra ACLs to make decision.

# we might need different treatment of DKIM
# for incoming and outgoing messages
# note that this acl only applied
# if DKIM-Signature is present
acl_smtp_dkim = ${if ={25}{$interface_port} \
                     {acl_dkim} \
                     {acl_dkim_submit} \
                }

# DKIM acls work only when DKIM-Signature is present
# in case of absence we have to validate it manually
# which can be done at DATA stage
acl_smtp_data = ${if ={25}{$interface_port} \
                     {acl_check_data} \
                     {acl_check_data_submit} \
                }

# SKIPPED

begin acl

  # SKIPPED

  acl_dkim:
    # let's reject all incoming mail messages
    # submitted to us with missing or invalid
    # DKIM-Signature
    deny dkim_status = none:invalid:fail
    accept

  acl_dkim_submit:
    # we definitely must reject outgoing messages
    # submitted by us from our local machine if
    # DKIM-Signature is missing or invalid
    deny dkim_status = none:invalid:fail
    accept

  acl_check_data:
    # here we don't require public incoming messages to
    # have DKIM-Signature, but it is possible in the same
    # way as for our outgoing messages below
    accept

  acl_check_data_submit:
    # deny our outgoing messages
    # if there is no DKIM-Signature header
    deny condition = ${if eq{$h_DKIM-Signature:}{} {yes}{no}}
    accept

Now we have DKIM based authentication and protection in our mail server.

Setting up mail system is not very complex task, once one understands how does it work and what to expect from it. Method of configuration, provided here, is for illustration purposes, to show how minimal could mail setup be, using non-common, alternative approaches.

We didn't mention few important topics here, which could complete the picture:

• STARTTLS we didn't discuss at all, because it is trivial task to automate certificate acquisition using tools like Let's Encrypt and its certbot
• we didn't cover SPAM filtering, that is subject for another article
• MUA tools to use by user for managing their mail, there are tons of choices, once we have our mails properly stored on local machine, it should be trivial to configure indexers like mu or ~notmuch to index emails and use clients like GNU Emacs or mutt, however their configuration will require additional article

In case if you really need to check new mails from your phone, user could easily write small script that will aggregate the contents of /var/mail/ and provide some additional web interface tool to view them, or probably use that phone to login server via ssh using termux for example.