Setting up a mail relay server with Postfix, DKIM, and a little Nebula trickery.

Given the proliferation of spam on just about every vaguely workable platform these days it seems sheer insanity to attempt to run your own mail server.  If it's out there, it's ripe for abuse in one way in another.  And yet, e-mail is still probably one of the best ways to get status reports from your machines every day (my SMTP bridge notwithstanding).  It is thus that the default configuration for mail servers these days defaults to "no way in hell will I relay a message for you," which is a net good for the the Internet as a whole, but by and large a huge pain in the ass if you actually want to set up a mail relay for some reason.  In my case, I wanted to set up Leandra (running in my rack at home) to relay outbound mail through another of my servers on the outside.  I further wanted to ensure that Leandra's outbound mail had the same kind of authentication and protection measures the rest of my machines have so that my servers wouldn't wind up on any spam blacklists and would be significantly difficult (because there is no 'impossible') to abuse.

The first thing I had to do was set up an A record in DNS for leandra.virtadpt.net pointing at the same IP address as the server I wanted to relay through.  If this were a sane and reasonable world I'd just set an alias with a CNAME record but it seems like nothing out there plays nicely with aliases anymore.  There is nothing that says at a single IP address can't have more than one hostname associated with it, so this isn't a big deal.

However, DKIM is kind of a big deal.  I don't fully understand it but I'll try explaining it as best I can, at least insofar as it applies to our use case.

The foundation of spamming is spoofing the origin of a message, meaning the sending e-mail address.  There are various and sundry ways of making this difficult, at the very least by preventing servers from relaying mail for anyone and everyone.  One of the latest ways to prevent this from happening is called DKIM (Domain Keys Identified Mail).  Without going into too much detail, DKIM uses public key encryption to help prove that a message came from a particular place.  When you set DKIM up on a mail server you generate a private key (which stays on the box) and a public key (which gets published in DNS as a TXT record).  You also plug some extra software into your mail server which generates and appends a digital signature to every message sent from or through that machine (parts of the message are hashed, the hash is encrypted with the private key on the mail server, and the resulting signature goes into a new SMTP header on the message).  When the signed message hits the receiving server, the mail server makes a couple of DNS queries to dig up the public key of the mail server that sent it and verifies the DKIM signature on the message.

The nice thing about DKIM?  End users don't have to worry about it.  It's all server side.  If a message fails DKIM authentication you might see a warning message when you open the message telling you that it might have been forged.

Okay.  So, what kind of sysadmin crap did I do to make this work?  To keep from messing things up too badly I did all of the steps that wouldn't impact anything else running on Exocortex first by following an excellent tutorial over at Linuxbabe.  I'm typing this process up in part because this blog is a manifestation of my external memory, and in part because there are folks out there (and I don't blame you) who are more comfortable doing something tricky once someone else has done it and explained what happened each step of the way.

For the record, all of these commands are run as the root user.  Also for the record, everything I'm going to show you is the public side of this setup.  All of the information has to be public by definition, because other mail servers on the Net need to be able to look this stuff up.  Anyone out there can go splunking through DNS or run a couple of searches on SHODAN and find it for themselves.

Of course, you'll want to adapt the configuration options and commands to match your own host- and domain names, IP addresses, and so forth.  Don't forget to make backup copies of your config files (you DO make backups, don't you?)

The first step was to install the OpenDKIM daemon on Exocortex.  As described above it ingests outbound e-mail on one side, digitally signs it, and barfs it out the other side by passing it off to Postfix.  This was probably the simplest part: apt-get install opendkim opendkim-tools

One of the things that's not terribly clear and is easy to trip over is that the postfix service account (which the various parts of Postfix run as) has to be added to the opendkim system group so that it can interact with the new parts you just installed.  Also, keep the OpenDKIM daemon out of your way while you work on other stuff: systemctl stop opendkim

I like to make a backup copy of the supplied config file before I edit anything, just in case I screw it up: cp /etc/opendkim.conf /etc/opendkim.conf.orig

While I could reiterate the instructions I followed originally, I've already linked to them so I'll start giving you the lazy way because it's easier to explain.  Edit the /etc/opendkim.conf file in your favorite text editor and add the following lines:

# How strictly to treat reorganization and reformatting of the
# headers and message body before considering the signature bad.
# Headers can be reformatted, but the message body can't.
Canonicalization   relaxed/simple

# What DKIM functions to fulfill: sign, verify
Mode               sv

# Also sign messages for recognized subdomains?
SubDomains         no

# If the opendkim daemon dies, restart it.
AutoRestart         yes

# If the opendkim daemon needs to be restarted, limit the attempts
# to 10 tries every minute.
AutoRestartRate     10/1M

# Run in the background?
Background          yes

# How long to wait before giving up on DNS (in seconds).
DNSTimeout          5

# Algorithm to use for the digital signatures.
SignatureAlgorithm  rsa-sha256

# Path to the file that contains the mappings between the DNS TXT
# record holding the pubkey and the location of the private key on
# the machine.
KeyTable           refile:/etc/opendkim/key.table

# Path to the file that contains the mappings of usernames on the
# server to DNS records containing the public keys.
SigningTable       refile:/etc/opendkim/signing.table

# Path to a file that contains a list of IP addresses, hostnames, and
# other system identifiers whose DKIM signatures won't be checked
# (because, by definition, they're known good because they're ours).
ExternalIgnoreList  /etc/opendkim/trusted.hosts

# Path to a file that contains a list of IP addresses, hostnames, and
# other system identifiers which may send mail through the server and
# get the DKIM signature.  If it's not in this list, it gets dropped.
InternalHosts       /etc/opendkim/trusted.hosts

Now we need to set up some directories to store stuff:

root@exocortex:/etc()# mkdir -p /etc/opendkim/keys
root@exocortex:/etc()# chown -R opendkim:opendkim /etc/opendkim

Time to write a few new config files that are specific to your server and domain.  The first file is /etc/opendkim/signing.table, with the following contents:

*@exocortex.virtadpt.net default._domainkey.exocortex.virtadpt.net
*@leandra.virtadpt.net default._domainkey.leandra.virtadpt.net

Of course, you'll want to make the contents reflect your situation.  Now create the /etc/opendkim/key.table file.

default._domainkey.exocortex.virtadpt.net exocortex.virtadpt.net:default:/etc/opendkim/keys/exocortex.virtadpt.net/default.private
default._domainkey.leandra.virtadpt.net leandra.virtadpt.net:default:/etc/opendkim/keys/leandra.virtadpt.net/default.private

Next file to create: /etc/opendkim/trusted.hosts

127.0.0.1
localhost
104.131.152.154
*.virtadpt.net
172.16.0.6

The IP address 104.131.152.154 is that of Exocortex, and is listed so that mail originating from the server will be DKIM signed.  You will, of course, put the IP address(es) of your server here instead.  See that IP address at the very bottom (172.16.0.6)?  That is Leandra's Nebula network IP address.  Leandra is going to hit Exocortex over a Nebula connection because mail relaying in Postfix is restricted, in part, by IP address of origin, and Leandra is on a connection with a dynamic IP address.  All things considered, I'd much rather not have to futz with dynamic hostnames, guessing netblocks, or anything like that.

Then I created two directories, one per hostname, to hold the DKIM key material: mkdir -p /etc/opendkim/keys/{exocortex,leandra}.virtadpt.net

Generate the key material for the two hostnames:

root@exocortex:/etc()# opendkim-genkey -b 2048 -d exocortex.virtadpt.net -D /etc/opendkim/keys/exocortex.virtadpt.net -s default -v
root@exocortex:/etc()# opendkim-genkey -b 2048 -d leandra.virtadpt.net -D /etc/opendkim/keys/leandra.virtadpt.net -s default -v

In each directory you will find two files, default.private (which contains the private key for the hostname) and default.txt (which contains the text of the entire TXT record that has to be copied into the DNS zone).  Here are the contents of one of mine:

root@exocortex:/etc/opendkim/keys/exocortex.virtadpt.net()# cat default.txt
default._domainkey      IN      TXT     ( "v=DKIM1; h=sha256; k=rsa; "
          "p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzOQVBHlkoI67vwT5Vh+PSo4UltdIiX+dI3iBTQRw99AseFCVvwVKjRmrfQ8fGCJ7NdMVoiHzZczN0h8oBTNPPS8VIoN5rMJn4sz5c+kSOr16iD/9R7kZBKMnGqSMu5+YMHkMrPJKGubHKz3pSVmBAKsB3ew2JgRpJ0NBPadeK9lI2eu46fe9BooNlYttCfFDNPQvJcNTQmiUEH"
          "7aPw3v8tdwifoY4EPB/XkWIMxxBxk1bHGN4vsB3NMdFmlStRO8ptVGGccsdBeMZv6s9WCLuIhnPlCxZGY1SD4cTDOc/qzEKSaBt7Tn6Hf3feaKgsSIN01dP6G47ojUT95r4lu6oQIDAQAB" )  ; ----- DKIM key default for exocortex.virtadpt.net

When I installed the record into the DNS zone files, it looked like this:

v=DKIM1; h=sha256; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzOQVBHlkoI67vwT5Vh+PSo4UltdIiX+dI3iBTQRw99AseFCVvwVKjRmrfQ8fGCJ7NdMVoiHzZczN0h8oBTNPPS8VIoN5rMJn4sz5c+kSOr16iD/9R7kZBKMnGqSMu5+YMHkMrPJKGubHKz3pSVmBAKsB3ew2JgRpJ0NBPadeK9lI2eu46fe9BooNlYttCfFDNPQvJcNTQmiUEH7aPw3v8tdwifoY4EPB/XkWIMxxBxk1bHGN4vsB3NMdFmlStRO8ptVGGccsdBeMZv6s9WCLuIhnPlCxZGY1SD4cTDOc/qzEKSaBt7Tn6Hf3feaKgsSIN01dP6G47ojUT95r4lu6oQIDAQAB

Note that I spliced together everything inside the double quotes into a single line and dropped it into a TXT record, default._domainkey.exocortex.virtadpt.net.  I don't know why it outputs the DNS record that way, but I suspect it has to do something with defaulting to BIND's zone file format because it's the most commonly used out there.  If you don't know what BIND is, don't worry about it.  It's not relevant to this post unless you run your own DNSes, in which case you know all about this anyway.

Now set file ownership on the private keys so that the opendkim daemon can read them (substitute as necessary, of course): chown opendkim:opendkim /etc/opendkim/keys/*.virtadpt.net/default.private

One of the things about DNS is that it takes time for changes to propagate, doubly so when you use a hosted DNS provider.  It takes both patience and periodic testing to see when the records are available.  I used the following command to look for the above DNS record: opendkim-testkey -d exocortex.virtadpt.net -s default -vvv

Before the DNS zone was reloaded at Dreamhost, the TXT record wasn't yet available, which resulted in the following output:

opendkim-testkey: using default configfile /etc/opendkim.conf
opendkim-testkey: checking key 'default._domainkey.exocortex.virtadpt.net'
opendkim-testkey: 'default._domainkey.exocortex.virtadpt.net' record not found

When the DNS record had successfully propagated this was the output:

opendkim-testkey: using default configfile /etc/opendkim.conf
opendkim-testkey: checking key 'default._domainkey.exocortex.virtadpt.net'
opendkim-testkey: key not secure
opendkim-testkey: key OK

("key not secure" means "DNSSEC isn't in use here."  Don't worry about it.)

By default, Postfix gets stuck into a chroot in the directory /var/spool/postfix when it starts up to minimize any damage it could do if something goes wrong.  This implies that the socket which Postfix uses to communicate with OpenDKIM must exist somewhere inside that directory structure.

root@exocortex:/etc/opendkim/()# mkdir /var/spool/postfix/opendkim
root@exocortex:/etc/opendkim/()# chown opendkim:postfix /var/spool/postfix/opendkim

Now OpenDKIM needs to be configured to find the socket.  There are two files (/etc/opendkim.conf and /etc/default/opendkim) which need to be edited in the same way: Search for the string "socket" (case insensitive) in each file and change its value to read "local:/var/spool/postfix/opendkim/opendkim.sock"

Now the tricky bit, where stuff can break.  Edit Postfix's primary configuration file /etc/postfix/main.cf and append the following lines at the bottom of the file:

# If the OpenDKIM milter isn't available, accept the message anyway.
milter_default_action = accept

# What milter communication protocol should be used to pass messages
# to and from OpenDKIM?  Just go with it.
milter_protocol = 6

# Where should the OpenDKIM milter be contact through?  Note that this
# is inside the /var/spool/postfix chroot.
smtpd_milters = local:opendkim/opendkim.sock

# Send mail that doesn't arrive from the network through the same milter
# as outbound mail.
non_smtpd_milters = $smtpd_milters

We're in the home stretch.  We can test the Postfix configuration file without having to restart the server and catch any errors early:

root@exocortex:/etc/postfix()# ls
dynamicmaps.cf    main.cf.bak.1  master.cf.bak.1  postfix-script
dynamicmaps.cf.d  main.cf.proto  master.cf.proto  post-install
main.cf           makedefs.out   postfix-files    sasl
main.cf.bak       master.cf      postfix-files.d  ssl
root@exocortex:/etc/postfix()# postfix check
postfix: Postfix is running with backwards-compatible default settings
postfix: See http://www.postfix.org/COMPATIBILITY_README.html for details
postfix: To disable backwards compatibility use "postconf compatibility_level=2" and "postfix reload"
root@exocortex:/etc/postfix()# echo $?
0

The last command (echo $?) means "show the exit status that postfix reload sent."  In this case a 0 means "it's all good."  If there were any problems, postfix check would have displayed errors.  The bit about "backwards-compatible default settings" seems pretty harmless - Postfix will run normally but it's telling you that you should probably look into updating your configuration settings a little.  I haven't really dug into this (though I should).

To adhere to best practices I edited /etc/postfix/master.cf to listen on a new port.  By RFC 4409 the port which is supposed to accept messages to relay on behalf of another server or user should be port 587/tcp.  In the /etc/services file this port is named "submission."  That said, the line I added to the /etc/postfix/master.cf file by copying the line above it and making a minor edit:

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
submission inet  n       -       y       -       -       smtpd

"Open a new port that is called 'submission' in /etc/services.  It is an inet service, meaning that it opens a port on at least one network interface.  Because it is an inet service it is not private (incidentally, Postfix requires this setting on inet services, so don't get creative with it).  It is an unprivileged service (the default), so it will run as the postfix user.  This service runs in the chroot, so apply those restrictions to it.  The service on this port is always running so don't worry about the wakeup feature.  The maximum number of workers servicing this port is 100 processes.  The worker which will handle requests on this port is the smtpd module."

I could have specified a specific port and not the string "submission" but I wanted to get things working and not dork around with hunting down bugs and mismatches.  I know I can cut the number of workers down from 100 but without a reason to do so I'm not worried about it just yet.

Now to kick everything off: Start everything up and send a test message with the command line mail utility while watching the system logs:

root@exocortex:/etc/postfix()# systemctl start opendkim
root@exocortex:/etc/postfix()# systemctl restart postfix
root@exocortex:/etc/postfix()# mail my-test-account@gmail.com && journalctl -xf
Subject: This is a test message.

This is a test message for the tutorial I'm writing.

--me
.
Cc: ^D
-- Logs begin at Sat 2020-07-25 13:51:49 UTC. --
...
Sep 27 22:42:03 exocortex.virtadpt.net run.sh[1482]: 127.0.0.1 - - [27/Sep/2020 22:42:03] "GET /Jayce HTTP/1.1" 200 -
Sep 27 22:43:55 exocortex.virtadpt.net postfix/pickup[5220]: ECE078203E: uid=0 from=<root>
Sep 27 22:43:55 exocortex.virtadpt.net postfix/cleanup[8058]: ECE078203E: message-id=<20200927224355.ECE078203E@exocortex.virtadpt.net>
Sep 27 22:43:55 exocortex.virtadpt.net postfix/qmgr[25609]: ECE078203E: from=<root@exocortex.virtadpt.net>, size=525, nrcpt=1 (queue active)
Sep 27 22:43:56 exocortex.virtadpt.net postfix/smtp[8061]: connect to gmail-smtp-in.l.google.com[2607:f8b0:400e:c09::1b]:25: Cannot assign requested address
Sep 27 22:43:56 exocortex.virtadpt.net postfix/smtp[8061]: ECE078203E: to=<my-test-account@gmail.com>, relay=gmail-smtp-in.l.google.com[74.125.20.27]:25, delay=0.96, delays=0.02/0.01/0.27/0.66, dsn=2.0.0, status=sent (250 2.0.0 OK  1601246636 t5si5096039pjm.145 - gsmtp)
Sep 27 22:43:56 exocortex.virtadpt.net postfix/qmgr[25609]: ECE078203E: removed
^C

^D means "hit control-d to end input" and ^c means "hit control-c to stop following the logfile."

Llet's look at the test message I sent to my Gmail address.  The reason I sent it there is because Gmail makes it really easy to look at the DKIM report for the message.  Open your test message in Gmail, click the three dots menu at the right of the message, and click Show Original.  Among other things, you will see in the report the result of the SPF test (don't worry about this, it's out of scope for this post), the DMARC test (also out of scope), and the test of our DKIM setup.  Which, if it worked, you should see that the Big G gave it a 'PASS'.  Success.

Now for the weird part of my setup, which is probably optional for you: Configure Nebula so that Leandra could contact Postfix over the VPN to send mail.  As mentioned earlier, Leandra's IP address on my Nebula VPN network is 172.16.0.6 and Postfix was configured to accept e-mail relay from that address.  To make this happen I had to add the SMTP submission service we just built to the Nebula configuration file on Exocortex.  This is actually really easy to do by editing /etc/nebula/config.yml and adding the following block:

    # Allow 587/tcp from any host in the home group
    - port: 587
      proto: tcp
      groups:
        - home

To make the changes take effect I restarted Nebula: systemctl restart nebula

I've had this setup running for a couple of weeks now and it has been stable and doing what I wanted, which is that system reports from root (at) leandra dot virtadpt dot net (spamblocked, and doesn't receive mail anyway) show up in my inbox elsewhere reliably, with the hostname I wanted, and pass anti-spam measures the way they're supposed to.