Server upgrade 2020

Regular readers of this blog have probably noticed that it is now served from a different IP address… Okay, okay, I'm kidding! This blog doesn't have any regular readers, of course.

Anyway, what I'm trying to say is that I recently spent quite some effort to move all my stuff — softwaremaniacs.org, highlightjs.org, their supporting databases and, most importantly, my personal email — to a new and shiny Linode instance under control of an up-to-date Ubuntu.

Here's the (heavily compressed) war story.

Ding!

It all started with an email from letsencrypt politely telling me my client software (certbot) must be too old as it uses an obsolete version of their protocol, and it's going to break when they drop support for it. Little did I know at that moment just how hairy a yak I was staring in the eyes…

Steps and stumbles

• Try upgrading the package via apt: no new versions available for this version of Ubuntu (I think it was 18.04).

• Try upgrading Ubuntu (long overdue) via do-release-upgrade: no more versions of Ubuntu available for the 32-bit architecture.

• Reach to Linode support asking why my 64-bit machine tells me it's 32 bit: get a few useful pointers revealing that my system is in fact a Frankenstein's monster: a 32-bit kernel with a 64-bit userspace. (By the way, Linode has the best support team!)

• Figure there's no way to upgrade the mess in place: need to provision a new machine and build all the services from scratch moving data from the old machine. Django apps, HTTP configs, Postgres databases and all the mail.

• Fire up a new instance, wait weeks for "when there's more time" to deal with it (you know how that ends).

• Meanwhile, software running highlightjs.org really needs an upgrade to support its new release process (it's always node.js for some reason): realize I can do it relatively easy on the new server without touching softwaremaniacs.org (thinking of migrating mail gives me headaches).

• Move entire highlightjs.org stuff in a semi-manual process, switch DNS, wait for the traffic on the old instance to die down in a day. Now I have two servers: really need to find more time to finish the transition!

• Move entire softwaremaniacs.org stuff in a semi-manual process, but don't switch DNS: I can't just move my mail server in a similar way because unlike mostly read-only Web apps, I will have to be accepting mail on both hosts while DNS is expiring.

• Anyway, need to set up a mail server on the new instance: turns out mail-stack-delivery package is not supported anymore. If you didn't know, mail-stack-delivery was a brilliant package maintained by Canonical that installed all packages needed for a personal mail server and then went ahead and wrote all the configs for you.

• But fret not, there's now the mail-server task, ostensibly doing the same: alas, it actually only installs postfix and dovecot but doesn't configure anything.

• Work up courage to read the 3-part personal email howto on ArsTehnica which I had opened in my browser for the past 80 years or so: realize it's opinionated, complicated and in places outdated (but still a good resource for orientation).

• Educate myself on "postfix", "dovecot", "Maildir", "sasl", "starttls", "spamassassin", "milters", "sieve", "fail2ban", "spf", "dkim": I guess I now know why Canonical doesn't want to support an out-of-the-box solution anymore.

• Set up mail, test the hell out of it by using my local /etc/hosts to point softwaremaniacs.org to the new server.

• Set up a temporary relay from the old server to the new one: realize postfix is not okay with relaying mail to another postfix that thinks it has the same name.

• Pretend the new server is called new.softwaremaniacs.org, switch DNS, wait for traffic on the old one to die down.

• Finally successfully install certbot on the new server, obtain new certificates. They were about to expire in just two days!

• Shut down the old server. RIP.

Outcomes

• I now have a cleaner, simpler setup which I understand. My quarantined spam now ends up in a regular mail folder (before, I had to SSH onto the machine and dig through some obscure file names).

• My outgoing mail has a better chance of reaching its destinations (but it's still an ongoing fight).

• My mail server doesn't use weird custom ports, so it's easier to set up mail clients.

• I'm getting less spam.

• My certificates are not in the immediate danger of expiring.

• My Python code is now using all the 3.8-isms I want! I even refactored quite a bit of site code, just for fun.

Next

I feel like I need to write my own mail server setup guide… Oh well…