SM.Org software update 2012

Over the course of a few recent weeks I updated this site to a more modern software and revised some previously made choices. This one was loooong overdue considering that I still ran Ubuntu 9.10 before the update meaning that the system was almost 3 years old.

Here are some mostly useless but probably fascinating notes about it.

Core system upgrades

Running a comparatively low-load site I had a luxury not to plan for complex procedures minimizing downtime. What I did was just SSHing on the host and running do-release-upgrade under "screen" five times in a row. Linode's upgrade docs were holding my hand during the process.

Currently the site runs on Ubuntu 12.04 "Precise" and 3.0 Linux kernel. It's good to have modern packages!

Notable moments (read: long downtimes) during the upgrades:

• Ubuntu's integrated mail stack was good back then and these days it's just wonderful! Upon installing a single package "mail-stack-delivery" you get yourself a completely set up mail server that can send mail, accept mail from outside and that provides an IMAP interface to your mailbox. It even does necessary SSL key generation magic for me, so I can do authorization from my mail client securely.

• What I struggled with during all the mail upgrades was Yandex' anti-spam software that relies on manual creative editing of /etc/postfix/master.cf. I still use it simply because it works but I should probably look for some more actively supported solutions eventually.

• Postgres updated to 9.x. Database conversion went without a hitch. The only problem was that now Ubuntu seems to be not as conservative in memory settings as it was before so Postgres refused to run due to low memory on my machine (756M) until I reduced the amount of "buffers" (I have no idea what it is). Seems to run pretty smooth so far :-).

• The default Python is now 2.7. The change reminded me harshly that I was stupid enough to install some non-debianized Python packages into the system directory of which Ubuntu hadn't a faintest idea. As a result new Python didn't see them. I'm still trying to figure out how best to solve this (more on that later).

• The longest downtime was caused by upgrading Django to the newest trunk version. I neglected it for the time long enough for many DeprecationWarnings to become exceptions. Spent some quality time refreshing my code base. One thing remaining unresolved is that now Django insist on MEDIA_URL and STATIC_URL to have different values. I have no idea why it has to be a hard requirement. My problem is that I don't use uploaded media and hence probably shouldn't use MEDIA_URL at all. Except that my forum mutants are generated on the server using an ImageField which does use MEDIA_URL for storage. Still, nothing what I can't deal with in due time, just have to figure out the best way.

New web stack

I moved from the old "lighttpd + FastCGI with flup" setup to the new one with "nginx + uwsgi". For some personal, completely subjective reason which I don't even remember anymore I always preferred lighttpd over nginx. However this was the case where the best maintainer won. It seems that nginx is more actively developed and uwsgi has most mind share. I also suspect all this is old news to everyone except me :-). But what actually bought me was the built-in support for uwsgi in nginx. I love integrated solutions! It means that I will have to write less of stupid glue-code in files that I will later forget.

The config file for nginx turned out to be much simpler than the one for lighttpd. It looks like it was specifically designed for the kind of tasks that web server admins do rather than being not-exactly-Turing-complete Perl-like code that happen to cover most use-cases with the enough amount of regexps.

OK, I just have to show you an example. Here's what I needed to do to handle some legacy redirects with lighttpd (heavily stripped and simplified):

fastcgi.server = (
  "/fcgi" =>  (
    (
      "socket" => "/var/run/sm_org/fcgi.socket" ,
      "check-local" => "disable",
    )
  )
)

url.redirect = (
  "^/soft/tags/(.*)" => "/soft/tagsfield/$1",
)

url.rewrite-once = (
  "^(/soft/tags/)(.*)" => "$1$2",
  "^/(.*)$" => "/fcgi/$1",
)

I don't even mind the infamous rewrite hack to connect the FastCGI backend (got used to it). But having a redirect and a corresponding no-op rewrite to make the former work… Seriously? (Before you ask, there is a reason why those redirects are not handled by Django code.)

Here's the nginx version that simply looks like a thing like that should look:

server {
  server_name softwaremaniacs.org;

  rewrite ^/soft/tags/(.*) /soft/tagsfield/$1 permanent;

  location / {
    include uwsgi_params;
    uwsgi_pass unix:/var/run/uwsgi/sm_org.socket;
  }

}

The uwsgi part of the story however wasn't that bright. What inspired my to try it out was this article about running multiple sites under uwsgi "Emperor mode". But since I don't run multiple sites I decided to run it first in a simple way.

With only one site to run I ditched the idea of having a separate upstart config for the master uwsgi process and a separate config for a site. Instead I put all the parameters in the upstart script itself as arguments to the uwsgi command.

Then I spent some time figuring out why it simply didn't run. Turns out that uwsgi no longer supports --module argument contrary to the statement that the config file keys are equivalent the command line arguments. The fact that Django uwsgi doc also refers to --module didn't help. Neither the fact that upstart has no diagnostics whatsoever (or I couldn't find one).

So I reconfigured everything toward the Emperor mode.

Then I spent some time trying to convince nginx running under "www-data" to talk to uwsgi running under my local user. Yes, this is uncommon, but solving that problem was way out of scope of my intentions. Anyway, apparently there's the wonderful chmod-socket option in uwsgi that solved it. Also I could probably use a TCP socket (by the way, does anyone know what's the practical difference and why everyone seems to prefer using unix sockets?)

Then I spent some time looking helplessly at nginx complaining that it doesn't get data from the backend and uwsgi writing some logs that didn't seem to have anything to do with it. Apparently the important line in those logs was:

-- unavailable modifier requested: 0 --

… which means "you don't have uwsgi-plugin-python installed". Obvious, right? :-)

Now, I don't exactly blame Ubuntu (or Debian?) maintainers who, after splitting uwsgi functionality into plugins, didn't think it necessary to include or recommend a single one of them to make uwsgi, you know, useful. Neither do I blame uwsgi maintainers for this cryptic error message. And neither do I blame myself for overlooking a warning in the uwsgi Quickstart guide that would solve my problem.

What I blame is the whole way of setting up computer software that we established over the last century. The culture of making every imaginable little thing configurable first, then forcing multitude of users to solve a few similar configuration task a million times over… But that's just frustration so please don't mind me!

OK, here are my uwsgi configs in case you wondered (but please use official docs whenever you can).

The upstart script responsible for running the uwsgi Emperor /etc/init/uwsgi.conf:

start on runlevel [2345]
stop on runlevel [06]

exec /usr/bin/uwsgi \
  --emperor /etc/uwsgi/apps-enabled \
  --uid maniac \
  --gid maniac

• You do want to run the Emperor mode even if you have only one backend simply because then you can restart it gracefully without using sudo. Just chmod your site config to own it yourself.
• You'll probably want to use "www-data" for uid and gid, though I personally find it more convenient to run my code under my local user.

The config describingn instance of the site backend /etc/uwsgi/apps-available/sm_org.ini:

[uwsgi]
master = 1
chdir = /home/maniac/sm_org
module = wsgi
processes = 5
max-requests = 1000
plugins = python
socket = /var/run/uwsgi/sm_org.socket
chmod-socket = 777

• I have no idea why you need the "master" option but it seems that everyone uses it. Let's keep everyone happy!
• chdir into the project's directory is needed because my code does relative imports of project apps located in that directory. Yours probably does it too.
• module = wsgi means the wsgi.py in the project directory that exports Django wsgi app instance. In your case it might be called projectname.wsgi if you follow current Django project layout.
• plugins = python is what makes uwsgi actually run Python code. You can skip "http" that some docs suggest here if you don't use your uwsgi backend as an HTTP server.
• chmod-socket = 777 is what allows my "www-data" owned nginx talk to the "maniac" owned uwsgi backend. You might not need it if you run them both under the same user or use a TCP socket.

virtualenv

I have one unresolved question right now. OK, there's actually more than one but this one occupies me most: to use or not to use virtualenv.

Here are my thoughts:

• I'm not a hosting company working for external clients so I don't run multiple sites with different set of packages. This is not my use-case for virtualenv. I'm quite happy maintaining my whole Django codebase using the trunk version of Django.

• I cannot rely only on system Python packages simply because some of them Ubuntu doesn't provide. PyPI is the place where all new packages live now and I want to use it conveniently.

• Installing packages with pip into the system site-packages is broken and out of the question. This is where the idea of using virtualenv comes up. But probably I could just tell pip to use a specific installation directory? I couldn't find the option for it.

• What I don't like about virtualenv is that it makes my life harder. I should either "activate" my single environment all the time or use explicit paths everywhere to run commands. If this is how everyone does this then the world definitely gone mad :-). I'd rather keep my current way with tweaking PYTHONPATH . But then I'll still have the problem of pip trying to install everything in site-packages :-(.

What a man to do?