nfp

So what happened was, I fed up manually uploading pictures I export from Darktable to my Flickr photo stream using the browser's file picker. So I decided to do something about it. The initial idea was to craft a FUSE file system which would automatically upload new files, but this turned out to be hard, so I switched to a much simpler solution: a little inotify watcher handing over new files to an upload script. I managed to code up a working solution over a weekend!

More interestingly, I made the watcher part — "nfp", for "New File Processor" — as a generic configurable tool which I published. It was only when I started writing this very blog post that I stumbled upon a standard Linux tool that does it, inotifywait :-)

Still, I hope there's something to be salvaged from this project. Read on!

Darktable

Darktable is my tool of choice for working with camera RAWs, and I just want to take a moment to share my appreciation for the folks making it. It's a ridiculously advanced, polished photo processor. A real testament to open-source software.

It actually used to have a Flickr export plugin, but it hasn't been working for a while, and got dropped in recent versions. Which is totally fair because it's very much out of scope for a photo editing software. Having a generic solution like nfp makes much more sense because it can connect arbitrary file producers and consumers. It doesn't even have to be about images.

Rust

Since "inotify" sounds very "systems" and "core", I immediately took it as an opportunity to play with Rust once more. That was the main reason. A nice side effect of it is that it builds into a small self-contained binary which you can bring with you anywhere. As long as it's Linux, anyway :-)

If I had to mention a single gripe with the language during this last foray, that would be implementing an ordered type with the PartialEq/Eq/PartialOrd/Ord trait family. This just feels unnecessarily hard. I still don't get what's the point of having partial variants, and why things couldn't be inferred from each other. Like, even the official docs on Ord recommend writing a boilerplate for PartialOrd that just calls out to Ord. I'm sure there are Reasons™ for it, but somehow Python can infer total ordering from just __eq__ and __lt__.

Debouncing

After using the tool for a week I noticed that the uploaded photos didn't have any metadata on them. After some digging this turned out to be due to the way Darktable writes exported files: it does it twice for every file. The second write, I assume, is specifically to add metadata to the already fully written JPEG. The problem was, nfp has been snatching the file away immediately after the first write.

The only way I know how to deal with this problem is "debouncing", a term familiar to programmers working with UI and hardware. Which means, adding a short grace period of waiting until a jittery signal stops appearing on the input or the user stops rapidly clicking a button. Or Darktable stops rapidly overwriting a file.

Quick search for a generic debouncer for Rust turned up only specific solutions tied to mpsc channels, or async streams, or hardware sensors. So I wrote my own debounce, which is a passive data structure with a couple of methods that doesn't want to know anything about where you get the data and what's the waiting mechanism. It just tracks time and removes duplicates.

I may yet turn it into a full-blown crate, and may be build a unixy-feeling debounce tool along the lines of:

inotifywait -m -e close_write /path | debounce -t 500 | python upload.py

Update: this has been implemented.

To do it properly though, I'll have to implement it as a two-threaded process, which will give me an opportunity to play with concurrency in Rust, something I haven't done yet. In nfp I cheated: it waits for new notifications on the same thread that sleeps for debounce timeouts, so it uses an ugly hack of sleeping in short chunks and constantly checking for new events:

loop {
    match debouncer.get() {
        State::Empty => break,
        State::Wait(_) => sleep(Duration::from_millis(50)),
        State::Ready(file) => { ... }
    }
    for event in inotify.read_events(&mut buffer)? {
        debouncer.put(...)
    }
}

Flickr uploader

The uploader script was a story in itself. Ironically I spent more time trying to make various existing solutions work for me than I did with nfp, but didn't have any luck. So I ended up cobbling together a Python script using flickrapi.

The ugly part of all these scripts is OAuth. More precisely, its insistence on having to register a client app to get a unique id and secret (apart from the user auth for whoever is going to be using it). It's totally fine for a web service, but in anything distributed to user-owned general-purpose computers it means that a determined user can fish out the client credentials and use them for something else (oh horrors!) I remember dealing with this problem when we worked on an OAuth service for Yandex around 2009, and we didn't come up with a good solution for it. These days I believe client credentials should be optional, akin to the User-Agent header in HTTP, and shouldn't be used for anything outside of coarse statistical data.

Anyway… Since I'm using this script only for myself, I registered it on Flickr, put the credentials in a config and forgot about it :-)

Here's the whole script for posterity:

import argparse
import logging
from pathlib import Path

import toml
import flickrapi


logging.basicConfig(level='INFO')
log = logging.getLogger()


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('filename', type=str)
    args = parser.parse_args()

    filename = Path(args.filename)
    config = toml.load(open(Path(__file__).parent / 'flickr.toml'))
    token = flickrapi.auth.FlickrAccessToken(**config['user'])
    flickr = flickrapi.FlickrAPI(**{**config['app'], 'token': token})

    log.info(f'Uploading {filename}...')
    title = filename.name.split('.', 1)[0]
    # TODO: use 'Xmp.darktable.colorlabels' to control visibility
    result = flickr.upload(filename, title=title, is_public=0)
    if result.attrib['stat'] != 'ok':
        raise RuntimeError(result)
    log.info(f'Successfully uploaded {filename}')


if __name__ == '__main__':
    main()

P.S. I love dict destructuring with **!

What's next

I'm not yet sure what to do with nfp. The good sense tells me to extract a debouncer out of it and drop the rest in favor of inotifywait, but it actually does add some extra value: it has a sensible config format and I can modify it further into being able to exec multiple processor scripts in parallel. Although I suspect the latter part can be handled by yet another unix voodoo :-)

And its best feature is that it works for me right now!