Software Maniacs blog » OAuthhttps://softwaremaniacs.org/blog/category/oauth/en/2022-01-09T13:13:40.753060-08:00ManiacIvan Sagalaev on programming and web developmenthttp://softwaremaniacs.org/media/sm_org/style/photo.jpgnfp
2022-01-09T13:13:40.753060-08:00https://softwaremaniacs.org/blog/2021/10/11/nfp/en/So what happened was, I fed up manually uploading pictures I export from Darktable to my Flickr photo stream using the browser's file picker. So I decided to do something about it. The initial idea was to craft a FUSE file system which would automatically upload new files, but this ...
<p>So what happened was, I fed up manually uploading pictures I export from <a href="https://www.darktable.org/">Darktable</a> to my <a href="https://www.flickr.com/photos/isagalaev/">Flickr photo stream</a> using the browser's file picker. So I decided to do something about it. The initial idea was to craft a FUSE file system which would automatically upload new files, but this turned out to be hard, so I switched to a much simpler solution: a little <a href="https://www.man7.org/linux/man-pages/man7/inotify.7.html">inotify</a> watcher handing over new files to an upload script. I managed to code up a working solution over a weekend!</p>
<p>More interestingly, I made the watcher part — "nfp", for "New File Processor" — as a generic configurable tool which I <a href="https://nest.pijul.com/isagalaev/nfp">published</a>. It was only when I started writing this very blog post that I stumbled upon a standard Linux tool that does it, <a href="https://www.man7.org/linux/man-pages/man1/inotifywait.1.html">inotifywait</a> :-)</p>
<p>Still, I hope there's something to be salvaged from this project. Read on!</p>
<p><a name=more></a></p>
<h2>Darktable</h2>
<p>Darktable is my tool of choice for working with camera RAWs, and I just want to take a moment to share my appreciation for the folks making it. It's a ridiculously advanced, polished photo processor. A real testament to open-source software.</p>
<p>It actually used to have a Flickr export plugin, but it hasn't been working for a while, and got dropped in recent versions. Which is totally fair because it's very much out of scope for a photo editing software. Having a generic solution like <code>nfp</code> makes much more sense because it can connect arbitrary file producers and consumers. It doesn't even have to be about images.</p>
<h2>Rust</h2>
<p>Since "inotify" sounds very "systems" and "core", I immediately took it as an opportunity to play with <a href="https://www.rust-lang.org/">Rust</a> once more. That was the main reason. A nice side effect of it is that it builds into a small self-contained binary which you can bring with you anywhere. As long as it's Linux, anyway :-)</p>
<p>If I had to mention a single gripe with the language during this last foray, that would be implementing an ordered type with the <code>PartialEq</code>/<code>Eq</code>/<code>PartialOrd</code>/<code>Ord</code> trait family. This just feels unnecessarily hard. I still don't get what's the point of having partial variants, and why things couldn't be inferred from each other. Like, even the <a href="https://doc.rust-lang.org/std/cmp/trait.Ord.html#how-can-i-implement-ord">official docs on <code>Ord</code></a> recommend writing a boilerplate for <code>PartialOrd</code> that just calls out to <code>Ord</code>. I'm sure there are Reasons™ for it, but somehow Python can infer <a href="https://docs.python.org/3/library/functools.html#functools.total_ordering">total ordering</a> from just <code>__eq__</code> and <code>__lt__</code>.</p>
<p><a name=debouncing></a></p>
<h2>Debouncing</h2>
<p>After using the tool for a week I noticed that the uploaded photos didn't have any metadata on them. After some digging this turned out to be due to the way Darktable writes exported files: it does it twice for every file. The second write, I assume, is specifically to add metadata to the already fully written JPEG. The problem was, <code>nfp</code> has been snatching the file away immediately after the first write.</p>
<p>The only way I know how to deal with this problem is "<em>debouncing</em>", a term familiar to programmers working with UI and hardware. Which means, adding a short grace period of waiting until a jittery signal stops appearing on the input or the user stops rapidly clicking a button. Or Darktable stops rapidly overwriting a file.</p>
<p>Quick search for a generic debouncer for Rust turned up only specific solutions tied to mpsc channels, or async streams, or hardware sensors. So <a href="https://nest.pijul.com/isagalaev/nfp:main/EPIHKWJDBRZLG.EQAAA">I wrote my own debounce</a>, which is a passive data structure with a couple of methods that doesn't want to know anything about where you get the data and what's the waiting mechanism. It just tracks time and removes duplicates.</p>
<p>I may yet turn it into a full-blown crate, and may be build a unixy-feeling debounce tool along the lines of:</p>
<pre><code>inotifywait -m -e close_write /path | debounce -t 500 | python upload.py
</code></pre>
<p class=note><small><b>Update:</b> this has been <a href="https://softwaremaniacs.org/blog/2021/10/25/debounce/en/">implemented</a>.</small></p>
<p>To do it properly though, I'll have to implement it as a two-threaded process, which will give me an opportunity to play with concurrency in Rust, something I haven't done yet. In <code>nfp</code> I cheated: it waits for new notifications on the same thread that sleeps for debounce timeouts, so it uses an ugly hack of sleeping in short chunks and constantly checking for new events:</p>
<pre><code>loop {
match debouncer.get() {
State::Empty => break,
State::Wait(_) => sleep(Duration::from_millis(50)),
State::Ready(file) => { ... }
}
for event in inotify.read_events(&mut buffer)? {
debouncer.put(...)
}
}
</code></pre>
<h2>Flickr uploader</h2>
<p>The uploader script was a story in itself. Ironically I spent more time trying to make various existing solutions work for me than I did with <code>nfp</code>, but didn't have any luck. So I ended up cobbling together a Python script using <a href="https://stuvel.eu/software/flickrapi/">flickrapi</a>.</p>
<p>The ugly part of all these scripts is <strong>OAuth</strong>. More precisely, its insistence on having to register a client app to get a unique id and secret (apart from the user auth for whoever is going to be using it). It's totally fine for a web service, but in anything distributed to user-owned general-purpose computers it means that a determined user can fish out the client credentials and use them for something else (oh horrors!) I remember dealing with this problem when we worked on an OAuth service for Yandex around 2009, and we didn't come up with a good solution for it. These days I believe client credentials should be optional, akin to the <code>User-Agent</code> header in HTTP, and shouldn't be used for anything outside of coarse statistical data.</p>
<p>Anyway… Since I'm using this script only for myself, I registered it on Flickr, put the credentials in a config and forgot about it :-)</p>
<p>Here's the whole script for posterity:</p>
<pre><code>import argparse
import logging
from pathlib import Path
import toml
import flickrapi
logging.basicConfig(level='INFO')
log = logging.getLogger()
def main():
parser = argparse.ArgumentParser()
parser.add_argument('filename', type=str)
args = parser.parse_args()
filename = Path(args.filename)
config = toml.load(open(Path(__file__).parent / 'flickr.toml'))
token = flickrapi.auth.FlickrAccessToken(**config['user'])
flickr = flickrapi.FlickrAPI(**{**config['app'], 'token': token})
log.info(f'Uploading {filename}...')
title = filename.name.split('.', 1)[0]
# TODO: use 'Xmp.darktable.colorlabels' to control visibility
result = flickr.upload(filename, title=title, is_public=0)
if result.attrib['stat'] != 'ok':
raise RuntimeError(result)
log.info(f'Successfully uploaded {filename}')
if __name__ == '__main__':
main()
</code></pre>
<p>P.S. I love dict destructuring with <code>**</code>!</p>
<h2>What's next</h2>
<p>I'm not yet sure what to do with <code>nfp</code>. The good sense tells me to extract a debouncer out of it and drop the rest in favor of <code>inotifywait</code>, but it actually does add some extra value: it has a sensible config format and I can modify it further into being able to exec multiple processor scripts in parallel. Although I suspect the latter part can be handled by yet another unix voodoo :-)</p>
<p>And its best feature is that it works for me right now!OAuth is not a protocol
2013-01-09T16:07:52.971000-08:00https://softwaremaniacs.org/blog/2012/07/30/oauth-is-not-a-protocol/en/Though this post is obviously triggered by the recent damnation of OAuth 2.0 by the (former) spec editor Eran Hammer, it's not directly related to it. These are my thoughts about the technical role of OAuth that I wanted to blog about a year ago but couldn't force myself to ...
<p>Though this post is obviously triggered by the recent <a href="http://hueniverse.com/2012/07/oauth-2-0-and-the-road-to-hell/">damnation of OAuth 2.0</a> by the (former) spec editor Eran Hammer, it's not directly related to it. These are my thoughts about the technical role of OAuth that I wanted to blog about a year ago but couldn't force myself to do it. But as of now, it seems surprisingly relevant!</p>
<p>My experience with OAuth comes from being a project manager of the <a href="http://api.yandex.com/oauth/">OAuth server at Yandex</a> where I took part in the design, closely watched the implementation (up to about draft 16 of the spec) and helped writing developer docs.</p>
<p><a name=more></a></p>
<h2>Too much freedom</h2>
<p>What always struck me as odd was the amount of details in the OAuth spec that are implementation dependent. Here are some from the top of my head that I still remember:</p>
<ul>
<li>Endpoint URLs are neither fixed nor is there any discovery mechanism for them.</li>
<li>Application registration is mentioned but doesn't have any properties upon which implementors could rely.</li>
<li>At one point the <code>client_secret</code> wasn't required leaving client authorization basically undefined too.</li>
<li>The authorization scope — both the set of permissions and their semantics — is completely service specific.</li>
<li>There are several ways to define a scope for a client: either during registration of an application or during an authorization request.</li>
<li>There are no guarantees about the rights implied by the access token: nothing prevents a service from limiting the authorization scope of a token based on meta data or, say, user's decision.</li>
</ul>
<p>Leaving those vast undefined grounds actually does make sense because real-world APIs differ immensely and creating a single detailed consistent authorization framework covering all their use cases is at the least impractical, if not impossible.</p>
<h2>Libraries</h2>
<p>All this means that it's impossible to devise such a thing as a "generic OAuth library" that could perform the complete authorization flow against an arbitrary API. As I understand it, current OAuth libraries provide service for encoding language-native data into GET requests and parsing JSON responses. The rest — designing all of the UI and setting up callback URLs — is still application specific. Actually those applications that talk to one particular API don't even need all that library code supporting different real-world flavors of OAuth and would do fine with plain HTTP and JSON libraries. The only thing that a dedicated OAuth library really buys them is the luxury of reading developer-oriented docs instead of a formal spec (which is no fun at all).</p>
<p>By the way, the very fact that so many different OAuths are still floating around in the wild is an indication that developers don't see the point of converging their code to match a single spec since most of said code will remain service-specific anyway.</p>
<h2>Not a protocol?</h2>
<p>The point of any Internet protocol is enabling interoperability between arbitrary parties without additional peer-to-peer agreements. Your HTTP client can (mostly) talk to any HTTP server on the Web without so much as even learning its name. Since it doesn't seem to be the case with OAuth, wouldn't it be better to treat it as a set of best practices instead of a protocol? An "architectural style" if you will…</p>
<p>And may be there should be really two different things:</p>
<ul>
<li>"WebAuth": a set of guidelines with all the different authorization flows and security considerations defined by current OAuth spec,</li>
<li>"OLogin": a protocol with concrete endpoints, common permissions and client identification intended for a use-case of "Login with Something" that seems to be one thing that many different APIs share.</li>
</ul>
<p>Does it make sense to anyone?Difference between OpenID and OAuth
2012-07-30T17:49:51.538000-07:00https://softwaremaniacs.org/blog/2011/07/14/openid-oauth-difference/en/Surprisingly there's much confusion between "OpenID" and "OAuth". People tend to use them as interchangeable terms or consider OAuth as a kind of a new and improved version of OpenID. In fact they are quite different things. I decided to write down my explanation to have a place to link ...
<p>Surprisingly there's much confusion between "OpenID" and "OAuth". People tend to use them as interchangeable terms or consider OAuth as a kind of a new and improved version of OpenID. In fact they are quite different things. I decided to write down my explanation to have a place to link to in the future.</p>
<p class=note><small>This post is <em>not about</em> which one of those is "better", or "evil", or "dead". Just bare facts.</small></p>
<p><a name=more></a></p>
<h2>Summary</h2>
<p class="picture right"><img src="/media/blog/openid-logo.png"></p>
<p><strong>OpenID</strong> allows a site to make sure that its user owns some personal URL (of a site, blog, profile). This fact is sufficient to use this unique URL to recognize the user next time he logs into the site. That's it. Everything else — registering accounts, obtaining emails and other data, permitting any activity on the site — is up to the site itself. In other words, OpenID is a pure <strong>authentication</strong> mechanism: you know who's come upon your site and you can do with this knowledge whatever you like.</p>
<p class="picture right"><img src="/media/blog/oauth-logo.png"></p>
<p><strong>OAuth</strong> allows a program (on the web or local) to obtain from a user certain rights to use a particular API. The rights are denoted with a token whose nature is not defined: it can be the same for different users or the same user can be granted different tokens at different times. The only thing guaranteed is that upon presenting the token to the service the program will be able to perform certain actions on it. In other words, OAuth is a pure <strong>authorization</strong> mechanism: you have certain rights but you can't infer their owner's identity.</p>
<p>Here's an analogy. OpenID is your driver's license: it says who you are but it doesn't imply you're going to drive a car whenever you're asked of your ID. OAuth is you car's key: a valet can drive your car with them but he doesn't need to know your name to do it.</p>
<h2>Signing into sites</h2>
<p class="picture right"><img src="/media/blog/twitter-facebook.png"></p>
<p>Given the summary above, OpenID seems to be the natural technology for implementing signing into sites with external accounts. And due to non-defined token properties such thing as "sign in with OAuth" shouldn't even be possible. However it's the OAuth that Twitter and Facebook use for their ubiquitous "Sign in" buttons. How's that?</p>
<p>Well, while both Twitter and Facebook use OAuth in their respective implementations they have absolutely nothing in common in everything else. The roles of protocols are as follows:</p>
<ul>
<li>OAuth provides sites with <strong>authorization</strong> to use Twitter/Facebook API</li>
<li>it's those APIs that provide then <strong>authentication</strong> since both of them give out some user identifying information</li>
</ul>
<p>Another analogy. "Sign in with OAuth" is a key from a car that happen to have owner's name on the insurance in the glove box.</p>
<p>It's important to understand that since authentication is done through proprietary API it's not a "sign in with OAuth" but rather a "sign in with Twitter/Facebook". One practical implication of this is that it's not possible to have a generic OAuth sign-in library that will work with any arbitrary site like Twitter and Facebook.</p>
<p>Unless…</p>
<h2>OpenID Connect</h2>
<p>Unless you standardize an open authentication API instead of relying on proprietary ones. Then the idea of a generalized OAuth-backed sign-in could work. And there was an attempt to do exactly that — <a href="http://openidconnect.com/">OpenID Connect</a>. It separates the core idea of OpenID — open authentication — from the protocol of exchanging keys replacing it with OAuth. It also defines accompanying things like discovery mechanism that OAuth lacks.</p>
<p>However OpenID Connect is not used anywhere in practice. And the reason to that is a completely different topic.