The config

This story is about an accomplishment at work. Due to it being work-specific I won't go into too much detail, so it won't be necessarily useful. It's just a personal piece. For various reasons (such as changing places, raising a child, workplace politics, poor luck) I haven't got any clear wins under my belt for quite some time, probably since building a Python team at Yandex 15 years ago. So this one feels a little special.

Inception

A couple of years ago our department — we're doing billing at Datadog — faced a point at which individual teams accumulated a lot of hard-coded, duplicate, undocumented, constantly diverging configuration about products. My team had it especially bad, as we had to maintain essentially identical pieces of config in two different languages across two different code repositories, and synchronize bits of it to yet another one.

So on the next department-wide gathering I volunteered to create a prototype of a solution in the form of a shared config service. The idea wasn't controversial, and there was no details yet to object to, so it became the moment where I put my stake in the ground.

Progress

It went way slower than I expected (and I thought about my initial estimates as conservative ones!) Sure, the prototype within our team took shape pretty quickly, and soon significantly simplified making ongoing changes to products. It was convincing other teams to bite the bullet and switch their code to the config that took most of the effort. Because by its nature, such a change makes things harder at first: you suddenly need to make network requests instead of simply relying on hard-coded data, and you now need to think about backwards compatibility of your data consumed by others. It's only after a while you (or not even you) will start profiting from reduced duplication and simplified processes due to data being available from a single source of truth.

But we got there. We got Product people on board, we acquired a manager, we started producing OKRs, we involved more engineers… And while there wasn't a clear moment of a "launch", on our latest department summit I realized people from various teams were mentioning "the config" not just as a nice idea, but as something definitely existing, and I was in the middle of all those conversations. It felt like a pivotal moment, and it was a warm, fuzzy feeling :-)

What worked

• I offered my idea in a very rough form, almost on a whim, as I wasn't even sure if I should may be first create a prototype before talking about it. I didn't have any slides or a prepared talk, I just shared my text editor on a screen during the "unconference" part and drafted a few ideas right there. Not many people were present, but it gave me an owning reference point for the effort.

• I named the project with a generic name claiming it as being a solution for the whole department. Not a fancy name, not a team-specific name, not a "draft" name. It's hard to prove how much it helped, but I believe it did.

• Finding early stakeholders and talking about their problems in their language instead of focusing on technology: managers want to streamline communications, product wants visibility, engineers want less churn in their code, bosses want to see a strategic vision.

• Making personal connections. Not only through docs, chat and zoom, but actual face-to-face meetings. When you win people over and they have trust in your project, they start inadvertently helping by mentioning it in meetings and tying their plans to its success. I saw this happening on a few occasions: an almost dead conversation suddenly turned around by someone credible saying "actually, we're going to use it in such and such way."

• Dumb architecture. None of the above would matter if I couldn't deliver a working prototype and quickly iterate it into something dependable. Since at first I was the only engineer/architect on the project, my approach was to use boring, dumb technology. Use static files instead of a database, serve them over HTTP via nginx, drop some plain Python modules to do validation at build time, put a number in a file name as a "versioning strategy", run deploys on a schedule instead of requiring teams to implement hooks, etc. Almost all of it is going to change over time, of course. But:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work — John Gall.

• Having a live spec. A single document with a high-level overview of the system's current state. Way better than relying on updating people via random meetings or producing a new half-complete document for every separate occasion. It does take effort to maintain, but it's worth it. Thank you Mr. Spolsky for teaching me this a long time ago.

And that's the story!