Hiring: you're doing it wrong

Hiring software engineers, that is…

I remember exactly one time in my career when I successfully passed a technical interview, in 2002, two years after my graduation. After that I failed every time I tried to do it "properly" and all my eventual employment came to be due to people already knowing me either by reference or by my public work. After having failed quite recently at another technical interview I quite naturally came to the conclusion that we have a problem in the industry: companies and people don't know how to hire engineers.

While it seems natural to myself, I understand it might not be obvious to everyone else what exactly makes me so sure that the problem is not, you know, with me (see, I do have a critical mind). Well… For the past twelve years or so I was developing business software for Windows, designing and implementing web services, writing tutorials (no proper books though), talking on international conferences, managing and shipping products, leading small and sizable teams, hiring engineers, teaching courses, maintaining open-source projects. And it's not solely my own opinion that many of those things were mostly successful. I won't bore you with my self-admiration any further but suffice it to say that I'm well past justifying my qualifications by the results of an hour-long interview.

Anyway, it's the words that matter, not the one who's talking. So here it is: what's wrong with hiring and how to fix it.

What's wrong

It seems that by now companies moved away from brain teasers of the "move-mount-fuji" sort and offer programming tasks instead. Good! However those are usually either basic CS testers or very abstract problems in graph theory, sets theory etc. It's nice to know if a candidate can revert a linked list and tell what big-O complexity the solution has. But it doesn't tell you anything about the skills that engineers use to do their job: understanding real world problems, evaluating trade-offs, finding and isolating bugs, optimizing performance, testing code, using tools, reading docs.

There's, in fact, a whole class of purely "interviewing" tasks. Like everyone's favorite where you're given a 3×4 field of cells and a marker in the left lower corner that can move up and right, and you have to count all the different routes to the upper right corner. Or anything involving Fibonacci numbers, possibly disguised as a robot walking up the stairs by one or two steps at a time…

Judging only be my own perceptions, I use completely different parts of my brain when solving these tasks and when working on anything practical. Practical tasks usually aren't limited by artificial conditions designed to make them "more interesting" or by a set of tools that you can use to solve them. They also often don't have a single correct answer. And I personally know a pretty accomplished programmer who periodically practices at interview question to not lose this skill. Heck, we even have whole books devoted to the subject!

A good engineering solution to the 3×4 field task is actually to hard-code the answer 10. Unless you have a variable-size field, in which case your classic straight-forward solution using recursion fails pretty soon depending on the abilities of your runtime. At which point you should really start asking what is it that we're trying to solve here to offer a useful solution.

A Google engineer once told me that there believed to be a correlation between people who's good at solving those tasks and people who can be good engineers. May be there is. But it doesn't say anything about those good engineers that you don't hire using this filter. Most of other companies simply don't have the luxury to lose talent like this (and I doubt even Google can afford it). It also seems to ignore the fact that there's no such thing as a generic "software engineer" position. Different jobs require both different level of qualification and different specialization. It's no use expecting a junior to understand business vs. engineering trade-offs as well as asking a senior to implement the infix-to-prefix conversion on the spot. Same goes for different fields: game devs generally aren't very familiar with exceptions and most JavaScript devs aren't much into memory management.

While we're at it, let me elaborate on that "implementing on the spot" thing…

Most interviews tend to be really tense conversations between a candidate and one or more employees. They are inherently tense because candidates are put into a stressful environment. People aren't generally comfortable with being evaluated, a trait which in programmers is further exaggerated by the fact that most of us are introverts. But what's really bad is that candidates are compelled to solve tasks as fast as possible. Even if they know that they have more than enough time overall they still expect that faster performance will be rewarded. Which is, incidentally, the biggest hindrance to any cognitive work (you probably saw that video already, but go watch it again).

How to do it

The main point that I'll try to convey here is this: asking right questions is only a half of a good interview story, you also need to interpret answers properly. I would even go as far as to say that the interviewer has more responsibility for uncovering candidate's abilities than the latter for showing them. Remember, you don't hire a sales person, don't expect a tech geek to look "impressive".

On to specifics.

As Joel Spolsky says in his Guerrilla Guide to Interviewing you should start by giving an introduction about the company and yourself and then asking about candidate's recent projects. It's all true and it really works for all the reasons that Joel has so eloquently spelled out. Also, while you can't alleviate the interview stress completely you can still reduce it this way.

Your actual technical question should be a simplified real-world task. It should be practical for all the reasons stated above and it should be simplified because working on minute details takes too much time and you don't have a whole week for this interview.

To give you an example, here's one of my long time favorite tasks:

• You're given a plain text log of IP addresses with the amount of bytes downloaded by them. The lines have a simple format of xxx.xxx.xxx.xxx \t <number>.
• The size of the log is several gigabytes, big enough not to fit comfortably in memory.
• You also have a table of IP address ranges assigned to different countries. The lines have a format <integer>-<integer> <country code>. Integers represent IP addresses.
• The size of the table is about 100 000 lines.
• You need to find a top-10 of countries downloading the most content.

It is simple but, surprisingly, it has enough corners and cul-de-sacs that can tell you quite a few things about the candidate.

After giving a task you should leave the room for 15 minutes or so. Seriously, don't sit on their head while they're trying to think. Let them close their eyes, gnaw at the pen, walk violently back and forth or in circles, bury their head under the table, sing aloud the problem's description or whatever it is that smart people do that helps them concentrate.

I knew a guy who literally sang the source code aloud while debugging it. It seemed to work well, so why not?

What you should also make clear, is that you don't expect them to have a complete solution by the time you're back. Tell them explicitly that there's no single correct answer to the task and you're more interested in the general approach to solving it. Not requiring the code to compile also makes it irrelevant if they use sheets of paper, a whiteboard or a laptop to do the coding. Use whatever you have in the room.

And if they do solve the task without any trouble you shouldn't be happy at all. Because it doesn't tell you much. May be you've got a really smart person here, or may be they just got lucky and the solution just "came up", or may be they have a particular tooth for exactly this kind of tasks. But since the engineer's job is solving problems that they don't know yet how to solve, your job as an interviewer is to put them into this state and see how they'll work their way out of it.

During this process it's okay to have a discussion. This is how problems are solved in practice. The discussion gives you a good insight into the person's reasoning process. Do they draw conclusions or randomly guess different answers hoping you'll pick the one for them? Do they dwell on insignificant details or can force themselves to reevaluate the task from a higher perspective? Do they ask productive questions? Can they abstract away already solved parts? This is the most productive part of the interview that should really give you an idea about the person.

It is also important to remember that engineering skills can't be arranged on a linear scale and you are not looking for a single cumulative "score" that puts a candidate left or right of some "hire/don't hire" point. People have different inclinations and different experience and you shouldn't expect everyone to have on the top of their head exactly the same things you do and probably consider "essential" and "must-know". In practical words, don't write people off just because they misjudge the big-O complexity of an algorithm, don't know a certain sorting method or don't remember the latency of a spinning hard drive in milliseconds. It only becomes suspicious if they haven't heard about any of such things at all.

One special note about hiring seniors. Don't bore them too much with solving your stupid tasks! A short warm-up task is fine but you'll get much more information about them by talking about their own experience. Simply because they usually have more of that than of pure "mad skillz" and textbook knowledge. This is, after all, what's valuable in them. Get down to details of their past projects, try to argue with them, play devil's advocate to force them into educating you about the decisions they made. This way you'll be able to see if they indeed took part in the work or were just hanging around. In the best case you might learn something useful for yourself :-).

Also accept that you will hire wrong people from time to time. It's just impossible to correctly evaluate a long-term performance during an interview. A person might be smart but may turn out insufferably lazy in a couple of months. This should be dealt with by other means but this is a whole another story.

Who can do it

This technically belongs to the "what's wrong" section but I saved it for dessert because it feels the most controversial.

I strongly believe that evaluating other people's performance — judging them, if you will — is a very particular set of skills. A good engineer is not necessarily a good interviewer. This job is also creative and you can't just formalize an interview process and let every engineer learn it. If you ask a good interviewer how they do it it's not uncommon to hear something like "I just talk to people and see if they're good or not". And this is it.

So instead of making every engineer on your team conduct interviews in turns as a chore (looking at you, Google) try to pick up those people whose hires consistently turn out to be good employees and let them do it.

Credits

This post was inspired by the hilarious piece "How I hire writers" by Hitesh Sarda. Specifically, the second-to-last paragraph in it.