Hiring software engineers, that is…
I remember exactly one time in my career when I successfully passed a technical interview, in 2002, two years after my graduation. After that I failed every time I tried to do it "properly" and all my eventual employment came to be due to people already knowing me either by reference or by my public work. After having failed quite recently at another technical interview I quite naturally came to the conclusion that we have a problem in the industry: companies and people don't know how to hire engineers.
While it seems natural to myself, I understand it might not be obvious to everyone else what exactly makes me so sure that the problem is not, you know, with me (see, I do have a critical mind). Well… For the past twelve years or so I was developing business software for Windows, designing and implementing web services, writing tutorials (no proper books though), talking on international conferences, managing and shipping products, leading small and sizable teams, hiring engineers, teaching courses, maintaining open-source projects. And it's not solely my own opinion that many of those things were mostly successful. I won't bore you with my self-admiration any further but suffice it to say that I'm well past justifying my qualifications by the results of an hour-long interview.
Anyway, it's the words that matter, not the one who's talking. So here it is: what's wrong with hiring and how to fix it.
It seems that by now companies moved away from brain teasers of the "move-mount-fuji" sort and offer programming tasks instead. Good! However those are usually either basic CS testers or very abstract problems in graph theory, sets theory etc. It's nice to know if a candidate can revert a linked list and tell what big-O complexity the solution has. But it doesn't tell you anything about the skills that engineers use to do their job: understanding real world problems, evaluating trade-offs, finding and isolating bugs, optimizing performance, testing code, using tools, reading docs.
There's, in fact, a whole class of purely "interviewing" tasks. Like everyone's favorite where you're given a 3×4 field of cells and a marker in the left lower corner that can move up and right, and you have to count all the different routes to the upper right corner. Or anything involving Fibonacci numbers, possibly disguised as a robot walking up the stairs by one or two steps at a time…
Judging only be my own perceptions, I use completely different parts of my brain when solving these tasks and when working on anything practical. Practical tasks usually aren't limited by artificial conditions designed to make them "more interesting" or by a set of tools that you can use to solve them. They also often don't have a single correct answer. And I personally know a pretty accomplished programmer who periodically practices at interview question to not lose this skill. Heck, we even have whole books devoted to the subject!
A good engineering solution to the 3×4 field task is actually to hard-code the answer 10. Unless you have a variable-size field, in which case your classic straight-forward solution using recursion fails pretty soon depending on the abilities of your runtime. At which point you should really start asking what is it that we're trying to solve here to offer a useful solution.
I was once called to ask a couple of Python questions on an interview for a Java position where a candidate mentioned he also knew some Python. After a quick chat I learned he was still a university student, was interested also in Erlang and was working with his school mates on a web site for their department. I also asked a few questions that I wasn't expected him to be familiar with, just to roughly evaluate his level. I proceeded then by giving him the Paul Graham's accumulator problem (scroll to "Appendix: Power") which is not trivial to implement in Python and requires knowledge of a pretty obscure implementation detail in CPython.
He couldn't solve it at first but with a fair number of hints from me he finally was able to crack it. But what's interesting is that the interviewer from the Java team and I have come to completely different conclusions. He said, "He didn't answer any questions by himself, we won't hire him", while I said "Let me hire him on my team then!"
Of course I wasn't expecting a student to know useless stuff like this. I was already almost convinced that he was a worthy person by the point he told me about that web project he was doing with friends in his free time. What else do you want from a guy with zero experience except passion and the basic ability to do this sort of work? The only reason I gave him that task was to affirm that he can post hypotheses, test them and analyze results.
I eventually went ahead and hired him and as far as I know he still works happily at Yandex.
While we're at it, let me elaborate on that "implementing on the spot" thing…
Most interviews tend to be really tense conversations between a candidate and one or more employees. They are inherently tense because candidates are put into a stressful environment. People aren't generally comfortable with being evaluated, a trait which in programmers is further exaggerated by the fact that most of us are introverts. But what's really bad is that candidates are compelled to solve tasks as fast as possible. Even if they know that they have more than enough time overall they still expect that faster performance will be rewarded. Which is, incidentally, the biggest hindrance to any cognitive work (you probably saw that video already, but go watch it again).
On my recent interview I was able to unfreeze myself and come up with a solution to a pretty simple task almost immediately after I left the room for a 5 minute restroom break. You just can't fake this effect.
And from the opposite point of view, I once had a pretty smart candidate sending me two interesting variants of the solution to a task that he wasn't able to crack half an hour before while sitting in the room with me. All it took was getting home and relaxing.
The main point that I'll try to convey here is this: asking right questions is only a half of a good interview story, you also need to interpret answers properly. I would even go as far as to say that the interviewer has more responsibility for uncovering candidate's abilities than the latter for showing them. Remember, you don't hire a sales person, don't expect a tech geek to look "impressive".
On to specifics.
As Joel Spolsky says in his Guerrilla Guide to Interviewing you should start by giving an introduction about the company and yourself and then asking about candidate's recent projects. It's all true and it really works for all the reasons that Joel has so eloquently spelled out. Also, while you can't alleviate the interview stress completely you can still reduce it this way.
Your actual technical question should be a simplified real-world task. It should be practical for all the reasons stated above and it should be simplified because working on minute details takes too much time and you don't have a whole week for this interview.
- You're given a plain text log of IP addresses with the amount of bytes downloaded by them. The lines have a simple format of
xxx.xxx.xxx.xxx \t <number>.
- The size of the log is several gigabytes, big enough not to fit comfortably in memory.
- You also have a table of IP address ranges assigned to different countries. The lines have a format
<integer>-<integer> <country code>. Integers represent IP addresses.
- The size of the table is about 100 000 lines.
- You need to find a top-10 of countries downloading the most content.
It is simple but, surprisingly, it has enough corners and cul-de-sacs that can tell you quite a few things about the candidate.
After giving a task you should leave the room for 15 minutes or so. Seriously, don't sit on their head while they're trying to think. Let them close their eyes, gnaw at the pen, walk violently back and forth or in circles, bury their head under the table, sing aloud the problem's description or whatever it is that smart people do that helps them concentrate.
I knew a guy who literally sang the source code aloud while debugging it. It seemed to work well, so why not?
What you should also make clear, is that you don't expect them to have a complete solution by the time you're back. Tell them explicitly that there's no single correct answer to the task and you're more interested in the general approach to solving it. Not requiring the code to compile also makes it irrelevant if they use sheets of paper, a whiteboard or a laptop to do the coding. Use whatever you have in the room.
And if they do solve the task without any trouble you shouldn't be happy at all. Because it doesn't tell you much. May be you've got a really smart person here, or may be they just got lucky and the solution just "came up", or may be they have a particular tooth for exactly this kind of tasks. But since the engineer's job is solving problems that they don't know yet how to solve, your job as an interviewer is to put them into this state and see how they'll work their way out of it.
During this process it's okay to have a discussion. This is how problems are solved in practice. The discussion gives you a good insight into the person's reasoning process. Do they draw conclusions or randomly guess different answers hoping you'll pick the one for them? Do they dwell on insignificant details or can force themselves to reevaluate the task from a higher perspective? Do they ask productive questions? Can they abstract away already solved parts? This is the most productive part of the interview that should really give you an idea about the person.
It is also important to remember that engineering skills can't be arranged on a linear scale and you are not looking for a single cumulative "score" that puts a candidate left or right of some "hire/don't hire" point. People have different inclinations and different experience and you shouldn't expect everyone to have on the top of their head exactly the same things you do and probably consider "essential" and "must-know". In practical words, don't write people off just because they misjudge the big-O complexity of an algorithm, don't know a certain sorting method or don't remember the latency of a spinning hard drive in milliseconds. It only becomes suspicious if they haven't heard about any of such things at all.
One special note about hiring seniors. Don't bore them too much with solving your stupid tasks! A short warm-up task is fine but you'll get much more information about them by talking about their own experience. Simply because they usually have more of that than of pure "mad skillz" and textbook knowledge. This is, after all, what's valuable in them. Get down to details of their past projects, try to argue with them, play devil's advocate to force them into educating you about the decisions they made. This way you'll be able to see if they indeed took part in the work or were just hanging around. In the best case you might learn something useful for yourself :-).
Also accept that you will hire wrong people from time to time. It's just impossible to correctly evaluate a long-term performance during an interview. A person might be smart but may turn out insufferably lazy in a couple of months. This should be dealt with by other means but this is a whole another story.
This technically belongs to the "what's wrong" section but I saved it for dessert because it feels the most controversial.
I strongly believe that evaluating other people's performance — judging them, if you will — is a very particular set of skills. A good engineer is not necessarily a good interviewer. This job is also creative and you can't just formalize an interview process and let every engineer learn it. If you ask a good interviewer how they do it it's not uncommon to hear something like "I just talk to people and see if they're good or not". And this is it.
So instead of making every engineer on your team conduct interviews in turns as a chore (looking at you, Google) try to pick up those people whose hires consistently turn out to be good employees and let them do it.
This post was inspired by the hilarious piece "How I hire writers" by Hitesh Sarda. Specifically, the second-to-last paragraph in it.