"Checking in" on the Coding Exercise

Earlier this year I wrote about how I was setting out to use a code exercise as part of the interview process for a software engineering position. Six weeks or so later, I haven't yet snagged my ideal candidate, but I sure as heck am sold on using this approach. I think I can safely say I'm never going to recruit without using an exercise like this in the future.

Here are my seven "lessons learned" from this approach to evaluating candidates. I had previously learned all of these lessons anyway, but it never hurts to have a refresher course...


1. Some people can "talk the talk," but not "walk the walk"

I'm looking to hire a "Principal Engineer" -- that may have some scope for interpretation when it comes to a precise meaning, but I'm fairly sure that by anybody's reasonable definition it includes the ability to write decent code. Quite frankly, to me, it clearly implies a software engineer at the top of their game. That's not to suggest that I didn't receive candidates that *sounded* like they were awesome...

But there was the thing...there is a surprisingly large number of people out there looking for a job at this level who aren't very good engineers. Some of these I kind of guessed might not be so hot from the initial phone screen. They fell into my "underwhelming" category, aka "didn't blow me away but I've no real concrete reason not to let them have a crack at the code exercise." But the other contingent here were those that phone screened like pros, that left me thinking "that's the one!" after I got off the phone with them. Sadly, quite a number of these turned in surprisingly mediocre work. (At this point may I offer my apologies to any former candidate reading this if I hurt your feelings. Sorry about that.) One person offered a solution that reminded me of when I first used to write code...around age 11. 'Nuff said.

Lesson learned: don't get carried away just because someone's a smooth talker and sounds impressive on the phone.


2. Some people can't read

I thought the exercise was pretty clearly worded. It wasn't too long or complex or ambiguous. I know some elements were not super-specific, but that was purposeful -- requirements often aren't clear in the business world anyway. The key aspects though were there. Or so I thought.

Moreover, I'm not too sure how you decide to undertake an exercise like this when looking for a job and don't pay really close attention to what you're being asked for. Maybe not doing so exposes that you're not a detail oriented person. That's certainly how my interview panel and I interpreted things when somebody sent us a "solution" that was completely different to what was required. As a reminder, the exercise requires an algorithmic exploration of a "maze" kind of environment. What we received was an interactive text adventure game. Which was cute and all, and it was clear that quite some effort had been put into it...to the extent that my initial reaction was that I wanted to cut the guy some slack. I was interested in having him come in for a face-to-face interview and finding out what happened. However, as my colleagues on the interview panel pointed out, submitting such a significantly wrong solution really did not bode well. If someone can't pay enough attention to get things right for a job interview, what hope is there gonna be when they have their feet under the desk?

Lesson learned: if it's wrong, it's wrong. Just because someone appears to have put a lot of effort into something doesn't mean they get away with producing completely the wrong deliverable.


3. Some people get stuck off in the weeds

This third point, moreso than any of the others, is likely to give prospective candidates some good advice. Lucky you huh?

I've seen this trait in engineers before, so it's perhaps not surprising to see it rear it's head amongst the candidates. I still don't quite understand it myself... Given that you don't want to spend forever on something like this that *might* help you land a new job, why oh why would you spend your time on things that are of low or no importance? At it's core the exercise requires you to parse an XML document, read in a text file and figure out some kind of algorithm to make your way through a maze looking for objects. Why then would you spend time on patently unimportant aspects or worry about odd or implausible corner cases? Some people think they're being "thorough" and there are times when that is appropriate. But one has to keep things in perspective...

Lesson learned: watch out for this behavior in candidates...because in someone on your actual team this can lead to a lot of waste from "over-engineering" to running around blind alleys (or mazes ;-).


4. Some people have surprisingly little pride in their work

Maybe some of what I interpret as lack of pride is more down to chaotic work practices, I'm not sure. Nonetheless, would you submit to a potential future employer code that was a mess as an example of your work? I'm talking about large swathes of code commented out, duplicate files, irrelevant files, stuff you've tried and abandoned etc. I wouldn't for sure...not exactly going to create a good first impression. Now I know I say to people "I'm not looking for a perfect solution" -- but I still would expect a certain degree of professionalism in what I do receive...

Lesson learned: another characteristic to watch out for in candidates. If this is what they show at an interview then goodness only knows what kind of stuff is going to come up when they're comfortably settled in their job.


5. Some people are incredibly slow

Perhaps this one shouldn't shock me. It's been known for decades that the best programmers are often an order of magnitude more productive than the worst.

Lesson learned: make sure you check how long someone spends on an exercise if you attempt something like this. Either that or provide clear instructions that they should only spend X amount of time (and even then check...) It's very telling.


6. Some people will give you whatever answer they think you want to hear

A gentlemen who shall remain nameless did a 180 on test-driven design during his interview because he thought we wanted to hear that this was the "One True Way" to approach software development. At this level of the game I'm expecting you to have well established and thoughtfully considered views on how to develop software along with an open mind that recognizes there is always room for learning something new. What I don't expect is you to roll over at the first challenge and alter your world view.

Lesson learned: yet another dubious behavior to guard against, at least in a senior position.


7. some people are awesome

I know it's been some rather grim and to some extent heavily laced with sarcasm and cheap shots at other peoples expense up to this point. But there is some good stuff too. There are some awesome people out there. I know this, I've interviewed them. Sadly they've not necessarily been the right fit or they've ended up going to other employers. However, they are out there. To at least see them during the recruitment process is a welcome reminder of their existence and of the fact that you've got to sort the wheat from the chaff if you ever want to bake some kick-ass bread.


And finally...
A few other interesting things that emerged. Out of many candidates so far only one has declined to undertake the coding exercise. Almost everyone has said they had a great deal of fun doing it, and a couple of said they're impressed that we take finding good candidates so seriously. A shocking number of people talk about TDD in their phone screen but then don't offer a solution with any unit tests ("well I didn't actually have time for that...")

Oh, and it's interesting how many people opt for the "stumbling drunk" algorithm whereby one wanders randomly around the maze looking for objects...