How to Create a Code Challenge to Evaluate a Developer’s Skills

Guest post by Andy Davis - super talented software engineer, guest blogger and awesome code challenge creator!

Recently I’ve been talking about coding challenges a lot. What’s makes for a good one? What makes for a bad one? How to make one? And even, how to create a code challenge.

Plenty of people out there are asked by their hiring team to come up with a challenge that will test candidates and find them their perfect matches.

If only it were that simple…

Since you are reading this, you have probably created a code challenge in the past, or maybe even need to make one now. So let’s spend a bit of time on it. It doesn’t have to be complicated and you don’t have to grab the first one you find online.

You shouldn’t rush it and you don’t need to create the best one in the world. Here’s why…

No one gets hired off the back of one coding challenge solution.

That said, it is usually the first opportunity to see what someone is like when actually writing code. So it is still totally worth the investment. A reliable developer hiring process shouldn’t only be about talking.

So, you need to make one, how should you do it?

Want to license a code challenge to analyse the REAL skills of your candidates? TRY FOR FREE

Set Goals for Your Code Challenge

Firstly, decide the goals behind your test before you create a code challenge. Goals are the easiest and most obvious way you can assess candidates quicker and more fairly. You’d be surprised at how many challenges out there are just Fizz Buzz - has any thought really been put into what they are looking for? or did they just google and use the first thing they found?

For example, do you want people who test drive their code, do you want hardcore optimisation specialists, do you just want people who can get the job done as fast as possible.

Once you have the goals, it is much easier to design a challenge.

I’m going to design a challenge, to show you what I mean….

But before I do… I want to leave you with one last suggestion to keep in mind when you make your challenge.

You should be trying to separate the wheat from the chaff when you create a code challenge, not the wheat from the wheat. What I mean by this is, filter out the people who can’t write code. Don’t filter at this stage by coding style or architectural approach. Those are the things that carry the value later in your hiring process when you are actually in the same room and talking to each other.

Without further ado, here is my attempt at creating a code challenge from scratch.

I wanted to pick something topical, so I scoured hacker news to see what a common topic was and if I could come up with something relevant.

I settled on Autonomous Vehicles (or Self Driving Cars).

Not only was it popular, it seemed kind of fun too. I wondered if I could code one up. Or better still, put together a part built framework/api and let the candidates come up with something.

After a few iterations and feedbacks, I had something quite useful in assessing candidates and working out which bucket they fall into. I say ‘bucket’ for a reason. Despite all companies thinking they only hire the best; I believe developers aren’t actually good or bad. There are no ‘best’ Developers. They just fall into different categories or ‘buckets’.

The best for one company is not the best for another. A junior, fresh into the industry, with no commercial experience are not bad developers for example.

OK, ok; there are some bad developers out there. We all know who there are, but someone hires them don’t they ? Let’s move on :)

TL;DR - If you think you are up to the challenge, please try out the Self Driving Car Challenge for yourself right now - We have C Sharp and Java versions for now, but we’ll add more if there is demand. Simply send an email to hello@geektastic.com stating whether you would like the C Sharp or Java version.

And now comes the science…. This is my detailed thinking and the process behind how I created this code challenge. The challenge is still a work in progress, but that is how I think all challenges should be and here is the detail and the journey of how I came up with it.

Want to test your coding skills and take a peer reviewed Geektastic code challenge? TRY FOR FREE

Where Do I Start?

Should I just google “coding challenges” and pick one at random, and maybe tweak it a little bit?

Here’s the snag…

Coding challenges are much more popular now, it’s getting more and more common to find:

1. Candidates already know the most popular ones.
2. Candidates are too busy to do your challenge.

Candidates Already Know the Most Popular Code Challenges

Let’s address the point A. As long as we twist it a little, it won’t matter if they already know it.

As long as the challenge isn’t 100% like for like with the one chosen from online, grabbing code from Google and Stack Overflow is totally acceptable. It will still show how the candidate likes to write code. After all in production code, a good developer rarely copies code without tweaking it to fit their needs.

Candidates are Too Busy for Your Code Challenge

On point B, if they are too busy to do it, that means we don’t want to hire them anyway, right?

Well… Not exactly! Too busy can be a good thing if it means they are in demand. After all, a good developer won’t necessarily be looking for their next role right now. However, they might be curious - we don’t want to quash that curiosity.

We want developers to actually do our challenge! So we need to make the bar of entry low enough keep the good guys interested despite how busy they are.

That’s not to say the challenge should be easy!

Testing Knowledge & Skills You Value

We are knowledge workers. There must be some knowledge that the good guys know, and the bad guys don’t? Something that comes with experience. Like a series of keys that get unlocked with experience?

The problem is that different hirers have different keys. For example, when I set challenges, all I really want to know is if they have the right type of attitude and can collaborate effectively with the team I’m hiring for. I favour TDD over Test after. I favour clean code and SOLID principles over other schools of thought. That’s not to say other schools of thought are wrong, it’s just not the way that I have found (over the years) to be the most efficient.

Let’s turn our attention to food for a second (because I’m hungry :)).

Do you know MasterChef The Professionals? I see the code challenge as the equivalent of a Monica’s or Marcus’ “Skills test”.

For those who don’t know, MasterChef is a cooking competition and “The Professionals” version of the show is for people who already work professionally as chefs, this is not amateurs. You see, it’s just like us - we’re professionals too.

In the skills test, a specific skill is put under test for around 15 minutes. It might be making a posh omelette, it might be filleting a whole salmon. The contestants don’t know in advance.

The skills test is rapid. Contestants mess up all the time, the nerves and the pressure makes them do silly things, but that is OK, the judges are looking for seeds of potential, not the finished article.

So if we think about this code challenge as a quick skills test like MasterChef, it puts it into context. Submissions don’t need to be perfect. Which means the test itself doesn’t need to be super hard in order to challenge them and separate them from each other.

So what am I looking for in a “Skills test”? Personally, I want to see:

• Test Driven Development
• Domain Driven Design (or an object model)
• Design Patterns
• Clean Code
• Extensible code

Remember: this list might not be the same for where you are hiring.

I also want to make it REALLY easy for a good candidate to breeze through.

Therefore I want the finished code to be:

• Fairly short to type out once a solution presents itself.
• Using a known design pattern.
• Fully unit tested (Ideally TDD).
• Using an object model.

Pro Tip: I want working code, one way of proving to yourself (and me) that it works is to have good tests around the code. I might not ask for tests in the challenge, but unless you know a better way of proving your code works, write tests.

Now we’re getting somewhere… To get cracking with my design of a new challenge, I need to:

• Pick an obvious domain people are aware of but haven’t necessarily worked in
• Pick a Gang of Four pattern

What? Pick a pattern? Yes, I will pick a pattern first and design the challenge after. One ideal solution would use that pattern - however, there might be even better solutions, only time will tell.

NOTE: If I was hiring for an online casino then my domain would be casino related. If I am writing a general challenge that is more about testing their skill in technology, I would try to pick something that everyone has heard of in order to avoid domain confusion.

I must keep reminding myself : How can I make it as easy as possible for the good developers to pass my challenge?

Clean code and TDD are cross cutting concerns in this context, I always want to see this. So I won’t make any template code that is full of hard dependencies as this would be unfair to expect them not to follow suit. Even though great developers would fix my poor template, this would be a risk so I will avoid that.

For extensibility, I’m also keeping in mind that I want something that can easily have “Feature requests” later - to give the candidate a chance to write code that lends itself to being open for extension (by the candidate predicting obvious next feature request and leaving the code open to that).

Pro Tip: Don’t actually implement that prediction (remember YAGNI).

Choosing a Code Challenge Idea

Here are some code challenge idea options:

• Shopping Cart - done to death.
• Mars Rover - a bit rarer but most good developers know it (not such a bad thing).
• A Phone Bill - nice little domain that can be modelled multiple ways, gets messy and long winded if parsing the file needed.
• Bowling Kata - takes far, far too long to do it well and solutions easy to download online.

Any of those can work when creating a code challenge if we tweak them to be short enough and focused enough. However that just seems too easy.

Let’s go wild and pick something topical at the moment, a quick scan of hacker news tells me that Self Driving Cars is pretty topical - e.g. George Hotz, an ex-Google developer just open sourced his project and there is of course Tesla who make the news almost daily at the moment.

The big players seem to think this is the future. A future they saw in Total Recall perhaps?

First shot at the description:

Your Luxury Limousine company has purchased its first Self Driving Car. The car has radar, cameras, GPS, and a computer onboard and accepts simple commands that control all movements of the car including steering, acceleration, braking etc.

Your car has an onboard assistant (that needs writing) called ALFIE.

The passenger in the car only needs to say a location and the car should drive itself there.

Right now these options are A and B.

Your first passenger gets in the car. Requests to go to Destination A. When she gets there, she changes her mind and realises she wants to go to Destination B. ALFIE takes her to destination A, then to destination B.

Background Information

From HQ….Destination A is NORTH (4 blocks).

From HQ….Destination B is NORTH (2 blocks), then EAST (2 blocks), then, NORTH (2 blocks).

Right now there are only 4 commands that work.

FORWARD (this moves the car forward 1 block).

TURN-LEFT (this turns the car 90 degrees left).

TURN-RIGHT (this turns the car 90 degrees right).

STOP.

At HQ the car automatically turns around and faces NORTH.

NOTE: This is an early prototype of the car and the manufacturer is releasing new available commands every week.

Assumptions:

You are only writing ALFIE.

You do not need to deal with voice control.

When the destination of A or B is chosen, the car sets off.

The car needs a new command for each block.

You can queue up these commands in one go.

The car will keep driving unless it has an explicit stop command.

The car will automatically turn and face the opposite way if you send one of the turn commands when.

Stopped.

There, that wasn’t too hard.

Does it satisfy my criteria?

1. Test Driven Development - Hmm, I’ve accidentally made it a closed system - I need to think about something that can be checked in order to prove the Car did its job
2. Domain Driven Design - Clearly there is a domain here - ALFIE, CAR, DESTINATION are some obvious entities
3. Clean Code - You can always make code clean in some way or another - it normally starts with following SOLID principles - it would be wise to invert the dependency to Car commands for starters.
4. Design Patterns - There are obvious design pattern choices here that will either be used or completely missed causing OCP violation
5. Easy for a Good Developer - It might be starting to get a little complicated - I’ll have to do a trial run at implementing it to see how long it takes me to come up with a solution. I’ll send it to some of my peers to try out.

Measuring Success

We need to be able to measure success with our code challenge. Perhaps some logging? That sounds too vague. I think that something in the domain would be better. What would tell us in the real world we are at the destination or where we are? GPS! Yes GPS is the answer… I need to add GPS to the question for verification purposes

“As the car moves, the GPS should update its current position. For simplicity, the GPS can be an [X,Y] coordinate.”

Good - that means some tests can be put in place too, which is good.

Next, find some guinea pigs…

Rick, the CEO for Geektastic, kindly put me in touch with 3 good developers who agreed to take the challenge.

How The First Round of Developers Performed

Firstly, thank you to all 3 of you for taking the code challenge. There is no job at the end of it, sorry, just this article instead. However, having now seen your code… I think I might just keep you in mind for the future.

The solutions submitted were all good in their own ways - but it was tackled in 3 different ways. One was quite familiar to me and quite similar to the way I code, another was a way I didn’t think of doing it and the 3rd really focused in on the algorithm. Although the pool is small, getting 3 different solutions is good news to me. It means the challenge can be used in different contexts to find different types of developers (all good ones of course).

The overwhelming feedback from the 1st candidate was that the challenge was decent. There was enough things to code and enough decisions to make about it and he was able to demonstrate how he writes code. However he missed the deadline - so, clearly, not enough time, fair enough, I only allowed him 30 minutes (plus Geektastic allows a little grace, which is good, human-to-human process after all). I sent it to him again and got it back within the hour. So in reality he had about an hour and a half. Not bad.

We decided that we need to increase the time limit as the challenge was OK for him but just not enough time. He also didn’t realise the API was a library reference (no source code), so said I should mention this as it was a little confusing. Consider it done :)

Changes made:

1. Increase to 2 hour time limit, more than enough. I think it can be done in about an hour, so 2 hours is a good cut off. I want people to have time to think, after all that is what we are really hiring, thinkers.
2. Make it clear that there is a compiled code library simulating the code and an API from a car manufacturer and that this code is not something the user can edit.
   1. Pro tip: Dependency Inversion Principle applies here, abstract it in case it changes.

The Second Attempt

Now we’re ready for the other 2 candidates to try. Let’s call them A and B.

This time, the time limit was no problem - fantastic. 2 hours is enough. It isn’t originally what I had in mind, as I think 2 hours can be off-putting. Since I think this created code challenge can be done in an hour or so, let’s stick with 2 hours.

Candidate A’s solution was very algorithm centric. Candidate B’s used a very basic algorithm but focused more on the coding style and expressiveness of the code. Both are 100% acceptable solutions in the right context. It really just depends what the client wants - i.e. these 2 developers fall into 2 different categories, but they are both good.

One of them (I won’t say which) gave feedback that they really didn’t know if what they had produced is what was wanted. Almost like it was too easy to just fill in a bit of code and a couple of tests, cleanup a few things and submit it. They were left not really feeling challenged.

At first, this worried me, what if everyone thinks this? what if it really is too easy?

It turns out, that developer is just really good! So, honestly, that is kind of what I want… a good developer to just come in and write the code and it seem like not a big deal.

When I review the code, I want to think “ok yep, that’s not bad, I see what they did there. Yep, they’ve cleaned up a few things quite nicely”. To me, that is a sign that the challenge is working - as long as everyone doesn’t feel this of course. It has hit the right balance.

Final Thoughts

It appears the challenge is working well now. Still, I am a perfectionist and there is always room for improvement. I do want to iterate the challenge and improve it more.

To self-critique my work, I think it is too wordy… under exam conditions your mind seems to race and seeing a long page for the description can be like a big mind fuzz. I don’t want the candidates to feel this - I think my next iteration would be to cut the length of instructions in half.

For now, it is what it is… I would love to see you have a go at the challenge.

We have C# and Java versions for now, but we’ll add more if there is demand. Simply send an email to hello@geektastic.com stating whether you would like the C Sharp or Java version

Once you have submitted your solution, we’ll get it reviewed by one of the review team.

Join Geektastic today to take on a code challenge yourself and show off your skills and knowledge.