What Makes a Good or a Bad Coding Challenge?

Guest post by Andy Davis

Anyone who has devised a coding challenge will agree with me when I say:

It’s REALLY hard to create that perfect coding challenge that filters out bad developers but keeps good developers interested.

Or is it?

Well, it turns out, you can dramatically increase the effectiveness of a coding challenge by using one simple rule.

And in today’s post I’m going to show you what that is… and exactly how apply it to the next coding challenge you design.

Want to test your coding skills and take a peer reviewed Geektastic code challenge? JOIN GEEKTASTIC

But before I tell you exactly what that is, let me tell you a little story…

Several years ago, a good friend of mine took his driving test for the 3rd time. He’d done a tonne of lessons and he’d had more than enough practice - he is a good driver. On the day of the test, the examiner asked him to turn left out the test centre. Under the pressure of the test, the instructions didn’t register and he turned right.

Now you could assume he failed for not following instructions. Instead some common sense was applied and his driving skills were still assessed - NOT his ability to follow instructions under intense examination pressure.

It should be this way with coding challenges.

Clean, well written code that communicates its intention, should be a pass even if it is not exactly what the specification asked for. Unfortunately it’s not always the case.

You or I might argue that understanding requirements is a big deal. And in some contexts it really is. If the job was to be fed requirements by email, have no interaction with anyone, and deliver within 1 hour, then I would have to agree with you. But how many jobs are like that? And let’s be honest, would we really want to work there? or shall we leave those jobs for elancers?

We all know that the real world isn’t like this. You don’t get fed a bunch or algorithms to code up during the job, so why do they send them to us as a coding challenge?

It’s because it is easier isn’t it? There is more less a set answer, does it work or not? And potentially an order of complexity (big O, anyone?)

Since algorithm questions have mathematically predictable answers, these sorts of tests can even be marked by a machine.

This is disrespectful. We took the time to do your coding challenge and you couldn’t take the time to have it marked by a human.

If you have your coding challenges marked by a machine - PLEASE STOP.

Machine marking is a binary solution to a non binary problem. People are not binary (not matter how geeky we talk).

Let me side-track you for a moment and tell you another story… (I’ll get back on topic soon, I promise)

At my university, there was a revolutionary new system called Coursemaster. The boffins that be had devised an automatic programming coursework marking system. It became our friend, foe and arch nemesis over those 3 years. At first it was fun, then we realised that our grades depended upon it. That’s right, our degree was marked by a machine (well, part of it anyway).

The professors built-in some flexibility, it allowed a 2nd and 3rd attempt. So what did students learn to do? That’s right, Group up and code it together, pooling their 2 spare lives as a sacrifice to the cause. A great application of Pair Programming - others might call it cheating!

Anyhow, like it or not, the Coursemaster system was definitely gamed by some students - but let’s be fair here, the system was out to game the students, it wasn’t really very good at marking their code - correctness is a very small part of the battle - so it probably levelled the playing field.

Flash forward 15 years and the technology has not moved on!

Only last week I was requested to do a test that was marked by a computer…

I got 0%….Why?

…I misinterpreted one word in an aggressively timed coding challenge and it left me with not enough time to correct my error. Total frustration. I couldn’t even prove it was good code, it wasn’t, I hadn’t refactored yet, I ran out of time -

Now there’s an idea, if the test is timed maybe there could be a refactoring window at the end where it doesn’t allow functionality to be changed but you can improve the code quality and make it communicate (which in my opinion would be a pass if you know how to do that).

Let’s cut to the chase…

The single biggest factor that will improve your coding challenge

So here is is… the killer idea you have been waiting for. I spent the last few weeks speaking to and surveying some of my peers on this topic. One guy stood out amongst the crowd, he’d done as many tests as me, but better than this, he’s also designed several coding challenges and saved a tonne of time for HR in the process.

It is with great pleasure that I mention Zeno Foltin, a London-based (polyglot) software developer. Zeno is a high quality developer and I rate his opinion on this topic extremely highly.

I met Zeno when both of our companies were working with WorldRemit, a London FinTech money transfer company. They were on a major hiring binge and Zeno took responsibility for making a coding challenge to help them in their quest for hiring good developers. Introducing this stage to their process was vital for them. The quality of candidates through the door went through the roof. I caught up with Zeno and asked him about World Remit coding challenge and his thoughts on how to design a good challenge.

We compared notes about the wealth of challenges we’ve done between us and some key points stand out.

So here it is, the number 1 stand out idea that you should apply immediately to any challenge you create:-

Make the challenge as close to the real world as you can.

It sounds too obvious doesn’t it? I wish it was. I can count on one hand the number of challenges I’ve done where I’ve been able to see the connection between the coding challenge and the company once I have started the role.

This is the single biggest factor that has kept us interested in various challenges over the years. There is nothing more frustrating that receiving a coding challenge to do, and seeing it is another boring algorithm like checking a string for parenthesis counts, or sorting a list without using inbuilt functions.

Generic coding challenges can be a drag, especially if you are applying for multiple roles. When the challenge is closer to the real role, it solves the “interest” problem. Candidates who aren’t that interested in the challenge will naturally not do their best - therefore it kills multiple birds with one stone.

One of the best ways of making your challenge close to real world is to make sure the challenge poses a problem within the general business domain of the company.

Want to license a code challenge to analyse the key skills of your candidates? Register now

IMPORTANT NOTE: If you take this to too far and make this too domain specific, you’ll end up alienating people and could end up with a coding challenge that only lets you hire people who already work for you - so remember to strike a balance. And remember if you use Geektastic for your coding challenge, there is an opportunity to license this challenge out of multiple companies to use. So if it covers a bit of the tech that needs testing as well as the general domain area, you’ll be onto a winner. If I were looking for a coding challenge to use for a logistics company and there were 2 available challenges but one was about sending parcels and one was about phone bills, I know which one I would chose.

To give you an idea. Challenges we’ve liked in the past, all fulfil the following

• Open for creative interpretation,
• Marked by a human
• Used for open discussion later
• Production quality code requested (Clean Code).
• More than input/output of a single method. Eg, a light UI, an object model, an API call.
• Simple and clear - can be summarised in a single sentence.
• Time suggestion, not limit

Several BIG Topics came up in the process of our discussions:

• Timed challenges
• IDE (Code Editor)
• Realisticness of it being like the actual job

So let’s look at each of those points in detail.

Timed challenges

Challenges with a hard cut off time. It seems obvious why companies might impose a time limits:

Create a level playing field for comparison - if they have the exact same amount of time and no prior knowledge, we can accurately compare candidate solutions.

Candidates know upfront that a fixed amount of of time is needed not a snow ball into hours/days and therefore are more likely to accept doing the challenge

Zeno and I put our heads together on this one and we think not enough thought is put into the downsides of that time limit.

For example:

It doesn’t actually measure how well they will do on the job as the time pressures in the job are not the same as examination pressures.

Faster developers are NOT better developers.

A candidate who is slightly slower but can model the domain well, TDD and produce something infinitely better than speedy gonzales (who only writes procedural code) may get unfairly compared as a worse developer if they couldn’t get that across in the timeframe.

Zeno said:

“I don’t think timed challenges provide any value in measuring how well the candidate will do on the job. If it’s about trying to find out how someone handles pressure, do it in a pair-programming session maybe. I think it’s silly to think in our creative industry that putting such pressure on people will bring the best out of them.”

So whilst we don’t think timed challenges are completely useless, it just might not be the type of comparison they think they are making - possibly just a superficial comparison. The compromise here is to have a time limit but make it generous enough that the candidate has time to try out a few ideas. The last thing you want to do is cut everyone off mid flow - this would remove the benefit of having a test in the first place.

IDE

Next up was the IDE. Believe it or not, some coding challenge systems actually force you to write and compile code in the browser - it seems to have become a bit of a fad recently. It’s pretty nifty but it only really works for simple code. It seems to have benefits of scale for the companies using them as the computer tests the inputs and outputs for correctness - but this shows very little respect to the candidate and actually makes their life much harder. It is unable to provide rich intellisense or access to commonly used frameworks or testing tools. There wasn’t really much debate here, we both hated coding in the browser. Developers like their tools and we are no different. Writing code in the browser is not part of our weaponry and will not be something required on the job. Browser based code editors are great for tutorials and quickly trying something out you’ve just been taught but as a candidate testing tool, this should seriously be reconsidered.

Zeno said: “They must use an IDE they are comfortable in. I used to ask candidates to bring their laptop with them for the face-to-face interview to do some pair-programming. You want to find out how the candidates work in their usual environment, not how they cope with a new and unfamiliar one. Unless the job is about writing code in the browser?” Compromise might be that you use your own IDE but then upload or paste the final solution.

Likeness to the actual job

We know that is it impossible to condense the real nature of the job down into a coding challenge. But there really doesn’t seem to be any point in asking a developer to complete something like an in-memory algorithm to return them the correct change from a vending machine. Sure, there are some algorithms in there and order of complexity to think about. However, if the job isn’t about solving little puzzles, it really doesn’t appear relevant.
I was once sent a zip file from an online fashion retailer and told to refactor it. Whilst this sort of request is a little bit lazy - at least this is probably closer to the real job. Refactoring legacy code is part of almost every job. Furthermore it was actually in their domain. So, big points to them for this challenge, it was enjoyable. The only downside of that is that I never heard anything back (after spending 2 hours on the test too) - this is one of the biggest bugbears we have as developers.

The bottom line around getting challenges close to the real world is:

• Avoid algorithms unless that really is what you are hiring for.
• Avoid tests that were clearly designed for a different industry
• Be more honest with your coding challenge, if you are hiring a bug fixer, how about a challenge where you need to fix a couple of bugs

Case Study: JUST EAT

One example we discussed in detail was the JUST EAT recruitment test. Zeno worked with them for a year and I went through the process last year, so I had also taken the same test.

We both had a great experience with it and rated it the number 1 test we’d done. You can see their test here:- https://github.com/justeat/JustEat.RecruitmentTest

Zeno said:

“I really liked the JUST EAT recruitment test because it is simple - list the takeaways for a given postcode using the API. It requires you write a simple UI to present the information, but it doesn’t tell you much more on how to go about it. It is directly to do with their business, it covers UI and logic (I don’t believe in back-end or front-end only roles), and it requires more effort than a couple of minutes figuring out or googling an algorithm. A well designed challenge, can give an opportunity for the candidate to showcase the best quality work that they can do without hard time constraints.”

The JUST EAT recruitment test is also a very enjoyable challenge. So let’s take a deeper look at it:

• It is a very clear and simple requirement. Use API to search for takeaways by postcode and display results
• It covers the full stack of API through to UI
• If you are a mobile developer, you do a mobile client. Web developers do a Web UI.
• It is about their domain - online takeaways. It uses their public API, so the takeaways you get back for a postcode search are real. This immediately hooks us into actually enjoying the challenge a bit more because it doesn’t feel contrived.
• Open source - JUST EAT decide to list their challenge on their github account - This alone sparked my curiosity got me looking at other public projects they had on there. With this, I found myself inside their published code and started to get a feel for the type of code they are already writing at JUST EAT. JustBehave, JustFakeIt and JustSaying were 3 frameworks right there in their account that I could look at and see some of the real code their teams produce - granted this is only the view of the code they want to be seen, but this isn’t code for show, it is code they actually use.

It’s great that even at the coding challenge stage, there is a bit of two way flow of information, just by being open to open-sourcing. There aren’t many places where you get to see some of the code before you go there. It is often a bait and switch of great stuff spoken about at interview followed by a shocking reality of legacy code once you start.

What can we learn from JUST EAT even if you don’t have a public API?

You might be fooled into thinking JUST EAT have unfair advantage of their domain being highly public facing. Even if you don’t “need a balti”, you still know some of the takeaways in your area that they service and when these familiar venues are returned by the API for your postcode. Not everyone can do this as their domain is not so publicly known - but I bet with a little thought, you can be creative enough and throw the candidates a little bone about the real job rather than have them implement a vending machine, phone bill or shopping cart. There is nothing worse than seeing another challenge that isn’t even related to the business sector that the company is in.

Another lesson that can be learned from JUST EAT is not be too strict about the solution. If you can allow the candidate to be creative, you will get a better picture than from an algorithm implementation with a right or wrong answer. Sure, it takes more effort to have a developer mark the solution, but that time will be infinitely better spent if you discover better candidates from it.

Here are some ideas for you, even if you aren’t as publicly glamourous as JUST EAT

• If you are Compare The Market - provide 3 insurer APIs and request the developer calls all 3 and display results in order of best quote.
• If you are Shazam, provide an API that accepts a music snippet and identifies it?
• If you are Uber, can you have an API showing closest car to a set of coordinates and have the developer write a little client to request a ride?
• If you are Tesco, yes yes ok, you can do an online shopping cart
• If you Buzz Fizzes, you can… well… nevermind.

It is a continual learning process

In agile software development, we are told to embrace change. Things will change, so build change into the process and not make it an afterthought.

Like anything else, accept that you won’t your challenge right the first time.

Developers WILL complain.

Some good candidates will slip away - refusing to do it.

Some good developers will ‘fail’ it.

Some bad candidates, will find their way past the challenge and waste precious time that you could be spending with Ninjas

Just a joke, there’s no such thing as Ninjas…

…and even if there is you don’t want them

…and even if you think you do, you just alienated them and half the developer community by using the word Ninja, whoops.

You should plan to get feedback from candidates taking your test and LISTEN to this feedback.

Warning: FEEDBACK WILL BE EMOTIONAL, especially if they ‘fail’.

Use any little nuggets inside their feedback to evolve and improve the challenge.

Finally, if they have a valid point - reconsider the candidate, I have successfully failed a challenge before and turned it into a pass - you should do the same (under the right circumstances) Continue to screen out cheats and chancers, however, is googling the answer really cheating? After all, good coders are good googlers and part of the job is knowing how to define problems well enough that finding the right answer on stackoverflow comes naturally. Pasting a snippet of code and using it effectively, is definitely not cheating, But I’ll leave this one up to you.

Don’t put all your eggs in one basket with a single piece of code.

A coding challenge should only be used to screen out really unsuitable candidates. Everyone else progresses to the next stage. And yes, there must be multiple stages if you have included a coding challenge. You cannot base a hiring decision on code alone.

Uncle Bob thinks a coding challenge is useful but only as part of an interview process, and values pair programming over and above it - I tend to agree, but that is another story - and probably a long one so I’ll save that for another day

Let’s wrap things up (that timer is ticking down)

I have covered a few key points here. I hope it sparks some ideas. If you are making a coding challenge or thinking about changing your current challenge, drop me a line, I would be interested to hear about it and be happy to pass comment on your current challenge.

I think it is important to remember that developers do like coding challenges. We’re coders, we like to code. We’re not so fond of having someone looking over our shoulder but we do like solving business problems and we do like writing code - so don’t be afraid to have one in your process.

Let’s conclude by reminding ourselves of some of the important points that we’ve explored

• Let candidates use a comfortable environment - do you really want them to write code in the browser or is their own IDE a better choice?
• If you impose a time limit, think carefully about the implication of that limit, are you going to cut off a good candidate mid flow. Is it really a fair comparison just because time is equal?
• Can you make it relevant to the job? We all now know that Google’s silly questions about painting jumbo jets and shrinking people into blenders didn’t find them any better candidates than traditional questioning - does the same apply to coding challenges?
• Can the coding challenge cover a little front-end and back-end
• Be very specific in the requirement. It is ok to say, you may present this on a simple web app. But requirements like “tests are optional” is a huge dilemma for a candidate - this is unfair and risky especially if you have in mind what you want (ie tests).
• If you value performance over clarity, tell the candidate this. Algorithm tests lend themselves to right or wrong (plus performance) - this makes it easier to mark, but making it easier to mark is NOT your goal. Your goal is to filter out the time-wasters, yet algorithms are the easiest to plagiarise

• Iterate, you won’t get it right first time. Keep tweaking it.

• Finally - Make it relevant to your business sector if you can. The best developers aren’t just coders, they care about solving real business problems. Make it clear you understand this by making your test relevant to your business.

Good luck in producing a great coding challenge. Personally, I recommend an open and creative challenge with a guide time limit. - where the candidate can show off their creativity and flare, domain modelling and good clean code. But then again, I would say that… because I once failed FizzBuzz.

About Andy Davis

Andy is a software developer. He’s been writing software for over 20 years and helps businesses solve interesting business problems, by helping them create the right software. He also helps teams with their developer hiring strategy as he believes that hiring the right people has a multiplier effect on awesome software teams.

Want to create your own? Perhaps you have a coding challenge you have created already already, or maybe you feel you can create a good one now from scratch. Whichever of those two, Geektastic make it easy to load it onto their platform and manage the whole process of inviting your candidates (who can complete it in their own IDE), submitting it and getting it reviewed by a real human.