Coding Assessments in a ChatGPT world

Given that it has already proven its prowess in law and medical licensing exams, it’s safe to assume that it would be a wiz at technical tests, being able to ace a multiple-choice coding test in less time than it would take to make a cup of coffee. And yet, we are still only in the early stages of this AI golden era. So, with ChatGPT and subsequent AI systems poised to only get bigger, better, and more advanced with time, what implications does this have for the technical assessment landscape? 

The Achilles heel of AI 

Despite their remarkable abilities, AI systems have their limitations in that they are mostly limited to the data and information that they have gleaned from the internet. In the realm of coding, this means that AI systems can readily solve common algorithms and test problems, albeit with many variant solutions. In essence, the more common and shorter the task, the more likely ChatGPT will provide an accurate answer. Therefore, the trick for challenging both human candidates and AI systems alike is to emphasize test length and uniqueness. 
 

The Intricate Dance of Length and Uniqueness in Coding Tests 

Coding test length and uniqueness are related but distinct aspects of coding assessments. The longer the test is, the more unique it is likely to be. While test length plays a role in uniqueness, it’s also crucial to incorporate uncommon problems and scenarios in the assessment. Unique tests are harder for AI systems like ChatGPT to solve as they may not have encountered similar problems during their training. However, measuring a test’s uniqueness can be challenging. Therefore, focusing primarily on test duration can often help kill these two birds with one stone. 
 
 
Codeaid is a great tool in this aspect because it is the longest and most in-depth coding test platform on the market to date. This granular approach, combined with its built-in plagiarism and similarity score features help to circumvent AI-generated tasks, as the tests are designed to evaluate both the test-taker’s understanding and problem-solving abilities and not just their ability to write structurally correct code. 
 

 
Detecting ChatGPT with ChatGPT 

Another way to assess the uniqueness of a test is to try putting ChatGPT itself to the challenge. 

By submitting a ChatGPT-generated solution and reviewing the score, you can gauge whether the test is a reliable measure of candidates’ skills.  
 

If ChatGPT struggles with the test, it’s likely a solid indicator of a candidate’s coding abilities, as it effectively challenges the AI system. Conversely, if ChatGPT scores well, you might still be able to use the test by looking for high plagiarism or similarity scores among test-takers, indicating the potential use of ChatGPT. However, it’s recommended to avoid such tests altogether to ensure a fair assessment of candidates’ skills. 
 

Time to re-think your screening strategy 

Screening tests, particularly multiple-choice tests, are possibly the most affected segment of the technical assessment market. While these tests are often timed, making them slightly more challenging to use ChatGPT, it still leaves a lot of room for AI-aided answers to slip through. 

While there is no clean-cut screening replacement for these tests, other than manual screening, one potential solution is to make coding tests used later in the hiring process more robust and challenging, making it difficult for AI systems to perform well. This would ensure that only the most qualified candidates make it through to the interview stage. By filtering out candidates before the interview stage, employers can maintain scalability and keep costs under control without sacrificing the quality of their hiring decisions. 
 
 

The Future of AI in the Technical Assessment Space 

This has led to and will continue to be a game of cat and mouse between employers and AI systems as they try to outsmart each other. And while blocking AI systems may seem like a logical solution at first, it is important to recognize that this is only a temporary fix. As AI technology continues to improve, it may become more beneficial to incorporate it into the software development ecosystem rather than trying to keep it out. 

In fact, just as Git has become an essential tool for software developers today, it is possible that AI systems will also become an integral part of the development process. This means that employers should not only be looking for ways to block AI systems but also consider how they can utilize them to their advantage. 

To do this, it is crucial for employers to ensure that their coding assessment vendors have a ChatGPT plan in place. This will help to ensure that they don’t find themselves bypassed by the very industry they are trying to serve.  

To find out more about how Codeaid’s robust testing system can help you find true, non-AI-generated talent, visit www.codeaid.io 

If you’re in technical recruitment, explore Codeaid. Our latest AI Interviewer system, combined with our robust coding testing system, can accurately evaluates developers’ technical knowledge, ensuring you hire the best candidate.