AI in Case Studies: Are Companies Assessing Different Skills Now?

I recently saw the same candidate receive completely different feedback during two hiring processes. 

One company praised them for using AI effectively throughout a case study. Another questioned the quality of the submission because they suspected AI had been used at all.

Neither company was wrong, they were simply assessing different skills.

As a recruiter specialising in data, AI, and software engineering recruitment, this is something I’m seeing increasingly often. The rise of AI tools is forcing hiring managers to ask what exactly they are trying to assess during the interview process.

For years, case studies have been designed to evaluate a candidate’s problem-solving ability, technical expertise, communication skills, and approach to decision-making.

AI has changed this, often dramatically.

Some organisations want to understand how candidates think independently without assistance, while others are starting to recognise that AI tools are becoming part of everyday work and want to see how effectively candidates can use them.

The result is that two companies hiring for similar data and AI jobs can arrive at completely different conclusions about the same candidate.

A candidate may use AI to research a problem, structure their response, challenge assumptions, improve clarity, and produce a stronger final submission.

From their perspective, they are simply demonstrating efficiency and modern ways of working. But a hiring may not see things the same way. Instead, they may ask ‘how much of this work is actually theirs?’

That’s a valid question, but is it missing the point?

The AI hiring debate is missing the point

When organisations discuss AI in recruitment, the conversation increasingly settles on one question: did the candidate use AI?

Increasingly, I think that’s the wrong question.

The more relevant questions might be: 

• Did they know how to use AI effectively?
• Did they validate and challenge the outputs? 
• Did they identify mistakes or gaps? 
• Did they understand the limitations of the tools? 
• Could they clearly explain their reasoning? 
• Did they add value beyond the first AI-generated response? 
• Could they solve the problem independently if required? 

These are the skills many companies are already looking for when hiring data scientists, machine learning engineers, AI engineers, data engineers, analytics leaders, and software engineers.

Using AI Is easy. Prompting takes a few short sentences, and outputs are near instant. But in such a world, judgement, critical thinking, validation, and decision-making become essential skills.

Hiring data and AI professionals in an AI-enabled world

This is no more relevant than when recrutiing data and AI professionals.

Many of the organisations I work with are actively investing in generative AI, machine learning, data platforms, and AI products.

In these environments, employees are expected to use AI tools daily. They’re using them to write code, analyse data, generate documentation, create prototypes, investigate issues, and accelerate delivery.

If that’s the reality of the role, does it really make sense to assess candidates as though those tools don’t exist, or should hiring processes evolve to test a candidate’s ability to work alongside AI effectively?

There’s no universal answer.

A company hiring for a highly specialised machine learning research role may prioritise deep individual expertise over the ability to prompt an AI tool.

A fast-growing technology business, on the other hand, may prioritise speed, adaptability, and the ability to leverage AI to improve productivity.

Both are reasonable. The problem arises when expectations aren’t clearly communicated, and that’s an issue I see all too often.

The real issue isn’t AI, it’s alignment

I don’t think most organisations have an AI problem. I think they have an expectations problem.

Candidates don’t always know whether they’re being assessed on independent problem-solving, AI utilisation, domain expertise, communication skills, or a combination of all four.

Hiring managers often assume candidates understand what’s being evaluated, but increasingly that’s no longer the case.

As AI becomes embedded across Data, AI, Analytics, and Software Engineering teams, organisations will need to become more explicit about what success looks like during the hiring process.

Otherwise, we’ll continue to see situations where one company rejects a candidate for using AI while another hires them because they used it exceptionally well.

Frequently asked questions

What skills are required when using AI?

Using AI effectively requires judgement, critical thinking, validation, and decision-making. Candidates should be able to challenge outputs, spot mistakes or gaps, explain their reasoning, and add value beyond the first AI-generated response. Independent problem-solving remains important.

Which roles are most likely to require AI?

Data scientists, machine learning engineers, AI engineers, data engineers, analytics leaders, and software engineers. Whether AI use matters in an assessment depends on the role and how the organisation works.

Should candidates use AI in interview tasks?

Organisations should clarify whether AI is permitted. Where AI is allowed, they should be ready to explain how they used it, how they checked the outputs, and which decisions reflect their own judgement. Where independent work is required, they should follow that brief.

How can hiring managers assess skills when candidates use AI?

Hiring managers should focus on the capabilities that matter for the role, not simply whether AI was used. Ask candidates to explain their approach, challenge assumptions, identify errors, and justify their decisions. If independent expertise is essential, include an opportunity to demonstrate it.

What should companies communicate before a case study?

Companies should explain whether AI is allowed, what kinds of assistance are acceptable, and what candidates will be evaluated on. The brief should make clear the balance between independent problem-solving, AI utilisation, domain expertise, and communication.