Testing AI on employment law · 111 min read

Testing AI on employment law: the first results

Twelve AI outputs marked against a sealed key on a conduct dismissal and unpaid wages scenario: nine consumer chatbots and assistants, and the assessment tool run three ways.

Introduction

The test is simple: a fictional client intake, given unchanged to a number of AI systems, with each answer marked against a key written before any of them were run. In effect, an AI employment law Olympics. This is the first round of results.

Twelve outputs were marked in the main comparison: nine from chatbots and assistants that most people could reach for free or on a consumer subscription (ChatGPT 6, Gemini, three Claude models, Meta AI, Microsoft Copilot, Grok and Perplexity), and three from the assessment tool.

The scenario

The first scenario is an unfair dismissal (conduct) and unlawful deduction from wages claim.

Mr Smith is a warehouse team leader with just over two years’ service and a clean record. At the end of a shift he took a patio heater in a crushed box from the disposal cage, where customer returns that could not be resold were kept. The heater carried a red “for disposal” tag. His shift manager had told him when he started that red-tagged items in that cage were going in the skip and staff could take them, and other people had done so; he had taken a damaged kettle himself months earlier, and a colleague on nights had taken a television.

The next morning the operations manager questioned him for ten minutes and suspended him on full pay. A week later he was invited to a hearing about “the removal of company property from site, and other matters which have come to light”, with nothing attached and no explanation of what the other matters were. At the hearing the same manager said a number of unnamed colleagues had reported stock going missing and had seen him taking things more than once; he was not shown any statements. He asked the manager to speak to the shift manager who had given the permission, and was told it would be “taken on board”.

A few days later Mr Smith received a letter informing him that he had been dismissed without notice for theft. It did not mention the shift manager. The letter was dated 15 October 2026, but he in fact read it on 17 October 2026.

He appealed in writing within the five days allowed and has had only an out-of-office reply. He has not been paid for the two and a half weeks of suspension, and has received nothing since his September wages, including notice pay.

In his enquiry form Mr Smith complained that he had not been told he had the right to bring a workplace colleague or trade union representative to the disciplinary hearing. He also mentioned that he had asked the same manager for a pay rise some months earlier and been refused, and wondered whether this had anything to do with his dismissal.

The scenario was set as if the enquiry had been received on 11 November 2026: six weeks after the time limit for most tribunal claims moved from three months to six, and before the 2027 changes to qualifying service and the compensation cap. A model advising on these facts has to know the law as it stood that week.

The task was an advice letter to the client in four sections — background facts, merits, limitation, and a conclusion with a percentage for each claim and next steps — in under 2,500 words, written for someone who is not a lawyer.

Expected answers

The key was written before any model was run. In outline, it wanted the following.

Three claims, and two conditional ones. Unfair dismissal, wrongful dismissal for the notice, and unlawful deduction for the unpaid suspension wages. The key also allowed for a possible holiday pay claim, on the basis that Mr Smith has received no final pay, which would ordinarily have included payment in lieu of untaken annual leave. A right-to-be-accompanied claim was also tentatively present, but the key expected the model to notice that Mr Smith did not appear to have made a request to be accompanied at the time, and to weigh it accordingly.

The merits, in the right order. The failure to interview the shift manager is the central defect: the employer never investigated the client’s only defence. Then the anonymous allegations used without safeguards, the “other matters” that were never specified and the “pattern” raised for the first time at the hearing, and the appeal that has gone unanswered, so nothing has been cured. The investigating manager also chairing the hearing was treated as a factor to be balanced; a model that treated it as unfair in itself was marked down. The key also expected the answer to address the counterweights: Mr Smith admits taking the item, so a tribunal could find he contributed to his dismissal, or would have been fairly dismissed anyway if he only succeeded on a procedural unfairness argument.

Wrongful dismissal on its own test. The question is whether Mr Smith in fact committed a repudiatory breach of contract serious enough to justify dismissal without notice. The tribunal decides that for itself on the evidence, and the employer’s belief, however reasonable, is not the test. A model that confused this with the band of reasonable responses was marked down.

The traps. The pay-rise refusal is not a claim and is not evidence of anything other than a tentative and unlikely ulterior motive. The colleague’s television helps as evidence that taking red-tagged items was accepted practice, but a model that over-valued it on the basis of a disparity in treatment argument was marked down. There is nothing in the facts to support discrimination, whistleblowing, equal pay or victimisation, and raising them was penalised.

Limitation. The dismissal took effect on 17 October, when he received the letter, and not on the date of the termination letter itself. Mr Smith was paid monthly on the 28th. The six-month limit applies, so unfair and wrongful dismissal by 16 April 2027 and the deductions claim by 27 April 2027 (counting from the missed payday).

Percentages. The key gave bands rather than figures: unfair dismissal 60–75 per cent, wrongful dismissal 50–65, wages 85–95, and no figure (or a conditional one) for holiday pay and the accompaniment point.

Marking. The key allocated 100 marks across the four sections, with penalties for errors of law, invented authority, raising claims the facts do not support, and advice that could put the client out of time. During marking I added a fifth section of 10 marks for things the legal key could not see: whether the letter was written for a non-lawyer, whether it followed the brief, and whether it was internally consistent and put figures on the claims. Totals are out of 110.

The models tested

The chatbot models used were mainly free-tier services: Gemini, Perplexity, Copilot, Grok, Claude Haiku and Meta AI. ChatGPT 6 and Claude Sonnet can also be reached on a free tier, with more restricted usage. Claude Opus was the most advanced model used and could only be accessed through a paid subscription.

The assessment tool was run in each of its three modes: local only, cloud only, and hybrid.

The results

Rank Model Total /110
1 Claude Sonnet 101
2 Claude Opus 99
3 Assessment tool, hybrid 95
4= Assessment tool, cloud 93
4= ChatGPT 6 93
6 Meta AI 85
7 Claude Haiku 71
8 Grok 67
9 Microsoft Copilot 65
10 Perplexity 64
11 Gemini 60
12 Assessment tool, local 56

Discussion

What everyone got right

Every one of the twelve identified the three claims that matter: unfair dismissal, wrongful dismissal, and unlawful deduction from wages. Every one saw that the shift manager had not been interviewed and that the anonymous evidence had never been put to the client, and all but the local run saw that “other matters” was not an allegation he could answer. Every one put the wages claim at or near the top of its prospects. Nobody invented a case or a statute. Nobody applied the 2027 reforms early.

What separated the tiers

Knowing the law had changed. The six-month limitation period was worth ten marks on its own and was the single largest discriminator. Seven of the twelve knew it applied. Four did not, and gave the client the old three-month deadline throughout; the local run applied the new limit to unfair dismissal and the old one to wrongful dismissal. An early deadline does no harm, so the key does not penalise it beyond the lost marks, but it is a clear sign of a model working from stale training. The clearest illustration is Meta AI: a careful, well-balanced letter that walked through the Burchell questions against the facts, set out both the Polkey and contributory fault reductions, got the wrongful dismissal test exactly right and kept notice pay separate from wages. It applied the three-month rule throughout, which alone cost it about thirteen marks and a place in the top tier. Perplexity was the mirror image: it found the right deadline and could not supply the analysis to go with it.

Telling the client what could go wrong. The top tier all did. Claude Opus set out contributory fault, Polkey and the Acas uplift in one short passage; the assessment tool’s cloud and hybrid runs and Meta AI did the same. Four of the lower-tier letters offered little or none. Gemini’s was the most striking: it described the heater as “worthless” (the facts do not say so), declared that dismissal “falls well outside the band of reasonable responses”, and put the wrongful dismissal claim at 80 per cent. It read as a pleading, not an assessment.

Keeping wrongful dismissal separate from unfair dismissal. Two letters slid into testing the notice-pay claim against the reasonableness of the employer’s belief or the quality of its investigation.

Resisting the urge to find more claims. The intake mentioned a refused pay-rise request and a colleague on the night shift who had taken a television. The key penalised raising discrimination, whistleblowing or victimisation on those facts. Claude Haiku raised discrimination, whistleblowing and equal pay, and asked the client to supply comparators’ sex. Perplexity listed every protected characteristic and then kept discrimination open as a row in its prospects table. The strongest letters either dismissed the point in a sentence or, better, used a short “claims not pursued” list to show the client the ground had been covered.

Where even the best fell short

The investigating manager also chaired the hearing and made the decision. The key’s view is that this is not unfair in itself, but may be relevant to fairness depending on the size of the organisation, a point made in the Acas Code. Four letters raised it, but only two (Sonnet and ChatGPT 6) said that it did not make the dismissal unfair on its own, which may say something about how far AI can be trusted with nuance.

Four of the five top-tier letters identified the correct six-month deadline and then advised the client to work to the earlier three-month date “to be safe”.

Only five of the twelve told the client that the compensatory award is capped, and Opus, which had the fullest remedies section, was not among them.

The assessment tool

The cloud and hybrid runs finished third and joint fourth, within four to eight points of the two Claude flagships. Their lost marks are systematic. Both correctly state that the dismissal took effect on 17 October, when the letter was read, and then count the deadlines from the 15th as the cautious date, which costs two days but may be the more sensible approach. Both assert a fact the client did not give: that he did not ask to bring a companion.

The cloud and hybrid runs scored within two points of each other, with the same structure, the same authorities and the same gaps. That is the result the hybrid design was built for: when a frontier model writes the letter, local extraction from the papers costs nothing against cloud extraction. The variable that matters is the model doing the writing.

The local run finished last, 39 points behind the hybrid on the same template, the same claim cards and the same rules. The difference is the model writing the letter: Gemma 4 31B, running on a desktop graphics card, in place of a frontier model through an API. The letter it produced identified the right claims, saw that the shift manager had not been interviewed and that the witness evidence had not been shown, kept notice pay separate from wages, and stated the rule on the right to be accompanied correctly. Its percentages — 70 for unfair dismissal, 60 for wrongful dismissal, 95 for wages — were all within the key’s bands. Everything else was thinner. It wrote about 1,300 words where the cloud and hybrid runs wrote around 2,400 to 2,500, and much of what separated the top tier was missing: no Polkey, no contributory fault, no analysis of the ignored appeal, no mention of the “other matters” or the pattern allegation, no holiday pay, and no figure in pounds for any claim. It treated the colleague’s television as straightforward inconsistent treatment, which it said “often makes a dismissal unfair” — the over-reading the key was set to catch. Its limitation section gave wrongful dismissal the old three-month deadline with no mention of the change in the law, gave unfair dismissal the new six-month one, gave no deadline at all for the wages claim, and listed the hearing invitation under the wrong date in a line of unedited template output. Its next steps told the client not to contact the employer directly, which cuts against chasing the appeal and the unpaid pay.

Conclusion

Every system tested, free or paid, local or cloud, found the claims and the main procedural defects; none produced a letter that would have seriously misled the client. The differences were in currency, balance and doctrinal precision: whether the model knew the law had changed six weeks earlier, whether it told the client what could go wrong as well as what had gone wrong, whether it kept the wrongful dismissal test separate from the unfair dismissal test, and whether it resisted building claims the facts did not support.

The next scenario will involve a more complex discrimination claim. From then on, I will publish the exact wording of the scenario and the output of each model, so that readers can see for themselves the results each one produces.