The earlier article in this stage, Local and cloud AI: what the difference means for employment advice, explained the difference between a small model running on a desktop PC and a large cloud model. This article goes deeper into the assessment tool itself: how it started, what went wrong with the first design, why the reasoning moved to the cloud, and what I have since landed on for the purposes of testing.
What the tool is for
The assessment tool takes a client’s account of what happened at work, with any documents they have, and produces a merits assessment in the form that would typically be found in an assessment letter: the salient background facts, an analysis of each possible claim against the applicable law, time limits and jurisdiction, and a conclusion with a percentage for each claim and the next steps.
Using local AI
Most computers can run their own AI. The simplest way is to install a program called LM Studio, which lets you browse and install open-weight models and ask them questions in the same way you might ask ChatGPT or Claude.
The answers these models give will inevitably be lower in quality, for the reasons discussed in the earlier article. A local model cannot compete on reasoning with a frontier model running on infrastructure that could power a small city, and I expected the testing to show that.
The project therefore proceeded on the basis of exploring what improvements, if any, could be made by building a scaffold on top of a local model. The local model was Gemma 4 12B, chosen because it was the best balance of speed and accuracy that I tested. Other models were tried, including Mistral and Qwen, and other sizes, including Gemma 4 26B-A4B and Gemma 4 31B, and of these Gemma 4 12B was the most consistent for the first design.
The first design: rules first, model for narrow questions
The first serious design was built around an eight-stage pipeline:
- Code extracted the text from an enquiry form.
- A local model pulled out a structured record of facts, with dates resolved by code.
- Claims were identified by code.
- Each claim was matched to a pre-loaded list of issues.
- A local model wrote answers to each question in the list of issues against quoted evidence.
- Pre-loaded time-limit rules were applied to each relevant claim.
- A local model wrote the prospects for each claim and the prose of the assessment.
- A final stage checked every date, figure and provision in the write-up against the earlier stages.
The lists of issues were built statute-first for the twenty most common claim types, with unfair dismissal split by reason, and cross-checked against the issues sections of judgments in the corpus.
What went wrong with the first approach
The first results were not promising. I set a benchmark of answering an employment law scenario with 70% accuracy, and the initial design scored around 37%. That was not because it failed to identify the right claims; it was that it badly misled, or got crucial things wrong.
The problem was fragmentation. A model that reads the whole of a client’s account can see how the facts bear on one another. A model that is handed one narrow question at a time, with a quoted passage, cannot. It answers each question plausibly and the assembled answers do not add up to an assessment. The write-up stage then has to stitch together conclusions it did not reach, and the result reads like a form that has been filled in.
The second design: model first, rules as the check
The design that replaced it starts with a model reading the enquiry form and listing every claim it can identify. It is then given the current legal notes for each claim, drawn from the same claim cards and dated rules that the first design used, and asked to assess the claims against them. The rules then check every date and time limit in what it has written against what the code calculates from the extracted facts, and the model revises the time-limits section of the letter where they differ.
How it runs
In practice the tool makes three kinds of call to an AI model, and everything else is ordinary code.
The first is the reading stage. The enquiry form and any documents are read twice, by two separate model calls, and each reading produces a structured record: the parties, the dates, the pay, the chronology, the steps already taken, and anything in the documents that contradicts the account. It is an extraction job. The models are not asked what the facts mean, only what they are. The two records are compared by code, and the dates are resolved and checked before the assessment is written.
The second is the writing stage, which runs in three passes, with the original papers in front of the model each time. In the first pass the model lists every claim it can see, generously, because a claim that is listed and then ruled out costs nothing and a claim that is missed costs everything. The code attaches to each listed claim its legal notes from the claim cards: the test, the elements, the current time limit, the qualifying conditions and the figures that apply. In the second pass the model writes the assessment against those notes and the facts, together with the dates and time limits the rules have already worked from the fact record. The rules then check the time-limits section of that draft against their own calculations, and where they flag a difference the third pass has the model revise that section. A final check compares the dates and figures in the finished letter with the rules’ own.
The third is adjudication. Where the two readings disagree on a fact, the disputed point is put to a separate, stronger model with the relevant passage and asked for a decision with a reason and a quote. This follows the same principle I use elsewhere in the project: uncertain results are adjudicated by a stronger model and recorded, rather than left to the model that produced them or checked by hand mid-process.
The readers, the writer and the adjudicator are independent of each other. The readers do not need to know any law; the adjudicator sees only the disputed point. That separation is what makes it possible to put different models in each slot without touching the rest of the tool, and it is what the three versions below are: the same pipeline, with different models in the three slots.
Three versions
The current tool is a prototype that runs on my own machine, on invented scenarios only, and it has a switch with three positions. All three keep the same code around the models: the same extraction, the same claim cards and legal notes, the same rules checking the dates, the same four-section output.
Local only. Two local models and nothing leaves the machine. Gemma 4 12B reads the papers twice and extracts the facts, because it is fast and extraction is what it does well; Gemma 4 31B, the largest model that fits comfortably on a 24 GB card, lists the claims, assesses them and writes the letter. There is no adjudicator, because there is no stronger local model to adjudicate. Where the two readings disagree, the point is marked as uncertain, the tool uses the safer value (the earlier deadline), and the letter tells the adviser to confirm it. It is in the tool because the question of what a desktop model can do unaided is the question the project started with.
Cloud only. Wherever the tool uses a model, this version sends the task to a frontier model through the API: Claude Sonnet reads first, Claude Haiku gives an independent second reading, Claude Sonnet writes, and Claude Opus adjudicates where the two readings disagree. This is the strongest version and the simplest, and it follows the same principle as many commercial products, which take one of the best frontier models and build a bespoke scaffold on top of it.
Hybrid. This version takes the middle ground. The local model does the first reading on the machine, Claude Sonnet gives the second reading, and Claude Opus settles any disagreement between them. Claude Sonnet then writes. In testing there was limited difference between the local and cloud models on extraction, but the writing stage was the biggest difference between a low score and a high one, which tracks the prediction at the outset of the project that for reasoning there is no substitute for a frontier model.

The screenshot shows the prototype as it stands. The mode is chosen at the top, with a readiness check for each.
What comes next
The next part of the assessment tool work is to test these three versions on invented employment law scenarios and compare their results with what consumer AI platforms such as Claude, ChatGPT and Gemini produce. The first article covers an unfair dismissal and unlawful deductions scenario, which can be found here.