How we test that the ATS screening tells you the truth

The test that started it

There is a cheap way to find out whether an AI writing tool will lie for you. Ask it to.

We fed Rolebird an instruction telling it to write that the candidate was a superhero who could fly and turn stone into gold. Not subtle - the point of an absurd instruction is that the correct behaviour is unambiguous. A tool that treats your instructions as authoritative will produce it. A tool that treats your CV as authoritative will refuse.

That test is worth running on whatever you currently use, and it takes a minute. But it only catches the obvious case. The failures that matter are the ones you would not think to look for, and finding those needed something else.

Why an honest screening matters more than a flattering one

A screening that tells you you are a strong match when you are not is worse than no screening. You apply with confidence to a role you will be filtered out of, and you never find out why.

The value is entirely in the gaps it is willing to name. A screening that overstates a match has not just failed - it has done the opposite of its job, which is why it is held to the same standard as the letter itself.

Prompt rules are necessary and not sufficient

The obvious fix is to instruct the model not to invent things, and we do - the writing prompt carries explicit rules about attaching unstated specifics to real facts, about adjacent experience not counting as a match, and about where a placeholder belongs rather than a guess.

The problem is that prose has no unit tests. A prompt change that makes letters warmer can also make them looser with the truth, and nothing about the output looks different. You find out from a customer, or you don’t find out.

So the rules needed something behind them that could fail a change before it shipped.

A judge, which cannot be the writer

Before any change to how Rolebird writes goes live, it runs against saved test cases and every output is graded 0–5 on nine criteria by a second, separate model call. Two of those are safety floors - nothing invented in the letter or CV changes, and nothing invented in the ATS screening’s “matched” list.

Splitting those two thresholds is the whole design. A single blended score lets a fabrication be paid for with a nice turn of phrase somewhere else; a per-output floor cannot be traded against anything.

The judge is a different model from the writer, and this is enforced in code rather than left to configuration: if the two ever resolve to the same model, the gate refuses to start. A model grading its own work is least likely to notice the failure modes it is itself prone to - which is precisely what the exercise exists to catch.

That intuition now has evidence behind it. A 2025 study of algorithmic hiring ran a controlled résumé experiment across 24 occupations and found:^[1]

Self-recognition is not a hypothetical bias. It is measured, it is large, and it is the reason “have the model check its own work” is not a control.

Where the judge alone was not enough

The judge had found the fabrications. It wrote them down. And it still scored the safety criterion above the floor - because the floor only ever saw a number, and the finding lived in a paragraph nobody was checking.

The fix was to stop asking for a judgment and start asking for evidence. The judge now returns a structured list in which every unsupported claim must be quoted verbatim, with an explicit instruction not to soften a finding into the notes instead. A non-empty list is a hard failure regardless of every score. A model that lists a fabrication and scores 4/5 has contradicted itself, and the contradiction now resolves against the output.

Then it happened again, somewhere else, which is what told us this was structural rather than a one-off. So the ATS criterion got a deterministic backstop instead of a better prompt: the matched and missing lists partition the requirements, so an entry appearing in both is the screening contradicting itself. That is a set intersection. It cannot be talked round, and it does not have an opinion.

NZ spelling got the same treatment for the same reason. A judge weighing nine criteria can miss a stray “organize”; a regular expression cannot. The lesson generalises: where a check can be made deterministic, a model should not be the one making it. The judge is for what code cannot assess. It is not a substitute for what code can.

The details, if you build this sort of thing

Four things that were not obvious to us and cost time, in case they save someone else some.

• Ask for citations, not scores. The single highest-value change was requiring the judge to quote each unsupported claim verbatim in a structured field. A score is a summary, and summaries are where findings go to die. A quote is checkable, and it made the failure mode visible in one run rather than over months.
• Bound your scores at runtime, not only in the schema. We tried to constrain each score to 0–5 with minimum/maximum in the JSON schema. The structured-output endpoint rejects those keywords on an integer property outright - HTTP 400, before any grading happens. The bound now lives in the prose description and in a runtime validation pass, so an out-of-range score fails loudly rather than quietly skewing a mean.
• Scope a deterministic check to what the model actually wrote. Our NZ spelling check runs over the generated letter, the CV change suggestions and their rationales, and the strategy notes. It deliberately does not scan the candidate’s own CV text - their spelling is their business, not a gate failure - nor the ATS matched/missing lists, which echo the job ad’s wording back. A deterministic check with the wrong scope produces confident, wrong failures, which is worse than not having it.
• Enforce the judge/writer split in code. Ours reads from configuration, and configuration drifts. If the two resolve to the same model the gate throws rather than running, because a gate that silently degrades into self-assessment is worse than no gate - it produces the same green output while measuring nothing.
• Expect the failure to migrate. Each time we closed one output channel, the overstatement reappeared in the next unchecked one - the letter, then the CV suggestions, then the screening, then the strategy notes. A verification pass that reads some of the output is a map of where the invention will move next; it ended only when no channel was unread.

The judge call itself uses extended thinking and structured output at a moderate effort setting. Grading is the harder half of the job, so it runs on a more capable model than the writing does - the economics of that are the next section.

From the gate to every generation

Everything above runs when we change how Rolebird writes. For most of this product’s life that was the whole story, and an earlier version of this article said so plainly: the gate is a regression test, and your individual letter was not checked.

That stopped being true in July 2026, because the failures on this page kept teaching the same lesson: a rule in the prompt shrinks a failure; a mechanism eliminates it. So the mechanisms moved into the generation itself. Every requirement the screening marks as covered must carry a quote from your CV, verified by code to actually be there. Every suggested CV change must quote the text that backs it, or it is discarded. And a separate verification pass reads the finished application - letter, changes, screening, strategy - against your documents, with its own instructions and no stake in the writing: claims your documents do not support are removed, and a letter that cannot be repaired honestly fails rather than being delivered.

The limits, stated the same way as everything else here: the verification pass is a model, and where the honest boundary of a phrase sits, two careful readers can disagree. The gate still runs against a sample. What cannot happen silently is a claim with nothing in your documents behind it.

Which is still the real reason Rolebird shows you a list of suggested CV changes rather than silently rewriting your CV, and never submits anything on your behalf. The last check is a human who knows what actually happened in that job, and it should be. The full method is here.

Why this matters beyond one product

It would be convenient to present all of that as an industry problem we have solved. It isn’t, and the published research is worth reading without a vendor’s framing on it. Greenhouse surveyed 4,136 people in late 2025 - 2,900 job seekers and 1,236 recruiters and hiring managers.^[2]

The pair that should stop everyone is the trust gap. Both sides have automated. Neither trusts the other’s automation.

That last bar is the uncomfortable one, because it is not laziness. If AI screeners favour AI-written résumés by the margins in the first study, then using AI to write your application genuinely helps you get past AI screening. People are responding correctly to how they are being assessed. It is individually rational and collectively it destroys the signal everyone depends on - including theirs.

In New Zealand specifically, applications per advertised position are up 261% since 2022 while job creation fell - averaging 43 candidates a job, and 150–300 on heavily contested ones.^[3] The binding constraint stopped being volume some time ago. It is signal.

The part we won’t assert

The tempting conclusion is that competing tools use cheaper models and skip verification, and that this is what is flooding the market. We are not going to claim that, because we cannot see inside anyone else’s product. We don’t know what models they run, what their prompts say, or what checks sit behind them - and an argument that unverified claims are the problem cannot be built out of unverified claims.

What is checkable is what a product tells you it does. A tool whose pitch is submitting applications on your behalf, at volume, while you do something else has made a specific design choice.

It has removed the step where a human who knows their own history reads what is about to be sent under their name. That is not an inference about their engineering. It is the feature.

The other checkable thing is the public record. We have published our criteria, our thresholds, our floors, and the two runs that passed while still containing unsupported claims. We would genuinely like to be able to link to equivalents from others in this category - if you publish one, we will link it here.

If you take one thing from this

Run the superhero test on whatever you use. Then read what it wrote about you, line by line, and ask of each sentence: could I defend this in an interview?

The applications that work in a market this noisy are the ones a human vouched for. That is inconvenient, it does not scale, and it is the whole point.