All posts
AI Performance2026-11-108 min readMartin Zillmann

Finding Inconsistencies Before You Publish: How Four AI Assistants Compared on a Complex Regulatory Document

AI review before publication can catch errors that are costly to fix afterward. Not all AI assistants perform equally.

Once a document is published, its errors are public. A dose stated one way on page 12 and another way on page 40, or a percentage that does not match its table, can lead to a correction, an awkward question from a reader, or, in serious cases, a retraction. Fixing these problems before publication is far easier than explaining them afterward.

This post makes one practical argument. AI assistants add the most value in the review stage before publication. There, they can act as a second reader that checks a draft for contradictions while there is still time to fix them. They do not replace the author or the human reviewer. A small benchmark we ran, however, shows that tools differ widely in how well they do this job, so the choice of tool matters.

The problem: the same fact, stated in several places

Long documents with many authors and repeated revisions are hard to review. The same fact (a dose, a count, a date, or a percentage) is often stated in several places, and small differences creep in. Checking by hand is slow and easy to get wrong. This kind of repetitive cross-checking is a natural task for AI assistants.

What the research says

Benchmarks suggest that document review is still hard for AI assistants, and that results depend on the benchmark and the model. In ContraDoc, GPT-4 performed best and could outperform human annotators, but it remained unreliable on subtle contradictions (1). In FIND, GPT-5 recovered 64% of expert-inserted inconsistencies, and even the best models missed almost half (2). In a peer-review study, detection of planted flaws ranged from 17.0% to 60.5% across four models (3). Studies on financial and regulatory documents report promising results (4, 5), though on different documents and tasks than the one tested here.

What we tested

Four AI assistants received the same prompt and the same input: the FDA’s multidisciplinary review document for NDA 214665, the LUMAKRAS (sotorasib) application (6). The assistants were bystro THiNK (Plus v1), Claude Opus 5.5, ChatGPT GPT-6 Astra, and Gemini Pro 3.1.

Each assistant was asked to find verifiable logical or numerical inconsistencies, where the same information appears differently in different parts of the document. Minor differences such as rounding and unit abbreviations were excluded. Each assistant also had to give page numbers for both occurrences, so that a person could check every item. Information contained only in images was excluded.

NDA 214665 is a complex document with body text, table text, figure legends, and images (Table 1).

MeasureValue
Body words (top-level paragraphs, excluding tables and text boxes)55,717
Table words (117 top-level and 22 nested tables, 139 in all)17,915
Body plus table words73,632
Text-box words (76 text boxes)1,005
Top-level body paragraphs5,231
Unique embedded image files (JPEG/PNG)210
Table 1. Summary statistics for NDA 214665.

We analyzed each assistant’s output in three ways:

  • Count: the number of items each assistant reported, and how many items were shared across assistants after matching.

  • Verification: each reported item was checked by hand against the source text and labeled correct or incorrect.

  • Overlap: items were matched by page reference and description to see which assistants reported the same item.

Because the true number of inconsistencies in the document is unknown, true recall cannot be calculated. We used pseudo-recall instead: the number of correct items for an assistant divided by the 47 unique items found across all four assistants. Pseudo-recall is likely to overstate true recall, because the true total is probably larger than 47.

What we found

Table 2 summarizes the results. bystro THiNK returned the most correct items: 33 of the 35 items it reported were correct. Claude Opus 5.5 had 30 correct items out of 36 reported, ChatGPT GPT-6 Astra had 13 of 16, and Gemini Pro 3.1 returned only 3, all of them correct. Figure 1 shows the correct and incorrect items for each assistant. Figure 2 shows how the items overlap.

Bystro THiNK (Plus v1)Claude Opus 5.5ChatGPT GPT-6 AstraGemini Pro 3.1
Reported3536163
Correct (verified)3330133
Incorrect2630
Pseudo-recall (%) *70.263.827.76.4
Share correct (%) †94.383.381.3100.0
Table 2. Raw counts for discrepancies identified across AI assistants. *Pseudo-recall = correct items ÷ 47, the number of unique items across all four assistants (the same 47 for every column). The true total in NDA 214665 is unknown and likely larger than 47. † Share correct = correct items ÷ reported items, calculated for this post from the counts above. The Gemini Pro 3.1 values are based on only three items.
Figure 1
Figure 1. Correct (+) and incorrect (−) discrepancies identified by each AI assistant.
Figure 2
Figure 2. Overlap of discrepancies returned by the four assistants, with counts for each region. Set areas are not drawn to scale. One of the three Gemini Pro 3.1 items is shared with all three other assistants, and two are shared only with bystro THiNK.

Three points stand out.

The assistants differed widely in output. Gemini Pro 3.1 returned far fewer items than the others. This study cannot say whether that reflects the tool, its settings, or the single run we performed.

The highest count was not the highest precision. bystro THiNK returned the most correct items, but 2 of its 35 reported items were incorrect, and Claude Opus 5.5 returned 6 incorrect items out of 36. The incorrect items shared a pattern: they could be rationalized using other information in the document, so they were easy mistakes for any reviewer to make. They were also often shared by two or more assistants. An AI flag is a lead to check, not a finding.

No single tool found everything. No assistant returned all 47 unique items. With the exception of Gemini Pro 3.1, each assistant returned correct items that no other assistant returned. This suggests that one run of one tool can miss items that another tool or another run would catch.

What AI review caught in this draft before publication

While revising this post, an AI-assisted read-through of the draft found the kinds of problems that a pre-publication review should catch. Each item was checked against its source:

  • Typos and naming. One AI assistant was spelled two different ways (“Genini” and “Gemini”), and the same tool appeared under three different names.

  • Factual claims. The draft described one cited study’s results as unquantified. That paper reports quantitative results, so the sentence was corrected.

  • Unsupported claims. The draft said the differences were “significant” and that AI review would “reduce error rates in published documents dramatically.” No statistical test was run, and the study did not measure error rates in published documents. Both claims were softened.

  • Tone. Sweeping phrases, such as AI assistants being “indispensable”, were replaced with statements the evidence supports.

  • Formatting. Drafting instructions, an empty reviewer section, and an incomplete sentence remained in the draft.

None of these fixes required new data. They required a careful reading of the text against its sources, which is the kind of cross-check AI assistants can perform quickly.

Where AI review falls short

AI review complements human judgment and accountability. It does not replace them. In this benchmark, even the best tool returned incorrect items, and no tool returned everything. A person still decides what counts as an inconsistency, checks each flag against the source, and is accountable for what is published.

The study has clear limits. A single analyst judged correctness by hand, so the labels could contain errors. Each assistant was run once, and outputs vary between runs. The results describe one document at one point in time, and model versions change. No statistical tests were run, so the differences are descriptive. An earlier analysis with a less specific prompt gave qualitatively similar results.

Final thoughts

Not all AI assistants are equal for this task, and the gap between the best and worst performers was larger than many professionals might expect. bystro THiNK was designed for accuracy and uses an agentic swarm to identify and cross-verify discrepancies. In this study it returned the most correct items, but it was not perfect, and it should be treated as a second reader, not the last one.

Standard AI benchmarks test general reasoning. For researchers, scientists, and technical writers, a check on a real document answers a more practical question: will this tool find the problems that matter in my document? Task-based, manually verified comparisons like this one are a useful complement to standard benchmarks.

A practical pre-publication routine

  1. Before publishing, run your near-final draft through an AI assistant with a specific prompt, asking it to list each repeated fact and where it appears.
  2. Check every flag against the source. Keep real issues, drop false alarms, and note why.
  3. Ask a human reviewer to check the claims, figures, and names that an AI tool cannot judge in context.
  4. Verify cited studies and their numbers against the original papers.
  5. Remember that you are responsible for what you publish.
Finding an inconsistency in your own draft is far better than hearing about it from a reader. Have you used AI review before publishing? What did it catch?