AI review before publication can catch errors that are costly to fix afterward. Not all AI assistants perform equally.
Once a document is published, its errors are public. A dose stated one way on page 12 and another way on page 40, or a percentage that does not match its table, can lead to a correction, an awkward question from a reader, or, in serious cases, a retraction. Fixing these problems before publication is far easier than explaining them afterward.
This post makes one practical argument. AI assistants add the most value in the review stage before publication. There, they can act as a second reader that checks a draft for contradictions while there is still time to fix them. They do not replace the author or the human reviewer. A small benchmark we ran, however, shows that tools differ widely in how well they do this job, so the choice of tool matters.
Long documents with many authors and repeated revisions are hard to review. The same fact (a dose, a count, a date, or a percentage) is often stated in several places, and small differences creep in. Checking by hand is slow and easy to get wrong. This kind of repetitive cross-checking is a natural task for AI assistants.
Benchmarks suggest that document review is still hard for AI assistants, and that results depend on the benchmark and the model. In ContraDoc, GPT-4 performed best and could outperform human annotators, but it remained unreliable on subtle contradictions (1). In FIND, GPT-5 recovered 64% of expert-inserted inconsistencies, and even the best models missed almost half (2). In a peer-review study, detection of planted flaws ranged from 17.0% to 60.5% across four models (3). Studies on financial and regulatory documents report promising results (4, 5), though on different documents and tasks than the one tested here.
Four AI assistants received the same prompt and the same input: the FDA’s multidisciplinary review document for NDA 214665, the LUMAKRAS (sotorasib) application (6). The assistants were bystro THiNK (Plus v1), Claude Opus 5.5, ChatGPT GPT-6 Astra, and Gemini Pro 3.1.
Each assistant was asked to find verifiable logical or numerical inconsistencies, where the same information appears differently in different parts of the document. Minor differences such as rounding and unit abbreviations were excluded. Each assistant also had to give page numbers for both occurrences, so that a person could check every item. Information contained only in images was excluded.
NDA 214665 is a complex document with body text, table text, figure legends, and images (Table 1).
| Measure | Value |
|---|---|
| Body words (top-level paragraphs, excluding tables and text boxes) | 55,717 |
| Table words (117 top-level and 22 nested tables, 139 in all) | 17,915 |
| Body plus table words | 73,632 |
| Text-box words (76 text boxes) | 1,005 |
| Top-level body paragraphs | 5,231 |
| Unique embedded image files (JPEG/PNG) | 210 |
We analyzed each assistant’s output in three ways:
Count: the number of items each assistant reported, and how many items were shared across assistants after matching.
Verification: each reported item was checked by hand against the source text and labeled correct or incorrect.
Overlap: items were matched by page reference and description to see which assistants reported the same item.
Because the true number of inconsistencies in the document is unknown, true recall cannot be calculated. We used pseudo-recall instead: the number of correct items for an assistant divided by the 47 unique items found across all four assistants. Pseudo-recall is likely to overstate true recall, because the true total is probably larger than 47.
Table 2 summarizes the results. bystro THiNK returned the most correct items: 33 of the 35 items it reported were correct. Claude Opus 5.5 had 30 correct items out of 36 reported, ChatGPT GPT-6 Astra had 13 of 16, and Gemini Pro 3.1 returned only 3, all of them correct. Figure 1 shows the correct and incorrect items for each assistant. Figure 2 shows how the items overlap.
| Bystro THiNK (Plus v1) | Claude Opus 5.5 | ChatGPT GPT-6 Astra | Gemini Pro 3.1 | |
|---|---|---|---|---|
| Reported | 35 | 36 | 16 | 3 |
| Correct (verified) | 33 | 30 | 13 | 3 |
| Incorrect | 2 | 6 | 3 | 0 |
| Pseudo-recall (%) * | 70.2 | 63.8 | 27.7 | 6.4 |
| Share correct (%) † | 94.3 | 83.3 | 81.3 | 100.0 |
The assistants differed widely in output. Gemini Pro 3.1 returned far fewer items than the others. This study cannot say whether that reflects the tool, its settings, or the single run we performed.
The highest count was not the highest precision. bystro THiNK returned the most correct items, but 2 of its 35 reported items were incorrect, and Claude Opus 5.5 returned 6 incorrect items out of 36. The incorrect items shared a pattern: they could be rationalized using other information in the document, so they were easy mistakes for any reviewer to make. They were also often shared by two or more assistants. An AI flag is a lead to check, not a finding.
No single tool found everything. No assistant returned all 47 unique items. With the exception of Gemini Pro 3.1, each assistant returned correct items that no other assistant returned. This suggests that one run of one tool can miss items that another tool or another run would catch.
While revising this post, an AI-assisted read-through of the draft found the kinds of problems that a pre-publication review should catch. Each item was checked against its source:
Typos and naming. One AI assistant was spelled two different ways (“Genini” and “Gemini”), and the same tool appeared under three different names.
Factual claims. The draft described one cited study’s results as unquantified. That paper reports quantitative results, so the sentence was corrected.
Unsupported claims. The draft said the differences were “significant” and that AI review would “reduce error rates in published documents dramatically.” No statistical test was run, and the study did not measure error rates in published documents. Both claims were softened.
Tone. Sweeping phrases, such as AI assistants being “indispensable”, were replaced with statements the evidence supports.
Formatting. Drafting instructions, an empty reviewer section, and an incomplete sentence remained in the draft.
None of these fixes required new data. They required a careful reading of the text against its sources, which is the kind of cross-check AI assistants can perform quickly.
AI review complements human judgment and accountability. It does not replace them. In this benchmark, even the best tool returned incorrect items, and no tool returned everything. A person still decides what counts as an inconsistency, checks each flag against the source, and is accountable for what is published.
The study has clear limits. A single analyst judged correctness by hand, so the labels could contain errors. Each assistant was run once, and outputs vary between runs. The results describe one document at one point in time, and model versions change. No statistical tests were run, so the differences are descriptive. An earlier analysis with a less specific prompt gave qualitatively similar results.
Not all AI assistants are equal for this task, and the gap between the best and worst performers was larger than many professionals might expect. bystro THiNK was designed for accuracy and uses an agentic swarm to identify and cross-verify discrepancies. In this study it returned the most correct items, but it was not perfect, and it should be treated as a second reader, not the last one.
Standard AI benchmarks test general reasoning. For researchers, scientists, and technical writers, a check on a real document answers a more practical question: will this tool find the problems that matter in my document? Task-based, manually verified comparisons like this one are a useful complement to standard benchmarks.