AI tools have become indispensable for knowledge workers. They draft documents, summarize reports, answer questions, and increasingly, help researchers survey scientific literature. These tasks used to consume hours of focused effort. But there is a well-known shadow hanging over all of this productivity: hallucinations.
An AI hallucination is not a small rounding error. It is when an AI assistant fabricates something such as a paper that never existed, an author who never wrote it, or a journal that never published it and then presents it with the same confident fluency as something completely accurate. In most professional settings, this is an inconvenience. In academic research, pharmaceutical development, regulatory writing, or government policy work, it can be far more damaging. A fabricated citation used in a drug submission, a legal brief, or a published paper is not just embarrassing, it can mislead downstream decisions that affect real outcomes.
We recognized a specific and practical gap: existing AI benchmarks tend to measure things like math reasoning, coding ability, or trivia recall. Very few evaluations test whether an AI can be trusted to produce a factually grounded, comprehensive, and verifiable literature search – a task professionals actually need done well. This evaluation was designed to examine this gap.
Each AI assistant was given the exact same prompts, asking it to generate literature searches across four technically demanding domains:
The prompts varied in style, ranging from a simple one-sentence request to a full LaTeX-formatted methods section introduction, mimicking the kinds of real inputs a researcher or engineer might actually provide. When an AI assistant's output was unclear about format, we requested a BibTeX file. Gemini, which initially returned very few references, was given a minimal follow-up prompt requesting more sources. Every single returned reference was then manually evaluated in two stages:
This is a rigorous, no-shortcuts evaluation that is an open-ended, expert-verified, assessment of factual reliability in a domain that matters to researchers.
AI assistant testing reveals a wide performance gap in their ability to gather and display the results of literature searches.
The three panels above quantify important features of AI assistant performance:
All of the models characterized in this work are good AI assistants for many applications, but it is important to select the right tool for the job: it is not effective to bash on a nail with a pencil since a hammer works better, but try writing with a hammer…
This benchmarking study makes a clear, practical case: when it comes to generating scientific literature searches, not all AI tools perform equally, and the differences are large enough to matter in practice. Bystro THiNK demonstrated both the broadest coverage and the highest factual precision with 177 real references, zero hallucinations, and zero errors. Its closest competitor returned fewer than one-third as many references and still had citation errors that a human reviewer would need to catch.
The broader implication is equally important: standard AI benchmarks that dominate headlines about GPT and Gemini performance are largely measuring the wrong things for researchers, scientists, and technical writers. Getting the right answer on a standardized reasoning test is not the same as reliably synthesizing and accurately citing real scientific evidence.
The task-grounded, expert-verified, benchmarking evaluation presented here represents an important complement to the standard evaluation landscape. For anyone using AI in technical or scientific work, it is this kind of evaluation that should matter most.
