A recent study looking at a technology to improve the performance of 4 LLMs (GPT-3.5, GPT-4, Vicuna-13B, and ChatGLM-6B) measured their performance on 4 multiple-choice medical question datasets: MedQA (US Medical Licensing Examination), PubMedQA (original and revised versions), MedMCQA, and EYE500 (Wang, et al. 2025). The authors found that accuracy across the data set, without augmentation using their technology, varied between 28-68% across the models and hallucinations varied between 35-68% across the models. Among hallucinations, the main types were factual inconsistencies (42%) and logical inconsistencies (24%).
With this backdrop, it is important to demonstrate that AI models can provide accurate results when given complex tasks. bystro THiNK, while containing elements of LLMs, utilizes four core agents in its workflow:
bystro THiNK was purpose-built to perform complex scientific tasks and should inherently have better performance. One of the key features of THiNK that should increase reliability is the cross-verification for the results. The actual performance of the THiNK tool will be explored below.
This evaluation was performed using a publication exploring the miRNA transcriptome and proteome in a Pompe disease mouse model (Merberg D, et al 2024) and the associated dataset available from Zenodo. This manuscript was selected because understanding miRNA and protein expression patterns in the Pompe disease mouse model involves the analysis of very large data sets using multiple tools to align and clean the raw data and multiple statistical tools to derive meaning from the data.
The initial prompt was:
bystro responded with the clarifying questions below rather than jumping into analyses as many similar systems do.
Here are a few questions that could be helpful to address:
Q1. Do you have the raw or processed data files (e.g., miRNA expression counts, proteomics abundance tables) from this study that you'd like me to analyze, or should I focus on reviewing the paper's methods and findings and attempt to replicate them using any publicly deposited data (e.g., from GEO or PRIDE/ProteomeXchange)?
Q2. When you say "replicate the findings," do you mean: (a) a full computational replication using the original or deposited data (re-running the statistical analyses from scratch), (b) a critical review of their methods and results to assess reproducibility and validity, or (c) applying their analytical approach to your own dataset? This will significantly affect the scope of the work.
Q3. If you do have your own data, could you share or describe the file formats and contents (e.g., normalized miRNA read counts matrix, TMT/label-free proteomics abundance file, sample metadata with disease vs. control labels)? Knowing the exact format will help me set up the correct analysis pipeline.
The follow up prompt addressed some of these questions:
With this clarification regarding the user's intentions, the bystro agents began the analysis and construction of a response, which was completed in roughly 32 minutes.
Given the very small amount of information and guidance that was given in the prompt, it surprised me that the model really approached the problem much like a research scientist might, starting with retrieval of the data set, deciding what methods to use to clean and annotate the data, and, most importantly, deciding what statistical tools to use. For statistical tools, it is often very difficult to identify the appropriate statistical tool for a problem given the sample size, the type of measurement, and the effect being sought — this is probably the heaviest lift in this task. In addition, there was a task that was performed by the agents that was not explicitly requested in the prompt, a critique of the original work — a bonus of sorts.
Some of the tasks that were performed by the agents to complete the task are listed below.
Below are the main outcomes of the analysis copied directly from the Results:
I have to show the figure below that was generated during analysis because we all know what it would take to do this ourselves (see above). And, I am guessing, based on personal experience, that several person-days of effort were saved using THiNK, creating the opportunity to do other, even more important things.
Figure 1. Panels A and B show volcano plots for serum miRNAs and the serum proteome, respectively, where down-regulated species are to the left (blue) and up-regulated species are to the right (red) and the y-axis is log of the p-value for the confidence of the observation. Panel C shows the measured concordance between data published by Merberg D, et al 2024 and this analysis. Panel D is a fold-change for key miRNAs, where blue indicates down-regulation and red indicates up-regulation.
It is important to note the study limitations identified by bystro THiNK.
For me, writing this, it is a bit of a slide rule vs calculator moment. Yes, the slide rule will get the job done, but the calculator is so much faster and more reliable. bystro THiNK is clearly the calculator and me doing the analysis by hand using conventional tools is the slide rule. The real beauty of AI tools is that they can act as efficiency amplifiers in the hands of experts. Why experts, you ask? As Plato said, knowing what question to ask is often more important than knowing the answer. However, if the model's responses are only correct 80% of the time, it is likely that whatever efficiency gain was achieved by shrinking the analysis time will be offset by the time needed for re-analysis to identify the 20% incorrect results.
We should always keep in mind that, by removing ourselves completely from the analysis through the use of sophisticated tools such as AI agents and LLMs, we give up both the elements of process understanding and control. A quick example to highlight the point: in this post, without any prior understanding of the complex workflow behind the required analysis, we were able to produce a very intricate and persuasive result. But, if we needed to verify the results ourselves, full process understanding and the ability to perform all of the tasks using proven tools would be required.
For an AI tool to be useful as an efficiency amplifier, one has to know that the tool can be trusted and that the model's output will be accurate and without hallucinations. Hopefully, this posting about bystro THiNK's capabilities to replicate known outcomes is a move in that direction.
