In 2017, a team of roughly 100 scientists, clinicians, bioinformaticians, and pathologists at Memorial Sloan Kettering Cancer Center published a landmark study in Nature Medicine. They had sequenced tumors from over 10,000 patients across 59 primary cancer types using a clinical platform called MSK-IMPACT. The paper has since accumulated more than 2,000 citations. It is, in short, the kind of work that sets the standard.
That made it the perfect benchmark.
A bystro AI white paper recently described an exercise in what you might call “rediscovery” — taking that same publicly available dataset, asking both an experienced human analyst and bystro THiNK (an agentic AI research platform) to independently reproduce the core analyses, and then comparing every single output.
The question was direct: Can an AI produce the same result as a skilled human analyst, from the same raw data, working from plain language instructions?
The publicly available MSK-IMPACT dataset on cBioPortal consists of 77 files totaling 63.4 megabytes of data. Four primary files were used: clinical sample data, mutation data, copy number alteration (CNA) data, and structural variant (SV) data, collectively containing over 12 million individual data points.
Both pipelines, the human using Excel and Python, and bystro THiNK using natural-language prompts, followed the same general process: transform each alteration file into a standardized format, combine them, merge in clinical data, recode alteration types to match the original paper’s categories, and exclude 30 patients whose records had conflicting cancer type assignments. Those 30 patients (65 samples) were removed specifically because counting cancer types at the patient level requires each patient to have exactly one cancer type, and no clear rule exists for choosing among multiple cancer types. Excluding them changed any summary statistic by less than 1%.
The result was a “master data table” of 104,705 rows.
The Core Finding: Manual = bystro THiNK ~ MSK-IMPACT
The bystro THiNK-generated master data table and the manually produced master data table were 100% identical — every row, every value, across all 104,705 rows (95% confidence interval: 99.9965% to 100%). Under standard statistical methods, this places a 95% upper bound of 0.003% on any undetected error rate which is the equivalent of no more than 29 errors per million data rows.
To put that in context: recent studies of leading AI systems on structured biomedical tasks have reported accuracy rates ranging from around 50% to 93%, depending on the task and dataset. Clinical trial matching tools, which are more constrained than open-ended data analysis, have achieved criterion-level accuracy of 87–93%. Tumor board decision tools have reported roughly 70% case-level agreement with expert oncologists. bystro THiNK’s performance on this structured data aggregation task was qualitatively different. It wasn’t just good, it was indistinguishable from manual analysis.
| Metric | Primary Data (cBioPortal) | bystro THiNK | Manual | DELTA (bystro vs manual) |
|---|---|---|---|---|
| Patients | 10336 | 10306 | 10306 | 0.00% |
| Samples | 10945 | 10880 | 10880 | 0.00% |
| Cancer Types | 58 | 58 | 58 | 0.00% |
| Samples with Mutations | 10129 | 10068 | 10068 | 0.00% |
| Genes with Mutations | 414 | 414 | 414 | 0.00% |
| CNA Samples | 5219 | 5195 | 5195 | 0.00% |
| SV Samples | 1667 | 1659 | 1659 | 0.00% |
While bystro THiNK and the manual analysis agreed perfectly with each other, both told a somewhat different story than the originally published manuscript, and that’s worth talking about.
To be clear: the white paper is explicit that this is not a criticism of the original research team. There are entirely reasonable explanations. The data deposited in cBioPortal may not be identical to the data that went into the final manuscript. Cancer type recoding decisions made by a 100-person team over multiple years are not always documented in the supplementary materials. Threshold choices (like whether to include samples with a CNA score of 0) can shift counts meaningfully without being wrong. The white paper’s analysis takes the publicly available data at face value, making as few assumptions as possible.
That said, the discrepancies are real, they are quantifiable, and they illustrate exactly why reproducibility in large-scale genomics research is hard.
The most striking example: Non-Hodgkin Lymphoma.The published paper’s Figure 2a lists 159 patients with Non-Hodgkin Lymphoma. The manual analysis of the data_clinical_sample.txt file returns exactly 5 patients with that cancer type. Five. The same five were identified independently by bystro THiNK — a 97% discrepancy for a single cancer type between what’s in the primary data file and what appeared in the published figure.
| Cancer Type | Published Fig 2a Count | Manual and bystro Count | Delta | Absolute Delta% |
|---|---|---|---|---|
| Bone Cancer | 42 | 129 | 87 | 207% |
| Non-Hodgkin Lymphoma | 159 | 5 | 154 | 97% |
| Embryonal Tumor | 76 | 6 | 70 | 92% |
| Gestational Trophoblastic Disease | 11 | 3 | 8 | 73% |
| Hepatobiliary Cancer | 242 | 345 | 103 | 43% |
| Ampullary Cancer | 25 | 16 | 9 | 36% |
| Gastrointestinal Neuroendocrine Tumor | 53 | 44 | 9 | 17% |
| Cancer of Unknown Primary | 160 | 181 | 21 | 13% |
Source: Appendix, Table 4. Cancer types with count discrepancies greater than 10%.
Several cancer types that appear in the published manuscript’s figures (Hepatocellular Carcinoma, Ewing Sarcoma, Chondrosarcoma, Meningothelial Tumor, Ependymomal Tumor, Chondroma, and Breast Sarcoma) are entirely absent from the CANCER_TYPE field in the primary data. Both the manual analysis and bystro THiNK correctly excluded them for that reason. Similarly, names used in the paper differ from names in the data: what the paper calls “Biliary Cancer” is “Hepatobiliary Cancer” in the source file; “Osteosarcoma” in the paper is “Bone Cancer” in the data.
At the aggregate level: the manuscript states that the cohort represents 62 principal cancer types and 361 detailed tumor subtypes. The primary data file, as analyzed here, contains 58 unique cancer types and 359 unique detailed subtypes. Summary tables downloaded from the MSK_IMPACT study Summary tab on cBioPortal are in agreement with the manual analysis.
These discrepancies don’t undermine the original research — the scientific findings of MSK-IMPACT stand on a much broader foundation than any single count in a summary figure. What they do illustrate, compellingly, is how a large-scale research dataset evolves over time. Data gets cleaned, recoded, and curated between analysis and publication. Sometimes those decisions are documented, sometimes they’re implicit in the domain expertise of the team.
And that’s precisely where a tool like bystro THiNK starts to look valuable not as a replacement for domain expertise, but as a reproducible, auditable second set of eyes. In a large collaborative study, if bystro THiNK and a primary analyst independently produce the same result, that’s a strong signal the analysis is right. If they don’t, that’s a flag for human review before the paper goes out, not after.
The white paper frames this as “supervisory quality control”: a parallel computational read that doesn’t require trusting the AI blindly but uses it as a reliable consistency check. In a field where large studies routinely involve data handoffs across institutions, format conversions, and iterative re-analyses, that kind of auditable redundancy has real value.
An AI platform, operating from plain language instructions on a 63-megabyte, multi-format clinical genomics dataset containing over 12 million data points, produced a master data table identical to one produced by an experienced human analyst: 104,705 rows with zero discrepancies. The error rate upper bound, statistically, is less than 3 in 100,000.
For researchers wondering whether agentic AI is ready to be part of serious scientific workflows, this is a data point worth knowing.