I recognize I’m biased, but I prefer the machine learning review system. Unlike many other fields (in statistics reviews can take a year!), in ML you submit and within 3 months you get a clear answer on whether your paper will be accepted or not. The double blind system reduces the salt in other fields, and if you’re rejected just modify and resubmit in a couple of months. This speed is due to the field’s requirement that authors also review papers, which also means that reviewers are highly busy (most of the other reviewers for papers I considered sent in their reviews the last day). As a result, reviewers are busy and decisions are rarely revisited even when authors write rebuttals.
One of our submissions, “Evidence Against Homogeneity: Identifying Process Mismatch for Copy Number Variation Detection,” scored a 3.5 initially (between weak and clear accept), with the lowest score a 2 (weak reject). The paper was likely to get in as-is, but after so much effort, we still wrote rebuttals (good practice) with Bystro’s assistance. Surprisingly, we were able to raise that score from a 2 to a 3. We wanted to describe how an LLM assisted us with one of the most miserable tasks in publishing.
You can read the paper yourself here. The TL;DR is that sometimes customers for CNVs group unrelated samples together for CNV calling. As a result, they’re highly unmatched resulting in erratic calls. We cluster the samples in the Bayesian evidence space and test whether there is more than one cluster present. If there is, it’s indicative of a process mismatch and requires further QC.
The review itself was constructive but pointed. The reviewer liked the writing and the breadth of datasets, but raised four concerns:
We gave Bystro the review and asked for help drafting the rebuttal. Note that in the prompt I provided some suggestions for directions to push back on. Namely:
Instead of immediately generating text, it came back with clarifying questions:
Figure 1: Before drafting a single line, Bystro asked scoping questions about what kind of help we wanted, which venue and length constraints applied, and whether other reviews should be considered, rather than guessing.
We told it plainly: we needed a full rebuttal, capped at 5,000 characters plus a title, addressing all four concerns, and that running new experiments wasn't realistic since this was an ML venue rebuttal window, not a new research cycle. That constraint mattered. It meant Bystro had to work entirely from evidence already in the manuscript, the same limitation we were facing ourselves.
Figure 2: The rebuttal Bystro produced, posted as our official response on OpenReview with one section per concern, each tied back to specific tables and figures in the manuscript.
What came back was structured the way a good rebuttal should be: one clearly labeled response per concern, each grounded in specific sections, tables, and figures rather than general reassurance.
For the biological-versus-technical separation concern, the response pointed directly at the within-batch subgroup analyses — in the thalassemia data, 48 of 94 samples carried CNVs, a substantial biological signal, yet the within-batch p-values stayed far from significance while the cross-batch tests on the same data were clearly significant. That contrast, stated plainly, is the whole argument: the method reacts to technical heterogeneity, not to biology, even when biological variation is strong.
For the 1-vs-2 components concern, rather than defending the limitation apologetically, the rebuttal reframed it as a deliberate design choice tied to real clinical batch sizes and the power constraints already documented in the paper's own figures, while also pointing out that the test still detects heterogeneity even when more than two subgroups exist — it just doesn't try to enumerate them.
For the missing baseline comparisons, it turned out we did have them, and the reviewer had simply missed the relevant table. The rebuttal pointed straight at the quantitative clustering comparisons against five established baselines, where our method matched or beat all of them across three datasets.
And for the electrophysiology dataset feeling disconnected, the response reframed it as the generalization test it was meant to be which was a demonstration that the method works from per-sample likelihoods in a completely different domain, not a genomics add-on that happened to wander into the paper.
None of this required new analysis. It required organizing what already existed into a response that a skeptical reader could follow concern by concern.
Writing a rebuttal well, under a deadline, while trying not to sound defensive, is genuinely hard emotionally as much as intellectually. Someone has just told you, in writing, that a claim you're proud of wasn't examined carefully enough. It's easy to either over-apologize or get combative, and neither serves you well with an area chair reading over the exchange.
What stood out about using Bystro here wasn't that it invented new arguments — it can't manufacture evidence that isn't in the paper. What it did was ask the right scoping questions upfront, then move fast from “here is our data and our draft rebuttal” to a structured, section-referenced response we could edit and stand behind. That's a meaningfully different task than staring at a blank page trying to remember which table had the number you need.
Reviewer scores moving in the ML community is rare enough that we don't want to oversell a single data point. But if a tool can reliably help turn a fair, pointed critique into a rebuttal that a reviewer reads and actually reconsiders, that's a real contribution to a part of the publishing process almost nobody enjoys.
