Reproduction report · July 2026
Fact-checking a medical AI paper · In plain language
We double-checked an AI that reads CT scans. Here's what held up.
Six big promises, one clever idea, and three missing pieces
A research paper claimed it could teach AI to write radiology reports — the documents a doctor writes after your scan — that don't make things up. Big claim. So we tried to rebuild their work from what they shared. Some of it checks out. The biggest numbers can't be checked at all, because key files were never released. This report tells you which is which.
From rollout to results.
01 · Why this matters to you
An AI that grades its own homework — honestly
If an AI ever helps write your radiology report, you want it punished for inventing problems and for missing real ones. That's exactly what this paper tries to do.
Hospitals are short on radiologists, and AI is starting to help draft scan reports. But today's AIs have a dangerous habit: they can write smooth, confident sentences about problems that aren't there — or skip problems that are. Both mistakes can hurt patients.
The paper we checked proposes a smarter way to train these AIs: a built-in report card that scores every sentence against a medical checklist — did it name the right body part? the right problem? did it invent anything? The AI trains to earn better scores, so it learns to be careful instead of just fluent.
Our job was simpler and more boring: fact-check it. Rebuilding the paper's experiments from shared materials is how science stays honest. Here's the short version:
- The core idea works. The report-card software the authors shared does exactly what their paper says it does.
- The big numbers can't be checked. The trained AI and its training recipe were never shared, so their headline results remain unverified — not proven wrong, just not checkable.
What we did vs. what a full redo would take
| What we did | Full redo (we didn't) |
|---|
Throughout this report, "we couldn't check it" means the materials weren't shared — it does not mean the paper is wrong.
02 · The scoreboard
Six promises, three kinds of answer
Each card is one promise from the paper, with our verdict. Click any card for the full story — what was promised, what blocked us, and what we tested instead.
- Confirmed — we tested it ourselves and it behaved as described
- Partly confirmed — the idea checked out at small scale; the full-size numbers remain untested
- Couldn't check — the materials needed to test it were never shared
The bars above are the paper's own grades, drawn with a striped, dashed "not verified" style on purpose: we could not recompute any of them. They're shown for context, and each is labeled with its source table.
03 · What we confirmed
The report card works the way the paper says
This is our strongest result: the scoring software the authors shared installs, runs, and behaves exactly as their central equation describes.
Think of the report card as two coaches watching the AI write, sentence by sentence:
- The miss-catcher — notices when the report forgets a real problem. Like a teacher marking "you skipped question 4."
- The invention-catcher — notices when the report makes something up. Like a teacher marking "this answer came from nowhere."
The paper's Equation 4, translated. We checked every piece against the shared code — it matches.
To test it, we fed made-up example reports into the real code, switching each coach on and off. The pattern matched the paper perfectly: each coach fired on exactly the mistake it was built to catch.
One honest limit: the paper's exact table numbers come from trained AIs, which we couldn't run. What we confirmed is the machinery underneath them — the part everything else depends on.
04 · The pop quiz
One invented problem, three judges
The paper claims its grader ranks reports the way a doctor would. We gave three grading methods the same five-report pop quiz — and watched word-matching reward a fib.
We wrote five versions of the same scan report. One perfect. One honest but missing a small problem. One honest but missing the big problem. One bare-bones. And one trap: a report that invents a problem that isn't there.
Then three judges ranked them: the paper's medical-checklist grader, and two classic methods (BLEU and ROUGE) that just count how many words match a reference report — the way a spell-checker "understands" a sentence.
The trap worked. Word-counting ranked the fibbing report above the more honest one — because fluent nonsense shares lots of words with the real thing. The paper's grader correctly dropped it two places. That single flip is the paper's whole argument, and we watched it happen with our own test.
Scale matters, so be careful: the paper tested thousands of real reports with an AI judge and found its grader agreed with doctors ≥90% of the time vs 65–75% for word-matching. Our five-report quiz confirms the direction of that claim, not its size.
05 · The test drive
We couldn't test the real AI — so we test-drove the plumbing
The paper's AI wasn't available. So we pointed a free, weaker stand-in at 20 real scans to prove the whole pipeline works end to end — and to give the next person a baseline to beat.
Picture it like testing a recipe with supermarket ingredients because the fancy ones are sold out. The meal won't taste the same, but you learn whether the oven works, the steps make sense, and the timer is right.
That's what this run was. In about 7.5 minutes on a free GPU, the stand-in AI read 20 real scans (3 slices each) and produced reports we could score. The pipeline — load scan, pick slices, write report, grade report — works.
Read these numbers carefully: the near-zero medical-checklist scores mean the stand-in AI is weak — a small model looking at a few 2D slices of 3D scans. They say nothing about whether the paper's method works. That's the point of a plumbing test.
06 · What's missing
Three missing pieces block four of six promises
Every unchecked promise in this report traces back to the same three gaps between the paper and what its authors actually shared.
A full redo is a genuinely big project: retrain a large AI on three medical datasets (one of them locked behind an application), on expensive computers, for days or weeks, at a cost of thousands of dollars — then rebuild the expert answer keys before you can even grade anything.
The good news: because the report card itself checks out, that expensive project is worth doing. The bad news: nobody can start it until the authors share the three pieces above.
07 · Check it yourself
Everything we ran is free to download and re-run
We published a complete bundle — our code, test scripts, data files, charts, and the test-drive results — as sisaacson/tif-grpo-repro-bundle. The main check takes under five minutes on an ordinary computer:
python verify_reward_and_concordance.py python generate_figures.py python scripts/stream_infer.py python scripts/evaluate_reports.pyThe bottom line
- The paper's core machinery is real. The report card behaves exactly as described — that's confirmed.
- The headline numbers are unverified, not refuted. They rest on files that were never shared.
- If the authors share the three missing pieces, every promise in this report becomes checkable. Until then, treat the big numbers as promising claims — not proven results.
Where everything comes from
Prepared by Techne Path · Logbook updated 2026-07-22 · Set in IBM Plex Sans ·
Original paper: arXiv:2605.20277 · Shared code: github.com/ZJU4HealthCare/TIF-GRPO · Every chart on this page is clickable — tap any bar, dot, or number for its source.