What the benchmark tests
Retrieval from long labels, exact warning and dosage extraction, multi-section synthesis, cross-label comparison, mixed batches, and calibrated refusal for near-miss questions.
LLM evaluation and grounded clinical QA
FDA-Gemini Benchmark
A reproducible benchmark for testing whether LLM agents can reason over FDA drug labels while preserving evidence, clinically important qualifiers, and refusal discipline.
The project moves beyond answer similarity with isolated Harbor tasks, hidden deterministic verifiers, fractional rewards, and trajectory-level failure analysis.
- FDA labels profiled
- 704
- source QA rows
- 17,207
- Harbor task suites
- 12
- scored hard-set trials
- 23
- mean fractional reward
- 0.700
Independent research benchmark. The 0.700 aggregate covers 23 recorded scored hard-set trials; the plots below compare a narrower three-run Gemini subset. Benchmark outputs are not medical advice.
01 / Problem
Correct words are not enough
In label-grounded QA, an answer can sound plausible while citing the wrong product, dropping a population restriction, changing a dose, or answering a claim the source never supports.
What counts as grounded
Every expected question ID must be present in valid JSON, cited passages must belong to the correct label, quotes must occur in those passages, and clinical claims must retain their qualifiers.
02 / Pipeline
From corpus to diagnosis
The workflow keeps data profiling, task construction, execution, scoring, and error analysis reproducible as separate stages.
01
Profile the corpus
Inspect 704 FDA labels and 17,207 QA rows to map label length, task type, citations, and refusal coverage.
02
Construct hard tasks
Generate seven challenge families spanning long retrieval, warnings, dosage, refusal, synthesis, comparison, and mixed batches.
03
Separate hidden gold
Expose questions and label passages while keeping expected facts, qualifiers, and citation checks inside hidden verifiers.
04
Execute with Harbor
Package each suite as an isolated Harbor environment and preserve job configuration, trajectories, timing, cost, and rewards.
05
Score partial correctness
Combine strict pass/fail with a deterministic fractional rubric for structure, content, qualifiers, citations, and refusals.
06
Analyze failure modes
Trace deductions back to answer content, clinical qualifiers, citation support, critical errors, and refusal decisions.
03 / Verifier
Strict outcomes, useful gradients
Binary pass/fail protects the safety-critical threshold, while fractional reward reveals whether a failure came from structure, content, qualifiers, citations, or refusal behavior.
Answer content
35%Clinical qualifiers
20%Citation support
25%Refusal correctness
10%Schema and coverage
10%A run passes only at an aggregate score of at least 0.85 with no critical errors. Wrong dose, unit, route, population, cross-label citation, unsupported answer, invalid JSON, or question-ID mismatch can force a failure even when partial reward remains high.
04 / Results
The hard set separated behavior
Refusal behavior was strongest. Warning retrieval and long-label retrieval were hardest, while dosage and mixed-batch outputs often looked close but missed strict qualifiers or structured fields.
| Task family | Trials | Mean reward | Pass@1 |
|---|---|---|---|
| Warning citations | 4 | 0.556 | 0.0% |
| Long-label retrieval | 4 | 0.565 | 0.0% |
| Cross-label comparison | 3 | 0.624 | 0.0% |
| Multisection synthesis | 3 | 0.751 | 0.0% |
| Numeric dosage | 3 | 0.801 | 0.0% |
| Mixed batch | 3 | 0.802 | 0.0% |
| Near-miss refusal | 3 | 0.897 | 100% |
05 / Evidence
Near-misses became visible
Fractional scoring exposed a gap that strict pass rates alone hide: outputs were frequently structurally valid and broadly relevant but still lost credit for content precision and clinically meaningful qualifiers.
06 / Judgment
Limits and next experiments
The benchmark is designed as an auditable research scaffold. Its claims stay bounded to the recorded corpus, verifier, and runs.
Current limitations
- The recorded hard-set aggregate contains 23 scored trials; it is a diagnostic study, not a broad model leaderboard.
- Plots comparing three Gemini runs use a narrower subset than the 23-trial aggregate and are labeled accordingly.
- Deterministic lexical and structured checks provide reproducibility but cannot capture every clinically equivalent phrasing.
- The benchmark evaluates answers against supplied labels; it does not validate medical safety or real-world clinical use.
Next experiments
- Run the same frozen suites across additional model and agent configurations.
- Add field-first validation for dosage and cross-label comparison before prose generation.
- Expand adversarial refusal cases and multi-document contamination checks.
- Report confidence intervals and repeated-run variance as the evaluation set grows.
07 / Repository
Inspect the benchmark
The repository includes the Harbor task environments, builders, hidden-verifier logic, tests, job artifacts, aggregate tables, and trajectory failure reports used for this case study.