The Benchmark for Retrieving Information in EHRs (BRIE) is an automatically generated QA dataset for EHRs, validated by 19 clinicians and continuously maintainable.
- 19 clinicians validated benchmark
- 9 LLMs evaluated
- Multi-document synthesis often missed
BRIE is created by a scalable framework that extracts question‑answer pairs from longitudinal notes, addressing the high cost and rapid obsolescence of manually curated benchmarks. Evaluation across nine large language models and five inference strategies shows that state‑of‑the‑art systems frequently omit clinically important details, especially when answers require synthesis across multiple documents and encounters. The dataset is designed for ongoing updates as clinical language evolves.





