OpenAI has introduced HealthBench, an open-source benchmark designed to assess the performance and safety of large language models (LLMs) in healthcare settings. Developed in collaboration with 262 physicians from 60 countries, HealthBench comprises 5,000 realistic health conversations, each evaluated using physician-created rubrics.

A Collaborative Effort

HealthBench was developed with input from a diverse group of physicians, ensuring that the benchmark reflects a wide range of medical practices and standards. The dataset includes multi-turn conversations that simulate real-world interactions between patients and healthcare professionals.

Evaluation Criteria

Each AI-generated response in HealthBench is assessed against a rubric designed by medical professionals, focusing on factors such as accuracy, relevance, and safety. These rubrics are weighted according to medical judgment, providing a nuanced evaluation of AI performance.

Model Performance

In initial evaluations, OpenAI’s o3 reasoning model achieved a top score of 60%, followed by Elon Musk’s Grok at 54%, and Google’s Gemini 2.5 Pro at 52%. HealthBench supports responses in 49 languages and covers 26 medical specialties, including neurology and ophthalmology.

Implications for Healthcare AI

HealthBench represents a significant step toward standardized evaluation of AI models in healthcare. By providing a comprehensive and medically-informed benchmark, it aims to enhance the reliability and safety of AI applications in medical contexts.

The benchmark is now available as an open-source resource, inviting further collaboration and development in the field of AI-driven healthcare solutions.