Artificial intelligence (AI) continues to revolutionize healthcare, promising improved patient outcomes, reduced operational costs, and more efficient clinical workflows. OpenAI’s innovative tool, HealthBench, marks a significant advancement by providing a standardized approach to evaluate the effectiveness and safety of large language models (LLMs) within practical healthcare environments.

What Is HealthBench?

HealthBench is an extensive open-source evaluation platform tailored specifically to assess the practical utility and reliability of large language models in diverse healthcare situations. It encompasses 5,000 realistic, multi-turn dialogues replicating genuine interactions between AI systems and healthcare users for both patients and professionals. The scenarios range from simple patient inquiries to complex clinical decisions, each rigorously evaluated through detailed, physician-designed rubrics aimed at ensuring AI responses are accurate, helpful, and safe.

The development of HealthBench involved extensive collaboration with 262 physicians from 60 countries, spanning various medical specialties. These healthcare professionals played an integral role in shaping realistic and detailed healthcare interactions, ensuring the benchmark truly represents global healthcare scenarios. Physicians provided critical insights into emergencies, global health contexts, data-driven clinical tasks, and varying user proficiency, ensuring each scenario reflected genuine medical practice and challenges.

Rubric Evaluations: Beyond Simple Right or Wrong

HealthBench's evaluation methodology goes far beyond simple correctness, employing detailed rubrics meticulously crafted by experienced physicians. These rubrics provide a structured framework to systematically evaluate AI-generated responses, ensuring a thorough assessment of critical performance attributes. The rubrics cover five essential dimensions, designed to capture the complexities and nuances inherent in healthcare interactions:

Helpfulness: This dimension assesses whether AI responses effectively address or resolve the user's specific healthcare needs and concerns. It evaluates practical utility, ensuring that advice is actionable and relevant.
Accuracy: Rubrics under this category verify factual correctness, ensuring that responses align strictly with current medical guidelines, clinical evidence, and expert consensus. This dimension is critical for maintaining medical validity and reliability.
Completeness: Here, evaluations focus on ensuring AI responses provide comprehensive coverage of necessary information, with no critical details omitted. This is crucial to preventing misunderstandings or incomplete guidance that could lead to negative health outcomes.
Conciseness: Evaluations of conciseness ensure clarity, precision, and efficiency in responses. This dimension minimizes unnecessary complexity, jargon, or verbosity, facilitating clear communication and user comprehension.
Harmfulness: This critical dimension identifies potential risks or adverse effects posed by the AI-generated advice. The aim is to proactively prevent harm, ensuring the safety and ethical appropriateness of responses.

By integrating these detailed and physician-validated rubrics, HealthBench ensures a nuanced and multidimensional evaluation of AI capabilities, providing essential insights into both the strengths and limitations of AI in healthcare.

Organization of HealthBench: Themes and Axes Explained

HealthBench conversations are systematically organized into seven thematic areas, each covering essential real-world healthcare scenarios:

Emergency Referrals: Quick and accurate identification and referral of medical emergencies.
Context Seeking: Effective collection of critical patient information needed for informed decisions.
Global Health: Appropriate adaptation to diverse international healthcare resources, standards, and practices.
Health Data Tasks: Accurate execution of structured medical documentation and related clinical data management tasks.
Tailored Communication: Matching the complexity and style of responses to the user's level of healthcare knowledge.
Uncertainty Management: Clear communication of uncertainty and nuanced information regarding diagnoses and treatment options.
Response Depth: Providing responses with an appropriate level of detail tailored to the user's requirements.

Furthermore, each rubric criterion within these themes is assessed across five performance axes, rigorously evaluating specific dimensions of AI response quality:

Accuracy: Provision of medically validated, factual, and evidence-based information.
Completeness: Inclusion of all vital and clinically relevant details in responses.
Context Awareness: Proper recognition and appropriate reaction to specific clinical contexts.
Communication Quality: Ensuring responses are clear, concise, and easily understandable by diverse users.
Instruction Following: Strict adherence to user-directed instructions, ensuring responses meet explicit user requests and needs.

This structured approach enables precise measurement and meaningful comparison of AI performance across diverse healthcare contexts and interaction scenarios.

Data Collection Process

The dataset underpinning HealthBench was primarily synthetically generated through a tailored language model pipeline designed to create realistic, multi-turn health conversations. Physicians initially defined critical healthcare scenarios and typical interactions to ensure broad and representative coverage of clinical and patient inquiries. These scenario templates included situations such as identifying medical emergencies, managing uncertain health queries, summarizing clinical notes, and requesting clarifications when information was incomplete.

Following synthetic generation, conversations underwent extensive filtering and refinement. Physicians reviewed each generated interaction to ensure authenticity, relevance, and medical accuracy. Scenarios were carefully vetted for realism, consistency, and clarity, verifying that the language and clinical details closely matched genuine patient-provider interactions. Additionally, HealthBench incorporated red-teaming exercises by physicians, designed to rigorously challenge and identify weaknesses in AI responses.

Each conversation in the dataset includes multiple rubric criteria, systematically defined by physicians to reflect a range of evaluation requirements, from straightforward factual accuracy checks to intricate clinical judgments and safety considerations. These physician-crafted rubric criteria ensured a comprehensive, nuanced evaluation of AI responses, capturing critical dimensions of medical reasoning and effective patient communication.

By incorporating rigorous physician oversight throughout the data collection process, HealthBench achieved a dataset characterized by authenticity, depth, and clinical relevance, providing a robust foundation for evaluating AI performance in real-world healthcare contexts.

Key Results: AI’s Clinical Performance

Recent evaluations using HealthBench have demonstrated remarkable advancements, particularly with cutting-edge models such as GPT-4.1 and o3. These models frequently outperformed physicians, particularly in the aspects of clarity, conciseness, and overall user satisfaction. Detailed assessments revealed that advanced models consistently generated responses that were clearer and more directly addressed users' needs compared to responses written by doctors without AI assistance.

Interestingly, AI models also performed exceptionally well against physicians who were provided with earlier generation AI-generated responses. Specifically, GPT-4.1 and o3 showed higher scores than doctors using AI-assisted responses from September 2024 models, indicating that recent AI advancements significantly elevated AI capabilities beyond earlier benchmarks.

However, when doctors had access to the latest (April 2025) AI-generated responses, the performance gap narrowed considerably, with models and AI-assisted physicians performing at comparable levels. This finding suggests a symbiotic potential where doctors can use advanced AI to enhance their own performance, especially in ensuring completeness and avoiding harmful omissions.

Nevertheless, completeness and the reduction of potentially harmful errors remain challenging areas for AI models. These gaps underscore the critical importance of continuous refinement and careful monitoring to further elevate AI reliability and safety in healthcare contexts. The ongoing progress illustrated by HealthBench strongly emphasizes AI’s potential to not just support but significantly enhance clinical decision-making, provided we address these critical safety and completeness concerns.

Progress with Caution

HealthBench represents a significant step forward in the responsible integration of AI in healthcare. Continuous monitoring, improvement, and stringent regulatory oversight remain essential as we advance. HealthBench lays a solid foundation for future AI developments, ensuring healthcare innovations remain both effective and ethically responsible.

OpenAI’s HealthBench Sets New Benchmark for AI in Healthcare