Installation

Create a virtual environment

MedHELM is compatible with Python 3.10, 3.11, and 3.12. We recommend Python 3.12 for best compatibility.

It is recommended to install MedHELM into a virtual environment to avoid dependency conflicts.

Using uv (Recommended)

uv is a fast Python package manager. To create and activate a Python virtual environment:

# Create a virtual environment with Python 3.12
uv venv --python 3.12 .venv

# Activate the virtual environment
source .venv/bin/activate

Using venv (Alternative)

# Create a virtual environment
python3.12 -m venv .venv

# Activate the virtual environment
source .venv/bin/activate

Install MedHELM

Choose one of the following tiers depending on which scenarios you want to run. After activating your environment, use pip for installation (it handles legacy packages better than uv).

Install from this repository (recommended if you cloned the repo)

If you cloned the MedHELM repository and want to run using the checked-out source:

uv pip install -e .

Standard (recommended to start)

Scenarios: PubMedQA, MedCalc-Bench, MedicationQA, MedHallu.

pip install medhelm

Clinical NLP tier ([summarization])

Adds heavier libraries (bert-score, rouge-score, nltk). Install may take 2–3 minutes.

Scenarios: DischargeMe (hospital course summaries), ACI-Bench (clinical transcripts), Patient-Edu (simplifying medical jargon).

pip install "medhelm[summarization]"

Gated / licensing tier ([gated])

Adds gdown so the code can download data from Google Drive. Install can also take longer.

Scenarios: MedQA (USMLE/Board exams), MedMCQA (AIIMS/NEET exams).

pip install "medhelm[gated]"

Install all tiers at once

To install all scenarios (standard, summarization, and gated):

pip install "medhelm[summarization,gated]"

Summary

See Quick Start for running benchmarks with medhelm-run (after activating your environment).