Developer Setup

This guide is for contributors working on the MedHELM repository. For end-user installation, see Installation.

Requirements

Python 3.10, 3.11, or 3.12 (3.12 recommended)
uv — used by CI and recommended for development

Check your Python version:

python --version

Set up the environment

From the repository root:

# Create and activate a virtual environment
uv venv --python 3.12 .venv
source .venv/bin/activate

# Install the package plus dev tools and CI dependencies (matches GitHub Actions)
uv sync --extra ci

The dev and types dependency groups are included automatically via [tool.uv] default-groups.

Run tests

CI runs unit tests excluding slow integration markers:

uv run pytest -m "not models and not scenarios" --durations=20

Scenario integration tests (network downloads):

uv sync --extra ci --extra scenarios
uv run pytest -m scenarios

Run a specific test file:

uv run pytest src/helm/benchmark/scenarios/test_medi_qa_scenario.py -vv

Run linter and type-checker

Install pre-commit hooks:

pre-commit install

Run manually:

./pre-commit.sh

Or individually:

black src scripts
flake8 src scripts
mypy src scripts

Executing commands with local changes

After uv sync, CLI entry points use your local checkout:

medhelm-run --run-entries med_qa:model=openai/gpt2 --suite dev --max-eval-instances 5
helm-summarize --schema src/helm/benchmark/static/schema_medhelm.yaml --suite dev
helm-server --suite dev

Without installing

From the repository root:

PYTHONPATH=src uv run python -m helm.benchmark.run --help

Checking in code

MedHELM uses pull requests against main. Typical workflow:

git checkout main && git pull origin main
Make changes and run tests locally
git checkout -b <your-handle>/<change-id>
./pre-commit.sh (or rely on pre-commit hooks on push)
git commit -a and git push origin <branch>
Open a PR on GitHub