Metrics
basic_metrics
BasicGenerationMetric(names: List[str])
Defines basic metrics which don’t require domain knowledge. This should be fairly comprehensive already, and we should try to use this as much as possible. If we need a different variant, try to generalize this or factor things out. It’s possible we don’t need to subclass this. names is a list of optional metrics to be specified by the user. Currently only exact_match is supported.
evaluate_generation(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService, eval_cache_path: str) -> List[Stat]
Compute all metrics.
BasicReferenceMetric()
Defines basic metrics for Scenarios that use one Request per Reference instead of one per Instance.
evaluate_references(adapter_spec: AdapterSpec, reference_request_states: List[RequestState], metric_service: MetricService, eval_cache_path: str) -> List[Stat]
Perform evaluation when we have made different requests for each reference. For each reference, we have a model score (log probability) and whether it’s correct.
InstancesPerSplitMetric
Report the average num_instances in each MetricContext across train_trials.
_compute_finish_reason_metrics(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService) -> List[Stat]
Record how often generation finished due to reaching token limit, stop token(s), or end of text
_compute_truncation_metrics(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService) -> List[Stat]
Record the number of training instances used in the prompt and whether even the prompt needed to be truncated (once we hit zero training instances).
compute_calibration_metrics(per_instance_stats: Dict[Instance, List[Stat]]) -> List[Stat]
compute_language_modeling_metrics(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService) -> List[Stat]
Compute the logprob and normalization factors for the first completion
compute_perplexity_metrics(stats: Dict[MetricName, Stat]) -> List[Stat]
compute_request_state_metrics(efficiency_metric: EfficiencyMetric, adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService) -> List[Stat]
Compute metrics that are common to both evaluate_generation and evaluate_references.
classification_metrics
ClassificationMetric(averages: Optional[List[Optional[str]]] = None, labels: Optional[List[str]] = None, scores: Optional[List[str]] = None, delimiter: Optional[str] = None)
Defines metrics for multi-class classification using the generation adapter.
Currently provides classification_macro_f1 and classification_micro_f1. These are population-level F1 measures to measure classification performance where each generation is a predicted class, and are different from the instance-level F1 measures in BasicMetrics that are intended to measure word overlap between the correct references and generations. The correct class should be provided by the normalized text of a correct reference. The predicted class for each instance is the normalized text of the generation.
Note: - It is highly recommended to specify the set of classes should be specified using the labels parameter. Otherwise, the set of classes is derived from the correct references from all the instances. This means that classes may be incorrectly omitted if they are never used as a correct reference. - The averages parameter is a list of averaging methods to be used. It has the same meaning average as in scikit-learn. - Generations that are not in any of the known classes are counted as a negative prediction for every class. - Perturbed classes are considered different classes from unperturbed classes. - Currently, multi-label classification is not supported.
:param delimiter: For multi-label classification, the string delimiter between classes in the model’s output. :param average: The list of scores to compute (e.g. “f1”, “precision”, “recall”). Defaults to [“f1”]. :param average: The averaging methods (e.g. “micro”, “macro”, “weighted”) to be used. It has the same meaning average as in scikit-learn. Defaults to [“macro”, “micro”]. :param labels: The set of labels. :return: A list of Stat objects.
evaluate_instances(request_states: List[RequestState], eval_cache_path: str) -> List[Stat]
MultipleChoiceClassificationMetric
Calculate population micro/macro F1 score for multiple_choice_* adapters. For generation adapters, please use ClassificationMetric.
evaluate_instances(request_states: List[RequestState], eval_cache_path: str) -> List[Stat]
dry_run_metrics
DryRunMetric()
Metrics for dry run.
efficiency_metrics
EfficiencyMetric()
compute_efficiency_metrics(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService) -> List[Stat]
Compute efficiency metrics for both inference and training. For inference, we record both the actual runtime and an estimated idealized runtime for the given request with an optimized software implementation run on A100 GPU(s), taking into account both the number of tokens in the prompt of the request, and the number of generated output tokens. For training, we report the estimated total metric tons of CO2 emitted to train the model. This is the same for each request.
ehr_sql_metrics
EhrSqlMetric
Metric for evaluating the EHR SQL dataset, focusing on: 1. Execution Accuracy – Whether the generated SQL query produces the same results as the ground truth. 2. Query Validity – Whether the generated SQL executes without errors. 3. Precision for Answerable Questions (Pans). 4. Recall for Answerable Questions (Rans).
evaluate_generation(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService, eval_cache_path: str) -> List[Stat]
Evaluate execution accuracy, query validity, and answerability metrics.
evaluate_instances_metric
EvaluateInstancesMetric
Metric that needs to examine all request states for all instances in the same split with the same perturbations in order to determine the Stats.
evaluate_instances(request_states: List[RequestState], eval_cache_path: str) -> List[Stat] abstractmethod
Evaluate all request states directly. Use only if nothing else works.
evaluate_reference_metrics
compute_reference_metrics(names: List[str], adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService) -> List[Stat]
Setup:
- Gold (correct references): G1 … Gm
- Predictions (completions): P1 … Pk
For each pair (G, P), we can define a ${score} (e.g., exact match, F1, BLEU).
We define the following stats:
- ${score}: max_i score(Gi, P1)
- ${score}@k: max_{i,j} score(Gi, Pj)
llm_jury_metrics
LLMJuryMetric(metric_name: str, scenario_name: str, annotator_models: Dict[str, AnnotatorModelInfo], default_score: float = 0.0)
Score metrics for LLM Jury.
evaluate_generation(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService, eval_cache_path: str) -> List[Stat]
medcalc_bench_metrics
MedCalcBenchMetric
Metric for evaluating the MedCalc Bench dataset, assessing the model’s ability to be a clinical calculator.
Exact match based on category: 1. Normal exact match: for categories “risk”, “severity” or “diagnosis”. 2. Variant exact match: for other categories, if the number calculated by the model falls between the values in the Lower limit and Upper limit columns, we mark it as accurate.
evaluate_generation(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService, eval_cache_path: str) -> List[Stat]
Evaluate a single generation against reference labels.
medec_metrics
MedecMetric
Metric for evaluating the MEDEC dataset, assessing medical error detection and correction.
- Error Flag Accuracy : Whether the model correctly identifies if a medical note contains an error.
- Error Sentence Detection Accuracy : Whether the model correctly identifies the erroneous sentence when an error is present.
evaluate_generation(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService, eval_cache_path: str) -> List[Stat]
Evaluate a single LLM generation against the ground truth labels.
mimiciv_billing_code_metrics
MIMICIVBillingCodeMetric
Metric for evaluating the MIMIC Billing Code dataset, assessing the model’s ability to match the reference ICD codes. Handles cases where raw prediction output contains additional text.
Calculates: 1. Precision: proportion of correctly predicted ICD codes among all predicted codes 2. Recall: proportion of correctly predicted ICD codes among all reference codes 3. F1 score: harmonic mean of precision and recall
ICD codes format: letter followed by 1-3 digits, optional period, optional additional digits
“J18.9”, “J45.909”, “J47.1”, “J96.01”
evaluate_generation(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService, eval_cache_path: str) -> List[Stat]
Evaluate a single generation against reference labels.
sct_bench_metrics
SCTBenchMetric
Metric for evaluating Script Concordance Test (SCT) responses.
Computes two metrics: - sct_score: Normalized expert panel weight for the model’s chosen rating (0 to 1). - sct_expert_set_membership: Whether the model’s response falls within the expert agreement region (binary, averaged across instances).
evaluate_generation(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService, eval_cache_path: str) -> List[Stat]
summarization_metrics
SummarizationMetric(task: str, language: str = 'en', device: str = 'cpu', bertscore_model: str = 'microsoft/deberta-large-mnli', rescale_with_baseline: bool = True, summac_new_line_split: bool = False)
Summarization Metrics
This class computes the following standard summarization metrics
- Rouge (1,2,L)
- Extractiveness (coverage, density, novel n-grams)
- Compression
- Faithfulness (SummaC)
evaluate_generation(adapter_spec: AdapterSpec, request_state: RequestState, metric_service: MetricService, eval_cache_path: str) -> List[Stat]
ultra_suite_asr_classification_metrics
UltraSuiteASRMetric
Score metrics for UltraSuite ASR.