metrics

Evaluation metrics for change-point detection algorithms.

This package provides a comprehensive suite of metrics for evaluating both offline and online change-point detection (CPD) algorithms. Metrics are organized into two evaluation scopes: single-run metrics that assess one detection trace against ground truth, and multiple-run metrics that aggregate results across many runs for benchmarking.

Public API

Abstract interfaces:

  • ISingleRunMetric – base class for metrics evaluated on one SingleRun. See pysatl_cpd.analysis.metrics.abstracts.

  • IMultipleRunMetric – base class for metrics evaluated over a sequence of SingleRun objects. See pysatl_cpd.analysis.metrics.abstracts.

Single-run metrics (operate on one SingleRun):

  • ClassificationPrimitive – base class for count-based classification metrics. See pysatl_cpd.analysis.metrics.single_run.

  • TruePositiveCount – counts true change points with matched detections.

  • FalsePositiveCount – counts unmatched detections.

  • FalseNegativeCount – counts true change points with no detection.

  • Delays – per-change-point detection delays for online algorithms.

  • RunLengths – distances between consecutive detections.

Multiple-run aggregation metrics (operate on a sequence of SingleRun):

  • AggregationMetric – abstract base that reduces per-run results via a user-defined method. See pysatl_cpd.analysis.metrics.multiple_run.

  • TotalSum – sums per-run numeric results.

  • TotalMean – arithmetic mean of per-run numeric results.

  • TotalMedian – median of per-run numeric results.

  • DerivedMetric – combines multiple multi-run metric outputs.

Multiple-run classification metrics:

  • TotalTP – total true positives across all runs.

  • TotalFP – total false positives across all runs.

  • TotalFN – total false negatives across all runs.

  • PrecisionMetric – micro-averaged precision.

  • RecallMetric – micro-averaged recall.

  • FScoreMetric – F-beta score (F1 when beta=1).

  • ClassificationReport – full classification summary dict.

Multiple-run online metrics:

  • ARLMetric – mean average run length across all runs.

  • MeanDelayMetric – mean detection delay across all runs.

  • MedianDelayMetric – median detection delay across all runs.

Subpackages

  • abstracts – abstract base classes ISingleRunMetric and IMultipleRunMetric that define the evaluation protocol.

  • single_run – metrics for evaluating a single detection run, including classification counts and online timing metrics.

  • multiple_run – metrics that aggregate results over many runs, including classification summaries and online delay statistics.

Each subpackage has its own docstring with detailed examples and notes.

Notes

  • Classification metrics use an error_margin tuple (left, right) to define a tolerance window around each true change point for matching detections.

  • Multiple-run classification metrics use micro-averaging: counts are summed across all runs before ratios are computed.

  • Online delay metrics require a max_delay parameter that caps both the matching window and the penalty for missed detections.

  • All metrics are generic over trace type, provider type, and result type. Type parameters are inferred from the SingleRun passed to evaluate.