benchmark

No-reset benchmark orchestrator.

type pysatl_cpd.benchmark.online.noreset.benchmark.DetectorDescription = ChangePointDetectorDescription
type pysatl_cpd.benchmark.online.noreset.benchmark.ClassificationTable = DataFrame
type pysatl_cpd.benchmark.online.noreset.benchmark.BisegmentsClassificationTable = DataFrame
type pysatl_cpd.benchmark.online.noreset.benchmark.ARLTable = DataFrame
type pysatl_cpd.benchmark.online.noreset.benchmark.SegmentState = StateDescriptor
class pysatl_cpd.benchmark.online.noreset.benchmark.OnlineNoResetBenchmark(dataset, registry, *, n_jobs=1, max_delay, global_policy, precision_policy=None, recall_policy=None, error_margin=None, policy_strict=True)[source]

Bases: Benchmark[DataT, OnlineDetectionTrace[Any]], Generic

Benchmark subclass specialised for no-reset online detectors.

Provides convenience methods for all no-reset scenario types: classification table (global and by transition), bisegments table, ARL by state, and PR-AUC computation.

Classification semantics are fixed at construction via max_delay and policy kind selectors; use build_classification_report() to build the same report configuration without instantiating a benchmark.

Parameters:

  • dataset (Dataset[TypeVar(DataT), TimeseriesAnnotation]) – Labeled dataset whose providers serve as detector inputs.

  • registry (BenchmarkRegistry[TypeVar(DataT), OnlineDetectionTrace[Any]]) – Registry that caches per-detector execution results.

  • n_jobs (int) – Number of parallel worker processes (default 1). Must be non-zero.

  • max_delay (int) – Maximum delay (steps) after the true change point used by bisegment policies and default right tolerance in error_margin when error_margin is omitted.

  • global_policy (NoResetPolicyKind) – Default bisegment policy kind for TP/FP/FN and as the fallback for precision/recall unless overridden.

  • precision_policy (NoResetPolicyKind | None) – Optional policy kind override for the precision metric (TP/FP bases).

  • recall_policy (NoResetPolicyKind | None) – Optional policy kind override for the recall metric (TP/FN bases).

  • error_margin (tuple[int, int] | None) – (left, right) tolerance for underlying classification metrics. When omitted, (0, max_delay) is used.

  • policy_strict (bool) – Whether policies compare detection values to the threshold with strict inequality (default True).

__init__(dataset, registry, *, n_jobs=1, max_delay, global_policy, precision_policy=None, recall_policy=None, error_margin=None, policy_strict=True)[source]
Parameters:
Return type:

None

static build_classification_report(*, max_delay, global_policy, precision_policy=None, recall_policy=None, error_margin=None, policy_strict=True)[source]

Build a NoResetClassificationReport from policy kinds and delay.

Return type:

NoResetClassificationReport[Any, Any]

Parameters:
get_classification_table(entries, *, collect_states=False, n_jobs=None, backend='loky')[source]

Run a global classification-table scenario across all entries.

Parameters:

  • entries (Sequence[OnlineNoResetBenchmarkEntry]) – Detector entries to benchmark.

  • collect_states (bool) – Whether to retain algorithm states during detection (default False).

  • n_jobs (int | None) – Worker count override; falls back to instance n_jobs when None.

  • backend (str) – Joblib parallel backend identifier (default "loky").

Returns:

Classification table per detector description.

Return type:

Mapping[ChangePointDetectorDescription, DataFrame]

get_classification_table_by_transition(entries, transition, use_arl, arl_length=None, *, collect_states=False, n_jobs=None, backend='loky')[source]

Run a transition-filtered classification-table scenario.

Parameters:

  • entries (Sequence[OnlineNoResetBenchmarkEntry]) – Detector entries to benchmark.

  • transition (TransitionDescriptor) – Target transition for bisegment filtering.

  • use_arl (bool) – Whether to include an ARL column.

  • arl_length (int | None) – Expected length of each no-change run; required if use_arl is True.

  • collect_states (bool) – Whether to retain algorithm states during detection (default False).

  • n_jobs (int | None) – Worker count override; falls back to instance n_jobs when None.

  • backend (str) – Joblib parallel backend identifier (default "loky").

Returns:

Classification table (with optional ARL column) per detector description.

Return type:

Mapping[ChangePointDetectorDescription, DataFrame]

static get_pr_auc_table(classification_tables)[source]

Compute PR-AUC from classification tables using trapezoidal integration.

Sorts by recall ascending, drops duplicate recall rows keeping the highest precision, and computes the area under the precision-recall curve via numpy.trapezoid.

Parameters:

classification_tables (Mapping[ChangePointDetectorDescription, DataFrame]) – Mapping of detector descriptions to DataFrames containing recall and precision columns.

Returns:

PR-AUC score per detector description. Entries with empty tables yield NaN.

Return type:

Mapping[ChangePointDetectorDescription, float]

Raises:

ValueError – If any table is missing the required recall or precision columns.

get_ARL_table_by_state(entries, state, arl_length, *, collect_states=False, n_jobs=None, backend='loky')[source]

Run an ARL-by-state scenario.

Parameters:

  • entries (Sequence[OnlineNoResetBenchmarkEntry]) – Detector entries to benchmark.

  • state (StateDescriptor) – Target state for no-change providers.

  • arl_length (int) – Expected length of each no-change run.

  • collect_states (bool) – Whether to retain algorithm states during detection (default False).

  • n_jobs (int | None) – Worker count override; falls back to instance n_jobs when None.

  • backend (str) – Joblib parallel backend identifier (default "loky").

Returns:

ARL table per detector description.

Return type:

Mapping[ChangePointDetectorDescription, DataFrame]

get_bisegments_table(entries, threshold, *, collect_states=False, n_jobs=None, backend='loky')[source]

Run a per-bisegment classification scenario at a fixed threshold.

Parameters:

  • entries (Sequence[OnlineNoResetBenchmarkEntry]) – Detector entries to benchmark.

  • threshold (float) – Detection threshold to apply.

  • collect_states (bool) – Whether to retain algorithm states during detection (default False).

  • n_jobs (int | None) – Worker count override; falls back to instance n_jobs when None.

  • backend (str) – Joblib parallel backend identifier (default "loky").

Returns:

Bisegment classification table per detector description.

Return type:

Mapping[ChangePointDetectorDescription, DataFrame]