scenarios

Scenarios for reset-online benchmark execution and analysis.

class pysatl_cpd.benchmark.online.reset.scenarios.OnlineResetWholeTimeseriesMetricScenario(entries, metric)[source]

Bases: BenchmarkScenario[DataT, OnlineDetectionTrace[Any], ResultT], Generic

A scenario that evaluates a multiple-run metric across whole timeseries.

For each detector entry, runs detection on every provider in the dataset and then evaluates the supplied metric over the collected set of runs.

Parameters:
entries: Sequence[OnlineResetBenchmarkEntry]
metric: IMultipleRunMetric[OnlineDetectionTrace[Any], LabeledData[DataT, TimeseriesAnnotation], ResultT]
prepare_benchmark_jobs(dataset)[source]

Create one job per entry, each running against all dataset providers.

Also records the set of provider annotations for later filtering in analyze.

Parameters:

dataset (Dataset[TypeVar(DataT), TimeseriesAnnotation]) – Dataset whose providers are used as detector inputs.

Returns:

One job per entry, each with the full provider list.

Return type:

Sequence[BenchmarkJob[TypeVar(DataT)]]

analyze(registry)[source]

Evaluate the metric for each detector entry using registry data.

Filters registry values to match the entry’s detector description and the provider annotations recorded during preparation, then evaluates the metric over the matching runs.

Parameters:

registry (BenchmarkRegistry[TypeVar(DataT), OnlineDetectionTrace[Any]]) – Registry containing cached execution results.

Returns:

Metric evaluation results keyed by detector description.

Return type:

dict[ChangePointDetectorDescription, TypeVar(ResultT)]

__init__(entries, metric)
Parameters:
Return type:

None