{ "cells": [ { "cell_type": "markdown", "id": "dd3861a6", "metadata": {}, "source": [ "# 06. PySATL CPD API: No-Reset Benchmarking\n", "\n", "## Section 1. Orientation\n", "\n", "The reset benchmark notebook evaluated detectors that restart after each declared change. No-reset benchmarking keeps the algorithm state moving continuously. That makes the benchmark focus on a different question: how should one continuous detection function be interpreted around true transitions and inside stable no-change regimes?\n", "\n", "The workflow in this chapter is: build one no-reset benchmark entry, run continuous traces for transition-centered bisegments, sweep thresholds over those traces, and use a classification policy to decide which thresholded points count as true positives, false positives, or false negatives. The policy controls that interpretation; it does not change the underlying trace.\n", "\n", "After the global threshold table, we narrow the same idea to one transition type, then inspect ARL by state for no-change robustness, then return to a concrete bisegment trace. From here on, outputs are table-oriented mappings keyed by detector description (`ChangePointDetectorDescription`). For a single `OnlineNoResetBenchmarkEntry`, that key is exactly `entry.description`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import plotly.io as pio\n", "pio.renderers.default = 'sphinx_gallery'\n" ] }, { "cell_type": "markdown", "id": "5bf39ae1", "metadata": {}, "source": [ "## What in this notebook\n", "\n", "By the end of the notebook, you should know how to build a compact no-reset benchmark campaign, define a no-reset classification policy, compute global classification tables, focus on one specific transition, compute ARL tables for one state, inspect bisegment-level results, optionally crop bisegment windows before detection with `bisegment_cut`, and connect those aggregate views back to one continuous trace.\n", "\n", "Scenario methods return mappings keyed by detector description (`ChangePointDetectorDescription`), not by a separate entry-id type.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "e562bbf0", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "from IPython.display import display\n", "\n", "from pysatl_cpd.algorithms.online import ShewhartControlChart\n", "from pysatl_cpd.analysis.visualization import DrawBackend, OnlineCpdPlotter\n", "from pysatl_cpd.analysis.visualization.benchmarking import (\n", " ARLBasedMetricVisualizer,\n", " BenchmarkPlotter,\n", " PrAucVisualizer,\n", " ThresholdBasedMetricVisualizer,\n", ")\n", "from pysatl_cpd.benchmark.online.noreset import (\n", " EventBasedPolicy,\n", " MixedPolicy,\n", " NoResetPolicyKind,\n", " OnlineNoResetBenchmark,\n", " OnlineNoResetBenchmarkEntry,\n", " PointBasedPolicy,\n", ")\n", "from pysatl_cpd.benchmark.online.noreset.thresholds.ranges import LinearThresholdsRange\n", "from pysatl_cpd.benchmark.online.noreset.tooling.bisegment_cut import BisegmentCut\n", "from pysatl_cpd.benchmark.registry import BenchmarkRegistry\n", "from pysatl_cpd.data.generator import preset_dataset\n", "from pysatl_cpd.data.providers.transformers import ColumnsSelectorTransformer" ] }, { "cell_type": "markdown", "id": "7f37a8d4", "metadata": {}, "source": [ "## Section 2. Building a No-Reset Benchmark Campaign\n", "\n", "The first no-reset benchmarking problem looks superficially similar to the reset case: choose a dataset, choose an algorithm, and choose a threshold sweep. But the campaign object is not a family of fully thresholded detectors. Instead, it is one algorithm entry together with a threshold range and an optional transformer.\n", "\n", "That difference matters because no-reset benchmarking treats thresholds as evaluation settings applied to continuous traces rather than as permanent baked-in detector identities at construction time.\n" ] }, { "cell_type": "markdown", "id": "258388ee", "metadata": {}, "source": [ "### 2.1 Choosing the dataset and feature view\n", "\n", "As in the reset chapter, we begin with a preset labeled dataset and a feature transformer so the observed signal is explicit. The important difference is interpretive rather than structural: in no-reset benchmarking, the same continuous run may later be analyzed under several thresholds and policies.\n", "\n", "A compact preset mean-shift dataset is still a good tutorial choice because it produces clear transitions and keeps the scenario tables readable.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "51a207c9", "metadata": {}, "outputs": [], "source": [ "FEATURE = \"feature_0\"\n", "ERROR_MARGIN = (0, 15)\n", "ARL_LENGTH = 60\n", "\n", "dataset = preset_dataset(\"mean_shifts\", n_series=8, seed=7, series_length=180)\n", "feature_transformer = ColumnsSelectorTransformer(columns=[FEATURE])\n", "\n", "print(\"Dataset size:\", len(dataset))\n", "print(\"Dataset states:\", [dict(state) for state in dataset.states])\n", "print(\"Dataset transitions:\", [repr(transition) for transition in dataset.transitions])\n", "print(\"Selected feature transformer annotation:\", feature_transformer.annotation)" ] }, { "cell_type": "markdown", "id": "c27bbd8f", "metadata": {}, "source": [ "### 2.2 Creating an OnlineNoResetBenchmarkEntry\n", "\n", "The next problem is packaging the no-reset campaign inputs. Instead of constructing one detector per threshold, the current API asks for one algorithm together with a threshold range object and an optional data transformer.\n", "\n", "`OnlineNoResetBenchmarkEntry` is that package. It represents a no-reset evaluation family rather than one already-thresholded detector instance.\n", "\n", "The `description` property is a `ChangePointDetectorDescription` for the unstated-threshold no-reset detector built from the same algorithm and transformer. Every `benchmark.get_*` method in this chapter returns `dict`/`Mapping` objects keyed by that description, which keeps analyzers and pickers aligned with the core detector-description model.\n", "\n", "When thresholds are inferred automatically from traces (ARL-based or max-detection grids), the resolver now uses only finite samples from `trace.detection_function`. Non-finite values no longer distort inferred bounds.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "f1ef1a62", "metadata": {}, "outputs": [], "source": [ "entry = OnlineNoResetBenchmarkEntry(\n", " algorithm=ShewhartControlChart(learning_period_size=20, window_size=10),\n", " thresholds=LinearThresholdsRange(start=1.5, end=3.0, count=6),\n", " data_transformer=feature_transformer,\n", " bisegment_cut=BisegmentCut.parse((8, 0)),\n", ")\n", "print(\"Detector description (entry.description):\", entry.description)\n", "print(\"Resolved threshold grid:\", list(entry.thresholds.thresholds_range))" ] }, { "cell_type": "markdown", "id": "2d341e9a", "metadata": {}, "source": [ "### 2.3 Creating the benchmark orchestrator\n", "\n", "The campaign still needs the same execution-management layer as the reset benchmark: a registry for cached runs and a benchmark object that knows how to prepare scenario-specific jobs and analyses.\n", "\n", "The no-reset benchmark object is more scenario-oriented than the reset one. Instead of one generic \u201cget metric\u201d method, it exposes dedicated methods for global classification tables, transition-focused tables, ARL-by-state tables, and bisegment tables.\n", "\n", "Detector-level crop is now configured on each `OnlineNoResetBenchmarkEntry` via `entry.bisegment_cut`, not as a separate argument on benchmark convenience methods.\n" ] }, { "cell_type": "markdown", "id": "ce63b32d", "metadata": {}, "source": [ "### 2.4 Cropping bisegment windows (`bisegment_cut`)\n", "\n", "Bisegment scenarios trim every transition-centered provider by discarding a fixed number of samples from the left edge and from the right edge before running the online detector. Set this crop on `OnlineNoResetBenchmarkEntry(bisegment_cut=...)` using a tuple via `BisegmentCut.parse(...)` or a ready `BisegmentCut` instance.\n", "\n", "Entry-level crop affects bisegment-backed scenarios: `get_classification_table`, `get_classification_table_by_transition`, and `get_bisegments_table`.\n", "\n", "ARL-by-state runs are built from no-change providers and are executed with no-op crop semantics, so ARL analysis does not trim those providers even if the original entry carries a non-noop `bisegment_cut`.\n", "\n", "Margins cannot remove the entire series: `left_trim + right_trim` must stay strictly below each provider length (otherwise validation raises)." ] }, { "cell_type": "code", "execution_count": null, "id": "50d5867f", "metadata": {}, "outputs": [], "source": [ "benchmark = OnlineNoResetBenchmark(\n", " dataset=dataset,\n", " registry=BenchmarkRegistry(),\n", " max_delay=ERROR_MARGIN[1],\n", " global_policy=NoResetPolicyKind.MIXED,\n", " error_margin=ERROR_MARGIN,\n", " policy_strict=False,\n", ")\n", "print(\"Benchmark object:\", benchmark)" ] }, { "cell_type": "markdown", "id": "244ab666", "metadata": {}, "source": [ "**Summary of Section 2:** A no-reset benchmark campaign is built around one algorithm entry plus a threshold range, not one detector per threshold. That design reflects the fact that continuous traces are generated first and interpreted under thresholds and policies later. Optional `entry.bisegment_cut` lets you crop bisegment providers before detection when you need the analysis window tighter than the raw labeled slice.\n" ] }, { "cell_type": "markdown", "id": "0da68d99", "metadata": {}, "source": [ "## Section 3. No-Reset Classification Semantics\n", "\n", "Before running any benchmark method, we need to answer the most important no-reset question: what exactly counts as a detection in a continuous trace? In reset mode, this question is mostly handled by the detector itself. In no-reset mode, it becomes a benchmark-time classification problem.\n", "\n", "This is why policy objects and no-reset classification reports are central to the chapter. They define how threshold crossings are translated into true positives, false positives, and false negatives around one true transition.\n" ] }, { "cell_type": "markdown", "id": "493517c9", "metadata": {}, "source": [ "### 3.1 Configuring classification on `OnlineNoResetBenchmark`\n", "\n", "No-reset classification (TP/FP/FN, precision, recall, F1) is fixed when you construct `OnlineNoResetBenchmark`. Pass `max_delay`, a `NoResetPolicyKind` for `global_policy`, and optionally `precision_policy` / `recall_policy` overrides, plus `error_margin` and `policy_strict` when you need non-default tolerances or threshold-compare strictness.\n", "\n", "The same report can be reproduced without a benchmark via the static helper `OnlineNoResetBenchmark.build_classification_report(...)`. The underlying `NoResetClassificationReport` type lives in `pysatl_cpd.benchmark.online.noreset.metrics` for advanced or experimental code paths.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6a01f9a5", "metadata": {}, "outputs": [], "source": [ "print(\"Classification configured with error margin:\", ERROR_MARGIN)\n", "print(\"Static helper returns the same report type as inside the benchmark:\")\n", "print(\n", " OnlineNoResetBenchmark.build_classification_report(\n", " max_delay=ERROR_MARGIN[1],\n", " global_policy=NoResetPolicyKind.MIXED,\n", " error_margin=ERROR_MARGIN,\n", " policy_strict=False,\n", " )\n", ")" ] }, { "cell_type": "markdown", "id": "27067e3d", "metadata": {}, "source": [ "### 3.2 Point-based, event-based, and mixed policies\n", "\n", "The practical problem solved by policies is subtle. A continuous detection function may spend many steps above threshold, and not every above-threshold step should necessarily count as a distinct detection event. The policy decides how to interpret those thresholded runs.\n", "\n", "The current public no-reset API exposes several policy styles. `PointBasedPolicy` treats threshold-satisfying points directly, `EventBasedPolicy` focuses on threshold-crossing events, and `MixedPolicy` combines event-style false-region behavior with point-style true-region behavior. The right choice depends on how repeated alarms should be interpreted in the task at hand.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "78a4c35c", "metadata": {}, "outputs": [], "source": [ "point_policy = PointBasedPolicy(max_delay=ERROR_MARGIN[1], strict=False)\n", "event_policy = EventBasedPolicy(max_delay=ERROR_MARGIN[1], strict=False)\n", "mixed_policy = MixedPolicy(max_delay=ERROR_MARGIN[1], strict=False)\n", "\n", "print(\"Point policy:\", point_policy)\n", "print(\"Event policy:\", event_policy)\n", "print(\"Mixed policy:\", mixed_policy)" ] }, { "cell_type": "markdown", "id": "b156d7a2", "metadata": {}, "source": [ "### 3.3 Why the error margin matters more in no-reset mode\n", "\n", "A reset detector declares one event and then restarts. A no-reset trace can spend a long interval near or above threshold, so the acceptable delay window around the true change point has a stronger effect on classification semantics.\n", "\n", "The error margin therefore deserves explicit attention. It defines the true region around the change point and directly interacts with the policy\u2019s `max_delay` setting when selecting which thresholded points count as valid true positives.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "91c556a5", "metadata": {}, "outputs": [], "source": [ "print(\"Error margin used by the report:\", ERROR_MARGIN)\n", "print(\"Policy max_delay:\", ERROR_MARGIN[1])" ] }, { "cell_type": "markdown", "id": "533cf6db", "metadata": {}, "source": [ "**Summary of Section 3:** No-reset classification is policy-driven because continuous traces do not naturally collapse into one event sequence on their own. The classification report and policy objects together define how thresholded continuous behavior becomes TP, FP, FN, precision, recall, and F1.\n" ] }, { "cell_type": "markdown", "id": "8538af2e", "metadata": {}, "source": [ "## Section 4. Global Classification Tables\n", "\n", "Once classification semantics are defined, the first broad analysis problem is global performance: how does the detector defined by the entry behave across all transition-centered bisegment runs in the dataset as the threshold changes? That is the role of the global classification table.\n", "\n", "Unlike the reset chapter, the main output here is already a threshold-indexed table. That makes no-reset benchmarking feel more natively table-oriented right from the beginning.\n" ] }, { "cell_type": "markdown", "id": "ef4753bf", "metadata": {}, "source": [ "### 4.1 get_classification_table(...)\n", "\n", "The motivation for the global table is to get one threshold sweep over all bisegment runs under the current classification report. It is the closest no-reset counterpart to the merged reset table from the previous notebook.\n", "\n", "`get_classification_table(...)` resolves thresholds for the entry, evaluates the chosen report across the matching runs, and returns one `DataFrame` per detector description. Cropping is controlled by `entry.bisegment_cut` (Section 2.4), not by a method argument.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "132ce6f8", "metadata": {}, "outputs": [], "source": [ "global_tables = benchmark.get_classification_table([entry])\n", "description, global_table = next(iter(global_tables.items()))\n", "print(\"Detector description:\", description)\n", "assert description == entry.description\n", "display(global_table.round(3))" ] }, { "cell_type": "code", "execution_count": null, "id": "81cd9d0a", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(8, 4))\n", "global_table.plot(x=\"threshold\", y=[\"precision\", \"recall\", \"f1\"], marker=\"o\", ax=ax)\n", "ax.set_title(\"Global no-reset threshold sweep\")\n", "ax.set_ylabel(\"Score\")\n", "ax.grid(True, alpha=0.3)\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "id": "571045ce", "metadata": {}, "source": [ "### 4.2 Reading the global table\n", "\n", "The global table contains both count-style and score-style summaries. TP, FP, and FN give raw event accounting under the chosen policy, while precision, recall, and F1 compress that accounting into more familiar detector-performance scores.\n", "\n", "The important interpretive point is that these numbers are not whole-timeseries reset metrics. They are derived from transition-centered no-reset evaluation under one chosen classification policy.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "df336858", "metadata": {}, "outputs": [], "source": [ "display(global_table[[\"threshold\", \"tp\", \"fp\", \"fn\", \"precision\", \"recall\", \"f1\"]].round(3))" ] }, { "cell_type": "markdown", "id": "383ea832", "metadata": {}, "source": [ "### 4.3 PR-AUC from global classification tables\n", "\n", "Threshold tables are useful, but sometimes we also want one high-level summary of the precision-recall tradeoff. That is the problem solved by PR-AUC in the no-reset benchmark API.\n", "\n", "`get_pr_auc_table(...)` derives one score per detector description from the global classification tables. It does not replace the threshold table, but it does provide a compact ranking-oriented view.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "61ed07fc", "metadata": {}, "outputs": [], "source": [ "pr_auc_scores = benchmark.get_pr_auc_table(global_tables)\n", "print(\"PR-AUC scores:\", pr_auc_scores)" ] }, { "cell_type": "markdown", "id": "acdc9ab8", "metadata": {}, "source": [ "**Summary of Section 4:** The global classification table is the first high-level no-reset summary. It describes policy-dependent threshold behavior across all transition-centered runs, and PR-AUC gives one compact precision-recall summary of that threshold sweep. These bisegment-backed tables honor `entry.bisegment_cut`; ARL-by-state runs are evaluated with no-op crop semantics.\n" ] }, { "cell_type": "markdown", "id": "80de9aed", "metadata": {}, "source": [ "## Section 5. Transition-Specific Analysis\n", "\n", "A global table is useful, but it can hide important structure. Some transitions are easy, some are hard, and some produce specific false-alarm patterns that disappear in the aggregate. The next problem is therefore transition specificity.\n", "\n", "No-reset benchmarking solves this by letting us evaluate one named transition at a time. This keeps the threshold sweep but narrows the underlying run set to one transition type.\n" ] }, { "cell_type": "markdown", "id": "252b83c3", "metadata": {}, "source": [ "### 5.1 get_classification_table_by_transition(...)\n", "\n", "The main motivation for this method is to answer questions like \u201chow well does the detector handle baseline-to-alternative_1 transitions specifically?\u201d Global tables cannot answer that cleanly because several transition types are aggregated together.\n", "\n", "The API accepts one transition descriptor, a flag controlling whether ARL should be included in the resulting table, and the ARL run length needed for that extra column. Crop settings come from `entry.bisegment_cut`.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3642ed94", "metadata": {}, "outputs": [], "source": [ "transition = next(iter(dataset.transitions))\n", "transition_tables = benchmark.get_classification_table_by_transition(\n", " [entry],\n", " transition=transition,\n", " use_arl=True,\n", " arl_length=ARL_LENGTH,\n", ")\n", "transition_table = next(iter(transition_tables.values()))\n", "print(\"Selected transition:\", transition)\n", "display(transition_table.round(3))" ] }, { "cell_type": "code", "execution_count": null, "id": "63fbd79f", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(8, 4))\n", "transition_table.plot(x=\"threshold\", y=[\"precision\", \"recall\", \"f1\"], marker=\"o\", ax=ax)\n", "ax.set_title(f\"Transition-specific threshold sweep: {transition}\")\n", "ax.set_ylabel(\"Score\")\n", "ax.grid(True, alpha=0.3)\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "id": "2e864a05", "metadata": {}, "source": [ "### 5.2 Why transition filtering matters\n", "\n", "The practical value of transition filtering is diagnostic specificity. A detector that looks strong globally may still fail badly on one specific regime change, and a detector that looks mediocre globally may be exactly the right tool for one targeted transition family.\n", "\n", "That is why no-reset benchmarking leans so heavily on transition-centered views. The continuous trace is interpreted around one true change point, and transition identity is part of that interpretation rather than an afterthought.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6f2ae4f5", "metadata": {}, "outputs": [], "source": [ "print(\"Transition-table columns:\", list(transition_table.columns))" ] }, { "cell_type": "markdown", "id": "0a0533f2", "metadata": {}, "source": [ "### 5.3 Optional ARL inside transition tables\n", "\n", "Transition-specific classification can be enriched with ARL when false-alarm robustness is also relevant to the interpretation. The table above includes an `arl` column precisely because we enabled `use_arl=True`.\n", "\n", "This is a good example of the scenario-oriented nature of the no-reset benchmark API: related but distinct analysis views can be combined where it makes interpretive sense.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "5bde6a32", "metadata": {}, "outputs": [], "source": [ "display(transition_table[[\"threshold\", \"precision\", \"recall\", \"f1\", \"arl\"]].round(3))" ] }, { "cell_type": "markdown", "id": "fece2a63", "metadata": {}, "source": [ "**Summary of Section 5:** Transition-specific tables let us keep the threshold-sweep logic while narrowing the analysis to one regime change type. This is one of the most important ways no-reset benchmarking becomes more diagnostic than a single aggregate summary.\n" ] }, { "cell_type": "markdown", "id": "c9c6b548", "metadata": {}, "source": [ "## Section 6. State-Based ARL Analysis\n", "\n", "The next no-reset problem is false-alarm behavior inside one regime. Classification tables focus on transition-centered runs, but ARL-by-state focuses on no-change windows from one chosen state. That is a different slice of the same benchmark campaign.\n", "\n", "This analysis is especially useful when the baseline behavior of the algorithm depends strongly on regime identity. A threshold that seems robust globally may still behave poorly inside one particular state.\n" ] }, { "cell_type": "markdown", "id": "b2cc02a8", "metadata": {}, "source": [ "### 6.1 get_ARL_table_by_state(...)\n", "\n", "The motivation for this method is to isolate false-alarm robustness in one state without mixing in transition difficulty. It extracts no-change windows associated with the chosen state and evaluates ARL across the threshold grid.\n", "\n", "The API needs the state descriptor and the expected ARL run length. The result is one ARL table per detector description. ARL uses no-change providers and is evaluated with no-op crop semantics (independent of any non-noop `entry.bisegment_cut`).\n" ] }, { "cell_type": "code", "execution_count": null, "id": "a93ba05a", "metadata": {}, "outputs": [], "source": [ "state = next(iter(dataset.states))\n", "arl_tables = benchmark.get_ARL_table_by_state([entry], state=state, arl_length=ARL_LENGTH)\n", "state_arl_table = next(iter(arl_tables.values()))\n", "print(\"Selected state:\", state)\n", "display(state_arl_table.round(3))" ] }, { "cell_type": "code", "execution_count": null, "id": "fdc9d97a", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(8, 4))\n", "state_arl_table.plot(x=\"threshold\", y=\"arl\", marker=\"o\", ax=ax, legend=False)\n", "ax.set_title(f\"ARL by state: {state}\")\n", "ax.set_ylabel(\"ARL\")\n", "ax.grid(True, alpha=0.3)\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "id": "372fa139", "metadata": {}, "source": [ "### 6.2 Interpreting ARL by state\n", "\n", "The interpretation here is simpler than in classification tables: larger ARL generally means longer expected time before an alarm, which is often desirable on genuine no-change windows. But the real point is comparative: how does ARL move as the threshold changes, and does the chosen state behave differently from what a global summary might suggest?\n", "\n", "This table is therefore less about precision-recall tradeoffs and more about robustness inside one stable regime.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "f7b6a89a", "metadata": {}, "outputs": [], "source": [ "display(state_arl_table[[\"threshold\", \"arl\"]].round(3))" ] }, { "cell_type": "markdown", "id": "b81b7931", "metadata": {}, "source": [ "### 6.3 Why ARL-by-state belongs in the no-reset story\n", "\n", "In a reset benchmark, ARL is one useful metric among several. In the no-reset story, ARL-by-state becomes more structurally important because the detector is continuously active and can repeatedly flirt with threshold inside stable regimes.\n", "\n", "That makes state-centered no-change analysis a natural counterpart to transition-centered classification analysis. The two table families answer different questions about the same continuous detector behavior.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "cb615855", "metadata": {}, "outputs": [], "source": [ "print(\"State-specific ARL table length:\", len(state_arl_table))" ] }, { "cell_type": "markdown", "id": "5e0391ee", "metadata": {}, "source": [ "**Summary of Section 6:** ARL-by-state isolates false-alarm robustness on no-change windows from one state. It complements the transition-centered classification tables and is one of the clearest examples of how no-reset evaluation differs structurally from reset evaluation.\n" ] }, { "cell_type": "markdown", "id": "507a2615", "metadata": {}, "source": [ "## Section 7. Bisegment-Level Analysis\n", "\n", "Global tables and transition-specific tables are still aggregate summaries. The next practical problem is local diagnosis: if the detector misbehaves, which concrete transition windows reveal that misbehavior most clearly? The bisegment table is the natural answer.\n", "\n", "Because no-reset classification is already transition-centered, bisegments are not an afterthought here. They are the local units from which many aggregate summaries are built.\n" ] }, { "cell_type": "markdown", "id": "8bf20968", "metadata": {}, "source": [ "### 7.1 get_bisegments_table(...)\n", "\n", "The motivation for the bisegment table is to inspect one threshold at local transition granularity. Instead of summarizing several runs into one row per threshold, we ask for the per-bisegment classification table at a chosen threshold.\n", "\n", "This is especially useful after a global threshold sweep has suggested one interesting operating point that deserves closer inspection. The scenario uses the same `entry.bisegment_cut` machinery as the global and transition-level classification tables when you need shorter windows per segment.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "c26d0ed9", "metadata": {}, "outputs": [], "source": [ "chosen_threshold = 2.4\n", "bisegment_tables = benchmark.get_bisegments_table([entry], threshold=chosen_threshold)\n", "bisegment_table = next(iter(bisegment_tables.values()))\n", "print(\"Chosen threshold:\", chosen_threshold)\n", "print(\"Bisegment table shape:\", bisegment_table.shape)\n", "display(bisegment_table.head(8))" ] }, { "cell_type": "markdown", "id": "f930ec1a", "metadata": {}, "source": [ "### 7.2 Why bisegments are central in no-reset mode\n", "\n", "A reset detector\u2019s event sequence can often be summarized at the whole-timeseries level. A no-reset detector\u2019s continuous trace, by contrast, becomes most interpretable around one true change point and the neighboring false region. That is exactly the structure a bisegment provides.\n", "\n", "This is why bisegments matter more here than they did in the reset chapter. They are a natural unit of no-reset diagnostic reasoning rather than just an auxiliary slice type.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3166264a", "metadata": {}, "outputs": [], "source": [ "print(\"Bisegment-table columns:\", list(bisegment_table.columns))" ] }, { "cell_type": "markdown", "id": "51f9ad90", "metadata": {}, "source": [ "### 7.3 Choosing informative local cases\n", "\n", "The bisegment table becomes most useful when we use it to choose representative or difficult cases for trace inspection. The goal is not to read every row, but to let the local table point us toward runs that explain aggregate behavior.\n", "\n", "A common strategy is to choose one threshold first, then pick one bisegment case that looks unusually good, unusually bad, or simply representative.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "7158c70b", "metadata": {}, "outputs": [], "source": [ "display(bisegment_table[[\"tp\", \"fp\", \"fn\", \"precision\", \"recall\", \"f1\"]].head(8))" ] }, { "cell_type": "markdown", "id": "42c9b44c", "metadata": {}, "source": [ "**Summary of Section 7:** Bisegment tables expose the local transition cases hidden inside aggregate summaries. In no-reset benchmarking, they are a natural diagnostic surface because the whole evaluation model is already centered on continuous traces around true change points.\n" ] }, { "cell_type": "markdown", "id": "bb6741d2", "metadata": {}, "source": [ "## Section 8. Visualizing No-Reset Benchmark Results\n", "\n", "At this point, we have several different no-reset tables: a global classification table, a transition-specific table, a state ARL table, and a local bisegment table. The next problem is summarizing the most important parts visually without losing the distinction between these table families.\n", "\n", "The plotting layer is the same one used earlier in the tutorial series, but the way we choose which table feeds which visualizer is more nuanced here because not every table supports the same columns or the same interpretation.\n" ] }, { "cell_type": "markdown", "id": "f016c670", "metadata": {}, "source": [ "### 8.1 Global threshold and PR-AUC views in matplotlib\n", "\n", "The global table is the natural source for PR-AUC and threshold-based precision-recall-F1 plots. Those views summarize the operating-point tradeoff under the chosen no-reset classification semantics.\n", "\n", "The code below uses `BenchmarkPlotter` just as in earlier notebooks, but now the benchmark table is a no-reset global classification table rather than a reset metric merge.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3b33727e", "metadata": {}, "outputs": [], "source": [ "global_plotter = (\n", " BenchmarkPlotter()\n", " .set_metrics(\n", " {\n", " \"pr_auc\": PrAucVisualizer(label=\"No-reset PR-AUC\"),\n", " \"thresholds\": ThresholdBasedMetricVisualizer(\n", " y_metrics=[\"precision\", \"recall\", \"f1\"],\n", " title=\"No-reset threshold curves\",\n", " ylabel=\"Score\",\n", " ),\n", " }\n", " )\n", " .set_benchmark_tables({\"global\": global_table})\n", ")\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(14, 5))\n", "global_plotter.draw(figure=fig, axes={\"pr_auc\": axes[0], \"thresholds\": axes[1]})\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "id": "fd02fc05", "metadata": {}, "source": [ "### 8.2 ARL-by-state views in matplotlib\n", "\n", "The state ARL table solves a different visual problem. It is not a precision-recall tradeoff view; it is a robustness curve over threshold. That is why it needs a different metric visualizer family.\n", "\n", "`ARLBasedMetricVisualizer` handles that table shape. The example below keeps the visual summary small and focused on the state-specific ARL story.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6ea3f335", "metadata": {}, "outputs": [], "source": [ "arl_plotter = (\n", " BenchmarkPlotter()\n", " .set_metrics(\n", " {\n", " \"arl_view\": ARLBasedMetricVisualizer(\n", " y_metrics=[\"threshold\"],\n", " title=\"Threshold as a function of ARL\",\n", " ylabel=\"Threshold\",\n", " )\n", " }\n", " )\n", " .set_benchmark_tables({\"state_arl\": state_arl_table})\n", ")\n", "\n", "fig, ax = plt.subplots(1, 1, figsize=(7, 5))\n", "arl_plotter.draw(figure=fig, axes={\"arl_view\": ax})\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "id": "a56fbee9", "metadata": {}, "source": [ "**Summary of Section 8:** No-reset visualization is table-family aware. Global classification tables support PR-AUC and threshold-score views, while state ARL tables support robustness-oriented ARL views. The plotting API is familiar, but the interpretive source tables are more differentiated than in reset mode.\n" ] }, { "cell_type": "markdown", "id": "6d7f9b1c", "metadata": {}, "source": [ "## Section 9. Explaining No-Reset Behavior With One Continuous Trace\n", "\n", "Aggregate tables tell us what happened across many runs, but the final interpretive problem is local and continuous: what does one no-reset trace actually look like around one transition, and how does that help explain the benchmark tables? This is where the chapter reconnects with the online core and visualization notebooks.\n", "\n", "Unlike the reset trace inspection, the emphasis here is not on one already-declared event sequence. It is on a continuous detection function that can later be interpreted under several thresholds and policies.\n" ] }, { "cell_type": "markdown", "id": "8821be3e", "metadata": {}, "source": [ "### 9.1 Selecting a cached local transition case\n", "\n", "Because no-reset benchmarking is transition-centered, the most natural concrete case for trace inspection is one bisegment provider rather than one arbitrary full time series. The bisegment table gives us the local case name, and the benchmark registry gives us the exact continuous trace that was already executed for that case.\n", "\n", "Using the registry avoids rerunning the detector and keeps the local plot tied to the same execution cache used by the aggregate tables." ] }, { "cell_type": "code", "execution_count": null, "id": "983162bd", "metadata": {}, "outputs": [], "source": [ "row = bisegment_table.iloc[10]\n", "name = row[\"bisegment_name\"]\n", "registry_items = list(benchmark.registry.executions_registry.items())\n", "\n", "matching_items = [\n", " (descr, single_run)\n", " for descr, single_run in registry_items\n", " if descr.detector_description == entry.description and descr.provider_description.name == name\n", "]\n", "\n", "if not matching_items:\n", " matching_items = [\n", " (descr, single_run) for descr, single_run in registry_items if descr.provider_description.name == name\n", " ]\n", "\n", "if not matching_items:\n", " available_names = sorted({descr.provider_description.name for descr, _ in registry_items})\n", " raise LookupError(f\"No run found for bisegment '{name}'. Available names count: {len(available_names)}\")\n", "\n", "key, inspection_run = matching_items[0]\n", "bisegment_provider = inspection_run.provider\n", "inspection_trace = inspection_run.trace\n", "transformer = entry.data_transformer\n", "transformed_bisegment_provider = (\n", " transformer.transform(bisegment_provider) if transformer is not None else bisegment_provider\n", ")\n", "\n", "print(\"Selected bisegment:\", name)\n", "print(\"Bisegment provider annotation:\", bisegment_provider.annotation)\n", "print(\"Bisegment change points:\", list(bisegment_provider.change_points))\n", "print(\"Registry key:\", key)" ] }, { "cell_type": "markdown", "id": "535a9417", "metadata": {}, "source": [ "### 9.2 Inspecting the cached continuous no-reset trace\n", "\n", "The cached trace is still an un-thresholded continuous detector output. The table-level classification result comes from applying threshold and policy logic to this trace; the trace itself stores the detection function and timing information.\n", "\n", "For no-reset runs, `detected_change_points` is usually empty because declarations are not baked into the detector run. In the next plot we explicitly overlay threshold-crossing candidates for the chosen threshold." ] }, { "cell_type": "code", "execution_count": null, "id": "d579dcac", "metadata": {}, "outputs": [], "source": [ "print(\"Detected change points stored in raw no-reset trace:\", inspection_trace.detected_change_points)\n", "print(\"Peak detection statistic:\", float(inspection_trace.detection_function.max()))" ] }, { "cell_type": "markdown", "id": "00b959ae", "metadata": {}, "source": [ "### 9.3 Visualizing the continuous trace with one chosen threshold\n", "\n", "The last problem is visual explanation. We want to look at the continuous trace and place one interesting threshold on top of it so the link between table-level summaries and local signal behavior becomes concrete.\n", "\n", "The figure below uses `OnlineCpdPlotter` for the provider-trace layout, then overlays the chosen threshold on the detection-function panel so the continuous no-reset story stays visible.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "4790da0d", "metadata": {}, "outputs": [], "source": [ "threshold_crossings = np.flatnonzero(np.asarray(inspection_trace.detection_function) >= chosen_threshold).tolist()\n", "threshold_crossings = [\n", " point\n", " for index, point in enumerate(threshold_crossings)\n", " if index == 0 or threshold_crossings[index - 1] != point - 1\n", "]\n", "\n", "plotter = OnlineCpdPlotter(\n", " backend=DrawBackend.MATPLOTLIB,\n", " data_provider=transformed_bisegment_provider,\n", " detection_trace=inspection_trace,\n", ")\n", "plotter.set_ground_truth(list(bisegment_provider.change_points), margin=10)\n", "plotter.set_legend_axis(\"detection_function\")\n", "fig, axes = plt.subplots(3, 1, figsize=(14, 10), sharex=True)\n", "fig.suptitle(f\"No-reset trace inspection at threshold={chosen_threshold:.2f}\", fontsize=16)\n", "fig = plotter.draw(\n", " figure=fig,\n", " axes={\"timeseries\": axes[0], \"detection_function\": axes[1], \"processing_time\": axes[2]},\n", ")\n", "axes[1].axhline(chosen_threshold, color=\"green\", linestyle=\"--\", linewidth=2, label=\"Chosen Threshold\")\n", "for index, point in enumerate(threshold_crossings):\n", " axes[0].axvline(point, color=\"green\", linestyle=\"--\", linewidth=2, alpha=0.8)\n", " axes[1].axvline(\n", " point,\n", " color=\"green\",\n", " linestyle=\"--\",\n", " linewidth=2,\n", " alpha=0.8,\n", " label=\"Threshold Crossing\" if index == 0 else None,\n", " )\n", "for ax in axes:\n", " handles, labels = ax.get_legend_handles_labels()\n", " if labels:\n", " ax.legend(loc=\"upper left\", fontsize=9)\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "id": "38ca6369", "metadata": {}, "source": [ "**Summary of Section 9:** No-reset trace inspection is about continuous signal interpretation. The benchmark tables summarize thresholded policy outcomes, but the raw no-reset trace explains how those outcomes arise from one evolving detection function around one concrete transition.\n" ] }, { "cell_type": "markdown", "id": "f70856a3", "metadata": {}, "source": [ "## Section 10. Final Recap\n", "\n", "This chapter rebuilt the no-reset benchmark story around the current API. We created one no-reset entry with a threshold range, defined policy-driven classification semantics, generated global and transition-specific classification tables, computed ARL-by-state tables, inspected bisegment-level results, learned where `entry.bisegment_cut` applies, visualized the major table families, and tied those summaries back to one continuous local trace.\n", "\n", "That is what distinguishes no-reset benchmarking in PySATL-CPD. It is not just \u201creset benchmarking without calling reset.\u201d It is a transition-aware, policy-dependent evaluation framework built around continuous detector behavior and scenario-specific analysis tables keyed by detector description.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.13" }, "mystnb": { "execution_mode": "force" } }, "nbformat": 4, "nbformat_minor": 5 }