{ "cells": [ { "cell_type": "markdown", "id": "5ea7e543", "metadata": {}, "source": [ "# 01. PySATL Data API\n", "\n", "## Section 1. Orientation\n", "\n", "The PySATL-CPD Data API is the layer that turns raw arrays, tables, and labels into a stable vocabulary for the rest of the project. Change point detectors do not want to care whether a signal started as a NumPy array or a pandas `DataFrame`; benchmark code does not want to care how states or transitions were encoded. The data layer is where those concerns are resolved once, in one consistent shape.\n", "\n", "This notebook is deliberately detailed because the whole tutorial series leans on the abstractions introduced here. We will start with plain providers, add labels and segment metadata, build labeled providers, explore slicing and querying operations, and finally move up to datasets, transformers, and file loaders. Every major concept is accompanied by executable code, and every subsection closes with a short summary so the notebook can serve both as a first tutorial and as a reference chapter.\n" ] }, { "cell_type": "markdown", "id": "c2f7de2c", "metadata": {}, "source": [ "## What in this notebook\n", "\n", "By the end of this notebook, you should be able to answer four practical questions.\n", "\n", "- How do I represent a raw univariate or multivariate time series before labels are available?\n", "- How do I attach ground-truth segment information and recover change points, states, and transitions from it?\n", "- How do I group several labeled providers into one benchmark-ready dataset and derive state-specific or transition-specific subsets from it?\n", "- How do I move between generated data, transformed feature views, and file-backed datasets without leaving the public data-layer model?\n", "\n", "The generator notebook will show how to produce synthetic series. This notebook explains what those generated objects become once they enter the common data API used by detectors and benchmarks.\n" ] }, { "cell_type": "markdown", "id": "cfe9a847", "metadata": {}, "source": [ "## Interface Overview\n", "\n", "Before we touch concrete classes, it helps to see the shape of the data layer as a ladder of abstractions. A `DataProvider` is the smallest building block: it is a sequential container with an annotation, a length, and the ability to cut and merge data. A `LabeledData` provider adds segment-aware behavior: it stores an ordered labeling, derives change points from segment boundaries, and can expose segment or bisegment views.\n", "\n", "Above individual providers sits `IDataset`, the shared collection abstraction. A dataset behaves like a sequence of labeled providers, but it also knows how to split, merge, and aggregate state and transition information across the collection. The concrete `Dataset` class is the general-purpose implementation for regular labeled time series collections, while `StateDataset` is the special case for no-change windows associated with one state.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "16567a30", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "from IPython.display import display\n", "\n", "from pysatl_cpd.data import (\n", " Dataset,\n", " FolderCsvColumns,\n", " IDataset,\n", " NDArrayMultivariateProvider,\n", " NDArrayUnivariateProvider,\n", " PandasLabeledData,\n", " PlainMultivariateLabeledData,\n", " PlainUnivariateLabeledData,\n", " StateDataset,\n", " TimeseriesAnnotation,\n", " TransitionDescriptor,\n", " UnlabeledTimeseriesAnnotation,\n", " load_folder_csv_dataset,\n", ")\n", "from pysatl_cpd.data.providers.labeled.segments_labeling import SegmentsLabeling\n", "from pysatl_cpd.data.providers.plain.pd_provider import PandasDataProvider\n", "from pysatl_cpd.data.providers.transformers.columns import (\n", " ColumnFeatureCreator,\n", " ColumnsSelectorTransformer,\n", ")\n", "from pysatl_cpd.data.typedefs import (\n", " BisegmentAnnotation,\n", " NoChangeSeriesAnnotation,\n", " SegmentAnnotation,\n", " SegmentInfo,\n", " StateDescriptor,\n", " frozendict,\n", ")\n", "\n", "\n", "def _find_repo_root(start=None):\n", " import pathlib\n", "\n", " cwd = pathlib.Path(start or pathlib.Path.cwd())\n", " for parent in [cwd] + list(cwd.parents):\n", " if (parent / \"pyproject.toml\").exists():\n", " return parent\n", " return cwd" ] }, { "cell_type": "markdown", "id": "d9acc6e4", "metadata": {}, "source": [ "## Section 2. Providers\n", "\n", "Providers solve the first practical problem in the data layer: how do we keep raw observations together with minimal metadata before ground-truth change information exists? The answer matters because later layers should be able to assume a common provider interface even when labels are still absent.\n", "\n", "```python\n", "class DataProvider:\n", " @property\n", " def annotation(self) -> AnnotationT: ...\n", "\n", " def __len__(self) -> int: ...\n", " def __iter__(self) -> Iterator[DataT]: ...\n", "\n", " def cut(self, start: int, stop: int, *, annotation=None) -> Self: ...\n", " @classmethod\n", " def merge(cls, providers, annotation_builder=None) -> Self: ...\n", "```\n", "\n", "\n", "In this section, we will look at the three most immediate representations: one-dimensional NumPy data, two-dimensional NumPy data, and pandas-backed tables. The concrete examples are intentionally small so the API shape is easy to read.\n" ] }, { "cell_type": "markdown", "id": "064b0f25", "metadata": {}, "source": [ "### 2.1 NDArrayUnivariateProvider\n", "\n", "When a time series has exactly one numeric observation per time step, the simplest useful provider is a one-dimensional NumPy wrapper. The motivation for this class is not just convenience. It gives the rest of the project a uniform interface for iteration, slicing, and annotation handling, while keeping the representation as lightweight as possible.\n", "\n", "The key API idea is that `NDArrayUnivariateProvider` behaves like a sequential container. You can iterate through observations, ask for its length, inspect the copied `raw_data`, and cut out an inclusive slice while optionally attaching a fresh annotation.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "311c0747", "metadata": {}, "outputs": [], "source": [ "univariate_values = np.array([0.5, 0.7, 1.1, 0.9, 1.3], dtype=np.float64)\n", "univariate_provider = NDArrayUnivariateProvider(\n", " univariate_values,\n", " UnlabeledTimeseriesAnnotation(name=\"univariate_demo\"),\n", ")\n", "\n", "print(\"Annotation:\", univariate_provider.annotation)\n", "print(\"Length:\", len(univariate_provider))\n", "print(\"Iterated values:\", list(univariate_provider))\n", "print(\"Raw data copy:\", univariate_provider.raw_data)\n", "\n", "univariate_slice = univariate_provider.cut(\n", " 1,\n", " 3,\n", " annotation=UnlabeledTimeseriesAnnotation(name=\"univariate_demo_slice\"),\n", ")\n", "print(\"Slice values:\", list(univariate_slice))\n", "print(\"Slice annotation:\", univariate_slice.annotation)" ] }, { "cell_type": "markdown", "id": "48a25c07", "metadata": {}, "source": [ "### 2.2 NDArrayMultivariateProvider\n", "\n", "The next problem is multivariate data. A detector or benchmark often wants several numeric features at each time step, but it still wants the same provider contract: iterate step-by-step, preserve annotation metadata, and allow consistent slicing. A plain two-dimensional array is a natural storage choice, but it still needs to be wrapped into the provider model.\n", "\n", "`NDArrayMultivariateProvider` treats rows as time steps and columns as features. Iteration yields one feature row at a time, while `raw_data` gives a copied matrix for inspection, plotting, or comparison with transformed versions later in the notebook.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "8b1bbcac", "metadata": {}, "outputs": [], "source": [ "multivariate_values = np.array(\n", " [\n", " [0.5, 10.0],\n", " [0.7, 10.2],\n", " [1.1, 10.4],\n", " [0.9, 10.7],\n", " ],\n", " dtype=np.float64,\n", ")\n", "multivariate_provider = NDArrayMultivariateProvider(\n", " multivariate_values,\n", " UnlabeledTimeseriesAnnotation(name=\"multivariate_demo\"),\n", ")\n", "\n", "print(\"Annotation:\", multivariate_provider.annotation)\n", "print(\"Length:\", len(multivariate_provider))\n", "print(\"Rows from iteration:\", [row.tolist() for row in multivariate_provider])\n", "print(\"Raw data shape:\", multivariate_provider.raw_data.shape)\n", "\n", "multivariate_slice = multivariate_provider.cut(\n", " 1,\n", " 2,\n", " annotation=UnlabeledTimeseriesAnnotation(name=\"multivariate_demo_slice\"),\n", ")\n", "print(\"Slice rows:\", [row.tolist() for row in multivariate_slice])" ] }, { "cell_type": "markdown", "id": "05217967", "metadata": {}, "source": [ "### 2.3 PandasDataProvider\n", "\n", "A raw NumPy matrix is compact, but many workflows are easier when features have names. That is the problem the pandas-backed provider solves. It keeps the same provider semantics while preserving column labels, which later becomes important for feature selection, plotting, and benchmark-time transformations.\n", "\n", "The important API distinction is that a `PandasDataProvider` still behaves like a provider first and a table second. You can iterate through it like any other provider, but you can also ask for the copied `dataset` and the list of `columns` to stay in a more tabular mindset.\n", "Beyond basic iteration, `select_columns()` returns a new `PandasDataProvider` with only the named columns. The optional `rename_provider=True` flag appends the selected column names to the annotation name, which makes the result easier to identify later in a workflow.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "1a420b42", "metadata": {}, "outputs": [], "source": [ "pandas_frame = pd.DataFrame(\n", " {\n", " \"value\": [0.5, 0.7, 1.1, 0.9],\n", " \"aux\": [10.0, 10.2, 10.4, 10.7],\n", " }\n", ")\n", "pandas_provider = PandasDataProvider(\n", " pandas_frame,\n", " UnlabeledTimeseriesAnnotation(name=\"pandas_demo\"),\n", ")\n", "\n", "print(\"Columns:\", list(pandas_provider.columns))\n", "print(\"Length:\", len(pandas_provider))\n", "print(\"Rows from iteration:\", [row.tolist() if hasattr(row, \"tolist\") else row for row in pandas_provider])\n", "display(pandas_provider.dataset)\n", "\n", "selected_plain = pandas_provider.select_columns([\"value\"], rename_provider=True)\n", "print(\"Selected annotation:\", selected_plain.annotation)\n", "display(selected_plain.dataset)" ] }, { "cell_type": "markdown", "id": "a8e9f987", "metadata": {}, "source": [ "**Summary of Section 2:** Unlabeled providers all solve the same interface problem, but they do it with different storage choices. Use the univariate NumPy provider for single-feature scalar signals, the multivariate NumPy provider for matrix-shaped numeric data, and the pandas provider when named columns make later operations easier to read and safer to configure.\n" ] }, { "cell_type": "markdown", "id": "43c27295", "metadata": {}, "source": [ "## Section 3. Labels, Segments, and Ground Truth\n", "\n", "A raw provider tells us what values were observed, but it does not tell us where the known change points are or what each regime means. This section solves that second problem by introducing the segment vocabulary that all labeled providers share.\n", "\n", "The central idea is simple: a labeled time series is built from an ordered sequence of segment descriptors. Once those descriptors exist, change points, states, transitions, segment slices, and transition slices all become derived views of one source of truth.\n" ] }, { "cell_type": "markdown", "id": "9389fd9a", "metadata": {}, "source": [ "### 3.1 StateDescriptor\n", "\n", "Before we describe a segment, we need a language for its state. A change-point benchmark rarely cares only that \"a segment exists\"; it usually cares whether the segment is baseline, shifted, anomalous, noisy, or part of some named regime family. The state descriptor is the lightweight key-value vocabulary used for that purpose.\n", "\n", "API-wise, `StateDescriptor` is intentionally simple. It behaves like an immutable mapping of small state attributes, so it can be compared, stored in sets, reused in filters, and printed in a compact way. The notebook series will keep reusing this same object when it talks about dataset states, transitions, and state-specific ARL slices.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "8b3599b1", "metadata": {}, "outputs": [], "source": [ "baseline_state = StateDescriptor(type=\"baseline\", regime=\"stable\")\n", "shifted_state = StateDescriptor(type=\"shifted\", regime=\"changed\")\n", "\n", "print(\"Baseline state:\", baseline_state)\n", "print(\"Baseline as dict:\", dict(baseline_state))\n", "print(\"Shifted state:\", shifted_state)\n", "print(\"States compare by content:\", baseline_state == StateDescriptor(type=\"baseline\", regime=\"stable\"))" ] }, { "cell_type": "markdown", "id": "c44f7d19", "metadata": {}, "source": [ "### 3.2 SegmentInfo\n", "\n", "Once a state vocabulary exists, the next problem is describing one concrete segment in one concrete time series. A segment needs boundaries, an order in the sequence, and a state label. That is exactly what `SegmentInfo` provides.\n", "\n", "The key API fields are `segment_num`, `segment_start`, `segment_end`, and `state`. Because change points in this project are zero-based, the segment boundaries are also expressed in zero-based coordinates. Later code never has to guess where a segment begins or ends; it reads those boundaries directly from the segment table.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6f7c77b6", "metadata": {}, "outputs": [], "source": [ "segment_rows = [\n", " SegmentInfo(segment_num=0, segment_start=0, segment_end=5, state=baseline_state),\n", " SegmentInfo(segment_num=1, segment_start=6, segment_end=10, state=shifted_state),\n", " SegmentInfo(segment_num=2, segment_start=11, segment_end=14, state=baseline_state),\n", "]\n", "\n", "segment_table = pd.DataFrame(\n", " {\n", " \"segment_num\": [segment.segment_num for segment in segment_rows],\n", " \"segment_start\": [segment.segment_start for segment in segment_rows],\n", " \"segment_end\": [segment.segment_end for segment in segment_rows],\n", " \"state\": [dict(segment.state) for segment in segment_rows],\n", " }\n", ")\n", "display(segment_table)" ] }, { "cell_type": "markdown", "id": "bc835480", "metadata": {}, "source": [ "### 3.3 Change Points, States, and Transitions\n", "\n", "The main motivation for storing segment rows instead of only raw change-point indices is that richer structure can be derived from them automatically. Once segments are ordered, change points are just the starts of later segments. States are the unique segment states. Transitions are the pairs of states that appear between adjacent segments.\n", "\n", "For the API user, this means you do not manually maintain three separate sources of truth. You define segments once, and labeled providers expose `change_points`, `states`, and `transitions` as derived properties.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "34a5e6d2", "metadata": {}, "outputs": [], "source": [ "manual_series = np.array([0.2, -0.1, 0.3, 0.1, 0.0, 0.2, 3.0, 3.3, 2.9, 3.1, 2.8, 0.1, -0.2, 0.0, 0.2])\n", "# Change points are the start indices of all segments except the first\n", "derived_change_points = tuple(segment.segment_start for segment in segment_rows[1:])\n", "# States are the unique state descriptors across segments\n", "derived_states = {segment.state for segment in segment_rows}\n", "# Transitions are adjacent pairs between consecutive segments\n", "derived_transitions = [\n", " TransitionDescriptor(curr_state=segment_rows[i].state, next_state=segment_rows[i + 1].state)\n", " for i in range(len(segment_rows) - 1)\n", "]\n", "print(\"Zero-based change points:\", derived_change_points)\n", "print(\"States:\", list(derived_states))\n", "print(\"Transitions:\")\n", "for tr in derived_transitions:\n", " print(tr)" ] }, { "cell_type": "markdown", "id": "f6fd28b5", "metadata": {}, "source": [ "Labeled providers store their segment table as a `SegmentsLabeling` \u2014 a `Sequence[SegmentInfo]` that also provides query and transformation methods. You will see it in action as the `.segments_labeling` property once we build labeled providers in \u00a74.\n" ] }, { "cell_type": "markdown", "id": "98a8c185", "metadata": {}, "source": [ "### 3.4 Annotations\n", "Every provider carries an annotation that describes what the data represents. PySATL defines a small hierarchy of annotation types, each tuned for a different stage of the data lifecycle.\n" ] }, { "cell_type": "markdown", "id": "216035ad", "metadata": {}, "source": [ "#### 3.4.1 TimeseriesAnnotation\n", "`TimeseriesAnnotation` is the base annotation for all labeled time series. It stores a `name`, an optional `source` (e.g. a file path or generator name), and optional immutable `metadata`. The more specific annotation types below extend it.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "4ddee3d4", "metadata": {}, "outputs": [], "source": [ "ts_ann = TimeseriesAnnotation(\n", " name=\"demo_series\",\n", " source=\"notebook\",\n", " metadata=frozendict(version=\"1.0\"),\n", ")\n", "print(\"Name:\", ts_ann.name)\n", "print(\"Source:\", ts_ann.source)\n", "print(\"Type:\", type(ts_ann).__name__)" ] }, { "cell_type": "markdown", "id": "19b4249d", "metadata": {}, "source": [ "#### 3.4.2 UnlabeledTimeseriesAnnotation\n", "`UnlabeledTimeseriesAnnotation` extends `TimeseriesAnnotation` for data without segment labels. We already used it in \u00a72 with raw providers. Its presence makes the lack of labels explicit while keeping the same base fields.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "8d669f61", "metadata": {}, "outputs": [], "source": [ "unlabeled_ann = UnlabeledTimeseriesAnnotation(name=\"raw_series\")\n", "print(\"Name:\", unlabeled_ann.name)\n", "print(\"Type:\", type(unlabeled_ann).__name__)" ] }, { "cell_type": "markdown", "id": "fc20306e", "metadata": {}, "source": [ "#### 3.4.3 SegmentAnnotation\n", "`SegmentAnnotation` extends `TimeseriesAnnotation` with a `state` field. This is the annotation attached to providers returned by `query_segments()` \u2014 it records which regime the segment belongs to.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "f6f8e931", "metadata": {}, "outputs": [], "source": [ "seg_ann = SegmentAnnotation(\n", " name=\"baseline_segment\",\n", " state=baseline_state,\n", ")\n", "print(\"Name:\", seg_ann.name)\n", "print(\"State:\", dict(seg_ann.state))\n", "print(\"Type:\", type(seg_ann).__name__)" ] }, { "cell_type": "markdown", "id": "46a3e9d6", "metadata": {}, "source": [ "#### 3.4.4 BisegmentAnnotation and TransitionDescriptor\n", "`BisegmentAnnotation` extends `TimeseriesAnnotation` with a `transition` field. Its type is `TransitionDescriptor`, a frozen dataclass with `curr_state` and `next_state`. This is what `query_bisegments()` attaches \u2014 it names both sides of a change point.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "aca2d422", "metadata": {}, "outputs": [], "source": [ "transition = TransitionDescriptor(\n", " curr_state=baseline_state,\n", " next_state=shifted_state,\n", ")\n", "biseg_ann = BisegmentAnnotation(\n", " name=\"baseline_to_shifted\",\n", " transition=transition,\n", ")\n", "print(\"Name:\", biseg_ann.name)\n", "print(\"Current state:\", dict(biseg_ann.transition.curr_state))\n", "print(\"Next state:\", dict(biseg_ann.transition.next_state))\n", "print(\"Type:\", type(biseg_ann).__name__)" ] }, { "cell_type": "markdown", "id": "8237bb02", "metadata": {}, "source": [ "#### 3.4.5 NoChangeSeriesAnnotation\n", "`NoChangeSeriesAnnotation` extends `SegmentAnnotation` for a series that contains no change points. It carries a `state` just like `SegmentAnnotation`, but signals that the entire span belongs to one regime. This is used by `StateDataset` to mark no-change windows.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "4634d4e8", "metadata": {}, "outputs": [], "source": [ "nochange_ann = NoChangeSeriesAnnotation(\n", " name=\"stable_run\",\n", " state=baseline_state,\n", ")\n", "print(\"Name:\", nochange_ann.name)\n", "print(\"State:\", dict(nochange_ann.state))\n", "print(\"Type:\", type(nochange_ann).__name__)" ] }, { "cell_type": "markdown", "id": "cfb25c8d", "metadata": {}, "source": [ "**Summary of Section 3:** `StateDescriptor` explains what regime a segment belongs to, `SegmentInfo` explains where that regime lives in the series, and everything else that matters for ground truth flows from those segment rows. Change points are zero-based boundaries derived from the same labeling that later powers segment and bisegment queries.\n", "The annotation hierarchy introduced in \u00a73.4 mirrors the lifecycle of data: `TimeseriesAnnotation` (base) \u2192 `UnlabeledTimeseriesAnnotation` (pre-label data), `SegmentAnnotation` (single segment with `state`), `BisegmentAnnotation` (transition window with `transition`), and `NoChangeSeriesAnnotation` (stable no-change series).\n" ] }, { "cell_type": "markdown", "id": "6836afa3-104c-4e00-94ea-4480df4d626a", "metadata": {}, "source": [ "### 3.5 SegmentsLabeling\n", "\n", "`SegmentsLabeling` is the container that wraps the ordered list of `SegmentInfo` into a validated, queryable sequence. Every labeled provider stores its segment information as a `SegmentsLabeling` internally \u2014 it is the glue between the `SegmentInfo` rows built in \u00a73.2\u2013\u00a73.3 and the concrete labeled provider constructors in \u00a74.\n", "\n", "The class validates that segments are contiguous and non-overlapping. Beyond validation, it powers the `query_segments()` and `query_bisegments()` methods that produce collections of segment views. Calling `provider.query_segments(filter_fn)` returns a list of labeled providers, each trimmed to one matching segment with its own annotation and change points. This makes per-segment analysis natural \u2014 iterate over segments, compute statistics per regime, or feed individual segment windows into a detector. Similarly, `query_bisegments()` returns two-segment windows centered on each change point, the natural unit for transition-aware evaluation.\n", "\n", "You rarely construct a `SegmentsLabeling` by hand in day-to-day use \u2014 the labeled provider factories do it automatically. But understanding it explains why the `segment_info` parameter of `from_unlabeled_data()` expects an iterable of `SegmentInfo`, what validation it undergoes, and how the same segment row indexing powers the query operations in \u00a75.\n", "\n", "The example below shows direct construction from the `segment_rows` we built in \u00a73.2 and a tabular view via `to_dataframe()`, which includes both the boundary columns and the state descriptor keys (e.g. `type`, `regime`) as columns.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "d6d5e815", "metadata": {}, "outputs": [], "source": [ "segments_labeling = SegmentsLabeling(segment_rows)\n", "print(\"Number of segments:\", len(segments_labeling))\n", "print(\"Segment states:\", [dict(s) for s in segments_labeling.states])\n", "print(\"Data length:\", segments_labeling.data_len)\n", "display(segments_labeling.to_dataframe())" ] }, { "cell_type": "markdown", "id": "14a3eeaf", "metadata": {}, "source": [ "## Section 4. Labeled Providers\n", "\n", "Now we can solve the real practical problem: how do we combine raw provider data with ordered segment labeling and keep the result easy to slice, inspect, and reuse? The answer is the family of labeled providers.\n", "\n", "The same logical model appears in several concrete implementations. The plain univariate and multivariate variants are NumPy-backed. The pandas-backed variant keeps named columns and integrates naturally with column selection and tabular inspection.\n" ] }, { "cell_type": "markdown", "id": "69461285", "metadata": {}, "source": [ "### 4.1 PlainUnivariateLabeledData\n", "\n", "The plain univariate labeled provider is the most compact complete labeled object in the data layer. It is useful when you want a scalar signal with explicit ground truth and you do not need named columns.\n", "\n", "Its API combines the raw-provider behavior with labeled-provider behavior. You can iterate through observations, access `raw_data`, inspect `change_points`, and query states or transitions from the same object.\n", "The `from_unlabeled_data()` class method takes an unlabeled provider, a sequence of `SegmentInfo`, and a `TimeseriesAnnotation`, and returns a fully connected labeled provider whose change points, states, and transitions are derived automatically. The `.segments_labeling` property exposes the underlying `SegmentsLabeling` container we discussed in \u00a73.5.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "a65ca7f1", "metadata": {}, "outputs": [], "source": [ "manual_series = np.array([0.2, -0.1, 0.3, 0.1, 0.0, 0.2, 3.0, 3.3, 2.9, 3.1, 2.8, 0.1, -0.2, 0.0, 0.2])\n", "manual_labeled = PlainUnivariateLabeledData.from_unlabeled_data(\n", " NDArrayUnivariateProvider(manual_series, UnlabeledTimeseriesAnnotation(name=\"manual_labeled_univariate\")),\n", " segment_rows,\n", " TimeseriesAnnotation(name=\"manual_labeled_univariate\"),\n", ")\n", "print(\"Annotation:\", manual_labeled.annotation)\n", "print(\"Length:\", len(manual_labeled))\n", "print(\"Raw data:\", manual_labeled.raw_data)\n", "print(\"Change points:\", list(manual_labeled.change_points))\n", "print(\"States:\", [dict(state) for state in manual_labeled.states])\n", "# The SegmentsLabeling container\n", "labeling = manual_labeled.segments_labeling\n", "print(f\"\\nSegmentsLabeling: {len(labeling)} segments, type={type(labeling).__name__}\")\n", "for seg in labeling:\n", " print(f\" Segment {seg.segment_num}: {seg.segment_start}--{seg.segment_end} state={dict(seg.state)}\")" ] }, { "cell_type": "markdown", "id": "2101d2fe", "metadata": {}, "source": [ "### 4.2 PlainMultivariateLabeledData\n", "\n", "A multivariate labeled provider solves the same problem, but now each time step is a feature row rather than a scalar. This is the most natural format for plain synthetic multivariate data when you want to stay NumPy-backed.\n", "\n", "The API difference is small but important: `raw_data` is now a matrix, and iteration yields rows. All higher-level label-derived behavior, including change points and segment queries, stays conceptually the same.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "e641ab73", "metadata": {}, "outputs": [], "source": [ "multivariate_labeling = [\n", " SegmentInfo(segment_num=0, segment_start=0, segment_end=2, state=baseline_state),\n", " SegmentInfo(segment_num=1, segment_start=3, segment_end=4, state=shifted_state),\n", "]\n", "multivariate_labeled = PlainMultivariateLabeledData.from_unlabeled_data(\n", " NDArrayMultivariateProvider(\n", " np.array([[0.0, 10.0], [0.2, 10.2], [-0.1, 9.8], [3.0, 20.0], [2.8, 19.7]], dtype=np.float64),\n", " UnlabeledTimeseriesAnnotation(name=\"manual_labeled_multivariate\"),\n", " ),\n", " multivariate_labeling,\n", " TimeseriesAnnotation(name=\"manual_labeled_multivariate\"),\n", ")\n", "\n", "print(\"Raw shape:\", multivariate_labeled.raw_data.shape)\n", "print(\"Rows:\", [row.tolist() for row in multivariate_labeled])\n", "print(\"Change points:\", list(multivariate_labeled.change_points))" ] }, { "cell_type": "markdown", "id": "f0486f1d", "metadata": {}, "source": [ "### 4.3 PandasLabeledData\n", "\n", "Pandas-backed labeled data is the richest form for day-to-day work because it preserves feature names. That makes it easier to inspect the data table, to select columns, and later to configure benchmarks that operate on specific feature subsets.\n", "\n", "The most important methods are `feature_columns`, `dataset()`, `select_columns()`, and `create_feature_column()`. They let you stay in one labeled-provider abstraction while still doing practical table-shaped manipulations.\n", "\n", "The example below constructs a `PandasLabeledData` from a manually created DataFrame with two feature columns (`value` and `aux`) and a three-segment baseline\u2013shift\u2013baseline labeling.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "38b96c7d", "metadata": {}, "outputs": [], "source": [ "pandas_data = pd.DataFrame(\n", " {\n", " \"value\": [0.2, -0.3, 0.5, -0.1, 0.4, 0.1, 3.1, 2.9, 3.3, 2.8, 3.0, 0.3, -0.2, 0.0, 0.1],\n", " \"aux\": [5.1, 4.9, 5.2, 5.0, 4.8, 5.1, 7.0, 7.2, 6.9, 7.1, 6.8, 5.0, 5.2, 4.9, 5.1],\n", " }\n", ")\n", "pandas_labeled = PandasLabeledData.from_unlabeled_data(\n", " PandasDataProvider(pandas_data, UnlabeledTimeseriesAnnotation(name=\"pandas_labeled_demo\")),\n", " [\n", " SegmentInfo(segment_num=0, segment_start=0, segment_end=5, state=baseline_state),\n", " SegmentInfo(segment_num=1, segment_start=6, segment_end=10, state=shifted_state),\n", " SegmentInfo(segment_num=2, segment_start=11, segment_end=14, state=baseline_state),\n", " ],\n", " TimeseriesAnnotation(name=\"pandas_labeled_demo\"),\n", ")\n", "print(\"Feature columns:\", list(pandas_labeled.feature_columns))\n", "print(\"Change points:\", list(pandas_labeled.change_points))\n", "display(pandas_labeled.dataset().head(8))" ] }, { "cell_type": "markdown", "id": "2aa48c9a", "metadata": {}, "source": [ "**Summary of Section 4:** Labeled providers are where raw observations and segment metadata finally become one object. The plain variants keep things lightweight, while the pandas-backed variant adds named columns and richer table operations. Builder helpers let generated series enter this model directly.\n" ] }, { "cell_type": "markdown", "id": "0d3911c4", "metadata": {}, "source": [ "## Section 5. Provider Operations\n", "\n", "Once data is labeled, the next question is not \"what is it?\" but \"how do I extract the exact view I need?\" This section solves that problem. We will slice a labeled provider directly, extract per-segment views, and extract per-transition bisegment views.\n", "\n", "These operations matter because later notebooks do not invent new data abstractions for training, inspection, or benchmarking. They reuse these same provider-level operations.\n" ] }, { "cell_type": "markdown", "id": "8f449d86", "metadata": {}, "source": [ "### 5.1 cut()\n", "\n", "The motivation for `cut()` is local inspection. Sometimes you want one contiguous window of a labeled series and you want the labels re-based to that window automatically. Doing this manually is error-prone because you would otherwise need to rewrite change-point positions and segment boundaries yourself.\n", "\n", "The `cut()` API takes inclusive start and end indices and returns a new labeled provider of the same concrete type. The sliced provider gets adjusted labeling and either a default or user-supplied annotation.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "1a1fd27f", "metadata": {}, "outputs": [], "source": [ "provider_cut = pandas_labeled.cut(2, 10)\n", "print(\"Cut annotation:\", provider_cut.annotation)\n", "print(\"Cut length:\", len(provider_cut))\n", "print(\"Cut change points:\", list(provider_cut.change_points))\n", "display(provider_cut.dataset())" ] }, { "cell_type": "markdown", "id": "080150f9", "metadata": {}, "source": [ "### 5.2 query_segments()\n", "\n", "The problem solved by `query_segments()` is selective state extraction. Instead of cutting raw index ranges by hand, you ask the provider for all segment-level views matching a predicate. That keeps the extraction logic tied to segment semantics rather than to memorized boundaries.\n", "\n", "Each returned provider is annotated as a `SegmentAnnotation` (introduced in \u00a73.4.3). That annotation carries the segment's `state`, making the result self-describing even after it has been separated from the original series.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "91326b3d", "metadata": {}, "outputs": [], "source": [ "baseline_segments = pandas_labeled.query_segments(lambda segment: segment.state[\"type\"] == \"baseline\")\n", "print(\"Number of baseline segments:\", len(baseline_segments))\n", "print(\"First baseline annotation:\", baseline_segments[0].annotation)\n", "print(\"First baseline change points:\", list(baseline_segments[0].change_points))\n", "display(baseline_segments[0].dataset())" ] }, { "cell_type": "markdown", "id": "da1f7e87", "metadata": {}, "source": [ "### 5.3 query_bisegments()\n", "\n", "Change-point evaluation often cares about local transition windows rather than about full time series. That is the motivation for `query_bisegments()`. A bisegment is the two-segment view centered on one true change point, and it is the natural unit for transition-aware no-reset evaluation.\n", "\n", "Each returned provider is annotated as a `BisegmentAnnotation` (introduced in \u00a73.4.4). Its `transition` field is a `TransitionDescriptor` containing `curr_state` and `next_state`, naming both sides of the change point.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "48e53921", "metadata": {}, "outputs": [], "source": [ "bisegments = pandas_labeled.query_bisegments()\n", "first_bisegment = bisegments[0]\n", "\n", "print(\"Number of bisegments:\", len(bisegments))\n", "print(\"First bisegment annotation:\", first_bisegment.annotation)\n", "print(\"First bisegment change points:\", list(first_bisegment.change_points))\n", "display(first_bisegment.dataset(state_columns={\"state\": \"regime\"}))" ] }, { "cell_type": "markdown", "id": "9d50cd20", "metadata": {}, "source": [ "### 5.4 merge()\n", "\n", "Sometimes the problem is the opposite of slicing: you have several compatible providers and want to stitch them back together into one longer labeled provider. That is exactly what `merge()` solves.\n", "\n", "The merge API is defined on the provider type itself. It validates compatibility, merges the raw unlabeled providers, merges segment labeling, and builds a merged annotation. This is the mechanism later reused by dataset-level `merge()` as well.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "08a7081d", "metadata": {}, "outputs": [], "source": [ "left = pandas_labeled.cut(0, 4)\n", "right = pandas_labeled.cut(5, len(pandas_labeled) - 1)\n", "merged_provider = type(pandas_labeled).merge([left, right])\n", "\n", "print(\"Merged length:\", len(merged_provider))\n", "print(\"Merged change points:\", list(merged_provider.change_points))\n", "print(\"Matches original length:\", len(merged_provider) == len(pandas_labeled))" ] }, { "cell_type": "markdown", "id": "9b965c3c", "metadata": {}, "source": [ "**Summary of Section 5:** Provider operations let you move between whole-series, segment-level, and transition-level views without leaving the labeled-provider abstraction. `cut()` is for contiguous windows, `query_segments()` is for state-aware extraction, `query_bisegments()` is for transition-aware extraction, and `merge()` reverses the process when compatible views need to be stitched back together.\n" ] }, { "cell_type": "markdown", "id": "69fbab01", "metadata": {}, "source": [ "## Section 6. Transformers\n", "The next problem is feature preparation. A labeled provider may already have the right labels and segmentation, but not the right feature view for a detector. This is common in multivariate data, where a benchmark or algorithm often wants one column or a small feature subset.\n", "Transformers solve that problem while staying inside the data-provider model. In this section we cover three transformer types: `ColumnsSelectorTransformer` for selecting feature columns, `ColumnFeatureCreator` for deriving new columns, and `ComposedTransformer` for chaining them together.\n" ] }, { "cell_type": "markdown", "id": "1fbe9161", "metadata": {}, "source": [ "### 6.1 ColumnsSelectorTransformer\n", "\n", "The motivation for `ColumnsSelectorTransformer` is simple: feature selection should be a first-class operation instead of ad hoc `DataFrame` slicing sprinkled through detector code. By packaging the selection as a transformer, you can inspect it, reuse it, and later pass the same object into benchmark entries.\n", "\n", "The API takes one column name or a sequence of column names. Its `annotation` property tells you how the transformation is described, and `transform()` applies the feature selection to a compatible pandas-backed labeled provider.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "de190564", "metadata": {}, "outputs": [], "source": [ "column_transformer = ColumnsSelectorTransformer(columns=[\"value\"])\n", "selected_provider = column_transformer.transform(pandas_labeled)\n", "\n", "print(\"Transformer annotation:\", column_transformer.annotation)\n", "print(\"Selected columns:\", list(selected_provider.feature_columns))\n", "print(\"Selected provider annotation:\", selected_provider.annotation)\n", "display(selected_provider.dataset().head())" ] }, { "cell_type": "markdown", "id": "00b0219a", "metadata": {}, "source": [ "### 6.2 ColumnFeatureCreator\n", "`ColumnFeatureCreator` appends a derived column computed row-wise from existing features. Use it when you need a transformation like a ratio, difference, or summary statistic that becomes a new feature column.\n", "The constructor takes the new column name and a callable that receives each pandas row and returns a value. The optional `rename_provider=True` flag updates the provider annotation name to reflect the added column.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "4506f857", "metadata": {}, "outputs": [], "source": [ "creator = ColumnFeatureCreator(\n", " name=\"value_sq\",\n", " mapping=lambda row: row[\"value\"] ** 2,\n", " rename_provider=True,\n", ")\n", "enhanced = creator.transform(pandas_labeled)\n", "print(\"Feature columns:\", list(enhanced.feature_columns))\n", "print(\"Provider name:\", enhanced.annotation.name)\n", "display(enhanced.dataset().head(3))" ] }, { "cell_type": "markdown", "id": "0cf7012c", "metadata": {}, "source": [ "### 6.3 ComposedTransformer\n", "`ComposedTransformer` chains multiple transformations into one reusable step using the `&` operator: the left transformer runs first, and its result becomes the input to the right transformer. This keeps the pipeline order visible and avoids temporary variables.\n", "The example below first selects a subset of columns, then appends a derived feature column computed from the remaining features.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ea3e93fb", "metadata": {}, "outputs": [], "source": [ "pipeline = ColumnsSelectorTransformer(columns=[\"value\", \"aux\"]) & ColumnFeatureCreator(\n", " name=\"product\",\n", " mapping=lambda row: row[\"value\"] * row[\"aux\"],\n", ")\n", "pipeline_result = pipeline.transform(pandas_labeled)\n", "print(\"Pipeline annotation:\", pipeline.annotation)\n", "print(\"Output columns:\", list(pipeline_result.feature_columns))\n", "print(\"Change points:\", list(pipeline_result.change_points))\n", "display(pipeline_result.dataset().head(4))" ] }, { "cell_type": "markdown", "id": "1b5f8e59", "metadata": {}, "source": [ "### 6.4 Why transformations belong in the data layer\n", "\n", "The same feature-selection problem appears in interactive inspection, in detector configuration, and in benchmarking. Keeping the transformation as a data-layer object means all three workflows can share one explicit transformation step instead of reimplementing the same slicing logic.\n", "\n", "In practice, the tutorial series will use transformers in two ways. Sometimes we apply them directly for inspection, as we did above. Later, benchmark entries will accept the same transformer so the feature view is part of the detector configuration rather than hidden in notebook-local preprocessing code.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ba67e6c5", "metadata": {}, "outputs": [], "source": [ "selected_aux_provider = ColumnsSelectorTransformer(columns=[\"aux\"], rename_provider=True).transform(pandas_labeled)\n", "print(\"Renamed selected annotation:\", selected_aux_provider.annotation)\n", "display(selected_aux_provider.dataset().head())" ] }, { "cell_type": "markdown", "id": "3affb5be", "metadata": {}, "source": [ "**Summary of Section 6:** Transformers keep feature preparation explicit and reusable. `ColumnsSelectorTransformer` selects feature columns, `ColumnFeatureCreator` appends derived columns, and `ComposedTransformer` chains steps together with the `&` operator. This becomes especially important once detectors and benchmarks start operating on multivariate labeled data.\n" ] }, { "cell_type": "markdown", "id": "5517d5d8", "metadata": {}, "source": [ "## Section 7. Datasets\n", "\n", "Individual labeled providers are useful for local inspection, but experiments and benchmarks normally work with collections. This section addresses that collection-level problem. We will create datasets, inspect their shared state and transition information, split them, filter them, merge them, and derive fixed-state no-change windows.\n", "\n", "The most important idea is that datasets do not invent a new data model. They are organized collections of the same labeled providers you have already seen.\n" ] }, { "cell_type": "markdown", "id": "8212833e", "metadata": {}, "source": [ "### 7.1 Creating a Dataset\n", "\n", "A `Dataset` is the general-purpose collection type for labeled providers. The motivation for this class is not just grouping objects together. It adds collection-level semantics such as the union of states and transitions, train/test splitting, and dataset-wide filtering.\n", "\n", "The API behaves like a sequence, so you can index or iterate over it directly. On top of that, properties such as `states` and `transitions` summarize the collection as a whole.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "82ef9177", "metadata": {}, "outputs": [], "source": [ "dataset = Dataset(\n", " [\n", " PlainUnivariateLabeledData.from_unlabeled_data(\n", " NDArrayUnivariateProvider(\n", " np.array([0.1, 0.3, -0.2, 0.4], dtype=np.float64),\n", " UnlabeledTimeseriesAnnotation(name=\"series_01\"),\n", " ),\n", " [\n", " SegmentInfo(segment_num=0, segment_start=0, segment_end=1, state=baseline_state),\n", " SegmentInfo(segment_num=1, segment_start=2, segment_end=3, state=shifted_state),\n", " ],\n", " TimeseriesAnnotation(name=\"series_01\"),\n", " ),\n", " PlainUnivariateLabeledData.from_unlabeled_data(\n", " NDArrayUnivariateProvider(\n", " np.array([-0.1, 2.9, 3.1, 2.8], dtype=np.float64),\n", " UnlabeledTimeseriesAnnotation(name=\"series_02\"),\n", " ),\n", " [\n", " SegmentInfo(segment_num=0, segment_start=0, segment_end=0, state=baseline_state),\n", " SegmentInfo(segment_num=1, segment_start=1, segment_end=3, state=shifted_state),\n", " ],\n", " TimeseriesAnnotation(name=\"series_02\"),\n", " ),\n", " PlainUnivariateLabeledData.from_unlabeled_data(\n", " NDArrayUnivariateProvider(\n", " np.array([0.5, 0.0, -0.3, 0.2, -0.1], dtype=np.float64),\n", " UnlabeledTimeseriesAnnotation(name=\"series_03\"),\n", " ),\n", " [\n", " SegmentInfo(segment_num=0, segment_start=0, segment_end=2, state=baseline_state),\n", " SegmentInfo(segment_num=1, segment_start=3, segment_end=4, state=shifted_state),\n", " ],\n", " TimeseriesAnnotation(name=\"series_03\"),\n", " ),\n", " PlainUnivariateLabeledData.from_unlabeled_data(\n", " NDArrayUnivariateProvider(\n", " np.array([2.9, 3.2, 3.0, 0.2, -0.1, 0.3], dtype=np.float64),\n", " UnlabeledTimeseriesAnnotation(name=\"series_04\"),\n", " ),\n", " [\n", " SegmentInfo(segment_num=0, segment_start=0, segment_end=2, state=shifted_state),\n", " SegmentInfo(segment_num=1, segment_start=3, segment_end=5, state=baseline_state),\n", " ],\n", " TimeseriesAnnotation(name=\"series_04\"),\n", " ),\n", " ]\n", ")\n", "print(\"Dataset type via public interface:\", isinstance(dataset, IDataset))\n", "print(\"Dataset size:\", len(dataset))\n", "print(\"Provider names:\", [provider.annotation.name for provider in dataset])\n", "print(\"Dataset states:\", [dict(state) for state in dataset.states])\n", "print(\"Dataset transitions:\", [repr(transition) for transition in dataset.transitions])" ] }, { "cell_type": "markdown", "id": "4fdcd35c", "metadata": {}, "source": [ "### 7.2 filter_by_annotation(), filter_by_segments(), and filter_by_bisegments()\n", "\n", "Once data is grouped into a dataset, the next problem is building focused subsets. Sometimes you filter by whole-provider annotation, sometimes by segment-level state, and sometimes by transition-level bisegment criteria. The dataset filtering helpers solve these three related but distinct collection problems.\n", "\n", "The APIs return new datasets whose providers are already in the right form for the requested view. That means segment filtering returns a dataset of segment providers, and bisegment filtering returns a dataset of bisegment providers, rather than merely returning raw indices.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "abdd19e8", "metadata": {}, "outputs": [], "source": [ "annotation_filtered = dataset.filter_by_annotation(\n", " lambda annotation: annotation.name.endswith(\"1\") or annotation.name.endswith(\"2\")\n", ")\n", "segment_filtered = dataset.filter_by_segments(lambda segment: segment.state[\"type\"] == \"baseline\")\n", "bisegment_filtered = dataset.filter_by_bisegments()\n", "\n", "print(\"Annotation-filtered size:\", len(annotation_filtered))\n", "print(\"Segment-filtered size:\", len(segment_filtered))\n", "print(\"Bisegment-filtered size:\", len(bisegment_filtered))\n", "print(\"First bisegment annotation from dataset filter:\", bisegment_filtered[0].annotation)" ] }, { "cell_type": "markdown", "id": "0a955d29", "metadata": {}, "source": [ "### 7.3 train_test_split() and merge()\n", "\n", "A benchmark-ready collection often needs two opposite operations: splitting into subsets and collapsing back into one provider. `train_test_split()` solves the first problem, and dataset-level `merge()` solves the second.\n", "\n", "The split API preserves the concrete dataset type, while `merge()` returns one labeled provider containing the data and labeling from all providers in sequence. Together, these two operations make it easy to move between collection-level and provider-level views.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "c0cba91d", "metadata": {}, "outputs": [], "source": [ "train_dataset, test_dataset = dataset.train_test_split(test_size=0.25, random_state=13)\n", "merged_dataset_provider = dataset.merge()\n", "\n", "print(\"Train/Test sizes:\", len(train_dataset), len(test_dataset))\n", "print(\"Merged provider length:\", len(merged_dataset_provider))\n", "print(\"Merged provider change points count:\", len(merged_dataset_provider.change_points))" ] }, { "cell_type": "markdown", "id": "78714674", "metadata": {}, "source": [ "### 7.4 StateDataset\n", "\n", "Some benchmark scenarios care specifically about no-change windows from one state. That is a different problem from general labeled-series collections, so the project provides `StateDataset` as a specialized abstraction.\n", "\n", "`StateDataset.from_dataset()` filters the source dataset by state, merges matching segments, slices them into fixed windows, and annotates the result as no-change providers associated with one state. This is a crucial bridge into later ARL-style evaluation workflows.\n", "The `keep_remainder=True` parameter includes a final shorter window when the segment length is not evenly divisible by `slice_length`; without it, any remainder is discarded.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "1948ec3c", "metadata": {}, "outputs": [], "source": [ "state_dataset = StateDataset.from_dataset(\n", " dataset,\n", " slice_length=4,\n", " state=next(iter(dataset.states)),\n", " keep_remainder=True,\n", ")\n", "\n", "print(\"StateDataset size:\", len(state_dataset))\n", "print(\"StateDataset state:\", state_dataset.state)\n", "print(\"First state-window annotation:\", state_dataset[0].annotation)" ] }, { "cell_type": "markdown", "id": "c6257762", "metadata": {}, "source": [ "**Summary of Section 7:** Datasets are the benchmark-facing collection abstraction of the data layer. They expose shared state and transition information, support focused filtering, allow reproducible splitting, and can derive specialized no-change collections through `StateDataset`.\n" ] }, { "cell_type": "markdown", "id": "2271c272", "metadata": {}, "source": [ "## Section 8. Loading Data From Files\n", "\n", "The last practical problem in the data layer is moving from a filesystem layout into the same in-memory dataset abstractions used everywhere else. That loader path is important because real projects rarely start from hard-coded arrays.\n", "\n", "The current public loader story centers on segmented CSV folders. The example uses a small pre-created dataset in `assets/userguide/examples/csv_dataset/` so you can inspect the file layout, the required columns, and the resulting labeled provider structure.\n" ] }, { "cell_type": "markdown", "id": "ed5c56b0", "metadata": {}, "source": [ "### 8.1 FolderCsvColumns and load_folder_csv_dataset()\n", "\n", "The loader needs two kinds of information: where to read the files from, and how to interpret their columns. `FolderCsvColumns` provides the column contract, and `load_folder_csv_dataset()` applies that contract to one root directory.\n", "\n", "The API assumes one or more subfolders under the root, each with a `metadata.yaml` file and one or more CSV files. Each CSV must include feature columns, state columns, and a segment-number column that identifies contiguous labeled regions.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "e3c1a51b", "metadata": {}, "outputs": [], "source": [ "ROOT = _find_repo_root()\n", "loaded_dataset = load_folder_csv_dataset(\n", " ROOT / \"assets/userguide/examples/csv_dataset\",\n", " FolderCsvColumns(\n", " feature_columns=[\"value\", \"aux\"],\n", " state_columns=[\"state_type\", \"state_regime\"],\n", " segment_num_column=\"segment_num\",\n", " ),\n", ")\n", "\n", "print(\"Loaded dataset size:\", len(loaded_dataset))\n", "if len(loaded_dataset) == 0:\n", " print(\"CSV example dataset contains no loadable providers in this docs environment.\")\n", "else:\n", " loaded_provider = loaded_dataset[0]\n", " print(\"Loaded provider annotation:\", loaded_provider.annotation)\n", " print(\"Loaded provider change points:\", list(loaded_provider.change_points))\n", " display(loaded_provider.dataset())\n" ] }, { "cell_type": "markdown", "id": "8bc4bd98", "metadata": {}, "source": [ "### 8.2 Why the loaded result matches the rest of the notebook\n", "\n", "A loader is only useful if it returns objects that can immediately participate in the same workflows as generated or manually constructed data. The loaded dataset above is not a special-case object; it is an ordinary `Dataset` of labeled providers.\n", "\n", "That means the same operations still apply. You can inspect states and transitions, filter by segments or bisegments, apply transformers, and pass the result into online detectors or benchmarks in later notebooks.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "2a90f386", "metadata": {}, "outputs": [], "source": [ "print(\"Loaded dataset states:\", [dict(state) for state in loaded_dataset.states])\n", "print(\"Loaded dataset transitions:\", [repr(transition) for transition in loaded_dataset.transitions])\n", "loaded_bisegments = loaded_dataset.filter_by_bisegments()\n", "print(\"Loaded dataset bisegment count:\", len(loaded_bisegments))" ] }, { "cell_type": "markdown", "id": "72d0d41f", "metadata": {}, "source": [ "**Summary of Section 8:** The folder CSV loader turns on-disk segmented data into the same in-memory labeled-provider and dataset abstractions used throughout the project. Once loaded, file-backed data behaves just like generated or manually constructed data.\n" ] }, { "cell_type": "markdown", "id": "3109232c", "metadata": {}, "source": [ "## Section 9. End-to-End Mini Workflow\n", "\n", "To close the notebook, we will solve the full local problem once: start from a generated labeled provider, build a dataset, extract a transition-focused subset, and prepare a single-feature view. The goal is not to introduce new APIs, but to show how the main pieces already fit together.\n" ] }, { "cell_type": "markdown", "id": "53a28f85", "metadata": {}, "source": [ "### 9.1 From generated series to benchmark-ready views\n", "\n", "The notebook has introduced many small building blocks, and the final challenge is to connect them without losing readability. A realistic data-preparation workflow does not use every class at once, but it does usually cross several abstraction boundaries in sequence.\n", "\n", "This short example goes from generation to labeled provider, from labeled provider to dataset, from dataset to bisegment subset, and from multivariate view to single-column view. That is the shape many later experiments will follow.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "bf5e8c75", "metadata": {}, "outputs": [], "source": [ "workflow_data = pd.DataFrame(\n", " {\n", " \"value\": [0.2, -0.1, 0.3, 0.0, 3.0, 2.9, 3.1, 0.1, -0.2, 0.2],\n", " \"aux\": [5.1, 4.9, 5.0, 5.2, 7.0, 7.2, 6.9, 5.0, 5.1, 4.8],\n", " }\n", ")\n", "workflow_provider = PandasLabeledData.from_unlabeled_data(\n", " PandasDataProvider(workflow_data, UnlabeledTimeseriesAnnotation(name=\"workflow_series\")),\n", " [\n", " SegmentInfo(segment_num=0, segment_start=0, segment_end=3, state=baseline_state),\n", " SegmentInfo(segment_num=1, segment_start=4, segment_end=6, state=shifted_state),\n", " SegmentInfo(segment_num=2, segment_start=7, segment_end=9, state=baseline_state),\n", " ],\n", " TimeseriesAnnotation(name=\"workflow_series\"),\n", ")\n", "workflow_dataset = Dataset([workflow_provider])\n", "workflow_bisegments = workflow_dataset.filter_by_bisegments()\n", "workflow_selected = ColumnsSelectorTransformer(columns=[\"value\"]).transform(workflow_provider)\n", "print(\"Workflow provider change points:\", list(workflow_provider.change_points))\n", "print(\"Workflow dataset size:\", len(workflow_dataset))\n", "print(\"Workflow bisegment size:\", len(workflow_bisegments))\n", "print(\"Workflow selected columns:\", list(workflow_selected.feature_columns))\n", "display(workflow_selected.dataset().head())" ] }, { "cell_type": "markdown", "id": "a70b15de", "metadata": {}, "source": [ "### 9.2 Final summary\n", "\n", "The PySATL data layer is not only a set of classes; it is a contract that keeps all later notebooks coherent. Raw providers solve storage and iteration, labeled providers solve ground-truth representation, datasets solve collection-level workflows, and transformers and loaders solve the practical edges where real data enters or changes shape.\n", "\n", "That is why this notebook is so detailed. Everything later in the tutorial series assumes the reader can recognize these abstractions on sight and move between them confidently.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.13" }, "mystnb": { "execution_mode": "force" } }, "nbformat": 4, "nbformat_minor": 5 }