> For the complete documentation index, see [llms.txt](https://docs.soda.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soda.io/guides-and-tutorials/data-steward/automate-anomaly-detection.md).

# Automate anomaly detection

Connect a data source to Soda Cloud, turn on Metric Monitoring with historical metric collection, and review the anomalies Soda detects.

This guide shows how to start monitoring the data quality of a data source in Soda Cloud without writing any checks. You connect the data source, onboard its datasets, and turn on Metric Monitoring. Soda then tracks metrics such as row counts, schema changes, and freshness for every dataset and flags the values that fall outside the range it expects.

With Historical Metric Collection, Soda calculates the metrics for past days and runs anomaly detection on them as soon as monitoring starts. The anomaly detection algorithm learns from that history right away, and you see past anomalies on the first day.

The examples use a PostgreSQL table, `daily_orders`, with one row per order and an `ordered_at` timestamp column. On a normal day the table receives about 200 orders.

## Prerequisites

* A Soda Cloud account with the **Manage data sources** global permission. See [Global and dataset roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles.md).
* A database user that Soda can use to read the data source. Use read-only access.
* A runner to connect to the data source: the Soda-hosted runner, or a self-hosted runner in your own environment. See [Deployment options](https://docs.soda.io/soda-documentation/connect/deployment-options.md).

## Connect a data source

{% stepper %}
{% step %}

### Create the data source

In Soda Cloud, go to **Data Sources** and click **New Data Source**. Select your data source provider, for example PostgreSQL.
{% endstep %}

{% step %}

### Name the data source and pick a runner

Under **Data Source Identifier**, enter a unique label. Soda generates the data source's immutable name from it.

Under **Default Runner**, select the runner that connects to the data source. The Soda-hosted runner needs no setup and is the quickest way to get started.
{% endstep %}

{% step %}

### Enter the connection details

Under **Connection Details**, fill in the host, port, user, password, and database. For the password and other sensitive values, use Soda Secrets instead of entering them directly.
{% endstep %}

{% step %}

### Test and connect

Click **Test connection**. When the test passes, click **Connect**.
{% endstep %}
{% endstepper %}

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-35cc1725954edfcaf8c9a2384f5769e66f2cfae9%2Fautomate-anomaly-detection-data-source.png?alt=media" alt="The Data Sources page in Soda Cloud with a connected PostgreSQL data source, its five onboarded datasets, and the Onboard Datasets button"><figcaption><p>A connected data source on the <strong>Data Sources</strong> page</p></figcaption></figure>

> For every field of the connection form, see [Onboard datasets on Soda Cloud](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud.md).

## Onboard datasets

After you connect, Soda runs a discovery scan that lists the datasets in the data source with their columns and column types. In **Onboard Datasets**, choose how to select datasets under **Dataset Selection**:

* **Manual** lets you browse the data source and select datasets one by one.
* **Rule-based** onboards every dataset that matches rules you set, such as a schema or a name prefix. Soda runs discovery once per day and onboards new datasets that match the rules.

Select the datasets and click **Next**. The next steps of the flow set up monitoring for all the selected datasets at once: turn on Metric Monitoring and Historical Metric Collection, set the monitoring schedule, suggest a time partition column, choose a dataset owner, and optionally enable profiling. The sections below show each of these settings for one dataset.

To onboard more datasets later, click **Onboard Datasets** next to the data source on the **Data Sources** page.

## Metric Monitoring settings

To review or change the monitoring of a dataset, open the dataset, go to the **Metric Monitors** tab, and click **Configure Metric Monitoring**. The **Update Metric Monitoring Configuration** panel has these settings:

* **Enable Metric Monitoring** turns monitoring on for the dataset.
* **Monitoring Schedule** sets when Soda scans the dataset to capture and evaluate the metrics, from hourly to weekly. A daily schedule suits a table like `daily_orders` that is loaded once per day.
* **Historical Metric Collection** calculates the metrics for past days, from a start date you choose, and runs anomaly detection on them.
* Each dataset monitor, such as **Total row count**, **Schema changes**, or **Partition row count**, has its own toggle.

Click **Save** to apply your changes.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-2d66b6d494339691e202e8c36b1c095db34d9557%2Fautomate-anomaly-detection-metric-monitoring-settings.png?alt=media" alt="The Update Metric Monitoring Configuration panel of daily_orders, with Enable Metric Monitoring on, a daily Monitoring Schedule, Historical Metric Collection on for 59 days, and all six dataset monitors on"><figcaption><p>Metric Monitoring settings of <code>daily_orders</code></p></figcaption></figure>

### Time partition column

Monitors based on data, such as **Partition row count** and **Most recent timestamp**, group the rows into partitions by a time partition column. Use a timestamp column that tells when each row arrived, for example `ordered_at`. During onboarding you can suggest this column; if you don't, Soda detects it from the column metadata, names, and content.

To check or change the column, go to the dataset's **Settings** tab. Under **Profiling & Metric Monitoring** > **Time Partition**, choose **Column** and the column name, then click **Save profiling**.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-e5ba553b924e603eb987089dddf11d5ec135c3cf%2Fautomate-anomaly-detection-time-partition.png?alt=media" alt="The Time Partition setting of daily_orders, partitioned by the column ordered_at"><figcaption><p>The time partition column of <code>daily_orders</code></p></figcaption></figure>

Profiling is optional. It collects column statistics, such as distinct and missing counts, on the dataset's **Profiling** tab. To turn it on or change it, click **Update Profiling Configuration** on that tab.

> For details on the schedule, the time partition column, and the other options, see [Onboard datasets on Soda Cloud](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud.md) and [Scan time strategy](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/scan-time-and-scan-time-strategy.md).

## Review the anomalies

Open a dataset and go to the **Metric Monitors** tab. Under **Dataset Monitors**, each monitor has a card with the value of the last scan, the expected range, and the trend of the last seven measurements. A red warning icon marks a monitor whose latest value is an anomaly.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-d9318bea8370f7e29f5bcce4b312ed2df4ee0516%2Fautomate-anomaly-detection-metric-monitors-tab.png?alt=media" alt="The Metric Monitors tab of daily_orders: Partition row count shows 40 with an expected range of 199 to 201 rows, and Most recent timestamp shows 19h 20m ago with an expected range of 39 to 41 minutes, both flagged"><figcaption><p>The <strong>Metric Monitors</strong> tab of <code>daily_orders</code></p></figcaption></figure>

In the example, the **Partition row count** monitor is flagged: the latest daily partition of `daily_orders` has 40 rows, while Soda expects 199 to 201 from the past days. **Most recent timestamp** is flagged as well: at scan time, the newest `ordered_at` value was more than 19 hours old, against an expected 39 to 41 minutes. Together, the two monitors show that the orders of the latest day stopped arriving early.

Click a card to open the monitor page. The plot shows the measurement of each scan, the expected range, and the anomalies. The table below the plot lists every result with its expected range.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-a26fd5a8d7aec571684b58a2f283a901cd496ece%2Fautomate-anomaly-detection-partition-row-count-monitor.png?alt=media" alt="The Partition row count monitor page of daily_orders: daily values of 200 for the last seven days, then a value of 40 marked as an anomaly, and the results table"><figcaption><p>The latest partition is flagged as an anomaly</p></figcaption></figure>

From the monitor page, you can act on the results:

* Use the menu of a result in the table to give feedback to the anomaly detection algorithm, for example to mark an anomaly as expected, or to link the result to an incident.
* Click **Set Threshold Strategy**, **Set Exclusion Values**, or **Set Sensitivity** to fine-tune when the monitor flags a value.

> Learn more about the [metric monitor page](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/metric-monitor-page.md) and the [dataset monitors](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors.md).

## Set up alerts

Soda does not send notifications for monitors until you opt in. To get an alert when a monitor flags an anomaly:

1. On the **Metric Monitors** tab, click the bell icon on the monitor's card.
2. In **Create Notification**, under **Notify**, keep **Dataset Owner** as the recipient, or click **+ Add recipient** to add others.
3. Click **Create**.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-a18cd9078fae7e3dd882461f62206bf3c3cb23cb%2Fautomate-anomaly-detection-notification-rule.png?alt=media" alt="The Create Notification dialog for the Partition row count monitor of daily_orders, with Dataset Owner as the recipient"><figcaption><p>A notification for the <strong>Partition row count</strong> monitor</p></figcaption></figure>

This notification covers one monitor. To cover many monitors with one rule, use the Notification Rules page.

> For more on rules and recipients, see [Notifications](https://docs.soda.io/soda-documentation/fix/notifications.md).

## Next steps

* Add column monitors for missing values, averages, and other column metrics. See [Column monitors](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors.md).
* Explore the column statistics that profiling collects. See [Profiling](https://docs.soda.io/soda-documentation/find/data-observability/profiling.md).
* Generate data contracts for the datasets you onboarded.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.soda.io/guides-and-tutorials/data-steward/automate-anomaly-detection.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
