Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Automate anomaly detection

Connect a data source to Soda Cloud, turn on Metric Monitoring with historical metric collection, and review the anomalies Soda detects.

This guide shows how to start monitoring the data quality of a data source in Soda Cloud without writing any checks. You connect the data source, onboard its datasets, and turn on Metric Monitoring. Soda then tracks metrics such as row counts, schema changes, and freshness for every dataset and flags the values that fall outside the range it expects.

With Historical Metric Collection, Soda calculates the metrics for past days and runs anomaly detection on them as soon as monitoring starts. The anomaly detection algorithm learns from that history right away, and you see past anomalies on the first day.

The examples use a PostgreSQL table, daily_orders, with one row per order and an ordered_at timestamp column. On a normal day the table receives about 200 orders.

Prerequisites

  • A Soda Cloud account with the Manage data sources global permission. See Global and dataset roles.

  • A database user that Soda can use to read the data source. Use read-only access.

  • A runner to connect to the data source: the Soda-hosted runner, or a self-hosted runner in your own environment. See Deployment options.

Connect a data source

1

Create the data source

In Soda Cloud, go to Data Sources and click New Data Source. Select your data source provider, for example PostgreSQL.

2

Name the data source and pick a runner

Under Data Source Identifier, enter a unique label. Soda generates the data source's immutable name from it.

Under Default Runner, select the runner that connects to the data source. The Soda-hosted runner needs no setup and is the quickest way to get started.

3

Enter the connection details

Under Connection Details, fill in the host, port, user, password, and database. For the password and other sensitive values, use Soda Secrets instead of entering them directly.

4

Test and connect

Click Test connection. When the test passes, click Connect.

The Data Sources page in Soda Cloud with a connected PostgreSQL data source, its five onboarded datasets, and the Onboard Datasets button
A connected data source on the Data Sources page

For every field of the connection form, see Onboard datasets on Soda Cloud.

Onboard datasets

After you connect, Soda runs a discovery scan that lists the datasets in the data source with their columns and column types. In Onboard Datasets, choose how to select datasets under Dataset Selection:

  • Manual lets you browse the data source and select datasets one by one.

  • Rule-based onboards every dataset that matches rules you set, such as a schema or a name prefix. Soda runs discovery once per day and onboards new datasets that match the rules.

Select the datasets and click Next. The next steps of the flow set up monitoring for all the selected datasets at once: turn on Metric Monitoring and Historical Metric Collection, set the monitoring schedule, suggest a time partition column, choose a dataset owner, and optionally enable profiling. The sections below show each of these settings for one dataset.

To onboard more datasets later, click Onboard Datasets next to the data source on the Data Sources page.

Metric Monitoring settings

To review or change the monitoring of a dataset, open the dataset, go to the Metric Monitors tab, and click Configure Metric Monitoring. The Update Metric Monitoring Configuration panel has these settings:

  • Enable Metric Monitoring turns monitoring on for the dataset.

  • Monitoring Schedule sets when Soda scans the dataset to capture and evaluate the metrics, from hourly to weekly. A daily schedule suits a table like daily_orders that is loaded once per day.

  • Historical Metric Collection calculates the metrics for past days, from a start date you choose, and runs anomaly detection on them.

  • Each dataset monitor, such as Total row count, Schema changes, or Partition row count, has its own toggle.

Click Save to apply your changes.

The Update Metric Monitoring Configuration panel of daily_orders, with Enable Metric Monitoring on, a daily Monitoring Schedule, Historical Metric Collection on for 59 days, and all six dataset monitors on
Metric Monitoring settings of daily_orders

Time partition column

Monitors based on data, such as Partition row count and Most recent timestamp, group the rows into partitions by a time partition column. Use a timestamp column that tells when each row arrived, for example ordered_at. During onboarding you can suggest this column; if you don't, Soda detects it from the column metadata, names, and content.

To check or change the column, go to the dataset's Settings tab. Under Profiling & Metric Monitoring > Time Partition, choose Column and the column name, then click Save profiling.

The Time Partition setting of daily_orders, partitioned by the column ordered_at
The time partition column of daily_orders

Profiling is optional. It collects column statistics, such as distinct and missing counts, on the dataset's Profiling tab. To turn it on or change it, click Update Profiling Configuration on that tab.

For details on the schedule, the time partition column, and the other options, see Onboard datasets on Soda Cloud and Scan time strategy.

Review the anomalies

Open a dataset and go to the Metric Monitors tab. Under Dataset Monitors, each monitor has a card with the value of the last scan, the expected range, and the trend of the last seven measurements. A red warning icon marks a monitor whose latest value is an anomaly.

The Metric Monitors tab of daily_orders: Partition row count shows 40 with an expected range of 199 to 201 rows, and Most recent timestamp shows 19h 20m ago with an expected range of 39 to 41 minutes, both flagged
The Metric Monitors tab of daily_orders

In the example, the Partition row count monitor is flagged: the latest daily partition of daily_orders has 40 rows, while Soda expects 199 to 201 from the past days. Most recent timestamp is flagged as well: at scan time, the newest ordered_at value was more than 19 hours old, against an expected 39 to 41 minutes. Together, the two monitors show that the orders of the latest day stopped arriving early.

Click a card to open the monitor page. The plot shows the measurement of each scan, the expected range, and the anomalies. The table below the plot lists every result with its expected range.

The Partition row count monitor page of daily_orders: daily values of 200 for the last seven days, then a value of 40 marked as an anomaly, and the results table
The latest partition is flagged as an anomaly

From the monitor page, you can act on the results:

  • Use the menu of a result in the table to give feedback to the anomaly detection algorithm, for example to mark an anomaly as expected, or to link the result to an incident.

  • Click Set Threshold Strategy, Set Exclusion Values, or Set Sensitivity to fine-tune when the monitor flags a value.

Learn more about the metric monitor page and the dataset monitors.

Set up alerts

Soda does not send notifications for monitors until you opt in. To get an alert when a monitor flags an anomaly:

  1. On the Metric Monitors tab, click the bell icon on the monitor's card.

  2. In Create Notification, under Notify, keep Dataset Owner as the recipient, or click + Add recipient to add others.

  3. Click Create.

The Create Notification dialog for the Partition row count monitor of daily_orders, with Dataset Owner as the recipient
A notification for the Partition row count monitor

This notification covers one monitor. To cover many monitors with one rule, use the Notification Rules page.

For more on rules and recipients, see Notifications.

Next steps

  • Add column monitors for missing values, averages, and other column metrics. See Column monitors.

  • Explore the column statistics that profiling collects. See Profiling.

  • Generate data contracts for the datasets you onboarded.

Last updated

Was this helpful?