Automate anomaly detection
Connect a data source to Soda Cloud, turn on Metric Monitoring with historical metric collection, and review the anomalies Soda detects.
This guide shows how to start monitoring the data quality of a data source in Soda Cloud without writing any checks. You connect the data source, onboard its datasets, and turn on Metric Monitoring. Soda then tracks metrics such as row counts, schema changes, and freshness for every dataset and flags the values that fall outside the range it expects.
With Historical Metric Collection, Soda calculates the metrics for past days and runs anomaly detection on them as soon as monitoring starts. The anomaly detection algorithm learns from that history right away, and you see past anomalies on the first day.
The examples use a PostgreSQL table, daily_orders, with one row per order and an ordered_at timestamp column. On a normal day the table receives about 200 orders.
Prerequisites
A Soda Cloud account with the Manage data sources global permission. See Global and dataset roles.
A database user that Soda can use to read the data source. Use read-only access.
A runner to connect to the data source: the Soda-hosted runner, or a self-hosted runner in your own environment. See Deployment options.
Connect a data source

For every field of the connection form, see Onboard datasets on Soda Cloud.
Onboard datasets
After you connect, Soda runs a discovery scan that lists the datasets in the data source with their columns and column types. In Onboard Datasets, choose how to select datasets under Dataset Selection:
Manual lets you browse the data source and select datasets one by one.
Rule-based onboards every dataset that matches rules you set, such as a schema or a name prefix. Soda runs discovery once per day and onboards new datasets that match the rules.
Select the datasets and click Next. The next steps of the flow set up monitoring for all the selected datasets at once: turn on Metric Monitoring and Historical Metric Collection, set the monitoring schedule, suggest a time partition column, choose a dataset owner, and optionally enable profiling. The sections below show each of these settings for one dataset.
To onboard more datasets later, click Onboard Datasets next to the data source on the Data Sources page.
Metric Monitoring settings
To review or change the monitoring of a dataset, open the dataset, go to the Metric Monitors tab, and click Configure Metric Monitoring. The Update Metric Monitoring Configuration panel has these settings:
Enable Metric Monitoring turns monitoring on for the dataset.
Monitoring Schedule sets when Soda scans the dataset to capture and evaluate the metrics, from hourly to weekly. A daily schedule suits a table like
daily_ordersthat is loaded once per day.Historical Metric Collection calculates the metrics for past days, from a start date you choose, and runs anomaly detection on them.
Each dataset monitor, such as Total row count, Schema changes, or Partition row count, has its own toggle.
Click Save to apply your changes.

daily_ordersTime partition column
Monitors based on data, such as Partition row count and Most recent timestamp, group the rows into partitions by a time partition column. Use a timestamp column that tells when each row arrived, for example ordered_at. During onboarding you can suggest this column; if you don't, Soda detects it from the column metadata, names, and content.
To check or change the column, go to the dataset's Settings tab. Under Profiling & Metric Monitoring > Time Partition, choose Column and the column name, then click Save profiling.

daily_ordersProfiling is optional. It collects column statistics, such as distinct and missing counts, on the dataset's Profiling tab. To turn it on or change it, click Update Profiling Configuration on that tab.
For details on the schedule, the time partition column, and the other options, see Onboard datasets on Soda Cloud and Scan time strategy.
Review the anomalies
Open a dataset and go to the Metric Monitors tab. Under Dataset Monitors, each monitor has a card with the value of the last scan, the expected range, and the trend of the last seven measurements. A red warning icon marks a monitor whose latest value is an anomaly.

daily_ordersIn the example, the Partition row count monitor is flagged: the latest daily partition of daily_orders has 40 rows, while Soda expects 199 to 201 from the past days. Most recent timestamp is flagged as well: at scan time, the newest ordered_at value was more than 19 hours old, against an expected 39 to 41 minutes. Together, the two monitors show that the orders of the latest day stopped arriving early.
Click a card to open the monitor page. The plot shows the measurement of each scan, the expected range, and the anomalies. The table below the plot lists every result with its expected range.

From the monitor page, you can act on the results:
Use the menu of a result in the table to give feedback to the anomaly detection algorithm, for example to mark an anomaly as expected, or to link the result to an incident.
Click Set Threshold Strategy, Set Exclusion Values, or Set Sensitivity to fine-tune when the monitor flags a value.
Learn more about the metric monitor page and the dataset monitors.
Set up alerts
Soda does not send notifications for monitors until you opt in. To get an alert when a monitor flags an anomaly:
On the Metric Monitors tab, click the bell icon on the monitor's card.
In Create Notification, under Notify, keep Dataset Owner as the recipient, or click + Add recipient to add others.
Click Create.

This notification covers one monitor. To cover many monitors with one rule, use the Notification Rules page.
For more on rules and recipients, see Notifications.
Next steps
Add column monitors for missing values, averages, and other column metrics. See Column monitors.
Explore the column statistics that profiling collects. See Profiling.
Generate data contracts for the datasets you onboarded.
Last updated
Was this helpful?
