> For the complete documentation index, see [llms.txt](https://docs.soda.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soda.io/data-observability/metric-monitoring-dashboard/dataset-monitors.md).
# Dataset monitors
## What is a dataset monitor?
A **dataset monitor** in Soda **tracks a specific high-level metric** for an entire table (or partition) over time. It helps detect **unusual patterns** or **unexpected changes** in overall data health, such as sudden spikes or drops in row count, delays in fresh data, or schema drift.
You can find dataset monitors by opening the **Metric Monitors** tab on any dataset and looking at the top section labeled “**Dataset Monitors**.” This section lists all active dataset monitors—both metadata-based and partition-based—in a clear overview of monitor cards. This overview provides, at a glance, critical information about the **status** of each monitor, the **value** of the last scan, and any detected **anomalies**, allowing you to have a one-look summary of the health of your data systems.
Unlike [column monitors](/data-observability/metric-monitoring-dashboard/column-monitors.md), which are configured at the dataset level but target individual columns, dataset monitors apply to the **entire table** (or its latest partition) and capture **broad indicators of data quality**. When the necessary data and metadata are available, dataset-level monitors work out of the box with no further configuration needed.
### Types of dataset monitors
Soda supports two categories of dataset‐level monitors: those that rely purely on **system metadata**, and those that compute values by **querying a designated time‐partition column**. Below is an in‐depth description of each built‐in monitor.
> For a more detailed discussion of monitors based on **querying the metadata** vs monitors based on **querying the data**, see the [Metadata vs Data-Based](#metadata-vs-data-based-monitors) section in this page.
| Dataset monitor type | Monitor | Description |
| ---------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------ |
| **Based on metadata** | Total row count | The total number of rows in the dataset at scan time. |
| | Total row count change | Change in total row count compared to the previous scan. |
| | Last modification time | Most recent time the data was changed relative to the last scan. |
| | Schema changes | Changes in the schema compared to the previous scan—any change is automatically flagged as an anomaly. |
| **Based on time partition column** | Partition row count | The number of rows in the last partition at scan time. |
| | Most recent timestamp | Time difference between scan time and the maximum timestamp in the partition column (at scan time). |
{% hint style="warning" %}
Monitors based on time partition columns look at data in the most recent partition **based on a timestamp**. If data is altered in an old partition, it will not be evaluated.
For example, data inserted today with timestamp of 2 days ago will not be evaluated if the partition interval is 1 day.
{% endhint %}
{% hint style="info" %}
For **Schema changes**, the expected result is always to have no schema changes, regardless of whether there have been frequent schema changes in the past or not.
{% endhint %}
## Understanding the monitor card
The dashboard provides a **health table** summarizing an overview of the monitors. Each monitor card is clickable and links to the **Monitor History** page of the metric.
Each monitor card will have the following information:
* **Monitor name**: the given name of the specific monitor.
* **Monitor explanation**: a brief description of the metric used.
* **Status**: ✅ healthy / ⚠️ violated
* **Today's value at scan time**: last recorded value.
* **Expected range**: calculated by the anomaly detection algorithm, based on historical data.
* **Trend line with last 7 observations**: a sparkline that shows an overview of the monitor plot.
* **Bell icon**: to enable/disable opt-in alerts.
## Metadata vs Data based monitors
Dataset-level monitors fall into two categories depending on their source of truth:
#### Metadata-based dataset monitors
[Metadata-based monitors](/data-observability/metric-monitoring-dashboard/dataset-monitors/metadata-based.md) rely solely on **system metadata exposed by your data warehouse**; fields like “row count,” “last modified time,” or “schema version” that the catalog provides without scanning table rows. Because they don’t touch actual data, **metadata monitors are extremely efficient and run quickly**. They alert you if your table grows, shrinks, stops updating, or changes structure.
#### Data-based dataset monitors
[Data-based monitors](/data-observability/metric-monitoring-dashboard/dataset-monitors/query-based.md) look directly at the **contents of a designated time-partition column** (e.g., a `date` or `timestamp` field) and compute a value from the rows in that partition. Examples include “Partition Row Count” (how many rows landed in today’s partition) or “Most Recent Timestamp” (the newest timestamp in that partition). Data-based monitors require a full scan of each partition they monitor, but **they capture freshness and volume signals that metadata alone cannot provide**. If your dataset has no time-partition column defined (or your warehouse can’t surface the needed metadata), Soda will disable the appropriate monitors so you only see the metrics that can be collected.
## Configure Dataset Monitors
Use the **Configure Metric Monitoring** panel to pick which built-in metadata and partition-based metrics you want Soda to track at the dataset level.
1. **Open the panel** → From any dataset’s Metric Monitors dashboard, click **Configure Metric Monitoring** on the top right.
2. **Enable or disable** → Toggle metrics on/off directly from here. If the data source doesn't support a given metric, it will be automatically off.
3. **Modify the monitor**
4. **Auto-apply** → Changes take effect immediately for the next scan. Simply close the panel when you’re done.
### Time partition column
Many data‐based monitors—such as **Partition Row Count** and **Most Recent Timestamp**—rely on a designated “time partition” column to know which slice of data to scan. The time partition column should be a date or timestamp field that naturally groups rows into discrete, regularly updated partitions (for example, a daily `order_date` or `event_time`). When Soda cannot detect a time partition column, metrics based on that data will not be available.
When Soda cannot detect a time partition column, a warning banner will appear on the Metric Monitors tab.
#### What's a good time partition column?
A good time partition column meets all of the following criteria:
1. **Date or timestamp type**: Each row contains a valid date (or timestamp) value.
2. **Regular arrival cadence**: New rows for each date/timestamp appear on a predictable schedule (e.g., daily, hourly).
3. **Reflects ingestion/arrival time**: The column’s value must correspond to when the record actually landed in this dataset, **not when it was originally created upstream**. The partition column should always show arrival date to the dataset so freshness checks remain accurate.
4. **Logical partition boundary**: It matches how you want to slice your data (e.g., `order_date` for daily sales, `event_time` for hourly logs).
When these conditions hold, partition-based monitors will reliably focus on the correct slice of data—namely, the rows that truly arrived during each time window—so any delays or backfills become immediately visible.
#### Suggest a time partition column during onboarding
When you onboard a new dataset from your data source, Soda attempts to **automatically detect** the most likely time partition column. You can:
* **Finish** onboarding without editing the Time Partition Column field, allowing Soda to detect it, or
* **Suggest** a Time Partition Column of your choice, forcing Soda to use that one for monitoring.
#### Find a time partition column
If you ever need to confirm or search for the right partition column:
1. Navigate to the **Datasets** page, select your dataset, and click the **Columns** tab.
2. Search columns with "timestamp" on them. Any column with a date or timestamp data type is a candidate.
#### Manually override time partition column
After onboarding, you can override the time partition column at any time. Changing it will reset Soda’s anomaly detection model for partition‐based metrics, so you’ll be retraining on historical data under the new partition definition. To override:
1. Acess the **Dataset Settings**
* Navigate to the **Datasets** tab
* From this list or from the dataset page itself, click on the (⋮) menu > **Edit Dataset**
2. Find Time Partition Columns
* Click on the **Profiling & Metric Monitoring** tab
Here you will see the current column being used for Time Partition.
* Reveal the Time Partition Column drop-down menu
This will show all **date** and **timestamp** columns that can be used as a Time Partition Column.
3. Select your new Time Partition Column
{% hint style="danger" %}
Changing this column resets the model and historical baselines.
{% endhint %}
4. Click **Save**. Soda will:
* Reset the partition‐based monitors (Partition Row Count, Most Recent Timestamp) to “training mode” and retrain baselines on the new partition.
* Preserve any metadata‐based monitors (Total Row Count, Schema Changes) unchanged.
By following these steps, you ensure that Soda’s data‐based monitors always reference the correct daily (or hourly) slice of your dataset, so partition‐level metrics and freshness checks produce accurate results.
## Missing metric values
Even when a metric is enabled and historical baselines exist, you may occasionally see gaps due to delayed or skipped scans. A “missing” metric indicates that Soda attempted to run the scan but did not receive a valid result for that metric, either because the scan runner was down, the query timed out, or metadata couldn’t be retrieved in time. Missing values do not count as anomalies; they simply mark a gap in the time series.
In Soda Cloud, you can identify these gaps as follows:
* On the **Metric Monitors** dashboard, any missing value is shown either as a grey point or an empty checkbox in the metric sparkline:
* In the detailed anomaly plot, missing points render as open circles (◯) along the timeline, and the trend line becomes dashed.
* In **Schema changes**, no plot is available since the expected value is always 0. Hovering over an empty checkbox will display “No measurement” in the tooltip, making it easy to distinguish a gap from a healthy measurement or a flagged anomaly.
These visual cues allow to immediately recognize when a scan didn’t complete successfully, enabling further investigation and restoration of full observability before critical issues go unnoticed.
***
{% if visitor.claims.plan === 'datasetStandard' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}
{% if visitor.claims.plan === 'enterprise' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}
{% if visitor.claims.plan === 'enterpriseUserBased' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}
{% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %}
{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**.
{% endhint %}
{% endif %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.soda.io/data-observability/metric-monitoring-dashboard/dataset-monitors.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.