> This is part 1 of 2 of the full documentation (pages 1–100 of 173).
> The content is paginated: fetch every part to see all of it.
> Next part: https://docs.soda.io/llms-full.txt/1
> Page index: https://docs.soda.io/llms.txt

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
This site has an agentic ask interface you may use to query the documentation dynamically by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.soda.io/index.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

---

# Home

<h2 align="center"><strong>Soda Docs</strong></h2>

<p align="center">Find what you need, explore the APIs, or start with the quickstart.</p>

<p align="center"><button type="button" class="button primary" data-action="search" data-icon="magnifying-glass">Search the docs…</button></p>

<p align="center"><a href="https://docs.soda.io/soda-documentation/get-started/quickstart" class="button primary medium">Quickstart</a><a href="https://docs.soda.io/soda-documentation/soda-ai/overview" class="button primary medium">Soda AI</a><a href="https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference" class="button primary medium">Contract Language reference</a><a href="https://docs.soda.io/apis/" class="button primary medium">Soda APIs</a></p>

***

<h3 align="center">Work with Soda</h3>

<p align="center">Find data quality issues, understand why they happened, and fix them at the source.</p>

<table data-column-title-hidden data-view="cards"><thead><tr><th>Stage</th><th>Capability</th><th>Description</th><th>Resources</th><th data-hidden data-card-target data-type="content-ref">Explore</th></tr></thead><tbody><tr><td><mark style="color:$primary;"><strong>1 → FIND</strong></mark></td><td><strong>Detect issues</strong></td><td>Catch the problems you expect and the ones you don't.</td><td><a href="https://docs.soda.io/soda-documentation/find/data-testing/cloud-managed-data-contracts" class="button secondary small">Data contracts</a> <a href="https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard" class="button secondary small">Metric monitoring</a></td><td><a href="https://docs.soda.io/soda-documentation/find/data-observability">Data observability</a></td></tr><tr><td><mark style="color:$primary;"><strong>2 → UNDERSTAND</strong></mark></td><td><strong>Investigate</strong></td><td>Dig into failures down to the rows and find the root cause.</td><td><a href="https://docs.soda.io/soda-documentation/understand/analyze-monitor-and-check-results" class="button secondary small">Analyze results</a> <a href="https://docs.soda.io/soda-documentation/understand/manage-issues/browse-datasets/dataset-page" class="button secondary small">Dataset page</a><a href="https://docs.soda.io/soda-documentation/understand/diagnostics-warehouse" class="button secondary small">Diagnostics warehouse</a></td><td><a href="https://docs.soda.io/soda-documentation/understand/analyze-monitor-and-check-results">Analyze monitor and check results</a></td></tr><tr><td><mark style="color:$primary;"><strong>3 → FIX</strong></mark></td><td><strong>Respond</strong></td><td>Alert the right people and track issues to resolution.</td><td><a href="https://docs.soda.io/soda-documentation/fix/notifications" class="button secondary small">Notifications</a> <a href="https://docs.soda.io/soda-documentation/fix/incidents" class="button secondary small">Incidents</a> <a href="https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities" class="button secondary small">Ownership</a></td><td><a href="https://docs.soda.io/soda-documentation/fix/notifications">Notifications</a></td></tr></tbody></table>

***

{% columns %}
{% column width="50%" %} <mark style="color:$primary;">**DATA RECONCILIATION**</mark>

### Prove your data arrived intact

Compare a target dataset against one or more sources, from row counts and aggregates down to individual records. Use it to sign off a migration, validate a pipeline run, or keep systems in sync.

<a href="https://docs.soda.io/soda-documentation/find/data-testing/data-reconciliation" class="button primary">Data reconciliation</a>

<a href="https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks" class="button secondary">Reconciliation checks syntax</a>
{% endcolumn %}

{% column width="50%" %}
{% code overflow="wrap" %}

```yaml
dataset: snowflake/analytics/public/orders

reconciliation:
  sources:
    - name: legacy
      dataset: mysql/shop/public/orders
  checks:
    - row_count_diff:
        source: legacy
        threshold:
          must_be_less_than: 1
```

{% endcode %}
{% endcolumn %}
{% endcolumns %}

***

<h3 align="center">Build with Soda</h3>

<table data-column-title-hidden data-view="cards"><thead><tr><th>Title</th><th>Description</th><th>Resources</th><th data-hidden data-card-target data-type="content-ref">Learn more</th></tr></thead><tbody><tr><td><strong>Soda AI</strong></td><td>Let AI agents write contracts and find root causes.</td><td><a href="https://docs.soda.io/soda-documentation/soda-ai/soda-mcp" class="button secondary small">Soda MCP</a> <a href="https://docs.soda.io/soda-documentation/soda-ai/bring-your-own-key-byok" class="button secondary small">BYOK</a></td><td><a href="https://docs.soda.io/soda-documentation/soda-ai/overview">Overview</a></td></tr><tr><td><strong>Integrate</strong></td><td>Connect Soda to your catalog, chat, and ticketing tools.</td><td><a href="https://docs.soda.io/soda-documentation/integrate/integrations" class="button secondary small">Integrations</a> <a href="https://docs.soda.io/soda-documentation/integrate/slack" class="button secondary small">Slack</a> <a href="https://docs.soda.io/soda-documentation/integrate/collibra" class="button secondary small">Collibra</a></td><td><a href="https://docs.soda.io/soda-documentation/integrate/integrations">Integrations overview</a></td></tr><tr><td><strong>Developer tools</strong></td><td>Work with Soda from code, the CLI, and the API.</td><td><a href="https://docs.soda.io/soda-documentation/developer-tools/soda-cli-reference" class="button secondary small">Soda CLI reference</a> <a href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core" class="button secondary small">Data source reference</a> <a href="https://docs.soda.io/soda-documentation/connect/deployment-options/soda-python-libraries" class="button secondary small">Python libraries</a></td><td><a href="https://docs.soda.io/soda-documentation/developer-tools/reference">Reference</a></td></tr></tbody></table>

***

<h3 align="center">Set up and manage Soda</h3>

<p align="center">Deploy Soda, connect your data, and decide who owns what.</p>

<table data-column-title-hidden data-view="cards"><thead><tr><th>Title</th><th>Description</th><th>Resources</th><th data-hidden data-card-target data-type="content-ref">Learn more</th></tr></thead><tbody><tr><td><strong>Deploy</strong></td><td>Choose where your scans run.</td><td><a href="https://docs.soda.io/soda-documentation/connect/deployment-options#soda-hosted" class="button secondary small">Soda-hosted Runner</a> <a href="https://docs.soda.io/soda-documentation/connect/deployment-options#self-hosted-runner" class="button secondary small">Self-hosted Runner</a> <a href="https://docs.soda.io/soda-documentation/connect/deployment-options#soda-python-libraries" class="button secondary small">Python libraries</a></td><td><a href="https://docs.soda.io/soda-documentation/connect/deployment-options">Deployment options</a></td></tr><tr><td><strong>Connect</strong></td><td>Bring your data sources and datasets into Soda.</td><td><a href="https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud" class="button secondary small">Via Soda Cloud</a> <a href="https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/via-soda-core" class="button secondary small">Via Soda Core</a> <a href="https://docs.soda.io/soda-documentation/get-started/quickstart/supported-data-sources" class="button secondary small">Supported data sources</a></td><td><a href="https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets">Onboard data sources &amp; datasets</a></td></tr><tr><td><strong>Govern</strong></td><td>Control who owns, sees, and changes what.</td><td><a href="https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings" class="button secondary small">Admin settings</a> <a href="https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles" class="button secondary small">Roles</a> <a href="https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities" class="button secondary small">Attributes &#x26; responsibilities</a></td><td><a href="https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings">Organization and Admin settings</a></td></tr></tbody></table>


# What is Soda?

**Soda is a data quality platform** that helps organizations make sure their data can be trusted. Soda gives teams a **shared workflow** to **monitor data quality**, **catch problems** early, **understand issues** at source, and **take action** quickly.

You can use Soda to:

* **Monitor production data** with automated, ML-powered observability that spots unexpected changes without needing to define every rule up front.
* **Define data contracts,** making expectations explicit and enabling producers and consumers to collaborate on reliable data at the source.
* **Test data earlier in the pipeline**, as part of CI/CD workflows or during development, to prevent bad data from reaching production.

Soda helps teams to *start right* and automatically detect anomalies in metrics that have already happened. And *shift left* to prevent issues from happening again with collaborative data contracts.

> <a href="https://www.soda.io/schedule-a-demo" class="button primary">Contact us</a> to learn more about Soda's capabilities.

### What is data quality?

**Data quality** refers to how well a dataset meets the expectations of completeness, accuracy, timeliness, uniqueness, and consistency. Good data supports business goals, drives confident decision-making, and is the base for great data products.

Poor data quality causes failed pipelines, incorrect reports, and broken AI models. Managing data quality means proactively validating assumptions and reactively monitoring for drift or degradation.

Soda helps you answer questions like:

* Is the data fresh and complete?
* Are there unexpected values or duplicates?
* Did values shift outside of expected ranges?
* Are schema or contract changes causing breakage?
* Are data quality metrics changing over time?

### Key Concepts

#### Data Observability

Data observability is a reactive approach to monitoring data in production and catching unexpected issues as they emerge. It helps answer the question: What is happening with my data right now, and how is that changing over time?

Use data observability to:

* Detect anomalies in data quality metrics such as freshness, row counts, null values or custom ones
* Monitor metric trends and seasonality
* Identify late-arriving or missing records
* Get alerted when values deviate from historical norms

#### Data Testing

Data testing is a proactive approach that validates known expectations about your data during development, deployment, or transformation. It helps you catch issues before they reach production, break reports, or impact downstream systems.

Use data testing to:

* Align on what “good data” looks like through data contracts
* Verify that your data meets those expectations, including schema, values, and transformations
* **Test data at every step of the pipeline** to prevent bad data from reaching downstream systems
* Integrate with CI/CD workflows for continuous quality checks during development

#### Data Contracts

**Data contracts define what a dataset should look like, including its schema, data types, value ranges, and other constraints.**\
They establish a shared agreement between data producers and consumers about what’s expected and what must be upheld.

Both **testing** and **observability** play a role in upholding data contracts:

* **Testing** validates that data meets the contract during development, pipeline execution, and on schedule.
* **Observability** monitors contract adherence in production and detects unexpected issues.

### Data Observability vs Data Testing

While data testing and observability are different in when and how they operate, they work best together as a **unified strategy**.

<table><thead><tr><th width="211.21875">Approach</th><th>Timing</th><th>Use case</th></tr></thead><tbody><tr><td><strong>Data Testing</strong></td><td>Proactive and preventative: Pre-production, during development or CI/CD</td><td>Prevent breakages before they happen: Validate known rules and enforce contracts</td></tr><tr><td><strong>Data Observability</strong></td><td>Reactive and adaptive: In production, runtime monitoring</td><td>Monitor data behavior and changes over time with automated detection of anomalies, schema changes, and other unexpected issues.</td></tr></tbody></table>

Together, they enable end-to-end data quality management: testing prevents problems, and observability detects those that escape prevention. At the same time, observability can help prioritize which issues to address and shift left to resolve them upstream.

### Data quality at scale across the enterprise

#### Divide and conquer

Managing data quality across hundreds or thousands of datasets requires a **scalable, federated approach**. Soda enables this through:

* **Metadata-driven observability** that adapts checks to each dataset's structure and context.
* **Role-based collaboration** so teams can take ownership of the data they know best.
* **An interface for both engineering and business users**, enabling collaboration through code, UI, or APIs, depending on user preference and role.
* **Integration with existing tools and workflows**, such as data catalogs and incident management systems.
* **Pipeline** **and** **CI/CD integration** to automate data quality checks.

#### Data quality as a team sport

Reliable data depends on collaboration across roles:

* **Data engineers** embed tests and monitor pipelines to catch issues early.
* **Data producers and consumers** align on expectations through data contracts.
* **Data consumers** report issues and collaborate with producers to interpret metrics and resolve problems.
* **Governance teams** define and enforce data quality standards.
* **Platform teams** deploy, manage, and secure the underlying infrastructure.

Soda Cloud acts as the shared workspace where these roles collaborate, triage incidents, and resolve issues.

### Deployment options

Soda offers three deployment models, depending on your infrastructure and data privacy needs.

| Deployment Model       | Description                                                                                                   | Ideal For                                                      | Key Features                                                                                                                                                                                  | Considerations                                                                                                                                                                       |
| ---------------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Soda Core**          | Open-source Python library (with commercial extensions) and CLI for running Data Contracts in your pipelines. | Data engineers integrating Soda into custom workflows.         | Full control over orchestration, in-memory data support, contract verification.                                                                                                               | <p><strong>No observability features</strong>.<br><br>Required for in-memory sources (e.g., Spark, DataFrames).<br><br>Data source connections managed at the environment level.</p> |
| **Soda-hosted Runner** | Managed version of Soda that runs observability features, executes Data Contracts and scheduled them.         | Teams seeking a simple, managed solution for data quality.     | Centralized data source access, no setup required, observability features enabled. Enables users to create, test, execute, and schedule contracts and checks directly from the Soda Cloud UI. | <p><strong>Required for observability features</strong>.<br><br>Cannot scan in-memory sources like Spark or DataFrames.</p>                                                          |
| **Self-hosted Runner** | Same as Soda-hosted Runner, but deployed and managed in your own Kubernetes environment.                      | Teams needing full control over infrastructure and deployment. | Similar to Soda-hosted Runner, but deployed within the customer’s environment; data stays within your network.                                                                                | <p><strong>Required for observability features.</strong></p><p>Cannot scan in-memory sources like Spark or DataFrames.</p><p><br>Kubernetes expertise required.</p>                  |

> Read more about [Deployment options](https://docs.soda.io/soda-documentation/connect/deployment-options)

### Supported data sources and integrations

Soda integrates with the modern data stack:

* **Data warehouses and databases**: Databricks, Snowflake, BigQuery, Redshift, PostgreSQL, MySQL, Spark, Presto, DuckDB, and more.
* **Orchestration platforms**: Airflow, Dagster, Prefect, Azure Data Factory.
* **Metadata tools**: Atlan, Alation, Collibra, data.world, Zeenea.
* **Cloud providers**: AWS, Google Cloud, Azure.
* **BI tools**: Looker, Tableau, Power BI.
* **Messaging and ticketing**: Slack, Microsoft Teams, Jira, PagerDuty, ServiceNow, Opsgenie.

### What’s next?

* To get started with Soda, check out the end-to-end [Quickstart](https://docs.soda.io/soda-documentation/get-started/quickstart) guide.

***

## Community & Support <a href="#community-and-support" id="community-and-support"></a>

Need help or want to contribute?

* **Join our Slack Community**: [Soda Community](https://join.slack.com/t/soda-community/shared_invite/zt-3epazj3kw-00z15nnW4KEt4j_vk8lbdQ)​
* **Browse GitHub Discussions**: [Soda on GitHub](https://github.com/sodadata/soda-core)​

> **Still have questions?** Use the search bar above or reach out through our community channels for additional help.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Quickstart

Learn how to get started with Soda.

This quickstart shows how Soda **detects unexpected data issues** by leveraging AI powered Anomaly Detection and **prevents future problems** by using data contracts.

### Scenario

A data engineer at a retail company needs to maintain the `regional_sales` dataset so their team can manage regional sales data from hundreds of stores across the country. The dataset feeds executive dashboards and downstream ML models for inventory planning. Accuracy and freshness are critical, so you need both:

* **Automated anomaly detection** on key metrics (row counts, freshness, schema drift)
* **Proactive enforcement** of business rules via data contracts

### Sign up

> Contact us at <support@soda.io> to get an account set up.

After signing up, you can follow the steps below to set up a data source and start improving data quality.

### Add a Data Source

Soda AI, paired with Soda Cloud’s no-code UI, lets you **connect** to any data source, **onboard** datasets, and **monitor** results in minutes.

{% stepper %}
{% step %}

#### Establish connection

Navigate to the **Data Sources** > Create a **New data source**

**F**ill out the connection details. Then, click on **Connect**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FbcAI6jiza4zQzDmpIXH6%2Fimage.png?alt=media&amp;token=922cbda8-31a2-4d06-b97a-71041d74152d" alt="" width="375"><figcaption></figcaption></figure>

> Learn more details on how to [Onboard data sources & datasets](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets).
> {% endstep %}

{% step %}

#### Onboard datasets

You can onboard datasets by asking the **Soda AI Agent** to do it for you.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FT73M6GqOzG06SIzafIM5%2Fimage.png?alt=media&amp;token=9c645348-0f94-4d7c-baac-11b2fa319af8" alt="" width="298"><figcaption></figcaption></figure>

> If you prefer to onboard datasets manually, head to the [onboarding page](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud#onboard-datasets).
> {% endstep %}

{% step %}

#### Enable Metric Monitoring

You can enable Metric Monitoring in several ways:

* While [manually onboarding](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud)
* By navigating to the dataset > **Metric Monitoring** tab
* By asking the Soda AI Agent to enable Metric Monitoring

{% hint style="info" %}
If you onboarded the dataset via Soda AI, the **dataset page** won't show any monitors yet.
{% endhint %}

Soda AI can **enable Metric Monitoring** for you:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FVf4gydkxCvu7n7of7fGG%2Fimage.png?alt=media&amp;token=ad2b74d5-ecce-4ec5-8066-9fb3743557f2" alt="" width="447"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

***

### Review Anomaly Detection results

Congratulations, you’ve onboarded your first dataset! Now let’s make sure you always know what’s happening with it.

That’s where Metric Monitoring comes in. It automatically tracks key metrics like volume, freshness, and schema changes, with no manual setup required. You’ll spot anomalies, detect trends, and catch unexpected shifts before they become problems.

{% stepper %}
{% step %}

#### Open the Metric Monitors dashboard

{% stepper %}
{% step %}
Go to **Datasets** → select the dataset to inspect.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FBFltKODftxkbN76RdVAy%2Fimage.png?alt=media&amp;token=5fe952d1-ec4d-4b50-a12f-61f7ac46a135" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Navigate to the **Metric Monitors** tab to learn more about the metrics calculated.

If enabled, key metrics are automatically monitored by default, helping you detect pipeline issues, data delays, and unexpected structural changes as they happen.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FrHXt2xyop3tQUa6RAFOE%2Fimage.png?alt=media&amp;token=2c94f46e-4b95-48a6-a4a5-544f0f07ffe3" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}
{% endstep %}

{% step %}

#### View anomalies in a specific monitor

In this guide, we will focus on the **Most recent timestamp** monitor. The panel shows that it was expected to be in a range of **0 - 5m 31s**, but the recorded value at scan time was **56m 49s**. In order to take a closer look:

{% stepper %}
{% step %}
Click the **Most recent timestamp** (or monitor of your choice) block.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FGrOIIMnoj1DrJXLsdGNl%2Fimage.png?alt=media&amp;token=92d0e9c5-1d83-44e7-af08-c82eccd9b99d" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
In the monitor page you’ll see:

* any red-dot anomalies flagged by the model,
* measured value vs. expected range,
* buttons to **Mark as expected**, **Create new incident**, etc.
  {% endstep %}

{% step %}
Flag an outlier as "expected" or investigate it further.
{% endstep %}
{% endstepper %}
{% endstep %}
{% endstepper %}

**Soda’s anomaly detection engine was built in-house** (no third-party libraries) and optimized for high precision. It continuously adapts to your data patterns, and it incorporates your feedback to reduce false alarms. Designed to minimize false positives and missed detections, it shows a 70% improvement in detecting anomalous data quality metrics compared to Facebook Prophet across hundreds of diverse, internally curated datasets containing known data quality issues.

The Anomaly Detection Algorithm offers complete control and transparency in the modeling process to allow for interpretability and adaptations. It features high accuracy while leveraging historical data, delivering improvements over time.

***

### Attack the issues in Soda Cloud

Our **automated anomaly detection** has just done the heavy lifting for you, identifying unusual patterns and potential data issues without any setup required.

But to **prevent** those issues from happening again, you must define exactly what your data should look like; every column, every rule, every expectation.

That’s where **Data Contracts** come in. They let you proactively set the standards for your data, so problems like this are flagged or even prevented before they impact your business.

{% stepper %}
{% step %}

#### Create a Data Contract

Create a new data contract to define and enforce data quality expectations.

{% stepper %}
{% step %}
In your [**dataset page**](https://docs.soda.io/soda-documentation/understand/manage-issues/browse-datasets/dataset-page), go to the **Checks** tab.
{% endstep %}

{% step %}
Click **Create Contract**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FpI4YqRAjmoaaRrn0qfZn%2Fimage.png?alt=media&amp;token=0c3cc0f6-6ae7-4e5d-b496-3cb4c54fe03e" alt=""><figcaption></figcaption></figure>

When creating a data contract, Contract Autopilot will connect to your dataset and build a data contract.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FknS54808uGh0lV4B23kD%2Fimage.png?alt=media&amp;token=3159fe07-928d-4ed4-a660-01076689577e" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
\[Optional] Ask [**Contract Copilot**](https://docs.soda.io/soda-documentation/soda-ai/contract-copilot) to edit the contract.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fx3ztVRmCuVPQwoM5W7GA%2Fimage.png?alt=media&amp;token=0b056d52-9bae-4eb0-a741-436fe82e9401" alt=""><figcaption></figcaption></figure>

With Soda, you can edit a contract:

* via [Contract Copilot](https://docs.soda.io/soda-documentation/soda-ai/contract-copilot),
* using a **no-code interface**, or
* **directly in code**.

This ensures an optimal experience for all users while also providing a **version-controlled code format** that can be synced with a Git repository.
{% endstep %}
{% endstepper %}
{% endstep %}

{% step %}

#### Publish & verify

{% stepper %}
{% step %}
Click **Test** to verify the contract executes as expected
{% endstep %}

{% step %}
When you are done with the contract, click **Publish**

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FzRM5UwkO13qY5eIEyAu0%2Fimage.png?alt=media&amp;token=9df3df66-055c-4c2f-a157-0e07f22fb833" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Verify all checks**.

Soda will evaluate your rules against the current data.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FgU5RzT9XcQiQhBmCpKkb%2Fimage.png?alt=media&amp;token=f026c56f-c198-46c9-a66f-dd2cff441129" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}
{% endstep %}

{% step %}

#### Review check results

Review the outcomes of the contract checks to confirm whether the data meets expectations.

You can drill into those failures in the **Checks** tab.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FXi1JRkTJvhns8VviwXo7%2Fimage.png?alt=media&amp;token=70e92950-9ec7-4328-9b06-c2cfa7da299b" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

***

### Attack the issues in your pipeline

The same contract you published in Soda Cloud can be verified from your **terminal** or from a **pipeline step**. Run it after a transformation and before anything downstream reads the table, and bad data stops where it was produced.

Three tools do this. Pick by where the verification runs.

| Tool                     | Use it when                                                                                                      |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| **Soda CLI** (`sodacli`) | You want one tool for the pipeline and the platform. A single Go binary, no dependencies, structured exit codes. |
| **Core CLI** (`soda`)    | You want local execution in a Python environment, with fine-grained control over which checks run.               |
| **Python API**           | Verification is one step inside Python code you already maintain, such as an Airflow task or a notebook.         |

{% tabs %}
{% tab title="Soda CLI" %}
{% stepper %}
{% step %}
**Install**

```bash
brew tap sodadata/tap
brew install sodacli
```

Confirm it with `sodacli version`.

> Windows and install-script options are on the [Soda CLI reference](https://docs.soda.io/soda-documentation/developer-tools/soda-cli-reference).
> {% endstep %}

{% step %}
**Authenticate**

```bash
sodacli auth login
sodacli auth status
```

`auth login` prompts for host, API key ID and secret, and stores them in `~/.soda/credentials`. In CI, pass `--api-key-id`, `--api-key-secret` and `--no-interactive` instead.

> Generate the keys from your profile: [Generate API keys](https://docs.soda.io/apis/generate-api-keys).
> {% endstep %}

{% step %}
**Verify the contract**

```bash
sodacli contract verify databricks_demo/unity_catalog/demo_sales_operations/regional_sales
```

This runs the checks on a Soda Runner. To run them locally through Soda Core instead, add `--local --datasource datasource.yml`.
{% endstep %}

{% step %}
**Review the results**

```bash
sodacli results list --status failing
sodacli job logs <scan-id>
```

Output is a table in a terminal and JSON when piped. Force either with `--output json` or `--output csv`.
{% endstep %}
{% endstepper %}
{% endtab %}

{% tab title="Core CLI" %}
{% stepper %}
{% step %}
**Install**

Install the `soda` package and the connector for your data source, for example `soda-databricks` or `soda-postgres`.

{% hint style="warning" %}
The `soda` package ships from a private PyPI index and needs a Team or Enterprise license, unless you run verifications on a Soda Runner. Contact <support@soda.io> for access.
{% endhint %}

> Installation flow and package list: [Soda Python Libraries](https://docs.soda.io/soda-documentation/connect/deployment-options/soda-python-libraries).
> {% endstep %}

{% step %}
**Connect to Soda Cloud**

```bash
soda cloud create -f sc_config.yml
```

Fill in the generated file:

{% code title="sc\_config.yml" %}

```yaml
soda_cloud:
  host: cloud.soda.io          # Use cloud.us.soda.io for the US region
  api_key_id: ${env.SODA_API_KEY_ID}
  api_key_secret: ${env.SODA_API_KEY_SECRET}
```

{% endcode %}

Test it with `soda cloud test -sc sc_config.yml`.
{% endstep %}

{% step %}
**Verify the contract**

Fetch the contract you published in Soda Cloud, then verify it:

{% code overflow="wrap" %}

```bash
soda contract fetch -d databricks_demo/unity_catalog/demo_sales_operations/regional_sales -f contract.yaml -sc sc_config.yml

soda contract verify -ds ds_config.yml -c contract.yaml --publish
```

{% endcode %}

`--publish` sends the results back to Soda Cloud, so a pipeline run shows up next to the scheduled ones. It requires the **Manage contract** permission on the dataset.
{% endstep %}

{% step %}
**Run a subset of checks**

Verifying the whole contract on every pipeline run is often more than you need. Select checks by type, column or attribute:

```bash
soda contract verify -ds ds_config.yml -c contract.yaml -cf attributes.severity=critical
```

> Filters, wildcards and check paths: [Core CLI reference](https://docs.soda.io/soda-documentation/developer-tools/cli-reference)
> {% endstep %}
> {% endstepper %}
> {% endtab %}

{% tab title="Python API" %}
{% stepper %}
{% step %}
**Install and configure**

Install the `soda` package, then create `sc_config.yml` as shown in the Core CLI tab. The Python API reads the same file. {% endstep %}
{% endstep %}

{% step %}
**Verify the contract**

Create `verify_contract.py`:

```python
from soda_core import configure_logging
from soda_core.contracts import verify_contracts_on_runner

configure_logging(verbose=False)

result = verify_contracts_on_runner(
    dataset_identifiers=["databricks_demo/unity_catalog/demo_sales_operations/regional_sales"],
    soda_cloud_file_path="sc_config.yml",
)

print(result.get_logs_str())
```

The runner fetches the published contract from Soda Cloud and executes it against your data source.

{% hint style="info" %}
Calling verification in a loop? Call `configure_logging()` inside the loop, before each verification. The logging setup does not reset between iterations.
{% endhint %}

> Every parameter and result attribute: [Python API](https://docs.soda.io/apis/python-api).
> {% endstep %}
> {% endstepper %}
> {% endtab %}
> {% endtabs %}

***

#### **Fail the pipeline when a check fails**

Verification is only worth putting in a pipeline if a failed check stops the run. Each tool reports the outcome in a way your orchestrator can act on.

{% tabs %}
{% tab title="Soda CLI" %}
`sodacli contract verify` returns:

| Code | Meaning                                       |
| ---- | --------------------------------------------- |
| `0`  | All checks passed.                            |
| `1`  | One or more checks failed. Fail the pipeline. |
| `2`  | Execution error. Retry or alert.              |
| `3`  | Authentication error. Check credentials.      |

A GitHub Actions step stops on a non-zero code with no extra wiring:

```yaml
- name: Verify data contract
  run: |
    sodacli auth login --host cloud.soda.io \
      --api-key-id ${{ secrets.SODA_KEY_ID }} \
      --api-key-secret ${{ secrets.SODA_KEY_SECRET }} \
      --no-interactive
    sodacli contract verify contracts/regional_sales.yml --no-interactive
```

{% endtab %}

{% tab title="Core CLI" %}
`soda contract verify` returns a different set of codes to `sodacli`. The same codes apply with `--use-runner`.

| Code | Meaning                                                                                                                                                                                           |
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `0`  | All checks passed.                                                                                                                                                                                |
| `1`  | One or more checks failed.                                                                                                                                                                        |
| `2`  | One or more checks warned and none failed.                                                                                                                                                        |
| `3`  | The verification could not run: a flag given more than once, a contract Soda Cloud could not return, a contract parse error, an engine error, or a scan that errored, was canceled, or timed out. |
| `4`  | Results could not be sent to Soda Cloud.                                                                                                                                                          |

Treat `2` deliberately. A warning is not a failure, so decide whether your pipeline continues on it:

```bash
soda contract verify -ds ds_config.yml -c contract.yaml --publish
status=$?
if [ $status -eq 1 ] || [ $status -ge 3 ]; then
  exit $status
fi
```

{% endtab %}

{% tab title="Python API" %}
Verification returns a `ContractVerificationSessionResult`. Read the outcome from it rather than parsing the logs:

```python
if not result.is_ok:
    print(result.get_errors_str())
    raise SystemExit(1)
```

| Attribute    | True when                                                               |
| ------------ | ----------------------------------------------------------------------- |
| `is_passed`  | No contract has failed checks.                                          |
| `is_failed`  | At least one contract has a failed check. Execution errors are ignored. |
| `is_warned`  | At least one contract has a warned check.                               |
| `has_errors` | At least one contract hit an execution error.                           |
| `is_ok`      | No failed checks and no errors.                                         |

For per-check detail, iterate `result.contract_verification_results` and read each `check_results` entry. {% endtab %} {% endtabs %}
{% endtab %}
{% endtabs %}

<i class="fa-hexagon-check" style="color:$success;">:hexagon-check:</i> **Your contract now runs wherever your data is produced.**

<p align="center">Prefer to try this in a notebook first? <a href="https://colab.research.google.com/drive/1zkV_2tLJ4ohdzmKGS3LgdFDDnTNTUXew?usp=sharing" class="button secondary medium" data-icon="up-right-from-square">Contract verification example notebook</a></p>

***

### What’s next?

<table data-view="cards"><thead><tr><th data-type="content-ref"></th><th></th></tr></thead><tbody><tr><td><a href="https://docs.soda.io/soda-documentation/find/data-observability/profiling">Profiling</a></td><td>Explore <strong>Profiling</strong> in the <strong>Discover</strong> tab to curate column selections for deeper analysis.</td></tr><tr><td><a href="https://docs.soda.io/soda-documentation/fix/notifications">Notifications</a></td><td>Set up <strong>Notification Rules</strong> (bell icon → <strong>Add Notification Rule</strong>) to push alerts to Slack, Jira, PagerDuty, etc.</td></tr><tr><td><a href="https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/custom-monitors">Custom monitors</a></td><td>Dive into <strong>Custom</strong> <strong>Monitors</strong> via <code>scan.yml</code> or the UI for even more tailored metrics.</td></tr></tbody></table>

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Overview

Discover how Soda AI can help you improve and automate your data quality seamlessly

<table data-header-hidden><thead><tr><th width="186.66668701171875">Feature</th><th>What it does</th></tr></thead><tbody><tr><td><a href="#chat-interface">Chat interface</a></td><td>Conversational entry point to Soda AI: <strong>performs</strong> actions in Soda, <strong>guides</strong> workflows, and <strong>answers</strong> questions, all in-product.</td></tr><tr><td>Contract Autopilot</td><td><strong>Creates</strong> a first version of a data contract.</td></tr><tr><td><a href="https://docs.soda.io/soda-documentation/soda-ai/contract-copilot">Contract Copilot</a></td><td>Helps <strong>edit</strong> and <strong>iterate</strong> on existing data contracts.</td></tr><tr><td><a href="https://docs.soda.io/soda-documentation/soda-ai/soda-mcp">MCP</a></td><td><strong>Connect your own agents</strong> to trusted data.</td></tr><tr><td>Root cause analysis (RCA) agent</td><td><strong>Investigates</strong> failing checks from Claude Code and finds the root cause.</td></tr><tr><td><a href="https://docs.soda.io/apis/rest-api">API</a></td><td><strong>Integrate</strong> Soda with your own systems.</td></tr><tr><td><a href="https://docs.soda.io/soda-documentation/developer-tools/soda-cli-reference">Soda CLI</a></td><td><strong>Run data quality programmatically</strong> in your existing pipelines.</td></tr></tbody></table>

**Soda AI gets your data AI-ready**. It includes several capabilities that serve one goal: trusted data that's ready for an AI-driven workflow. With Soda AI, your team and AI agents can turn data into actionable, understandable and trustworthy information.

The following capabilities make this work:

<table data-header-hidden><thead><tr><th width="186.66668701171875">Feature</th><th>What it does</th></tr></thead><tbody><tr><td><a href="https://docs.soda.io/soda-documentation/soda-ai/soda-mcp">Soda MCP</a></td><td><strong>Connect your own agents</strong> to trusted data.</td></tr><tr><td>Root cause analysis (RCA) agent</td><td><strong>Investigates</strong> failing checks from Claude Code and finds the root cause.</td></tr><tr><td><a href="https://docs.soda.io/soda-documentation/soda-ai/soda-ai-agent">Soda AI Agent</a></td><td>Conversational entry point to Soda AI: <strong>performs</strong> actions in Soda, <strong>guides</strong> workflows, and <strong>answers</strong> questions, all in-product.</td></tr><tr><td>Contract Autopilot</td><td><strong>Creates</strong> a first version of a data contract.</td></tr><tr><td><a href="https://docs.soda.io/soda-documentation/soda-ai/contract-copilot">Contract Copilot</a></td><td>Helps <strong>edit</strong> and <strong>iterate</strong> on existing data contracts.</td></tr><tr><td><a href="broken://pages/jsZPg1FqxguLLnwjLeHe">API</a></td><td><strong>Integrate</strong> Soda with your own systems.</td></tr><tr><td><a href="https://docs.soda.io/soda-documentation/developer-tools/soda-cli-reference">Soda CLI</a></td><td><strong>Run data quality programmatically</strong> in your existing pipelines.</td></tr></tbody></table>

## What is Soda AI?

**Soda AI is the collection of AI-powered features** that make your data quality AI-ready. It isn't a single tool but the umbrella over everything in Soda that uses AI to define, maintain, and reason about data quality: the **Soda AI Agent**, **Autopilot**, **Copilot**, and the **tools** that connect your contracts and data-quality status to **external agents**.

Soda AI's features fall into a few groups:

<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-cover data-type="image">Cover image</th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Soda AI Agent</strong></td><td>The <strong>Soda AI Agent</strong> is the in-product way of interacting with Soda AI in natural language. It answers questions about Soda and your data, guides you through platform workflows, and proposes/performs actions on your behalf. As a Soda AI interface, the Agent reaches for the tools below to carry out tasks.</td><td><a href="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FGayhWCvMT0CwU9XbSPl3%2Fsoda_chat.gif?alt=media&#x26;token=623faa81-59a0-4984-9cd9-a2e00eeaf5b5">https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FGayhWCvMT0CwU9XbSPl3%2Fsoda_chat.gif?alt=media&#x26;token=623faa81-59a0-4984-9cd9-a2e00eeaf5b5</a></td><td><a href="https://docs.soda.io/soda-documentation/soda-ai/soda-ai-agent">Soda AI Agent</a></td></tr><tr><td><strong>Contract Copilot</strong></td><td><strong>Edit and iterate on contracts</strong> in plain English (adding a freshness check, tightening a rule, handling a nested field) without hand-writing code lines. Contract Copilot will show you exactly what changed.</td><td><a href="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FH1WDk6863pSYzVg8LVWg%2Fsoda_copilot_short.gif?alt=media&#x26;token=bad2660c-1d07-4ec5-a50c-bf7d08830fb8">https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FH1WDk6863pSYzVg8LVWg%2Fsoda_copilot_short.gif?alt=media&#x26;token=bad2660c-1d07-4ec5-a50c-bf7d08830fb8</a></td><td><a href="https://docs.soda.io/soda-documentation/soda-ai/contract-copilot">Contract Copilot</a></td></tr><tr><td><strong>Agent-facing interfaces</strong></td><td><strong>Connect agents to trusted data.</strong> MCP, Soda CLI and API put your contracts and data-quality status in front of the tools your team and your agents already use. Engineers can run quality checks inside their existing pipelines, and agents act on verified data.</td><td><a href="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fibl0Rsrn6CAGgsTkEjub%2FFrame%203518(4).png?alt=media&#x26;token=fce24d27-42c5-4dc6-8dc2-72c9854d7b43">https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fibl0Rsrn6CAGgsTkEjub%2FFrame%203518(4).png?alt=media&#x26;token=fce24d27-42c5-4dc6-8dc2-72c9854d7b43</a></td><td><a href="https://docs.soda.io/soda-documentation/soda-ai/soda-mcp">Soda MCP</a></td></tr></tbody></table>

Across all of these, **Soda AI is designed to**:

* **Reduce manual effort** in defining and maintaining data quality expectations
* **Help** users **reason about data** and contracts faster
* **Integrate with existing workflows**, enabling AI agents to interact with Soda Cloud in your own pipelines.
* **Keep humans in control**: every proposed action is shown and approved by a person before it is applied
* **Operate safely** within strict privacy, security, and governance boundaries

### Limitations

* Soda AI features may not always produce complete or perfectly accurate answers.
* Users remain responsible for reviewing any proposed action before it is applied.
* The Soda AI Agent, Contract Autopilot, and Contract Copilot do not operate outside the context of your Soda Cloud workspace.

{% hint style="success" %}
**The Soda AI Agent, Contract Autopilot, and Contract Copilot are assistive, not autonomous**. These features are designed as a control surface for human-in-the-loop data quality, scoped to the same operations you could perform yourself, subject to your role and permissions.
{% endhint %}

***

## Enable/Disable AI features

AI features in Soda can be configured from the **Organization Settings**.

{% stepper %}
{% step %}
Navigate to **Settings** > **Organization Settings**.
{% endstep %}

{% step %}
Under the **Organization** tab **> Soda AI features:** Check/Uncheck the feature checkbox to enable/disable it.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FehVJec3tG1doQbuy2pPK%2FFrame%203518(1).png?alt=media&#x26;token=a87b42f6-dbd1-47ad-8c2d-aebf75e27da3" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Save** on the top right corner.
{% endstep %}
{% endstepper %}

***

## Soda AI security & privacy measures

Soda's features are built with **privacy and security as first-class design principles**. The following safeguards apply to all AI/ML-powered features in Soda.

### Security and privacy

Soda AI features follow the same security and privacy principles as the rest of Soda:

* **Your data isn't used to train models.** Neither Soda nor the model provider trains or fine-tunes models on your data, metadata, or results.
* **Source data isn't sent by default.** Row-level data and PII only reach a model provider for features you opt in to.
* **Soda AI acts within your permissions.** It can't do anything you couldn't do yourself with your role.
* **Failed rows stay in your warehouse.** Rows stored by Diagnostics Warehouse aren't available to Soda AI.

> For what each feature sends to the model provider, what is stored, and which settings admins control, see [AI security, compliance & privacy](https://docs.soda.io/soda-documentation/soda-ai/ai-security-compliance-and-privacy).

#### **Confidentiality & access control**

* Soda AI's actions are subject to your **existing** **Soda roles and permissions**. None of the features can perform anything you could not perform yourself.
* Requests are **not used for model training**.

### AI guardrails and ethics

* Foundation models include built-in safety guardrails from the provider
* Additional **system-level guardrails** restrict responses to the intended Soda use cases
* Soda AI features do not generate harmful, discriminatory, or user-targeted content

Because Soda AI features do not evaluate users or make decisions, traditional bias and fairness risks are minimal.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Soda MCP

Use Soda from your preferred AI agent.

{% hint style="info" %}
**The MCP server does not require Soda AI enablement**. It provides agentic access to the Soda Cloud API, meaning your organization can use it without enabling Soda AI features.

The MCP will not be able to reach the capabilities that depend on Soda AI if the latter is not enabled.
{% endhint %}

Soda MCP is available to Soda v4 users. Connect your AI client to Soda Cloud to inspect datasets and checks, manage contracts, and work with other data quality resources.

[Connect remotely](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/connect-remotely) for the recommended setup: sign in with your Soda Cloud account, without installing a server or creating API keys. [Run locally](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/run-locally) if you need control over the MCP server and its security configuration.

`soda-mcp` is a Model Context Protocol (MCP) server for Soda Cloud. **It lets MCP-capable AI clients use Soda Cloud's public API** to inspect and manage data quality resources such as datasets, checks, contracts, scans, monitors, incidents, attributes, data sources, users, and user groups.

Reach for Soda MCP **when you want an AI agent to help with data workflows**, for example:

* **List datasets** and **inspect** their health status.
* **Retrieve** checks, contracts, scans, and incidents.
* **Create** or **update** contracts and monitors.
* **Trigger** discovery, contract verification, historical metric collection, or other supported Soda Cloud actions.
* **Manage** **Soda Cloud resources** such as attributes, roles, users, and user groups.
* **Build complex workflows** across your entire AI layer, connecting to other agents and MCPs

> To view the full list of MCP tools, navigate to the [Tools documentation.](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/mcp-tools)

#### MCP vs. Soda AI Agent <a href="#mcp-vs.-soda-ai" id="mcp-vs.-soda-ai"></a>

Soda MCP connects your own AI agent to supported Soda Cloud operations. [Soda AI Agent](https://docs.soda.io/soda-documentation/soda-ai/soda-ai-agent) provides an agent inside Soda Cloud via a chat, combining supported Cloud actions with Soda documentation and applicable webpage context. Their capabilities overlap, but their tools and execution environments differ.

```mermaid
flowchart TB
    own["Your AI agent"] --> mcp["Soda MCP"]
    chat["Soda AI chat in Soda Cloud"] --> agent["Soda AI agent"]
    agent --> embedded["Embedded MCP tools"]
    agent --> docs["Soda documentation retrieval"]
    agent --> page["Webpage context and UI integration"]
    mcp --> operations
    embedded --> operations
    subgraph operations["Supported Soda Cloud operations"]
        standard["Datasets, contracts, checks, and more"]
        autopilot["Contract Autopilot<br/>Requires enablement"]
    end
```

| Workflow               | Your agent with Soda MCP                                                                                                          | Soda AI chat                                                                                                                              |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| **Setup**              | Connect remotely with your Soda Cloud login, or run locally with API keys.                                                        | Nothing to install. Requires Soda AI chat to be enabled.                                                                                  |
| **Permissions**        | Remote access uses your signed-in Soda Cloud permissions; local access uses the permissions associated with your API key.         | Acts as you, with your permissions.                                                                                                       |
| **Soda AI enablement** | Not required. Contract Autopilot requires enablement; other MCP operations remain available.                                      | Required.                                                                                                                                 |
| **Background work**    | Enables scheduled, parallel, and background work through your agent or orchestration system.                                      | Supports concurrent conversations and continued responses while the panel is hidden. Does not provide durable unattended agent execution. |
| **Product guidance**   | Depends on your agent and connected tools. Includes contract and data-standard authoring guidance, but no general Soda Q\&A tool. | Retrieves Soda documentation and uses applicable webpage context.                                                                         |

### Installation instructions

Choose [Connect remotely](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/connect-remotely) for the recommended setup, or [Run locally](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/run-locally) for the local alternative.

#### Requirements

See [local requirements](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/run-locally#requirements).

#### Install from Soda's private PyPI

See [private package installation](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/run-locally#install-from-sodas-private-pypi).

#### Configure an MCP client

See [local client configuration](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/run-locally#configure-an-mcp-client).

#### Verify the server

See [local server verification](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/run-locally#verify-the-server).

### Usage instructions

#### Use the tools

Once connected, ask your AI client to perform Soda Cloud tasks naturally. Examples:

```
List the datasets in my Soda Cloud organization and summarize unhealthy datasets.
```

```
Can you show me the latest failing checks?
```

```
Verify the contract for the dataset datasource/database/schema/table
```

```
Use Contract Autopilot to create a contract for the dataset datasource/database/schema/table
```

Some tools modify Soda Cloud resources or trigger scans. Review the client's proposed tool call before approving write, delete, or trigger actions.

#### Usage notes

`create_secret`, `update_secret`, and `create_runner` are available only when you [run locally](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/run-locally). For long-lived secrets, prefer the Soda Cloud Web UI so plaintext stays out of chat history. See the [shared tools reference](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/mcp-tools).

MCP servers expose tools and instructions on how to use them. Some MCP clients, like Codex and OpenCode, do not load the instructions, only the tools. This may affect performance. Claude Code loads both tools and instructions.

#### Troubleshooting

See [remote connection troubleshooting](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/connect-remotely#troubleshooting) or [local server troubleshooting](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/run-locally#troubleshooting).

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Connect remotely

Connect your AI client to Soda MCP with your Soda Cloud login.

Connect to Soda MCP with your Soda Cloud account. Remote access is available to Soda v4 users and requires no local server, Python, private package index, or API keys.

The steps below configure Claude Code for your own user account, across projects on this machine. Start with [Claude Code installed](https://code.claude.com/docs/en/setup) and access to your Soda Cloud organization.

{% hint style="warning" %}
Soda MCP includes tools that create, update, and delete resources or start scans. Actions remain limited by your Soda Cloud permissions. Review proposed tool calls before approving them.
{% endhint %}

## Connect Claude Code

{% stepper %}
{% step %}

### Choose your region

Use the region where you sign in to Soda Cloud. If you are unsure, check your Soda Cloud browser address rather than guessing.

| Region        | Soda Cloud address         | MCP server URL                 |
| ------------- | -------------------------- | ------------------------------ |
| EU            | `https://cloud.soda.io`    | `https://cloud.soda.io/mcp`    |
| US            | `https://cloud.us.soda.io` | `https://cloud.us.soda.io/mcp` |
| {% endstep %} |                            |                                |

{% step %}

### Add your connection

In the terminal where you run Claude Code, inspect existing connections first:

```bash
claude mcp list
```

If Soda is already configured, inspect that entry with `claude mcp get <name>`. Reuse it if it has the correct URL and user scope; continue to sign-in. If it uses local stdio, a different region, or another scope, follow [Existing connections](#existing-connections) before adding anything. Check for Soda entries under other names or installed plugins too.

If no Soda connection exists, run only the command for your region:

{% tabs %}
{% tab title="EU" %}

```bash
claude mcp add --transport http --scope user soda-mcp https://cloud.soda.io/mcp
```

{% endtab %}

{% tab title="US" %}

```bash
claude mcp add --transport http --scope user soda-mcp https://cloud.us.soda.io/mcp
```

{% endtab %}
{% endtabs %}

`--scope user` keeps this configuration personal. Do not add it to a shared project configuration or change organization-wide settings. The same single-line command works in Bash, PowerShell, and Windows Command Prompt; run it on the host where your client runs, including inside WSL if applicable.
{% endstep %}

{% step %}

### Sign in

Start or restart your Claude Code session, then enter:

```
/mcp
```

Select your Soda connection (`soda-mcp` if added above) and follow the authentication flow. Sign in to Soda Cloud in your browser and review the consent request yourself. Use the normal OAuth login and consent flow; do not add API key headers. If the browser does not open, open the URL Claude Code provides. Return to Claude Code after consent.
{% endstep %}

{% step %}

### Verify your connection

In `/mcp`, confirm that your Soda connection (`soda-mcp` if added above) is connected. Then ask Claude Code:

```
Use Soda MCP to call list_datasets with page 0 and size 10. Report whether the call succeeds without changing any resources.
```

A successful dataset response, including an empty result, confirms a working Cloud-backed connection. Seeing tools in the list is not enough. A tool error, including `isError: true` inside an HTTP 200 response, means verification failed.
{% endstep %}
{% endstepper %}

> Continue with the shared [MCP tools reference](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/mcp-tools). `create_secret`, `update_secret`, and `create_runner` are local-only.

## Setup prompt

Copy this prompt into the AI client you want to configure. The client must support remote HTTP MCP and OAuth; this is general setup guidance, not a compatibility guarantee for every client.

{% code overflow="wrap" %}

```
Set up remote Soda MCP for this client and my personal user account using https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/connect-remotely. Adapt to my host and shell. Inspect existing Soda connections, preserve unrelated configuration, and avoid duplicates. Ask which Soda region I use if unclear; never guess. Make only the necessary setup changes, minimizing manual steps. Let me complete login and consent myself. Verify with a successful read-only tool call and report the result. Do not change shared or organization settings or install unrelated software. Offer local fallback only if necessary and after my confirmation, not for transient failures. If tools or client support are unavailable, give minimal next steps without claiming success.
```

{% endcode %}

## Existing connections

Do not silently overwrite an existing registration or leave both local and remote Soda connections active. Confirm the intended change before replacing or disabling a connection. Preserve unrelated entries and keep a recoverable copy of the entry you change, without exposing its credentials. A same-name entry in local or project scope takes precedence over user scope.

For an existing personal user-scoped entry named `soda-mcp`, once you have confirmed replacement, remove only that entry, then add the remote connection above:

```bash
claude mcp remove soda-mcp --scope user
```

For a plugin-managed or shared entry, use the client's supported controls to disable it for your own use, or ask your administrator if personal changes are restricted. Do not alter shared configuration as part of this setup.

## Troubleshooting

* **Authentication required:** open `/mcp` and complete sign-in for the configured Soda connection. Confirm that the endpoint matches your Soda Cloud region.
* **Permission denied:** the tool still requires the corresponding Soda Cloud permissions. Ask your Soda administrator if you need additional access.
* **Connection timeout or service error:** retry later without changing your region, deleting valid credentials, or switching automatically to local setup.
* **Unsupported client or no execution tools:** follow that client's personal remote HTTP MCP and OAuth setup instructions, using the URL above. If it cannot connect remotely, consider [Run locally](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/run-locally) only after confirming that alternative and its prerequisites.

> For client-specific details, see the [Claude Code MCP reference](https://code.claude.com/docs/en/mcp).

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Run locally

Run Soda MCP locally with your Soda Cloud API keys.

Run Soda MCP locally if you need control over the MCP server and its security configuration. For the recommended setup without Python, package installation, or API keys, [connect remotely](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/connect-remotely).

The server runs as a local stdio MCP process. Your AI client starts `soda-mcp`, passes Soda Cloud credentials through environment variables, and then calls Soda tools through the MCP protocol. On startup, `soda-mcp` authenticates with Soda Cloud and loads organization context so the client can use the exposed tools against the correct tenant.

Follow the instructions in this section to install the Soda MCP server for your host client.

#### Requirements

* Python 3.10+
* [`uv`](https://docs.astral.sh/uv/) python package on `PATH`
* A Soda Cloud API key ID and secret
* A Soda Cloud host URL

If you don't have API keys already you can create them following the instructions [here](https://docs.soda.io/apis/generate-api-keys).

Set the runtime environment variables before starting the server. Use `cloud.soda.io` for EU or `cloud.us.soda.io` for US; the examples below use EU:

```bash
export SODA_CLOUD_HOST="cloud.soda.io"
export SODA_API_KEY_ID="your-key-id"
export SODA_API_KEY_SECRET="your-key-secret"
```

<i class="fa-exclamation" style="color:$warning;">:exclamation:</i> **Note:** If you are using Windows without WSL you will need to modify the commands to work with either the Windows command prompt:

```shellscript
set SODA_CLOUD_HOST=cloud.soda.io
set SODA_API_KEY_ID=your-key-id
set SODA_API_KEY_SECRET=your-key-secret
```

or Windows Powershell:

```powershell
$env:SODA_CLOUD_HOST = "cloud.soda.io"
$env:SODA_API_KEY_ID = "your-key-id"
$env:SODA_API_KEY_SECRET = "your-key-secret"
```

You will also need to modify the commands of the following sections as is appropriate.

#### Install from Soda's private PyPI

Stable releases are published to Soda's private PyPI indexes for Team and Enterprise customers. Choose the private index that matches your license and region, then set `UV_INDEX` using the same Soda API key credentials:

<table><thead><tr><th width="138.66665649414062">License</th><th width="122.16668701171875">Region</th><th>Index URL</th></tr></thead><tbody><tr><td>Team</td><td>EU</td><td><code>https://team.pypi.cloud.soda.io</code></td></tr><tr><td>Team</td><td>US</td><td><code>https://team.pypi.us.soda.io</code></td></tr><tr><td>Enterprise</td><td>EU</td><td><code>https://enterprise.pypi.cloud.soda.io</code></td></tr><tr><td>Enterprise</td><td>US</td><td><code>https://enterprise.pypi.us.soda.io</code></td></tr></tbody></table>

Pick the appropriate command below according to your plan and region. Note that this command depends on the Soda API key environment variables mentioned above in order to execute successfully.

Pick the proper one according to your plan and region:

{% tabs %}
{% tab title="Team EU" %}
{% code overflow="wrap" %}

```bash
export UV_INDEX="https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@team.pypi.cloud.soda.io"
```

{% endcode %}
{% endtab %}

{% tab title="Team US" %}
{% code overflow="wrap" %}

```bash
export UV_INDEX="https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@team.pypi.us.soda.io"
```

{% endcode %}
{% endtab %}

{% tab title="Enterprise EU" %}
{% code overflow="wrap" %}

```bash
export UV_INDEX="https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.cloud.soda.io"
```

{% endcode %}
{% endtab %}

{% tab title="Enterprise US" %}
{% code overflow="wrap" %}

```bash
export UV_INDEX="https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.us.soda.io"
```

{% endcode %}
{% endtab %}
{% endtabs %}

Smoke-test that `uvx` can resolve and start the package. Stop the server with `Ctrl-C` after startup:

```bash
uvx -qq --no-progress soda-mcp@latest
```

If everything is set up correctly you will see the following output:

```
Authenticated as FirstName LastName (userEmail@soda.io) in org Soda TestOrg
Starting soda-mcp server (base_url=https://cloud.soda.io)
```

#### Configure an MCP client

Configure your MCP client to start `soda-mcp` over stdio and pass the required environment variables to the child process.

Before adding a connection, inspect your client's existing Soda registrations. Reuse a suitable connection; do not silently replace one or leave duplicate active connections. Preserve unrelated settings when editing configuration.

For **Claude Code:**

```bash
claude mcp add soda-mcp \
  --transport stdio \
  --scope user \
  -e SODA_CLOUD_HOST="$SODA_CLOUD_HOST" \
  -e SODA_API_KEY_ID="$SODA_API_KEY_ID" \
  -e SODA_API_KEY_SECRET="$SODA_API_KEY_SECRET" \
  -e UV_INDEX="$UV_INDEX" \
  -- uvx -qq --no-progress soda-mcp@latest
```

For **OpenCode**, add a local MCP server entry to your user-scope `opencode.jsonc`:

```jsonc
{
  "mcp": {
    "soda-mcp": {
      "type": "local",
      "command": [
        "env",
        "SODA_CLOUD_HOST=<host>",
        "SODA_API_KEY_ID=<key-id>",
        "SODA_API_KEY_SECRET=<key-secret>",
        "UV_INDEX=<private-index-url-with-auth>",
        "uvx",
        "-qq",
        "--no-progress",
        "soda-mcp@latest"
      ]
    }
  }
}
```

For a generic stdio MCP client:

```json
{
  "mcpServers": {
    "soda-mcp": {
      "command": "uvx",
      "args": ["-qq", "--no-progress", "soda-mcp@latest"],
      "env": {
        "SODA_CLOUD_HOST": "<host>",
        "SODA_API_KEY_ID": "<key-id>",
        "SODA_API_KEY_SECRET": "<key-secret>",
        "UV_INDEX": "<private-index-url-with-auth>"
      }
    }
  }
}
```

#### Verify the server

After configuring the client, confirm that `soda-mcp` appears in the client's MCP server list and is connected. Then invoke a low-cost read tool such as `list_attributes`.

A successful response confirms that:

* The MCP client can start the server.
* Soda Cloud credentials are reaching the server.
* Authentication succeeds against the configured Soda Cloud tenant.
* The client can see and call the registered Soda tools.

#### Troubleshooting

* If `uvx` cannot resolve `soda-mcp`, confirm `UV_INDEX` is set in the MCP client's environment and points at the correct private index.
* If private PyPI returns `401` or `403`, confirm the API key ID and secret are valid for the selected Team or Enterprise index.
* If startup fails with missing credential errors, confirm the MCP client passes `SODA_API_KEY_ID`, `SODA_API_KEY_SECRET`, and `SODA_CLOUD_HOST` .
* If Soda Cloud authentication fails, confirm the API key belongs to the same tenant as the configured host or URL.
* If the client does not show any tools, refresh or restart the MCP client so it reconnects and reloads tool descriptions.

> For available tools and usage guidance, see [MCP tools](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/mcp-tools) and [Soda MCP](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp#usage-instructions).

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# MCP tools

Soda MCP exposes Soda Cloud operations as Model Context Protocol tools. This catalog describes the 103 tools in Soda MCP 0.10.0 and covers both [remote](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/connect-remotely) and [local](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp/run-locally) connections.

Tools marked **Local only** are not available remotely: `create_secret`, `update_secret`, and `create_runner`. Tool use is subject to your Soda Cloud permissions and organization features.

Most tools map to documented operations within Soda v4. The contract and data-standard knowledge tools provide packaged authoring guidance without calling Soda Cloud.

<details>

<summary>Attributes - labels for organizing datasets, checks, and columns</summary>

Attributes categorize and organize resources, such as marking checks as critical for notification rules.

* `list_attributes` — List attribute definitions available in your Soda Cloud organization for datasets, checks, columns, data standards, and data products.
* `create_attribute` — Create a new attribute definition for datasets, checks, columns, data standards, or data products.
* `update_attribute` — Update an attribute's label, description, or allowed values.
* `delete_attribute` — Delete an attribute definition. The attribute is removed from any resources where it is set.

</details>

<details>

<summary>Checks - data quality validations from multiple sources</summary>

Checks are Soda Cloud data quality validations and can originate from Soda products, dbt, data contracts, data standards, or metric monitoring. Contract-authored checks can use fixed thresholds; metric-monitoring checks use dynamic thresholds. There is no generic public create-check endpoint.

In 0.10, check discovery and check history exclude monitors. Use `list_monitors` to find a dataset's monitors and `list_monitor_results` for their history. For the latest results of known IDs, `list_checks(check_ids=[...])` accepts 1–100 check or monitor IDs; do not combine that batch with filters, pagination, or sorting. Ordinary SodaCL anomaly checks remain checks.

* `list_checks` — List checks in your Soda Cloud organization, including current status and latest result, associated datasets, agreements, and linked incidents.
* `list_check_results` — List historical results for one check, including result IDs, evaluation statuses, and data timestamps.
* `delete_check` — Delete a specific check by its ID. Checks are the general Soda Cloud validation resource and can originate from contracts, data standards, Soda products, dbt, or metric monitoring.

</details>

<details>

<summary>Datasets - tables tracked in Soda Cloud</summary>

* `list_datasets` — List datasets in your Soda Cloud organization, including data source, incidents, attributes, health status, and contract metadata.
* `get_dataset` — Get a single dataset by dataset ID.
* `get_dataset_by_dataset_qualified_name` — Get a single dataset by dataset qualified name (DQN), e.g. \[datasource\_name]/\[database\_name]/\[schema\_name]/\[dataset\_name].
* `update_dataset` — Update dataset properties (label, tags, attributes, owners, profiling, metric monitoring, diagnostics warehouse, compute warehouse, time partition, data delay).
* `list_dataset_columns` — List active columns of a dataset with current assigned column attribute values.
* `set_column_attributes` — Set attribute values on columns of a dataset.
* `delete_dataset` — Delete a dataset. This action is permanent and cannot be undone.
* `list_dataset_roles` — List available dataset roles. Use dataset roles to manage access to individual datasets.
* `create_dataset_role` — Create a custom dataset role.
* `update_dataset_role` — Update the name or permissions of a custom dataset role.
* `delete_dataset_role` — Delete a custom dataset role. This action is permanent and cannot be undone.
* `get_dataset_compute_warehouse` — Get the compute warehouse configuration for a dataset.
* `get_dataset_diagnostics_warehouse` — Get diagnostics warehouse information for a dataset.
* `get_dataset_profiling` — Get profiling information for a dataset (structure, statistical summaries, data characteristics).
* `list_dataset_responsibilities` — List user and user group permissions assigned to a dataset, and their associated roles.
* `update_dataset_responsibilities` — Update user/group permissions and their associated roles for a dataset.

</details>

<details>

<summary>Monitors - ML-based dynamic-threshold metrics</summary>

Monitors track metrics over time with anomaly detection. Dataset monitors cover built-in metrics, while column and custom SQL monitors are user-defined.

General monitor management requires a supported V4 dataset and **Manage contracts** permission; creating monitors also requires the relevant monitoring features and dataset setup. Dataset monitors can be disabled with `update_monitor`, but cannot be deleted.

For grouped monitors, read history using the group member IDs; group parents have no results of their own. Record-level Anomaly Detection (RAD) requires feature and runner support. Its diagnostics describe partition-level distribution changes, not individual anomalous rows or proven business causes.

Feedback records your judgment about a result's classification. Pass 1–100 result IDs, not monitor or check IDs, to `submit_monitor_result_feedback`. Investigating a result does not submit feedback; confirm any judgment proposed by your agent before it submits feedback.

{% hint style="warning" %}
Re-enabling freshness, rows-inserted, or RAD monitoring can reset other monitors' history and start collecting historical data again. Clearing a disabled monitor's `isEnabled` override with `null` can also re-enable it.

Only after you authorize this broader reset, pass `allow_dataset_history_reset=true` as a separate argument to `update_dataset` or `update_monitor`, not inside `body`. Otherwise, affected edits return `dataset_history_reset_required` before any write. SQL query, result-column, or grouping changes can also clear affected monitor history. Let automatic history collection finish rather than starting it again; starting collection does not mean training is complete.
{% endhint %}

* `list_monitors` — List a V4 dataset's metric monitors and latest outcomes, across dataset, column and custom SQL types.
* `list_monitor_results` — List one metric monitor's historical results and available diagnostics, including RAD explanations.
* `submit_monitor_result_feedback` — Replace classification feedback on 1-100 V4 metric-monitor results with the user's judgment.
* `get_dataset_metric_monitoring` — Get metric monitoring configuration for a dataset, including all three monitor types and the dataset-level scan schedule.
* `create_monitor` — Create a V4 dataset, built-in column, or custom SQL metric monitor using ML-based dynamic thresholds.
* `update_monitor` — Update a metric monitor by ID; the backend resolves its dataset and family.
* `delete_monitor` — Delete a column or custom SQL metric monitor, permanently removing its results and group children.
* `run_historical_metric_collection` — Trigger a historical metric collection scan for a dataset. The v4 API calls this a historical metric collection scan; in user-facing terms it backfills metric history so monitors (ML-based anomaly detection) have a baseline to learn from.

</details>

<details>

<summary>Contracts - per-dataset sources of truth for contract-authored checks</summary>

A data contract defines its contract-authored checks for one dataset. See Editing contracts for the safe edit workflow.

* `list_contracts` — List data contracts in your Soda Cloud organization. A data contract is the declarative source-of-truth for contract-authored checks on a dataset.
* `create_contract` — Create and publish a data contract on a dataset from the YAML `contents` you supply.
* `get_contract` — Retrieve a specific data contract by ID, including its YAML content, with an optional contract-editing mode.
* `edit_contract` — Edit an existing data contract's YAML using its contract ID and the actual `edit_revision` from a completed `get_contract(for_editing=true)` result. This is not a general editor for check YAML: data-standard YAML, including threshold patches and previews, uses `update_data_standard`. Never substitute a data-standard ID or invent a revision. Operations-mode diffs compare the normalized baseline with the patched candidate; contents-mode diffs compare the exact fetched and caller-supplied strings.
* `get_contract_editing_knowledge` — Get packaged contract-editing knowledge, returning the authoring guide by default.
* `list_contract_versions` — List published versions of a specific data contract. Each publish creates a new version; this tool returns the version history.
* `verify_contract` — Trigger a contract verification scan for the given contract. Runs the checks defined in the contract against the dataset.
* `create_skeleton_contract` — Trigger async generation of a skeleton contract for a dataset, derived from its warehouse schema. Use this to bootstrap a contract's YAML body when no contract has been published yet.
* `get_skeleton_contract_status` — Get the status of a skeleton contract generation operation started via `create_skeleton_contract`.
* `generate_contracts` — Trigger async AI-powered generation of full contracts for one or more datasets.
* `get_contract_generation_status` — Get the status of a contract generation operation started via `generate_contracts`.

</details>

<details>

<summary>Data Standards - organization-wide YAML policies applied by scope</summary>

A data standard applies checks to every dataset matching its scope, unlike a contract, which belongs to one dataset. Targeted YAML editing preserves supported public configuration and validates the whole candidate locally. See Editing data standards for modes, preview, and publication boundaries.

* `list_data_standards` — List data standards in your Soda Cloud organization. A data standard is an organization-wide YAML policy whose checks are automatically applied to every dataset matching its scope.
* `get_data_standards_activity` — Get an organization-wide rollup of data standards activity, including counts of active and total standards, matched datasets, and aggregated check results.
* `get_data_standard` — Retrieve a specific data standard by ID, including its YAML contents, attributes, and scope.
* `get_data_standard_authoring_knowledge` — Get packaged knowledge for creating and updating data standards and previewing their scope; the guide is the default.
* `list_data_standard_checks` — List the aggregated checks generated by a specific data standard across the datasets it matches.
* `list_data_standard_datasets` — List the datasets currently matched by a specific data standard's scope.
* `evaluate_data_standard_scope` — Re-evaluate which datasets currently match a data standard's scope and return their dataset IDs.
* `preview_data_standard_scope` — Preview which datasets a candidate data standard scope matches while leaving saved standards unchanged.
* `create_data_standard` — Create an organization-wide data standard whose YAML checks apply to every dataset matching its scope.
* `update_data_standard` — Edit a data standard's YAML checks or configuration by `data_standard_id`.
* `update_data_standard_status` — Transition a data standard to a new status without modifying its contents, attributes, scope, owners, or schedule. Use this to pause or activate an existing standard.
* `delete_data_standard` — Delete a data standard.
* `execute_data_standards` — Trigger a scan that runs the active data standards linked to a dataset. Identify the dataset by `datasetId`.
* `dry_run_data_standard` — Test an unsaved data standard (a dry run) by running its checks once against one dataset's live data, without creating or modifying a public data standard. Provide the standard YAML in `contents` and the target dataset in `datasetId`.

</details>

<details>

<summary>Data Products - input datasets grouped behind a report or dashboard</summary>

A data product groups the input datasets behind a report, dashboard, or other deliverable and reports their combined health. It is defined in YAML and addressed by its `key`, not its display label. It owns no checks: contracts and data standards define them.

These tools require the `dataProductsEnabled` organization feature and the relevant data product permissions.

Create and update accept the complete YAML in `body.contents`. Before updating, read `get_data_product` and preserve unchanged fields: update replaces the definition, so omitted inputs, attributes, or descriptions are removed. Use `list_attributes(resource_type="dataProduct")` to resolve attribute definitions, and review any warnings returned by create or update. Deleting a product leaves its input datasets, contracts, and checks unchanged.

* `list_data_products` — List data products in your Soda Cloud organization. A data product groups the input datasets behind one report, dashboard, or other deliverable and reports their combined health.
* `get_data_product` — Retrieve one data product by `data_product_name`, its YAML `key`, including its YAML `contents`, owner, attributes, and health.
* `list_data_product_inputs` — List the input datasets of one data product, with each input's check-result counts and last evaluation time.
* `create_data_product` — Create a data product from its YAML definition in `body.contents`.
* `update_data_product` — Replace a data product's YAML definition. `data_product_name` is the product's YAML `key`.
* `delete_data_product` — Delete a data product by `data_product_name`, its YAML `key`.

</details>

<details>

<summary>Datasources - warehouse connections that own datasets</summary>

A datasource is a configured warehouse connection, such as Snowflake or Postgres, that owns datasets.

Reading datasource configuration YAML can return stored literal credentials to the MCP host; those values enter model context when the host or embedder forwards the tool result. Soda secret references remain `${secret.NAME}` placeholders. Prefer Soda secrets and the Soda Cloud Web UI for future credential changes. This server does not redact configuration-read results or control host or model-provider retention.

* `list_datasources` — List datasources in your Soda Cloud organization. A datasource is a configured connection to a warehouse (e.g. Snowflake, Postgres) that owns datasets.
* `run_datasource_discovery` — Trigger a discovery scan on a datasource. Detects new tables and schemas in the warehouse on-demand (instead of waiting for the scheduled cron).
* `create_datasource` — Create a new datasource from a YAML configuration document. The datasource type (e.g. Snowflake, Postgres) is extracted from the YAML contents.
* `get_datasource` — Get datasource metadata by ID: ID, name, label, type, owner, and creation/update timestamps.
* `get_datasource_configuration_file_contents` — Get the current datasource connection configuration YAML by ID. Use this read before a targeted edit: supplying `configurationFileContents` to `update_datasource` replaces the whole YAML, so preserve every unchanged returned value.
* `update_datasource` — Update a datasource. Top-level fields are PATCH-like: omitted fields keep their existing values. Supplying `configurationFileContents` instead replaces the entire YAML; for a targeted YAML edit, first call `get_datasource_configuration_file_contents` and preserve every unchanged returned value, including existing literals.
* `delete_datasource` — Delete a datasource and all its associated resources (datasets, checks, scans, incidents).
* `list_datasource_roles` — List available datasource roles. Use datasource roles to manage access to individual datasources.
* `create_datasource_role` — Create a custom datasource role.
* `update_datasource_role` — Update the name or permissions of a custom datasource role.
* `delete_datasource_role` — Delete a custom datasource role. This action is permanent and cannot be undone.
* `get_datasource_diagnostics_warehouse` — Get the diagnostics warehouse configuration for a datasource. The diagnostics warehouse collects scan-related data (failed rows, scan results) and forwards it to the customer's warehouse for storage and analysis.
* `update_datasource_diagnostics_warehouse` — Update the diagnostics warehouse configuration for a datasource. The diagnostics warehouse collects scan-related data (failed rows, scan results) and stores it in your warehouse for analysis.
* `list_datasource_responsibilities` — List user and user group permissions assigned to a datasource, and their associated roles.
* `update_datasource_responsibilities` — Update user/group permissions and their associated roles for a datasource.
* `test_datasource_connection` — Trigger an async connection test for a datasource configuration. Use this to validate a YAML + runner combination without creating a datasource, e.g. before calling `create_datasource`.
* `get_datasource_connection_test_status` — Get the status of an async datasource-connection-test operation started via `test_datasource_connection`.
* `onboard_discovered_datasets` — Trigger async onboarding of one or more discovered datasets for a datasource. Onboarding promotes a discovered-but-not-yet-tracked dataset into a regular Soda dataset that can carry contracts, checks, and monitors.
* `get_onboard_discovered_datasets_status` — Get the status of an async dataset-onboarding operation started via `onboard_discovered_datasets`.

</details>

<details>

<summary>Discovered Datasets - tables found by discovery scans</summary>

Discovered datasets have been found by Soda but have not yet been onboarded as regular Soda datasets.

* `list_discovered_datasets` — List datasets Soda has discovered in your data sources. Discovered datasets have been found during a discovery scan but may not yet be onboarded as first-class Soda datasets.

</details>

<details>

<summary>Runners - execute scans and datasource operations</summary>

Self-hosted runners require API key credentials. Create them through the local-only `create_runner` tool, or follow the [Soda Runner deployment guide](broken://pages/Db0zxXDkPOUSKNNBuQXE#deploy-using-cli-only) to obtain them in Soda Cloud.

* `list_runners` — List Soda runners in your organization.
* `create_runner` — **Local only.** Create API key credentials for a new self-hosted Soda runner deployment.
* `get_runner` — Get a runner by ID, including online status, runner type, version information, and last-seen timestamp.
* `delete_runner` — Delete a self-hosted runner by ID.

</details>

<details>

<summary>Secrets - encrypted credentials referenced from datasource YAML</summary>

Datasource YAML can reference stored credentials as `${secret.NAME}`. The Soda Cloud Web UI is the safer place for a person to enter a secret because it keeps plaintext out of chat history and the MCP client/server flow.

`create_secret` and `update_secret` are available only through the local server. They accept plaintext locally and encrypt the value before transmitting it to Soda Cloud. Remote connections can list secret metadata and delete secrets, but cannot create or update their values.

* `list_secrets` — List secrets in your organization.
* `create_secret` — **Local only.** Create a new secret that can be referenced in datasource YAML as `${secret.NAME}`.
* `delete_secret` — Delete an existing secret.
* `update_secret` — **Local only.** Update the value of an existing secret. The secret name cannot be changed.

**Local secrets threat model**

* Plaintext exists in local agent process memory and on the local stdio pipe, but is never logged or sent to Soda Cloud unencrypted.
* The server uses AES-256-GCM and wraps the key with Soda Cloud's RSA public key. It caches that public key for about an hour and refreshes it once after an `encryption_failed` response.
* This is defense in depth over TLS. It does not protect against a compromised MCP client or local agent process.

</details>

<details>

<summary>Incidents - triage state on data quality failures</summary>

* `update_incident` — Update an incident's title, severity, status, description, resolution notes, lead, or linked check results.

</details>

<details>

<summary>Scans - observe and control scan executions</summary>

Contract, datasource, monitor, and data-standard tools trigger scans. These tools inspect or cancel those scan executions.

* `get_scan_status` — Get the current state of a scan. Use this to monitor scan progress during execution.
* `get_scan_logs` — Get log details for a scan in a final state. Use this to investigate issues with scan execution; log availability does not establish successful completion.
* `cancel_scan` — Cancel a running scan.

</details>

<details>

<summary>Users - organization users and groups</summary>

* `list_users` — List users in your Soda Cloud organization. Search by first name, last name, or email.
* `create_user` — Invite one or more users to your Soda Cloud organization.
* `disable_user` — Disable a user in your Soda Cloud organization.
* `list_user_groups` — List user groups in your Soda Cloud organization, including lists of members. Fuzzy search on group name.
* `get_user_group` — Get a single user group by ID, including its members. Returns the full group details (non-paginated).
* `create_user_group` — Create a new user group. Use this tool whenever the user asks to create a user group (typically naming a new group), even when the prompt also lists initial members.
* `update_user_group` — Update an existing user group's members. Use this tool only when the user references an existing group (by ID or by name); for a brand-new group, use `create_user_group` instead.
* `delete_user_group` — Delete a user group by ID. This action is permanent and cannot be undone.

</details>

<details>

<summary>Licensing - SPU consumption</summary>

SPUs (Soda Processing Units) are the consumption unit of SPU-based licensing. Consumption accumulates per day. An organization on any other licensing model is refused with a `400 organisation_not_on_spu_plan` rather than reading zero, so a zero total always means nothing was consumed over the period asked about.

* `get_spu_consumption` — Get the number of SPUs (Soda Processing Units) your organization consumed over a period.

</details>

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Soda AI Agent

Ask questions, get guidance, and let Soda perform actions for you in natural language, without leaving Soda Cloud.

The Soda AI Agent is **the in-product, conversational face of Soda AI**. It lives alongside your work in Soda Cloud and helps **you understand data**, **author data contracts**, and **accelerate onboarding** without replacing human decision-making.

You can request it to **perform actions** on your behalf, get **step-by-step guidance** on platform workflows, or ask it **questions** about Soda, all in **natural language, without leaving the page you're on**. As an agentic assistant within Soda Cloud, **the Soda AI Agent can access the tooling** necessary to perform actions on your behalf, such as Contract Autopilot or Soda MCP.

## Chat window

The Soda AI Agent sits in a chat window that stays at the bottom of every page in Soda Cloud. Type your request in the **Ask Soda AI anything...** field and send it.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FDe4ze9tEMvSRqEE4XnMC%2Fimage.png?alt=media&amp;token=f523b01b-900d-4eb4-b6ad-b37a18443ff6" alt=""><figcaption></figcaption></figure>

### Capabilities

The Soda AI Agent's actions are scoped to the **same operations you can perform** yourself in Soda Cloud, subject to your role and permissions. **The Soda AI Agent can:**

* **Perform** safe, reviewable **actions in Soda Cloud** on your behalf. For example:
  * **Generate data contracts.** Using Contract Autopilot, you can point the Soda AI Agent at your datasets and generate fully populated contracts with recommended checks, drawn from your data's own profile. Data teams can get full coverage in an afternoon, instead of manually creating contracts for weeks.
  * **Edit and iterate over data contracts** in plain English. Add a freshness check, tighten a rule, handle a nested field, all without hand-writing code. Within the chat, you will see exactly what changed, so you keep full agency over the editing process.
* **Guide** **you through platform workflows** (configuring datasets, authoring contracts, investigating incidents)
* **Answer product and docs questions.** Ask what a check, monitor, or incident means, or tell it to make a change. It proposes the action, scoped to what your role already allows, and waits for you to approve.
* **Agentic navigation.** Ask the Soda AI Agent to take you to another place in the Web UI and it will take you there.
* **Context awareness.** The Soda AI Agent knows basic information about the page you are visiting to be better able to answer related questions.

{% hint style="info" %}
Every change is proposed, shown, and approved by a human user. The Soda AI Agent is an assistant and control surface, and is designed to perform **human-in-the-loop processes**.
{% endhint %}

### Conversations and history

Each chat is saved as a **conversation** in the Soda AI Agent chat window. You can resume a previous conversation or start a new one from any page on Soda Cloud.

{% hint style="info" %}
**Soda does not store prompts or conversations.** Your prompts and conversation history are stored in your browser local storage. If you change browsers or open Soda Cloud in a different device, you will not have access to your conversation history.
{% endhint %}

***

## Data usage and privacy

* To answer questions and propose or perform actions, the Soda AI Agent analyzes your **natural-language request**, Soda **documentation content**, basic information about the **page you are on**, and, when needed, **dataset schemas** and **non-identifiable column-level metadata**.
* The Soda AI Agent works within an **authenticated Soda Cloud session** only.
* Every action taken by the Soda AI Agent is subject to your **existing Soda roles and permissions**.
* When the Soda AI Agent uses Contract Autopilot or [Contract Copilot](https://docs.soda.io/soda-documentation/soda-ai/contract-copilot) to carry out a request, those features send the data described on their own pages.
* If your organization brings its own key (BYOK), Soda AI Agent requests go to your own OpenAI project instead of Soda's.

***

## Enable/disable the Soda AI Agent

An organization admin enables or disables the Soda AI Agent for the whole organization.

{% stepper %}
{% step %}
Navigate to **Settings** > **Organization Settings**.
{% endstep %}

{% step %}
Under the **General** tab **> AI** tab, check or uncheck **Enable Soda AI Agent**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FGikGhJF24pt64sbfKc3R%2Fimage.png?alt=media&amp;token=898c9fc3-19da-401b-890d-47ff1b54d28e" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Save** on the top right corner.
{% endstep %}
{% endstepper %}

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Unstructured input

Attach images, PDFs and text files to a Soda AI Agent message so Soda AI can read them alongside your question.

Attach a file to a Soda AI chat message and Soda AI reads it together with your question. Use it for context that does not live in a dataset: a requirements document, a screenshot of a report, a CSV export or a SQL query.

For example, you can:

* Attach a PDF with data requirements and ask Soda AI to turn them into checks on a contract.
* Attach a screenshot of a failing report and ask what could explain the numbers.
* Attach a CSV sample or a SQL query and ask which checks would catch the problems in it.

## Attach a file

You can add files to a message in three ways:

* Click the **paperclip** button in the chat input and pick one or more files.
* Paste a file or an image into the chat input.
* Drag and drop files anywhere onto the Soda AI chat panel. Dropping files on the collapsed chat bar opens the panel and attaches them.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FHXh3D4WpGzoB4v1HFGds%2Fimage.png?alt=media&#x26;token=b8b41733-0e3f-4c04-a842-05e3159dde09" alt="The Soda AI chat bar at the bottom of the page, with the paperclip button next to the send button"><figcaption><p>The paperclip button in the chat input</p></figcaption></figure>

Attached files appear as tiles above the input. Remove a file with the **x** on its tile. You can send a message with files and no text: Soda AI uses the first file name as the conversation title.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FN6SQsNKLegRZzw5azuWr%2Fsoda-ai-attach-composer.png?alt=media" alt="A CSV file attached in the chat input, shown as a tile with its name and size above the typed question"><figcaption><p>An attached file, ready to send with the question</p></figcaption></figure>

After you send the message, the file tile stays above your question in the conversation.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FU09Mj3XRuLSk0QlrN22e%2Fsoda-ai-attach-sent.png?alt=media" alt="A sent chat message with the attached CSV file tile above the question"><figcaption><p>The attached file in the conversation</p></figcaption></figure>

Soda AI answers from the file content. For example, attach a CSV sample and ask for a data contract: Soda AI points out the problems it finds in the rows and proposes contract YAML with a check for each one, ready to copy or publish directly from Soda AI.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F5FNjlrFuce90HppZPbaL%2Fimage.png?alt=media&#x26;token=39e47c65-8f87-4858-9d01-6a79c11dffe4" alt="Soda AI answering a question about an attached CSV file with a proposed data contract in YAML"><figcaption><p>A proposed contract for the attached CSV</p></figcaption></figure>

### Supported files

| Kind      | Formats                                                     |
| --------- | ----------------------------------------------------------- |
| Images    | PNG, JPEG, GIF, WebP                                        |
| Documents | PDF                                                         |
| Text      | TXT, LOG, CSV, TSV, Markdown, YAML, JSON, NDJSON/JSONL, SQL |

Soda checks the content of each file, not only its extension. A file whose content does not match its type, for example a renamed executable, is refused.

## Size limits

| Limit                                | Value                        |
| ------------------------------------ | ---------------------------- |
| Files per message                    | 3                            |
| Size of one image file               | 20 MB before Soda resizes it |
| Size of one image after resizing     | 5 MB                         |
| Size of one PDF                      | 5 MB                         |
| Size of one text file                | 512 KB                       |
| Total size of a message's files      | 8 MB                         |
| Total size of a message's text files | 512 KB                       |

Your browser resizes and re-encodes large images before it sends them. The 5 MB limit and the 8 MB total apply to an image after that step. GIFs are sent as a still image.

PDFs and text files are not changed. Their limits apply to the file as it is.

When a file breaks a limit or its type is not supported, Soda AI does not attach it and tells you which file was refused and why.

### Files in long conversations

A conversation also has a size limit of its own. Some model providers do not store the conversation, so Soda AI sends the whole conversation again with each new message, including every file attached earlier. These providers are Anthropic and Google Gemini, and Portkey with Anthropic or another provider that is not OpenAI behind the gateway.

On these providers, a large file takes up part of the conversation's size limit for the rest of the conversation. When the files of a message would fill the remaining space, Soda AI warns you before you send. When the conversation reaches the limit, the chat asks you to start a new conversation.

On OpenAI and Azure OpenAI, and on Portkey with OpenAI behind the gateway, the provider stores the conversation. Soda AI does not send earlier files to the provider again.

## File handling and privacy

* The file content is sent to the model provider together with your message, so the model can read it. Soda AI can refer back to an attached file in follow-up questions in the same conversation.
* Soda Cloud does not store your files. Your browser keeps each file in the conversation history, on your device, like the rest of your [conversations](https://docs.soda.io/soda-documentation/soda-ai/soda-ai-agent#conversations-and-history). The browser needs the file to send it again on providers that do not store the conversation.
* Soda AI reads only the files you attach. Attach only content you are allowed to share with Soda AI.

{% hint style="warning" %}
Do not attach files that contain personal data, credentials or other sensitive information unless your organization allows sharing them with the model provider.
{% endhint %}

> Learn more about [Soda AI security & privacy measures](https://docs.soda.io/soda-documentation/soda-ai/ai-security-compliance-and-privacy).

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Contract Copilot

Learn more about Contract Copilot, a Soda AI-powered feature to edit data contracts.

{% hint style="success" %}
**Do this with the** [**Soda AI Agent**](https://docs.soda.io/soda-documentation/soda-ai/soda-ai-agent)**.** Describe the change you want in plain English and the Soda AI Agent will update your contract from schema and metadata. You approve the diff.

Read more about [Soda AI](https://docs.soda.io/soda-documentation/soda-ai/overview)'s capabilities.
{% endhint %}

**Contract Copilot** is an AI assistant that helps teams **understand**, **edit and evolve existing data contracts** using natural language. It acts as an intelligent guide throughout the contract lifecycle, making it easier to define expectations, detect gaps, and keep contracts aligned with real-world data behavior, **without ever accessing or processing your raw data**.

It allows users to:

* Add, modify, or remove checks
* Update contracts faster without manually writing checks
* Understand how a contract was changed

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F2W2h7m1jT3JgavGlnYXr%2Fsoda_copilot_long.gif?alt=media&amp;token=c91399d1-a367-47da-85e5-3c922c561287" alt=""><figcaption></figcaption></figure>

#### <i class="fa-camcorder">:camcorder:</i> Watch Contract Copilot in action

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FN7xN0mrOVhgz7vPew2p5%2FGrabaci%C3%B3n%20de%20pantalla%202025-12-15%20174046.mp4?alt=media&token=4dec17b6-abfb-46bf-83e9-f98a2fecdb1b>" %}

Contract Copilot is designed to support both:

* **Business and analytics users**, who need assistance drafting data contracts and establishing data expectations, and
* **Data engineers and platform teams**, who need precise, maintainable, and scalable contracts.

## How does Contract Copilot work?

{% hint style="warning" %}
This feature requires the **Manage contract** permission on the dataset. Learn more about permissions here: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

{% stepper %}
{% step %}
**You provide a natural-language request** (e.g. “Add freshness checks”)
{% endstep %}

{% step %}
Soda retrieves relevant **dataset metadata**, such as:

* Column names and types
* Available check attributes
* Data source type
  {% endstep %}

{% step %}
This metadata, together with the request, is sent to the model
{% endstep %}

{% step %}
**The model proposes contract updates** directly on the existing contract
{% endstep %}

{% step %}
The user reviews and applies the changes
{% endstep %}
{% endstepper %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F2W2h7m1jT3JgavGlnYXr%2Fsoda_copilot_long.gif?alt=media&amp;token=c91399d1-a367-47da-85e5-3c922c561287" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
**Traceability**

Contract Copilot allows you to see what changed directly in the Soda Cloud UI. Requests and outputs can be logged for review and troubleshooting, allowing for cross-team traceability and transparency.
{% endhint %}

***

## Enable/Disable Contract Copilot

Soda Contract Copilot can be configured from the **Organization Settings**.

{% stepper %}
{% step %}
Click on the **profile icon** on the top right corner → Navigate to **Organization Settings.**

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FkZCGJNF6NAp7BrbqXPZe%2Fimage.png?alt=media&amp;token=96c20ef1-6bed-4284-9660-e2f7cbfac5c7" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Under the **Organization** tab **> Soda AI:** Check/Uncheck the "Enable Contract Copilot powered by OpenAI" checkbox to enable/disable the feature.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FNrFNQqrfSN7xVvwbpIRt%2Fimage.png?alt=media&amp;token=d4f37cdb-223e-4982-9cbe-abe1091e90cb" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Save** on the top right corner.
{% endstep %}
{% endstepper %}

***

## Privacy & security

{% columns %}
{% column %} <i class="fa-square-check">:square-check:</i> Contract Copilot uses AI/ML techniques to **help you write data contracts**, analyzing:

* Dataset schemas
* Non-identifiable column-level metadata
* Statistical summaries
* Existing contract definitions and checks
  {% endcolumn %}

{% column %} <i class="fa-xmark">:xmark:</i> Contract Copilot **does not** analyze or inspect:

* Raw table data
* Individual records
* Row-level values
* PII or sensitive fields
  {% endcolumn %}
  {% endcolumns %}

### Confidentiality & access control

* Metadata is retrieved from Soda Cloud on a **need-to-know basis**
* Only users with **permission to edit or propose contract changes** can access Contract Copilot
* All users must authenticate

> Learn more about [Soda's AI security & privacy measures](https://docs.soda.io/soda-documentation/soda-ai/ai-security-compliance-and-privacy).

{% hint style="info" %}
Contract Copilot can generate contracts that include [`column_expression`](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference#column-expression) when detecting:

* Nested fields
* Type mismatches
* Casting requirements
  {% endhint %}

## Limitations

* Copilot does not guarantee correctness of checks
* Users remain responsible for validating contract logic
* Copilot does not operate outside the context of an existing contract

{% hint style="success" %}
Soda Contract Copilot focuses on **assisting decision-making**, not replacing human ownership of contracts. The output is a set of **suggestions, explanations, or guidance** that you can review and apply.
{% endhint %}

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Bring your own key (BYOK)

Learn how to run the Soda AI features on your own account with OpenAI, Anthropic, Google Gemini, Azure OpenAI or Portkey

{% hint style="info" %}
Bring your own key is **only available in organizations fully on Soda v4**.

[Migrate to Soda v4](https://docs.soda.io/soda-documentation/govern/migrate-from-v3-to-v4) to have access to the newest Soda capabilities.
{% endhint %}

Bring your own key (BYOK) runs the Soda AI features on an account that your organization holds with a model provider. Soda supports these provider types:

* OpenAI
* Anthropic
* Google Gemini, through the Gemini API or Vertex AI
* Azure OpenAI (Microsoft Foundry)
* Portkey AI Gateway

With a provider of your own:

* **Requests go to your provider account.** Soda sends the requests of the Soda AI Agent and Contract Autopilot to the account behind your key. Your agreement with the provider applies, together with the retention, privacy and regional settings of that account. The provider bills your organization for the usage.
* **You keep control of the key.** Soda stores a reference to the key and never the key itself. The reference points to an environment variable on your runner or to a Soda Cloud secret. You rotate or revoke the key in the system that holds it.

> Learn what the AI features send to the provider in [AI security, compliance & privacy](https://docs.soda.io/soda-documentation/soda-ai/ai-security-compliance-and-privacy).

## AI configuration

Admins manage providers in the **AI** tab of **Organization settings**.

### Organization settings - AI tab visibility

The **AI** tab shows when BYOK is available to your organization and at least one AI feature is available to it. If the tab is missing, contact <support@soda.io>.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FGCCnH1SC9VGkauiHdk22%2Fimage.png?alt=media&amp;token=62726a0c-31a4-4096-a013-0b7e04ee91af" alt=""><figcaption></figcaption></figure>

The tab has two parts:

* The **Providers** list. It shows each provider with its type, its model, who added it, who made it active, and its status.
* The **Soda AI** box. It holds the switches that turn the AI features on or off.

### Providers

One provider is **Active** at a time. The Soda AI Agent and Contract Autopilot both run on the active provider. Every other provider has the status **Available**.

#### **Soda's built-in provider**

The **Soda (OpenAI)** row is the OpenAI account that Soda operates. It shows only in organizations that may use Soda's key. In those organizations it:

* is the active provider until you make another provider active,
* is included in your Soda subscription,
* cannot be edited or removed, and
* has no settings. Soda chooses the model.

In an organization without the **Soda (OpenAI)** row, the AI features stay off until you add a provider.

#### **Add a provider**

{% stepper %}
{% step %}
Click **Add provider**. The **Provider configuration** panel opens.
{% endstep %}

{% step %}
The first time, read the notice **Before you add your own key** and click **Agree and continue**. Soda records your organization, your user and the time.
{% endstep %}

{% step %}
Choose the provider type.
{% endstep %}

{% step %}
Enter a **Name**. The name identifies the provider in the list, for example `Team OpenAI`.
{% endstep %}

{% step %}
Fill in the fields of the provider type. The sections below list them.
{% endstep %}

{% step %}
Click **Save**.
{% endstep %}
{% endstepper %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FmuxZiz4VbfpXhhRxMcK4%2Fimage.png?alt=media&amp;token=05961465-3689-42f0-94d6-61430a8cd060" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
**A new provider becomes the active provider when you save it.** The Soda AI Agent and Contract Autopilot use it from their next request.
{% endhint %}

Every key field takes a reference, not the key itself. See [API key reference formats](#api-key-reference-formats).

**OpenAI**

* **API key**: a reference to an OpenAI API key.
* **Model**: optional. Contract Autopilot runs this model. Leave it empty to run the runner's default model. The Soda AI Agent ignores this field: on an OpenAI provider it runs the model that Soda chooses.

{% hint style="warning" %}
**The Soda AI Agent does not support OpenAI projects with zero data retention.** It continues a conversation from the responses that OpenAI stores, and zero data retention turns that storage off.
{% endhint %}

**Anthropic**

* **API key**: a reference to an Anthropic API key. Use a key scoped to a workspace. Anthropic rejects keys created at the organization level.
* **Model**: optional. The Soda AI Agent and Contract Autopilot both run this model. Leave it empty to run the default model of each. The hint under the field names the default model of the Soda AI Agent.

Soda sends requests to the Anthropic API directly. A custom base URL, such as a gateway in front of Anthropic, is not supported.

**Google Gemini**

Choose how Soda authenticates in **Authentication**:

* **Vertex AI service account**
  * **Service account key**: paste the JSON key of the service account into a Soda Cloud secret, then reference the secret here. The service account needs the **Vertex AI User** role.
  * **Project ID**: optional. Leave it empty to use the project in the key file.
  * **Location**: **Global**, **EU (multi-region)** or **US (multi-region)**.
* **Gemini API key**
  * **API key**: a reference to a Gemini API key. Use a key from a project with billing enabled. With a free-tier key, Google can use your prompts to improve its products.

For both options:

* **Model**: optional. The Soda AI Agent and Contract Autopilot both run this model. Leave it empty to run the default model of each. The hint under the field names the default model of the Soda AI Agent.

**Azure OpenAI (Microsoft Foundry)**

* **Endpoint**: the endpoint of your Azure OpenAI or Microsoft Foundry resource. Soda accepts `https://<resource>.openai.azure.com`, `https://<resource>.services.ai.azure.com` and `https://<resource>.cognitiveservices.azure.com`, each optionally followed by `/openai/v1`. Other hosts, such as a gateway in front of the resource, are not accepted.
* **Authentication**:
  * **API key**
    * **API key**: paste one of the keys of the resource into a Soda Cloud secret, then reference the secret here.
  * **Entra ID service principal**
    * **Tenant ID**: the directory (tenant) ID, or one of the domains of the tenant, such as `contoso.onmicrosoft.com`.
    * **Client ID**: the application (client) ID of the service principal.
    * **Client secret**: a reference to the client secret of the service principal. The service principal needs the **Cognitive Services OpenAI User** role on the resource.
* **Deployment**: required. Enter the name of the deployment on your Azure resource, not the name of the model. The Soda AI Agent and Contract Autopilot both call this deployment.

Requirements for the Soda AI Agent:

* The deployment must serve a reasoning model. A deployment of a model without reasoning fails.
* If the resource restricts network access, allow Soda's IP addresses on it.
* The resource stores each response, and the Soda AI Agent continues a conversation from the stored responses. Soda has no setting that turns this storage off.

**Portkey AI Gateway**

* **Provider behind the gateway**: the provider behind the slug of your model string in the Portkey model catalog.
  * **OpenAI**: OpenAI itself.
  * **Anthropic**: the Anthropic API itself.
  * **Other (OpenAI-compatible)**: every other provider. This includes Claude on Amazon Bedrock or Vertex AI, and Azure OpenAI.
* **API key**: paste your Portkey API key into a Soda Cloud secret, then reference the secret here. Portkey holds the key of the provider behind it. Soda never receives that key.
* **Model string**: required. Enter the model string from your Portkey model catalog in the form `@provider-slug/model-name`, for example `@openai-prod/gpt-5-mini`. The Soda AI Agent and Contract Autopilot both run this model.

Soda refuses a response that comes from a provider other than the one you chose in **Provider behind the gateway**.

#### **API key reference formats**

A key field accepts one of two reference formats: an environment variable or a Soda Cloud secret. Soda resolves the reference at run time and never writes the key to Soda Cloud's storage. The field refuses any other value, including the key itself.

**1. Environment variable: `${env.NAME}`**

Use the `${env.NAME}` format when the key is set as an environment variable on a self-hosted runner.

```bash
${env.LLM_API_KEY}
```

At run time, the runner reads the environment variable. The key stays in your own infrastructure and never enters Soda Cloud.

Use this format when:

* you already **manage credentials through the environment of your platform**, for example Kubernetes secrets mounted as environment variables, or
* you want the key to **stay outside Soda Cloud**.

{% hint style="info" %}
An environment variable does not work for the Soda AI Agent. The Soda AI Agent runs in Soda Cloud, where you cannot set environment variables. Use a Soda Cloud secret if the provider also runs the Soda AI Agent.
{% endhint %}

**2. Soda Cloud secret: `${secret.NAME}`**

Use the `${secret.NAME}` format to keep the key in Soda Cloud's secrets manager. The secret must exist under **Organization settings → Secrets**. In the form, the **Secret** button on the field inserts the reference.

```
${secret.llm-api-key}
```

At run time, Soda reads the secret from the encrypted secret store of your organization.

Use this format when:

* the provider runs the Soda AI Agent,
* you **do not run a self-hosted runner**, or
* you want to **manage every credential in one place**.

{% hint style="info" %}
**A `${secret.…}` reference links the provider to the secret.** While a provider references a secret, Soda Cloud prevents the deletion of that secret. The deletion dialog lists the providers to update first.
{% endhint %}

#### Manage existing providers

Each provider that you added has a three-dot menu:

* **Make active**: makes the provider the active provider. For an Azure OpenAI or Portkey provider, the panel opens first and asks for the deployment or the model string.
* **Edit**: changes the name, the key reference and the other fields. For the active provider, it also changes the model. The provider type cannot change. To use another type, add a new provider.
* **Remove**: removes the provider after you confirm.

If you remove the active provider, the AI features go back to **Soda (OpenAI)** when your organization has that row. Without that row, the AI features have no provider until you make another provider active.

### What runs on the active provider

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FtG8PlHICzybGcb4dsP4o%2Fimage.png?alt=media&amp;token=9f53f321-3d9b-43ae-85fa-8f350faa449a" alt=""><figcaption></figcaption></figure>

The active provider runs two capabilities:

* **Soda AI** runs the **Soda AI Agent**, the chat in Soda Cloud.
* **Runner** runs **Contract Autopilot**. A runner makes these requests: the Soda-hosted runner or a self-hosted runner.

The model that each capability runs depends on the provider type:

| Provider type      | Field in the form          | Soda AI Agent                                      | Contract Autopilot                               |
| ------------------ | -------------------------- | -------------------------------------------------- | ------------------------------------------------ |
| OpenAI             | **Model**, optional        | The model that Soda chooses. The field is ignored. | The model in the field, or the runner's default. |
| Anthropic          | **Model**, optional        | The model in the field, or Soda AI's default.      | The model in the field, or the runner's default. |
| Google Gemini      | **Model**, optional        | The model in the field, or Soda AI's default.      | The model in the field, or the runner's default. |
| Azure OpenAI       | **Deployment**, required   | The deployment in the field.                       | The deployment in the field.                     |
| Portkey AI Gateway | **Model string**, required | The model string in the field.                     | The model string in the field.                   |

The form has no field for the reasoning effort.

#### Runner support

Every runner supports an OpenAI provider. An Anthropic, Google Gemini, Azure OpenAI or Portkey provider needs a runner that supports that provider type. Contract Autopilot skips a dataset when its runner does not support the active provider. If that happens on a self-hosted runner, upgrade the runner.

#### Make a provider active

{% stepper %}
{% step %}
In the **Providers** list, open the three-dot menu of the provider.
{% endstep %}

{% step %}
Click **Make active**.
{% endstep %}

{% step %}
Confirm. For an Azure OpenAI or Portkey provider, the panel opens instead: enter the deployment or the model string and click **Save**.
{% endstep %}
{% endstepper %}

The Soda AI Agent and Contract Autopilot use the new active provider from their next request.

#### Switch back to the built-in provider

If your organization has the **Soda (OpenAI)** row:

{% stepper %}
{% step %}
Open the three-dot menu of **Soda (OpenAI)**.
{% endstep %}

{% step %}
Click **Make active**.
{% endstep %}

{% step %}
Confirm.
{% endstep %}
{% endstepper %}

#### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

Setup:

| Problem                                                                 | What to do                                                                                                                                                                                                      |
| ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The **AI** tab is missing from Organization settings.                   | BYOK or the AI features are not available to your organization. Contact <support@soda.io>.                                                                                                                      |
| A key field shows "Must be a ${secret.NAME} or ${env.NAME} reference."  | Enter a `${env.NAME}` or `${secret.NAME}` reference. The field refuses a key pasted directly and every other format.                                                                                            |
| The **Endpoint** field of an Azure OpenAI provider refuses the URL.     | Enter the URL of the resource on `openai.azure.com`, `services.ai.azure.com` or `cognitiveservices.azure.com`, over `https`, with no port and no path other than `/openai/v1`.                                  |
| A `${secret.NAME}` reference does not resolve at run time.              | Make sure the secret exists under **Organization settings → Secrets** and that its name matches the reference exactly. Secret names are case-sensitive.                                                         |
| A `${env.NAME}` reference does not resolve at run time.                 | Make sure the environment variable is set on the self-hosted runner and that the name matches exactly. Environment variable names are case-sensitive. The Soda AI Agent cannot read your environment variables. |
| Contract Autopilot skips datasets after you change the active provider. | The runner of the data source does not support the provider type. Upgrade the self-hosted runner, or make a provider active that the runner supports.                                                           |

Messages in the Soda AI Agent chat. Each message can also quote the provider's own error text.

| Message                                                                                                                                 | What to do                                                                                                                                                                         |
| --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "\<Provider> rejected your organization's API key."                                                                                     | The provider refused the key. Check that the key is valid and not revoked, then update the secret or the key reference.                                                            |
| "Azure OpenAI refused your organization's API key."                                                                                     | Check that the key belongs to the resource at the configured endpoint. If the resource restricts network access, allow Soda's IP addresses.                                        |
| "Azure OpenAI refused your organization's Entra ID service principal."                                                                  | Check the tenant ID, the client ID and the client secret. Check that the service principal has the **Cognitive Services OpenAI User** role on the resource, and the network rules. |
| "The model configured for your organization is not available with its AI provider API key."                                             | The provider does not know the model, or your account cannot use it. Edit the provider and correct the **Model**.                                                                  |
| "The deployment configured for your organization was not found on its Azure OpenAI resource, or it does not serve a reasoning model."   | Edit the provider. Check the **Deployment** name and the **Endpoint**, and use a deployment of a reasoning model.                                                                  |
| "Portkey could not serve the model configured for your organization"                                                                    | Check the **Model string** and **Provider behind the gateway**. The slug must exist in your Portkey catalog, and the provider behind it must be the one you chose.                 |
| "Your organization's \<Provider> account has no remaining quota."                                                                       | The quota or the budget of the provider account is used up. Raise it in the provider account.                                                                                      |
| "We couldn't decrypt your organization's stored \<Provider> key." or "We couldn't securely access your organization's \<Provider> key." | Soda could not read the key. Check that the secret still exists, then contact <support@soda.io>.                                                                                   |
| "This conversation has reached its length limit. Start a new conversation to continue."                                                 | Start a new conversation. See [Unstructured input](https://docs.soda.io/soda-documentation/soda-ai/soda-ai-agent/unstructured-input#size-limits) for the effect of attached files. |

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# AI security, compliance & privacy

Learn what data is sent to a model provider by Soda AI features, what Soda stores, and what your organization remains responsible for.

Each Soda AI feature sends requests to a large language model (LLM) hosted by a **third-party provider**. The built-in Soda provider is **OpenAI**. With bring your own key (BYOK), the provider is the one your organization configured: OpenAI, Anthropic, Google Gemini, Azure OpenAI or Portkey. The provider's terms that apply depend on which of the two you use. This page covers:

* what the requests contain
* what Soda and the provider store
* which settings admins control
* what your organization remains responsible for when it relies on AI output

<h4 align="center">AI features covered in this page</h4>

<p align="center"><a href="https://docs.soda.io/soda-documentation/soda-ai/soda-ai-agent" class="button primary medium">Soda AI Agent</a><a href="https://docs.soda.io/soda-documentation/soda-ai/contract-copilot" class="button primary medium">Contract Copilot</a></p>

{% hint style="info" %}
[Soda MCP](https://docs.soda.io/soda-documentation/soda-ai/soda-mcp) and the root cause analysis (RCA) agent run inside AI tools **you choose and configure**, such as Claude Code. Soda returns data to your agent, and **your agent's provider and your agreement with that provider govern what happens to it next**.
{% endhint %}

> Learn more about [Soda AI](https://docs.soda.io/soda-documentation/soda-ai/overview) and its features.

***

## Model providers

Every AI feature runs through a provider. Admins manage providers in the **AI** tab of **Organization Settings**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FoUGKJWaQiIGTRzkBd48P%2Fimage.png?alt=media&amp;token=5912013a-a9cb-4de6-882f-a8406475970e" alt=""><figcaption></figcaption></figure>

<h4 align="center">Which account receives the requests?</h4>

<table data-card-size="large" data-view="cards"><thead><tr><th align="center"></th><th></th><th></th></tr></thead><tbody><tr><td align="center"><h4>Bring your own key (BYOK)</h4></td><td><p>With <a href="https://docs.soda.io/soda-documentation/soda-ai/bring-your-own-key-byok">bring your own key (BYOK)</a>, requests go to <strong>your organization's own account with the provider you configured</strong>, under your agreement with that provider.</p><div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p><strong>Enterprise plans use BYOK only.</strong></p></div></td><td><ul><li>The retention, privacy, and regional settings of your own provider account apply.</li><li>Soda stores a reference to where the API key lives and never the key itself.</li></ul></td></tr><tr><td align="center"><h4>Soda-managed AI</h4></td><td>With the built-in Soda provider, requests go to an OpenAI account that Soda operates, under Soda's agreement with OpenAI.</td><td><ul><li>Usage is included in your Soda license.</li><li>Soda chooses the models.</li></ul></td></tr></tbody></table>

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FCPk4HSG5z2Zz9fJOPf7J%2FSoda%20AI%20vs%20BYOK.png?alt=media&amp;token=39efe2a2-19e6-4c5b-8719-ec20f6d22847" alt="" width="563"><figcaption><p>Soda-managed AI provider vs Bring your own key (BYOK)</p></figcaption></figure>

> Using your own supported provider account, agreement, and data-handling settings, Soda AI can fit easily within **your organization's existing security and governance** arrangements.
>
> Set up your own provider by [bringing your own key (BYOK)](https://docs.soda.io/soda-documentation/soda-ai/bring-your-own-key-byok).

### Data sent to the model provider

**Row-level data**, such as individual records or personal data in your datasets, **is not sent** unless an admin enables it for Contract Autopilot.

| Feature                                                                                  | Data sent to the model provider                                                                                                                                                                                                                                                                          |
| ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [**Soda AI Agent**](https://docs.soda.io/soda-documentation/soda-ai/soda-ai-agent)       | <ul><li>Your messages</li><li>Files you attach to a message: images, PDFs and text files</li><li>Soda documentation content</li><li>Basic information about the Soda Cloud page you have open</li><li>Dataset schemas</li><li>Non-identifiable column-level metadata when a request needs them</li></ul> |
| [**Contract Copilot**](https://docs.soda.io/soda-documentation/soda-ai/contract-copilot) | <ul><li>Your request</li><li>Column names and data types</li><li>Available check attributes</li><li>The data source type</li><li>The existing contract</li></ul>                                                                                                                                         |
| **Contract Autopilot**                                                                   | <ul><li>Computed profiling metrics</li><li>The data source type</li><li>The dataset schema</li><li>A sample of up to 100 rows when <strong>Send row-level data to Soda</strong> is enabled</li></ul>                                                                                                     |

* A contract you generate from the chat runs through Contract Autopilot, so the Contract Autopilot row applies to that request.
* Contract Autopilot computes its profiling metrics from a sample of about 10,000 rows. **Only the resulting metrics go to the model provider**, not the sample.
* Failed rows stored in Diagnostics Warehouse stay in your own data warehouse and **are not available to any AI feature**.

> Learn more about [Diagnostics Warehouse](https://docs.soda.io/soda-documentation/understand/diagnostics-warehouse).

## Storage and retention

**Soda does not store your prompts or chat conversations on its servers**. The chat keeps the conversation history in **your browser**, on your device. The browser stores it in IndexedDB, or in local storage when IndexedDB is not available. The files you attach are kept with the history in IndexedDB. In local storage, only the name of a file and a small preview are kept.

The browser keeps up to 50 conversations and drops a conversation 30 days after its last message. The history does not carry over to another browser or device, clearing the browser's site data deletes it, and anyone using the same browser profile can read it.

What the model provider stores depends on the provider:

* **OpenAI and Azure OpenAI.** The chat and Contract Copilot continue a conversation by referring back to responses stored with the provider. Requests and responses stay in the provider account for that provider's retention period. The same applies to Portkey with OpenAI behind the gateway.
* **Anthropic and Google Gemini.** The provider does not store the conversation for the chat. The browser sends the whole conversation again with each message, including the files attached earlier, and Soda passes it to the provider. The same applies to Portkey with Anthropic or another provider behind the gateway.

With the built-in Soda provider, the provider account is Soda's OpenAI project, covered by [OpenAI's API data policies](https://platform.openai.com/docs/guides/your-data). With BYOK, it is your own provider account, under your settings and your agreement with the provider.

{% hint style="warning" %}
**Soda AI features do not support OpenAI projects with zero data retention**. Zero data retention disables the stored responses that the chat and Contract Copilot depend on.

On Azure OpenAI, the resource stores each response of the chat. Soda has no setting that turns this storage off.
{% endhint %}

## Model training

**Soda does not use your data, metadata, prompts, or results to train or fine-tune any AI model**. With the built-in Soda provider, OpenAI does not train its models on API request data, and Soda's OpenAI project has input and output sharing with OpenAI turned off. With BYOK, your agreement with your provider and the settings of your provider account decide whether the provider can use your requests.

Soda improves its AI features through development work that does not use your prompts. In some AI features, you can rate a response, and Soda records the rating in its product analytics.

## Access and human oversight

**AI features work only inside an authenticated Soda Cloud session and act with your permissions**. They cannot do anything in Soda Cloud that you could not do yourself.

Contract Autopilot publishes each generated contract as soon as its scan succeeds, with no separate approval step. Generated contracts carry a purple checkmark until someone edits them. Review them before you rely on them, especially when a contract gates a pipeline.

## Admin controls

Admins with the **Manage Organization Settings** global role control AI features from **Organization Settings**:

* In the **AI** tab, under **Soda AI**, enable or disable each AI feature. In an organization without the **AI** tab, these switches are in the **Organization** tab.
* For Contract Autopilot, enable or disable **Send row-level data to Soda**, which controls the 100-row sample.
* In the **AI** tab, add your own provider and make it the active provider.

> Contact <support@soda.io> to request access to an AI feature or setting that does not appear in your organization.

## Terms and compliance

Your agreement with Soda governs your use of Soda AI features, and nothing on this page changes that agreement. Whether AI features are available to your organization can depend on the AI terms in that agreement.

AI processing within the Soda's services detailed on this page runs under the same security controls as the rest of Soda Cloud and is covered by **Soda's SOC 2 Type 2 compliance**. Data is encrypted in transit and at rest.

> Read the [Soda Privacy Policy](https://www.soda.io/privacy-policy).

{% hint style="info" %}
**Model providers operate under their own separate security policies**. Please note **users are responsible for their arrangements with the model provider**, including the provider agreement, the data processing agreement, appropriate account settings, and provider charges.

For the built-in Soda provider, go to [OpenAI's security information](https://openai.com/security-and-privacy/). With BYOK, refer to the security information of your own provider.
{% endhint %}

## Risks and your responsibilities

**Language models are probabilistic**, so output varies between requests and can contain errors or omissions. A generated check can look plausible and still be wrong for your data. **Your organization remains responsible for checking that results suit its data and intended use**, including results used in automated workflows.

* **Keep meaningful human oversight** of any decision with legal or similarly significant effects.
* **Do not enter personal data**, **credentials**, or other **confidential information** in a prompt. Your messages and the files you attach go to the model provider.
* Soda monitors its AI features for **availability** and **technical errors**. It does not check whether each individual answer or suggestion is correct.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Deployment options

Soda offers flexible deployment options to suit your team’s infrastructure, scale, and security needs. Whether you want to embed Soda directly into your pipelines, use a centrally managed deployment, or rely on Soda’s fully-hosted solution, there’s an option for you.

This guide provides an overview of the three main deployment options: **Soda Python Libraries**, **Soda-hosted Soda Runner, and Self-hosted Soda Runner**, to help you choose the right setup for your organization.

***

## Overview of Deployment Options

<table data-full-width="true"><thead><tr><th width="112">Deployment Model</th><th width="148">Description</th><th width="138">Ideal For</th><th>Key Features</th><th>Considerations</th><th>Plans</th></tr></thead><tbody><tr><td><strong>Soda-hosted Soda Runner</strong></td><td>Fully-managed Soda Runner, hosted by Soda.</td><td>Teams seeking a simple, managed solution for data quality.</td><td><ul><li>Centralized data source access</li><li>No setup required</li><li>Observability features enabled</li></ul><p>Enables users to create, test, execute, and schedule contracts and checks directly from the Soda Cloud UI.</p></td><td><strong>Required for observability features</strong>.<br><br>Cannot scan in-memory sources like Spark or DataFrames.</td><td>Available for Free, Team and Enterprise Plans.</td></tr><tr><td><strong>Self-hosted Soda Runner</strong></td><td>Same as Soda-hosted Soda Runner, but deployed and managed in your own Kubernetes environment.</td><td>Teams needing full control over infrastructure and deployment.</td><td><p>Similar to Soda-hosted Runner, but deployed within the customer’s environment; data stays within your network.<br></p><ul><li>Full control over deployment</li><li>Integration with secret managers</li><li>Customization to meet your organization’s specific requirements</li></ul></td><td><strong>Required for observability features</strong>.<br><br>Cannot scan in-memory sources like Spark or DataFrames.<br>Kubernetes expertise required.</td><td>Available for Enterprise Plan. Contact us at <a href="mailto:support@soda.io">support@soda.io</a></td></tr><tr><td><strong>Soda Python Libraries</strong></td><td>Open-source Python library (with commercial extensions) for programmatic configuration and enforcement of data contracts in your pipelines.</td><td>Data engineers integrating Soda into custom workflows.</td><td><ul><li>Full control over orchestration</li><li>In-memory data support</li><li>Contract verification</li></ul></td><td><strong>No observability features</strong>.<br><br>Required for in-memory sources (e.g., Spark, DataFrames).<br><br>Data source connections managed at the environment level.</td><td>Open Source. Available for Free, Team and Enterprise Plans.</td></tr></tbody></table>

## Deployment options in detail

### Soda Runner

#### Soda-hosted Runner

The Soda-hosted Runner is a fully-managed deployment of the Soda Runner, hosted by Soda in our infrastructure. It allows you to connect to your data sources and manage data quality directly from the Soda Cloud UI without any infrastructure setup on your end. You need only whitelist the IP address of the Soda-hosted runner so that it can connect to your data.

Key points:

* **No setup or management required**. Soda handles deployment and scaling.
* **Data source connections are centralized** in Soda Cloud, and users can leverage the Soda Runner to execute scans across those data sources.
* Enable **observability features** in Soda Cloud, such as profiling, metric monitoring, and anomaly detection.
* Enables users to create, test, execute, and schedule **contracts** and checks directly from the **Soda Cloud UI**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F6LjauzTHCS6qj3IUfV4f%2Fwith-managed-agent.png?alt=media&amp;token=a244e53d-97cd-45dd-ac20-c7e4baec9b80" alt=""><figcaption></figcaption></figure>

***

> Onboard your datasets in Soda Cloud with Soda-hosted runner: [Via Soda Cloud](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud)

***

#### Self-hosted Runner

The Self-hosted Runner offers the same capabilities as the Soda-hosted Runner, but it is deployed and managed by your team within your own Kubernetes environment (e.g., AWS, GCP, Azure). This model provides full control over deployment, infrastructure, and security, while enabling the same centralized data source access and Soda Cloud integration for scans, contract execution, and observability features.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FbzBmprHresxWJ2WAn1IW%2Fwith-agent.png?alt=media&amp;token=5ac5b22b-7199-40f7-bc64-821d0c451203" alt=""><figcaption></figcaption></figure>

> Learn how to deploy the Self-hosted Soda Runner: [Broken mention](broken://pages/DVzqeL43wc3c9AsZ4TFi).

> Onboard your datasets in Soda Cloud with self-hosted runner: [Via Soda Cloud](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud).

### Soda Python Libraries

Soda Core is an **open-source Python library and CLI** that allows you to **embed Soda directly in your data pipelines**. You can orchestrate scans using your preferred orchestration tools or pipelines, and execute them within your own infrastructure. Additional commercial extensions are available via extensions packages, such as `soda-groupby` , `soda-reconciliation`, etc.

> See detailed installation instructions here: [Install Soda Python Libraries](https://docs.soda.io/soda-documentation/connect/deployment-options/soda-python-libraries)

Key points:

* Ideal for teams who want **full control** over scan orchestration and execution.
* Data source connections are configured and managed **at the environment level**.
* Required for working with **in-memory data sources** like Spark and Pandas DataFrames.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FPcT7jbdMcuhbPa9Ltooy%2Fwith-library.png?alt=media&amp;token=889757cb-a054-42b9-be5a-d517b7cff7f3" alt=""><figcaption></figcaption></figure>

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Soda Python Libraries

This page describes how to install the Soda Python packages, which are required for running Soda scans via the CLI or Python API.

## Installation

#### Requirements

To use Soda, you must have installed the following on your system.

* **Python 3.10, 3.11, or 3.12**.\
  To check your existing version, use the CLI command: `python --version` or `python3 --version`. If you have not already installed Python, consider using `pyenv` to manage multiple versions of Python in your environment.

{% hint style="info" %}
While Python 3.12 is the highest officially supported version, there are no known constraints preventing use of Python 3.13+.
{% endhint %}

* **Pip 21.0 or greater.**\
  To check your existing version, use the CLI command: `pip --version`
* **A Soda Cloud account**; see how to [sign up](https://docs.soda.io/soda-documentation/get-started/quickstart#sign-up).

Best practice dictates that you **install the Soda CLI using a virtual environment**. If you haven't yet, in your command-line interface tool, create a virtual environment in the `.venv` directory using the commands below.

Soda recommends using [`uv`](https://github.com/astral-sh/uv):

```bash
uv venv .venv
source .venv/bin/activate
```

If you prefer the standard Python tooling (not using `uv`), depending on your version of Python, you may need to replace `python` with `python3` in the first command (or with the Python version you want your virtual environment to run, e.g. `py -3.11`):

```sh
python -m venv .venv
source .venv/bin/activate
```

#### Choose an installation flow

Before you install the Soda CLI, decide which installation flow applies to your environment and license type. The two flows available serve different purposes:

<table><thead><tr><th width="215.00006103515625">Use Case</th><th width="123.199951171875">Installation Flow</th><th>Description</th></tr></thead><tbody><tr><td>Executing data contracts with basic data quality checks on enterprise data sources.</td><td><a href="#public-pypi-installation-flow"><strong>Public PyPI</strong></a></td><td><p>Use this installation method if you’re just getting started.</p><p>The Public PyPI index hosts Soda Core packages for all supported data sources.</p></td></tr><tr><td>Same as above, plus: group by checks, contract scopes, reconciliation checks, migrating checks from v3 to v4, running checks on Oracle data, and capturing failed rows with the Diagnostics Warehouse.</td><td><a href="#private-pypi-installation-flow"><strong>Private PyPI</strong></a></td><td>Private PyPI repositories are region-specific and require authentication using your API key credentials. This method ensures secure access to licensed components, enterprise-only extensions, and region-compliant hosting.</td></tr></tbody></table>

{% hint style="info" %}
Different installations will support **different packages**. Learn more about which packages are supported in [public](#supported-packages) and [private](#supported-packages2) PyPI.
{% endhint %}

{% hint style="success" %}
**How to differentiate between free open-source Soda, and paid licensed Soda?**

**Soda V3**: package names included `core` if the package was free open-source. E.g.:

* `soda-core-postgres` (free open-source)
* `soda-postgres` (paid licensed Soda).

**Soda V4**: no differentiation using `core` in package names. Differentiation will be **based on the installation flows** listed above.
{% endhint %}

***

### Public PyPI installation flow

To use the open source Soda Core python packages, you must install them from the public Soda PyPi registry: `https://pypi.cloud.soda.io/simple` .

1. **Install the Soda Core package** for your data source. This gives you access to all the basic CLI functionality for working with contracts.

```bash
uv pip install -i https://pypi.cloud.soda.io/simple --pre -U "soda-postgres>4"
```

Replace `soda-postgres` with the appropriate package for your data source. See the [Data source reference for Soda Core](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core) for supported packages and configurations.

<i class="fa-octagon-check">:octagon-check:</i> Now you can connect to Soda Cloud.

#### Supported packages

* `soda-core`: "umbrella" package (does not include Diagnostics Warehouse)
* Data-source-specific packages: naming pattern is `soda-<datasource>` (e.g. `soda-postgres`, `soda-bigquery`, `soda-sparkdf`, etc.)

***

### Private PyPI installation flow

If you wish to use commercial extensions to the Soda Core python package, you must install them from one of the private Soda PyPi registries below. The private PyPI installation process adds an authentication layer and region-based repositories for license-based access control of **Team** and **Enterprise** customers.

> To get access to the private PyPi registries, Soda Cloud, Diagnostics Warehouse, and more Soda licensed features, contact us at <support@soda.io>.

{% stepper %}
{% step %}
\[If not using `uv`] **Upgrade `pip`** inside your new virtual environment.

```sh
pip install --upgrade pip
```

{% endstep %}

{% step %}
**Choose the correct repository** based on your license and region.

<table><thead><tr><th width="197.59999084472656">License</th><th width="100">Soda Region</th><th>Repository URL</th></tr></thead><tbody><tr><td>Team<sup>1</sup></td><td>EU</td><td><a href="http://team.pypi.cloud.soda.io/">team.pypi.cloud.soda.io</a></td></tr><tr><td>Team</td><td>US</td><td><a href="http://team.pypi.us.soda.io/">team.pypi.us.soda.io</a></td></tr><tr><td>Enterprise<sup>2</sup></td><td>EU</td><td><a href="http://enterprise.pypi.cloud.soda.io/">enterprise.pypi.cloud.soda.io</a></td></tr><tr><td>Enterprise</td><td>US</td><td><a href="http://enterprise.pypi.us.soda.io/">enterprise.pypi.us.soda.io</a></td></tr><tr><td></td><td></td><td></td></tr></tbody></table>

<sup>1</sup> Team: Any license except "Trial" or "Enterprise" (see below)\ <sup>2</sup> Enterprise: one of `enterprise` , `enterprise_user_based` , `dataset_standard` , `premier` licenses.
{% endstep %}

{% step %}
**Set your credentials**.

```bash
export SODA_API_KEY_ID="your_key_id"
export SODA_API_KEY_SECRET="your_key_secret"
```

> See how to generate your own [API key values](https://docs.soda.io/apis/generate-api-keys).
> {% endstep %}

{% step %}
Based on your license and region, **choose and execute one of the following commands**, **replacing** `soda>=4` with the package that you need to install.

{% tabs %}
{% tab title="Team EU" %}
{% code overflow="wrap" %}

```bash
uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@team.pypi.cloud.soda.io" --extra-index-url=https://pypi.cloud.soda.io
```

{% endcode %}
{% endtab %}

{% tab title="Team US" %}
{% code overflow="wrap" %}

```bash
uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@team.pypi.us.soda.io" --extra-index-url=https://pypi.cloud.soda.io
```

{% endcode %}
{% endtab %}

{% tab title="Enterprise EU" %}
{% code overflow="wrap" %}

```bash
uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.cloud.soda.io" --extra-index-url=https://pypi.cloud.soda.io
```

{% endcode %}
{% endtab %}

{% tab title="Enterprise US" %}
{% code overflow="wrap" %}

```bash
uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.us.soda.io" --extra-index-url=https://pypi.cloud.soda.io
```

{% endcode %}
{% endtab %}
{% endtabs %}

* `soda`: required for the contract generator (includes Diagnostics Warehouse)

  <pre class="language-sh" data-overflow="wrap"><code class="lang-sh">uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.soda.io" --extra-index-url=https://pypi.cloud.soda.io
  </code></pre>
* `soda-groupby`
* `soda-migration`
* `soda-reconciliation`
* `soda-scopes`
* `soda-oracle`
* `soda-mysql`
  {% endstep %}
  {% endstepper %}

## Available extensions

The extensions you need to install depend on the **features** you want to use and your Soda subscription plan. The table below lists the supported features, the **required extension** for each one, and which plans include access.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Onboard data sources & datasets

Before you can monitor, test, or enforce data contracts in Soda, you need to **connect your data source and onboard your datasets**.

**Onboarding** is the process of:

* **Connecting Soda to your data source** (e.g., Snowflake, BigQuery, Databricks, PostgreSQL, etc.)
* **Discovering** available datasets (tables or views)
* **Selecting** which datasets to monitor and/or validate
* **Configuring** observability, testing, and scheduling behavior

How you onboard datasets depends on how you deploy Soda.

***

### Choose your onboarding path

Soda supports two main ways to onboard datasets:

{% columns %}
{% column %} <a href="https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud" class="button primary">Via Soda Cloud</a>

This approach is ideal if you want:

* No-code dataset onboarding
* Automated dataset discovery
* Built-in metric monitoring
* Data contracts created in the UI
* Centralized scheduling and alerting
  {% endcolumn %}

{% column %} <a href="https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/via-soda-core" class="button primary">Via Soda Core</a>

This approach is ideal if you want:

* Full control in code
* Git-based workflows
* CI/CD integration
* Contract verification inside pipelines
* Support for in-memory datasets
  {% endcolumn %}
  {% endcolumns %}

You can also combine both approaches: use Soda Cloud for **centralized observability and governance**, and Soda Core for **pipeline-level validation**.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Via Soda Cloud

Connect a data source to Soda Cloud and onboard its datasets, step by step, from the connection form to monitoring and profiling.

## Onboard a new data source

Before you can define contracts, you need to **connect Soda Cloud to your data source**. This allows Soda to access your datasets for profiling, metric monitoring, and contract verification.

Use read-only access for your data source by default. Before adding a data source to Soda Cloud, you are responsible for configuring the database role and permissions that Soda will use. The exact permissions depend on your data source and setup.

Diagnostics Warehouse is an exception: grant write access only to its target location. See the [required permissions for Diagnostics Warehouse](broken://pages/dWoYeagXP1e77Kf8TS5K#diagnostics-warehouse-required-permissions).

First, **navigate to the Data Sources page** from the left navigation bar in Soda Cloud. On this page, you'll see a list of connected sources and a **New Data Source** button.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FuXP2q7Clh09MekKlMgms%2Fdata_sources.png?alt=media&amp;token=7f39a71a-8d98-44db-b4a4-2d89d8fd8b4e" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
You need the "**Manage data sources**" global permission to add a new data source.

Learn more about [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

{% stepper %}
{% step %}

### Establish connection

{% stepper %}
{% step %}
On the **Data Sources** page, click on **New Data Source**

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F3HwvS2BxSrTjaIKQgzRH%2Fnew_data_source_button.png?alt=media&amp;token=fd428309-4a52-4778-9b10-ad0d8509c625" alt="" width="485"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Select your data source provider

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FbkMDXyt4z5CZZwFKELTw%2Fselect_data_source.png?alt=media&amp;token=8d3a1e88-efca-4038-a15c-8f96150f58c8" alt="" width="375"><figcaption></figcaption></figure>

After selecting a source, you'll be presented with a configuration form.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F7dbs7FW3k060iGov7INM%2Fdata_source_configuration_form.png?alt=media&amp;token=b667a49e-267f-4ba4-aea0-af518df01a7a" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Name your data source** under Data Source Identifier

Enter a friendly, unique label. A unique identifier will be automatically generated from this label. This becomes the immutable ID of the data source and can also be used to reference the same connection in Soda Core.

The label must be between 2 and 45 characters.
{% endstep %}

{% step %}
**Set a default runner**

You’ll be asked to select a runner. This is the component that connects to your data source and runs scans.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FJZ7eRfHk0s87k7WthASb%2Fimage.png?alt=media&amp;token=1d300049-5681-4830-870c-a6bb8fd955e4" alt="" width="563"><figcaption></figcaption></figure>

You can choose from:

* **Soda-hosted runner** – Quickest option, fully managed by Soda (recommended for getting started)
* **Self-hosted runner** – For custom or secure deployments where you manage the runner yourself

> Learn more about deployment options: [Deployment options](https://docs.soda.io/soda-documentation/connect/deployment-options)
> {% endstep %}

{% step %}
\[Optional] **Set a default schedule** for contracts

Toggle on **Default schedule** to pre-fill contract schedules for this data source. When off, contracts created against this data source default to **midnight UTC**.

{% hint style="info" %}
**Existing contracts are not affected.** Contracts keep their own schedule once saved, so changes to the Default schedule only apply to contracts that have not been configured yet.
{% endhint %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F4zntq79qlR4ry5jCYfkS%2Fdefault_schedule.png?alt=media&amp;token=ed407708-a7ad-436d-81ef-429f27da2ba2" alt=""><figcaption></figcaption></figure>

The default schedule is used by:

<table data-header-hidden><thead><tr><th width="186"></th><th></th></tr></thead><tbody><tr><td><strong>Contract Editor</strong></td><td>When the default schedule is enabled, the time zone and cron pre-fill from the Default schedule.</td></tr><tr><td><strong>Contract Autopilot</strong></td><td><ul><li>During dataset onboarding: when the default schedule is enabled, the <strong>Add schedule to generated contracts</strong> section pre-fills with the Default schedule.</li><li>When generating data contracts in bulk: you can choose <strong>Use the data source's default schedule</strong> so each contract picks up its own data source's default, or <strong>Manual schedule</strong> to apply one schedule to all selected datasets.</li></ul></td></tr></tbody></table>
{% endstep %}

{% step %}
Secure your credentials with secrets

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FXDKAvHlu2sNE021ncpwi%2Fimage.png?alt=media&amp;token=358c0102-dd32-4773-b667-e89c1276b544" alt="" width="375"><figcaption></figcaption></figure>

You’ll need to fill in the connection details. Soda uses the **official Python packages** for each supported data source, which means you can define any properties required by those libraries, flexibly and reliably.

This includes common fields like host, port, database name, username, and more, depending on the data source.

#### **Using secrets for sensitive credentials**

For sensitive values such as passwords, tokens, or keys, you should use **Soda Secrets** instead of entering them directly in the configuration.

* Secrets are **encrypted and securely stored** in Soda Cloud.
* They can be safely referenced in your data source configuration without exposing them in plain text.

To add secrets:

{% stepper %}
{% step %}
Navigate to the **Data Sources** tab in the top navigation.
{% endstep %}

{% step %}
Click the **Secrets** tab.
{% endstep %}

{% step %}
Define key-value pairs for your sensitive credentials.
{% endstep %}
{% endstepper %}

You can then reference a secret in your data source configuration using this syntax:

```javascript
${secret.SECRET_NAME}
```

This ensures your sensitive values stay secure while still being accessible to the runner at runtime.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FEVYC4hZsDQlba964H0fa%2Fsecrets_page.png?alt=media&amp;token=ac6f4c24-c9a5-4d74-b0fe-a46433befdb4" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Test** and **Connect**

Once the form is complete:

* Click **Test Connection** to validate that Soda can successfully connect to your data source.
* If the test passes, click **Connect** to finalize the setup.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FpI0zWMMWR2ErRoiqswbp%2Fconnected_successfully.png?alt=media&amp;token=32bfa228-1580-4ce6-b94d-da8c6cd4c534" alt="" width="350"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}
{% endstep %}

{% step %}

### \[Optional] Enable Diagnostics Warehouse

**Enable** [**Diagnostics Warehouse**](broken://pages/UZCAHd8ZBgl01336NkZa) to store record-level diagnostics, such as failed rows, in a warehouse you own.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FTgRDjj6kNdGfTeiPRxjd%2Fenable_diagnostics_warehouse.avif?alt=media&amp;token=0715f217-c21e-4f52-971a-a8cedf1d3a62" alt="" width="563"><figcaption><p>Enable Diagnostics Warehouse</p></figcaption></figure>

{% stepper %}
{% step %}
Choose the Diagnostics Warehouse connection

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F2ZTarUeBy7keickpe3sC%2Fimage.png?alt=media&amp;token=86b2cb7b-11c1-47d5-87ce-6299bb7d18e0" alt="" width="375"><figcaption><p>Choose the Diagnostics Warehouse connection</p></figcaption></figure>
{% endstep %}

{% step %}
\[Optional] Diagnostics Warehouse **Advanced Configuration**

* Provide a **schema configuration**

This will be the default schema where Soda creates diagnostic tables. If not provided, Soda will create a new schema with the default name `soda_diagnostics`.

* Provide **table names**

For your own organization, you can add prefixes/suffixes to the table names that Soda will create in your diagnostics warehouse.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FL0wuzERRwn3gU5F7Vfxv%2Fddwh_advanced_configuration.png?alt=media&amp;token=84823aaa-c41a-4de6-afe3-6a51cdf6a20b" alt="" width="505"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
\[Optional] Allow **collections of check results & scans**

You can allow Soda to collect check results and scans (only for this data source) in order to store them securely on your diagnostics warehouse.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FZZO1MwrlfMIAOA40VMTa%2Fddwh_allow_collection_check_results_and_scans.png?alt=media&amp;token=52ad0339-5547-45cc-8072-376e060272b8" alt="" width="505"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
\[Optional] Allow **failed row collection**

You can allow Soda to collect failed rows (only for this data source) in order to store them securely on your diagnostics warehouse.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FJVDxH7Q2gMQ6XXNN445K%2Fddwh_allow_failed_row_collection.png?alt=media&amp;token=388b80c5-233b-4ed5-ab04-549231a2d86a" alt="" width="505"><figcaption></figcaption></figure>

> Learn more about this options when [setting up Diagnostics Warehouse](broken://pages/E1KTiEL4c6zytpedbEZg).
> {% endstep %}

{% step %}
Click **Test connection**, then click **Save**
{% endstep %}
{% endstepper %}

{% hint style="warning" %}
The Diagnostics Warehouse connection **must have write access** in order to perform correctly.
{% endhint %}

> Learn more about [setting up Diagnostics Warehouse](broken://pages/E1KTiEL4c6zytpedbEZg#set-up-diagnostics-warehouse).
> {% endstep %}

{% step %}

### Onboard datasets

After connecting, Soda will perform an **automated dataset discovery**. Soda triggers a scan that analyzes the datasets and retrieves their metadata, including columns and column data types. This **reduces manual setup** efforts, **ensures data coverage** in your environment and **keeps Soda's dataset inventory aligned with your data sources**. This feature allows other Soda features to work seamlessly:

* **Contract generation**
* Automated discovery of **time partition column**

- Automated discovery of **Primary Keys** for [Diagnostics Warehouse](https://docs.soda.io/soda-documentation/understand/diagnostics-warehouse)

{% stepper %}
{% step %}
Choose a dataset selection strategy

Dataset selection can be **manual** or **rules-based.**

<details>

<summary><strong>Manual dataset selection</strong></summary>

**Manual** selection allows you to browse a directory view of all the datasets in your data source.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FSZGpcMU7AOn1wPO1Dz3q%2Fhttps___files.gitbook.com_v0_b_gitbook-x-prod.appspot.com_o_spaces_2FA2PmHkO5cBgeRPdiPPOG_2Fuploads_2Fda45hBJtQVU6tJ3qVlSH_2Fimage.avif?alt=media&amp;token=5a904f53-be90-4d1a-abed-fb286b46eabc" alt=""><figcaption></figcaption></figure>

The **Scope** can range from the entire data source to a specific schema. Any element selected on the left panel becomes the **scope** of the dataset search.

The **manual selection is made for scale**; it can easily handle thousands of schemas and hundreds of thousands of datasets.

{% hint style="warning" %}
Datasets that have already been onboarded **will not be visible** in the manual dataset selection.
{% endhint %}

</details>

<details>

<summary><strong>Rules-based dataset selection</strong></summary>

**Rules-based** selection allows you to automate the dataset onboarding process, only selecting those which match specified rules.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FDBZbzPyea7IwKrxHwV6W%2Fhttps___files.gitbook.com_v0_b_gitbook-x-prod.appspot.com_o_spaces_2FA2PmHkO5cBgeRPdiPPOG_2Fuploads_2FecABXrkajltel2gDwI0U_2Fimage.png?alt=media&amp;token=6e7a166d-7f9e-4fc8-b9cc-90e23175904e" alt="" width="375"><figcaption></figcaption></figure>

{% hint style="info" %}
Rules-based selection includes **existing** and **future** datasets that match the conditions.

Soda will run discovery scans once per day at a random time on the data source. When Soda discovers a **new dataset that matches the conditions** set in the rules, it will **automatically onboard** it.
{% endhint %}

* You can choose the specific schemas where your datasets of interest are located.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FsziISOx4xLkF9wL5VV30%2Fhttps___files.gitbook.com_v0_b_gitbook-x-prod.appspot.com_o_spaces_2FA2PmHkO5cBgeRPdiPPOG_2Fuploads_2FLEob6nMNeKBu9Ps9Vzvx_2Fimage.avif?alt=media&amp;token=ca941dc5-f131-4882-9682-01cc9c2aa2ee" alt="" width="375"><figcaption><p>Choose the specific schemas where your datasets of interest are located.</p></figcaption></figure>

* You can add rules to include or exclude datasets that match certain conditions, such as "name contains" or "name starts with", or provide your own regex pattern.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FFColtCjRiby1Ibr4xnhx%2Fhttps___files.gitbook.com_v0_b_gitbook-x-prod.appspot.com_o_spaces_2FA2PmHkO5cBgeRPdiPPOG_2Fuploads_2FEoZSOhzkguDVDlTVVBy9_2Fimage.avif?alt=media&amp;token=93d83ff3-861d-40b8-b71e-c9f0cb9bf8ef" alt="" width="375"><figcaption><p>After selecting a scope, you can filter datasets by different conditions, including or excluding rules.</p></figcaption></figure>

* To create a rule, choose a condition that datasets must match in order to be onboarded.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FHDNz6A9yAdOC7vbAZTZR%2Fhttps___files.gitbook.com_v0_b_gitbook-x-prod.appspot.com_o_spaces_2FA2PmHkO5cBgeRPdiPPOG_2Fuploads_2FvteYFfPcA5A1tSuTgDKo_2Fimage.avif?alt=media&amp;token=c0034d8f-d4b8-4a77-b30d-6abc7b403da5" alt="" width="375"><figcaption><p>Choose a condition that datasets must match in order to be onboarded.</p></figcaption></figure>

* In the example below, only datasets whose name **does not** start with "`dwh`" from the `public` schema will be onboarded.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FAntgg0Y33ikhTV41s7Sb%2Fhttps___files.gitbook.com_v0_b_gitbook-x-prod.appspot.com_o_spaces_2FA2PmHkO5cBgeRPdiPPOG_2Fuploads_2F3fpyYVrAtm4t2rbfY7bH_2Fimage.avif?alt=media&amp;token=57f55093-fe6c-42ac-9c3c-1e48d78911d4" alt="" width="375"><figcaption><p>A Rule Name can be provided to identify this dataset selection rule.</p></figcaption></figure>

Once you click on **Validate rule**, Soda will calculate how many datasets currently match the defined conditions:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FX51aA1A4AHdSpKLTVnPU%2Fimage.png?alt=media&amp;token=e7b0772d-3670-4984-8d5b-b8f2494431de" alt="" width="563"><figcaption></figcaption></figure>

***

#### Onboarding rules review

**Once the onboarding process is finished** (after [Enabling Metric Monitoring](#id-3.-enable-metric-monitoring-and-profiling-optional)), **an overview of the Onboarding Rules will be provided**. From this view, rules can be edited or deleted:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F0UWAhb7qUmrrAEIkf0jS%2Fimage.png?alt=media&amp;token=097dd9a8-0992-46ab-8338-ca6a37a1d184" alt="" width="563"><figcaption></figcaption></figure>

* Rules will be executed in order of appearance on this view.
* The order of the rules can be changed. **As soon as a dataset matches a rule, it will be onboarded** automatically; datasets can only be onboarded once.

</details>
{% endstep %}

{% step %}
Confirm onboarding

<i class="fa-octagon-check">:octagon-check:</i> Click on **Next** to finish the process.

Once onboarded, datasets will appear in your Soda Cloud UI and become available for contract creation or metric monitoring.

{% hint style="info" %}
**Refresh dataset discovery:** Soda runs discovery scans once per day at a random time to get the latest view of tables and schemas within a data source. **By pressing on the** <i class="fa-arrow-rotate-right">:arrow-rotate-right:</i> **icon** on the top right of the page, you can run the scan on demand.

**To configure the discovery scan schedule, go to** [#configure-discovery-scan](#configure-discovery-scan "mention").
{% endhint %}
{% endstep %}

{% step %}
Enable/disable **Contract Autopilot**

Enable Contract Autopilot to auto-generate contracts tailored to your data. This is a **hands-off, AI-powered feature** that allows you to have a first contract draft that is made to fit your dataset's columns.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FSeQy78MoZMrBvajPLPwI%2Fimage.png?alt=media&amp;token=85c4f13d-5524-455a-9a3f-30245465c8f2" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
\[Optional] Schedule generated contracts

You can **add a schedule to generated contracts** to define when Soda will run them, including frequency and time. Default is daily at 00:00 UTC.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FLy4HOgHV3KHt9FLnyDOM%2Fimage.png?alt=media&amp;token=43193ec6-ad07-49a7-8957-3dd5c9ab63fc" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
\[Optional] Enable **Metric Monitoring** & **Profiling**

Through Metric Monitoring, you can enable built-in monitors to automatically track row counts, schema changes, freshness, and more across your datasets. This step is optional but recommended. **This can be enabled in bulk when onboarding data sources and datasets**.

> Learn more about Metric Monitoring: [Metric monitoring](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard)

{% stepper %}
{% step %}
Toggle on **Metric Monitoring**

When metric monitoring is enabled it's possible to later add column monitors on dataset level or overwrite any of the settings.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FGQDClF2Bgo9s2DlNuIVh%2Fimage.png?alt=media&amp;token=abf84a84-6711-481b-a04d-71816ff5e5a4" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Set a **Monitoring Schedule**

The monitoring schedule defines when Soda scans a dataset to capture and evaluate metrics. While scans may run slightly later due to system delays, Soda uses the **actual execution time,** not the scheduled time, when visualizing **time-sensitive metadata metrics** like insert lag or row count deltas. This ensures accuracy.

Data-based metrics like averages or null rates are not affected by small delays, as Soda only scans complete partitions, keeping these metrics stable and reliable.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FIIsHn7RINvdneD8WFXvB%2Fimage.png?alt=media&amp;token=fd65c7ab-8258-465d-9f44-6728c1989632" alt="" width="375"><figcaption></figcaption></figure>

Scans can be scheduled to occur from **hourly** to **weekly**, depending on your needs.

> Learn more about [how to pick a scan time](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/scan-time-and-scan-time-strategy).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FCxGrBfQmBjE6TPNhX6QG%2Fimage.png?alt=media&amp;token=ee8e856b-7f40-4e81-bf01-f354205fced8" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Toggle on/off **Historical Metric Collection**

When **Historical Metric Collection** is enabled, Soda automatically calculates past data quality metrics through backfilling and applies the anomaly detection algorithm to that historical data through backtesting. This gives you immediate visibility into past data quality issues, even before monitoring was activated.

The historical data also helps train the anomaly detection algorithm, improving its accuracy from day one. You can specify a start date to control how far back the backfilling process should begin.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FmaJ6AyjMhgGJRdBL4F3x%2Fimage.png?alt=media&amp;token=6f6b380f-a8a6-40f9-bf2f-1edf2e0264d8" alt="" width="375"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Suggest a **Time Partition Column**

Metrics that are not based on metadata require a **time partition column** to group data into daily intervals or 24-hour buckets, depending on the monitoring schedule. This column must be a **timestamp field**, ideally something like a `created_at` or `last_updated` column. It's important that this timestamp reflects **when the data arrives in the database**, rather than when the record was originally created.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FleDBgO9yFyu6J0GWlpje%2Fimage.png?alt=media&amp;token=456a07c0-2364-48cb-8525-6761457e883f" alt="" width="375"><figcaption></figcaption></figure>

Soda uses a list of **suggested time partition columns** to determine which column to apply. If multiple columns are suggested, Soda checks them **in the order they are listed**, starting with the first. It will try to match one by validating that the column is a proper timestamp and suitable for partitioning.

If none of the suggested columns match, Soda falls back to a **heuristic approach**. This heuristic looks at metadata, typical naming conventions, and column content to infer the most likely time partition column.

If the heuristic fails to find a suitable column or selects the wrong one, the time partition column can be manually configured after onboarding under **dataset settings**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FpvgOmjdKj6OKRNDRdpm8%2Fimage.png?alt=media&amp;token=6a8d0c4a-fe09-4fe3-b9d9-69ad73fff87e" alt="" width="563"><figcaption><p>To access Dataset Settings: Navigate to a dataset > Click on "<strong>⋮</strong>" > <strong>Edit Dataset</strong> > <strong>Profiling &#x26; Metric Monitoring</strong> tab.</p></figcaption></figure>
{% endstep %}

{% step %}
\[Optional] Set a delay for **Late-arriving data**

Rows often land a while after the moment they describe. Without a delay, scans analyze a period before all of its data has arrived. Set a delay and each scan analyzes an older period instead of the one that just ended.

Use the gap between the time a row belongs to and the time it lands. Leave this on **No delay** if there is no such gap. You can change it per dataset later.

> Learn more about [Late-arriving data](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/late-arriving-data).
> {% endstep %}

{% step %}
\[Optional] Toggle on **Advanced monitor configuration**

Enable or disable each monitor to suit your data quality needs.
{% endstep %}

{% step %}
Click on **Next**
{% endstep %}
{% endstepper %}
{% endstep %}

{% step %}
Define a **dataset owner**

**Define a dataset owner** by choosing a user from the drop-down menu.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FDbOtQNOTWfACbSCDGmJO%2Fdataset_owner_dropdown.png?alt=media&amp;token=c11c2ce0-0171-4ccc-8762-7842345fd889" alt="" width="505"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
\[Optional] Enable **check results & scans collection**

From this onboarding view, you can also enable **check results & scans collection** if Diagnostics Warehouse is enabled for this data source.
{% endstep %}

{% step %}
\[Optional] Enable **failed row collection**

From this onboarding view, you can also enable **Failed row collection** if Diagnostics Warehouse is enabled for this data source. When you enable **Collect failed rows**, select a **storage** and a **detection strategy**.

{% hint style="info" %}
Profiling scheduled execution is set at 12:00AM UTC by default.
{% endhint %}
{% endstep %}

{% step %}
\[Optional] Enable **Profiling**

> Learn more about [Profiling](https://docs.soda.io/soda-documentation/find/data-observability/profiling).
> {% endstep %}

{% step %}
Click on **Finish**

If you used **Rules-based** **selection** to onboard datasets, an **Active Onboarding Rule Pipeline** view will appear now to confirm the conditions.
{% endstep %}
{% endstepper %}
{% endstep %}

{% step %}
**Access the datasets**

Once onboarding is completed, your data source will appear in the **Data Sources** list. You can click the **Onboarded Datasets** button to access the connected datasets.
{% endstep %}
{% endstepper %}

Soda Cloud will now spin up its **Soda-hosted Runner** and perform an initial **Profiling** & **Historical Metric Collection scan**, if configured. This usually takes only a few minutes.

🎉 **Congrats! You've successfully onboarded your data source and datasets.**\
You're now ready to create data contracts and start monitoring the quality of your data.

***

## Onboard datasets from an existing data source

Note that you can repeat the datasets onboarding process at any time to add more datasets from the same data source. Datasets that previously have been onboarded will not re-appear in the data selection step.

Simply return to the data source page and click **Onboard Datasets** to update your selection. You will be prompted to follow the [dataset onboarding steps](#onboard-datasets).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FvZjx3punyqnAiSy3jesr%2Fonboard_datasets_button.png?alt=media&amp;token=95835052-9325-41bb-9b47-41a58034cea8" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
You need the **Manage data sources** global permission to add a new data source. Learn about [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

## Connect a data source onboarded with Soda Core to a Runner

When you create or push a [new data source from **Soda Core**](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/via-soda-core), it becomes available in Soda Cloud, but it is **not automatically connected to a Soda Runner**.

Connecting the data source to a Runner enables Soda Cloud features that require a Runner runtime, such as:

* **Metric Monitoring**
* **Profiling**
* Running **Data Contracts on the Soda Runner** (via scheduling and the Soda Cloud interface)

{% stepper %}
{% step %}
Locate the partially onboarded data source in Soda Cloud

* Navigate to **Data Sources**. Find the data source that was created or pushed from Soda Core.
* Click on "**⋮**" > **Edit connection**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FHZ6F3nL1T8gcFjNiivKz%2Fedit_partial_connection.png?alt=media&amp;token=4b34c84b-b2b1-4446-acbb-d1ef2b3b7855" alt="" width="190"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Edit the connection and select a Runner

{% stepper %}
{% step %}
Fill in the connection form:

* **Select the Runner** you want to use ([Soda-hosted](https://docs.soda.io/soda-documentation/connect/deployment-options#soda-hosted) or [self-hosted](https://docs.soda.io/soda-documentation/connect/deployment-options#self-hosted-runner))
* **Provide the required connection details** and credentials (use Soda Secrets for sensitive values)
  {% endstep %}

{% step %}
Click **Test Connection**.
{% endstep %}

{% step %}
Click **Connect** to save the configuration.
{% endstep %}
{% endstepper %}
{% endstep %}

{% step %}
Enable Runner-powered features on your datasets

<i class="fa-octagon-check">:octagon-check:</i> Once the data source is connected to a Runner, you can enable Runner-powered features on your datasets, including:

* Metric Monitoring
* Profiling
* Running Contracts on the Runner
  {% endstep %}
  {% endstepper %}

## Configure discovery scan

By default, Soda runs discovery scans once per day at a random time; if you need a different cadence, you can configure a custom schedule for each data source.

{% stepper %}
{% step %}
Navigate to **Scans** and click on the scan you want to configure.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FQD5eVXkuqZaLwMNZHLiY%2Fscans_page.png?alt=media&amp;token=9e866474-26cb-4d29-9c22-8299813807e8" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click on **Edit Scan Definition** (top right).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FzOuTAywvGktcLLpvwOuL%2Fscan_page_w_edit_scan_definition.png?alt=media&amp;token=5eec0a1f-82d1-4065-8204-0bd60ca7ba1a" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Edit** the schedule.

You can choose to trigger the scan **on schedule** or **externally** using a Soda Core CLI command.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FmFwU9BMf9Ee5iBqUUK95%2Fedit_scan_deifinition_popup.png?alt=media&amp;token=1faf3d6a-1f03-4b82-b826-b008a8043bfd" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click on **Save**.
{% endstep %}
{% endstepper %}

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Additional settings

## Test a contract on a sample

{% hint style="warning" %}
Currently, **this feature is only supported in Snowflake data sources**.
{% endhint %}

When testing a data contract, Soda allows you to **run contract validation on a sample of your dataset** instead of the full data. This feature helps you quickly and cost-efficiently verify that your contract runs correctly before executing full scans.

Running a test contract on a sample enables you to:

* Validate that your contract **syntax**, **checks**, and **filters** work as expected.
* **Reduce data warehouse compute cost** while verifying new or updated contracts.
* **Iterate faster** on contract definitions in development environments.

Results from sampled runs reflect only a subset of your data and **may not represent its actual quality**. Use full verification once your contract logic is validated.

### Enable sampling for test contracts <a href="#fe3e56b4-c951-4d56-9b18-e7e4a92ce408" id="fe3e56b4-c951-4d56-9b18-e7e4a92ce408"></a>

This feature can be enabled at the **data source level**, applying to **all datasets** that use that connection.

{% hint style="warning" %}
You need the "**Manage data sources**" global permission to add a new data source. Learn more about [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles).
{% endhint %}

To enable this feature:

1. Go to **Data sources**.
2. Click **Edit connection** for a data source.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FURoxSsfsC9TDB9mJH65D%2Fimage.png?alt=media&amp;token=08eb4560-8ba1-4095-8851-c2f21e466a82" alt=""><figcaption></figcaption></figure>

3. Under the **Connection Details** section, toggle **Data Sampling**.
4. Specify your **sample size** on the **Limit** field.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fey6zFYu73Ys12Ut6ivUO%2Fimage.png?alt=media&amp;token=387f86dc-c019-4187-8fac-de12fa0ddef2" alt="" width="375"><figcaption></figcaption></figure>

5. <i class="fa-octagon-check">:octagon-check:</i> Click **Connect**.


# Via Soda Core

> Go to the [data source reference for Soda Core](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core) for more in-depth details about supported data sources and connection parameters.

**A data source can be partially onboarded** programmatically via Soda Core. Once a data source is fully onboarded via Soda Cloud, the engineering team can onboard datasets programmatically.

In a nutshell, a dataset is pushed onto Soda Cloud by:

1. Creating a contract YAML that points to that dataset
2. Pushing the results of the contract onto Soda Cloud

{% hint style="warning" %}
**Soda does not support pushing a full data source configuration programmatically**. This flow will create an empty data source that needs to be configured later in Soda Cloud.
{% endhint %}

## Partially onboard a data source via CLI

This flow shows how to partially onboard a data source programmatically in order to finish setting it up in Soda Cloud.

{% stepper %}
{% step %}

#### Set up your connections

Follow the [CLI reference](https://docs.soda.io/soda-documentation/developer-tools/cli-reference) to:

1. Perform [**installation**](https://docs.soda.io/soda-documentation/developer-tools/cli-reference#installation) **steps**
2. [**Connect to Soda Cloud**](https://docs.soda.io/soda-documentation/developer-tools/cli-reference#connect-to-soda-cloud)
3. [**Configure a data source**](https://docs.soda.io/soda-documentation/developer-tools/cli-reference#configure-a-data-source)
   {% endstep %}

{% step %}

#### Create a contract

Run the following:

{% code overflow="wrap" %}

```shellscript
soda contract create --dataset datasource/db/schema/table --file contract.yaml --data-source ds_config.yml --soda-cloud sc_config.yml
```

{% endcode %}

{% hint style="warning" %}
Unlike the `create` command in the CLI reference, to push a data source via CLI **you must not use the `--use-runner` flag**.

**The `--use-runner` flag** would attempt to find the data source in Soda Cloud, which has not been yet set up in your environment at this time.
{% endhint %}

**When should I use the `--use-runner` flag?**

In some organizations, the data source Admin can create the data source connection in Soda Cloud without onboarding any datasets. Then, **engineers can create and publish new contracts to onboard datasets via CLI**. Since the data source exists already in Soda Cloud, the `--use-runner` flag should be used so that the datasets are pushed from Soda Core onto the existing data source through the Runner.
{% endstep %}

{% step %}

#### Test, verify & publish the contract

1. **Test that the contract is correct**

   ```shellscript
   soda contract test --contract contract.yaml
   ```
2. **Verify the contract**

   ```shellscript
   soda contract verify --contract contract.yaml --data-source ds_config.yml
   ```
3. **Publish the contract**

   ```shellscript
   soda contract publish --contract contract.yaml --soda-cloud sc_config.yml
   ```

{% endstep %}
{% endstepper %}

<i class="fa-octagon-check">:octagon-check:</i> After successfully publishing the contract, you will see a success output:

```
  __|  _ \|  \   \\
\__ \ (   |   | _ \\
____/\___/___/_/  _\\ CLI v4.0.4b24
Fetching datasets configurations from Soda Cloud for datasets '[DatasetIdentifier(data_source='CLI_testing', prefixes=['postgres', 'aldi_local'], dataset='retail_orders')]'
Verifying contract 📜 contract.yaml 🤞

### Contract results for CLI_testing/postgres/aldi_local/retail_orders
+-----------------+-----------------------------------+-------------------------------+-----------+---------------+   
| Column          | Check                             | Threshold                     | Outcome   | Diagnostics   |   
+=================+===================================+===============================+===========+===============+   
| [dataset-level] | Schema matches expected structure | level: fail                   | ✅ PASSED |               |   
|                 |                                   | must be less than or equal: 0 |           |               |   
+-----------------+-----------------------------------+-------------------------------+-----------+---------------+   
# Summary:
|----------------|---|----|
| Checks         | 1 |    |
| Passed         | 1 | ✅ |
| Failed         | 0 | ✅ |
| Warned         | 0 | ✅ |
| Not Evaluated  | 0 | ✅ |
| Excluded       | 0 | ✅ |
| Runtime Errors | 0 | ✅ |

👌 Results sent to Soda Cloud
To view the dataset on Soda Cloud, see https://cloud.us.soda.io/o/<datasetID>/datasets/ab1bc55d-c49a-441e-a0dc-c01857c71b21
Updating post processing stage 'diagnosticWarehouse' to state 'completed' for scan <scanID>
Updated post processing stage 'diagnosticWarehouse' to state 'completed' for scan <scanID>

```

## Complete onboarding via Soda Cloud

{% hint style="info" %}
A data source **must be connected to Soda Runner** to access contract verification in Soda Cloud. Connection to Soda Runner is performed via Soda Cloud.
{% endhint %}

* If you attempt to use contract verification, you will find the following warning:

  <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F7N3amM2VBr0urjmVdKwT%2Fimage.png?alt=media&amp;token=4f849563-2f0e-4242-b9af-665cf879cb5c" alt=""><figcaption></figcaption></figure>
* If you attempt to access Metric Monitoring capabilities in Soda Cloud, you will find the following warning:

  <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FUxmoloUIT0qb9mmMeL9Z%2Fimage.png?alt=media&amp;token=c13fc44c-a981-41ed-ba29-90119c8d51ae" alt=""><figcaption></figcaption></figure>

{% hint style="success" %}
**In order to finish the onboarding,** [**complete the connection in Soda Cloud**](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud#connect-a-data-source-onboarded-with-soda-core-to-an-agent)**.**
{% endhint %}

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Data testing

## What is data testing?

Data testing is the practice of validating that your data meets the expectations you’ve defined for it before it reaches stakeholders, dashboards, or downstream systems. Just like software testing ensures your code behaves as intended, data testing safeguards the quality and reliability of your data.

At Soda, we see data testing as **the foundation of data trust**. Whether you’re verifying row counts, checking for missing or invalid values, or enforcing schema integrity, the goal is the same: catch issues early, reduce incidents, and keep your data consumers confident.

## What is a Data Contract?

A **Data Contract** is a formal agreement between data producers and data consumers that defines what “good data” looks like. It sets expectations about schema, freshness, quality rules, and more, and makes those expectations explicit and testable.

With a data contract in place, producers commit to delivering data that meets certain standards. Consumers, in turn, can rely on that contract to build reports, models, or pipelines without second-guessing the data.

At Soda, Data Contracts are **testable artifacts** that can be authored, versioned, verified, and monitored, whether in code or in the UI. They’re the connective tissue between producers and consumers, aligning teams and eliminating ambiguity.

### What is Contract Verification?

Defining a contract is only the first step. Verifying that your data actually **meets** the expectations is where the value is realized. Contract verification is the process of testing whether the data in your datasets aligns with the rules, thresholds, and schema defined in the contract.

At Soda, contract verification is **fully automated**. Whether triggered manually, on a schedule, or as part of your CI/CD pipelines, each verification run checks that:

* The schema matches the contract definition (columns, data types, structure)
* The data complies with checks like missing, duplicate, invalid values, and custom rules

This helps you catch issues early, ensure data quality over time, and build trust across your organization.

## Authoring and managing Data Contracts: choose your style

Soda supports two complementary ways to author and manage data contracts. They are designed to fit the way your team works.

### Cloud-managed Contracts (Soda Cloud UI)

If you’re a data analyst, product owner, or stakeholder, who prefers intuitive interfaces over code, **Soda Cloud** is the ideal workspace.

With the Soda Cloud UI, you can:

* Browse datasets and view profiling insights
* Define a contract with a no-code Editor
* Schedule and monitor contract verifications
* Collaborate with your team and publish contracts with a click

There’s no setup or YAML required, just fast, visual workflows that enable domain experts to contribute directly to data quality.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F27fgQ68C9qZMzbtbQzMu%2FScreenshot%202025-05-20%20at%203.05.13%20PM.png?alt=media&amp;token=1b60a020-a07b-4ce7-8685-1d41e480c7fb" alt="" width="563"><figcaption></figcaption></figure>

### Git-managed Contracts (Soda Core CLI)

If you live in your terminal and manage your data pipelines as code, you’ll want to use **Soda Core** and the **Soda CLI**.

With this setup, you can:

* Define contracts in YAML
* Run contract verifications in CI/CD
* Push the contract and verification results to Soda Cloud for visibility
* Use Git as the source of truth for version control, collaboration, and reviews
* Collaborate with non-technical users that use Soda Cloud and integrate with engineering workflows via Git

This path offers full control, transparency, and seamless integration into your dev tooling.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FHbx4NYN4P3I7l2qTpLFO%2FScreenshot%202025-05-21%20at%2010.56.28%20AM.png?alt=media&amp;token=d3263306-f094-4d87-bb0d-38e1756c743b" alt=""><figcaption></figcaption></figure>

***

### Combine UI and Git for cross-team collaboration

Soda gives you the flexibility to blend both approaches. For example, non-technical users can define or adjust contracts visually in Soda Cloud for the datasets they manage, while engineers can use Git-managed contracts for the datasets they own.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FBqzaygUEugWcFP82mzYE%2Fimage.png?alt=media&amp;token=2e21e0fa-34e1-453b-9f51-68b78687f5b9" alt=""><figcaption></figcaption></figure>

This hybrid model enables collaboration across teams:

* Business users bring domain expertise directly into the contract
* Engineers maintain quality, consistency, and governance
* Each dataset follows the authoring method that best suits the team responsible for it

You can mix and match—using the UI for some contracts, and code for others—depending on your team's structure and preferences.

And even if Data Contracts are managed in Git, you can still involve non-technical users who can propose changes to a contract in the UI. These approved changes can be embedded into engineering workflows and synced to Git, ensuring that every update follows your organization’s quality and deployment standards.

Choose the model, or combination of models, that works best for your organization.

***

## Soda Runner vs. Soda Core for execution

Once a contract is published, you’ll want to verify that the actual data meets the contract’s expectations. This verification can be done in two ways:

* **Soda Runner** is our managed runner that lives in your environment and connects to Soda Cloud. It handles contract verification, scheduling, and execution securely, without exposing your data externally. It is great for teams who want central management without maintaining CLI infrastructure.
* **Soda Core** is our open-source engine you can run anywhere: locally, in CI, or data pipeline. It’s lightweight, customizable, and great for teams that prefer full control or have strict environment constraints.

Both approaches support the same Data Contract logic. Choose the one that best fits your deployment model.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Git-managed Data Contracts

## Define, version, and test contracts as code

For teams that manage data like software, Git-managed data contracts offer a code-first way to define and enforce data quality expectations.

In this model, contracts are written in YAML and stored in your Git repository, right alongside your data models, transformation logic, and CI/CD workflows. You write, version, test, and promote contracts just like any other code artifact.

This approach gives engineers full control, reproducibility, and integration into development pipelines. And with the right setup, you can still collaborate with non-technical users via Soda Cloud and even sync UI-authored changes into Git using our future proposal workflow.

***

## Why Git-managed?

* **Full version control**\
  Track every change, roll back when needed, and manage contracts with the same discipline as application code.
* **Code-first workflow**\
  Keep contracts close to your data models and transformations for better alignment, automation, and traceability.
* **CI/CD integration**\
  Run contract verifications in your existing pipelines; on every commit, PR, or deployment.
* **Team governance**\
  Ensure all changes are reviewed, tested, and approved through standard Git workflows (pull requests, approvals, branching).
* **Hybrid collaboration**\
  Combine Git workflows with Soda Cloud for monitoring, visualization, and cross-functional input via contract proposals.

***

If you're already managing your data infrastructure in Git, Git-managed contracts are the natural extension for bringing **data quality under control** without adding friction or silos.

In the next sections, we’ll walk you through how to set up, author, and run Git-managed contracts using the Soda CLI.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Configure your environment

Before you can define, test, or verify Git-managed data contracts, you need to install the Soda CLI and configure your environment.

This setup gives you full control over your contracts, letting you version them in Git, execute them locally or remotely, and integrate them into your CI/CD pipelines.

## 1. Install the Soda Core Python Package

> If you haven't already, follow the [Soda Core installation flow](https://docs.soda.io/soda-documentation/connect/deployment-options/soda-python-libraries) to set up your environment, set your credentials, and install the necessary Soda extensions.

## 2. \[Optional] Connect to Soda Cloud

If you want to interact with Soda Cloud to **publish contracts** and **view verification results**, or to **use Soda Runner**, you’ll need to connect the CLI to your Soda Cloud account.

> Don’t have an account? [Book a demo](https://soda.io/demo?utm_source=documentation\&utm_medium=cloud_docs\&utm_campaign=demo_request\&utm_content=connect_soda_cloud) to get started.

### Create the config file

Run the following command:

```shellscript
soda cloud create -f sc_config.yml
```

This generates a basic Soda Cloud configuration file:

{% code title="sc\_config.yml" %}

```yaml
soda_cloud:
  host: cloud.soda.io        ##change to cloud.us.soda.io, depending on your Soda region
  api_key_id: ${SODA_CLOUD_API_KEY_ID}
  api_key_secret: ${SODA_CLOUD_API_KEY_SECRET}
```

{% endcode %}

#### Add your API keys

To add your API keys, you can do **one** of the following:

* Set environment credentials, as directed in the [Private PyPi installation flow](https://docs.soda.io/soda-documentation/connect/deployment-options/soda-python-libraries#private-pypi-installation-flow), or
* Open `sc_config.yml` and fill in your API key and organization details.

> Learn more about how to [generate API keys](https://docs.soda.io/apis/generate-api-keys).

### Test the connection

Run the following command:

```shellscript
soda cloud test -sc sc_config.yml
```

<i class="fa-octagon-check">:octagon-check:</i> This ensures the CLI can authenticate and communicate with Soda Cloud. A successful connection will return:

```
  __|  _ \|  \   \\
\__ \ (   |   | _ \\
____/\___/___/_/  _\\ CLI v4.0.4b24
Testing soda cloud file sc_config.yml
✅ Success! Tested Soda Cloud credentials in 'sc_config.yml'
```

## 3. Configure your data source

To verify a contract, Soda needs to know how to connect to your data source. You have **two options**:

### Connect with Soda Core

If you prefer to define your own connection locally (or aren’t using a Soda Runner), you can create a data source config file for **Soda Core**.

Install the required package for your data source.\
For example, for PostgreSQL:

```shellscript
pip install -i https://pypi.cloud.soda.io/simple --pre -U "soda-postgres>4"
```

> See the [Data source reference for Soda Core](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core) for supported packages and configurations.

#### Create the config file

```shellscript
soda data-source create -f ds_config.yml
```

Open `ds_config.yml` and provide the necessary credentials.

For example, with PostgreSQL:

{% code title="ds\_config.yml" %}

```yaml
type: postgres
name: postgres
connection:
  host: host_name
  port: 5432
  user: user_name
  password: ${env.SODA_DEMO_POSTGRES_PW}
  database: db_name
```

{% endcode %}

> Refer to the [Data source reference for Soda Core](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core) for the configurations of each data source type.

{% hint style="info" %}
**Avoid hardcoding secrets**. Use environment variables or a secrets manager where possible.
{% endhint %}

#### Test the connection:

```shellscript
soda data-source test -ds ds_config.yml
```

### Use an existing Data Source via Soda Runner

If your data source is already connected to Soda Cloud using a **Soda Runner** (hosted or self-hosted), you can reuse that connection without managing credentials or configs locally.

You just need to ensure you have [set up the connection with Soda Cloud](#id-2.-optional-connect-to-soda-cloud).

***

Choose the method that best fits your setup:

* **Soda Runner** for a centralized, cloud-managed connection.
* **Local configuration** if you want full control within your environment.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Create and edit contracts

With Git-managed contracts, you define your expectations as code using YAML. This gives you full control over how your data is validated, and allows you to manage contracts just like any other code artifact: versioned, tested, and deployed via Git.

> To learn all about the structure and supported features, refer to the full specification in the [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference).

The contract structure includes:

* Dataset and column structure
* Available check types (`missing`, `invalid`, `duplicate`, `freshness`, etc.)
* Filters (dataset-level and check-level) — Optional
* Threshold configuration — Optional
* Use of variables — Optional
* Scheduling — Optional
* ...and more

## Create a contract

{% hint style="warning" %}
Creating a contract requires having installed the `soda` package using **the** [**Private PyPi installation flow**](https://docs.soda.io/soda-documentation/connect/deployment-options/soda-python-libraries#private-pypi-installation-flow) available both for Team and Entreprise license, unless you are using a Runner.

Need access to the private PyPI? Please [contact us](mailto:support@soda.io).
{% endhint %}

`soda contract create` bootstraps a contract by introspecting a dataset's schema. It writes a YAML file that lists every column with its data type and adds a single `schema:` check. You then edit the file to add data-quality checks (`missing`, `invalid`, `duplicate`, `freshness`, etc.).

### Prerequisites

* A data source configuration file (for local execution), or a Soda Cloud configuration file (for runner execution).
* Access to Soda's private PyPI (see include above) for installing the library.

### Local execution

Runs introspection from your machine against the data source.

{% code overflow="wrap" %}

```shellscript
soda contract create -d <DATA_SOURCE>/<DATABASE>/<SCHEMA>/<TABLE> -ds ds_config.yml -f contract.yml
```

{% endcode %}

### Runner execution

Delegates introspection to a running Soda Runner. Useful when the data source is only reachable from the runner's network.

{% code overflow="wrap" %}

```shellscript
soda contract create -d <DATA_SOURCE>/<DATABASE>/<SCHEMA>/<TABLE> -sc sc_config.yml -f contract.yml --use-runner
```

{% endcode %}

Omit `-f` to send the generated skeleton to Soda Cloud as a draft contract instead of writing a local file.

### Parameters

| Parameter                    | Required    | Description                                                                                                         |
| ---------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------- |
| `--dataset, -d`              | Yes         | Fully qualified dataset name, slash-separated: `<DATA_SOURCE>/<DATABASE>/<SCHEMA>/<TABLE>`.                         |
| `--file, -f`                 | Conditional | Path to the contract YAML file to create. Directories are created if needed. Required unless `--soda-cloud` is set. |
| `--data-source, -ds`         | Conditional | Path to a data source YAML config file. Required for local execution; ignored when `--use-runner` is set.           |
| `--soda-cloud, -sc`          | Conditional | Path to a Soda Cloud YAML config file. Required when `--use-runner` is set, or when `--file` is omitted.            |
| `--use-runner, -a`           | No          | Run introspection on Soda Runner instead of locally. Default: `false`.                                              |
| `--no-type-parameters, -ntp` | No          | Omit data type parameters (precision, scale, length) from generated columns.                                        |
| `--verbose, -v`              | No          | Display detailed logs during execution. Default: `false`.                                                           |

### What the skeleton contains

A skeleton generated for a `retail_orders` dataset in PostgreSQL looks like this:

{% code title="contract.yml" %}

```yaml
dataset: <DATA_SOURCE>/<DATABASE>/<SCHEMA>/<TABLE>
columns:
  - name: order_id
    data_type: character varying
    character_maximum_length: 50
  - name: product_id
    data_type: character varying
    character_maximum_length: 50
  - name: customer_id
    data_type: character varying
    character_maximum_length: 50
  - name: order_quantity
    data_type: integer
  - name: discount
    data_type: real
  - name: shipping_address
    data_type: character varying
    character_maximum_length: 64
  - name: total_order_value
    data_type: real
checks:
  - schema:
```

{% endcode %}

{% hint style="info" %}
The skeleton contains **only a `schema:` check**. No column-level checks (`missing`, `invalid`, `duplicate`, etc.) are generated — add these yourself after reviewing the columns.
{% endhint %}

Use `--no-type-parameters` to suppress `character_maximum_length`, `numeric_precision`, `numeric_scale`, and `datetime_precision` entries if you prefer a shorter contract.

### Common errors

* `Either --file or --soda-cloud must be provided` — supply one as an output destination.
* `--use-runner requires --soda-cloud` — runner mode needs a Soda Cloud config to reach the runner.

You can now edit the contract and add checks by modifying the file.

> Get the most out of your data contracts: head to the [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference) to learn more.

## Test your contract

Before publishing or verifying your contract, you can run a **test command** to ensure the contract is correctly defined. It reads no data, so it needs no data source configuration.

```shellscript
soda contract test --contract contract.yml
```

This will:

* Validate the YAML syntax
* Check for any issues with variables, filters, or structure

{% hint style="info" %}
Run this as part of your development workflow or CI to catch errors early.
{% endhint %}

## Verify your contract

Before publishing your contract, you may want to execute it to verify that it runs correctly.

> Read more about how to [Verify a contract](https://docs.soda.io/soda-documentation/find/data-testing/git-managed-data-contracts/verify-a-contract)

## Publish the Contract

If you have Soda Cloud, once your contract is finalized and tested, you can publish it to Soda Cloud, making it the authoritative version for verification and scheduling.

{% hint style="warning" %}
This action requires the "**Manage contract**" permission on the dataset; the user is identified based on the API key provided in the Soda Cloud configuration.

Learn more about permissions here: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

```sh
soda contract publish --contract contract.yml --soda-cloud sc.yml
```

> Learn how to connect the CLI to Soda Cloud: [Core CLI reference](https://docs.soda.io/soda-documentation/developer-tools/cli-reference#connect-to-soda-cloud)

Publishing:

* Uploads the contract to Soda Cloud
* Makes it available for manual, scheduled, or programmatic verifications
* Enables visibility and collaboration through the UI

Once published, the contract becomes the **source of truth** for the dataset until a new version is published.

You’re now ready to start verifying your contract and monitoring your data.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Verify a contract

Once your contract is authored and published (or available locally), you can verify whether the actual data complies with the defined expectations. Soda provides two execution options:

* **Soda Core** – run verifications locally, typically in CI/CD pipelines or dev environments.
* **Soda Runner** – run verifications remotely using a runner deployed in your environment, triggered via Soda Cloud.

Both approaches support variable overrides, publishing results to Soda Cloud, and integration into automated workflows.

> Learn more about [Deployment options](https://docs.soda.io/soda-documentation/connect/deployment-options)

## Using Soda Core

Soda Core runs the verification locally, connecting to your data source using the defined data source configuration file.

```javascript
soda contract verify --data-source ds_config.yml --contract contract.yaml
```

This command:

* Connects to your database using the local config
* Loads the contract
* Runs all checks and returns a pass/fail result

### With variable overrides

You can pass variables defined in the contract using the `--set` flag:

```javascript
soda contract verify --data-source ds_config.yml --contract contract.yaml --set START_DATE=2024-05-01
```

> Learn about variables in Data Contract: [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference#make-contracts-dynamic-with-variables)

### Publish results to Soda Cloud

To send verification results to Soda Cloud for visibility and reporting.

Add the flag `--publish` to the command.

{% hint style="warning" %}
This action requires the "Manage contract" permission on the dataset; the user is identified based on the API key provided in the Soda Cloud configuration.

Learn more about permissions here: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

```bash
soda contract verify --data-source ds_config.yml --contract contract.yaml --publish --soda-cloud sc_config.yml
```

> Learn how to connect the CLI to Soda Cloud: [Core CLI reference](https://docs.soda.io/soda-documentation/developer-tools/cli-reference#connect-to-soda-cloud)

This is recommended if you want stakeholders to see the outcomes in Soda Cloud or include them in dashboards and alerting.

### Run a subset of checks

Use `--check-filter` (or `-cf`) to run only specific checks from your contract. Each filter is a `key=value` expression that matches checks by their properties such as type, column, name, or custom attributes.

{% code overflow="wrap" %}

```bash
# Run only missing checks
soda contract verify --data-source ds_config.yml --contract contract.yaml -cf type=missing
```

{% endcode %}

```bash
# Run only checks on the id column with severity=critical
soda contract verify --data-source ds_config.yml --contract contract.yaml \
  -cf column=id -cf attributes.severity=critical
```

Write a filter as `key!=value` to leave out the checks that match it instead:

```bash
# Run every check except the schema check
soda contract verify --data-source ds_config.yml --contract contract.yaml -cf type!=schema
```

You can also use `--check-paths` (or `-cp`) to select checks by their YAML path.

> Learn more about check filters, check paths, wildcards, and combinations in the [CLI reference](https://docs.soda.io/soda-documentation/developer-tools/cli-reference#run-a-subset-of-checks).

## Using Soda Runner

Soda Runner executes verifications using data sources configured in Soda Cloud.

```bash
soda contract verify --contract contract.yaml --use-runner --soda-cloud sc_config.yml
```

This setup:

* Runs verifications through the Soda Runner connected to your data source
* Fetches the published contract from Soda Cloud
* Returns the result locally in the CLI

### With variable overrides

You can pass variables defined in the contract using the `--set` flag:

```bash
soda contract verify --contract contract.yaml --use-runner --soda-cloud sc_config.yml --set START_DATE=2024-05-01
```

> Learn about variables in Data Contract: [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference#make-contracts-dynamic-with-variables)

### Publish results to Soda Cloud

You can also push results to Soda Cloud from the runner-based run.

Add the flag `--publish` to the command.

{% hint style="warning" %}
This action requires the "Manage contract" permission on the dataset; the user is identified based on the API key provided in the Soda Cloud configuration.

Learn more about permissions here: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

```bash
soda contract verify --dataset datasource/db/schema/table --publish --use-runner --soda-cloud sc_config.yml
```

This is recommended if you want stakeholders to see the outcomes in the Soda Cloud or include them in dashboards and alerting.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Cloud-managed Data Contracts

Cloud-managed Data Contracts let you define and manage expectations for your data directly in the Soda Cloud UI.

This approach is perfect for data analysts, product owners, and business stakeholders who know what “good data” looks like but prefer intuitive tools over code. It’s also ideal for teams that want to move fast, collaborate visually, and integrate seamlessly with engineering workflows when needed.

With Soda Cloud, you can browse datasets, add quality rules, test and publish contracts, and set up scheduled or on-demand verification. All from your browser.

## Why Cloud-managed?

* **Faster time to value** – no setup required
* **Accessible to everyone** – empower domain experts, not just engineers
* **Built for collaboration** – share, comment, and propose changes in a shared UI
* **Easily operationalized** – schedule tests and trigger verifications programmatically

Cloud-managed contracts are a powerful way to bring your organization together around trusted data.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F1ek7cp49FWQYZgVsBac0%2Fimage.png?alt=media&amp;token=dd732f32-ec9b-41d0-8bf7-b19eab6ef1d9" alt=""><figcaption></figcaption></figure>

## Prerequisites

Before creating contracts in Soda Cloud, make sure:

* You have a **Soda Cloud account**
* You have access to an organization in Soda Cloud
* You have connected at least one **data source** via a Soda Runner [Via Soda Cloud](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud)

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Author a contract in Soda Cloud

Once your dataset is onboarded, you can begin defining the expectations that make up your Data Contract.

## Creating a Contract

To create a Data Contract, navigate to any **onboarded dataset** and click **Create Contract**.

{% hint style="warning" %}
This action requires the **Manage contract** permission on the dataset. Learn more about permissions here: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FQb5XYVm85WYaeASBnEUb%2Fimage.png?alt=media&amp;token=1cf296d0-34b0-42eb-9d28-337b5bbfd51b" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
**Create Contract** will return a **contract template skeleton** based on your data. This will facilitate check creation, since the bones of the contract are already in place. If you wish to start with an **empty contract file**, simply click on **Skip to empty contract** when prompted.
{% endhint %}

You’ll be taken to the **Contract Editor**, a powerful interface where you can define your contract in two ways:

* **No-code view**: Point-and-click UI to add quality checks and configure settings
* **Code view**: YAML editor for full control and advanced use cases.

> See language reference: [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference)

You can switch between views at any time using the **editor toggle** in the top right corner.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F16IN5cOuanjIdaAtftjn%2Fcontracttoggle.gif?alt=media&amp;token=82b84fe4-68c4-46b7-be73-a94e97848622" alt="" width="563"><figcaption></figcaption></figure>

{% hint style="success" %}
You can edit your contract **manually**, whether it is with the code or no-code interface, **or you can use** [**Contract Copilot**](https://docs.soda.io/soda-documentation/soda-ai/contract-copilot) to help you implement the checks you need automatically.
{% endhint %}

## Key concepts in contract authoring

Understanding how to structure your contract is essential. Soda supports several types of checks and configuration options:

* **Filter**: applies a global filter to limit which rows are considered across the entire contract (e.g., only the latest partition or rows from the past 7 days.)
* **Variables:** help you **parameterize** your contract, making it flexible and adaptable to different contexts (e.g., environments, schedules, or partitions.)
* **Dataset-level Checks**: rules that apply to the dataset as a whole, like row count, freshness, or schema checks.
* **Column-level Checks**: rules that apply to individual columns, like missing values, uniqueness, ranges, or regex formats.

All visible columns are detected during onboarding. You can also manually add columns if needed.

### Use variables

Variables allow **dynamic substitution of values** in contracts. They help you:

* **Parameterize** values that differ across environments, datasets, or schedules.
* **Reuse values** in multiple places within the same contract to reduce duplication and improve maintainability.

You can define variables at the top of your contract:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FPZF5sBHC0TSTE2YR5ilu%2Fimage.png?alt=media&amp;token=c9080d51-3a35-4641-9de8-cfa0677880f1" alt=""><figcaption></figcaption></figure>

Then use them throughout your contract using the `${var.VARIABLE_NAME}` syntax.

For example:

```javascript
filter: timestamp = "${var.partition_timestamp}";
```

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FLVWRVjItiKkcVNEdPMQ1%2Fimage.png?alt=media&amp;token=bf7d58a6-d316-4724-9214-36a935898c41" alt=""><figcaption></figcaption></figure>

You can **test the contract with variables**:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fti7uSlAfcwtIbHRvQTdu%2Fimage.png?alt=media&amp;token=e28a9f14-11b8-46aa-9d6b-8d63ae4ce8f1" alt=""><figcaption></figcaption></figure>

When running the contract, variable values must be provided unless a default is defined.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fuok4RWVWSeGCVT17pN7e%2Fimage.png?alt=media&amp;token=796e7eea-e0e8-459b-b78b-33746a97cef7" alt="" width="563"><figcaption></figcaption></figure>

Variables are ideal for partitioned datasets, date-based rules, or customizing checks based on context.

#### Out of the box variables

**Now**: You can use `${soda.NOW}` in your Contract to access the current timestamp.

***

### Define Attributes

Use attributes to **label**, **sort**, and **route** your checks in Soda Cloud. Attributes help you organize checks by properties such as domain, priority, location, and sensitivity (e.g., PII).

> Learn how to leverage attributes with [Notifications](https://docs.soda.io/soda-documentation/fix/notifications) and [Browse datasets](https://docs.soda.io/soda-documentation/understand/manage-issues/browse-datasets).

**Set default attributes at the top level**

You can define default attributes at the dataset level. These attributes apply to **all checks**, unless overridden at the individual check level.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fml61k90fMKfNQlRLKq5B%2Fimage.png?alt=media&amp;token=46fb5250-bec7-4907-af28-5d2d7cc37674" alt="" width="563"><figcaption></figcaption></figure>

**Apply attributes to checks**

You can add attributes directly to individual checks. For example:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FJtPjltdKzcW887RWRrvl%2FScreenshot%202025-07-11%20at%2011.46.14%E2%80%AFAM.png?alt=media&amp;token=d9e6933f-67af-4d36-90c7-2b5bf66b8756" alt="" width="375"><figcaption></figcaption></figure>

#### **Attribute Validation in Soda Cloud**

When publishing contract results to Soda Cloud, **all check attributes must be pre-defined in Soda Cloud**. If any attribute used in a contract is not registered in your Soda Cloud environment, the results will **not be published**, and the data contract scan will be **marked as failed**.

> Learn how to configure attributes in Soda Cloud: [Check and dataset attributes](https://docs.soda.io/soda-documentation/govern/check-and-dataset-attributes).

***

## Testing the Contract

Before publishing, click **Test** (top right) to simulate a contract verification against your live data. Soda will:

* Run all defined checks
* Display which rules pass or fail
* Surface profiling and diagnostic insights

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fc40q8MCMB9zLuX7CkhvA%2Fimage.png?alt=media&amp;token=686795cd-4892-4590-863a-1a7e5751376e" alt="" width="563"><figcaption></figcaption></figure>

This dry run helps ensure your contract behaves as expected, before making it official.

{% hint style="warning" %}
This action requires the "**Manage contract"** permission on the dataset. Learn more about permissions here: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

### Test a contract on a data sample

> You can test a contract on a sample of your data. Learn more at the onboarding [Additional settings](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud/additional-settings#test-a-contract-on-a-sample).

## Publishing the Contract

Once you're happy with the results, click **Publish** (top right).

Publishing sets this version as the **source of truth** for that dataset. From this point on:

* Verifications will use the published version
* All users see this contract as the authoritative definition of data quality for that dataset
* Changes will require a new version or a proposal (depending on permissions)

Publishing ensures your data expectations are versioned, visible, and enforceable.

{% hint style="warning" %}
This action requires the **Manage contract** permission on the dataset. Learn more about permissions here: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

***

You’re now ready to start verifying your contract and monitoring your data.

## Contract history

**Contract history** provides a snapshot view of all changes that have been made to a contract.\
No retention policy applies to contract history. All contract records are preserved in full without time limitation.

To access contract history:

{% stepper %}
{% step %}
**Navigate to a dataset** with an existing data contract.
{% endstep %}

{% step %}
**Click on the** <i class="fa-clock">:clock:</i> **icon** next to **Edit Contract**, on the top right (or click on **Edit Contract >** <i class="fa-clock">:clock:</i>).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FfK5nTlHsdTao3IbGEJbB%2Fimage.png?alt=media&amp;token=ca9bf482-1f90-4850-976e-ba0053f2968e" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Review contract history** by choosing a version on the left panel and inspecting it on the right panel.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FWjeqShwwrkl0kIPfNav6%2Fimage.png?alt=media&amp;token=c1637a61-8bf7-4306-9972-561faf45a8b7" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

Just as when a contract is being created or edited, you can toggle between the code and no-code views.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fe4bWjpnFbo4LWgyHp8Ao%2Fimage.png?alt=media&amp;token=851e3f2c-6176-4a74-9fae-be7d18f7d446" alt=""><figcaption></figcaption></figure>

The code view allows to toggle diff and toggle split view.

{% columns %}
{% column %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F8kLd8Igou4Ysef9wa9Tp%2Fimage.png?alt=media&amp;token=d9493bce-03e5-4b99-84f6-4c3c82eff6da" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FqbByA5R8QA2at8AuUta8%2Fimage.png?alt=media&amp;token=3a640705-34dc-4a03-b27f-d367c92f6bf0" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

### Request history

While contract history allows to see the changes that a contract has undergone, **request history** provides an overview of the **change requests** that have been made over a specific contract.

#### View dataset request history

To access the request history of any dataset, navigate to the dataset > **Requests** tab.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FazCUu3syCCuutOT32LF1%2Fimage.png?alt=media&amp;token=f6eaa861-8e1f-420e-864b-56e9bf237072" alt=""><figcaption></figcaption></figure>

The list of requests can be filtered by **title key word** and by **state** (**Open**, **Done**, **Won't do**, **Draft**, **Ready for review**, and **Waiting for feedback**)

> From this view, you can also [create a request](https://docs.soda.io/soda-documentation/find/data-testing/contract-collaboration#initiating-a-request).

This view provides a snapshot of each request, making visible:

* The title, description (if any), and time of creation of the request
* The state of the request (**Open**, **Done**, **Won't do**, **Draft**, **Ready for review**, and **Waiting for feedback**)
* The <i class="fa-lightbulb">:lightbulb:</i> icon, which indicates that the request has a proposal
* The <i class="fa-message-lines">:message-lines:</i> icon, which indicates that the request has comments

{% columns %}
{% column %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F9QnBGZ4OTPD0imIoh8za%2Fimage.png?alt=media&amp;token=0ad89a8e-ed81-4a2f-a528-2335aeb2866c" alt=""><figcaption></figcaption></figure>
{% endcolumn %}

{% column %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FAtqYPcklaYQ7xFEz8pTs%2Fimage.png?alt=media&amp;token=16047d72-0f6c-4070-a664-bd0fc9fce09f" alt=""><figcaption></figcaption></figure>
{% endcolumn %}
{% endcolumns %}

#### View organization request history

To access all request history in an organization, navigate to tab **Requests** on top of the page.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FNQmx2fOCZ9DHqsYh6Kwn%2Fimage.png?alt=media&amp;token=9f00bf17-0782-4744-beb2-0ccb3997012a" alt=""><figcaption></figcaption></figure>

This page provides an overview of **all requests** made within the organization. The requests can be filtered by:

* Title key word(s)
* Status
* User that created the request
* Users that are participants in the request

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Verify a contract

Once a Data Contract is published, the next step is to **verify** that the actual data matches the expectations you’ve defined. Soda offers several flexible ways to execute contract verifications, depending on your needs and technical setup.

***

## Manual execution (from the dataset page)

You can manually verify a contract at any time from the dataset page in Soda Cloud.

Simply open the dataset and click **Verify Contract**. This will:

* Execute the checks in the published contract
* Use the latest available data
* Display pass/fail results directly in the UI

This is especially useful for one-off validations, exploratory testing, or during incident investigations.

If the contract has scopes, the button follows the pill you select on the **Checks** tab, such as **Verify scope** to run only the checks of one scope.

{% hint style="warning" %}
This action requires the "Manage contract" permission on the dataset. Learn more about permissions here: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Ficgr7nMZ0cnzgctuEKwA%2FScreenshot%202025-05-29%20at%209.13.41%20PM.png?alt=media&amp;token=5823357d-bf18-428e-b3c5-d4b2123feb49" alt=""><figcaption></figcaption></figure>

## Scheduled execution (from the Contract Editor)

To monitor data quality over time, set up **scheduled verifications** directly in the contract editor.

When editing or viewing a contract:

1. Go to the **Schedule** section
2. Choose how often you want the contract to be verified (e.g., hourly, daily, weekly)
3. Choose your time zone
4. Choose a cron expression

Soda Cloud will automatically run the contract at the specified intervals, using the selected runner. All results will be stored and visualized in Soda Cloud, with alerts triggered when rules fail (if configured).

{% hint style="info" %}
If the data source has a **default schedule** configured, the schedule, timezone and cron expression pre-fill from it; pre-filled values display an **A** (auto) indicator. You can override either value before saving. Once saved, the contract keeps its own schedule independent of the data source default.

If no default schedule is set on the data source, the schedule defaults to **midnight UTC**.
{% endhint %}

{% hint style="info" %}
If the contract has scopes, this schedule runs only the checks without a scope. Each scope has its own schedule, or runs only manually or programmatically.
{% endhint %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F7p4GZR2fVkzzKsLJMB6r%2Fimage.png?alt=media&amp;token=d70168a8-93b4-42cb-8575-6fe283482771" alt="" width="563"><figcaption></figcaption></figure>

## Programmatic execution (via CLI)

For advanced workflows and full automation, you can verify contracts programmatically using the Soda CLI and a Soda Runner.

This is ideal for:

* CI/CD pipelines
* Custom orchestration (e.g., Airflow, dbt Cloud, Dagster)
* Triggering verifications after data loads

#### **Step 1: Connect to Soda Cloud**

First, create a Soda Cloud configuration file:

```bash
soda cloud create -f sc_config.yml
```

This generates a basic config file. Open it and fill in your API key and organization details.

> Learn how to [generate API keys](https://docs.soda.io/apis/generate-api-keys).

You can test the connection:

```bash
soda cloud test -sc sc_config.yml
```

#### **Step 2: Verify a Contract**

Now you can run a verification using the CLI and a remote Soda Runner.

To verify a dataset **without pushing the results to Soda Cloud**:

```shellscript
soda contract verify --dataset datasource/db/schema/table --use-runner --soda-cloud sc_config.yml
```

This allows you to verify that the contract produces the expected results before pushing results to Soda Cloud.

To verify and **also push the results to Soda Cloud**:

```bash
soda contract verify --dataset datasource/db/schema/table --publish --use-runner --soda-cloud sc_config.yml
```

This makes the verification results available in the UI for stakeholders, trigger notifications and monitoring dashboards.

{% hint style="warning" %}
This action requires the "**Manage contract**" permission on the dataset; the user is identified based on the API key provided in the Soda Cloud configuration. Learn more about permissions here:[Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

If Soda Cloud cannot return a contract for the dataset, for example because none is published yet, the command verifies nothing and exits with code 3. One error line names the dataset and the reason:

```
🚨 Could not fetch the contract for dataset 'datasource/db/schema/table': the dataset has no published contract in Soda Cloud
```

> Learn what each exit code means: [Core CLI reference](https://docs.soda.io/soda-documentation/developer-tools/cli-reference#exit-codes)

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Review checks

This page describes the Soda Cloud check page navigation.

After creating and verifying a Cloud-managed data contract, you can review the results of its checks directly in Soda Cloud. The **Checks** page provides visibility into how each check performed during the most recent verification and over time.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FSsKr7njg0SZaJqDXX4PJ%2Fimage.png?alt=media&amp;token=515a430c-582a-4d5f-9018-d0ba57d8f469" alt=""><figcaption></figcaption></figure>

In the **Checks** page, next to each check, Soda displays the overall verification outcome. The contract can have one of the following states:

* **Passed**: All checks met their defined expectations.
* **Failed**: One or more checks violated the contract.
* **Error**: The contract could not be evaluated due to configuration, connectivity, or runtime issues.

Each individual check is listed with its status. This allows you to quickly identify which rule failed and assess its impact.

### Accessing check results

You can access a check page in Soda Cloud in multiple ways:

* Navigate to the dataset > **Checks** tab > Select the check you want to inspect
* Navigate to the **Checks** page (top menu) and search for the check you want to inspect

### Understanding individual check results

Selecting a check opens its detailed view. This page includes:

#### Historical plot

Check results are stored for each verification run. You can review previous runs to understand how a check has behaved over time. This helps identify recurring issues, assess the impact of upstream changes, and validate that fixes have resolved the problem.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FL9F3PhFbXBBHgPfazBfC%2Fimage.png?alt=media&amp;token=0f86e6b2-74a9-457b-8191-864be9ec1d39" alt=""><figcaption><p>Check page for the <code>Email must be valid</code> check in the <code>dim_customer</code> dataset</p></figcaption></figure>

#### Result tab

The **Result** tab provides a concise summary of the outcome of the most recent contract verification for the selected check. It answers the primary question: did this rule pass or fail, and what was the measured value?

At the top of the tab, Soda displays the timestamp of the most recent verification run. Below, a result table summarizes the execution with four fields: the check status, the measured value, the time of execution, and any linked incident.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fhg4mSVsii2wHZo5gmXyN%2Fimage.png?alt=media&amp;token=0cb38dd8-0d15-4347-8faa-1d93bb0b12a5" alt=""><figcaption></figcaption></figure>

#### Diagnostics tab

Each check includes a **Diagnostics** tab that explains how the result was computed. Diagnostics provides execution metadata that supports transparency and troubleshooting by showing exactly how Soda evaluated the contract condition.

The tab displays a list of diagnostic properties for the most recent run:

<table><thead><tr><th width="220">Property</th><th>Description</th></tr></thead><tbody><tr><td><code>Failed rows count</code></td><td>The number of rows that failed the check in this run.</td></tr><tr><td><code>Failed rows percent</code></td><td>The failed rows count expressed as a percentage of <code>Check rows tested</code>. Populated when <code>Check rows tested</code> is available.</td></tr><tr><td><code>Dataset rows tested</code></td><td>The total number of rows in the dataset evaluated during the run.</td></tr><tr><td><code>Check rows tested</code></td><td>The number of rows directly tested by the check. For row-level checks (missing, invalid, duplicate), this is populated automatically. For failed rows query checks, this is only populated when <code>rows_tested_query</code> is configured on the check.</td></tr></tbody></table>

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FZCzGXvWJfaBhT3ezeNJw%2Fimage.png?alt=media&amp;token=6986967d-0846-4252-9801-72c417b46e13" alt=""><figcaption></figcaption></figure>

#### Failed rows tab

When available, failed rows are stored externally in your own data source to guarantee security and privacy. On this tab, Soda provides **failed rows queries** and, if configured, direct access your Diagnostics Warehouse to view failed rows.

If [Diagnostics Warehouse](https://docs.soda.io/soda-documentation/understand/diagnostics-warehouse) is enabled, the **Failed Rows** tab provides access to the specific records that violated the rule. This allows you to:

* Inspect the raw data that caused the failure.
* Validate business assumptions.
* Share concrete examples with data producers.
* Accelerate root cause analysis.

When failed row storage is not enabled, this tab remains visible but inactive.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FA7Ap6cqMKwadsOit3Zda%2Fimage.png?alt=media&amp;token=d228d5c9-ebc2-4659-8c8c-8c128d18f32c" alt=""><figcaption></figcaption></figure>

***

Reviewing checks in Soda Cloud closes the feedback loop between **defining** expectations and **enforcing** them. Data contracts establish what “good data” means. Each **check** page shows whether your data currently meets that standard and provides the information needed to investigate and resolve any violations.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Contract collaboration

Data contracts define the expectations between data producers and data consumers, ensuring that the data delivered is fit for purpose and aligned with business needs. However, data requirements evolve, and consumers often identify gaps or new use cases that require adjustments.

To support this, Soda provides a collaborative process that allows **data consumers to request changes** to existing data contracts or propose the creation of new ones. Consumers can directly propose changes by editing the data contract with **Soda's no-code editor**, suggesting concrete modifications for the dataset owner to review.

This approach enables data consumers to express their requirements not just in abstract terms but in actionable, implementable contract changes. By doing so, the consumer helps the dataset owner by:

* Making their needs clearer and more concrete.
* Supporting faster alignment between producers and consumers.
* Contributing to quicker and smoother implementation.
* Reducing unnecessary communication overhead.

The dataset owner remains the final decision-maker, reviewing proposed changes, iterating with the consumer as needed, and then publishing the updated contract once consensus is reached.

This collaborative workflow ensures that **data contracts remain living agreements** that continuously adapt to evolving business use cases while maintaining producer accountability.

> In Soda Cloud, you can access a view of the **contract history**, which allows to inspect all changes made to a specific contract.
>
> **Learn more about** [Author a contract in Soda Cloud](https://docs.soda.io/soda-documentation/find/data-testing/cloud-managed-data-contracts/author-a-contract-in-soda-cloud#contract-history)**.**

## Explore requests

The **Requests** dashboard centralizes all data contract change requests in one place, making collaboration and review explicit and traceable.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FOMaEBWxXDMJ50x0647pR%2Fimage.png?alt=media&amp;token=e7e791c6-2a07-41d0-8eff-7dc52094adc2" alt=""><figcaption></figcaption></figure>

#### Browse requests

Browse all requests across datasets, with key context such as status, author, participants, comments, and creation date. You can **view**, **edit** and **update** any request by clicking on it from the list and navigating the side panel.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FPXo7B1gVFFbJ5cdHOCA5%2Fimage.png?alt=media&amp;token=2bd56641-eb1b-4b62-a1f5-332b5f8a492b" alt=""><figcaption></figcaption></figure>

#### Filter requests

You can filter requests by:

* <i class="fa-lightbulb">:lightbulb:</i> **Request:** the name of the request
* <i class="fa-table">:table:</i> **Dataset:** the location of the request
* <i class="fa-circle-info">:circle-info:</i> **Status:** "Open", "Done", or "Won't do"
* <i class="fa-user">:user:</i> **Author:** the user who created the request
* <i class="fa-user-group">:user-group:</i> **Participants:** the users that took part in the request

#### Save request collections

**Save filtered views as collections** for quick access and share them with your organization.

## Initiate a request

{% hint style="warning" %}
This action requires the **Propose checks** permission on the dataset. Learn more about permissions here: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

Users can:

* **Request a change** by simply describing their needs and use cases.
* **Propose changes directly** by editing the data contract, suggesting concrete modifications for the dataset owner to review.

When a request is created, dataset owners automatically receive an email notification, ensuring they can promptly review and collaborate with the requester.

### Initiate a request with a proposal

To propose a change or create a new contract, data consumers can initiate a request directly from the dataset page.

1. **Navigate to a dataset**\
   Go to any onboarded dataset in Soda.
2. **Start editing**
   * If the dataset does not yet have a contract, click **Create Contract**.
   * If a contract already exists, click **Edit Contract**.
3. **Make changes**\
   Update the contract based on your needs and use case. You can add, modify, or remove elements to ensure the contract reflects the requirements you want to address.
4. **Create a new request**\
   After making your edits, click **Create a Request**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F9bv0DFTfx6qNxRuFGWXS%2Fimage.png?alt=media&amp;token=7a4b52d1-fad5-4359-8efa-da0419307996" alt=""><figcaption></figcaption></figure>

5. **Provide details**\
   You will be prompted to:
   1. Enter a **title** for the request.
   2. Provide a **reasoning** or description of the changes, explaining why they are needed.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FuoPq5Kf3bFyiPXvycXvU%2Fimage.png?alt=media&amp;token=34610ea4-ea84-4e6a-ad1f-ae9fbe4fbd34" alt=""><figcaption></figcaption></figure>

6. **Save the request**\
   Once you click **Save**, a new request is created containing your proposed changes. This proposal is then shared with the dataset owner for review and follow-up.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F8HuESYwALQysfGdCa80p%2FScreenshot%202025-08-20%20at%2010.13.20%E2%80%AFAM.png?alt=media&amp;token=742f84aa-9550-4e73-ae74-1a1165742e66" alt=""><figcaption></figcaption></figure>

### Initiate a request without a proposal

In some cases, data consumers may want to request changes without directly editing the contract themselves. This allows them to highlight a need while leaving the implementation details to the dataset owner.

1. **Navigate to the dataset**\
   Open the dataset in Soda.
2. **Go to the Requests tab**\
   Select the **Requests** tab for that dataset.
3. **Create a new request**\
   Click **Create a Request**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fn925aiwjdL1bDkUpWwuN%2FScreenshot%202025-08-20%20at%2010.18.33%E2%80%AFAM.png?alt=media&amp;token=40b0e157-ea50-4ad1-8759-6fd245cefffb" alt=""><figcaption></figcaption></figure>

4. **Provide details**\
   You will be prompted to:
   1. Enter a **title** for the request.
   2. Provide a **reasoning** or description of the changes, explaining why they are needed.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FC01ftOwFapQnAhhEq26G%2FScreenshot%202025-08-20%20at%2010.20.09%E2%80%AFAM.png?alt=media&amp;token=23eebfbe-822f-4242-9018-e37ebbe4566d" alt=""><figcaption></figcaption></figure>

5. **Save the request**\
   Once you click **Save**, the request is created. The dataset owner will be notified and can review, clarify, and propose changes to the contract based on your input.

## Collaborate over a Request

Each dataset page includes a **Requests** tab where all requests related to that dataset are listed. From here, users can:

* **Search** for a request by name.
* **Filter** requests by status: *Open*, *Done*, or *Won’t Do*.
* **Click on any request** to access collaboration tools.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FNtegOcUCRqqaCo7wCJFG%2FScreenshot%202025-08-20%20at%2010.31.53%E2%80%AFAM.png?alt=media&amp;token=b3a5b9f6-8a7b-4883-afa3-9df337af2be3" alt=""><figcaption></figcaption></figure>

Once inside a request, users can collaborate in the following ways:

### Review Proposals

Click **View Proposal** to examine an existing proposal associated with the request.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FMQjyTBSJ1zpAzNSCsrbS%2Fezgif-8070ba2848fa4e.gif?alt=media&amp;token=45f2b96d-b739-495d-b072-8bd6bc5ecc99" alt=""><figcaption></figcaption></figure>

When viewing a proposal, visual indicators show exactly what has changed in the contract:

* **Blue icon** → element was **modified** (M).
* **Red icon** → element was **removed** (R).
* **Green icon** → element was **added** (A).
* **Blue dot** → a parent element has one or more **children that were updated**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fs0HGjRDFGB9bRt7yWHGb%2FScreenshot%202025-08-20%20at%2010.54.51%E2%80%AFAM.png?alt=media&amp;token=a34343af-7b7e-41a1-8a77-df387d36d5c0" alt="" width="375"><figcaption></figcaption></figure>

### Exchange Messages

Participants can post text messages within the request to clarify needs, align on requirements, and discuss next steps.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FRyopv5DkaNxPf4uvmjSb%2FScreenshot%202025-08-20%20at%2010.36.30%E2%80%AFAM.png?alt=media&amp;token=e4c7b216-a063-41cc-a800-a803bdc6a3de" alt=""><figcaption></figcaption></figure>

### Create a New Proposal

Users can contribute new proposals to move the request forward.

**Iterate on an existing proposal:** while viewing a proposal, click the **pen icon** to edit and build upon it.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F7FVMrDdv6VvwqU4G7VDk%2FScreenshot%202025-08-20%20at%2010.58.50%E2%80%AFAM.png?alt=media&amp;token=7c2fadf1-4ff5-463d-ba1c-de4c72f0d8dc" alt="" width="563"><figcaption></figcaption></figure>

**From scratch:** click **Add Proposal** to create a brand-new proposal.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FgVgHDpNZhPQC9Oa49iiB%2FScreenshot%202025-08-20%20at%2010.59.20%E2%80%AFAM.png?alt=media&amp;token=dbf57a0f-97e0-477d-82a2-af97c2f10456" alt="" width="375"><figcaption></figcaption></figure>

In both cases:

1. Make your edits.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FLyLKG0WofaZYHfs3dte3%2FScreenshot%202025-08-20%20at%2011.03.57%E2%80%AFAM.png?alt=media&amp;token=12c2ead5-6c08-4bd6-bbbf-644875061bb9" alt=""><figcaption></figcaption></figure>

2. Click **Save**.
3. Provide a message to explain what you have done.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FIlDOtobkSnKgf3IERy3E%2FScreenshot%202025-08-20%20at%2011.01.00%E2%80%AFAM.png?alt=media&amp;token=ff0f473b-2509-41dd-bd2a-09af80e5ae72" alt=""><figcaption></figcaption></figure>

4. Click **Save** again

All participants are automatically **notified by email** when a new proposal is created or an iteration is made, ensuring everyone stays aligned and can respond promptly.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FJ3DsVelotOqhtH6ZHZWy%2FScreenshot%202025-08-20%20at%2011.02.41%E2%80%AFAM.png?alt=media&amp;token=3aae0d85-fc6e-4b54-824a-e9e59dcf0b23" alt=""><figcaption></figcaption></figure>

## Publish a proposal

{% hint style="warning" %}
This action requires the **Manage Contract** permission on the dataset. Learn more about permissions here: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

### Publishing to Soda Cloud

After reviewing a proposal, you can **publish it** by clicking the **Publish** button. Once published, all participants associated with the request will automatically receive a notification, ensuring they are informed of the update.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FWxHgya8XxgRQT4qyv38F%2FScreenshot%202025-08-20%20at%2011.41.15%E2%80%AFAM.png?alt=media&amp;token=809bd98f-747d-4b12-87ae-688e56f803f2" alt="" width="375"><figcaption></figcaption></figure>

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FpV8hynfGZF3Dm4lL88Jz%2FScreenshot%202025-08-20%20at%2011.43.02%E2%80%AFAM.png?alt=media&amp;token=924489d2-3d4d-4146-92c1-632fa2b4b5c4" alt=""><figcaption></figcaption></figure>

#### Resolving conflict

In case a new version of the contract has been published, it is required to sync the proposal with the latest version to publish it.

1. When reviewing a proposal, click on **Sync to latest**

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F92chxEWjOulDtCSToP45%2FScreenshot%202025-08-20%20at%2011.43.54%E2%80%AFAM.png?alt=media&amp;token=d28f9c08-5417-4e59-bc4c-2ef2889660a5" alt=""><figcaption></figcaption></figure>

2. There are then 2 scenarios that can arise:
   1. Soda can **automatically merges the 2 versions**. You can then proceed to the next step
   2. **There are conflicts that Soda cannot resolve**. In this case, you will be required to resolve the conflict. Soda offers a tool allowing you to compare the latest published version (left side) with the version of the proposal (right side). You can then edit the proposal version to resolve the conflicts. Click **Continue** to proceed

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FcMEYXkTxVWgQdTINmbfX%2FScreenshot%202025-08-20%20at%2011.52.26%E2%80%AFAM.png?alt=media&amp;token=4fa08dc8-36af-4ad8-a265-70c3872bf0d4" alt=""><figcaption></figcaption></figure>

3. Optionally, do extra edits
4. Click **Save** to create a new proposal, which you can now publish

### Publish to Git

You can fetch the content of a proposal from Soda Cloud and save it as a contract file, which can then be published to Git. This allows you to incorporate approved changes into version-controlled data contracts.

```bash
soda request fetch -r 7 -p 1 -sc soda-cloud.yaml --f ./contracts/ecommerce_orders.yamlRe
```

<table><thead><tr><th width="225.433349609375">Parameter</th><th width="123.8499755859375">Required</th><th>Description</th></tr></thead><tbody><tr><td><code>-r</code></td><td>Yes</td><td>The request number. Identifies the request to fetch. Request numbers can be found when reviewing a proposal. See screenshot below.</td></tr><tr><td><code>-p</code></td><td>No</td><td>The proposal number. Defaults to the latest proposal if not specified. Proposal numbers are shown as the decimal part when reviewing a proposal. See screenshot below.</td></tr><tr><td><code>--soda-cloud, -sc</code></td><td>Yes</td><td>Path to the Soda Cloud config file (e.g., <code>soda-cloud.yaml</code>).</td></tr><tr><td><code>--f</code></td><td>Yes</td><td>Path to the output file where the contract will be written.</td></tr></tbody></table>

Request and proposal numbers can be found on Soda Cloud when reviewing a proposal. The first number is the request, and the decimal is the proposal.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F06tHO36Erwk9fxuT5NXe%2FScreenshot%202025-08-20%20at%2011.35.57%E2%80%AFAM.png?alt=media&amp;token=238404f7-735c-4e35-9dfc-8f13be255e05" alt="" width="375"><figcaption></figcaption></figure>

After fetching the proposal, you can optionally use the **publish command** to publish it from Soda Cloud to Git: [Core CLI reference](https://docs.soda.io/soda-documentation/developer-tools/cli-reference#publish-a-contract)

## Close a request

Each request in Soda has a **status** to reflect its lifecycle. Initially, a request is created in the **Open** state. Once the requested changes have been implemented and published, the request can be moved to **Done**. If the decision is made not to implement the request, it can be transitioned to **Won’t Do**. Whenever a request’s status is updated, all participants are automatically notified by email, ensuring transparency and alignment across the collaboration process.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FuT1rYEWv6eujhyQtVPI7%2FScreenshot%202025-08-20%20at%2012.21.41%E2%80%AFPM.png?alt=media&amp;token=f7648275-ded5-4033-9a23-fcfe379ebf5e" alt=""><figcaption></figcaption></figure>

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Data reconciliation

This page describes how Soda handles data reconciliation through different types of reconciliation checks.

**Reconciliation checks** are a **validation step** used to ensure that data remains consistent and accurate when moving, transforming, or syncing between different systems. The core purpose is to confirm that the **target data matches one or more source datasets,** whether that's during a one-time migration, a recurring data pipeline run, or ongoing synchronization across environments.

For instance, if you are migrating from a MySQL database to Snowflake, reconciliation checks can verify that the data transferred into Snowflake staging is intact and reliable before promoting it to production. This **minimizes the risk of data loss, duplication, or corruption during critical migrations**.

Beyond migrations, reconciliation checks are also used in **data pipelines and integrations**. They help validate that transformations applied in-flight do not compromise accuracy, and that downstream datasets remain coherent with upstream sources.

Other use cases include **regulatory compliance**, where organizations must prove that financial or operational data has been faithfully replicated across systems, and **system upgrades**, where schema changes or infrastructure shifts can introduce unexpected mismatches.

By systematically applying reconciliation checks, teams can maintain trust in their data, reduce operational risk, and streamline incident detection when anomalies arise.

## Multiple source datasets

Reconciliation supports comparing your target dataset against **multiple source datasets** simultaneously. This is useful when:

* **Multiple upstream systems** feed into a single target: for example, validating that a consolidated data warehouse matches both a staging environment and an operational database.
* **Cross-environment validation**: ensuring consistency across regional databases, different cloud providers, or separate business units that all contribute to one target.
* **Phased migrations**: during gradual cutovers, you may need to reconcile against both the old and new source systems until the migration is complete.

Each source is defined with a `name` and a `dataset`, and each check specifies which source it targets using the `source:` field. This gives you full control over which validations run against which upstream system.

> Learn more about the syntax in the [reconciliation checks reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks).

## Defining source datasets

Before defining reconciliation checks, you specify one or more **source datasets** in the `sources:` list. Each source represents a system of record against which you want to validate consistency. It is possible to define a **filter** on each source dataset, allowing you to reconcile only a subset of records that match certain criteria (for example, only transactions from the current month, or only rows belonging to a specific business unit).

For the **target dataset**, the reconciliation check applies the **dataset filter defined at the top of the contract** (see [Author a contract in Soda Cloud](https://docs.soda.io/soda-documentation/find/data-testing/cloud-managed-data-contracts/author-a-contract-in-soda-cloud#key-concepts-in-contract-authoring)).

Ensure that both source and target are constrained to the same logical scope before comparisons are made, keeping the validation consistent and relevant.

> A filter always applies to one **side** of the comparison. For the complete picture of which filter applies where, see the [filter application matrix](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks#filter-application-matrix).

<div align="center"><figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FqnZYM9jwi8Prdg3n0Xh7%2Fimage.png?alt=media&amp;token=43e754ab-65f8-4cf7-ab41-381589bc0f9e" alt="" width="450"><figcaption><p>In a data contract, configure a reconciliation check by defining the <strong>source dataset</strong> and applying <strong>aligned filters</strong></p></figcaption></figure></div>

## **Metric-Level Reconciliation**

At this level, aggregate metrics from the source and target datasets are compared. Examples include totals (e.g., revenue, number of rows), averages, or other summary statistics. This approach is efficient and provides a high-level signal that the data remains consistent. It is especially useful for large-scale migrations or pipelines where exact row-by-row comparison may not be necessary at all times.

### Thresholds

Comparisons at the metric level are evaluated against a defined threshold, which represents the **acceptable difference between source and target**. This tolerance can be set depending on the business context. Some use cases may allow small discrepancies (e.g., rounding differences), while others require exact equality.

When comparing integrity checks such as missing values, duplicates, or invalid entries, you can reconcile either by looking at the **raw count** of affected records or by comparing the **percentage metric** (e.g., the percentage of rows with missing values in each dataset). This flexibility ensures that reconciliation is meaningful regardless of dataset size or distribution.

### Check-level filter

In addition to dataset-level filters, reconciliation checks support **check-level filters**, which are applied consistently to both the source and target within the scope of a specific check. These filters make it possible to validate a **subset of the data** relevant to the context of the check. The check-level filter is applied **on top of any existing source or target dataset filters**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FDbeHODWtAgAy7XKGDhvu%2Fimage.png?alt=media&amp;token=2a43b107-2721-4f91-b3b5-6f916882c4c9" alt="" width="563"><figcaption></figcaption></figure>

## **Row-level reconciliation**

For more granular validation, reconciliation can be performed at the **row level**. This type of check surfaces detailed differences such as **missing records**, **mismatched values**, or **unexpected duplicates**. Row-level reconciliation is critical in scenarios where accuracy at the record level is non-negotiable—such as record that address financial transactions, user data, or regulatory reporting.

This requires specifying a **primary key (or a composite key)** to uniquely identify rows between the source and the target. Once rows are aligned, you can define a **list of columns to test** for exact matches or acceptable tolerances. If no column list is provided, the check defaults to comparing **all columns in order**. This flexibility ensures that comparisons can range from broad validation across the entire dataset to focused checks on only the most critical attributes.

{% hint style="warning" %}
[Rows diff](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks#rows-diff) reconciliation checks are **not supported in Synapse**.
{% endhint %}

{% hint style="warning" %}
A narrower set of reconciliation checks applies when the source is Salesforce. See [Salesforce](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/salesforce).
{% endhint %}

### Thresholds

Row-level reconciliation supports thresholds expressed either as the **count of differing rows** between source and target, or as the **percentage of differing rows relative to the source dataset row count**. These thresholds determine the acceptable level of variance before the check is considered failed, giving you fine control over sensitivity and tolerance.

This dual approach allows teams to adapt reconciliation logic to different contexts, using absolute counts when every record matters, and percentages when evaluating proportional differences in large datasets.

### Check-level filter

As with metric-level checks, you can define a **check-level filter** that is applied on top of any existing dataset filters. This allows you to reconcile only a targeted segment of data within the context of the specific check—for example, testing only a single business unit, product family, or date range.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FFvSbXSZrelE5DZ8PaQ0m%2Fimage.png?alt=media&amp;token=4c580abd-2af0-42b9-a845-952bc8824b7d" alt="" width="563"><figcaption></figcaption></figure>

#### Filtering each side independently

A single filter expression only works when the same SQL is valid against both datasets. That is often not the case: a legacy source and its migrated copy routinely differ in column names, status encodings, or partitioning conventions.

Every check type except schema diff can therefore filter each side on its own terms, with a **source filter** and a **target filter** on the same check. Both are optional and independent — set one, the other, or both. The source filter is combined with the filter declared on the source dataset itself, so both constraints apply.

The single both-sides filter remains available and unchanged; the two forms are mutually exclusive on the same check.

> Learn more about the syntax and the rules in [Filters](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks#filters).

### Keys and NULL values

Row-level reconciliation aligns the two datasets on their key columns, so a `NULL` key cannot be matched with anything. Rows with a `NULL` in a key column are **excluded from the comparison** on the side where they occur, counted in the check's diagnostics, and reported as a warning in the scan log. The check still evaluates on the remaining rows.

Pick keys that are unique and not nullable. A high count of excluded rows means a large part of the dataset was never actually compared.

### Values stored in different shapes

Two systems routinely hold the same fact in two shapes. A legacy source pads its text with trailing spaces where the migrated target does not; a source records a discount as `95` where the target records `0.95`. A plain row-level comparison reports every one of those rows as changed, even though the data agrees.

A row-level check can therefore carry a **per-column SQL expression** on either side — `TRIM(name)`, `amount / 100`, a cast — applied before the values are compared. Everything downstream sees the transformed value: the comparison itself, the check's diagnostics, and the failed-row samples in Soda Cloud, so what you look at is what was compared. Take care with expressions on **key** columns: normalizing a key can merge values that used to be distinct, and row-level reconciliation expects keys to be unique.

The same expressions are available on the [referential-integrity check](#referential-integrity-reconciliation), where they normalize the values before the two sides' distinct sets are compared — `UPPER(TRIM(code))` on both sides makes a code that differs only in case or padding count as the same code, rather than as one value missing on each side. There are no key columns there, so the caution above does not apply.

The aggregate and freshness checks name one column per side and take a single expression per side instead of a map — rescale a cents column before summing it, or cast a text timestamp before taking its maximum.

Between the three ways of shaping what a check compares, take the lightest one that fits: a **filter** chooses which rows are compared, a **per-column expression** reshapes one column's values, and a **custom query** replaces what is read on that side.

> Learn more in [Transform values before comparing](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks#transform-values-before-comparing) and [Choosing between filters, expressions, and custom queries](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks#choosing-between-filters-expressions-and-custom-queries).

### Reconciling derived data

Sometimes the two things you want to compare are not two tables. The source may need a join to be comparable, or the comparison may be meaningful only at an aggregated grain.

* **Views are the default path.** Create a view over the derived data on either side and onboard it as a dataset. Views behave like any other dataset in Soda: they are discovered and profiled, they can carry their own contract and data standards, and other checks can reuse them.
* **Custom SQL queries are the power-user path.** When creating a view is not possible in your organization, or the derivation genuinely belongs in the contract, a row-level check can take a full SQL query for the source side, the target side, or both. This buys flexibility at the cost of the guarantees a dataset gives you: the query is yours to keep deterministic, it is re-executed once per page, and it never gains discovery or profiling coverage.
* **For aggregate-level comparisons**, compare the result of two SQL queries as a single metric instead of a row set. That is far cheaper than a row-level diff when you only need to know *whether* the two systems agree, not *which* records disagree.

> Learn more in [Compare derived data with custom SQL queries](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks#compare-derived-data-with-custom-sql-queries) and [Metric diff](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks#metric-diff).

### Performance considerations

Row-level reconciliation is inherently **heavier** than metric-level reconciliation, as it requires comparing records across potentially large datasets. To enable comparisons even when data lives in different systems, data is loaded into memory from both the source and the target, where the diff is executed. A **paginated approach** is used to maintain scalability; this ensures that memory usage remains stable, but execution time will increase as the dataset size and column count grow.

### Benchmarks

| Dataset Shape          | Change Rate | Memory Usage | Execution Time |
| ---------------------- | ----------- | ------------ | -------------- |
| 10 columns, 500K rows  | 1% changes  | <80MB RAM    | **9s**         |
| 360 columns, 100K rows | 1% changes  | <80MB RAM    | **1m**         |
| 360 columns, 1M rows   | 1% changes  | <80MB RAM    | **35m**        |

**Recommendations**

* **Leverage filters to scope checks to new or incremental batches of data** wherever possible, rather than repeatedly reconciling the entire dataset. This reduces both execution time and operational overhead.
* Use **metric-level reconciliation as a first line of validation**. It is significantly more efficient and scalable, and can quickly highlight whether deeper row-level analysis is even necessary.

## Referential-integrity reconciliation

Sometimes the question is narrower than "are these two datasets the same". You only need to know whether every key on one side still has a counterpart on the other: foreign keys, code tables, and dimension keys that must stay aligned with a system of record living in another database. A reference reconciliation check compares the **distinct values** of one or more columns between source and target, and reports the values that exist on one side only.

Pick the direction that matches the relationship — the target constrained to the values of the source (the usual case when the source is the system of record), the source constrained to the target, or the two required to hold exactly the same set. Because the comparison is over distinct values, duplicates are irrelevant and no key column has to be declared; rows whose compared columns are `NULL` are left out on the side where they occur.

> Learn more in [Reference diff](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks#reference-diff).

## Schema-level reconciliation

Before comparing values, it is often worth asking whether the two datasets still have the **same shape**. A schema reconciliation check compares the actual columns and data types of the source dataset with those of the target, and reports which columns are missing on either side and which types diverge. Renamed columns can be mapped explicitly, and technical columns that exist on one side by design can be excluded.

This is distinct from the [schema check](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference#schema-check), which compares a **single** dataset against the structure declared in its contract. Schema reconciliation compares **two live datasets** against each other, which is what you want when validating that a migration or replication reproduced the source faithfully — regardless of what any contract declares. The two are complementary, and a contract can carry both.

Schema reconciliation is also the fastest way to explain a row-level check that cannot align its two sides: if the source and target project a different number of columns, a schema check tells you exactly which ones.

> Learn more in [Schema diff](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks#schema-diff).

## Caveats

### Data should be quiescent during a scan

Row-level reconciliation reads both sides page by page, and each page is a separate query execution — Soda does not hold a snapshot across them. If rows are inserted, deleted, or re-ordered while the scan runs, rows can be read twice or skipped, and the check reports differences that exist in neither dataset.

Schedule reconciliation for a window in which neither side is being written to, or constrain both sides to a **closed partition** whose rows can no longer change.

### One dataset can back several sources

Entries in the source list may reference the same dataset, each through its own filter — for example, two differently filtered slices of one dataset compared against the target. Where the slicing only differs per check, a single source with per-check filters keeps the contract shorter.

> See [Caveats and limitations](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks#caveats-and-limitations) for the full list, including data-source-specific restrictions.

***

## Implement reconciliation checks programmatically

Soda is suitable for no-code and programmatic users alike. If you are implementing checks programmatically, you can learn more about the **contract language syntax for reconciliation** on the [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks). Reconciliation checks can be used for both metric- and row-level validation.

> Learn more in the [reconciliation checks reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks).

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Failed Rows check

{% hint style="info" %}
In the Soda Cloud no-code builder, Failed Rows Checks are labelled **SQL Expression**, **SQL Query – Failed Rows**, and **SQL Query – Failed Keys**
{% endhint %}

## `expression` vs `query` vs `keys_query`

A `failed_rows` check flags individual rows that break a rule and sends them to the Diagnostics Warehouse (if it's set up).

* You must provide **exactly one** of `expression`, `query`, or `keys_query`.
* The parser rejects zero or more than one (`failed_rows_check_yaml.py`).
* All three support `metric: count` (default, threshold `= 0`) or `metric: percent`.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FYGBKisKpmqmbEzOrmUfj%2Fimage.png?alt=media&amp;token=63bdd1a3-5898-463f-acbf-e6a1e53a0236" alt=""><figcaption></figcaption></figure>

|                                  | **SQL Expression** (`expression`)                        | **SQL Query – Failed Rows** (`query`)          | **SQL Query – Failed Keys** (`keys_query`)                                                                                                  |
| -------------------------------- | -------------------------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| **What you write**               | A **boolean predicate** that is TRUE for a failing row   | A **full `SELECT`** returning the failing rows | A **full `SELECT`** returning only the key column(s) of the failing rows, which should be mapped to the Diagnostics Warehouse key column(s) |
| **How Soda counts failures**     | `COUNT(CASE WHEN <expr> THEN 1)` — Soda builds the query | Counts rows returned by your query             | Counts rows returned by your query (identical to `query`)                                                                                   |
| **Where failed rows are stored** | **Shared** diagnostics-warehouse tables (`fk_`/`fr_`)    | **Per-check** table (`frq_`)                   | **Shared** diagnostics-warehouse tables (`fk_`/`fr_`)                                                                                       |

#### When to use

* **Boolean condition on one dataset?** → **`expression`**. Simplest; Soda applies filters, derives passing rows, and computes rows-tested for you.
* **Need custom SQL (joins/aggregations) and want the failing rows themselves, stored per-check?** → **`query`**.
* **Same, but you want only key identifiers stored (hashed, in the shared diagnostics warehouse) for privacy/correlation, or your rule is key-based?** → **`keys_query`**.

### 1. SQL Expression (`expression`)

A **boolean SQL expression that evaluates to TRUE for a failing row**. It functions similar to a `WHERE`-clause predicate. Soda counts a row as failing only when the expression is TRUE, via `COUNT(CASE WHEN <expression> THEN 1)`; rows where it's FALSE **or NULL/UNKNOWN** pass (`failed_rows.py`, `failed_rows_check.py`).

The expression is transparent to Soda, which means this is the most capable form:

* Soda can **derive the passing (complement) set** of rows.
* Soda **applies the dataset filter and check-level `filter:`** at scan time. For a check in a [contract scope](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference#contract-scopes), Soda applies the scope's `filter` in place of the dataset filter.
* Soda **computes `check_rows_tested` automatically**, so `metric: percent` needs no extra query.

{% code overflow="wrap" %}

```yaml
checks:
  - failed_rows:
      expression: "discount > list_price"   # rows where this is TRUE are failures
```

{% endcode %}

{% hint style="success" %}
**Use it when** the rule is a row-level boolean condition on columns of the *same* dataset (ranges, allowed values, cross-column comparisons, null logic). It's the simplest and the only form where Soda understands and can scope the query.
{% endhint %}

### 2. SQL Query – Failed Rows (`query`)

A **verbatim, full `SELECT`** you supply; every row it returns is a failed row. Soda treats it as opaque SQL, which means it cannot compute the complement and **cannot express it as a dataset+filter AST**, so the **dataset filter is not applied** (`failed_rows.py`). The failing rows land in a **per-check diagnostics table** (`frq_`), whose view exposes the check ID as a `__soda_check_id` column.

{% code overflow="wrap" %}

```yaml
checks:
  - failed_rows:
      query: |
        SELECT o.* FROM orders o
        LEFT JOIN customers c ON o.customer_id = c.id
        WHERE c.id IS NULL          -- orphaned orders
      metric: percent
      rows_tested_query: "SELECT COUNT(*) FROM orders"   # required for percent
```

{% endcode %}

{% hint style="success" %}
**Use it when** the failing logic needs joins, aggregations, CTEs, or cross-dataset lookups **and you want the actual failing rows** captured/displayed. For `metric: percent` you must supply `rows_tested_query` for the denominator (the parser errors otherwise).
{% endhint %}

### 3. SQL Query – Failed Keys (`keys_query`)

Also a **verbatim, full `SELECT`**, but it returns only the **key/identifier column(s)** of the failing rows. **Counting is identical to `query`**. The returned keys are **hashed by Soda and routed into the&#x20;*****shared*****&#x20;diagnostics-warehouse tables (`fk_`/`fr_`)** rather than a per-check `frq_` table (`failed_rows_check.py`, `failed_rows_check_yaml.py`).

It additionally supports **`key_column_mapping`**, which maps each diagnostics-warehouse key-column name to the output column of your query that supplies it (identity mapping when omitted).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fpa28rdkSIL2fnhQwglqO%2Fimage.png?alt=media&amp;token=0c97fe1a-9dea-42dc-a834-29506450429e" alt=""><figcaption></figcaption></figure>

{% code overflow="wrap" %}

```yaml
checks:
  - failed_rows:
      keys_query: |
        SELECT order_id, line_no          -- composite key of failing rows
        FROM order_lines
        GROUP BY order_id, line_no
        HAVING COUNT(*) > 1               -- duplicates
      key_column_mapping:
        dwh_order_id: order_id
        dwh_line_no: line_no
      metric: count
```

{% endcode %}

{% hint style="success" %}
**Use it when** you want **key-based diagnostics stored centrally.**

The keys are hashed and share a warehouse, so this suits **not persisting full row payloads** (privacy/sensitive tables), and identifier-centric rules (dedup, orphans, cross-source mismatches). Same limitations as `query`: opaque to Soda, dataset filter not applied, `rows_tested_query` required for `percent`.
{% endhint %}

## Shared behavior & limitations

* Diagnostics carry three SQL fields end-to-end: `failedRowsSourceQuery`, `failedRowsDataQuery`, and `failedRowsKeysQuery` (`soda_cloud_failed_rows_diagnostic.py`).
* **Sample collection must be enabled** organization wide (Organization Settings → *Allow Soda to collect sample data and failed row samples*); default sample volume is **100 rows**, adjustable via the samples limit (set `0` to disable).
* **Dataset filters do not apply** to `query`/`keys_query`. Scope with a `WHERE` clause inside the query yourself. Only `expression` respects the dataset filter and check `filter:`.
* **Contract scope filters do not apply** to `query`/`keys_query` either. A scoped `expression` check uses its scope's `filter` in place of the dataset filter.
* In a [contract scope](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference#contract-scopes), a `missing`, `invalid`, `duplicate` or `failed_rows` expression check takes its failed rows from its scope: the samples, its keys in `fk_` and its rows in `fr_` all read the scope's `filter`. When Soda reads the source to store full rows and to check the unique keys, it covers the filters of every scope with at least one check that stored keys in `fk_`. When a `keys_query` check stores keys, scoped or not, Soda reads the whole source with no filter, since the query can return keys outside any filter.
* `rows_tested_query` is **only** used with `query`/`keys_query`
  * Supplying it with `expression` just logs a warning.
  * `key_column_mapping` is only used with `keys_query`.


# Data observability

An overview of Soda's key observability features and how they help catch data issues early.

## What is data observability?

Data observability is the ongoing process of **monitoring and assessing** the health of your data throughout its lifecycle. It focuses on analyzing metadata, metrics, and logs to detect issues as they arise, helping teams maintain trust in their data.

At the core of data observability are **monitors** that track key data quality metrics over time. When a metric behaves unexpectedly, anomaly detection algorithms analyze historical patterns to determine whether an alert should be triggered.

Typical data quality metrics to monitor are:

* **Schema changes** to surface structural modifications
* **Row counts** to detect unexpected changes in data volume
* **Most recent timestamps** to detect data freshness, missing or delayed data
* **Missing values** to track data completeness
* **Averages** to observe shifts in distributions

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fq1SthfKRzsmZ5jVWN2Zk%2Finteractive_plot.gif?alt=media&amp;token=aef7dbc0-27de-452c-8c7e-a127c1731d43" alt=""><figcaption></figcaption></figure>

## Soda’s practical approach to Data Quality

Soda embraces pragmatism over purity: practical outcomes and effectiveness are more important than ideal, unidimensional approaches. **Effective data quality comes from combining data observability and data testing**. Each serves a different purpose. Observability is about speed and broad coverage. Testing is about precision, enforcement, and prevention.

**Benefits of Data Observability**

* Enables broad coverage quickly, even across large data sources
* Surfaces unknown issues without needing to define every rule
* Requires minimal configuration to get started
* Leverages existing metadata for fast and efficient monitoring
* Provides early signals when something might be wrong

**Limitations of Data Observability**

* Serves only as a signal. An anomaly suggests an issue but doesn’t confirm it
* Can generate false alerts, since detection is driven by algorithms
* Requires further investigation to validate and resolve alerts
* Does not prevent issues. It flags them after they’ve happened
* May result in extra work to follow up and interpret alerts

**Start with Observability, but rely on Testing**

**Observability is a fast and efficient way to get initial coverage**. It helps surface unknown issues with minimal setup and delivers immediate value across many datasets. However, **for lasting reliability and trust in your data, testing is more important**.

Testing requires more effort up front. It involves **defining explicit expectations and rules** for your data. But that investment pays off. When a test fails, you know there is a real data quality issue, no guesswork, no false alerts. When an anomaly is detected, it doesn't necessarily mean there is an underlying data quality issue, and more investigation effort is required.

**For long-term reliability, testing is essential**. It adds rigor by enforcing defined standards and helps prevent bad data going into production. Start with your most critical datasets, then expand gradually using a collaborative approach, where business users help by proposing checks. This creates a **scalable framework** that grows with your organization while ensuring **lasting data quality**.

## What makes Soda’s Data Observability so useful?

Soda’s data observability allows teams to **monitor data health** across large environments **without manual setup**. All anomalies are surfaced in a single, easy-to-navigate dashboard, making it simple to spot issues and investigate patterns. Behind the scenes, a **proprietary anomaly detection algorithm** ensures high precision by minimizing false positives and focusing on meaningful deviations. Notifications are opt-in and alerts are only triggered when they matter, helping teams stay focused without being overwhelmed by noise.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fzi3r1ejQUBAdKFviXss7%2FScreenshot%202025-06-05%20at%2021.22.13.png?alt=media&amp;token=46e578c5-aaab-4858-a458-7b1985641851" alt=""><figcaption><p>Metric Monitoring Dashboard</p></figcaption></figure>

### Effortlessly monitor thousands of tables

Soda enables **large-scale observability** with ease. Instead of configuring each table manually, monitoring is applied at the data source level and automatically extends to all datasets underneath. This allows teams to activate observability across **hundreds or even thousands of tables** in minutes.

By leveraging metadata such as row counts, schema evolution, and insert activity, **Soda delivers lightweight and efficient monitoring**. There is no need to scan entire datasets or write custom logic for each table. You can do that if needed, but it is not required. Observability starts working immediately and is built to handle even the largest data platforms.

### Start today—look one year back

Observability is not just about what happens next. With built-in backfilling and backtesting, **Soda instantly analyzes historical metadata and metric trends**. From the moment observability is enabled, teams gain visibility into past data quality metrics and can detect potential anomalies that may have gone unnoticed.

This historical context is essential. It helps determine whether a current anomaly is truly new or part of an ongoing pattern. It also allows the anomaly detection algorithm to establish baselines immediately, which improves the quality of alerts from the very beginning.

### High precision alerts with fewer false positives

Soda’s **proprietary anomaly detection algorithm** is specifically designed for data quality monitoring. Every component has been developed entirely in-house without relying on third-party frameworks. This gives Soda full control over the modeling stack and ensures **transparency**, **customization**, and **explainability**. These attributes are especially important in production environments where trust in alerts is essential.

The algorithm is built on a **proprietary evaluation framework** that rigorously tests its performance using hundreds of internally curated datasets with known data quality issues. This framework enables structured, repeatable experimentation and continuous benchmarking of new techniques. It prioritizes reducing false positives to ensure alerts are accurate, meaningful, and reliable.

In benchmark testing, Soda’s algorithm demonstrated a **70 percent improvement in anomaly detection accuracy compared to Facebook Prophet**. Unlike generic forecasting tools that rely on rigid assumptions, Soda’s model is tailored to the real-world challenges of monitoring data quality at scale.

The system is flexible and adapts to different team needs. It can run autonomously with smart defaults or be fine-tuned through a user-in-the-loop approach. Teams can improve detection by providing feedback and adjusting sensitivity. This flexibility ensures that alerts remain focused, useful, and aligned with the needs of each organization.

## Metric Monitoring

Soda’s **Metric Monitoring feature** is the foundation of Data Observability, allowing users to automatically **track key dataset and column-level statistics** over time, detect deviations, and get alerted before data issues impact downstream analytics. While quality checks also keep track of measurements over time, metric monitors use that history of measurements to learn from them and automatically adjust thresholds to inform about expected values or alert about anomalies.

### Implement Metric Monitoring at scale

Metric Monitoring is developed to be a hassle-free feature. You can unlock organization‐wide observability through Soda Cloud’s [no-code dataset onboarding](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets):

* **Manual** dataset onboarding: allows you to browse and select specific databases, schemas or datasets to onboard them in bulk.
* **Rules-based** dataset onboarding: allows you to set rules and automate the dataset onboarding process, not only for current datasets, but also for future ones.

This instantly provides automated metric monitoring across hundreds of thousands of tables by simply selecting all the datasets you care about and defining a shared schedule in one step. No more configuring each table by hand: stay ahead of pipeline failures, data delivery delays, and structural changes with consistent, centralized monitoring that grows as fast as your data.

> Learn more about [dataset onboarding](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FZcIyIDaml6bTmnv8WON9%2Fimage.png?alt=media&amp;token=5c240247-339f-4aab-8ff2-9e87a1752c22" alt=""><figcaption></figcaption></figure>

***

> Learn more about how roles and permissions affect Metric Monitoring capabilities via the [Organization and Admin settings](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings).

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Metric monitoring

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FK3d17AjioFxE5kmyk6tx%2Fimage.png?alt=media&amp;token=c434fb09-5d4e-4d8e-b656-c1da85a977f3" alt=""><figcaption></figcaption></figure>

As soon as a data source is connected, the metric monitoring dashboard is available and will have **historical information**. Soda establishes a **statistical baseline** for each metric and continually compares new scan results against that baseline, **flagging anomalies** according to the sensitivity, exclusions, and threshold strategy you’ve configured.

## What are Metric Monitors?

**Metric monitors** are the foundation of data observability in Soda. Monitors track data quality metrics over time and leverage historical values for analysis. Soda automatically collects these metrics and examines how they evolve over time through a [proprietary anomaly detection algorithm](https://docs.soda.io/soda-documentation/find/data-observability#high-precision-alerts-with-fewer-false-positives) to identify when metrics deviate from expected patterns and trigger alerts. These deviations are surfaced and recorded in the **Metric Monitors** tab for each dataset.

### Monitors vs metrics: what's the difference?

The main difference between monitors and metrics is that **monitors are configurable, while metrics are not**.

Monitors build on top of metrics by wrapping their static measurement in a configurable context. Each monitor is customizable, so the user can select **scan time**, **scan frequency**, **thresholds**, and **metric** to be used.

Metrics, on the other hand, are only a part of the monitor. They are built-in, static definitions of data properties; it is not possible to alter how a metric is computed at its source, but it is possible to select which metric to track through a **metric monitor**.

## Types of monitors

A metric monitor tracks one data quality metric over time. Soda learns a baseline from the metric's history and flags new values that deviate from it.

Soda has four types of metric monitors. They differ in what they measure and how much setup they need.

| Monitor                                                                                                                            | Tracks                                             | Setup                                                                               | Example                                              |
| ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------- | ---------------------------------------------------- |
| [Dataset monitors](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors)   | A high-level metric for a whole table or partition | None for metadata-based monitors. Data-based monitors need a time partition column. | Row count drops, schema changes, data stops arriving |
| [Column monitors](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors)     | A statistical metric for one column                | Pick the columns and metrics to monitor                                             | Spike in missing values, shift in an average         |
| [Group By monitors](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/group-by-monitors) | A column metric, broken down per category          | Pick the metric and the column(s) to group by                                       | Missing values per region                            |
| [Custom monitors](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/custom-monitors)     | A metric you define with your own SQL query        | Write the SQL query. Enterprise only.                                               | A ratio computed across several tables               |

Start with dataset monitors to cover many datasets with no setup. Add column, Group By, and custom monitors on the datasets and columns that matter most.

All active monitors for a dataset appear on its **Metric Monitors** tab: dataset monitors at the top, column and Group By monitors at the bottom.

## Configure Opt-in Alerts

{% hint style="info" %}
Metric monitors do not send notifications on their own. Anomalies always appear on the dataset’s **Metric Monitors** tab, but nobody receives an email, Slack, or Microsoft Teams message until a notification rule covers the monitor. The default notification rules of an organization cover user-defined checks only, not metric monitors.
{% endhint %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FE4IxmcIjexuKhftoluzR%2Fopt_in_alert.gif?alt=media&amp;token=6941f2a1-b141-4c9d-ad2e-497d566e85d3" alt=""><figcaption></figcaption></figure>

You can turn any metric monitor into a proactive alert by clicking its **bell icon** on the Metric Monitors dashboard and selecting **Add Notification Rule**. This brings up the **Add Notification Rule panel**:

1. **Name**\
   Enter a descriptive title for your rule (e.g. “Row-Count Alerts – Prod Sales”).
2. **Data source**\
   Choose the warehouse or connection to scope your rule.\
   Then, search for and check the specific tables (or columns) this rule should cover. The “Matches X datasets” badge updates in real time so you know exactly what you’ll be alerting on.
3. **Applies to**\
   Pick which check type you want to alert on.
4. **Recipients**\
   Select one or more notification targets:

* Email addresses
* Slack channels
* Other integrations

This dialog lets you reuse a single rule for multiple datasets or checks, ensuring your team only gets the notifications they care about.

You can also create the same rule from the **Notification Rules** page in your profile menu: click **New Rule** and choose **All Monitors** or **Specific Monitors** as the rule scope. See [Notifications](https://docs.soda.io/soda-documentation/fix/notifications).

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Dataset monitors

Learn more about Metric Monitors that run scans at a dataset level.

## What is a dataset monitor?

A **dataset monitor** in Soda **tracks a specific high-level metric** for an entire table (or partition) over time. It helps detect **unusual patterns** or **unexpected changes** in overall data health, such as sudden spikes or drops in row count, delays in fresh data, or schema drift.

You can find dataset monitors by opening the **Metric Monitors** tab on any dataset and looking at the top section labeled “**Dataset Monitors**.” This section lists all active dataset monitors—both metadata-based and partition-based—in a clear overview of monitor cards. This overview provides, at a glance, critical information about the **status** of each monitor, the **value** of the last scan, and any detected **anomalies**, allowing you to have a one-look summary of the health of your data systems.

Unlike [column monitors](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors), which are configured at the dataset level but target individual columns, dataset monitors apply to the **entire table** (or its latest partition) and capture **broad indicators of data quality**. When the necessary data and metadata are available, dataset-level monitors work out of the box with no further configuration needed.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F8NsXlTxiWePUCFC5btia%2Fimage.png?alt=media&amp;token=4c832d09-07dd-49f1-9af9-c2423d0b8dce" alt=""><figcaption></figcaption></figure>

### Types of dataset monitors

Soda supports two categories of dataset‐level monitors: those that rely purely on **system metadata**, and those that compute values by **querying a designated time‐partition column**. Below is an in‐depth description of each built‐in monitor.

> For a more detailed discussion of monitors based on **querying the metadata** vs monitors based on **querying the data**, see the [Metadata vs Data-Based](#metadata-vs-data-based-monitors) section in this page.

| Dataset monitor type               | Monitor                | Description                                                                                            |
| ---------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------ |
| **Based on metadata**              | Total row count        | The total number of rows in the dataset at scan time.                                                  |
|                                    | Total row count change | Change in total row count compared to the previous scan.                                               |
|                                    | Last modification time | Most recent time the data was changed relative to the last scan.                                       |
|                                    | Schema changes         | Changes in the schema compared to the previous scan—any change is automatically flagged as an anomaly. |
| **Based on time partition column** | Partition row count    | The number of rows in the last partition at scan time.                                                 |
|                                    | Most recent timestamp  | Time difference between scan time and the maximum timestamp in the partition column (at scan time).    |

{% hint style="warning" %}
Monitors based on time partition columns look at data in the most recent partition **based on a timestamp**. If data is altered in an old partition, it will not be evaluated.

For example, data inserted today with timestamp of 2 days ago will not be evaluated if the partition interval is 1 day.

If your data always arrives late like this, set a delay. Soda then waits for the data before it analyzes a period. See [Late-arriving data](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/late-arriving-data).
{% endhint %}

{% hint style="info" %}
For **Schema changes**, the expected result is always to have no schema changes, regardless of whether there have been frequent schema changes in the past or not.
{% endhint %}

## Understanding the monitor card

The dashboard provides a **health table** summarizing an overview of the monitors. Each monitor card is clickable and links to the **Monitor History** page of the metric.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FXnMovu7HkB9L926C0rVi%2Fimage.png?alt=media&amp;token=42712ed6-106f-4193-9511-29b364c94010" alt=""><figcaption></figcaption></figure>

Each monitor card will have the following information:

* **Monitor name**: the given name of the specific monitor.
* **Monitor explanation**: a brief description of the metric used.
* **Status**: ✅ healthy / ⚠️ violated
* **Today's value at scan time**: last recorded value.
* **Expected range**: calculated by the anomaly detection algorithm, based on historical data.
* **Trend line with last 7 observations**: a sparkline that shows an overview of the monitor plot.
* **Bell icon**: to enable/disable opt-in alerts.

## Metadata vs Data based monitors

Dataset-level monitors fall into two categories depending on their source of truth:

#### Metadata-based dataset monitors

[Metadata-based monitors](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors/metadata-based) rely solely on **system metadata exposed by your data warehouse**; fields like “row count,” “last modified time,” or “schema version” that the catalog provides without scanning table rows. Because they don’t touch actual data, **metadata monitors are extremely efficient and run quickly**. They alert you if your table grows, shrinks, stops updating, or changes structure.

#### Data-based dataset monitors

[Data-based monitors](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors/query-based) look directly at the **contents of a designated time-partition column** (e.g., a `date` or `timestamp` field) and compute a value from the rows in that partition. Examples include “Partition Row Count” (how many rows landed in today’s partition) or “Most Recent Timestamp” (the newest timestamp in that partition). Data-based monitors require a full scan of each partition they monitor, but **they capture freshness and volume signals that metadata alone cannot provide**. If your dataset has no time-partition column defined (or your warehouse can’t surface the needed metadata), Soda will disable the appropriate monitors so you only see the metrics that can be collected.

## Configure Dataset Monitors

Use the **Configure Metric Monitoring** panel to pick which built-in metadata and partition-based metrics you want Soda to track at the dataset level.

1. **Open the panel** → From any dataset’s Metric Monitors dashboard, click **Configure Metric Monitoring** on the top right.

   <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FWOIBHIpXfmXLlAgLRxEV%2Fimage.png?alt=media&amp;token=dac71766-2ec4-44f6-9d14-327abf57e3be" alt=""><figcaption></figcaption></figure>
2. **Enable or disable** → Toggle metrics on/off directly from here. If the data source doesn't support a given metric, it will be automatically off.
3. **Modify the monitor**
4. **Auto-apply** → Changes take effect immediately for the next scan. Simply close the panel when you’re done.

### Time partition column

Many data‐based monitors—such as **Partition Row Count** and **Most Recent Timestamp**—rely on a designated “time partition” column to know which slice of data to scan. The time partition column should be a date or timestamp field that naturally groups rows into discrete, regularly updated partitions (for example, a daily `order_date` or `event_time`). When Soda cannot detect a time partition column, metrics based on that data will not be available.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FPEdTKImZCxcRsZOEnbFI%2Fimage.png?alt=media&amp;token=79ed8abd-9864-40ad-8397-0ddaa4e1ce77" alt=""><figcaption><p>When Soda cannot detect a time partition column, a warning banner will appear on the Metric Monitors tab.</p></figcaption></figure>

#### What's a good time partition column?

A good time partition column meets all of the following criteria:

1. **Date or timestamp type**: Each row contains a valid date (or timestamp) value.
2. **Regular arrival cadence**: New rows for each date/timestamp appear on a predictable schedule (e.g., daily, hourly).
3. **Reflects ingestion/arrival time**: The column’s value must correspond to when the record actually landed in this dataset, **not when it was originally created upstream**. The partition column should always show arrival date to the dataset so freshness checks remain accurate.
4. **Logical partition boundary**: It matches how you want to slice your data (e.g., `order_date` for daily sales, `event_time` for hourly logs).

When these conditions hold, partition-based monitors will reliably focus on the correct slice of data—namely, the rows that truly arrived during each time window—so any delays or backfills become immediately visible.

#### Suggest a time partition column during onboarding

When you onboard a new dataset from your data source, Soda attempts to **automatically detect** the most likely time partition column. You can:

* **Finish** onboarding without editing the Time Partition Column field, allowing Soda to detect it, or
* **Suggest** a Time Partition Column of your choice, forcing Soda to use that one for monitoring.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FB9ZllCUj0zuXsVz9WlsI%2F61f04dbd-51df-40d2-8d27-727615be3928.png?alt=media&amp;token=d1cbf0d3-1732-4048-a62f-b5562a46e501" alt=""><figcaption></figcaption></figure>

#### Find a time partition column

If you ever need to confirm or search for the right partition column:

1. Navigate to the **Datasets** page, select your dataset, and click the **Columns** tab.
2. Search columns with "timestamp" on them. Any column with a date or timestamp data type is a candidate.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FDFF1U1GpJf3HXGD7Hfrj%2Fimage.png?alt=media&amp;token=fb1fbf23-5c86-43ba-a679-daadceeffdcf" alt=""><figcaption></figcaption></figure>

#### Manually override time partition column

After onboarding, you can override the time partition column at any time. Changing it will reset Soda’s anomaly detection model for partition‐based metrics, so you’ll be retraining on historical data under the new partition definition. To override:

1. Acess the **Dataset Settings**

* Navigate to the **Datasets** tab

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FWJewUMMZbKkLeUhOlwL7%2Fimage.png?alt=media&amp;token=755d0f35-cbba-4b9b-891b-6af25fe9a27d" alt=""><figcaption></figcaption></figure>

* From this list or from the dataset page itself, click on the (⋮) menu > **Edit Dataset**

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FQth7XmpMQX97nKOXAJtW%2Fimage.png?alt=media&amp;token=4b16c822-b0b2-4e43-9e01-27b54ea9a3f5" alt=""><figcaption></figcaption></figure>

2. Find Time Partition Columns

* Click on the **Profiling & Metric Monitoring** tab

Here you will see the current column being used for Time Partition.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FOiCoyZDAYaLUMstmD9xb%2Fimage.png?alt=media&amp;token=26926c05-c5b3-4286-89ec-9c793336a4e3" alt=""><figcaption></figcaption></figure>

* Reveal the Time Partition Column drop-down menu

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fel55auWfYj12OA32JuE1%2Fimage.png?alt=media&amp;token=5d43cc98-d024-4c26-ac9c-968c7fcc67b5" alt=""><figcaption></figcaption></figure>

This will show all **date** and **timestamp** columns that can be used as a Time Partition Column.

3. Select your new Time Partition Column

{% hint style="danger" %}
Changing this column resets the model and historical baselines.
{% endhint %}

4. Click **Save**. Soda will:
   * Reset the partition‐based monitors (Partition Row Count, Most Recent Timestamp) to “training mode” and retrain baselines on the new partition.
   * Preserve any metadata‐based monitors (Total Row Count, Schema Changes) unchanged.

By following these steps, you ensure that Soda’s data‐based monitors always reference the correct daily (or hourly) slice of your dataset, so partition‐level metrics and freshness checks produce accurate results.

## Missing metric values

Even when a metric is enabled and historical baselines exist, you may occasionally see gaps due to delayed or skipped scans. A “missing” metric indicates that Soda attempted to run the scan but did not receive a valid result for that metric, either because the scan runner was down, the query timed out, or metadata couldn’t be retrieved in time. Missing values do not count as anomalies; they simply mark a gap in the time series.

In Soda Cloud, you can identify these gaps as follows:

* On the **Metric Monitors** dashboard, any missing value is shown either as a grey point or an empty checkbox in the metric sparkline:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fzul7JXiTMsCVlJ0WZ4xI%2Fimage.png?alt=media&amp;token=e3bcde18-474d-40da-8e09-6b6f71fe0c07" alt=""><figcaption></figcaption></figure>

* In the detailed anomaly plot, missing points render as open circles (◯) along the timeline, and the trend line becomes dashed.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FcgyBHs9oTkCuyBAGTzd5%2FCaptura%20de%20pantalla%202025-06-06%20214547.png?alt=media&amp;token=40447991-450a-4060-8775-7f360277bbe4" alt=""><figcaption></figcaption></figure>

* In **Schema changes**, no plot is available since the expected value is always 0. Hovering over an empty checkbox will display “No measurement” in the tooltip, making it easy to distinguish a gap from a healthy measurement or a flagged anomaly.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fg1p1cWRaj4M2h8Run1Jg%2Fimage.png?alt=media&amp;token=1546647a-b3a4-4d2a-8250-d2713955bd23" alt=""><figcaption></figcaption></figure>

These visual cues allow to immediately recognize when a scan didn’t complete successfully, enabling further investigation and restoration of full observability before critical issues go unnoticed.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Based on metadata

Learn more about metadata-based dataset monitors.

These monitors are derived directly from the data platform’s system metadata, without scanning row-level values. They surface structural signals, such as:

* [**Last modification time**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors/metadata-based/last-modification-time): when the dataset was last updated
* [**Schema changes**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors/metadata-based/schema-changes): any alterations to the schema
* [**Total row count**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors/metadata-based/total-row-count): the overall number of records in the dataset
* [**Total row count change**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors/metadata-based/total-row-count-change): the delta in row count compared to the previous observation

Because they read only metadata, these monitors are extremely lightweight to compute and ideal for continuous, real-time dashboarding of dataset activity.

> Learn more about which features are supported in **each** [**metadata data source**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/metadata-data-sources), such as historical backfilling or partition row count.

## Metric Monitoring on table views

{% hint style="warning" %}
Depending on your data source(s), **metadata-based Metric Monitors may only be supported on Tables**, and not on other data objects e.g. **Views**.

Alternatives could be to setup these metadata-based Metric Monitors on the source Tables of your non-Table data objects, and/or store these data objects as Tables instead.
{% endhint %}

Metric Monitoring on table views is currently available on:

* BigQuery
* Databricks
* Postgres
* Redshift
* Snowflake

{% hint style="info" %}

* **Metric Monitors on table views do not use metadata**; metadata is not provided for views.
* All monitors are supported except [Last modification time](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors/metadata-based/last-modification-time).
  * Last modification time must be **turned off manually**; Soda will attempt to enable it automatically as there is no structural difference between tables and table views.
    {% endhint %}

## Unavailable metadata

When Soda Cloud cannot obtain the underlying metadata required to calculate a dataset-level metric, it prevents you from configuring or viewing a metric that would always fail. There are two cases:

### **Non-retrievable metadata from data source**

If a connected data source cannot provide the required metadata for a given dataset-level metric, such as row counts or schema timestamps, **Soda will automatically disable that metric** both on the Metric Monitors dashboard and in the **Configure Dataset Monitors** panel so you only see and configure metrics that your source can actually collect.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FXx5db1L1Rytw5wAciRII%2Fimage.png?alt=media&amp;token=70bc5d6a-cfa6-455e-bfc3-38d14d2da614" alt=""><figcaption></figcaption></figure>

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fhhdqn0QzFk8zUB3pXdct%2Fimage.png?alt=media&amp;token=2ed308f5-20ed-4d60-a8d3-bf20b6973d53" alt=""><figcaption></figcaption></figure>

### Unavailable metadata (history)

Some warehouses expose current metadata but don’t provide historical snapshots (for example, systems that only track the latest row count). In this case, **Soda will compute the metric starting from your very first scan**, but it cannot backfill any history prior to that point. As a result, anomaly detection baselines for that metric begin only at scan #1 and there is **no retroactive historical data** to train against.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FmPyU7YNXaNNVcZFWmVsE%2Fimage.png?alt=media&amp;token=c92388d6-d9d7-4b27-9976-207772121977" alt=""><figcaption></figcaption></figure>

The **Schema changes** monitor will not add historical metadata and backfilling will not be available, unlike with other metrics. The monitor only starts recording from the moment the dataset is onboarded.

## Supported data sources

<table><thead><tr><th width="185.79998779296875">Data source</th><th width="171.20001220703125">Onboarding</th><th>Metadata</th><th>Metadata history</th><th>Querying data</th></tr></thead><tbody><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/amazon-athena">Athena</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/bigquery">BigQuery</a></td><td>✅</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/databricks">Databricks SQL</a></td><td>✅</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/dremio">Dremio</a></td><td>✅</td><td>Upcoming</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/duckdb">DuckDB</a> (<a href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/pandas">Pandas</a>, <a href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/polars">Polars</a>, etc.)</td><td>Partial/Python only</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/microsoft-fabric">Fabric</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td>MariaDB</td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr><tr><td>MySQL</td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/oracle">Oracle</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/postgresql">PostgreSQL</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/amazon-redshift">Redshift</a></td><td>✅</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>SAP HANA</td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/spark-dataframe">Spark Dataframe</a></td><td>Partial/Python only</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/snowflake">Snowflake</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/sql-server">SQL Server</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/supabase">Supabase</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/synapse">Synapse</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/trino">Trino</a></td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr></tbody></table>

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Last modification time

#### Definition <a href="#definition" id="definition"></a>

The elapsed interval between the scan time and the timestamp of the most recent change to the database. This includes any change to the data (inserts, updates, deletes) as well as any change to the schema.

#### Source <a href="#source" id="source"></a>

metadata

#### Computation <a href="#computation" id="computation"></a>

The compute method depends on the database. Soda requires specific metadata fields that are different for every database.No sampling used.

#### Exceptions <a href="#exceptions" id="exceptions"></a>

In **Redshift**, adding columns is not part of `last_modification_time`.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Schema changes

### Definition

The count of schema alterations (column additions, removals, or data-type changes) detected since the previous scan. Any schema change is treated as an anomaly.

### Source

metadata

### Computation

No sampling used. The value is calculated through the difference of two full table definitions.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Total row count

### Definition

The total number of rows in the dataset at scan time.

### Source

metadata

### Computation

Through the row count value provided by the metadata, which is calculated differently for every database.

{% hint style="info" %}
For example:

In Oracle, the total row count is calculated by doing a `count(*)`. As in Oracle, this is not available by default.

BigQuery, on the other hand, provides metadata information through the `INFORMATION_SCHEMA.TABLE_STORAGE` metadata table. Specifically, Soda uses the `total_rows` column from that table.
{% endhint %}

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Total row count change

### Definition

The difference in total row count between the current scan and the immediately preceding scan (`current_count – previous_count`).

### Source

metadata

### Computation

Soda keeps track of the previous total row count, fetches total row count again at scan time and subtracts both.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Based on data

Learn more about data-based dataset monitors.

Monitors based on data require executing **queries** against the data itself to surface usage and content recency patterns, for example:

* [**Most recent timestamp**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors/query-based/most-recent-timestamp-dataset): the latest event or ingestion time across all rows
* [**Partition row count**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors/query-based/partition-row-count): the number of records within the current partition (e.g. today’s data)

These query-based monitors give you a window into data flow and freshness, helping detect lags in ingestion pipelines or staleness in source systems.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Most recent timestamp (dataset)

### Definition

The interval between scan time and the maximum timestamp in the partition column (within the latest partition).

When a dataset has a [delay](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/late-arriving-data), the interval is measured from the end of the period being analyzed instead of the scan time. Soda also reads this monitor's history to suggest a delay.

### Source

data (time partition)

### Computation

Via `MAX(column)` for any time related column.

No sampling used.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Partition row count

### Definition

The number of rows in the most recent time partition at scan time (e.g. all rows where `partition_col = current_partition`).

### Source

data (time partition)

### Computation

Through `count(*)` for the partition.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Column monitors

Learn more about Metric Monitors that run scans at a column level.

## What is a Column Monitor?

A **column monitor** in Soda tracks a specific statistical metric for a given column over time. It helps detect unusual patterns or unexpected changes in column behavior, such as spikes in missing values or shifts in averages.

You can find column monitors by opening the **Metric Monitors** tab on any dataset and scrolling to the bottom of the page. This section lists all active column monitors in a structured, searchable view. The list can be sorted by recency or by the number of detected anomalies, allowing you to quickly focus on the most relevant issues.

Unlike dataset-level monitors, which can be applied at the data source level, column monitors are configured at the dataset level and are tailored to specific use cases. It is recommended to add column monitors only to columns where changes are likely to reflect actual data quality issues. Adding too many monitors may increase false positives and create unnecessary noise.

For column monitors to work, a **time partition column** must be defined. Soda uses this column to divide the data into time-based partitions, typically by day, and calculates the selected metrics within each partition. The column must be a timestamp and should reflect when records arrive in the database to ensure accurate and meaningful results.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FuVADUOeEo7yNWGr4JBOG%2Fimage.png?alt=media&amp;token=2c093a6b-4541-4e8b-8a06-64265021bd13" alt=""><figcaption></figcaption></figure>

For each dataset, you’ll see a scrollable list that includes:

* Result of the anomaly detection: Anomaly, Expected or Unkown (not evaluated yet)
* Column name
* Metric name (e.g. "Missing values percentage", "Average")
* Column being tracked
* Latest value
* Trend sparkline

At the bottom of the list it is possible to load more monitors. From this view, every monitor can be deleted or configured with opt-in notifications.

### Types of column monitors

<table><thead><tr><th valign="top">Data type</th><th>Metric</th><th>Description</th></tr></thead><tbody><tr><td valign="top"><strong>All data types</strong></td><td>Count</td><td>Detects anomalies in the number of non-missing (non-NULL) values in a column.</td></tr><tr><td valign="top"></td><td>Duplicate percentage</td><td>Detects anomalies in the percentage of duplicate values in a column.</td></tr><tr><td valign="top"></td><td>Maximum value</td><td>Detects anomalies in the maximum (highest) value in a column.</td></tr><tr><td valign="top"></td><td>Minimum value</td><td>Detects anomalies in the minimum (lowest) value in a column.</td></tr><tr><td valign="top"></td><td>Missing values percentage</td><td>Detects anomalies in the maximum (highest) value in a column.</td></tr><tr><td valign="top"></td><td>Unique count</td><td>Detects anomalies in the number of distinct (unique) values in a column.</td></tr><tr><td valign="top"><strong>Timestamp</strong></td><td>Most recent timestamp</td><td>Detects anomalies in the most recent (latest) timestamp value in a column.</td></tr><tr><td valign="top"><strong>Numeric</strong></td><td><br>Average</td><td>Detects anomalies in the average (mean) value of a column.</td></tr><tr><td valign="top"></td><td>Standard deviation</td><td>Detects anomalies in the standard deviation of values in a column.</td></tr><tr><td valign="top"></td><td>Sum</td><td>Detects anomalies in the total (sum) of values in a column.</td></tr><tr><td valign="top"></td><td>Variance</td><td>Detects anomalies in the variance (spread) of values in a column.</td></tr><tr><td valign="top"></td><td>Q1</td><td>Detects anomalies in the 25th percentile (first quartile) value of a column.</td></tr><tr><td valign="top"></td><td>Median</td><td>Detects anomalies in the 50th percentile (median - Q2) value of a column.</td></tr><tr><td valign="top"></td><td>Q3</td><td>Detects anomalies in the 75th percentile (third quartile) value of a column.</td></tr><tr><td valign="top"><strong>Text</strong></td><td>Average length</td><td>Detects anomalies in the average character length of text values.</td></tr><tr><td valign="top"></td><td>Maximum length</td><td>Detects anomalies in the shortest character length of text values.</td></tr><tr><td valign="top"></td><td>Minimum length</td><td>Detects anomalies in the longest character length of text values.</td></tr></tbody></table>

{% hint style="info" %}
We are working on releasing more metrics and monitors.
{% endhint %}

## Add Column Monitors

Column monitors can be added one by one or in bulk. When multiple columns are selected only metrics that are applicable to all columns will be shown.

### 1. Open the Column Monitor wizard

1. In the **Metric Monitors** dashboard, click **Add Column Monitors**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FbOgndd1ZYvTZcQ7vjqwO%2FCaptura%20de%20pantalla%202025-06-05%20152806.png?alt=media&amp;token=f7638843-13a3-47c5-8872-80f8dee6fafd" alt=""><figcaption></figcaption></figure>

### 2. Select columns

1. Search or scroll your table’s columns.
2. Check one box, or several boxes to select columns in bulk.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F8ooiWddWGPuQgqIHjoB0%2Fimage.png?alt=media&amp;token=b20dfbd3-8756-444a-8dc7-0846db8339a8" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
**Column monitors are typed**: metrics will appear as long as the necessary data type is available.

For example, if a column type is `str` (text based), it will not be possible to enable numeric metrics.
{% endhint %}

### **3. Choose metrics**

{% hint style="info" %}
For all column metrics:

* Data is not sampled.
* Missing values are ignored (except in [Missing values percentage](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/all-data-types/missing-values-percentage)).
  {% endhint %}

1. Select the metrics of interest.
2. Search or expand metrics for further configuration:
   1. **Valid Range:** define **MIN** and **MAX** values the metric can take (defaults to –∞/∞ or 0–∞ for time-based metrics).
   2. **Threshold Strategy:** choose whether to alert on the **Upper range**, the **Lower range**, or both.
   3. **Exclusion Values: s**pecify literal values or ranges to ignore when marking anomalies.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FmoDKXi0gcU3vG1o3g88r%2Fimage.png?alt=media&amp;token=bf36acda-ae37-4f36-8266-ad59cb35f836" alt=""><figcaption></figcaption></figure>

### 4. Add monitors

1. Once you’ve selected your columns and toggled the desired metrics on, click **Add Monitors**.
2. Empty monitors will be added to the list.
3. At the top of the page, you will be prompted to **run a Historical Metric Collection Scan**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F2jbdrAhgQQKkv5kP9w5Y%2Fimage.png?alt=media&amp;token=6489a58f-9bde-4d17-a4b1-21f47d3c8742" alt=""><figcaption></figcaption></figure>

{% hint style="success" %}
**Tip**: add all your column monitors first, then run the historical scan in one go. This will save time and computing costs, and ensures every monitor shares the same look-back window.
{% endhint %}

## Configure and fine-tune Column Monitors

Column Monitors can be configured when setting them up as well as while they're in production. To fine-tune the monitor to your specific needs, **go to the page for each specific metric**.

> Learn more about [How to fine-tune Metric Monitoring →](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/metric-monitor-page#how-to-fine-tune-metric-monitoring)

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# All data types

Metrics that support all data types are foundational checks that apply to any column regardless of its data type:

* [**Count** ](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/all-data-types/count)**of non-NULL values**
* [**Duplicate percentage**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/all-data-types/duplicate-percentage)
* [**Minimum**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/all-data-types/minimum-value) **and** [**maximum**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/all-data-types/maximum-value) **values**
* [**Missing values percentage**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/all-data-types/missing-values-percentage)
* [**Unique count**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/all-data-types/unique-count) **of distinct entries**

These metrics form the backbone of data completeness and consistency monitoring, ensuring **every column meets basic quality expectations**.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Count

### Definition

The total number of non-NULL values in the monitored column. Useful for identifying unexpected drops or spikes in data completeness.

### Source

data

### Computation

`COUNT(<column>)`

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Duplicate percentage

### Definition

Percentage of all duplicate non-NULL records in a column.

***

For example: `duplicate_percent(id)` on this table is 0.66 (or 66%):

| id   | name |
| ---- | ---- |
| 1    | a    |
| 1    | b    |
| 2    | c    |
| null | d    |
| null | e    |

### Source

data

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Maximum value

### Definition

The maximum value within a specific column.

### Source

data (any datatype with an order defined for max/min)

### Computation

`MAX(column)`

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Minimum value

### Definition

The minimum value within a specific column.

### Source

data (any datatype with an order defined for max/min)

### Computation

`MIN(column)`

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Missing values percentage

### Definition

Number of missing values relative to the number of rows in the partition, expressed as percentage → `(number of null or missing values in column ÷ total rows in partition) × 100`

### Source

data

### Computation

`1 - (count(column) / count(*)) × 100`

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Unique count

### Definition

The number of distinct non-NULL values in the monitored column. It highlights unexpected changes in cardinality (e.g., new user IDs, codes).

### Source

data

### Computation

`COUNT(DISTINCT <column>`

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Timestamp

Timestamp metrics highlight recency and time‐based anomalies, which is crucial for validating timeliness in event streams and incremental loads:

* [**Most recent timestamp**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/timestamp/most-recent-timestamp-column)

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Most recent timestamp (column)

### Definition

Interval between scan time and the maximum value in a date/time/time-stamp column (within the partition). Supported only on date/datetime/time columns.

### Source

data (timestamp)

### Computation

`MAX(timestamp)`

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Numeric

Numeric metrics capture central tendency and dispersion in numerical columns, such as:

* **Mean (**[**average**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/numeric/average)**)**
* [**Standard deviation**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/numeric/standard-deviation) **&** [**variance**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/numeric/variance)
* [**Sum**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/numeric/sum)
* **Percentiles (**[**Q1**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/numeric/q1)**,** [**median**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/numeric/median)**,** [**Q3**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/numeric/q3)**)**

Numeric metrics enable you to detect outliers, shifts in scale, or drifts in distribution over time.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Average

### Definition

Arithmetic mean of all non-NULL values in the column, computed per partition.

### Source

data (numeric)

### Computation

Using the built in `AVG(column_name)` function of every database.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Standard deviation

### Definition

The standard deviation of the values within a specific column.

### Source

data (numeric)

### Computation

Uses the SQL standard for all databases `STDDEV_SAMP()`, which is a sampling based method.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Sum

### Definition

Detects anomalies in the total (sum) of all non-NULL values in a given column over the latest partition. It flags unexpected increases or decreases in the aggregate amount.

### Source

data (numeric)

### Computation

`SUM(<column>)`

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Variance

### Definition

The variance of the values within a specific column.

### Source

data (numeric)

### Computation

Uses the SQL standard for all databases `VAR_SAMP()`, which is a sampling based method.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Q1

{% hint style="warning" %}
This metric is not supported in MySQL.
{% endhint %}

### Definition

Quartiles divide all non-NULL values in the column within the latest partition into four equal parts based on value: Q1 represents the 25th percentile.

### Source

data (numeric)

### Computation

* For data sources supporting exact percentiles (e.g. PostgreSQL’s `PERCENTILE_DISC(0.25)`), Soda uses that function.
* For data sources that provide approximations (such as BigQuery, SQLServer, Redshift and Trino), Soda uses those approximated values.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Median

{% hint style="warning" %}
This metric is not supported in MySQL.
{% endhint %}

### Definition

Quartiles divide all non-NULL values in the column within the latest partition into four equal parts based on value: Q2 represents the 50th percentile, that is, the median.

### Source

data (numeric)

### Computation

* For data sources supporting exact percentiles (e.g. PostgreSQL’s `PERCENTILE_DISC(0.5)`), Soda uses that function.
* For data sources that provide approximations (such as BigQuery, SQLServer, Redshift and Trino), Soda uses those approximated values.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Q3

{% hint style="warning" %}
This metric is not supported in MySQL.
{% endhint %}

### Definition

Quartiles divide all non-NULL values in the column within the latest partition into four equal parts based on value: Q3 represents the 75th percentile.

### Source

data (numeric)

### Computation

* For data sources supporting exact percentiles (e.g. PostgreSQL’s `PERCENTILE_DISC(0.75)`), Soda uses that function.
* For data sources that provide approximations (such as BigQuery, SQLServer, Redshift and Trino), Soda uses those approximated values.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Text

**Text metrics** help catch formatting issues, truncated values, or unexpectedly long/free-form entries.

With Soda, it's possible to assess the character‐length properties of string columns:

* [**Average length**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/text/average-length)
* [**Minimum length**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/text/minimum-length)
* [**Maximum length**](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/column-monitors/text/maximum-length)

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Average length

### Definition

The average (`avgLength`) of non-NULL string values in the column.

### Source

data (value length in characters)

{% hint style="info" %}
SQLServer: instead of length in characters, it uses data length (number of bytes).
{% endhint %}

### Computation

For each value, Soda encapsulates the length in the aggregation metric: `AVG(LENGTH(column))`

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Maximum length

### Definition

The maximum (`maxLength)` of non-NULL string values in the column.

### Source

data (value length in characters)

{% hint style="info" %}
SQLServer: instead of length in characters, it uses data length (number of bytes)
{% endhint %}

### Computation

For each value, Soda encapsulates the length in the aggregation metric: `MAX(LENGTH(column))`

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Minimum length

### Definition

The minimum (`minLength`) of non-NULL string values in the column.

### Source

data (value length in characters)

{% hint style="info" %}
SQLServer: instead of length in characters, it uses data length (number of bytes)
{% endhint %}

### Computation

For each value, Soda encapsulates the length in the aggregation metric: `AVG(LENGTH(column))`

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Group By monitors

**Group By monitors** enable you to track data quality metrics across specific segments of your dataset. Instead of monitoring a metric for a column as a whole, you can break it down per category (for example, per region, per school year, per status).

This functionality is especially valuable when:

* You want to detect anomalies at a more granular level, within each segment or category.
* You need visibility into how data quality differs across categories.
* You want to monitor trends and patterns that would otherwise be hidden in aggregated metrics.

{% hint style="warning" %}
**Only one Group By monitor can be configured at a time**.

Because a Group By monitor spawns multiple monitors (one per category), limiting this to a single configuration helps manage performance.
{% endhint %}

## Explore Group By monitors

When a Group By monitor is active in a dataset, results are displayed at the bottom of the **Metric Monitors tab**, on the **Column Monitors table:**

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FDMXlpbfDearS7onYOcBQ%2FGrabacindepantalla2025-09-04144353-ezgif.com-video-to-gif-converter.gif?alt=media&amp;token=790ff78d-6f94-4a4e-9358-a36d748182d7" alt=""><figcaption></figcaption></figure>

There, you will see:

#### Entry in Column Monitors table

A Group By monitor is listed like any other monitor, but its description indicates the Group By column(s) and the metric being measured (e.g. "Maximum length of `Bus_No` grouped by `Breakdown_or_Running_Late`").

{% hint style="info" %}
Group By monitors will **always be displayed at the top** of the Column Monitors table, even when no anomalous groups were detected.
{% endhint %}

From the Column Monitors table, it is possible to **turn on notifications at the column level** by clicking on the **bell icon**. Note that notifications at a category level are not available at the moment.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fox8Wv5G9TVguiaoMXPoZ%2Fimage.png?alt=media&amp;token=29e54f30-6697-42ac-91c3-ada797e15469" alt=""><figcaption></figcaption></figure>

#### Groups table

Expanding the monitor displays a groups table, which shows the results for each group or category. Each row corresponds to one category (or combination of categories if multiple columns are grouped).\
From the groups table, it is possible to **delete specific categories** by clicking on the **bin icon** on the right.

{% hint style="danger" %}
Deleting from the groups table is intended to **remove groups/categories that are no longer present in the data**.

* Deleting a category **removes the history** for that monitor.
* If the group/category is still present in the data, **the monitor will be re-created on the next scan**. **It will not be backfilled**, unless a historical metric collection scan is triggered.
  {% endhint %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FHU0xqzbNYnw5jCfppqjR%2Fimage.png?alt=media&amp;token=93bcf904-7601-43f5-bb1d-2d9a492369a3" alt=""><figcaption></figcaption></figure>

**Example:**

**Group By** `Breakdown_or_Running_Late` **+ metric** `Maximum length of Bus_No` → a row for each `Breakdown_or_Running_Late` value, with the maximum bus number length observed in that category, alongside its anomaly detection status.

## Add Group By monitors

You can add a Group By monitor from the **Metric Monitors** section of the dataset page.

1. Scroll to the **Column Monitors table** and click **Add Column Monitors**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FZK5rNaiPwrl7UXsNPlfD%2Fimage.png?alt=media&amp;token=43110082-5f29-47a6-93fd-96338f3d20d8" alt=""><figcaption></figcaption></figure>

2. In the **Add Column Monitors** panel, toggle on **Group By**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FmMUAJ0JgMo88TkdG7koA%2Fimage.png?alt=media&amp;token=4032a5fc-bf4d-4224-a6a9-7067f7904ef3" alt=""><figcaption></figcaption></figure>

3. Select one or more columns to group by.

{% hint style="warning" %}
For the time being, only columns with a maximum of **50 distinct categories** are eligible for Group By monitoring.
{% endhint %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FwrqyWG0tWuQj4tdHBCf0%2Fimage.png?alt=media&amp;token=d67d9845-56f0-472b-a52a-1238099c523d" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
Multiple columns can be selected, but note that the resulting categories are **combinatory** (e.g., `Column A × Column B`).
{% endhint %}

4. (Optional) **Exclude specific categories (segments)** that you don’t want to monitor.
5. Select one or more columns to monitor under **Column Selection**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FzOcAMh5C6eW3hfGw0twJ%2Fimage.png?alt=media&amp;token=ce88440f-f042-4342-8415-fbceec65c0b2" alt=""><figcaption></figcaption></figure>

6. Enable one (or more) metric from the right-hand list.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F0ou7hy0AseYtlXnDgVec%2Fimage.png?alt=media&amp;token=a473e9db-58e2-4250-a0d3-2f1958badaa8" alt=""><figcaption></figcaption></figure>

7. Click **Add 1 Monitor** on the top right to save.

The monitor now appears in the **Column Monitors table** and starts tracking anomalies across each category.

### Category management

* Categories can be excluded when configuring the monitor. See Step 4 on [#add-group-by-monitors](#add-group-by-monitors "mention").
* Categories can be deleted after creation from the [#groups-table](#groups-table "mention") if you decide they should no longer be monitored.

***

## Key Considerations

* **One Group By monitor at a time**\
  Only one configuration is allowed, since Group By monitors expand into many underlying monitors.
* **Multiple Group By columns**\
  More than one column can be selected, but the categories generated are combinatory.
* **Category limits**\
  Columns with more than 50 categories cannot be used for Group By monitoring.
* **Exclusions and deletions**\
  You can exclude categories at configuration time or delete them later from the [#groups-table](#groups-table "mention").
* **Notifications**\
  Notifications are configured at the **column level**, not yet at the per-category level.

***

With Group By monitors, you gain more granular visibility into your data quality, while keeping control over compute cost and category management.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Custom monitors

Learn more about how to use your own SQL queries to build custom SQL Metric Monitors.

{% hint style="warning" %}
Custom SQL Monitors are available for **Enterprise** users and can be configured via both the **Soda Cloud UI** and the **Monitoring Configuration API**.
{% endhint %}

**Custom SQL Monitors** enable you to define monitoring logic using **your own SQL queries**. This is ideal when built-in Soda metrics or anomaly checks don’t meet your needs; for example, when you must aggregate data across multiple tables, compute ratios, or detect anomalies in grouped datasets.

A **Custom SQL Monitor** runs your SQL query against your connected data source and evaluates its results on a schedule, just like any other Soda monitor.

This feature can be used to:

* Define metrics using **custom aggregations or joins**.
* **Compute grouped results** (e.g., `GROUP BY` customer, institution, or region).
* Apply **filters, CTEs, and where clauses** to narrow down data.
* Integrate results with **notification rules** to alert your team when certain conditions are met.

## How to create a custom SQL monitor

**Example scenario:** an organization needs to monitor **daily incidents** per **borough** and **reason** in their `Bus Breakdowns and Delays` dataset, and flag unusual spikes/drops via notification rules.

The goal is to know which boroughs are the ones suffering the most incidents and why that's happening.

#### Prerequisites

* **Enterprise** plan.
* A dataset connected in Soda Cloud.
* An **API token** with permission to **author** monitors (if creating [#in-the-column-monitoring-configuration-api](#in-the-column-monitoring-configuration-api "mention")).

### In the Soda Cloud UI

1. Navigate to **Datasets → Custom Monitors** at the bottom of the page. Click on **Add Column Monitor**.

   <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FYQhldxcMuMUIZYF39lj5%2Fimage.png?alt=media&amp;token=417ec643-21f7-4498-91d8-8caf3a63d5d1" alt=""><figcaption></figcaption></figure>
2. **Name** your custom monitor and **provide the custom SQL query**.\
   In this case, we are monitoring **incident count** by **borough** and **reason**.

   <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FvvlKkU7sZRJmJFHVGgd5%2Fimage.png?alt=media&amp;token=860903f0-4947-4b0e-9b8d-29c0eb9355d5" alt="" width="563"><figcaption></figcaption></figure>
3. Provide a **Result metric** and a **Valid range**, and define a **Threshold strategy**.\
   In this case, the **result metric** is `incident_count`; we want to **group by** `Boro` and `Reason`, and the **valid range** cannot be negative, so the minimum value is `0`. Both **Upper range** and **Lower range** anomaly detection are enabled to catch unusual spikes/drops per group.

   <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FXodSoKrnkslP01zSXSe0%2Fimage.png?alt=media&amp;token=101088c1-2b4b-4923-b36d-707b806ea598" alt="" width="563"><figcaption></figcaption></figure>
4. Click on **Add Monitor** on the top right.

{% hint style="warning" %}
The **result metric** column must return a number. Soda stores every measurement as a double, so a query whose result column is text, a boolean, or a timestamp, or whose value is `NaN` or infinity, is not measured: the scan logs an error naming the monitor and the value, and skips that monitor. Text is not converted, even when it holds a number such as `'12'`, so cast the result column to a numeric type in the query. `NULL` means no data for that run and is fine.
{% endhint %}

The monitor will now be visible at the bottom of the Metric Monitoring dashboard.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FGrH5WS6RMkAPw3zXUI85%2Fimage.png?alt=media&amp;token=e9b3e0f9-fafc-41ef-8c44-b0680ed722b5" alt=""><figcaption></figcaption></figure>

This monitor will:

* Run daily and compute `incident_count` for every `(Boro, Reason)` pair within the partitioned time window.
* Store grouped results so you can see which areas and causes are trending.
* Trigger notifications (based on your organization’s [**notification rule**](https://docs.soda.io/soda-documentation/fix/notifications)) when anomaly detection flags a group.

***

### In the **Column Monitoring Configuration API**

{% hint style="info" %}
**Coming soon**
{% endhint %}

***

#### Supported variables

List of all the variables currently supported using `${soda.<variable>}` syntax:

* `SCAN_TIME`: time for which the scan is running; has the same value as `PARTITION_END_TIME` (note this is different from **when** the scan is running)
* `PARTITION_COLUMN`: column used to perform time-based partitioning
* `PARTITION_START_TIME`: start time for the partition time window
* `PARTITION_END_TIME`: end time for the partition time window
* `PARTITION_INTERVAL`: duration of the partition time window
* `TABLE`: qualified name of the table being analyzed, e.g. `"my-schema"."my-table"`

***

## Valid range vs. expected range

Metric monitors in Soda use **two different types of ranges** that serve distinct purposes: **expected range** and **valid range**.

### Expected range

The **expected range** represents the values that Soda’s anomaly detection algorithm *expects* a metric to fall within, based on historical patterns.

* It is **automatically learned** by Soda’s Metric Monitoring model.
* It adapts over time as more data is observed.
* It is the range used to **detect anomalies**.
* If the measured value falls **outside the expected range**, Soda flags an anomaly.

{% hint style="success" %}
**Expected range = what usually happens.**
{% endhint %}

***

### Valid range

The **valid range** defines the **absolute bounds of sensible values** for a metric.

* It is **not learned** by the model.
* It is **manually defined**, either:
  * implicitly by Soda for most built-in monitors, or
  * **explicitly by the user for custom monitors** and certain built-in monitors (for example, `average`).

    <div data-gb-custom-block data-tag="hint" data-style="warning" class="hint hint-warning"><p>For <strong>custom monitors</strong> and <strong>context-dependent metrics</strong> (such as averages), Soda <strong>cannot infer a sensible valid range</strong>. In these cases, the valid range must be defined by the user.</p></div>
* It is used to **constrain the model’s predictions**, not to detect anomalies.

If the anomaly detection model predicts an expected range that falls outside the valid range, Soda **clips the expected range to the valid range**.

{% hint style="success" %}
**Valid range = what is possible or meaningful.**
{% endhint %}

***

#### How they work together

1. Soda’s anomaly detection model predicts an **expected range**.
2. Soda checks whether that expected range violates the **valid range**.
3. If it does, Soda **clips** the expected range so it stays within valid bounds.
4. The clipped expected range is then used to evaluate anomalies.

***

#### Example: Row count monitor

* **Valid range**: `(0, +∞)`\
  (A dataset cannot have a negative number of rows.)
* **Model prediction**: `(-2, 8)`
* **Clipped expected range**: `(0, 8)`

Without a valid range, Soda could surface predictions that do not make sense for the use case. The valid range prevents this.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Metric monitor page

Every monitor in Soda has its own dedicated detail page. This page is designed to help you explore the monitor's history, understand its behavior over time, and take action when needed. From here, you can investigate anomalies, give feedback to improve the detection algorithm, create incidents, and fine-tune the monitor's sensitivity or configuration.

The page consists of two main components:

1. **An interactive plot** that visualizes metric trends, anomalies, and historical context
2. **A results table** that lists all metric values and events visible in the plot

## **The Interactive Monitor Plot**

The interactive plot gives you a time-based view of how the monitor metric has evolved. It combines metric values, expected behavior, and any detected anomalies in a single visual.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FbEQu1QSvzdvPxrwTghTe%2Fimage.png?alt=media&amp;token=09f0d74c-b79f-4016-b4d7-64d073d6e690" alt=""><figcaption></figcaption></figure>

#### **Navigating the Plot**

* **Select a time window** using the range slider below the plot to zoom in or out on a specific period

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FXDYpxgTY3afUuBDWakSJ%2Ftime_window.gif?alt=media&amp;token=2ccd6eb4-d50e-467e-9a2d-d22e9314f43e" alt=""><figcaption></figcaption></figure>

* **Click and drag** to zoom into a custom time range

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FnlSKZaid0yCPgtVuwRPR%2Ftime_window_select.gif?alt=media&amp;token=87deac07-847f-455c-bd55-827d0a3a6e25" alt=""><figcaption></figcaption></figure>

* **Hover over data points** to view detailed information for each result

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F9ilNRkgcKMg21Jf2wYWA%2Fplot_hover.gif?alt=media&amp;token=a42c6e9a-1ef9-4f08-bee8-1daae1f18049" alt=""><figcaption></figcaption></figure>

#### **What the plot shows**

* **Expected range:** the shaded area that represents the predicted normal behavior, as defined by the anomaly detection model
* **Measurement**: the actual metric value for each scan
* **Anomaly:** points marked when the metric falls outside the expected range and is flagged by the algorithm
* **Missing:** scans where no metric could be collected, typically due to unavailable data or delayed scans
* **Feedback:** shows if the user provided feedback on a result (e.g. confirmed or dismissed an anomaly)
* **Configuration change:** visual markers indicating when the monitor’s configuration was updated

When a dataset has a [delay](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/late-arriving-data), the plot for a partition-based monitor stops at the last period Soda analyzed, not at today. Periods whose data is not due yet are left out instead of being drawn as missing.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FswQAJsB58brYc7eg7sPx%2Fimage.png?alt=media&amp;token=6fc0ef43-8a66-454d-84ea-a5004d75b087" alt=""><figcaption></figcaption></figure>

#### **Key events in the plot**

* **Monitor created:** marks the date the monitor is created
* **Initial configuration:** shows the starting settings used when the monitor was first enabled
* **Configuration updated:** marks changes to thresholds, exclusions, or sensitivity applied over time

## **Using the Results Table**

Below the plot, the table lists all historical scan results, including metric values, anomaly status, and any user actions (like feedback or incidents). The plot is aligned with the table, so each data point in the plot directly corresponds to a result in the table

This makes it easy to correlate visual trends with specific events, compare changes, and drill into the context of any anomaly or data quality issue.

With the three dots it is possible to give feedback to the anomaly detection algorithm, in bulk for multiple results at once or create and link the results to an incident

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FllGFTKs9Wf9beWF7cU4C%2FScreenshot%202025-06-06%20at%2019.30.50.png?alt=media&amp;token=28b51123-2e38-4395-9e5d-4e022c7dc0b2" alt=""><figcaption></figcaption></figure>

## How to fine-tune Metric Monitoring

Soda's Observability tools work out of the box with predefined baselines, but you can fine-tune them to your specific needs. Do this from the page for each specific metric.

### Set Threshold Strategy

By default Soda uses an adaptive statistical method, but you can control which sides of the expected range should trigger an anomaly alert:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F1KQeSAuhd77Ayl65PmoB%2Fthreshold_strategy.gif?alt=media&amp;token=499a0db4-c5f0-47d5-9150-fef4944739ed" alt=""><figcaption></figcaption></figure>

1. **Open the panel**

Click on "Set Threshold Strategy" button on the metric of your choice

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FtLMxDRHZk0rhFISem3co%2Fimage.png?alt=media&amp;token=8254d2bf-1cdf-42c5-9e65-db0b7623bdc8" alt=""><figcaption></figcaption></figure>

2. **Choose your alert ranges**

* **Upper range**: when checked, Soda will flag any metric value that exceeds the upper bound of its statistical baseline.
* **Lower range**: when checked, Soda will flag any metric value that falls below the lower bound.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FmefveEumu99j7z96AKG7%2Fimage.png?alt=media&amp;token=0e25e1a6-70af-4c98-8dc8-08a0660d6dc0" alt=""><figcaption></figcaption></figure>

3. **Apply your settings**

Click **Set Threshold Strategy** to save.

With this simple toggle interface you can, for example, watch only for unexpectedly high values, only for drops below your baseline, or both.

### Set Exclusion Values

Exclude values from monitoring to tell Soda which specific values or ranges should be ignored when evaluating anomalies (e.g. test rows, manual overrides). The available inputs depend on the metric type.

1. **Open the panel**\
   Click on **Set Exclusion Values** button on the metric of your choice

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fe70IzhMl75yvSPlolscb%2Fimage.png?alt=media&amp;token=3b641cdc-e898-473d-a241-5f00cac52627" alt=""><figcaption></figcaption></figure>

2. **Define your exclusions**\
   Click on **+ Add exclusion**

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FOntcRHL1DoZDrla7qzJG%2Fimage.png?alt=media&amp;token=9d51cf85-86d3-4d80-a7bb-4d6ec859c7c2" alt=""><figcaption></figcaption></figure>

* **Numeric metrics** (Total row count, Total row count change, Partition row count):
  * **Type**: **Value** or **Value range**
  * **Value**: enter the exact metric value (e.g. `205`) or, for a range, specify both lower and upper bounds.
* **Time-based metrics** (Last modification time, Most recent timestamp):
  * **Type**: **Value** or **Value range**
  * **Value**: enter the cutoff you want to ignore (e.g. `0 days, 10 hours, 49 minutes`) or, for a range, specify both lower and upper bounds.
* **Schema changes**: exclusions are not supported for schema-drift monitors.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fujgrs2SAtgsKnDdAOFnZ%2Fimage.png?alt=media&amp;token=d86fe7ea-4c87-40ea-8d56-a46a84d0c8c4" alt=""><figcaption></figcaption></figure>

You can stack multiple rules by clicking **+ Add exclusion**.

3. **Apply**\
   Click **Set Exclusion Values** to save your rules.

{% hint style="warning" %}
This will not retroactively change past results. It only affects future anomaly evaluations.
{% endhint %}

### Set Sensitivity

Soda uses a statistical baseline to define an “expected range” for anomaly detection. You can adapt how tight or loose that range is.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F1ta9TgcrnY03px9Wvmtp%2Fsensitivity%20(2).gif?alt=media&amp;token=dab2af70-d565-402e-b1d5-a7dc56d29ab6" alt=""><figcaption></figcaption></figure>

1. **Open the panel**\
   Click on **Set Sensitivity** button on the metric of your choice

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FynOP9wr7y1jEwvtAVMke%2Fimage.png?alt=media&amp;token=37650bc7-81ef-4647-8c3c-e795fb235172" alt=""><figcaption></figcaption></figure>

2. **Adjust the sensitivity**

* **Provide a z-score**: enter a value between **0.3** and **6** to control the exact width of the expected range OR use the slider to drag between **Narrow** (lower z-score) and **Wide** (higher z-score).
* **Default**: `z = 3`

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FPgTtkfPYt4xyC8K5wuxU%2Fimage.png?alt=media&amp;token=8d4c0ec7-3796-45d3-9998-435f422bceea" alt=""><figcaption></figcaption></figure>

Preview how changing sensitivity widens or narrows the gray “expected” band in the plot

3. **Apply**

Click **Apply** sensitivity to save.

{% hint style="warning" %}
This will not retroactively change past results. It only affects future anomaly evaluations.
{% endhint %}

### Give feedback to improve detection

Our home-brewed anomaly detection algorithm draws trends from historical data, but it can also learn from your input as you give it feedback.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FujGXlrRyaxl7RbAYRPFy%2Fimage.png?alt=media&amp;token=a571680b-5371-4b69-859b-9f046fe913f9" alt=""><figcaption></figcaption></figure>

When a monitor flags an anomaly you can:

1. **Mark as expected**\
   Teach Soda that this deviation is acceptable: future similar variations will no longer trigger alerts.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FTx6Mh5gsfoVAFaQcYBQx%2Fmark_as_expected.gif?alt=media&amp;token=801f2bfe-0a65-46d9-96c5-9efb4981bc30" alt=""><figcaption></figcaption></figure>

2. **Mark as anomaly**\
   Explicitly flag a point as an anomaly, even if it fell inside the baseline. This helps refine your alert definitions.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FUH91Ksi9O2krWKHXHhb5%2Fmark_as_anomaly.gif?alt=media&amp;token=dd1c61e2-2132-4a5f-a80d-2f0c7a11daaf" alt=""><figcaption></figcaption></figure>

### Create incidents

* **Create new incident**\
  Create a ticket in your incident management tool directly from the panel.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FKiHQtAK373E9Iljs84QY%2Fcreate_incident.gif?alt=media&amp;token=f1e73f5a-4ffa-46c6-8fb6-c9a22f918151" alt=""><figcaption></figcaption></figure>

* **Link to existing incident**\
  Add this scan to an unresolved incident. The incident can be on this monitor's dataset or, with **Search incidents across all datasets**, on any other dataset, so one incident can track an issue across several monitors and checks.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FKjwqMHEMYNXuJHF1yfOQ%2Flink_to_incident.gif?alt=media&amp;token=01f884ad-c385-4d31-8f75-1120d29b3c76" alt=""><figcaption></figcaption></figure>

* **Unlink from incident**\
  On a scan that is linked to an incident, remove the monitor from that incident. This removes all of the monitor's scans from the incident. An incident always keeps at least one check or monitor, so to replace the only one, link the new one first.
* **Bulk feedback**\
  More than one scan can be added to an incident or feedback. Simply check the boxes of the scans you want to add.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FerjFRXEsiegX8mQiZlPJ%2Fbulk_feedback.gif?alt=media&amp;token=22bec08a-7f16-4425-8184-8a39a3878726" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
All feedback and incident links become part of the scan history, providing an auditable trail for both data engineers and business stakeholders.
{% endhint %}

> Learn more about [Incidents](https://docs.soda.io/soda-documentation/fix/incidents).

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Scan time strategy

Selecting the right scan time is essential for accurate data monitoring and reliable metric collection. Scans that occur too early may run before the data has been fully loaded into the database, leading to false positives or misleading results. This guide outlines how to determine the best scan time based on your data load patterns and operational needs.

## Scan frequency

Scans can be scheduled to occur from **hourly** to **weekly**. The time jumps are meant to fit into a 24-hour cycle that matches **hourly/daily seasonalities** related to how humans organize their day. Metric Monitoring can happen every:

|                       |                          |
| --------------------- | ------------------------ |
| <ul><li>1 h</li></ul> | <ul><li>8 h</li></ul>    |
| <ul><li>2 h</li></ul> | <ul><li>12 h</li></ul>   |
| <ul><li>3 h</li></ul> | <ul><li>1 day</li></ul>  |
| <ul><li>4 h</li></ul> | <ul><li>1 week</li></ul> |
| <ul><li>6 h</li></ul> |                          |

## Key considerations

### Data load completion time

When is the database load expected to be complete?

* Determine when the relevant tables or datasets are expected to be fully loaded.
* Factor in common variances: if a load is expected to complete by `00:00 UTC` but occasionally finishes at `00:10 UTC`, account for the expected, albeit sporadic, delay.

Knowing this helps avoid scanning too early and capturing incomplete data.

### Acceptable load delay tolerance

When is a delayed load considered late or "problematic"?

* If data arriving by `02:30 UTC` is still valid for monitoring purposes, it may be better to delay the scan to reduce false alerts.
* Scanning immediately after the earliest expected load time is not always necessary.

Understanding what qualifies as "late data" helps define the tolerance window for scan timing.

Moving the scan does not help when your data itself arrives late. The period a scan analyzes ends at the scan, so a later scan time moves that period along with it, and the same stretch of data is still missing. Set a delay instead: it moves the period without moving the scan. See [Late-arriving data](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/late-arriving-data).

### Response window & team availability

How fast after the load can someone respond to issues flagged by monitors?

* If nobody can take action until `09:00 UTC`, scanning earlier may not be useful unless scans feed downstream processes or dashboards.

Choose a scan time that aligns with both data readiness and team readiness.

{% hint style="success" %}
**Consistency is key**

Running scans at the **same time every day** allows to build up a reliable baseline of expected behavior. This helps surface anomalies clearly when something deviates from the norm.
{% endhint %}

***

## Example scenario

* **Scan frequency**: daily
* **Expected load completion:** `00:00 UTC`
* **Occasional load delay:** up to `00:10 UTC`
* **Team available from:** `08:00 UTC`

#### Scan options

<table><thead><tr><th width="183.81668090820312">Strategy</th><th width="126.4000244140625">Scan time</th><th>Rationale</th></tr></thead><tbody><tr><td>Minimal buffer</td><td><code>00:15 UTC</code></td><td>Captures data soon after load with minor delay tolerance.</td></tr><tr><td>Conservative buffer</td><td><code>01:30 UTC</code></td><td>Allows extra time for delayed loads, reduces risk of false positives.</td></tr><tr><td>Operationally aligned</td><td><code>07:30 UTC</code></td><td>Ensures scan results are fresh and complete when the team starts reviewing.</td></tr></tbody></table>

## Scan scheduling at scale

When scanning large volumes of tables:

* It is acceptable to configure scans for the **same scheduled time** (e.g. `00:00 UTC`).
* Scans that are scheduled in large volumes (thousands of tables) may be configured to run at the same logical time, but **the system naturally distributes execution** based on queuing and available resources, so the actual execution will be **staggered**.

***

## Historical scans

* **Historical metric collection scans** (for metric baseline backfilling) run only **once** at configuration time.
* These scans are **not governed** by the scan schedule. They occur once and they are typically the most resource-intensive.

***

## Best practices

* **Consistency is key:** Using the same scan daily establishes a stable baseline for anomaly detection.
* **Early scans should be avoided:** Scheduling scans before the last acceptable load time is not recommended unless business needs require it.
* **Time zones should be centralized:** Aligning scan time with the database time zone is ideal, especially when your time partitioning column is based on the insert/load time in that time zone.
* **Monitoring and adjusting:** If load patterns or SLAs change, scan times should be revisited and adjusted accordingly.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Late-arriving data

Make Soda wait for late data before it analyzes a period, so your monitors see complete data.

Each scan analyzes the period that just ended. That works when the data for that period has already landed. Often it has not. Rows tend to land a while after the moment they describe, so the scan reads a period that is still filling up.

The monitors still report a value. But that value describes an incomplete period. Partition row count comes out low. Most recent timestamp comes out stale. The anomaly detection algorithm learns from those values, so it learns the wrong normal.

A **delay** fixes this. Each scan then analyzes an older period instead of the one that just ended. Set a delay of two days, and Monday's data is analyzed on Wednesday. By then it has arrived.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-96eb35868775129494f70ffb744db8c543035031%2Fdata-delay-timeline.svg?alt=media" alt=""><figcaption><p>A dataset with daily partitions, whose rows land up to six hours after the moment they describe. A later scan time moves the analyzed period along with it, so the same six hours stay missing. A delay moves the period back instead.</p></figcaption></figure>

## Delay length

A delay should be as long as your data's lag, and no longer. A longer delay still catches all your data, but you wait longer for the results of each period.

To measure the lag, take one row. Note the time its time partition column says it belongs to. Then note when that row actually landed in your dataset. The gap between the two is your delay.

Two examples:

* Rows stream in about two hours behind their own timestamp. Use a two-hour delay.
* Your supplier sends each day's file three days later. Use a three-day delay.

What counts is the size of the gap, not how it compares to your partition. Two hours of lag on daily partitions still needs a two-hour delay. Without it, every scan misses the last two hours of the day it analyzes.

## Scan time versus delay

Moving the scan later is a tempting alternative, but it does not work. The period a scan analyzes ends at the scan itself. Move the scan two hours later, and the period moves two hours later too. The same two hours are still missing, just later in the day.

A delay moves the period without moving the scan. That is what closes the gap. Your scans keep running on the same schedule, at the same frequency. Only the period each one looks at moves back.

Use the [scan time](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/scan-time-and-scan-time-strategy) to decide when your scans run. It sets which compute window they fall in, and whether your team is around to act on the results. Use a delay for data that arrives late.

## Affected monitors

A delay applies to the monitors that read your data through the **time partition column**:

* Partition row count
* Most recent timestamp
* All column monitors

Metadata monitors are not affected: total row count, total row count change, last modification time, and schema changes. They read the data source directly at scan time, so there is no period to move.

{% hint style="info" %}
A delay needs a [time partition column](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard/dataset-monitors#time-partition-column). Without one there are no partitions to move. Soda saves the delay, but it does nothing until you configure a time partition column.
{% endhint %}

## Configure a delay

You can set a delay in three places:

* **During onboarding**, for all your datasets at once. See [onboard datasets](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud).
* **In bulk**, from the Datasets list. Select your datasets, then click **Configure Metric Monitoring**.
* **For a single dataset**, on its **Metric Monitors** tab. Click **Configure Metric Monitoring**, then open **Late-arriving data**.

Then pick one of two options:

* **No delay**: each scan analyzes the period that just ended. This is the default.
* **Delayed analysis**: each scan analyzes an older period. Type how much older in the **Delay** field.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-8dbd14c37bac8143550fe7cfca617d5235fc5a5c%2Fdata-delay-configuration.png?alt=media" alt=""><figcaption><p>Delayed analysis with a two-day delay. The preview underneath shows the next scan, and the exact period that scan will analyze.</p></figcaption></figure>

Write a delay as a number and a unit. Use `w` for weeks, `d` for days, `h` for hours, and `m` for minutes. Combine them in any order, separated by spaces: `2d`, `36h`, `1w 2d 3h`. The longest delay you can set is 365 days.

Soda previews the result under the field. It shows the delay, the time of the next scan, and the exact period that scan will analyze. Compare that period with the time your data usually lands, then save.

{% hint style="warning" %}
**Changing the delay clears the monitoring history**

Old measurements describe a different period than new ones, so Soda cannot compare them. When you change the delay, Soda clears this dataset's monitoring history, and the anomaly detection algorithm starts learning again. Changing the time partition column does the same.

Anomaly detection stays quiet until the new baseline builds up.
{% endhint %}

## Let Soda recommend a delay

Soda can suggest a delay for you. It watches the **Most recent timestamp** monitor, which tracks how far behind your newest row is at each scan. When that lag is stable and bigger than half a partition, Soda shows a suggestion on the **Metric Monitors** tab.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-b55ae5ad8acd9dd71dc0161186c95f1def335e15%2Fdata-delay-recommendation.png?alt=media" alt=""><figcaption><p>The suggestion appears on the Metric Monitors tab, above the monitors it is about.</p></figcaption></figure>

**Delay analysis by ...** opens the panel with the delay already filled in. Nothing is saved yet. Review the period it will analyze, change the value if you want, or close the panel to leave the dataset as it was.

**Dismiss** hides the banner. It only comes back if Soda arrives at a clearly different delay, so a dismissed 2-day suggestion will not reappear tomorrow.

You may see no suggestion at all. The common reasons:

* **Too little history.** Soda needs at least a week of measurements, so a newly onboarded dataset has nothing to go on yet.
* **The lag jumps around.** If data is one day late one time and four days late the next, no single delay fixes it. Soda does not guess.
* **The lag is small.** Soda only suggests a delay when the lag is more than half a period, so it does not nudge you over a few minutes. A shorter delay is still yours to set by hand.
* **The data is very old.** If the newest row is over a year old, the dataset has probably stopped loading. That is a different problem.

## Set a delay through the API

The dataset update endpoint accepts `dataDelaySeconds`, the delay in seconds. Send `null` to go back to analyzing the period that just ended. See the [REST API](https://docs.soda.io/apis/rest-api).

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Metadata data sources

## Supported data sources

Below, find a snapshot of all the capabilities supported for each data source, including **metadata** and **metadata history**:

<table><thead><tr><th width="185.79998779296875">Data source</th><th width="171.20001220703125">Onboarding</th><th>Metadata</th><th>Metadata history</th><th>Querying data</th></tr></thead><tbody><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/amazon-athena">Athena</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/bigquery">BigQuery</a></td><td>✅</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/databricks">Databricks SQL</a></td><td>✅</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/dremio">Dremio</a></td><td>✅</td><td>Upcoming</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/duckdb">DuckDB</a> (<a href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/pandas">Pandas</a>, <a href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/polars">Polars</a>, etc.)</td><td>Partial/Python only</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/microsoft-fabric">Fabric</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td>MariaDB</td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr><tr><td>MySQL</td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/oracle">Oracle</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/postgresql">PostgreSQL</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/amazon-redshift">Redshift</a></td><td>✅</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td>SAP HANA</td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/spark-dataframe">Spark Dataframe</a></td><td>Partial/Python only</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/snowflake">Snowflake</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/sql-server">SQL Server</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/supabase">Supabase</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/synapse">Synapse</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/trino">Trino</a></td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr></tbody></table>

## Limitations & edge cases

Some data sources present **limitations** when it comes to **metadata** collection or retrieval. Keep in mind these caveats when choosing which data source(s) to connect to Soda.

### Oracle

* **Historical backfilling:** not possible.
* **Row count:** metadata row counts are calculated via `count(*)`. Soda does not use metadata for this metric in Oracle. It requires an additional package and/or is unreliable based on the schedule of that package.
* **Last modification time:** Soda uses metadata

Note that past data is only available for a limited amount of time, which varies depending on the system. The minimum goes back 120 h.

{% hint style="warning" %}
**Non-UTC timestamps are not recommended** when connecting Soda to Oracle data sources. Soda uses timezone data when available, but assumes UTC when the timezone is not provided by the data source.

Some databases convert timestamps to UTC, but **Oracle does not do any implicit conversions** and stores timestamps and timezone information as the user inputs them. Because of Oracle Python client limitations, all timezone information is stripped when Soda retrieves it, which means that Soda will read all timestamps as if they were UTC regardless of the original input.
{% endhint %}

***

### Postgres

Metadata is supported, but it requires some additional setup on Postgres's side.

* **Historical backfilling:** not possible.
* **Row count:** enabled out-of-the-box.
* **Last modification time:** `track_commit_timestamp` must be enabled: <https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-TRACK-COMMIT-TIMESTAMP>
  * If `track_commit_timestamp` is not enabled, Soda will return a warning.

***

### BigQuery

Metadata metrics are available and supported in BigQuery.

* **Historical backfilling:** possible.
* **Partition column:** can be suggested based on metadata available in BigQuery.
  * Soda will prioritize user-suggested columns.
  * If there are no user-suggested columns, Soda will try a metadata approach to find the partition column automatically.
  * If there are no columns found in the metadata of BigQuery, Soda will fall back on its own heuristic.

{% hint style="info" %}
**Partition column availability in BigQuery:**

If the user has configured a partitioning column on BigQuery's side, Soda will use it (given that it is a date/timestamp column).

Otherwise, Soda will fall back on a standard sampling method to detect the partition column.
{% endhint %}

***

### Redshift

* **Historical backfilling** is supported on Redshift and it is limited to 7 days for the metadata.
* **Modification time** does not include schema changes. Only:
  * inserts
  * updates
  * deletes

***

### Synapse

Synapse does not provide metadata history tables.

* **Historical backfilling:** not possible.
* **Last modification time:** not possible.
* **Row count:** current row counts are calculated via `count(*)`.\
  Soda does not use metadata for this metric in Synapse.
* **Quartile metrics** (Q1, median, Q3): not possible.\
  Synapse does not support quartile metrics.

***

### MariaDB

MariaDB does not expose usable warehouse metadata. `information_schema.TABLES.TABLE_ROWS` is an estimate rather than a count, so Soda does not read it.

* **Historical backfilling:** not possible for the metadata.\
  MariaDB exposes no change log for Soda to reconstruct metadata history from. A backfill still runs, on data-based metrics only.
* **Last modification time:** not possible.
* **Row count:** current row counts are calculated via `count(*)`.\
  Soda does not use metadata for this metric in MariaDB.
* **Quartile metrics** (Q1, median, Q3): not possible.\
  MariaDB has neither a percentile function nor `WITHIN GROUP`.

### MySQL

MySQL does not expose usable warehouse metadata. `information_schema.TABLES.TABLE_ROWS` is an estimate for InnoDB tables rather than a count, so Soda does not read it.

* **Historical backfilling:** not possible for the metadata.\
  MySQL exposes no change log for Soda to reconstruct metadata history from. A backfill still runs, on data-based metrics only.
* **Last modification time:** not possible.
* **Row count:** current row counts are calculated via `count(*)`.\
  Soda does not use metadata for this metric in MySQL.
* **Quartile metrics** (Q1, median, Q3): not possible.\
  MySQL has neither a percentile function nor `WITHIN GROUP`.

***

### SAP HANA

SAP HANA exposes no metadata provider to Soda, so Metric Monitoring runs on data-based metrics only. Monitors that read the warehouse's own catalog are unavailable.

* **Historical backfilling:** not possible for the metadata.\
  There is no change log for Soda to reconstruct metadata history from. A backfill still runs, on data-based metrics only.
* **Last modification time:** not possible.
* **Row count:** current row counts are calculated via `count(*)`.\
  Soda does not use metadata for this metric in SAP HANA.
* **Quartile metrics** (Q1, median, Q3): supported.\
  Soda splits the query when a percentile and a `DISTINCT` aggregate over the same column would otherwise land in one `SELECT`, which SAP HANA rejects. The same approach is used for Redshift.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Profiling

Profiling provides a quick and comprehensive overview of a dataset’s structure and key statistics.

Profiling helps you understand the shape, quality, and uniqueness of your data before creating checks or metric monitors.

With profiling, you can explore metadata about your dataset, such as **column names, data types, distinct counts, null counts, and summary statistics**. You can also quickly search for specific columns to focus on the attributes that matter most to your analysis.

Profiling is useful for:

* **Business teams**: Gain a fast understanding of what’s inside a dataset, its completeness, and potential anomalies.
* **Data teams**: Validate schema, data types, and distributions before writing quality tests or transformations.
* **Data owners**: Quickly identify unexpected values, nulls, or structural changes in a dataset.

### Key features

* **Dataset overview**: Displays a structured view of all columns, their types, and counts.
* **Interactive navigation**: Scroll through the dataset structure or jump directly to a column of interest.
* **Search and filter**: Quickly locate a column by name to review its profiling details.
* **Column-level insights**:
  * **Statistics**
    * Column name
    * Column data type
    * Number of distinct values
    * Number of missing (null) values
    * Minimum, maximum, mean (for numeric columns)
    * Length, patterns, or categories (for text columns)
  * **Histogram** for numeric columns
  * **Frequent values**
  * **Extreme values**, for numeric columns
  * **Data checks** that exist for this column

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FKDHgXMsr46VeCmZmIftu%2Fhttps___files.gitbook.com_v0_b_gitbook-x-prod.appspot.com_o_spaces_2FA2PmHkO5cBgeRPdiPPOG_2Fuploads_2FLTpyRbrafUcyJ5jtphv5_2FScreenshot_202025-05-29_20at_205.09.23_20PM.avif?alt=media&amp;token=9ea82194-9415-4e07-9204-002796616cbf" alt=""><figcaption></figcaption></figure>

***

## Enable & configure Profiling

### 1. Enable Profiling

You can enable Profiling during [dataset onboarding](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud).

If you want to enable Profiling on an existing dataset, follow the next steps:

1. Click on **Datasets** > The dataset of your choosing
2. Navigate to the **Columns** tab in the dataset view
3. Click on **Update Profiling Configuration**

   <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FvlTEM40UjGAIOsdZjnQ9%2Fimage.png?alt=media&amp;token=33231714-98f2-41dd-96cb-7b8bf81d16a2" alt=""><figcaption></figcaption></figure>
4. Toggle on **Enable Profiling**

   <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FvG4oS3THsseyyKU7VLjL%2Fimage.png?alt=media&amp;token=dd546172-2d44-4a8c-96d4-8988a3b1eafd" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
When Profiling is enabled **during** [**onboarding**](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud), an automatic scan for Profiling will be executed **regardless of manual or scheduled execution**.
{% endhint %}

### 2. Configure Profiling

Once Profiling has been enabled, you can configure it to adapt to your organization's needs.

1\. Choose a **Profiling schedule**

Profiling happens every 24 hours. **Choose a UTC time** from the dropdown menu to pick a specific hour when the scan will be scheduled.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FVkfdKvy0i20lKVrCpssM%2Fimage.png?alt=media&amp;token=e455dbb3-4e04-4860-b560-5dba0459aec6" alt="" width="563"><figcaption></figcaption></figure>

2. Choose a **Profiling strategy**
   * **Use sampling:** To perform Profiling, Soda will use a **sample of up to 1 million rows** from the dataset.
   * **Use a time window:** To perform Profiling, Soda will use data present in a **30-day time window**, based on the dataset time-partition column.

{% hint style="info" %}
The **time-partition column** is specified **above the columns table**, on the **Columns** tab of any given dataset.
{% endhint %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F7wBUNb3lGMLKIKSwu60H%2Fimage.png?alt=media&amp;token=c91bebb2-c095-4d6b-80ac-b5342376a3dd" alt=""><figcaption><p>In the Bus Breakdown and Delays dataset, the time-partition column is Last_Updated_On</p></figcaption></figure>

3. Click on **Finish**

Now, Profiling will be scheduled.

### Disable Profiling

#### Disable column profiling at the organization level <a href="#disable-column-profiling-at-the-organization-level" id="disable-column-profiling-at-the-organization-level"></a>

If you wish to disable column profiling at the organization level, you must possess **Admin privileges** in your Soda Cloud account. Once confirmed, follow these steps:

1. Navigate to your avatar.
2. Click on **Organization settings**.
3. Uncheck the box labeled **Allow Soda to collect column profile information**.

***

### How it works

When you open Profiling for a dataset:

1. Soda runs a lightweight scan of the dataset’s metadata and a sample of the data (depending on configuration).
2. It calculates summary statistics for each column.
3. Results are displayed in the Profiling view for exploration.

#### Key considerations

* Soda can only profile columns that contain `NUMBERS` or `TEXT` type data; it cannot profile columns that contain `TIMESTAMP` data except to create a freshness check for the anomaly dashboard.
* Soda performs the Discover datasets and Profile datasets actions independently, relative to each other. If you define `exclude` or `include` rules in the Discover tab, the Profile configuration does not inherit the Discover rules. For example, if, for Discover, you exclude all datasets that begin with `staging_`, then configure Profile to include all datasets, Soda discovers and profiles all datasets.

***

## Next Steps

After reviewing profiling results, you can:

* Create **tests** based on profiling insights (e.g., "column should not have nulls").
* Set up **monitors** to track data quality over time.
* Export profiling information to support documentation and governance processes.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Manage data quality issues

Browse your datasets and checks in Soda Cloud to see the state of your data quality and find the issues that matter to your team.

**Detecting a data quality issue is only the start**. The harder part is the work that follows: knowing **which** issues matter, **who** should look at them, **what** the business impact is, and **how** to drive them to resolution before downstream consumers are affected.

**Soda Cloud** gives data and platform teams **a single place** to make that work manageable **at scale**, turning thousands of individual check results into a coherent operational picture of data health across your organization.

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Browse datasets</strong></td><td>See every dataset onboarded into Soda Cloud, with failing checks, detected anomalies, and the time of the last scan.</td><td><a href="https://docs.soda.io/soda-documentation/understand/manage-issues/browse-datasets">Browse datasets</a></td></tr><tr><td><strong>Dataset page</strong></td><td>Open a dataset to see its checks, metric monitors, profiling, scans, incidents, and requests.</td><td><a href="https://docs.soda.io/soda-documentation/understand/manage-issues/browse-datasets/dataset-page">Dataset page</a></td></tr><tr><td><strong>Browse checks</strong></td><td>See the health of every check across datasets, and group them with filters such as data source, dataset, owners, or status.</td><td><a href="https://docs.soda.io/soda-documentation/understand/manage-issues/browse-checks">Browse checks</a></td></tr></tbody></table>

### Narrow down to what matters

Filter datasets and checks by data source, owners, attributes, or status, such as failing checks or detected anomalies. Save a filtered view as a **collection** and share it with your organization, so each team starts from its own slice of the same data.

> Once you've found a failing check or monitor, see how its results evolve over time in [Analyze monitor and check results](https://docs.soda.io/soda-documentation/understand/analyze-monitor-and-check-results).

***

## Soda for issue management

Most data quality tooling stops at "the check failed." That leaves the rest of the workflow—triage, ownership, communication, and resolution—scattered across Slack threads, tickets, and tribal knowledge. **Soda closes that gap**.

{% columns %}
{% column %}

#### See the whole picture at a glance

Track the health of every onboarded dataset and every active check from one view. Spot trends, identify the datasets most at risk, and measure how data quality is evolving over time across teams, domains, and data sources.
{% endcolumn %}

{% column %}

#### Focus on what matters to your team

Slice the picture by domain, owner, criticality, environment, or any custom dimension that reflects how your organization actually works. Save the views your team relies on and share them with stakeholders.
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

#### Investigate without leaving the platform

When something fails, drill from a high-level health score down to the dataset, the column, the check, and the historical trend behind the failure, with the diagnostic context needed to understand scope and impact.
{% endcolumn %}

{% column %}

#### Get the right people involved, fast

Route alerts to the teams and channels that own the affected data, open incidents that track resolution to closure, and keep a record of what was broken, what was done, and why.
{% endcolumn %}
{% endcolumns %}

## Soda Cloud capabilities

Soda Cloud is the layer that sits on top of your data quality checks, whether they're authored as data contracts, monitored as metrics, or run from [Soda Core](https://github.com/sodadata/soda-core), and turns them into an **operational workflow**.

At a high level, teams use it to:

* **Monitor data health across the organization** with executive-level dashboards covering scan activity, failing checks, and overall health scores, filterable by the dimensions that matter to your business.
* [**Browse datasets and checks**](https://docs.soda.io/soda-documentation/understand/manage-issues/browse-datasets) with rich filtering, search, and saveable views, so each team can carve out the slice of the estate they're responsible for.
* **Tag datasets and checks with custom metadata** (domain, criticality, data layer, owning team, sensitivity) and use that metadata to drive filtering, reporting, and alert routing consistently across the platform.
* [**Analyze check and monitor results**](https://docs.soda.io/soda-documentation/understand/analyze-monitor-and-check-results) **over time** with historical trend views and per-check-type diagnostics that surface the *why* behind a failure, not just the *what*.
* **Send the right alert to the right place** through a [notification system](https://docs.soda.io/soda-documentation/fix/notifications) that targets recipients based on the properties of the affected data, rather than blasting every failure to a single channel.
* **Run investigations as** [**incidents**](https://docs.soda.io/soda-documentation/fix/incidents) with assigned ownership, status tracking, resolution notes, and integrations into the tools your on-call and operations teams already use.

## Collaboration across roles

Issue management in Soda is designed around the reality that **data quality is a team sport**. Data engineers, platform owners, analysts, and business stakeholders all need different views of the same underlying signal.

{% hint style="info" %}
Saved views, custom attributes, and scoped notification rules let each persona work from a **tailored slice of the same source of truth**, without forking dashboards or duplicating logic.
{% endhint %}

### Integrate Soda with your existing tools

The built-in dashboards cover the most common needs, but every organization eventually wants to roll data quality metrics into its own reporting layer. **Soda exposes its data through a public** [**REST API**](https://docs.soda.io/apis/rest-api), so you can pull health scores, check results, and incident data into **Power BI**, **Tableau**, **Looker**, or any internal tool your business already trusts.

## Soda Core vs Soda Cloud

**Soda Core** gives you a powerful, scriptable way to run checks against your data and get pass/fail output wherever you invoke it: perfect for **CI pipelines**, **orchestrators**, and **local validation**.

**Soda Cloud provides the layer above in a user-friendly UI**: cross-team dashboards, historical trends, routing of alerts to the right recipients, incident workflow, and shared metadata model. Those are the problems Soda Cloud solves once a single team's checks become an organization's data quality program.

If you've outgrown the "check ran, here's the result" loop and need to manage data quality as an ongoing operational discipline across many teams and many datasets, the Soda Cloud features are what makes that step possible.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Browse datasets

The **Datasets** page displays **all datasets that have been onboarded into Soda Clou**d—either through publishing a contract or via the [onboarding process](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud).

It provides a quick overview of each dataset’s health, showing at a glance if a dataset has **issues**, **how many checks** from its contract **are failing**, how many **anomalies** were detected through metric monitoring, and **when the last scan was executed**.

You can filter datasets by properties like data source, owners, arrival time, attributes, or flags such as failures or anomalies. Use the search bar to quickly find a specific dataset by name, and sort the list by name, creation time, or data quality status.

> Learn more about [custom attributes](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FcTNeK4YHihYIYfho5MCb%2Fimage.png?alt=media&amp;token=233a7707-c8ea-4fe5-a9af-f8dfb89abb97" alt=""><figcaption></figcaption></figure>

You can also sort the datasets list by name, creation date, check failures, or arrival time, among other options, to prioritize your focus.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FrIJzEeNM6GaoKQwCyx9i%2Fimage.png?alt=media&amp;token=61da53c9-d684-4ac7-8415-eb2036edfd3d" alt=""><figcaption></figcaption></figure>

## Customize your datasets' view

You can **tailor the Datasets view** to focus on the areas that matter most to you:

{% stepper %}
{% step %}
Use the **filter options** to narrow down the view.
{% endstep %}

{% step %}
Click the **Save** button to store your current filter configuration as a **collection**. Enter a **name** for the collection and click **Save**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fg0KZ0RFb5ZtvhFjJlmYp%2Fimage.png?alt=media&amp;token=3b39c0d9-3a2f-45a7-a2f7-4789f0643151" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Once saved, your collection will be available in the **dropdown at the top right** of the dashboard. Simply select it to switch views.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FXJMTBB9ejmXX3UX3durD%2Fimage.png?alt=media&amp;token=904deb6e-e1c5-4ae5-b669-833c1e52e447" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Use the **context menu** next to the collection name to:

* **Delete** the collection if it’s no longer needed.
* **Share** the collection with others in your organization.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FRVHwgubl1gzt9NksiD0g%2Fimage.png?alt=media&amp;token=fff62184-c582-4e53-9d70-4a3460025725" alt="" width="377"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Dataset page

The **dataset page** provides a detailed view of each dataset’s health and monitoring information in Soda Cloud. It includes several tabs to help you explore and manage data quality at the dataset level.

## Checks tab

The **Checks** tab displays the **results of all checks defined in the dataset’s contract**. Checks are grouped by column, with column-level checks nested under their respective columns. **Columns with failed checks are automatically expanded** so you can spot issues quickly. You can filter the view to show only failed checks and search for specific checks or columns by name for faster navigation and troubleshooting.

> Learn more about [Data Testing](https://docs.soda.io/soda-documentation/find/data-testing).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FzNiwUkc1HftjjbpuX6db%2Fimage.png?alt=media&amp;token=2d9591e3-e32f-40bd-8060-bf2bcb9bc142" alt=""><figcaption></figcaption></figure>

## Metric Monitors tab

The **Metric Monitors** tab shows the **metrics that are actively monitored for the dataset**, helping you track trends and detect anomalies over time.

> Learn more about the [Metric monitoring](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FkEnxkjWuOJdRQdHigtWD%2FScreenshot%202025-05-29%20at%205.09.51%20PM.png?alt=media&amp;token=bae6d41c-f1dd-4232-9e4c-8486057a7d08" alt=""><figcaption></figcaption></figure>

## Columns tab - Profiling

The **Columns** tab provides an **overview of the dataset’s structure**, including column names, data types, distinct counts, and other statistics. You can search for a specific column by name to quickly locate and review its profiling details.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FzkPvLbPABh3XYCRRO7im%2Fimage.png?alt=media&amp;token=6b413cf1-3af3-4c64-9974-45356c90a3b0" alt=""><figcaption></figcaption></figure>

## Scans tab

The **Scans** tab lists **all scans that have been executed for the dataset**. This gives you a dataset-level view of **what has been scanned**, **why it was scanned**, and **what the outcome was**.

Typical scan types include:

* **Contract verification** scans to validate that the dataset matches the expectations defined in its contract.
* **Metric monitoring** scans that compute and track dataset metrics over time.
* **Profiling** scans to generate profiling results for the dataset (such as statistics and distribution insights).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FT1UIe9wRIxzGKmxTlyQV%2Fimage.png?alt=media&amp;token=a6883e96-6b3b-4dfd-a3b2-1d6147b0d287" alt=""><figcaption></figcaption></figure>

From the Scans tab, you can:

* **Review the list of all scans** related to the dataset.
* **Open a specific scan** to inspect its details and results.
* **Access the scan execution logs** to support troubleshooting and root-cause analysis.

## Incidents tab

The **Incidents** tab lists **incidents related to the dataset**, helping you track issues and collaborate on resolution. This includes incidents that also link checks from other datasets. You can filter incidents based on criteria such as **user lead**, **status**, or **severity** to focus on the most important or urgent cases, and click **Create incident** to open a new incident for one of the dataset's checks.

> Learn more about [incidents](https://docs.soda.io/soda-documentation/fix/incidents).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FsEZbveilg5VRMlNcQXMp%2Fimage.png?alt=media&amp;token=cafd7acb-d8d2-4186-9690-cc9db8f3c2a1" alt=""><figcaption></figcaption></figure>

## Requests tab

The **Requests** tab displays **all contract change requests associated with the dataset**. It enables dataset owners and data consumers to collaborate on improving the dataset contract over time.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fn925aiwjdL1bDkUpWwuN%2FScreenshot%202025-08-20%20at%2010.18.33%E2%80%AFAM.png?alt=media&amp;token=40b0e157-ea50-4ad1-8759-6fd245cefffb" alt=""><figcaption></figcaption></figure>

In the **Requests** tab, you can:

* **Browse all requests** for the dataset.
* **Search and filter requests** by status (such as **Open**, **Done**, or **Won’t Do**).
* **Open a request** to review and collaborate via:
  * Messages and comments between participants
  * Proposals showing suggested contract changes (including added/modified/removed elements)

This workflow supports structured **contract collaboration** where consumers can propose changes directly, and dataset owners remain the final decision-makers for publishing updates.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Browse checks

The **Checks** page displays all checks defined in a data contract and tracked in Soda Cloud. It provides a quick overview of check health across datasets, allowing you to create **custom groupings** by applying filters such as data source, dataset, owners, or status (pass, fail, warning). This helps you focus on specific areas or teams that matter most.

You can also review key details like the check type, the dataset it belongs to, and the time of the last scan. Use the search bar to quickly find a specific check by name, and sort the list by name, last run time, or check status.

You can filter checks by properties such as data source, dataset, owners, attributes, or status (pass, fail, warning). Use the search bar to quickly find a specific check by name.

> Learn more about custom attributes: [Check and dataset attributes](https://docs.soda.io/soda-documentation/govern/check-and-dataset-attributes)

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FvDxHQk1qbcI2iIMcbl7R%2Fimage.png?alt=media&amp;token=8e7f784d-6916-41bb-877f-1cd3ae729ce8" alt=""><figcaption></figcaption></figure>

You can also sort the list by name, last run time, or check status.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FFOtO6gTbGJsfL0Rab7QT%2Fimage.png?alt=media&amp;token=4d790d3f-e22f-418d-9cf0-115654b41dfc" alt=""><figcaption></figcaption></figure>

## Customize your check view

You can tailor the Checks view to focus on the areas that matter most to you:

{% stepper %}
{% step %}
Use the filter options to narrow down the view
{% endstep %}

{% step %}
Click the **Save** button (top right) to store your current filter configuration as a **collection**. **Enter a name** for the collection and click **Save**

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F5yOa5SrRVw1tL06Wd8p5%2Fimage.png?alt=media&amp;token=36181198-0bdb-4593-8ac6-298a55475921" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Once saved, your collection will be available in the **dropdown** at the top right of the dashboard. Simply select it to switch views.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FNxunUjd3lrB8l4WuCLDM%2Fimage.png?alt=media&amp;token=c0895941-fbe0-42bf-bc4d-20149c38c45c" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Use the context menu next to the collection name to:

* **Delete** the collection if it’s no longer needed.
* **Share** the collection with others in your organization.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F6qM0NtcHpABTkLkgdiG6%2Fimage.png?alt=media&amp;token=639a7525-9bc5-4be9-aa3d-1cec6c512281" alt="" width="388"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Analyze monitor and check results

When you access a **monitor** (from Metric Monitoring) or a **check** (from the Contract), Soda Cloud provides a **time series view** that shows how the monitored metric or check result evolves. This helps you explore the history of data quality issues, spot trends, and understand changes in your data.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FHARPpXJraqVzHkRWt0G3%2Fimage.png?alt=media&amp;token=dee5fbb6-3748-42ca-b78e-9e083c5450ed" alt=""><figcaption></figcaption></figure>

## Diagnostics

For certain types of checks, **additional diagnostic information** is also available for check results to help you investigate issues in more detail.

For example:

* **Schema Checks**: View a side-by-side comparison of the **actual vs. expected schema** to identify differences.
* **Missing, Duplicate, or Invalid checks**: See the percentage of failed rows vs. passing rows, including a breakdown of **valid**, **missing**, **invalid**, and **duplicate** rows, as well as the **number of tested rows**, to understand the scale and impact of the issue.
* **Reconciliation checks**: View reconciliation insights from both the **source** and the **target** datasets to compare results, identify gaps, and better understand mismatches.

This view helps you drill down into specific data issues, explore context, and take informed action.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FIzLAxrYhOWUaMKzxZ9v4%2FScreenshot%202026-01-26%20at%204.01.52%E2%80%AFPM.png?alt=media&amp;token=27601b17-9d1b-4ad2-a3dd-2a30ca80bb27" alt=""><figcaption></figcaption></figure>

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Diagnostics Warehouse

Diagnostics Warehouse provides a clear, detailed view of the state of data checks while allowing access to failed rows in order to take a closer look and resolve data quality issues.

## Overview

### What is Diagnostics Warehouse?

The **Diagnostics Warehouse is a place inside your data warehouse** (in one or multiple schemas) **reserved for Soda to store data quality information**.

***

With Diagnostics Warehouse, Soda stores all scans, failed records, and historical data quality issues directly in your own data warehouse. **Nothing is stored outside.** This gives you the ability to run diagnostics, resolve issues, and see exactly why problems happen. You can go as deep as you need, from a single record to a full dataset.

Each time a Soda scan runs, Diagnostics Warehouse stores failed rows together with **check**, **scan**, and **dataset attributes & metadata**. With that information, data teams can quickly **diagnose and resolve issues at both row and dataset level**. Additionally, Soda's Diagnostics Warehouse makes it easier for teams to build on top of Soda's outcomes to set up operation workflows, and connect to BI tools you already know and trust.

### Features & capabilities

* **Full diagnostic information in one place**, including attributes.
* **Faster root-cause analysis:** jump from a failed check to the exact failed rows, affected datasets/columns, and prior history to see if it’s a one-off issue or a pattern.
* **Open & portable features:** it’s just tables in your warehouse. Query with SQL, power dashboards, join with lineage, incident, or cost data, and automate workflows.
* **Privacy and transparency:** Diagnostics Warehouse stores tables in your own warehouse, giving you full control over security, retention and access.

### Security & governance

* **Data minimization:** Diagnostics Warehouse stores metadata about runs and checks and, **for row-level checks**, it only stores failed rows when the option is enabled.
* **Warehouse residency:** Diagnostics are **not stored in Soda**. They live in your analytics warehouse, respecting your access controls, encryption, and audit trails.
* **No destructive access**: Soda does not require permissions to delete data. It only **evaluates** checks, **stores** metadata, and **moves** failed rows into a designated location.\
  Cleanup and garbage-collection queries are provided, but Soda never drops tables itself.

### Supported data sources

Diagnostics Warehouse integrates with a range of data sources so that you can centralize, analyze, and act on diagnostics data across your organization.

| Data source                | Support status    | Notes                               |
| -------------------------- | ----------------- | ----------------------------------- |
| **Athena**                 | ✅ Fully supported | Full functionality and scalability. |
| **BigQuery**               | ✅ Fully supported | Full functionality and scalability. |
| **Databricks**<sup>1</sup> | ✅ Fully supported | Full functionality and scalability. |
| **Dremio**                 | ✅ Fully supported | Views are not supported in Dremio.  |
| **MySQL**                  | ✅ Fully supported | Full functionality and scalability. |
| **OracleDB**               | ✅ Fully supported | Full functionality and scalability. |
| **Postgres**               | ✅ Fully supported | Full functionality and scalability. |
| **Snowflake**              | ✅ Fully supported | Full functionality and scalability. |
| **SQL Server**             | ✅ Fully supported | Full functionality and scalability. |
| **Redshift**               | ✅ Fully supported | Full functionality and scalability. |
| **MS Fabric**              | ✅ Fully supported | Full functionality and scalability. |
| **Synapse**                | ✅ Fully supported | Full functionality and scalability. |

<sup>1</sup> Supports Spark DataFrame scans in Databricks when Diagnostics Warehouse is configured on Databricks/Unity Catalog.

***

### Get started

1. **Enable Diagnostics Warehouse** in your Soda data source settings.
2. Grant the service **identity permission** to create and write to the Diagnostics Warehouse schema in your warehouse.
3. **Run your checks**; Diagnostics Warehouse tables populate automatically.
4. Query your warehouse and connect to your BI tools to **start exploring**.

> **Next**: to enable Diagnostics Warehouse in your organization, **reach out to Soda at** [**sales@soda.io**](mailto:sales@soda.io).

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Incidents

Incidents help you track, investigate, and resolve data quality issues when they occur. An incident is created when a data issue, such as a failed or warning check, has been confirmed and assigned to someone for resolution.

An incident is linked to one or more checks through their check results. The checks can belong to different datasets and data sources, so a single incident can track an issue that surfaces in several places, such as a failed load that breaks checks on several downstream tables.

## How to create an incident

{% hint style="warning" %}
Only users with the **Manage incidents** permission on a check's dataset can create an incident for that check. Read about [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles) and [who can view and manage an incident](#who-can-view-and-manage-an-incident).
{% endhint %}

You can start an incident from several places in Soda Cloud:

* On the **Incidents** page, click **New Incident**.
* On a **Dataset Page**, open the **Incidents** tab and click **Create incident**.
* In a list of checks, such as the **Checks** tab of a dataset, open the **context menu** of a check and select **Create Incident**.
* On a check or monitor page, open the **context menu** of a result, such as the one on the **Result** tab of a check, and select **Create Incident**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-9bbf991c49129ab18aa110a1bcf3c2838d5a5000%2Fincidents-check-context-menu.png?alt=media" alt="Context menu of a check result on the Result tab of a check page, showing Create Incident and Link To Incident"><figcaption></figcaption></figure>

{% stepper %}
{% step %}

#### Select the check

Select the **data source**, the **dataset**, and the **check** the incident is about. If you start from a dataset, the dialog opens with that dataset selected; if you start from a check or a result, it opens with that check selected.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-c95ba5e3215e51b711dc26c13e1cf8c7225465f3%2Fincidents-create-dialog.png?alt=media" alt="Create incident dialog with the data source, dataset, check, and check result selection above the title, severity, lead, and description fields" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Connect the incident to check results

Select one or multiple **check results** of that check to associate with the incident. A check result can belong to only one incident, so results that are already linked to another incident can't be selected.
{% endstep %}

{% step %}

#### Describe the incident

Provide a **title**, a **severity**, and a **description**, and optionally assign a **lead**.
{% endstep %}

{% step %}

#### Create the incident

<i class="fa-octagon-check">:octagon-check:</i> Click **Save** to proceed
{% endstep %}
{% endstepper %}

An incident starts with one check. To add checks from the same dataset or from other datasets, link them to the incident.

## Link a check to an incident

You can link a check to an existing incident from the check or from the incident page.

{% hint style="info" %}
Only unresolved incidents accept new checks. A resolved incident can't be changed, and the link dialog doesn't list resolved incidents.
{% endhint %}

### From a check

{% stepper %}
{% step %}

#### Open the link dialog

In a list of checks, open the **context menu** of a check and select **Link To Incident**. On a check or monitor page, you can do the same for a single result.
{% endstep %}

{% step %}

#### Select the incident

The list shows the unresolved incidents on the check's dataset, newest first. To pick an incident from another dataset, select **Search incidents across all datasets**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-e8e175b1e40af53df230d242f4e9ec5b491aae6d%2Fincidents-link-to-incident-dialog.png?alt=media" alt="Link to Incident dialog with Search incidents across all datasets selected and the warning about adding a dataset to the incident" width="490"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Select the check results

If you start from a check whose latest result is already linked to an incident, select which of its results to link. Otherwise, Soda links the result you started from: the check's latest result, or the result you selected on a check or monitor page.
{% endstep %}

{% step %}

#### Link the check

Click **Link**.
{% endstep %}
{% endstepper %}

When the incident doesn't include the check's dataset yet, the dialog warns you before you link. Users without access to that dataset still see the incident, but not this check or its results.

### From the incident page

{% stepper %}
{% step %}

#### Open the link dialog

On the incident page, open the **context menu** next to **Checks** and select **Link A New Check**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-2e94cade6be2a9495d621301c216cafe6a062986%2Fincidents-checks-context-menu.png?alt=media" alt="Context menu next to the Checks dropdown on the incident page, showing Link A New Check and Unlink This Check"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Select the check and its results

Select the **data source**, the **dataset**, and the **check**, then select the check results to link. The dialog warns you when the dataset is not part of the incident yet.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-f91f40e5b277d75806f00e3bae2328c4a436f7b6%2Fincidents-link-new-check-dialog.png?alt=media" alt="Link a new check dialog with the data source, dataset, check, and check result selection" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Save the link

Click **Link check and save**. Soda saves the link right away, without the need to click **Save** on the incident page.
{% endstep %}
{% endstepper %}

## Review the checks of an incident

The **Checks** section of the incident page shows one linked check at a time. Use the dropdown to switch between checks, or type in it to filter the list. Each entry shows the check name, the column the check applies to, if any, and the outcome of the check's latest result, so you can see which checks have recovered since the incident was created. When the incident spans more than one dataset, each entry also starts with its data source and dataset.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-c3d352db2e6cf0afbad4d377c9635e0ef560188e%2Fincidents-checks-dropdown.png?alt=media" alt="Checks dropdown on the incident page listing checks from two datasets, each with its latest result outcome, data source, and dataset"><figcaption></figcaption></figure>

Below the dropdown, a table lists the results of the selected check, with the results that are linked to the incident selected. To change which results are linked, select or clear results and click **Save**. If you clear every result of a check, saving unlinks the check from the incident.

The **Relates to** panel groups the linked checks by dataset. Each group shows the dataset with its data source, followed by the checks linked from that dataset, each with the column it applies to, or **Dataset check** for a check that isn't tied to a column.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-36bcec737cf2c30933c8e89b7e6d6d78e69f1e42%2Fincidents-incident-page.png?alt=media" alt="Incident page of an incident linked to checks from several datasets, with the Checks section and the Relates to panel grouping the checks per dataset"><figcaption></figcaption></figure>

## Unlink a check from an incident

Unlinking a check removes all of its check results from the incident. You can unlink a check from the incident page, or from a check or monitor page.

On the incident page, select the check in the **Checks** dropdown, open the **context menu** next to **Checks**, and select **Unlink This Check**. Confirm with **Unlink and save**; Soda saves the change right away.

On a check or monitor page, open the **context menu** of a result that is linked to an incident, select **Unlink From Incident**, and confirm with **Unlink**.

{% hint style="info" %}
An incident always keeps at least one check. **Unlink This Check** is unavailable while only one check is linked, and unlinking the only check from a check or monitor page is refused. To replace an incident's only check, link the new check first, then unlink the old one.
{% endhint %}

## Who can view and manage an incident

Access to an incident follows the datasets of its linked checks. When an incident spans several datasets, you need a permission on only one of them to open or manage the incident, but you only see the checks on datasets you can view.

| Action                                                                | Required permission                                                      |
| --------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| Open an incident                                                      | **View dataset** or **Manage incidents** on at least one of its datasets |
| See a linked check and its results                                    | **View dataset** on the check's dataset                                  |
| Create an incident                                                    | **Manage incidents** on the check's dataset                              |
| Edit the title, description, severity, lead, or status, or resolve it | **Manage incidents** on at least one of the incident's datasets          |
| Link or unlink a check                                                | **Manage incidents** on that check's dataset                             |

When part of an incident is hidden from you, a banner at the top of the incident page says so. The hidden checks stay linked, whatever you change on the page, and the root cause analysis report is not shown. If you don't have **Manage incidents** on any of the incident's datasets, the page is read-only.

{% hint style="info" %}
Linking a check from another dataset also opens the incident to users of that dataset. They can see the incident's title, description, and status, but only the checks on datasets they can view.
{% endhint %}

## View incidents for a dataset

Once created, the incident will appear in the **Incidents tab** of the corresponding **Dataset Page**. An incident that links checks from several datasets appears in the Incidents tab of each of them.

It is possible to filter incidents based on lead, status, reporter, and severity.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FsEZbveilg5VRMlNcQXMp%2Fimage.png?alt=media&amp;token=cafd7acb-d8d2-4186-9690-cc9db8f3c2a1" alt=""><figcaption></figcaption></figure>

## View incidents across the organization

Incidents can also be seen in a central place in Soda Cloud. In the top navigation, click on Incidents to see all the incidents of the organization.

Use the title search and the filters to find relevant incidents. You can filter on data source, dataset, lead, reporter, severity, and status.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-f64010c78bbd538beb1e52eae567831a33e6d645%2Fincidents-organization-filters.png?alt=media" alt="Incidents page with the title search, filtered on one data source with the Data Source filter"><figcaption></figcaption></figure>

## Updating an incident

### Assign a lead

Assign a **lead:** the user responsible for resolving the issue.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F3rDRBTHpSxqqFbdlRDD0%2Fimage.png?alt=media&amp;token=690057f2-aab4-417d-8315-ed19928192c3" alt=""><figcaption></figcaption></figure>

### Update status

Track progress by updating the incident’s status as the investigation and resolution evolve.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FNnGNdPg0FtAf0wC08yWi%2Fimage.png?alt=media&amp;token=a1d727e7-fbc3-4453-bf6b-f1194ab9137f" alt=""><figcaption></figcaption></figure>

### Add a resolution note

When marking an incident as **resolved**, a resolution note is mandatory to document what was done.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F6rGdlsd10Od3HVaLRswL%2Fimage.png?alt=media&amp;token=8a662186-dd8b-4218-919d-ffe6f92d3671" alt=""><figcaption></figcaption></figure>

### Include more check results

If new results are failing, you can include them in the incident: select them in the results table of their check on the incident page, or link another check.

> Read how in [Review the checks of an incident](#review-the-checks-of-an-incident) and [Link a check to an incident](#link-a-check-to-an-incident).

{% hint style="success" %}
After changing fields or check results, click **Save** or **Resolve** on the top right corner to apply them. Linking and unlinking a check through the context menu next to **Checks** is saved right away.
{% endhint %}

## Delete a check that is linked to an incident

Deleting a check removes its check results from every incident they're linked to. An incident that has other checks linked keeps them; an incident whose only check you delete is deleted as well. Before you confirm, the delete dialog shows how many incidents will be deleted and how many will only lose the check.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-c409bcdf5e5e9fa86fba423db328a59a13545073%2Fincidents-delete-check-dialog.png?alt=media" alt="Delete check confirmation listing the incidents that will be deleted and the incidents that will also be affected" width="563"><figcaption></figcaption></figure>

## Integration with external systems

You can integrate incidents with Slack, MS Teams, or other external systems using Soda’s **webhook** capabilities or the **Incidents API**.

> Learn on how to integrate with Soda: [Integrations](https://docs.soda.io/soda-documentation/integrate/integrations).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fe1qtfzmyPAQpomswkZTC%2Fimage.png?alt=media&amp;token=5bae36ba-064b-4d4f-8e4d-2178a72e9861" alt=""><figcaption></figcaption></figure>

The Incidents API returns each incident with its linked checks, datasets, and data sources, and can link check results to an existing incident or unlink them. It offers the same changes as the incident page, including assigning a lead.

> For endpoints and request fields, refer to the [REST API](https://docs.soda.io/apis/rest-api) reference.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Notifications

Soda’s **notification system** helps you stay informed when data issues occur—whether it’s a failed check or an anomaly detected through metric monitoring. Notifications are dynamically dispatched using **Notification Rules**, allowing you to target alerts based on specific properties, attributes, or datasets.

## How Notification Rules work

Notification Rules define **when** and **to whom** a notification is sent. Rules can be configured to match specific checks or anomalies, ensuring the right people are notified at the right time.

## Create a Notification Rule

{% hint style="warning" %}
Only users with the **Manage Notification Rules** permission can create or edit rules. All users can view rules. Read about [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

To create a new Notification Rule:

{% stepper %}
{% step %}
Click on your profile in Soda Cloud and select **Notification Rules** from the menu.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FkoMdTsQcPHu3vO3jgeS2%2Fimage.png?alt=media&amp;token=f6df81ca-5417-4f78-8408-163d0f97a3f8" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **New Rule**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FiM7gjMeykxpXC3QvQRGm%2Fimage.png?alt=media&amp;token=7a1576eb-69a5-487e-ad8d-d7ab280fccc3" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Provide a **name** for the rule.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FI1MLVT3xRLh8QkuEyiQ6%2Fimage.png?alt=media&amp;token=8f837193-4979-4e41-bdde-68c437316e0b" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Define the **Rule Scope**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F8k2cfvI3WggP5VZUTWIp%2Fadd-notification-rule.png?alt=media&amp;token=9eb6733a-45dc-428d-84eb-c5e45a9094e2" alt=""><figcaption></figcaption></figure>

**Checks**:

* **All checks**: The rule applies to every check in your organization.
* **Specific checks**: Build custom rules by filtering on check properties, dataset properties, or attributes.
* **All monitors**: The rule applies to every monitor in your organization.
* **Specific monitors**: Build custom rules by filtering on monitor metric type, dataset properties or attributes.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FrIzopV4iefVNq0JJ6nyu%2Fimage.png?alt=media&amp;token=cd70dced-96bf-47b7-9ab9-9275dd3bd735" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Define the **recipients** (users, groups, or integrations like Slack, Teams, or webhooks, see [Integrations overview](https://docs.soda.io/soda-documentation/integrate/integrations)).
{% endstep %}

{% step %}
...and choose the **alert type**: Warn & Fail, Fail, Not Evaluated, Excluded, or all results.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FunJrgRsoYADnGmlzANo8%2Fimage.png?alt=media&amp;token=0e17af30-8ab7-4543-9f65-5225cec247ec" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
\[Optional] **Advanced options**

* Enable **Granular Results** to include column-level details and check results.
* Enable **Only checks matching this rule** to report just the checks the rule matched, instead of the whole contract verification.
* Enable **Personalized Message** to include context for this notification rule.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FIFb2DhgvXWhrUE1CE9hU%2Fimage.png?alt=media&amp;token=0b4bc25e-bc17-4b68-98cc-afd2e113c04d" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Save** to create the Notification Rule.
{% endstep %}
{% endstepper %}

***

## Pause Notification Rules

You can **pause a Notification Rule** at any time to temporarily disable alerts without deleting the rule.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FhG9Jo7IkOcZuw9ozCPnn%2Fimage.png?alt=media&amp;token=48683a9c-9e60-496e-ab03-c1440504c1cd" alt=""><figcaption></figcaption></figure>

***

## Disable legacy agreement notifications

If your organization previously used agreement-level notifications (Soda v3), an organization **admin** can disable them in **Organization Settings** → **Agreement notifications**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fb60aWG9tOXsFtyG0w5Ov%2Fimage.png?alt=media&amp;token=dc992c6d-435f-44f5-823f-e5de84cde425" alt=""><figcaption></figcaption></figure>

With agreement notifications off, all alerts are dispatched through Notification Rules instead:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fo4f46LMRoYr48e7lQLRP%2Fimage.png?alt=media&amp;token=e5484ffc-1bd0-4a75-8637-56f0a100b621" alt=""><figcaption></figcaption></figure>

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Check and dataset attributes

Attributes allow you to add descriptive **metadata** to your datasets and checks. This metadata can then be:

* **Used for filtering** in Soda Cloud, making it easier to search and organize datasets and checks based on specific criteria (e.g., business domain, sensitivity, criticality).
* **Leveraged in reporting**, enabling you to group datasets, track ownership, and monitor data quality across different categories or dimensions.

Adding meaningful attributes enhances discoverability, governance, and collaboration within Soda and its integrations.

## Creating an attribute

{% hint style="warning" %}
Only users with the **Manage Attributes** permission can create or edit attributes. [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

To create a new attribute:

{% stepper %}
{% step %}
Click your profile icon in the top-right corner and select **Attributes** from the menu.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FweSnRBc9NfkiGuhk1pt1%2Fimage.png?alt=media&amp;token=d7996a95-40b5-4365-b5fd-6a6848f0bda2" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **New Attribute**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FaVXi2n81tQs9WDjn7nW0%2Fimage.png?alt=media&amp;token=e5ba1208-b6e2-491f-b278-8f8c4375a4f7" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Provide a **Label** for the attribute. Note that a unique **name** will be generated from this label. This name is immutable and is used in Data Contract definitions to reference the attribute.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FrxCYMHEp0g94qyuyuy2Z%2Fimage.png?alt=media&amp;token=a6c7a385-f197-4a49-ad43-6062e786e9c0" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Select the **Resource Type** where the attribute applies: **Dataset**, **Check**, or **Column**

{% hint style="info" %}
The **Column** resource type is part of **Data Standards** and is only available when Data Standards is enabled for your organization.
{% endhint %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FZ4zVDs7o6Yhiqd2r46WU%2Fimage.png?alt=media&amp;token=6fd5eb06-a587-4c3c-8382-b3adadbe0574" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Choose the **Type** of attribute: Single select, Multi select, Checkbox, Text, Number, Date

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FP50xIKLuG7Y4PdI9DAO8%2Fimage.png?alt=media&amp;token=f351457b-ebd1-415e-8079-667a1efef533" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Add a **Description** for context.
{% endstep %}

{% step %}
Click **Save**
{% endstep %}
{% endstepper %}

## Edit attributes

To edit an attribute, use the context menu next to the attribute name and select **Edit Attribute**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FECamWYcOsjPnLIztJKIy%2Fimage.png?alt=media&amp;token=3ba92fd6-a31e-42f9-a4a6-dea821b6b96f" alt=""><figcaption></figcaption></figure>

Note that the name property and the assigned resource type cannot be changed.

## Attributes in datasets

> Learn how to set attributes for datasets: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)

## Attributes in columns

> Learn how to set attributes on individual columns: [Column attributes](https://docs.soda.io/soda-documentation/govern/column-attributes)

## Attributes in checks

Attributes for checks will be defined as part of the **Data Contract**.

> Learn how to set attributes for datasets:
>
> * Authoring in Soda Cloud: [Author a contract in Soda Cloud](https://docs.soda.io/soda-documentation/find/data-testing/cloud-managed-data-contracts/author-a-contract-in-soda-cloud#define-attributes)
> * Data Contract as code: [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference#check-attributes)

## Attributes in filters

Once an attribute has been assigned at least once, either to a dataset or a check, it becomes available as a filter in Soda Cloud. Attributes that have not yet been used will not appear in filter options.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F7l7WpMKIu4iIA2qzpyvY%2Fimage.png?alt=media&amp;token=2661f09e-f01f-49b1-be3b-6d1eeb6ba930" alt=""><figcaption><p>The attribute <strong>Team</strong> has several possible values (core, platform, backend, etc), which allows to filter datasets by <strong>Team</strong></p></figcaption></figure>

You can also filter check results by attribute directly on the dataset page's **Checks** tab, with the health score updating to reflect the filtered subset.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Dataset attributes & responsibilities

Dataset settings allow you to define key metadata, ownership, and business context for your datasets. This information helps ensure data governance, accountability, and seamless integration with other tools like your data catalog.

## Dataset Owner

#### Purpose of the Dataset Owner

Each dataset should have a designated **dataset owner**: a person or team responsible for the dataset's quality, availability and usage.

**Typically, the role of a Dataset Owner includes:**

* Defining and maintaining the dataset's purpose and documentation.
* Ensuring the dataset meets data quality standards and contract requirements.
* Responding to issues, such as failed checks or data quality alerts.
* Reviewing and approving changes to the dataset schema or contract.

#### Updating the Dataset Owner

{% hint style="warning" %}
Only users with the **Configure Dataset** permission on the dataset can update the dataset owners. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

To assign a **Dataset Owner**:

{% stepper %}
{% step %}
Open the **dataset page**.
{% endstep %}

{% step %}
Click the **context menu** (⋮) in the top-right corner and select **Edit Dataset**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FLfhVVZ8KkKQllFYiPrdj%2Fimage.png?alt=media&amp;token=cef4f909-73ad-4085-873a-7a1fa9f971d8" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
In the **Owned by** section, select one or more users and/or user groups.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Ff48t1eFD34mT5riwfGmn%2Fimage.png?alt=media&amp;token=203796ed-6814-4d79-aef8-b259992d6587" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Fill in the optional information and click **Save** (top right) to apply the changes.
{% endstep %}
{% endstepper %}

## Responsibilities

#### What are Responsibilities?

**Responsibilities** allow you to assign permissions to users or user groups, ensuring they have the access they need to work with a dataset.

A **Responsibility** is a combination of:

* A **User** or **User Group**.
* A **Dataset Role**, which is a predefined collection of permissions (such as the ability to edit contracts, view checks, or manage settings).

By assigning Responsibilities, you define **who can do what** for each dataset, supporting clear ownership, governance, and collaboration.

> Learn about defining custom roles: [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles).

#### How to Add Responsibilities

Managing responsibilities requires the following dataset role: "Manage dataset responsibilities"

To assign a Responsibility to a user or group:

{% stepper %}
{% step %}
Open the **dataset page**.
{% endstep %}

{% step %}
Click the **context menu** (⋮) in the top-right corner and select **Edit Responsibilities**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FQLlq87Ey0BN2ZHNxf0ba%2Fimage.png?alt=media&amp;token=02b31f04-6763-4a22-bfd7-ed4311e2e3f7" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Add the desired **users** or **user groups**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F9Qi3GKkiDGenYC7rGS21%2Fadd-user-to-responsibility.png?alt=media&amp;token=8c55d521-cc36-4ba1-a844-4c6a55aa4a91" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Select the appropriate **Dataset Role** for each.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FXrCdqhhVDcoHX9voGAXW%2Fselect-role.png?alt=media&amp;token=5d1ecee4-dfe7-4bb0-9169-6b380550c525" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Save** to apply the changes.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FDTT2fEFy9woIC6ziJJyb%2Fedit-responsibility-save.png?alt=media&amp;token=7fb47bda-7e3a-4f4b-b01f-2a57d2a54818" alt="" width="535"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### Default Dataset Owner role

Every dataset has a **default Dataset Owner role**, automatically assigned to the designated Dataset Owner(s).

* This role provides essential permissions to manage and maintain the dataset.
* The Dataset Owner role **cannot be removed**, but it can be combined with other roles for additional permissions.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fw7sLOR7VpUICVdFK7iRY%2Fdefault-owner-responsibility.png?alt=media&amp;token=011c8a42-43b0-42a1-bc59-b1a783cb62ab" alt="" width="535"><figcaption></figcaption></figure>

{% hint style="info" %}
The **default permissions** granted to the Dataset Owner role are customizable at the organization level. For more details on configuring the default Dataset Owner role and other roles, see the [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles#define-default-responsibilities)
{% endhint %}

## Data source responsibilities

Responsibilities can also be assigned at the **data source** level, mapping a user or user group to a **data source role**. A user must have a data source responsibility to see the datasets and checks onboarded from that data source.

{% hint style="warning" %}
Only users with the **Manage responsibilities** permission on the data source can manage data source responsibilities. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

To assign a responsibility on a data source:

{% stepper %}
{% step %}
Open the **data source page**.
{% endstep %}

{% step %}
Click the **context menu** (⋮) in the top-right corner and select **Edit Responsibilities**.
{% endstep %}

{% step %}
Add the desired **users** or **user groups** and select a **data source role** for each.
{% endstep %}

{% step %}
Click **Save** to apply the changes.
{% endstep %}
{% endstepper %}

## Dataset Attributes

{% hint style="info" %}
Only users with the **Configure Dataset** permission on the dataset can update the attributes. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

#### Purpose of Attributes

Dataset attributes allow you to add descriptive **metadata** to your datasets. This metadata can then be:

* **Used for filtering** in Soda Cloud, making it easier to search and organize datasets and checks based on specific criteria (e.g., business domain, sensitivity, criticality).
* **Leveraged in reporting**, enabling you to group datasets, track ownership, and monitor data quality across different categories or dimensions.

Adding meaningful attributes enhances discoverability, governance, and collaboration within Soda and its integrations.

Learn how to define attribute types: [Check and dataset attributes](https://docs.soda.io/soda-documentation/govern/check-and-dataset-attributes)

#### Adding Dataset Attributes

You can add or modify dataset attributes in the Dataset Settings page:

{% stepper %}
{% step %}
Click the **context menu** (⋮) in the top-right corner and select **Edit Dataset**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FQLlq87Ey0BN2ZHNxf0ba%2Fimage.png?alt=media&amp;token=02b31f04-6763-4a22-bfd7-ed4311e2e3f7" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Set a value for the existing attribute type. They are all optional.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FvXDPpfxdvvOLy3qqpEJZ%2FEdit-attributes.png?alt=media&amp;token=5a8b0fbf-ee13-4f95-b80c-0d6005bae258" alt="" width="536"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Save** your changes.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FwMlq6qShftDbaCUx9za4%2Fedit-attributes-save.png?alt=media&amp;token=0bdd9d16-f955-4f6d-9b54-acd84c55999c" alt="" width="536"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

## Bulk edit of Attributes and Responsibilities

When managing multiple datasets, you can save time by applying changes in bulk using the **Bulk Edit** feature.

#### How to bulk edit Datasets

{% stepper %}
{% step %}
Go to the **Datasets** page.
{% endstep %}

{% step %}
Select the datasets you want to edit using the checkboxes. Click **Edit** in the action menu.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FRwNdvte1dndDUJbB3nwP%2Fimage.png?alt=media&amp;token=a52139c4-088f-466a-ad44-b7f1ca67bf3d" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Define attributes you want to add or modify across the selected datasets.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FZXw68W1zrBkgDr7OrBLf%2FScreenshot%202025-05-29%20at%2011.21.53%20AM.png?alt=media&amp;token=209544f2-2690-4661-8990-f6a81a1f7c94" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Define responsibilities you want to add or modify across the selected datasets.

Choose whether to **update existing** responsibilities (add new without removing existing) or **reset** (replace all existing responsibilities with the new definition).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FgoZDGE2iTsOWJUHN2qcd%2Fbulk-responsibility.png?alt=media&amp;token=f631db39-7764-4ac7-a4bc-eb132f206f8d" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Continue** to review your changes.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FSeGisSnoLyiuWKeUFBUr%2Fbulk-save.png?alt=media&amp;token=5415a624-1d9f-411c-b54c-9614a6fdc8f8" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

## Integrate with data catalog

You can automate the management of **dataset attributes** and **responsibilities** in Soda Cloud using our **REST API**. This allows you to:

* Programmatically **set or update attributes** for multiple datasets.
* Assign **responsibilities** (users, groups, and roles) to datasets at scale.
* Keep your Soda Cloud configuration in sync with your data catalog or external metadata management systems.

This automation ensures that your metadata stays up-to-date and consistent across your ecosystem, supporting seamless governance and discoverability.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Column attributes

Column attributes let you attach descriptive **metadata to individual columns** of a dataset, in the same way that [dataset attributes](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities#dataset-attributes) describe whole datasets and check attributes describe checks.

Use column attributes to capture column-level business context such as **sensitivity** (e.g. PII, confidential), **data domain**, **classification**, or **ownership**. Once set, column attributes can be:

* **Viewed** on the dataset's column cards and in the column information panel.
* **Set programmatically** through the Soda Cloud REST API, to keep Soda in sync with a data catalog or governance tool.
* **Used in Data Standards** to target columns by their attribute values instead of (or in addition to) their names. See [#use-column-attributes-in-data-standards](#use-column-attributes-in-data-standards "mention").

{% hint style="info" %}
Column attributes are part of **Data Standards** and are only available when Data Standards is enabled for your organization. If you do not see the options described below, Data Standards is not enabled on your plan.
{% endhint %}

## Defining a column attribute

Column attributes use the same attribute catalog as datasets and checks. Before you can set a value on a column, an attribute with the **Column** resource type must exist.

> Learn how to create one: [Check and dataset attributes](https://docs.soda.io/soda-documentation/govern/check-and-dataset-attributes). When creating the attribute, select **Column** as the **Resource Type**.

## Setting column attribute values

{% hint style="warning" %}
Only users with the **Configure Dataset** permission on the dataset can edit column attributes. [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

You set values per column from the dataset page.

{% stepper %}
{% step %}
Open the **dataset page** and locate the column you want to annotate in the **Columns** list.
{% endstep %}

{% step %}
Open the column's **context menu** (⋮) and select **Edit Attributes**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-70b97cfc677789aa29ca5e3642078e7759c9cd9d%2Fcolumn-edit-attributes-menu.png?alt=media" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
In the **Edit Attributes** dialog, set a value for each column attribute. All attributes are optional, and the available input depends on the attribute type (single select, multi select, checkbox, text, number, or date).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-cce4cc6c95d71536b3e1bf2a309c10b5a80d2fe8%2Fcolumn-edit-attributes-dialog.png?alt=media" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Save** to apply the changes.
{% endstep %}
{% endstepper %}

## Viewing column attributes

Once set, a column's attribute values appear:

* As **tags** on the column card, next to the column's data type.
* In the **Custom Column Attributes** section of the column information panel.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fgit-blob-ce058f98d347f42b588ac0cb47fd097bb8b49291%2Fcolumn-attributes-display.png?alt=media" alt=""><figcaption><p>Column attribute values appear as tags on each column card.</p></figcaption></figure>

## Set column attributes via the REST API

You can set or update column attribute values in bulk with the Soda Cloud REST API, which is the recommended way to keep column metadata in sync with an external data catalog or governance tool.

Use the `POST /api/v1/datasets/{datasetId}/columnAttributes` endpoint. The request body maps each column name to a map of attribute name → value. Setting one attribute on a column leaves the column's other attribute values untouched. If any column name, attribute name, or value type is invalid, the **entire request is rejected** and no changes are persisted.

The caller must have the **Configure Dataset** (`CONFIGURE_DATASET`) permission on the dataset.

> See the full request and response schema in the auto-generated API reference: [REST API](https://docs.soda.io/apis/rest-api).

## Use column attributes in Data Standards

Column attributes can drive **Data Standards**: a standard's `match` block can target columns by their attribute values (`equals`, `any_of`, grouped with `all_of` / `any_of`, and `exclude`) instead of, or in addition to, their names. This lets a single standard apply, for example, to every column tagged with a given sensitivity.

> The full Data Standards `match` syntax is documented with Data Standards.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Organization and Admin settings

Configure Soda Cloud to match how your organization governs access, identity, and automation, with enterprise-grade controls and network security.

The **Organization and Admin Settings** in Soda Cloud provide a centralized interface for managing your organization’s configuration, roles, user access, and integrations. From setting your organization’s name to defining global roles, managing user groups, and enabling the Soda-hosted Runner, these settings help you tailor Soda Cloud to your team’s needs and governance policies.

To access the settings, click on your avatar, and then click **Organization Settings**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FLSw8EoT4ag8jjVIpiPWj%2FScreenshot%202025-05-29%20at%202.21.12%E2%80%AFPM.png?alt=media&amp;token=c9140617-f96d-4b1e-b578-6753fd5b46a5" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}
Only users with the **Manage Organization Settings** global role can access and modify organization settings.
{% endhint %}

***

## Built for enterprise governance

Adopting a data quality platform across an organization raises the same questions as adopting any other system of record: Who can log in? How is their identity managed? What can each role actually do? How do we let CI/CD pipelines act without handing out a person's credentials?

Soda Cloud is designed to answer all of those out of the box.

{% columns %}
{% column %}

#### Identity, your way

Federate authentication through your existing SAML 2.0 identity provider (including Azure AD, Okta, and Google Workspace) and let your IdP remain the source of truth for who works at your company and which teams they belong to.
{% endcolumn %}

{% column %}

#### Permissions that match your org chart

Define what each role can do across the organization and per dataset, with a fully customizable role model. Grant exactly the access each team needs without forcing your structure into a rigid template.
{% endcolumn %}
{% endcolumns %}

{% columns %}
{% column %}

#### Network-level access controls

Restrict access to Soda Cloud to your corporate network, VPN, or any defined set of IP ranges. Requests from anywhere else are blocked before they ever reach authentication.
{% endcolumn %}

{% column %}

#### Automation-ready by design

Run scans, verify contracts, and call the API from CI/CD pipelines using service accounts: machine identities that are separate from any individual user and revocable in one click.
{% endcolumn %}
{% endcolumns %}

### Customizable platform

Soda Cloud allows organizations to have full control over each user and group permissions:

* **Organization-level configuration:** set your organization's identity in the platform, control which Soda capabilities are enabled (including AI-powered assistants and dataset profiling), and manage the storage of data source secrets used by your scans.
* **User management:** invite, deactivate, and organize users into groups, with group-based permission assignment so access scales as the team grows.
* **Single Sign-On (SSO):** connect Soda Cloud to your identity provider, with optional one-way sync of user groups directly from the IdP.
* **Global and dataset roles (RBAC):** design the role model that fits your organization, with separate scopes for what users can do across the platform versus what they can do on a specific dataset.
* **Service accounts:** issue API-only identities for pipelines and integrations, with their own permissions and lifecycle independent of any human user.
* **Network controls:** restrict platform access by IP address or CIDR range, enforced before authentication.
* **Multiple environments:** operate separate organizations for production, staging, and development from the same user profile, so each environment has its own data, roles, and credentials.

### Security and access control at a glance

Soda Cloud treats identity as the security perimeter. Every request is authenticated, scoped to the user's organization, and evaluated through role-based access control before any data is returned.

The result is a platform that supports both **least-privilege access** for sensitive datasets and **broad data democratization** for the rest — without forcing one approach on the entire organization.

> Contact <support@soda.io> to learn more about Soda Cloud and what it can do for your organization.

## Add multiple organizations <a href="#add-multiple-organizations" id="add-multiple-organizations"></a>

You may find it useful to set up **multiple organizations in Soda Cloud** so that **each corresponds with a different environment** in your network infrastructure, such as production, staging, and development. Such a setup makes it easy for you and your team to access multiple, independent Soda Cloud organizations using the same profile, or login credentials.

Note that **Soda Cloud associates any** [**API keys**](https://docs.soda.io/apis/generate-api-keys) **that you generate within an organization** with both your profile *and* the organization in which you generated the keys. **API keys are not interchangeable between organizations.**

> Contact <support@soda.io> to request multiple organizations for Soda Cloud.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# General settings

The **General Settings** page allows you to configure foundational settings for your organization in Soda Cloud. These settings impact how your organization operates and how users interact with the platform.

## Organization name

Set the name of your organization. This name appears throughout Soda Cloud, such as in dashboards, reports and notifications.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FRwi93TobRxrfTygkI9c4%2Fimage.png?alt=media&amp;token=a8581160-d008-4070-be08-bf803bcaeb31" alt=""><figcaption></figcaption></figure>

## Allow "Login As"

Enable the **Login As** feature to allow the Soda Support team to log in as an admin within your organization. This can be useful when troubleshooting issues or providing assistance.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F9SSzl9o6O67vo37Hjo9B%2Fimage.png?alt=media&amp;token=d12de627-5dc2-48cc-a4ec-c447f0f6e086" alt=""><figcaption></figcaption></figure>

## Enable or disable the Soda-hosted Runner

You can choose whether to use the Soda-hosted Runner by enabling or disabling it in the Organization Settings:

* Check the **Soda-hosted Runner** option to **enable** or **disable** the runner for your organization.
* Disabling the runner prevents Soda Cloud from running scans or checks via the managed runner. You’ll need to use a self-hosted runner or Soda Core in your environment instead.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FJDjVg3FEvpOozLjqb37b%2Fimage.png?alt=media&amp;token=023635d4-be1e-46c3-aef4-e977e2c31669" alt=""><figcaption></figcaption></figure>

***

## IP Allowlist

{% hint style="warning" %}
This feature requires the **Manage organization settings** permission. Learn more about [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles).
{% endhint %}

IP allowlisting lets you **restrict access** to your Soda Cloud organization to a defined set of **IP addresses** or **CIDR ranges**. Once enabled, any request originating from an IP address not on the list is blocked regardless of whether the user has valid credentials.

Use IP allowlisting when your organization requires network-level access controls. For example, to enforce that Soda Cloud can only be accessed from a corporate network or VPN.

### Enable IP allowlisting

{% stepper %}
{% step %}
**Open Organization Settings**

Click your avatar in the top right corner, then select **Organization Settings**. Navigate to the **Organization** tab.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F4wLQ8pa6d1XHriPn2SGu%2Fimage.png?alt=media&amp;token=ea1af389-d989-4a5d-af54-80a2268b145a" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Enable the restriction**

Check **Restrict access to specific IP addresses**. An empty IP allowlist appears below the checkbox.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FSjPC8TX35Qo7VYIrCqzX%2Fimage.png?alt=media&amp;token=28ac9f5e-a6b8-460e-922e-ee406e49128f" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Add IP addresses or ranges**

Add entries to the list. Each entry can be either:

* An exact IP address, for example `193.121.86.177`
* A CIDR range, for example `192.0.2.0/24`

Add one entry per row.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FKFqAbDyduPRVDetqs1mL%2Fimage.png?alt=media&amp;token=cad05955-35c0-4497-9102-d073ae51d69d" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
**Click Save**
{% endstep %}
{% endstepper %}

Once saved, Soda Cloud will only accept traffic from the addresses on your list. Requests from any other IP are blocked at the network level, before authentication.

{% hint style="danger" %}
**Changes made to the allowlist take effect immediately after saving**. Before enabling IP allowlisting or removing entries, confirm your current IP address is included in the list, if applicable.
{% endhint %}

### Disable IP allowlisting

In order to disable allowlisting and remove all IP restrictions, uncheck the same box: **Restrict access to specific IP addresses** in **Organization Settings > Organization**. All users with valid credentials will immediately regain access regardless of their IP address.

***

## Profiling

Soda can collect dataset profiling information, such as column-level statistics, and display it in Soda Cloud to help users better understand the datasets they are working with.

> Read more about [Profiling](https://docs.soda.io/soda-documentation/find/data-observability/profiling).

If you prefer not to send profiling data to Soda Cloud, you can disable this feature for your organization. Uncheck **Profiling** to turn profiling off.

This ensures that no profiling information is collected or pushed to Soda Cloud. Only check and monitor results, along with the dataset schema required for Data Contract and Metric Monitoring, will be processed.

You can also **manage access to profiling information at the dataset level** in Soda Cloud by configuring dataset-level permissions and applying row-level access controls where supported.

> Learn more about [global and dataset roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FYur5mUQjyIlS2zsKY1Yh%2Fimage.png?alt=media&amp;token=5b48498d-7be0-4fc9-a521-2593cba0f0f6" alt=""><figcaption></figcaption></figure>

## Soda AI

Soda AI settings allow you to **enable AI-powered assistants in Sod**a to help users get answers faster and collaborate more effectively on data contracts.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FFvlJK0TCumJfLx7Vx4tc%2Fimage.png?alt=media&amp;token=b370d910-8f76-4500-9df3-aa6212ba6d4e" alt=""><figcaption><p>Enable or disable Soda AI features on the <strong>organization settings</strong></p></figcaption></figure>

### Enable Soda AI Agent (powered by OpenAI)

When enabled, the **Soda AI Agent** (the in-app AI assistant in Soda Cloud) answers your questions about Soda, and works in Soda Cloud on your behalf, from checking data quality status to taking actions in your org.

> Learn more about [Soda AI Agent](https://docs.soda.io/soda-documentation/soda-ai/soda-ai-agent)

### Enable Contract Copilot (powered by OpenAI)

When enabled, **Contract Copilot** helps users create and improve Soda Contracts with AI-assisted suggestions.

> For more information, see [Soda Contract Copilot](https://docs.soda.io/soda-documentation/soda-ai/contract-copilot).

### Enable Contract Autopilot (powered by OpenAI)

When enabled, **Contract Autopilot** helps you automatically create Soda Contracts tailored to your data.

#### Send row-level data to Soda

To generate a contract via Autopilot, Soda sends dataset metadata to the OpenAI API.

To improve the quality of your auto-generated data contract, you can enable this option to **allow Soda to send a small sample of row data to OpenAI** in addition to the dataset metadata.

## Enable data source secrets

Manage secure storage of secrets such as API keys, credentials or connection details. Secrets can be used in data source configurations, checks, and other automated processes.

> For more information on how to enable secrets, see the [Soda Cloud onboarding flow](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fav89bFAFyiTNEzHfwWR1%2Fimage.png?alt=media&amp;token=300dc0b0-6504-479c-9ef1-291d2231762b" alt=""><figcaption></figcaption></figure>

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# User management

The **Users and User Groups** in Soda Cloud settings allows you to control access to your organization by managing individual users and user groups. This ensures that team members have the appropriate permissions to use Soda Cloud effectively.

With **SSO enabled**, users and groups can be synced directly from your identity provider, reducing manual effort and ensuring alignment with your organization’s access policies. Learn more about SSO configuration [User and user group management with SSO](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/user-and-user-group-management-with-sso).

## Invite Users

{% hint style="warning" %}
Only users with the **Manage organization settings** permission on the organization can invite users. [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

The **Invite Users** feature is only available when **SSO is not enabled**.

To invite users manually:

{% stepper %}
{% step %}
Go to the **Users** tab in **Settings**.
{% endstep %}

{% step %}
Click the **+** icon at the top of the user list

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FuNR02xp4bjos8hMAhhL6%2Fimage.png?alt=media&amp;token=5edfa21e-9645-403c-b0a6-ea42b060d1f2" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Enter the email addresses of the users you want to invite.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FpTdzhOGzebJDWUqVDAKG%2Fimage.png?alt=media&amp;token=46aaf7d0-6f46-458f-bcad-48018aba9af4" alt="" width="469"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

Invited users will receive an email with a link to set their password and join your organization in Soda Cloud. Once they complete the setup, they will have access to Soda Cloud based on the roles and permissions you assign.

## Deactivate Users

{% hint style="warning" %}
Only users with the **Manage organization settings** permission on the organization can deactivate users. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

Deactivating a user **blocks their access to Soda Cloud** and **disables any existing API keys** associated with their account. This is useful when a user no longer needs access, but you want to retain their account for record-keeping or future reactivation.

To deactivate a user:

{% stepper %}
{% step %}
Go to the **Users** tab in **Settings**.
{% endstep %}

{% step %}
Find the user you want to deactivate.
{% endstep %}

{% step %}
Click the context menu for the user and select **Deactivate From This Organization**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fid4P4MFML4wYxec21E0k%2Fimage.png?alt=media&amp;token=36363407-bec6-4e6f-9506-7a5fc7c42849" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

You can reactivate a user later if they need access again.

## Assign User to Groups

{% hint style="warning" %}
Only users with the **Manage organization settings** permission on the organization can manage user groups. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

Assigning users to groups allows you to manage access and permissions more efficiently by applying global roles to groups rather than individual users.

To assign users to groups:

{% stepper %}
{% step %}
Go to the **Users** tab in **Settings**.
{% endstep %}

{% step %}
Find the user you want to assign to a group.
{% endstep %}

{% step %}
Click the content menu next to their name and select **Edit User Groups**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FcVhNJq6J46yuMyJqIg4O%2Fimage.png?alt=media&amp;token=aa0a0819-8325-470c-ab4a-662daa9f8816" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Select one or more user groups from the list.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FsAkMNoxGVc5ntYNyrZ6d%2Fimage.png?alt=media&amp;token=8ec0713d-ff27-4b57-99b8-31b9c0729b54" alt="" width="469"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Save**
{% endstep %}
{% endstepper %}

## Assign Global Roles to Users

{% hint style="warning" %}
Only users with the **Manage organization settings** permission on the organization can assign global roles. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

Global roles define a user’s permissions across Soda Cloud. Assigning global roles directly to users allows you to grant them specific access rights, such as managing datasets, running scans, or configuring organization settings.

To assign a global role to a user:

{% stepper %}
{% step %}
Go to the **Users** tab in **Settings**.
{% endstep %}

{% step %}
Find the user you want to assign a role to.
{% endstep %}

{% step %}
Click the context menu for the user and select **Assign Global Roles**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FWO34IwG4j5GrFlSF2hlm%2Fimage.png?alt=media&amp;token=06354415-fd7a-41b7-8df5-a2d329b7382f" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Choose one or more global roles from the list.
{% endstep %}

{% step %}
Click **Save**
{% endstep %}
{% endstepper %}

## User Groups

{% hint style="warning" %}
Only users with the **Manage organization settings** permission on the organization can manage user groups. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

User groups allow you to manage access and permissions for multiple users at once, helping you simplify and scale permission management across your organization. By assigning global roles to groups, you ensure that all members of the group have consistent access rights, without the need to assign permissions individually.

If you have Single Sign-On (SSO) enabled, user groups can also be synced automatically from your identity provider, ensuring your Soda Cloud user management aligns with your existing access policies. [Learn more about SSO integration](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/user-and-user-group-management-with-sso) .

Note that by default, there is an Everyone group which is not editable and contains all the users from the organization

### Create User Groups

You can manually create user groups in Soda Cloud, whether you’re importing user groups SSO or not.

To create a user group:

{% stepper %}
{% step %}
Go to the **User Groups** tab in **Settings**.
{% endstep %}

{% step %}
Click **Create User Group** at the top of the user group list.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FX9C1eInc5bwvnTK8oVIp%2Fimage.png?alt=media&amp;token=68283778-9867-4d4c-b426-2b8142b2693b" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Enter a **name** for the group.
{% endstep %}

{% step %}
\[Optional] **Add users** to the group immediately, or add them later.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F5rixI3hhmIw68pIHi5wq%2Fimage.png?alt=media&amp;token=a91ff3db-a6cc-4277-8002-7b0dd3480db6" alt="" width="469"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### Edit Group Members

You can edit the members of user groups that you have created on Soda Cloud.

{% hint style="warning" %}
**IDP-managed user groups cannot be edited.** You can recognize them in the **User groups** table by the flag in the **IDP-managed** column.
{% endhint %}

To edit a user group:

{% stepper %}
{% step %}
Go to the **User Groups** tab in **Settings**.
{% endstep %}

{% step %}
Find the group you want to modify.
{% endstep %}

{% step %}
Click the context menu next to the group and select **Edit Members**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FpLRKbtpSRQhEWDADAzoB%2Fimage.png?alt=media&amp;token=b233f13b-c441-49fb-bd65-7bfb6442beac" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Select the users that should be in the user group and click **Save**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FmCaRjdR8eAieheCcnviH%2Fimage.png?alt=media&amp;token=3140101e-7cb2-483a-ad32-3d9ab63b4dd4" alt="" width="469"><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### View Group Members

You can view the list of users in a group to understand who has access through that group and to help manage permissions across your organization.

To view the members of a group:

{% stepper %}
{% step %}
Go to the **User Groups** tab in **Settings**.
{% endstep %}

{% step %}
Click the group name or the row to open its details.
{% endstep %}

{% step %}
The list of users assigned to the group will be displayed.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F5n2UN0AJzdU8zmZZvB3K%2Fimage.png?alt=media&amp;token=9f9005d7-d9c5-413b-9dfb-5db1e1a492a7" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### Assign Global Roles to User Group

Global roles define a user’s permissions across Soda Cloud. Assigning global roles directly to user groups allows you to grant them specific access rights, such as managing datasets, running scans, or configuring organization settings.

To assign a global role to a user group:

{% stepper %}
{% step %}
Go to the **User Groups** tab in **Settings**.
{% endstep %}

{% step %}
Find the user group you want to assign a role to.
{% endstep %}

{% step %}
Click the context menu for the user and select **Assign Global Roles**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FK9s6oWwivDqLBiZe8HP1%2Fimage.png?alt=media&amp;token=f4568e65-6378-45d5-bf02-92ae7020e25b" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Choose one or more global roles from the list.
{% endstep %}

{% step %}
Click **Save**
{% endstep %}
{% endstepper %}

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# User and user group management with SSO

Organizations that use a Security Assertion Markup Language (SAML) 2.0 single sign-on (SSO) identity provider (IdP) can add Soda Cloud as a service provider.

Once added, employees of the organization can gain authorized and authenticated access to the organization’s Soda Cloud account by successfully logging in to their SSO. This solution not only simplifies a secure login experience for users, it enables IT Admins to:

* grant their internal users’ access to Soda Cloud from within their existing SSO solution
* revoke their internal users’ access to Soda Cloud from within their existing SSO solution if a user leaves their organization or no longer requires access to Soda Cloud
* set up one-way user group syncing from their IdP into Soda Cloud (tested and documented for Azure Active Directory and Okta)

## Compatibility <a href="#compatibility" id="compatibility"></a>

Soda Cloud is able to act as a service provider for any SAML 2.0 SSO identity provider. In particular, Soda has tested and has written instructions for setting up SSO access with the following identity providers:

* [Azure Active Directory](#add-soda-cloud-to-azure-a-d)
* [Okta](#add-soda-cloud-to-okta)
* [Google Workspace](#add-soda-cloud-to-google-workspace)

Soda has tested and confirmed that SSO setup works with the following identity providers:

* OneLogin
* Auth0
* Patronus

## SSO access to Soda Cloud <a href="#sso-access-to-soda-cloud" id="sso-access-to-soda-cloud"></a>

When an employee uses their SSO provider to access Soda Cloud for the first time, Soda Cloud automatically assigns the new user to roles and groups according to the [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles)for any new users. Soda Cloud also notifies the Soda Cloud Admin that a new user has joined the organization, and the new user receives a message indicating that their Soda Cloud Admin was notified of their first login. A Soda Cloud Admin or user with the permission to do so can adjust users’ roles in Organization Settings. See [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles) for details.

When an organization’s IT Admin revokes a user’s access to Soda Cloud through the SSO provider, a Soda Cloud Admin is responsible for updating the resources and ownerships linked to the User.

Once your organization enables SSO for all Soda Cloud users, Soda Cloud blocks all non-SSO login attempts and password changes. If an employee attempts a non-SSO login or attempts to change a password using “Forgot password?”, Soda Cloud presents a message that explains that they must log in or change their password using their SSO provider.

Optionally, you can set up the SSO integration Soda to include a one-way sync of user groups from your IdP into Soda Cloud which synchronizes with each user login to Soda via SSO. [#sync-user-groups-from-an-idp](#sync-user-groups-from-an-idp "mention")

Soda Cloud supports both **Identity Provider Initiated (IdP-initiated)**, and **Service Provider Initiated (SP-initiated)** single sign-on integrations. Be sure to indicate which type of SSO your organization uses when setting it up with the Soda Support team.

## Add Soda Cloud to Azure AD <a href="#add-soda-cloud-to-azure-a-d" id="add-soda-cloud-to-azure-a-d"></a>

1. Email <support@soda.io> to request SSO set-up for Soda Cloud and provide your Soda Cloud organization identifier, accessible via **your avatar** > **Organization Settings**, in the **Organization** tab.\
   Soda Support sends you the `samlUrl` that you need to configure the set up with your identity provider.
2. As a user with sufficient privileges in your organization’s Azure AD account, sign in through [portal.azure.com](http://portal.azure.com/), then navigate to **Enterprise applications**. Click **New application**.
3. Click **Create your own application**.
4. In the right pane that appears, provide a name for your app, such as Soda Cloud, then select the **(Non-gallery)** option. Click **Create**.
5. After Azure AD creates your app, click **Single sign-on** in the left nav under the **Manage** heading, then select the **SAML** tile.
6. In the **Basic SAML Configuration** block that appears, click **Edit**.

   <figure><img src="https://docs.soda.io/assets/images/basic-saml.png" alt=""><figcaption></figcaption></figure>

   <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FsQA29oQzvJrgRhWCpvMv%2Fimage.png?alt=media&amp;token=e348b2f4-463b-461c-af6a-9680624a2c0b" alt=""><figcaption></figcaption></figure>
7. In the **Basic SAML Configuration** panel, there are two fields to populate:
   * **Identifier (Entity ID)**, which is the value of `samlUrl` from step 1.
   * **Reply URL**, which is the value of `samlUrl` from step 1.
8. Click **Save**, then close the confirmation message pop-up.
9. In the **User Attributes & Claims** panel, click **Edit** to add some attribute mappings.
10. Configure the claims as per the following example. Soda Cloud uses `familyname` and `givenname`, and maps `emailaddress` to `user.userprincipalname`.\
    (Optional) Follow the additional steps to enable one-way user group syncing to your SSO configuration; see [#add-soda-cloud-to-azure-a-d](#add-soda-cloud-to-azure-a-d "mention")).

    <figure><img src="https://docs.soda.io/assets/images/additional-claims.png" alt=""><figcaption></figcaption></figure>

    <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F9KezOAZbj8y27ZtKSxBU%2Fimage.avif?alt=media&amp;token=1ce9c95a-43c3-4e79-abff-182ea3212457" alt=""><figcaption></figcaption></figure>
11. Scroll down to collect the values of three fields that Soda needs to complete the Azure AD SSO integration:
    * **Azure AD Identifier** (Section 4 in Azure). This is the IdP entity, ID, or Identity Provider Issuer that Soda needs
    * **Login URL** (Section 4 in Azure). This is the IdP SSO service URL, or Identity Provider Single Sign-On URL that Soda needs.
    * **X.509 Certificate**. Click the **Download** link next to **Certificate (Base64)**.
12. Email the copied and downloaded values to <support@soda.io>. With those values, Soda completes the SSO configuration for your organization in cloud.soda.io and notifies you of completion.
    * Soda Cloud supports both Identity Provider Initiated (IdP-initiated), and Service Provider Initiated (SP-initiated) single sign-on integrations; be sure to indicate which type of SSO your organization uses.
    * (Optional) Ask Soda to enable one-way user group syncing to your SSO configuration; see [#set-up-user-group-sync-in-azure-a-d](#set-up-user-group-sync-in-azure-a-d "mention"))
13. Test the integration by assigning the Soda application in Azure AD to a single user, then requesting that they log in.
14. After a successful single-user test of the sign in, assign access to the Soda Azure AD app to users and/or user groups in your organization.

## Add Soda Cloud to Okta <a href="#add-soda-cloud-to-okta" id="add-soda-cloud-to-okta"></a>

1. Email <support@soda.io> to request SSO set-up for Soda Cloud and provide your Soda Cloud organization identifier, accessible via **your avatar** > **Organization Settings**, in the **Organization** tab.\
   Soda Support sends you the `samlURL` that you need to configure the set up with your identity provider.
2. As an Okta Administrator, log in to Okta and navigate **Applications** > **Applications overview**, then click **Create App Integration**. Refer to [Okta documentation](https://help.okta.com/en-us/content/topics/apps/apps_app_integration_wizard_saml.htm) for full procedure.
3. Select **SAML 2.0**.
4. Provide a name for the application, Soda Cloud, and upload the Soda logo.
5. Click **Next**. In the **Configure SAML** tab, there are two fields to populate:
   * **Single sign on URL**, which is the value of `samlUrl` from step 1.
   * **Audience URI (SP Entity ID)**, which is also the value of `samlUrl` from step 1.\
     The values for these fields are unique to your organization and are provided to you by Soda and they follow this pattern: `https://cloud.soda.io/sso/<your-organization-identifier>/saml`.
6. Be sure to use an email address as the application username.
7. Scroll down to **Attribute Statements** to map the following values, then click **Next** to continue.
   * map `User.GivenName` to `user.firstName`
   * map `User.FamilyName`to `user.lastName`
   * map `User.Email` to `user.email`
   * (Optional) Follow the additional steps to enable one-way user group syncing to your SSO configuration; [#set-up-user-group-sync-in-okta](#set-up-user-group-sync-in-okta "mention").
8. Select the following options, then click **Finish**.
   * I’m an Okta customer adding an internal app.
   * This is an internal app that we have created.
9. In the **Sign On** pane of the application, scroll down to click **View Setup Instructions**.
10. Collect the values of three fields that Soda needs to complete the Okta SSO integration:
    * **Identity Provider Single Sign-On URL**
    * **Identity Provider Issuer**
    * **X.509 Certificate**
11. Email the copied and downloaded values to <support@soda.io>. With those values, Soda completes the SSO configuration for your organization in cloud.soda.io and notifies you of completion.
    * Soda Cloud supports both Identity Provider Initiated (IdP-initiated), and Service Provider Initiated (SP-initiated) single sign-on integrations; be sure to indicate which type of SSO your organization uses.
    * (Optional) Ask Soda to enable one-way user group syncing to your SSO configuration; see [#set-up-user-group-sync-in-okta](#set-up-user-group-sync-in-okta "mention").
12. Test the integration by assigning the Soda application in Okta to a single user, then requesting that they log in.
13. After a successful single-user test of the sign in, assign access to the Soda Okta app to users and/or user groups in your organization.

## Add Soda Cloud to Google Workspace <a href="#add-soda-cloud-to-google-workspace" id="add-soda-cloud-to-google-workspace"></a>

1. Email <support@soda.io> to request SSO set-up for Soda Cloud and provide your Soda Cloud organization identifier, accessible via **your avatar** > **Organization Settings**, in the **Organization** tab.\
   Soda Support sends you the `samlURL` that you need to configure the set up with your identity provider.
2. As an administrator in your Google Workspace, follow the instructions in [Google Workspace documentation](https://support.google.com/a/answer/6087519?hl=en\&ref_topic=7559288) to **Set up your own custom SAML application**.
3. Optionally, upload the Soda logo so it appears in the app launcher with the logo instead of the first two letters of the app name.
4. On the **Google Identity Provider details** page, be sure to copy or download the following values:
   * **SSO URL**
   * **Entity ID**
   * **IDP metadata**
   * **Certificate**
5. On the **SAML Attribute mapping** page, add two Google directory attributes and map as follows:
   * Last Name → User.FamilyName
   * First Name → User.GivenName
6. Email the copied and downloaded values to <support@soda.io>. With those values, Soda completes the SSO configuration for your organization in cloud.soda.io and notifies you of completion. Soda Cloud supports both Identity Provider Initiated (IdP-initiated), and Service Provider Initiated (SP-initiated) single sign-on integrations; be sure to indicate which type of SSO your organization uses.
7. In the Google Workspace admin portal, use Google’s instructions to [Turn on your SAML app](https://support.google.com/a/answer/6087519?hl=en\&ref_topic=7559288) and verify that SSO works with the new custom app for Soda.

## Sync user groups from an IdP <a href="#sync-user-groups-from-an-idp" id="sync-user-groups-from-an-idp"></a>

If you wish, you can choose to regularly one-way sync the user groups you have defined in your IdP into Soda Cloud.

Doing so obviates the need to manually create user groups in Soda Cloud that you have already defined in your IdP, and enables your team to select an IdP-managed user groups when assigning ownership access permissions to a resource, in addition to any user groups you may have created manually in Soda Cloud. See: [User management](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/user-management#user-groups)

* Soda has tested and documented one-way syncing of user groups with Soda Cloud for Okta and Azure Active Directory. [Contact Soda](https://community.soda.io/slack) to request tested and documented support for other IdPs.
* Soda synchronizes user groups with the IdP every time a user in your organization logs in to Soda via SSO. Soda updates the user’s group membership according to the IdP user groups to which they belong at each log in.
* You cannot manage IdP user group settings or membership in Soda Cloud. Any changes that you wish to make to IdP-managed user groups must be done in the IdP itself.

### **Set up user group sync in Azure AD**

1. In step 10 of the SAML application setup procedure [above](https://docs.soda.io/soda-cloud/sso.html#add-soda-cloud-to-azure-ad), in the same **User Attributes & Claims** section of your Soda SAML Application in Azure AD, follow [Microsoft’s instructions](https://learn.microsoft.com/en-us/entra/identity/hybrid/connect/how-to-connect-fed-group-claims#add-group-claims-to-tokens-for-saml-applications-using-sso-configuration) to add a group claim to your Soda SAML Application.
   * For the choice of which groups should be returned in the claim, best practice suggests selecting **Groups assigned to the application**.
   * For the choice of Source attribute, select **Cloud-only group display names**.

     <figure><img src="https://docs.soda.io/assets/images/group-claims.png" alt="" width="375"><figcaption></figcaption></figure>

     <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FjZqm1Vm7YDW5exDUrVvp%2Fimage.png?alt=media&amp;token=7344d7e6-30c7-4900-8bd4-3f15463ab549" alt=""><figcaption></figcaption></figure>
2. After saving the group claim, navigate to **Users and Groups** in the left menu, and follow Microsoft’s instructions to [Assign a user or group to an enterprise application](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/assign-user-or-group-access-portal?pivots=portal). Add any existing groups to the Soda SAML Application that you wish to make available in Soda Cloud to manage access and permissions.
3. In your message to Soda Support or your Soda Customer Engineer, advise Soda that you wish to enable user group syncing. Soda adds a setting to your SSO configuration to enable it.
4. When the SSO integration is complete, you and your team can select your IdP user groups from the dropdown list of choices available when assigning ownership or permissions to resources.

<br>

### **Set up user group sync in Okta**

1. In step 7 of the SAML application integration procedure [above](https://docs.soda.io/soda-cloud/sso.html#add-soda-cloud-to-okta), follow Okta’s instructions to [Define group attribute statements](https://help.okta.com/en-us/content/topics/apps/define-group-attribute-statements.htm).
   * For the Name value, use **Group.Authorization**.
   * Leave the optional Name Format value as **Unspecified**.
   * Use the Filter to find a group that you wish to make available in Soda Cloud to manage access and permissions. Exercise caution! A broad filter may include user groups you do not wish to include in the sync. Double-check that the groups you select are appropriate.

     <figure><img src="https://docs.soda.io/assets/images/group-attribute.png" alt="" width="375"><figcaption></figcaption></figure>

     <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FcC3Fc5dJ185yXP47miCr%2Fimage.png?alt=media&amp;token=a6f714b4-e326-49c0-a37d-9cff196f58fd" alt=""><figcaption></figcaption></figure>
2. Use the **Add Another** button to add as many groups as you wish to make available in Soda Cloud.
3. In your message to Soda Support or your Soda Customer Engineer, advise Soda that you wish to enable user group syncing. Soda adds a setting to your SSO configuration to enable it.
4. When the SSO integration is complete, you and your team can select your IdP user groups from the dropdown list of choices available when assigning ownership or permissions to resources.

### Renew SSO certificate

To renew an SSO certificate, **you need to provide Soda with the new X.509 certificate**, with which Soda will update your Soda organization's SSO configuration. Since Soda can only validate SSO against one certificate, there will be downtime between you deactivating the old certificate, and Soda updating the SSO configuration.

Depending on your organization's process of renewing the certificate, you could notify Soda (or arrange for a call) in advance of the specific datetime you want to renew, so Soda can be prepared for your update and minimize the mentioned downtime.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Global and Dataset Roles

Soda Cloud uses **Global Roles** and **Dataset Roles** to manage access and permissions. These roles ensure users and user groups have the right level of access based on their responsibilities.

## Global Roles

{% hint style="warning" %}
Only users with the **Manage organization settings** permission on the organization can define and assign global roles. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

Global roles define permissions across the entire organization in Soda Cloud.

By default, Soda Cloud provides to Global Roles: Admin and Users. You can create custom roles with a subset of the permissions

<table><thead><tr><th>Permission Group</th><th>Description</th><th width="100">Admin</th><th width="100">User</th></tr></thead><tbody><tr><td>Create data sources and manage runners<sup>1</sup></td><td><ul><li>Allow to deploy a new Soda Runner and manage runner configuration in Soda Cloud.</li><li>Allow the creation of new data source connections in Soda Cloud.</li><li>Does <strong>not</strong> grant access to existing data sources. Access to read or modify a data source is governed by Data Source Roles.</li></ul></td><td>✓</td><td></td></tr><tr><td>Create new datasets and data sources with Core/Library</td><td><ul><li>Allow the creation of new data sources in Soda Cloud when using the Core/Library.</li><li>Allow to onboard datasets in Soda Cloud on data sources connected with Soda Runner. See <a data-mention href="https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud">Via Soda Cloud</a></li><li>Onboarding a dataset is permitted when a user has <strong>either</strong> this global permission <strong>or</strong> the <strong>Create datasets</strong> permission on the target data source via a Data Source Role.</li></ul></td><td>✓</td><td>✓</td></tr><tr><td>Manage attributes</td><td><ul><li>Allow to define which datasets and check attributes are available to use in the organization.</li></ul></td><td>✓</td><td></td></tr><tr><td>Manage notification rules</td><td><ul><li>Allow to manage how notifications are sent.</li></ul></td><td>✓</td><td>✓</td></tr><tr><td>Manage organization settings</td><td><ul><li>Manage organization settings</li><li>Deactivate users</li><li>Create, edit, or delete user groups</li><li>Create, edit, or delete dataset roles</li><li>Create, edit, or delete global roles</li><li>Assign global roles to users or user groups</li><li>Add, edit, or delete integrations</li><li>Access and download the audit trail</li></ul></td><td>✓</td><td></td></tr><tr><td>Manage scan definitions</td><td><ul><li>Update scan definition</li><li>Run scan definition manually</li></ul></td><td>✓</td><td></td></tr></tbody></table>

<sup>1</sup> If your organization hasn't enabled [data source roles](#data-source-roles), the permission group "Create data sources and manage runners" is called **"Manage data sources and runners".**

* Description: Allow to deploy a new Soda Runner as well as configure data source connections in Soda Cloud.

### Create Custom Global Roles

You can create custom global roles to match your organization’s needs.

To create a global role:

{% stepper %}
{% step %}
Go to the **Global Roles** section in **Settings**.
{% endstep %}

{% step %}
Click Add Global Role to create a new role.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FEC3VHQ7eZtAaktpUia0n%2Fimage.png?alt=media&amp;token=39cc451e-93cb-4da3-afc6-bbd3886de8c7" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Enter a **name** for the role.
{% endstep %}

{% step %}
Select the **permissions** the role should have.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FNPmMShuTwwW04wCAFlTM%2Fglobal-role-create-1%20(1).png?alt=media&amp;token=d3db155e-d01a-4247-b533-73698c324461" alt="" width="519"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Save**.
{% endstep %}
{% endstepper %}

### Edit Custom Global Roles

You can edit global roles at any time to adjust permissions as your organization’s needs evolve.

To edit a global role:

{% stepper %}
{% step %}
Go to the **Global Roles** section in **Settings**.
{% endstep %}

{% step %}
Find the global role you want to modify.
{% endstep %}

{% step %}
Click the context menu next to the role and select Edit Global Role.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FDpF9WMIcE0sbktixQRRB%2Fimage.png?alt=media&amp;token=1f6843c2-0452-40ee-bf27-c7aabc541da2" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Adjust the role’s **name** and **permissions** as needed.
{% endstep %}

{% step %}
Click **Save** to apply your changes.
{% endstep %}
{% endstepper %}

### Assign Members to Global Roles

You can assign roles to individual users or user groups to grant them the associated permissions.

To assign a global role:

{% stepper %}
{% step %}
Go to the **Global Roles** section in **Settings**.
{% endstep %}

{% step %}
Find the global role you want to assign.
{% endstep %}

{% step %}
Click the context menu next to the role and select **Assign Members**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Ft32LO4SZ9o4w68WcCvmg%2Fimage.png?alt=media&amp;token=ec3169c5-8845-4b17-bd75-8c07233ca32c" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Select the users or user groups that should have the global roles.
{% endstep %}

{% step %}
Click **Save** to apply your changes.
{% endstep %}
{% endstepper %}

You can also assign roles on the Users and User groups tabs:

* For users: [User management](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/user-management)
* For user groups: [User management](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/user-management)

## Data source roles

{% hint style="info" %}
Data Source Roles are available as a feature preview. **Existing organizations must opt in**, so that current workflows are not interrupted.

Contact <support@soda.io> to enable this feature on Soda Cloud.
{% endhint %}

Data source roles define permissions at the level of an individual data source. They control who can see, configure, and manage a data source, and who can manage other users' access to it.

By default, Soda Cloud provides three data source roles: **Manager**, **Editor**, and **Viewer**. You can create custom roles with a subset of the permissions.

<table><thead><tr><th>Permission</th><th>Description</th><th width="116.6666259765625">Manager</th><th width="100">Editor</th><th width="100">Viewer</th></tr></thead><tbody><tr><td><strong>View data source</strong></td><td>See the data source and the datasets, checks, and results onboarded from it. Always enabled for any user who holds a role on the data source.</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td><strong>Create datasets</strong></td><td>Onboard new datasets on this data source.</td><td>✓</td><td>✓</td><td></td></tr><tr><td><strong>Manage data source settings</strong></td><td>Edit the connection, diagnostics warehouse, and onboarding rules for the data source.</td><td>✓</td><td>✓</td><td></td></tr><tr><td><strong>Delete data source</strong></td><td>Remove the data source.</td><td>✓</td><td>✓</td><td></td></tr><tr><td><strong>Manage responsibilities</strong></td><td>Assign and remove other users' permissions on this data source.</td><td>✓</td><td></td><td></td></tr></tbody></table>

{% hint style="warning" %}
**Data source roles and dataset roles are evaluated separately; their permissions are not merged.**

**A user must have a data source role** on a data source **to see** any of its **datasets**, **checks**, **or results.** Without a role on the data source, the data source and everything onboarded from it are **not visible** to the user, even if that user holds a dataset role on one of its datasets.
{% endhint %}

### Migration to data source roles

When data source roles are enabled for an organization, existing access is migrated automatically:

* Users with the **Create data sources and manage runners** permission (previously "Manage data sources and agents") are granted the **Manager** role on all existing data sources.
* If an **Everyone** user group exists, it is granted the **Viewer** role on all existing data sources.

{% hint style="info" %}
In large organizations, this migration may take several minutes to complete, because a permission is created for each data source.
{% endhint %}

## Dataset roles

Dataset roles define permissions for specific datasets.

{% hint style="warning" %}
Only users with the **Manage organization settings** permission on the organization can define and update dataset roles, as well as default responsibilities. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

By default, Soda Cloud provides to Dataset Roles: Manager, Editor, and User. You can create custom roles with a subset of the permissions

<table><thead><tr><th>Permission Group</th><th>Description</th><th width="100">Manager</th><th width="100">Editor</th><th width="100">Viewer</th></tr></thead><tbody><tr><td>View dataset</td><td>Access the dataset and view checks</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Access dataset profiling and samples</td><td>Allow users to see insights about the data</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Access failed row samples for checks</td><td>Allow users to see samples of rows that are considered invalid</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Configure dataset</td><td>Allow users to define dataset attributes and owner, change settings, and add/enable/configure metric monitors at a dataset level</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Manage dataset responsibilities</td><td>Allow users to grant and remove permissions through responsibilities.</td><td>✓</td><td></td><td></td></tr><tr><td>Manage Contracts</td><td>Allow users to modify as well as verifying the Data contract</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Propose checks</td><td>Allow users to propose changes in the Data Contract</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Manage incidents</td><td>Allow users to create, edit, and close incidents, and to link the dataset's checks to incidents or unlink them.</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Delete dataset</td><td>Allow users to remove a dataset and its checks.</td><td>✓</td><td></td><td></td></tr></tbody></table>

### Create Custom Dataset Roles

You can create custom dataset roles to match your organization’s needs.

To create a dataset role:

{% stepper %}
{% step %}
Go to the **Dataset Roles** section in **Settings**.
{% endstep %}

{% step %}
Click **Add Dataset Role** to create a new role.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FRQ5sZo9Cpady6JZ5UUaY%2Fimage.png?alt=media&amp;token=434d8319-9e3e-4b09-ad00-a6d2462b8a84" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Enter a **name** for the role.
{% endstep %}

{% step %}
Select the **permissions** the role should have.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F0AYj4tAWQEuceOoUupg4%2Fimage.png?alt=media&amp;token=10f85e19-b4ff-4d07-a28b-e07150ac9ef5" alt="" width="563"><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Save** to apply your changes.
{% endstep %}
{% endstepper %}

### Edit Dataset Roles

You can edit dataset roles at any time to adjust permissions as your organization’s needs evolve.

To edit a dataset role:

{% stepper %}
{% step %}
Go to the **Dataset Roles** section in **Settings**.
{% endstep %}

{% step %}
Find the dataset role you want to modify.
{% endstep %}

{% step %}
Click the context menu next to the role and select **Edit Dataset Role**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FDWbQLbCCRWPL1wUhr1tO%2Fimage.png?alt=media&amp;token=2f8ee2ad-2a48-4f80-884f-5bcc2850b304" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Adjust the role’s **name** and **permissions** as needed.
{% endstep %}

{% step %}
Click **Save** to apply your changes.
{% endstep %}
{% endstepper %}

### Assign dataset responsibilities

**Responsibilities** in Soda Cloud define who has access to a dataset and what they are allowed to do. They are assigned by mapping **users** or **user groups** to a **dataset role.**

This ensures that the right people have the appropriate permissions for each dataset, such as the ability to manage checks, propose new rules, or view profiling information.

For example:

* Assign a **Manager** role to a dataset owner who needs full control.
* Assign a **Viewer** role to a business user who only needs to monitor data quality results.

By assigning responsibilities, you ensure clear access control, accountability, and governance across your datasets.

Learn about how to set up responsibilities on a dataset: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)

### Define default responsibilities

#### For the dataset owner

Soda Cloud allows you to define **default responsibilities** for the dataset owner, which will automatically be granted for all dataset owners. This ensures that all users have a consistent baseline level of access unless you choose to customize it.

By default, all dataset owners have the "Manager" role.

**How to Configure Default Responsibilities**

{% stepper %}
{% step %}
Go to the **Organization Settings** page in Soda Cloud.
{% endstep %}

{% step %}
Locate the **Datasets Roles** section.
{% endstep %}

{% step %}
Select the **dataset role** to assign to the Dataset Owners

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FPsef38rL1fIxnm5kbdpp%2Fdataset-role-owner.png?alt=media&amp;token=ddea07f8-1566-4595-9bd0-c4fa5491dea7" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Save** on the top right of the page to apply changes

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Flhaw3WUs0BhlHUBMfhA1%2Fsave-default.png?alt=media&amp;token=b166408d-47c0-4731-91b5-95b28c850647" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

#### **For everyone**

Soda Cloud allows you to define **default responsibilities** for the **Everyone** group, which will automatically apply to all newly onboarded datasets. This ensures that all users have a consistent baseline level of access unless you choose to customize it.

By default:

* The **Everyone** group is assigned as a "Viewer" for all new datasets.
* This setting applies to **all users** in your organization unless disabled.

You can either customize the default role or **disable** the default responsibilities if you do not want the Everyone group to receive any automatic access to new datasets.

**How to Configure Default Responsibilities**

{% stepper %}
{% step %}
Go to the **Organization Settings** page in Soda Cloud.
{% endstep %}

{% step %}
Locate the **Datasets Roles** section.
{% endstep %}

{% step %}
Select the **dataset role** to assign to the Everyone group for new datasets.
{% endstep %}

{% step %}
To disable default responsibilities, toggle the feature **off**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F37H9sY4cjRAUlnRUMUIc%2Fdataset-role-everyone.png?alt=media&amp;token=cd241d00-454d-4bc1-b765-7b6873a73552" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Click **Save** on the top right of the page to apply changes

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Flhaw3WUs0BhlHUBMfhA1%2Fsave-default.png?alt=media&amp;token=b166408d-47c0-4731-91b5-95b28c850647" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

***

## Role-Based Access Control (RBAC)

Role-Based Access Control (RBAC) in Soda Cloud ensures that users can only access and interact with data according to their assigned roles; roles and permissions are fully customizable to adapt to your organization's needs.

RBAC is designed to:

* Enforce least-privilege access
* Prevent direct user-to-permission grants
* Scope access by organization and role
* Ensure that all access is authenticated and authorized

Here’s how these features can benefit your organization on the journey to governed data democratization:

#### Customizable roles and permissions

With Soda, **you can tailor access to data** to align with your specific needs. For instance, you can create a role for Product Marketing that allows the team to view sample data and propose data quality checks on particular datasets, while restricting editing capabilities on others.

If a **default role** is not quite what you're looking for, **you can easily edit its permissions** to add new capabilities.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FukWyDm6vwtIGYuIRav2S%2Froles.gif?alt=media&amp;token=55d59923-d67e-4154-acfd-c822155ffcdc" alt="" width="375"><figcaption></figcaption></figure>

#### Streamlined user management

You can **enable user group synchronization from your Identity Provider (IdP) to Soda Cloud**, reducing the administrative burden of ensuring consistent permissions. This saves time during onboarding and offboarding while minimizing human error.

#### Bulk editing for enhanced efficiency

You can assign roles and permissions to multiple datasets in one go through Soda Cloud UI or via [API](https://docs.soda.io/apis/rest-api).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FKbwC3DYqaSEKzYy9f06c%2Fbulk-edit.gif?alt=media&amp;token=68a49c27-4a09-421d-ae10-43174e34d65a" alt="" width="563"><figcaption></figcaption></figure>

### Architecture

#### Strong identity as the security perimeter

Access to critical systems is federated via an Identity Provider (IdP) with:

* Multi-factor authentication (MFA) enforcement
* Role-based access control (RBAC)
* Unique user identification
* Least-privilege, role-scoped permissions

Roles are:

* Managed centrally in the IdP
* Mapped to system permissions

User group synchronization from the IdP to Soda Cloud is supported to streamline onboarding and offboarding while minimizing human error.

#### Authentication and enforcement

Every request to Soda Cloud:

1. Is authenticated
2. Is scoped to the user’s organization
3. Passes through RBAC enforcement

{% hint style="info" %}
Soda Cloud does **not** provide a capability to publish content or files that can be accessed by users who are not authenticated members of the organization. Public link sharing and anonymous access capabilities are not allowed to **prevent exposure of Soda Cloud content outside the authenticated organization**.
{% endhint %}

### **How access control helps data democratization**

Access control empowers data owners to efficiently manage data requests while ensuring that data is accessible.

* **Accelerates value creation** by enabling quick access to a wide range of dataset
* **Improves decision making** by ensuring users can easily identify and use the most relevant data

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Service accounts

Learn how to create and manage service accounts in Soda Cloud to authenticate automated pipelines and API integrations without tying credentials to individual users.

{% hint style="warning" %}
This feature requires the **Manage organization settings** permission. Learn more about permissions here: [Dataset attributes & responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities)
{% endhint %}

Service accounts are organization-bound identities designed for automated pipelines and API integrations. Unlike regular users, they authenticate exclusively via API key. They have no email/password login and are not tied to any individual's SSO credentials.

Use service accounts when you want API keys that are independent of any individual user, for example in CI/CD pipelines, scheduled scans, or data engineering workflows.

| A service account can...                                                                          | A service account cannot                                                                |
| ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| <i class="fa-check">:check:</i> Run `soda-core` scans (results attributed to the service account) | <i class="fa-x">:x:</i> Log in to the Soda Cloud UI                                     |
| <i class="fa-check">:check:</i> Call the Soda Cloud REST API                                      | <i class="fa-x">:x:</i> Participate in agreements, discussions, or incidents via the UI |
| <i class="fa-check">:check:</i> Be assigned to datasets and user groups                           |                                                                                         |
| <i class="fa-check">:check:</i> Receive notifications                                             |                                                                                         |
| <i class="fa-check">:check:</i> Create and verify data contracts from a pipeline                  |                                                                                         |
| <i class="fa-check">:check:</i> Log incident activity via API                                     |                                                                                         |

## Create a service account

{% stepper %}
{% step %}

#### Navigate to Service Accounts

Click on your avatar > **Organization Settings** > **Service Accounts** tab

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FJnglh3wyc9sUw3s6Yhq8%2Fimage.png?alt=media&amp;token=286fc454-ad89-4b5b-a206-f1302bc2407a" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Create a new service account

Click on <i class="fa-plus">:plus:</i> (top right) to create a new service account.

Enter a name and a unique email address for the service account, then confirm.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FYSucg7VtYvnpzko9lzv8%2Fimage.png?alt=media&amp;token=e51393bc-dcda-47e3-b1e8-e0b9112b5075" alt="" width="383"><figcaption></figcaption></figure>

{% hint style="info" %}
A unique email is required but it does not represent a real login.
{% endhint %}
{% endstep %}

{% step %}

#### Store the API key

Copy the API key that is displayed. **This is the only time the key is shown**. It cannot be retrieved after you navigate away.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FhwIqPLQmIr7t9pSu7TjL%2Fimage.png?alt=media&amp;token=91328285-542b-476e-824d-1e173917c2cc" alt="" width="563"><figcaption></figcaption></figure>

Use the API key ID and secret as credentials wherever you would normally configure Soda API keys, for example in `soda-core` scan configurations or REST API calls.
{% endstep %}
{% endstepper %}

### Deactivate a service account

In the **Service Accounts** tab, open the context menu for the account and select **Deactivate**. The API key is immediately invalidated.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FSLzZg2MEM72H9gytvsaX%2Fimage.png?alt=media&amp;token=b08514f1-f0a8-4d06-8dd4-eac21eaf15c1" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
**Note:** Service accounts cannot be deleted by design. They are only deactivated until activated again.
{% endhint %}

***

## Default permissions

New service accounts are automatically assigned a default role configured under **Organization Settings > Global Roles > Responsibilities**. This default role excludes UI-bound permissions that don't apply to non-human accounts.

You can also:

* Add a service account to a **user group** to inherit that group's dataset permissions.
* Assign a service account a direct **dataset role** via the dataset's Edit Responsibilities panel.

When both a group-inherited and a direct dataset role exist, the higher permission takes precedence.

{% hint style="info" %}
Bound runners appear on the Service Accounts page once connected.
{% endhint %}

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# User provisioning with SCIM

Automatically provision and deprovision Soda Cloud users and user groups from Microsoft Entra ID with SCIM

**Soda Cloud supports user provisioning through SCIM** (the System for Cross-domain Identity Management). With SCIM enabled, **your identity provider manages Soda Cloud users for you**: when someone joins your organization, Entra ID creates their Soda Cloud account, and when they leave, their access is revoked in the same sync. Your IT team administers Soda Cloud access from the identity provider they already use, with no manual user management in Soda Cloud.

{% hint style="info" %}
**SCIM provisioning is an opt-in feature** available to organizations with a Soda Enterprise plan. To enable it, contact Soda Support at <support@soda.io>.
{% endhint %}

### Prerequisites

* A Soda Cloud organization (Enterprise license) with SCIM provisioning enabled.
* The **Manage organization settings** permission. See [Global and dataset roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles).
* Administrator access to your tenant in Microsoft Entra ID.

## Generate a SCIM token in Soda Cloud

Entra ID authenticates against Soda Cloud with a bearer token that you generate in your organization settings.

{% stepper %}
{% step %}

#### Open the SCIM tab

In Soda Cloud, navigate to **your avatar**→ **Organization Settings**, then open the **SCIM** tab.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FYcDvOuegVtp4SsCmEiUt%2Fimage.png?alt=media&amp;token=58f9dcf5-9601-4d36-8d57-efeffbb60b56" alt="" width="563"><figcaption><p>Navigate to the SCIM tab on your org settings to see all available actions and configurations.</p></figcaption></figure>
{% endstep %}

{% step %}

#### Generate the token

Click **Generate token**. Soda Cloud generates a new token and displays it once.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FnqOHLSSRHPdQk888wEdP%2Fimage.png?alt=media&amp;token=6a40e238-74f2-470a-a25b-08225929480a" alt="" width="563"><figcaption><p>After clicking on "Generate token", Soda will display a randomly generated token.</p></figcaption></figure>
{% endstep %}

{% step %}

#### Copy and store the token

Copy the token and store it securely, then click **Done**.

{% hint style="warning" %}
After you click **Done**, Soda Cloud never displays the token again. If you lose the value, you can [regenerate the token](#regenerate-token) (see below).
{% endhint %}
{% endstep %}
{% endstepper %}

#### Regenerate token

Soda Cloud never displays a generated token after leaving the bearer token screen. If you lose the value, click **Regenerate token**; this deletes the previous token and replaces it with a new one, so any existing SCIM connection that uses the old token stops working until you update it.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fq7ZgXia2YtIC3RTYb91r%2Fimage.png?alt=media&amp;token=dbdd9a6e-4b67-439d-ac8e-11ecf8be6138" alt="" width="563"><figcaption><p>Click on "Regenerate token" to delete previous token and replace it with a new one.</p></figcaption></figure>

## Configure SCIM in Entra ID

{% hint style="info" %}
This step must be performed by an Azure Entra ID administrator in your organization.
{% endhint %}

{% stepper %}
{% step %}

#### Open provisioning for the application

In the Microsoft Entra admin center, navigate to **Enterprise applications**, then select the application that represents your Soda Cloud organization.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FIAB3cSUIf0WWoMRSb3QD%2Fimage.png?alt=media&amp;token=f0ecb2a0-c807-49f3-9619-ce2f42377753" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Start the connection flow

Select **Provisioning**, then start the flow by selecting **Connect your application**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FdCJwR5ThLnzcHgtG9wbN%2Fimage.png?alt=media&amp;token=7612de78-164c-4c10-9e6b-ea924bdb2030" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Configure the provisioning connection

Configure the provisioning connection as follows:

| **Authentication method** | Bearer authentication                                                                                                                    |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **Tenant URL**            | <ul><li>Soda EU: <code><https://cloud.soda.io/scim/v2></code></li><li>Soda US: <code><https://cloud.us.soda.io/scim/v2></code></li></ul> |
| **Secret token**          | Paste the SCIM token you generated in Soda Cloud.                                                                                        |

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fw9NIQG4LZ4oyNIwsdApz%2Fimage.png?alt=media&amp;token=5445bd45-61f8-4a8a-adb1-ff270c870509" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Test the connection

Click **Test connection** and verify that the connection test succeeds.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FEmsxu5cABlqTJkjh6LPq%2Fimage.png?alt=media&amp;token=fa985a49-5ae4-4707-ad11-6cf85e8a50fb" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

## Verify that provisioning works

{% hint style="info" %}
This step must be performed by an Azure Entra ID administrator in your organization, and a Soda Cloud user with **Manage organization settings** permission.
{% endhint %}

### Verification using provision on demand

The quickest way to confirm the integration is Entra ID's **provision on demand**, which syncs a single user **immediately** instead of waiting for a scheduled provisioning cycle.

{% stepper %}
{% step %}

#### Provision a test user

Provision a test user on demand on Entra ID.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F7NMAQ8JBvTSxHlKutZFW%2Fimage.png?alt=media&amp;token=024001e4-7f0f-454f-9cbf-b03c0c20d017" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Confirm user

Confirm in Soda Cloud that the user appears in **Organization Settings** under the **Users** tab.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FJ9aENzcCxFPrKLG7dW00%2Fimage.png?alt=media&amp;token=5ae87d8c-56f3-4a70-bb21-3a2e9251b125" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

### Verification using the full provisioning

Once regular provisioning runs, you can observe synced users and their actions, such as "Create" or "Disable", in the **Entra ID provisioning logs**.

{% hint style="info" %}
**Entra ID runs provisioning on its own schedule**, so changes to users and user groups in your identity provider can take some time to appear in Soda Cloud; refer to [Microsoft's documentation](https://learn.microsoft.com/en-us/entra/identity/app-provisioning/application-provisioning-when-will-provisioning-finish-specific-user#how-long-will-it-take-to-provision-users) for details on the provisioning cycle.
{% endhint %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FHbUHjo7i3lwTsiIBRkUw%2Fimage.png?alt=media&amp;token=97edadbb-9e0a-45f3-97a3-39860b27952b" alt=""><figcaption><p>The Entra ID provisioning logs will expose the actions taken for each user</p></figcaption></figure>

In Soda Cloud, the **Users** and **User Groups** tabs in **Organization Settings** display a **Managed by SCIM** column that identifies which users and user groups are provisioned through your identity provider.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FUfav12p2buyzymc47U1d%2Fimage.png?alt=media&amp;token=acdae7f9-2bba-4053-bb62-2bb69c9e5057" alt=""><figcaption><p>The column <strong>managed by SCIM</strong> appears when SCIM provisioning is enabled and working.</p></figcaption></figure>

> For single sign-on setup, refer to [User and user group management with SSO](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/user-and-user-group-management-with-sso).

> For details on the roles and permissions of provisioned users, refer to [Global and dataset roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles).

***

## Limitations

Soda has tested and documented SCIM provisioning with **Microsoft Entra ID**. It is currently the **only supported identity provider** for SCIM.


# Migrate from v3 to v4

This guide details the steps required to migrate datasets from **v3** to **v4 contracts.**

The migration needs to be enabled by the Soda team. Contact us to ensure that the feature flag (`v3ToV4DatasetMigrationEnabled`) is enabled for your organization.

{% hint style="warning" %}
This action requires the global **Admin** role and can be [enabled for users with **Manage contract** permission](#enable-dataset-owners-to-migrate-datasets-from-v3-v4). Learn more about permissions here: [Global and Dataset Roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles)
{% endhint %}

## Using Soda Cloud

### Start the migration

The migration can be performed in two ways:

1. **Bulk migration:** On the datasets pag&#x65;**,** you can select multiple datasets to migrate them all at once.
2. **Single migration:** On a dataset page, you can migrate the dataset individually.

<div><figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FgLX9SbDVwLZNqcNjaZU7%2F0-bulk.png?alt=media&amp;token=f4eb37aa-2c3b-467c-a8d6-4610b035dd2c" alt=""><figcaption></figcaption></figure> <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F7rv6nTzb5jD86FcD9Din%2F0-single.png?alt=media&amp;token=3c384104-6869-405c-9c31-413bd6513a88" alt=""><figcaption></figcaption></figure></div>

Once initiated, Soda guides you through each step of the process to ensure a safe and transparent migration.

### Select a v4 data source

In the first step, you’ll select which **v4 data source** the dataset(s) should be migrated to.

1. Choose an existing **v4 data source** from the dropdown list.
2. Or click **Add a new v4 Data Source** to create a new one.
3. Click **Next**

{% hint style="warning" %}
You can only migrate v3 datasets to a **data source of the same type**. For example, from Snowflake to Snowflake.

The migration process in Soda Cloud requires a **valid connection** configuration to your data source **through Soda Runner**.
{% endhint %}

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FAOyhU4WNWTyvC5p7VNE0%2F1-select-data-source.png?alt=media&amp;token=0bba810f-fca1-41ca-a7ae-4500da20d0fc" alt=""><figcaption></figcaption></figure>

### Configure migration settings

In this step, you can customize optional settings before running the migration.

**Available configurations:**

* **Contract Schedule:** Set a schedule for automatic contract verification. [Verify a contract](https://docs.soda.io/soda-documentation/find/data-testing/cloud-managed-data-contracts/verify-a-contract)
* **Migrate History:** include 90 days of historical check results in the migration.
* **Migrate Responsibilities:** Migrate the dataset's ownership and assigned responsibilities.
* **Migrate Attributes:** Migrate the dataset's attributes.

Once your settings are configured, click **Continue** to proceed.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FUjCACKk4ZYwomo3noARG%2F2-configure-migration.png?alt=media&amp;token=3dd16665-b7a5-4e06-a90f-ce8ad0a2740d" alt=""><figcaption></figcaption></figure>

### Review migration

Before migrating, you’ll see a detailed summary of what will be migrated and what won’t.\
Each dataset displays:

* **Number of checks** that were:
  * **Successfully translated:** The behavior of the new checks matches that in v3.
  * **Translated with a warning:** Translated, but not an exact one-to-one match. Review recommended.
  * **Not translated:** Checks are either not supported in v4 or cannot be translated. See [#current-limitations](#current-limitations "mention")
* For each check with a warning or that was not translated, you can view the **definition of the v3 check,** along with an **indicative remark** explaining the reason or impact.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FYahckD5TKgp4JmytP3aT%2F3.0-review-migration.png?alt=media&amp;token=a7ea4002-7f84-4996-8e19-aa09210b5568" alt=""><figcaption></figcaption></figure>

Click the **eye icon** to open a preview of the generated contract before finalizing the migration.It lets you verify the contract structure, filters, variables, and checks. You can test it directly from this view to ensure everything runs as expected before confirming migration.

<div><figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FtSfZpZ3abtmTScPNmYyJ%2F3.1-view-contract.png?alt=media&amp;token=74390993-0310-4ded-8943-9fc023ce69ba" alt=""><figcaption></figcaption></figure> <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FBYjLLltdUZ2w0zv6YexM%2F3.2-review-contract-migration.png?alt=media&amp;token=6375e02a-3cad-4d61-bcb9-ce7c5e2a098a" alt=""><figcaption></figcaption></figure></div>

### Complete migration

After reviewing, use the checkboxes to deselect any datasets that are not ready for migration, then click **Migrate** to finalize the process.

This action starts a **background process** to migrate the selected datasets. The migration may take a few seconds or minutes, depending on the volume being migrated, to complete.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FkYJTZk2NugDFuyCoGcTL%2FScreenshot%202025-11-07%20at%2011.11.02%E2%80%AFAM.png?alt=media&amp;token=beacefb4-7c07-4d5a-9bc5-89b94e085cc7" alt=""><figcaption></figcaption></figure>

Once started, you can navigate to the **v3 datasets** to monitor progress.\
Refresh the page after a few moments to view the migration results.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FNBS7lksfyTXHY2GBTA6Q%2FScreenshot%202025-11-07%20at%2011.19.33%E2%80%AFAM.png?alt=media&amp;token=8eb145f0-ceca-4ecd-b3c0-1113b0273169" alt=""><figcaption></figcaption></figure>

### Post-migration results

Once migration is complete:

* Your original **v3 dataset** remains accessible but marked as *migrated*.
* Your new **v4 dataset** is fully active and ready for contract-based validation.

The v3 dataset page shows its migration status as **Completed migration**.

In the v4 dataset view, you can now see:

* Migrated checks under **Dataset checks** and **Column checks**.
* Contract verification results and coverage metrics.
* The option to edit or version your new data contract.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FQRnI1aH0gbK5acsW8nt1%2F5.result-v4.png?alt=media&amp;token=f645f850-e71f-4182-956a-3bc11ef05adc" alt=""><figcaption></figcaption></figure>

## Using Soda CLI

### Install the migration CLI

In a virtual environment, install the Soda migration package as well as the Soda Core package for your data source (see [Data source reference for Soda Core](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core)).

{% hint style="warning" %}
The migration process requires having installed **the `soda-migration` package using the** [**private PyPI**](#private-pypi-installation-flow) with a **Team** or **Enterprise license.**

Need access to the PyPI repository ? Please [contact us](https://www.soda.io/contact).
{% endhint %}

Choose your organization host to install the migration package:

{% tabs %}
{% tab title="Team EU" %}
{% code overflow="wrap" %}

```bash
pip install soda-migration -i "https://<api_key_id>:<api_key_secret>@team.pypi.cloud.soda.io"
```

{% endcode %}
{% endtab %}

{% tab title="Team US" %}
{% code overflow="wrap" %}

```bash
pip install soda-migration -i "https://<api_key_id>:<api_key_secret>@team.pypi.cloud.us.soda.io"
```

{% endcode %}
{% endtab %}

{% tab title="Enterprise EU" %}
{% code overflow="wrap" %}

```bash
pip install soda-migration -i "https://<api_key_id>:<api_key_secret>@enterprise.pypi.cloud.soda.io"
```

{% endcode %}
{% endtab %}

{% tab title="Enterprise US" %}
{% code overflow="wrap" %}

```bash
pip install soda-migration -i "https://<api_key_id>:<api_key_secret>@enterprise.pypi.cloud.us.soda.io"
```

{% endcode %}
{% endtab %}
{% endtabs %}

Next to the migration package, **it is also required to install the necessary package to connect to your data source**. See [Data source reference for Soda Core](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core). This is required because Soda connects to your data source to generate the contract skeleton before translating existing checks.

### Configuration

Create a `migration.json` file to map v3 dataset IDs to v4 DQNs (Dataset Qualified Names). This file defines which datasets to migrate and their corresponding new DQNs.\
\
Example structure:

```json
[
    {
        "dqn": "v4_data_source/postgres/public/dim_employee",
        "v3DatasetId": "6d706754-4d0d-4aa0-9c7d-25fbe07cb3f6"
    },
    {
        "dqn": "v4_data_source/postgres/public/dim_employee",
        "v3DatasetId": "6d706754-4d0d-4aa0-9c7d-25fbe07cb3f6"
    }
]
```

> Learn more about DQNs in [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference#dataset-fully-qualified-name)

#### Retrieve the v3 dataset IDs

To retrieve the v3 dataset IDs, you want to migrate from Soda Cloud:

* Use the [REST API](https://docs.soda.io/apis/rest-api) to fetch the datasets' information and their IDs
* The IDs can also be found in the dataset URL for a given dataset.

<details>

<summary>Example Python script to generate the configuration file</summary>

```python
#!/usr/bin/env python3
"""
Soda Cloud Dataset Migration Configuration Generator

This script fetches dataset information from Soda Cloud API and generates
a migration.json file for dataset migration. This file is used to migrate
datasets from V3 to V4.
"""

import argparse
import base64
import json
import os
import subprocess
import sys
from dotenv import load_dotenv


def generate_basic_auth_token(api_key, api_secret):
    """Generate Basic Auth token from API key and secret."""
    token = f"{api_key}:{api_secret}"
    return base64.b64encode(token.encode()).decode()


def execute_curl_with_auth(url, auth_token):
    """Execute curl command with Basic Auth and return JSON response."""
    try:
        headers = f"Authorization: Basic {auth_token}"
        command = ['curl', '-s', url, '--header', headers]
        result = subprocess.run(command, capture_output=True, text=True, check=True)
        return json.loads(result.stdout)
    except subprocess.CalledProcessError as e:
        print("Curl failed with error:\n", e.stderr)
        return None
    except json.JSONDecodeError as e:
        print(f"Failed to parse JSON response: {e}")
        return None


def fetch_datasets(api_key, api_secret, v3_data_source_name=None):
    """Fetch datasets from Soda Cloud API."""
    url = "https://cloud.soda.io/api/v1/datasets"
    params = []
    if v3_data_source_name:
        params.append(f"datasourceName={v3_data_source_name}")
    params.append("size=1000")
    if params:
        url += "?" + "&".join(params)
    
    auth_token = generate_basic_auth_token(api_key, api_secret)
    
    print(f"Fetching datasets from {url}...")
    response = execute_curl_with_auth(url, auth_token)
    
    if response is None:
        print("Failed to fetch datasets from API")
        return None
    
    if "content" not in response:
        print("Unexpected API response format")
        return None
    
    return response["content"]


def generate_migration_data(datasets, v4_data_source_name):
    """Generate migration data from datasets."""
    migration_data = []
    
    for dataset in datasets:
        migration_entry = {
            "v3DatasetId": dataset["id"],
            "dqn": v4_data_source_name + "/" + dataset["datasource"]["prefix"].replace(".", "/") + "/" + dataset["name"]
        }
        migration_data.append(migration_entry)
    
    return migration_data


def save_migration_file(migration_data, output_path):
    """Save migration data to JSON file."""
    try:
        with open(output_path, 'w') as f:
            json.dump(migration_data, f, indent=2)
        print(f"Migration file saved to: {output_path}")
        return True
    except Exception as e:
        print(f"Failed to save migration file: {e}")
        return False


def main():
    """Main application entry point."""
    parser = argparse.ArgumentParser(
        description="Generate Soda Cloud dataset migration file"
    )
    parser.add_argument(
        "v4_data_source_name",
        help="V4 data source name for migration"
    )
    parser.add_argument(
        "v3_data_source_name",
        help="V3 data source name to filter datasets (used as query parameter)"
    )
    parser.add_argument(
        "--output",
        "-o",
        default="migration.json",
        help="Output file path (default: migration.json)"
    )
    parser.add_argument(
        "--env-file",
        default="../.env",
        help="Environment file path (default: ../.env)"
    )
    
    args = parser.parse_args()
    
    # Load environment variables
    load_dotenv(args.env_file)
    
    # Get API credentials from environment
    api_key = os.getenv("SODA_CLOUD_API_KEY")
    api_secret = os.getenv("SODA_CLOUD_API_SECRET")
    
    if not api_key or not api_secret:
        print("Error: SODA_CLOUD_API_KEY and SODA_CLOUD_API_SECRET must be set in environment")
        print("You can set them in a .env file or as environment variables")
        sys.exit(1)
    
    print(f"Generating migration for data source: {args.v4_data_source_name}")
    print(f"Filtering datasets by V3 data source: {args.v3_data_source_name}")
    
    # Fetch datasets from API
    datasets = fetch_datasets(api_key, api_secret, args.v3_data_source_name)
    if datasets is None:
        sys.exit(1)
    
    print(f"Found {len(datasets)} datasets")
    
    # Generate migration data
    migration_data = generate_migration_data(datasets, args.v4_data_source_name)
    
    # Save to file
    if save_migration_file(migration_data, args.output):
        print("Migration generation completed successfully!")
    else:
        sys.exit(1)


if __name__ == "__main__":
    main()

```

</details>

***

### Generate contracts

Run the following command, replacing the paths and file names with your setup. Note that this action requires a valid connection configuration to your data source and Soda Cloud.

```sh
soda contract-migrator generate-bulk \
--bulk-config-file migration.json \
--output-directory contracts/ \
--v4-data-source ds.yml \
--soda-cloud soda-cloud.yml \
--schedule "0 0 * * *" \
-v
```

#### Parameters

| **Parameter**        | **Required** | **Description**                                                                                                                                                                                                                                                                                          |
| -------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--bulk-config-file` | Yes          | Path to the JSON configuration file (`bulk.json`) that maps v3 dataset IDs to v4 DQNs.                                                                                                                                                                                                                   |
| `--output-directory` | Yes          | Directory where the generated v4 contracts will be saved. The structure within this location mirrors the DQN hierarchy.                                                                                                                                                                                  |
| `--v4-data-source`   | Yes          | <p>Path to the local v4 datasource YAML file (e.g., <code>ds.yml</code>). Used to retrieve schema metadata during contract generation.<br><br>See <a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core">/pages/Ovp9NbJpqBhQz07AImSa</a></p> |
| `--soda-cloud`       | Yes          | Configuration required to connect to Soda Cloud.                                                                                                                                                                                                                                                         |
| `--schedule`         | No           | Optional cron expression defining the schedule for the generated contracts (e.g., `"0 0 * * *"`).                                                                                                                                                                                                        |
| `--verbose`          | No           | Enables verbose output for detailed logs during the migration process.                                                                                                                                                                                                                                   |

### Review migration

Ensure that each generated contract includes:

* Correct `v4_dqn` in the `dataset` property
* Correct `v3_dataset_id` property referencing v3 dataset
* Correct **columns and types**
* Checks migrated from `sodacl` into `contracts` checks
* v3 check IDs present in the `qualifier` field for each check
* Accurate **check filters**, expressions, and metadata

See [#current-limitations](#current-limitations "mention") to know which checks cannot be automatically migrated yet. Those checks can still be added manually, and the history can be migrated by setting a v3 check ID in the `qualifier`

### Complete migration

Once contracts are verified, publish them to Soda Cloud with the following command:

```sh
soda contract-migrator publish \
--contract contracts/ \
--soda-cloud soda-cloud.yml \
--migrate-attributes \
--migrate-responsibilities \
--migrate-test-results 
```

#### Parameters

| Parameter                    | Required |                                                                                                                                                                                                                                                                                                                                                                        |
| ---------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--contract`                 | Yes      | <p>Specifies the path to the folder containing the contracts or a specific contract file to be migrated.<br><br>This parameter supports <strong>recursive directory traversal</strong>.</p><p><br>⚠️ <strong>Note:</strong> If the folder includes contracts for datasets that have already been migrated, those datasets will be <strong>migrated again</strong>.</p> |
| --`soda-cloud`               | Yes      | Configuration required to connect to Soda Cloud.                                                                                                                                                                                                                                                                                                                       |
| --`migrate-test-results`     | No       | <p>Include test history from v3 to v4. A maximum of 90 days of history is migrated.<br><br>Default is <code>false</code>.</p>                                                                                                                                                                                                                                          |
| --`migrate-responsibilities` | No       | <p>Copy responsibilities from v3 datasets to v4 datasets.<br><br>Default is <code>false</code>.</p>                                                                                                                                                                                                                                                                    |
| --`migrate-attributes`       | No       | <p>Copy attributes from v3 datasets to v4 datasets<br><br>Default is <code>false</code>.</p>                                                                                                                                                                                                                                                                           |
| --`verbose`                  | No       | Enable verbose mode                                                                                                                                                                                                                                                                                                                                                    |

### Post-migration results

After publishing, confirm the following:

* **Logs:** Check migration status (overall and per dataset).
* **UI Overview:** Verify datasets in the Soda Cloud UI.
* **Checks:** Ensure checks appear with full history.
* **Flags:** Confirm presence of the following indicators:
  * **v4 flag** present
  * **v3 link** available
  * **Migration completed** flag present

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F4HxS2efWThnpNjtzXtzI%2FScreenshot%202025-10-21%20at%2010.54.22%E2%80%AFAM.png?alt=media&amp;token=52ada23c-851c-44ec-93db-db602b7b53b6" alt=""><figcaption></figcaption></figure>

## Notes and recommendations

* Migration does not delete v3 datasets; it simply marks them as migrated. Once migration is completed, you will be required to update your pipelines or agreements to stop executing v3 checks. Then you can remove the v3 datasets or the v3 data source to remove all its datasets.
* Verify that your v4 data source has a valid connection configuration before migrating. Soda connects to your data source to generate the contract structure.
* Review the migrated contract in detail before finishing the migration. However, it is possible to re-run the migration if necessary [#re-running-a-migration](#re-running-a-migration "mention")

## Enable dataset owners to migrate datasets from v3→v4

An organization **Admin** can enable users with **Manage Contract** permission to migrate datasets.

{% hint style="info" %}
**Dataset owners** have the **Manage Contract** permission by default.
{% endhint %}

{% stepper %}
{% step %}
Click on your profile and navigate to **Organization Settings**

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FsKkpLmS50N59SYXfPKZa%2Fimage.png?alt=media&amp;token=aefe8606-6025-4ddd-a52f-92c8c9086380" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}
Under **Dataset migration**, check the option "Allow users with Manage Contract permission to migrate datasets"

This option is disabled by default.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FSeZi1uhCjvWFn7n7MjCx%2Fimage.png?alt=media&amp;token=522712c7-5982-44d2-9f74-1c525012fe52" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

Users with permission will now be able to see the migration tool:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F5uk9JvSKa7o1rdj4TazB%2Fimage.png?alt=media&amp;token=c735a90a-4fe0-4cff-863f-b09e5cf75c20" alt=""><figcaption></figcaption></figure>

***

## View and filter migrated datasets

After migration, you can review and manage your datasets from the **Datasets** page in Soda Cloud.

Use the filters to easily identify datasets based on their **migration status** and **version**:

* **Migration status filter** — view datasets that are *Pending*, *In progress*, or *Completed migration*.
* **Version filter** — filter by dataset version (**v3** or **v4**) to focus on datasets still awaiting migration or already upgraded.

This makes it simple to track migration progress, validate completed transitions, and identify datasets that still require attention.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F5GbGfDPnQBSLOoAYWbwS%2FScreenshot%202025-11-07%20at%2011.22.04%E2%80%AFAM.png?alt=media&amp;token=3e3e6aff-b1a3-4a75-8581-fac3e927e317" alt=""><figcaption></figcaption></figure>

## Re-running a migration

Once a dataset has been successfully migrated to version 4 (v4), Soda blocks the migration for the dataset to happen again. To re-run the migration, you will need to delete the v4 dataset and run the migration again.

## How thresholds are translated

A check with both a `fail` and a `warn` alert condition translates into a v4 threshold that carries the fail condition, with the warn condition as an `additional` threshold marked `level: warn`:

```yaml
threshold:
  must_be_less_than: 10
  additional:
    must_be_less_than: 5
    level: warn
```

A check with only a `warn` condition becomes a single threshold with `level: warn`. See [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference#warn-and-fail-thresholds).

When the fail and warn conditions disagree on a shared threshold property — a different `metric`, or units that cannot be converted to one (for example, a unit set by a variable) — they cannot share one v4 threshold. Only the fail condition is migrated, and the migration report carries a remark for the dropped warn condition; re-add it manually.

A v3 alert level that carries multiple `when` conditions is not translated by the migrator. See [#current-limitations](#current-limitations "mention").

## How checks are matched

The migration process uses the **qualifier** ( [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference#check-qualifiers)) field to identify which v3 checks should be migrated. The qualifier values are set to the v3 check IDs.\
\
Because the qualifier is part of the check's identity algorithm, it is important **not to change the qualifier** after migration.\
\
Changing it would result in a **loss of history for the checks** in Soda Cloud.

## Current limitations

### Translation step

#### **Check types**

The following check types are not yet supported:

* [Group By check](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference#group-by-check)
* Any [reconciliation](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference/reconciliation-checks) checks
* Reference check
* Any checks using [anomaly score](https://docs.soda.io/soda-documentation/soda-v3/sodacl-reference/anomaly-score) or [anomaly detection](https://docs.soda.io/soda-documentation/soda-v3/sodacl-reference/anomaly-detection)

#### **Alert conditions**

A v3 alert level that carries more than one `when` condition is not translated by the migrator. Most such checks are dropped from the generated contract with a remark in the migration report. When the conditions are written as a YAML list, the check migrates with no threshold at all and no remark: it then fails verification for check types that require a threshold (`freshness`, `aggregate`, `metric`), and silently falls back to the check type's default threshold for the rest. If the check also carries a single-condition level, that level is translated and the multi-condition level is dropped silently.

Re-author these checks in the contract. A two-sided condition is usually a single `must_be_between` or `must_be_not_between` range.

#### **Dataset filters**

Dataset filters ([/spaces/oV0A6Eua8LUIyWgHxsjf/pages/yBqe6yM8bkyesdvcy0Pd#in-check-vs.-dataset-filters](https://docs.soda.io/spaces/oV0A6Eua8LUIyWgHxsjf/pages/yBqe6yM8bkyesdvcy0Pd#in-check-vs.-dataset-filters "mention") ) are currently not migrated.

#### Column casting

Data Contract does not support casting yet.\
When casting is detected in a check, the check will not be translated and will be excluded from the migration.

#### **Variables**

**Variables in names**

If variables are used in the column name, the check will not be translated and will be excluded from the migration.

Exampl&#x65;**:**

```yaml
checks for ECOMMERCE_ORDERS [test]:
  - missing_count(${ORDER_ID_COL}) = 0:
      name: Must not have null values
```

**Variables default values**

Variables used in SodaCL are automatically added to the data contract. They will not have a default value. The default values can be added manually by the users.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Security, licensing & compliance

Use these pages to answer security, privacy, and procurement questions about Soda.

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Soda Trust Center</strong></td><td>Review Soda's security and compliance practices, policies, and certifications.</td><td><a href="https://trust.soda.io/">https://trust.soda.io/</a></td></tr><tr><td><strong>Data flows between Soda &#x26; user</strong></td><td>Follow each step, from the runner picking up instructions to results reaching Soda Cloud, for contract verification and metric monitoring with a self-hosted runner.</td><td><a href="https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/data-flows-between-soda-and-user">Data flows between Soda &amp; user</a></td></tr><tr><td><strong>AI security, compliance &#x26; privacy</strong></td><td>See what Soda AI features send to the model provider, what Soda and the provider store, and which settings admins control.</td><td><a href="https://docs.soda.io/soda-documentation/soda-ai/ai-security-compliance-and-privacy">AI security, compliance &amp; privacy</a></td></tr></tbody></table>

## Licensing

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><h4>Licensing &#x26; documentation access</h4></td><td>Find out how to get full access to the documentation.</td><td><a href="https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing">Licensing &amp; documentation access</a></td></tr><tr><td><h4>Soda licenses</h4></td><td>Learn how dataset-based licensing works and how Soda counts your datasets.</td><td><a href="broken://pages/nEeVAUUx8V57By8ALc2b">Broken link</a></td></tr></tbody></table>

## Access control

Control who can see and change what in Soda Cloud.

<p align="center"><a href="https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles" class="button primary">Global and dataset roles</a><a href="https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/user-and-user-group-management-with-sso" class="button primary">User and user group management with SSO</a><a href="https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/user-provisioning-with-scim" class="button primary">User provisioning with SCIM</a><a href="https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/service-accounts" class="button primary">Service accounts</a></p>

> For a step-by-step setup that keeps sensitive data inside your network, see [Manage sensitive data](https://docs.soda.io/guides-and-tutorials/data-governance/manage-sensitive-data).

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Data flows between Soda & user

A description of the data flows that take place when the user interacts with Soda in an enterprise deployment with a self-hosted runner.

#### Platform architecture

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FOlhjIHAJKa5GfNBW2035%2Fversion2__1_.png?alt=media&amp;token=75c3085f-5d44-47af-b148-d68902dc0e1f" alt=""><figcaption><p>Soda platform architecture diagram</p></figcaption></figure>

All flows start with instruction retrieval and a Kubernetes Job launch:

## 1. Instruction retrieval by the Runner

{% stepper %}
{% step %}
The Soda Runner deployed in the customer's Kubernetes cluster (within their VPC) continuously polls the Soda Cloud API for any new "instructions."
{% endstep %}

{% step %}
If instructions are available, the Runner retrieves required resources from Soda Cloud, which include:

* The contract file defining the data quality checks.
* Data source connection details, where sensitive credentials are replaced by placeholders for environment variables.
* Diagnostic warehouse connection details for failed-row storage.
  {% endstep %}
  {% endstepper %}

## 2. Launching a Kubernetes Job

{% stepper %}
{% step %}
Once the resources are retrieved, the Runner launches a Kubernetes Job inside the customer environment to execute the data quality scan.
{% endstep %}

{% step %}
Environment variables (or workload identities via ServiceAccounts) are injected into the Pod to supply secure credentials at runtime.
{% endstep %}

{% step %}
Files such as the contract and configuration are retrieved by the scan process using a call to the Soda Cloud API.
{% endstep %}
{% endstepper %}

***

## Contract Verification

### 3. Data Source Connections

* The Soda process running inside the Job uses the provided details to connect directly to the customer's data sources (e.g., databases or data warehouses).
* This connection stays entirely within the customer's network, ensuring no row-level data leaves the environment.

### 4. Running the Data Quality Contract

{% stepper %}
{% step %}
The Soda Runner translates the data contract into optimized SQL statements
{% endstep %}

{% step %}
The Soda process executes the contract verification, validating the datasets against the defined checks.

During this process:

* Data quality metrics (aggregated statistics, not raw data) are calculated.
* Any rows failing validation checks are isolated for further inspection.
  {% endstep %}
  {% endstepper %}

### 5. Reporting Results

{% stepper %}
{% step %}
Metrics and summaries are sent back to Soda Cloud via an API call.

* These metrics include table/dataset names, check names, logic, and aggregated results.
  {% endstep %}

{% step %}
No row-level data is transmitted to Soda Cloud.
{% endstep %}

{% step %}
Failed rows (if applicable) are written to the diagnostic warehouse inside the customer network, making them accessible for troubleshooting without ever leaving the environment.
{% endstep %}
{% endstepper %}

### 6. Job Completion and Termination

Once the Soda process completes:

{% stepper %}
{% step %}
The Kubernetes Job terminates automatically.
{% endstep %}

{% step %}
The Runner is aware of the termination and signals completion of the instruction back to Soda Cloud through an API call.
{% endstep %}
{% endstepper %}

***

## Metrics Monitoring

### 3. Data source connections

* The Soda process connects directly to the customer's data source and executes SQL to calculate the latest metric value (e.g. row count, null rate). This runs entirely within the customer's network.

### 4. Running the Metrics Monitoring Scan

{% stepper %}
{% step %}
The Runner fetches the metric's history (time-series data) from Soda Cloud via API.
{% endstep %}

{% step %}
The anomaly detection algorithm runs locally inside the Runner.

The algorithm uses the fetched history and the latest value to determine whether the result is anomalous.
{% endstep %}

{% step %}
The Runner sends only the metric value and the anomaly result (pass/fail) back to Soda Cloud.

No raw row data is transmitted.
{% endstep %}
{% endstepper %}

### 5. Job completion and termination

Once the Soda process completes:

{% stepper %}
{% step %}
The Kubernetes Job terminates automatically.
{% endstep %}

{% step %}
The Runner signals completion of the instruction back to Soda Cloud through an API call.
{% endstep %}
{% endstepper %}

***

### Diagnostics Warehouse

The behavior of the Diagnostics Warehouse depends on whether the target is the same data source as the origin or a different one.

> Learn more about the [Diagnostics Warehouse data model](broken://pages/x7jNHrVrxhXlQ9WFF0Qa).

#### Same data source (in-source transfer)

When the Diagnostics Warehouse target is configured as the same connection as the source data source:

* The Runner sends SQL instructions to the data source to move failed rows directly within the warehouse.
* The Runner does **not** extract row-level data—only aggregated check results pass through the Runner.
* By default, all failed rows are transferred; use the [**guardrails**](broken://pages/E1KTiEL4c6zytpedbEZg#set-up-diagnostics-warehouse-for-your-data-source) feature to cap the volume if desired.

#### Different data source (between-source transfer)

When the Diagnostics Warehouse target is a different data source from the origin:

* The Runner queries and extracts the failed rows from the source (all rows by default).
* The Runner then writes those rows into the target Diagnostics Warehouse.
* By default, all failed rows are transferred; use the [**guardrails**](broken://pages/E1KTiEL4c6zytpedbEZg#set-up-diagnostics-warehouse-for-your-data-source) feature to cap the volume if desired.

***

## Security and Isolation

* All communication between Soda Cloud and the Runner is encrypted at rest and in transit.
* Soda Cloud operates in a multi-tenant environment but does not store any record-level data.

## Summary

In this architecture, Soda Cloud orchestrates the scans and collects only aggregated metrics, while all sensitive data stays in the customer's network. For contract verification, failed rows go to a customer-controlled diagnostics warehouse. For metrics monitoring, only the metric value and anomaly result are transmitted to Soda Cloud. In all cases, no row-level data ever leaves the customer's environment.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Licensing & documentation access

Learn how Soda features are documented and how to gain full access to the documentation.

Soda’s documentation is designed to be accessible and relevant to all users; from those exploring Soda for free, to large-scale enterprise customers. **Documentation visibility depends on your license plan** and your **login status**.

**Soda maintains two types of documentation:**

### Public documentation

Public documentation is accessible to **anyone**, without requiring a Soda account.\
It covers general topics such as:

* getting started guides;
* feature overviews and descriptions;
* Soda Cloud and Soda Runner setup;
* common configurations and syntax references;
* open-source integrations and community features;
* and more.

**Public documentation can be accessed at any time, without login.**

***

### Private documentation

Private documentation is **conditional**, meaning access depends on:

* Your **Soda license type**, and
* Whether you are **logged in to Soda** in your browser.

Private documentation includes advanced or licensed-only features such as:

* comprehensive description of features, such as architecture and integration details;
* curated documentation unique to your organization;
* custom data quality policies.

If your plan includes a feature, you **automatically gain access to its related documentation.**

***

## How access works

Access to private documentation is handled **securely and automatically** using **signed cookies** from your Soda session.

1. Log in to [**Soda Cloud**](https://cloud.soda.io/) **using the same browser** where you plan to read the documentation.
2. [Soda Docs](https://docs.soda.io/) reads your **login token (JWT)** from the browser cookies.
3. The documentation content is unlocked automatically according to your license permissions.

**No extra steps are needed.** As long as you’re **logged in on the same browser**, Soda Docs will recognize your plan and grant access accordingly.

### Troubleshooting

If you can’t access documentation you expect to see:

* Make sure you are **logged in to Soda Cloud** on the same browser.
* Refresh the page or clear cookies if access doesn’t update.
* Confirm your **license type** under your Soda account settings.
* Confirm which version of the documentation you are seeing at the **bottom of any page**.
* [**Contact** **Soda**](https://www.soda.io/contact) if your organization has access to private features but can't see the corresponding documentation.

{% hint style="info" %}
**Access is dynamic:** as your organization’s license changes, your documentation access updates automatically.
{% endhint %}

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Integrations overview

Soda offers seamless integrations with many tools across your data stack. Whether you're aligning data governance efforts, collaborating across teams, or triggering workflows, you can enhance Soda’s observability capabilities with the following connections:

## Supported integrations

### Messaging and collaboration

* [MS Teams](https://docs.soda.io/soda-documentation/integrate/ms-teams)
* [Slack](https://docs.soda.io/soda-documentation/integrate/slack)
* [Jira](https://docs.soda.io/soda-documentation/integrate/jira)
* [ServiceNow](https://docs.soda.io/soda-documentation/integrate/servicenow)
* [Webhook](https://docs.soda.io/soda-documentation/integrate/webhook)

> For more details on notification rules, see the [Notification rules documentation](https://docs.soda.io/soda-documentation/fix/notifications).

### Catalogs and governance tools

* [Alation](https://docs.soda.io/soda-documentation/integrate/alation)
* [Atlan](https://docs.soda.io/soda-documentation/integrate/atlan)
* [Collibra](https://docs.soda.io/soda-documentation/integrate/collibra)
* [data.world](https://docs.soda.io/soda-documentation/integrate/data.world)
* [Metaphor](https://docs.soda.io/soda-documentation/integrate/metaphor)
* [Purview](https://docs.soda.io/soda-documentation/integrate/purview)

### BI tools

* [Sigma](https://docs.soda.io/soda-documentation/integrate/sigma)

### Data transformation and code repositories

* [GitHub](https://docs.soda.io/soda-documentation/integrate/github)

***

## Create an integration

To create an integration:

{% stepper %}
{% step %}
Go to the **Integrations** section in **Settings**.
{% endstep %}

{% step %}
Click the **+** button to add a new integration.
{% endstep %}

{% step %}
Select the integration type (**Slack**, **Microsoft Teams**, or **Webhook**).
{% endstep %}

{% step %}
Follow the setup steps for the chosen integration
{% endstep %}

{% step %}
Click **Save** to activate the integration.
{% endstep %}
{% endstepper %}

***

## Edit an integration

You can update existing integrations if connection details or configurations change.

To edit an integration:

{% stepper %}
{% step %}
Go to the **Integrations** section in **Settings**.
{% endstep %}

{% step %}
Find the integration you want to update.
{% endstep %}

{% step %}
Click the context menu and select **Edit Integration Settings**.
{% endstep %}

{% step %}
Update the configuration as needed.
{% endstep %}

{% step %}
Click **Save** to apply the changes.
{% endstep %}
{% endstepper %}

## Pause an integration

You can temporarily pause an integration if you want to stop sending notifications and incident updates without fully deleting the configuration. The integration will no longer be available in notification rules.

To pause an integration:

{% stepper %}
{% step %}
Go to the **Integrations** section in **Settings**.
{% endstep %}

{% step %}
Locate the integration you want to pause.
{% endstep %}

{% step %}
Change the status to "Paused" in the table
{% endstep %}

{% step %}
Select **Pause**.
{% endstep %}
{% endstepper %}

While paused, the integration will no longer send any notifications. You can resume it at any time by following the same steps and selecting **Active**.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Alation

Learn how to integrate Alation with Soda.

{% hint style="info" %}
You will be prompted to [contact support](mailto:support@soda.io) as a last step to finish setting up the Alation integration.
{% endhint %}

Integrate Soda with Alation to access details about the quality of your data from within the data catalog.

* Run data quality checks using Soda and visualize quality metrics and rules within the context of a data source, dataset, or column in Alation.
* Use Soda Cloud to flag poor-quality data in lineage diagrams and during live querying.
* Give your Alation users the confidence of knowing that the data they are using is sound.

> 🎥 Watch a [5-minute overview](https://vimeo.com/563765014) showcasing the integration of Soda and Alation.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FDW8OjhaQVXgtQbU8fQsJ%2Fimage.png?alt=media&amp;token=b769152a-dc48-4422-a91e-fa42d553c857" alt=""><figcaption><p>Alation dashboard when integrated with Soda</p></figcaption></figure>

## Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* You have verified some contracts and published the results to Soda Cloud.
* You have an Alation account with the privileges necessary to allow you to add a data source, create custom fields, and customize templates.
* You have a git repository in which to store the integration project files.

***

## Local setup <a href="#set-up-the-integration" id="set-up-the-integration"></a>

{% stepper %}
{% step %}

#### Credentials

**1.1. Log in to Soda Cloud**

Sign into your Soda Cloud account and confirm that you see the datasets you expect to see in the data source you wish to test for quality.

**1.2. Create an `.env` file**

To connect your Soda Cloud account to your Alation Service Account, create an `.env` file in your integration project in your git repo and include details according to the example below. Refer to [Generate API keys](https://docs.soda.io/apis/generate-api-keys) to obtain the values for your Soda API keys.

{% code title=".env" %}

```dotenv
ALATION_HOST=yourcompany.alationcatalog.com
ALATION_USER=<your username for your Alation account>
ALATION_PASSWORD=<your password for your Alation account>
SODA_HOST=cloud.soda.io            ## or cloud.soda.us.io, depending on your organization
SODA_API_KEY_ID=<your Soda Cloud pubic key>
SODA_API_KEY_SECRET=<your Soda Cloud private key>
```

{% endcode %}

**Optional Configurations:**

* `SSL_VERIFY` - `True` / `False` - set whether ssl certificates should be verified (default `True`). Disabling ssl verify should only be used for POCs and temporarily, not for production.
* `SSL_VERIFY_PATH` - string, a path to local certificate to be used if custom/self-signed ssl certificates are used.
* `REFRESH_TOKEN_PATH` - string, storage path for the Alation refresh token
* `CONFIGURED_REFRESH_TOKEN_PATH` - string, path to a directory where the Alation refresh token is located. This is used to initially configure a Alation refresh token. The file in this path is copied into the `REFRESH_TOKEN_PATH`
* `SODA_DATASOURCE_MAPPING_FILE` - string, path with file name to the datasource mapping file
* `SODA_DESCRIPTION_ATTRIBUTE` - string, Soda check attribute name to use for the custom description fields in Alation
  {% endstep %}

{% step %}

#### Data source mappings

**Create a mapping file**

To sync a data source and schema in the Alation catalog to a data source in Soda Cloud, you must map it from Soda Cloud to Alation. Create a `.datasource-mapping.yml` file in your integration project and populate it with mapping data according to the following example. The table below describes where to retrieve the values for each field.

{% code title=".datasource-mapping.yml" %}

```yaml
 - name: Cars
   soda:
     datasource_id: 2d33bf0a-9a1c-4c4b-b148-b5af318761b3
     datasource_name: adventureworks
     # optional dataset_mapping   soda: catalog
     dataset_mapping:
        Cars_data: Cars
   catalog:
     type: "alation"
     datasource_id: "31"
     datasource_container_name: "soda"
     datasource_container_id: "1"
 - name: Soda Demo
   soda:
     datasource_id: 8505cbbd-d8b3-48a4-bad4-cfb0bec4c02f
   catalog:
     type: "alation"
     datasource_id: "37"
     datasource_container_name: "public"
     datasource_container_id: "2"
```

{% endcode %}

| Field                                                                    | Retrieve value from                                                                                                                                                                                                                               |
| ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                                                                   | A name you choose as an identifier for an integration between Soda Cloud and a data catalog.                                                                                                                                                      |
| <p><code>soda:</code><br><code>datasource\_id</code></p>                 | The data source information panel in Soda Cloud.                                                                                                                                                                                                  |
| <p><code>soda:</code><br><code>datasource\_name</code></p>               | The data source information panel in Soda Cloud.                                                                                                                                                                                                  |
| <p><code>soda:</code><br><code>dataset\_mapping</code></p>               | <p>(Optional) When you run the integration, Soda automatically maps all of the datasets between data sources. However, if the names of the datasets differ in the tools you can use this property to manually map datasets between tools.<br></p> |
| <p><code>catalog:</code><br><code>type:</code></p>                       | The name of the cataloging software; in this case, “alation”.                                                                                                                                                                                     |
| <p><code>catalog:</code><br><code>datasource\_id</code></p>              | Retrieve this value from the URL on the data source page in the Alation catalog; see image below.                                                                                                                                                 |
| <p><code>catalog:</code><br><code>datasource\_container\_name</code></p> | The schema of the data source; retrieve this value from the data source page in the Alation catalog under the subheading **Schemas**. See image below.                                                                                            |
| <p><code>catalog:</code><br><code>datasource\_container\_id</code></p>   | The ID of the `datasource_container_name` (the schema of the data source); retrieve this value from the schema page in the Alation catalog. See image below                                                                                       |

* Retrieve the Alation `datasource_id` from the URL<br>

  <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F5AV9O7SwS8q01Us1sbxV%2Fimage.png?alt=media&amp;token=8c51d7cb-7edf-4c65-91e4-01720180d74d" alt=""><figcaption><p>The <code>datasource_id</code> is in the URL of the data source page</p></figcaption></figure>
* Retrieve the Alation `datasource_container_name` (schema) from the data source page<br>

  <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F53h9EgrhkVK623RaWZdM%2Fimage.png?alt=media&amp;token=cf603fe7-5e3d-4131-90b6-a0d513f1e3cf" alt=""><figcaption><p>The <code>datasource_container_name</code> is the data source schema name</p></figcaption></figure>
* Retrieve the Alation `datasource_container_id` for the `datasource_container_name` from the URL in the **Schema** page.\ <br>

  <figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FtuIbWOigQKpRGvqm3g0A%2Fimage.png?alt=media&amp;token=0b91679c-400b-406c-8156-b42665cce21a" alt=""><figcaption></figcaption></figure>

{% endstep %}

{% step %}

#### Onboarded datasets

Make sure that datasets you wish to sync data quality data from Soda Cloud to catalog are fully onboarded on both sides.
{% endstep %}

{% step %}

#### \[Optional] Enable API access to Alation with SSO

If your Alation account employs **single sign-on (SSO)** access, you must [Create an API service account](https://developer.alation.com/dev/docs/creating-an-api-service-account) for Soda to integrate with Alation.

If your Alation account **does not use SSO**, skip this step and proceed to [Customize the catalog](https://docs.soda.io/soda-documentation/integrate/alation#customize-the-catalog).
{% endstep %}

{% step %}

#### Customize the catalog

{% hint style="info" %}
Some catalogs require manual customization for this integration to work.
{% endhint %}

Alation custom fields are created in global context on Alation -> Settings -> Customize Catalog -> Custom Fields page. These fields can then be attached to any entity on Custom Templates tab on the same page.

Set up the following custom fields and then attach them to the Table entity:

* `Has DQ` - Picker with `True` and `False` values (!) Make sure to use these values exactly as Alation API is case-sensitive.
* `Profile - Last Run` - Date
* `Soda DQ Overview` - Rich Text

**6.1. Custom fields**

Create custom fields in Alation that reference information that Soda Cloud pushes to the catalog. These are the fields the catalog users will see that will display Soda Cloud data quality details.\
In your Alation account, navigate to **Settings** > **Catalog Admin** > **Customize Catalog**. In the **Custom Fields** tab, create the following fields:

* Under the **Pickers** heading, create a field for “Has DQ” with Options “True” and “False”. The Alation API is case sensitive so be sure to use these exact values.
* Under the **Dates** heading, create a field for “Profile - Last Run”.
* Under the **Rich Texts** heading, create the following fields:
  * “Soda DQ Overview”
  * “Soda Data Quality Rules”
  * “Data Quality Metrics”

**6.2. Add custom fields to Custom Templates**

Add each new custom field to a **Custom Template** in Alation. In **Customize Catalog**, in the **Custom Templates** tab, select the **Table** template, then click **Insert…** to add a custom field to the template:

* “Soda DQ Overview”

**6.3. Add "Data Quality Info" grouping**

In the **Table** template, click **Insert…** to add a **Grouping of Custom Fields**. Label the grouping “Data Quality Info”, then **Insert…** two custom fields:

* “Has DQ”
* “Profile - Last Run”

1. In the **Column** template, click **Insert…** to add a custom field to the template:
   * “Has DQ”

**6.4. Add "Soda Data Profile Information" grouping**

**6.4.** In the **Column** template, click **Insert…** to add a **Grouping of Custom Fields**. Label the grouping “Soda Data Profile Information”, then **Insert…** two custom fields:

* Data Quality Metrics
* Soda Data Quality Rules
  {% endstep %}
  {% endstepper %}

## Kubernetes setup <a href="#run-the-integration" id="run-the-integration"></a>

The Soda-Alation integration can be run in a Kubernetes cluster. Contact <support@soda.io> to run the integration in your organization.

***

## Run the integration <a href="#run-the-integration" id="run-the-integration"></a>

Contact <support@soda.io> directly to acquire the assets and instructions to run the integration and view Soda Cloud details in your Alation catalog.

***

## Use the integration <a href="#use-the-integration" id="use-the-integration"></a>

Access Soda Cloud to [create no-code checks](https://docs.soda.io/soda-documentation/find/data-testing/cloud-managed-data-contracts) or [initiate a request](https://docs.soda.io/soda-documentation/find/data-testing/contract-collaboration#initiate-a-request) in order to **execute checks against datasets** in your data source each time you run a Soda scan manually or orchestrate a scan using a data pipeline tool, such as Airflow. Soda Cloud **pushes data quality scan results** to the corresponding data source in Alation so that users can **review data quality information from within the catalog**.

In Alation, beyond reviewing data quality information for the data source, users can access the **Joins** and **Lineage** tabs of individual datasets to examine details and investigate the source of any data quality issues.

#### Open in Soda <a href="#open-in-soda" id="open-in-soda"></a>

In a dataset page in Alation, in the **Overview** tab, users have the opportunity to review Soda information on Alation, or directly access Soda Cloud to scrutinize data quality details.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FnZjZg0zqu3QaXV0UjGgb%2Fimage.png?alt=media&amp;token=8c93dc3a-0081-498e-8415-9055116478f2" alt=""><figcaption><p>Under the <strong>Soda DQ Overview</strong> heading in Alation, click <strong>Open in Soda</strong> to access the dataset page in Soda Cloud</p></figcaption></figure>

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Atlan

Learn how to integrate Atlan with Soda.

Integrate Soda with Atlan to access details about the quality of your data from within the data catalog.

* Run data quality checks using Soda and visualize quality metrics and rules within the context of a data source, dataset, or column in Atlan.
* Use Soda Cloud to flag poor-quality data in lineage diagrams.
* Give your Atlan users the confidence of knowing that the data they are using is sound.

***

When integrating Soda with Atlan, you can access Soda information directly in your Atlan dashboards and assets:

#### Check results overview

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F8dvWZMyED0Kn06wSbphL%2Fimage.png?alt=media&amp;token=90d8fe67-4cc3-41ed-90b6-78fdbc54540f" alt=""><figcaption><p>In asset overview, Atlan displays Soda check results overview (top right).</p></figcaption></figure>

#### Soda scan & checks details

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FQXP8J29VJa7Q4UPBmZxi%2Fimage.png?alt=media&amp;token=8f79d5ee-0229-4089-858e-44d7854ed714" alt=""><figcaption><p>Atlan displays a Soda icon on the vertical right menu. The panel shows Soda scan details, such as check results, last scan and sync time. From this screen, users can directly open the dataset in Soda Cloud (top right).</p></figcaption></figure>

#### Lineage + Soda check results

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FMnBncZwTKIx9mzGTeHh5%2Fimage.png?alt=media&amp;token=880a9b24-7496-4c9a-801c-da2b8f27889a" alt=""><figcaption><p>Lineage diagrams will display a Soda icon over integrated datasets, allowing to directly view Soda check results without leaving the Atlan Lineage page.</p></figcaption></figure>

## Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* You have verified some contracts and published the results to Soda Cloud.
* You have an Atlan account with the privileges necessary to allow you to set up a Connection in your Atlan workspace.

## Set up the integration <a href="#set-up-the-integration" id="set-up-the-integration"></a>

You can integrate Atlan and Soda from your Atlan account.

{% stepper %}
{% step %}

#### Add a new workflow

On the main page, create a **New workflow**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FbN18zsqa5MnrfWqaKVR8%2Fimage.png?alt=media&amp;token=1688da4c-0afb-4e67-b73c-4e9985f8430c" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Select the Soda package

Creating a new workflow will redirect you to the **Marketplace** page. There, look for the **Soda package** and select it.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FohG1omCqbOOf3hlWvW7q%2Fimage.png?alt=media&amp;token=c8b1c4a5-707b-4e3a-a2f3-b33d460f0b10" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Create a new connection

After selecting the Soda package, click on **Setup Workflow** to create a new connection between Atlan and Soda.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FQRfL63jUp0av7RWtfqDM%2Fimage.png?alt=media&amp;token=522c58ba-cbb0-4def-8135-ad5a876fcaf5" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Enter connection details

Follow the instructions to [Generate API keys](https://docs.soda.io/apis/generate-api-keys) in Soda Cloud to get your API keys.

<i class="fa-octagon-check">:octagon-check:</i> Test Authentication > Authentication successful > **Next** (bottom right)

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FaBUmmi06lRdPYdeEKrq9%2Fimage.png?alt=media&amp;token=082d059d-eb32-4603-af0d-4496dac57dab" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Name the connection and select admins

Enter a connection name and, optionally, add or remove connection admins. Click on **Next**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FjP0fOkFF5thMiu3Y2dXk%2Fimage.png?alt=media&amp;token=302e2aa5-b4b7-4239-8743-e0e8f329b437" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Advanced configuration

You can include or exclude **filters**, select **specific connections** for assets to which you want to map Soda checks, or run **preflight checks**.

> For more details on advanced configuration, visit [Atlan's documentation](https://docs.atlan.com/apps/connectors/observability/soda/how-tos/crawl-soda).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FxoJcEFRav0HedoNgynuA%2Fimage.png?alt=media&amp;token=de4ab3d0-8bf6-49a3-b7cc-6ef0c348adc5" alt=""><figcaption></figcaption></figure>
{% endstep %}

{% step %}

#### Run the connection

**Run** the connection immediately or **Schedule & Run** to run periodically.
{% endstep %}

{% step %}

#### <i class="fa-octagon-check">:octagon-check:</i> Review the monitoring page

Once the workflow has run (you will be redirected if you choose to **Run** immediately), you will be able to see the status in the **Workflow center**.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FV786gQV7sTVP2IHbQqpQ%2Fimage.png?alt=media&amp;token=bd2458ca-8d46-4b10-8a06-affe01a5e817" alt=""><figcaption></figcaption></figure>
{% endstep %}
{% endstepper %}

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Collibra

Overview of the bi-directional integration between Soda and Collibra: what it does, what it creates, and what you need to run it.

Bring Soda's data quality results into Collibra, and Collibra's governance context into Soda.

The Soda ↔ Collibra integration connects data quality execution in Soda with data governance in Collibra. Soda knows whether a table is trustworthy right now; Collibra is where people go to find out. The integration keeps the second answer honest.

## Capabilities

**Soda → Collibra.** Every check in scope becomes a governance asset in Collibra, carrying its latest result, its definition, when it last ran, a link back to Soda, and (where Soda reports them) the rows tested, passed and failed. Checks are linked to the table or column they test, to their data quality dimension, and to the data standard that generated them.

**Collibra → Soda.** Ownership, tags, dataset attributes and column-level business terms flow the other way, so the people and definitions that live in your glossary show up next to the checks in Soda.

Each direction is independently configurable, and every Collibra → Soda sync is off by default.

## Created assets

| Soda             | Collibra asset type                                               |
| ---------------- | ----------------------------------------------------------------- |
| Check or monitor | Data Quality Rule                                                 |
| Group-by check   | Data Quality Rule (the check) + one Data Quality Metric per group |
| Data standard    | Business Rule                                                     |

All three are configurable if your operating model uses custom types.

## Run sequence

The integration runs as a scheduled job — typically daily, shortly after your Soda scans finish. Each run reads your Soda Cloud organization, writes data quality assets into Collibra, and writes a small amount of governance metadata back into Soda.

It holds no state of its own: every run reconciles Collibra against whatever Soda currently says, so runs are repeatable and a second run over unchanged data does almost nothing. A dataset that fails does not stop the run.

## Requirements at a glance

* Python 3.10 or later, or the container image
* A Soda Cloud API key
* A Collibra user or OAuth application with permission to create and edit assets, attributes and relations
* Network access from wherever the job runs to both Soda Cloud and Collibra

Contact <support@soda.io> for access to the distribution.

## Documentation

|                                                                 |                                                                    |
| --------------------------------------------------------------- | ------------------------------------------------------------------ |
| [How it works](broken://pages/N2olF6JBzvvSr1UHT89H)             | What a run does, its scope, and what it does not do                |
| [Prerequisites](broken://pages/fo6AUNgXnLXoA6VOYVyW)            | What must exist in Soda and Collibra before you configure anything |
| [Setup](broken://pages/46l8iohZtc5njWj0UNyh)                    | Install, configure, and complete your first run                    |
| [Collibra operating model](broken://pages/wjJw7d9QpqB3HUXdVW0M) | Asset types, relation widening, and data quality scores            |
| [Configuration reference](broken://pages/GRr9RYqbuI74nuV3YAow)  | Every field in `config.yaml`                                       |

### Synchronized data

**Soda → Collibra**

|                                                            |                                                                            |
| ---------------------------------------------------------- | -------------------------------------------------------------------------- |
| [Checks and monitors](broken://pages/IBT09HEObxu54U38Jhih) | Every check becomes a Collibra asset with its result, definition and links |
| [Group-by checks](broken://pages/thQwowztU9ad5fkKfIY3)     | One rule asset plus one metric asset per group                             |
| [Data standards](broken://pages/ChiqAZX6I8gKJ9rhN4ih)      | Each standard becomes a Business Rule, linked to the checks it generates   |
| [Deleted checks](broken://pages/oRgsV9nQi5gDCoUVPfAO)      | Delete or mark obsolete when a check disappears from Soda                  |

**Collibra → Soda**

|                                                              |                                            |
| ------------------------------------------------------------ | ------------------------------------------ |
| [Dataset metadata](broken://pages/CVPHqS3m64xHBvVa0hAj)      | Owners, tags and dataset attributes        |
| [Column business terms](broken://pages/qnWgO9gKTfibLFBkNCym) | Glossary terms onto Soda column attributes |

### Running it

|                                                        |                                                |
| ------------------------------------------------------ | ---------------------------------------------- |
| [Deployment](broken://pages/wQUaS1SCQ6oEB5clHJqb)      | Kubernetes, Docker, AWS Lambda, and scheduling |
| [Troubleshooting](broken://pages/P4zL6ZJiMYQoCgIn1U5A) | Symptoms, causes, and what to check            |
| [FAQ](broken://pages/zvo38191o7eU8WkP1B0N)             | Common questions                               |

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# data.world

Learn how to integrate data.world with Soda.

Integrate Soda with data.world to surface data quality check results directly within your data.world catalog.

* Run data quality checks using Soda and push check results and quality scores to the corresponding tables in your data.world catalog.
* Give your data.world users the confidence of knowing that the data they are using is sound.
* Drill down from a data.world table directly into Soda Cloud to investigate check details, diagnostics, and historical trends.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fe9nOxwTmVz6bwqqcqlSc%2Fimage.png?alt=media&amp;token=6d6f70bd-0ac0-4f11-9c1c-c97c246faa4c" alt=""><figcaption><p>Example view of the Data Quality results tab for a table in Data.world</p></figcaption></figure>

### How it works

The integration is a lightweight CLI tool (`soda-dw-sync`) that you install and schedule. On each run it:

1. Reads your data source mapping from `ds_config.yml` to determine which Soda Cloud data sources to process and how they map to data.world collections.
2. For each data source, retrieves all datasets from the Soda Cloud API.
3. For each dataset, retrieves all checks and their latest evaluation results.
4. Transforms those results into the format expected by the data.world Check Runs and Badges APIs.
5. Posts check results and a dataset-level quality badge (Good / Moderate / Poor) to the matching table in data.world. Tables are identified by their fully-qualified name (`database.schema.table`) extracted from Soda Cloud.

***

## Prerequisites

* Python 3.10+
* You have completed at least one Soda scan and confirmed that the datasets you wish to sync appear in Soda Cloud.
* You have a data.world account with the privileges necessary to write to the target catalog collection.
* You have a git repository (or local directory) in which to store the integration project files.
* Access to the data.world repository. Contact us at <support@soda.io> to receive access.

## Set up the integration

{% stepper %}
{% step %}

#### Install

{% hint style="info" %}
Installing and using the data.world integration requires access to the integration repository. **Contact us at** [**soda@support.io**](mailto:soda@support.io) **to get access to the repository.**
{% endhint %}

Best practice dictates that you **install the integration using a virtual environment** and that you use **environment variables** to store secrets. If you haven't yet, in your command-line interface tool, create a virtual environment in the `.venv` directory.

```bash
# Optional: create a virtual environment first
python3.10 -m venv .venv
source .venv/bin/activate
```

Clone or download the data.world integration repository, then install from the repo root:

```bash
pip install .
```

Alternatively, install directly from a release wheel:

```bash
pip install soda_dw_integration-0.1.0-py3-none-any.whl
```

{% endstep %}

{% step %}

#### Configure Soda Cloud credentials

Create a `sc_config.yml` file with your Soda Cloud connection details.

{% code title="sc\_config.yml" %}

```yaml
soda_cloud:
  host: cloud.soda.io               # or cloud.us.soda.io for US region
  api_key_id: ${env.SODA_API_KEY_ID}
  api_key_secret: ${env.SODA_API_KEY_SECRET}
  dimension_attribute: quality_dimension   # optional; see note below
```

{% endcode %}

> Refer to [Generate API keys](https://docs.soda.io/apis/generate-api-keys) to create the `api_key_id` and `api_key_secret` values.

**Note:** The `dimension_attribute` field is optional. When set, the tool reads the named check attribute from Soda Cloud and uses its value as the quality dimension label in data.world. If the attribute is absent or the field is omitted, the check's `checkType` is used instead.
{% endstep %}

{% step %}

#### Configure data source mappings

Create a `ds_config.yml` file that maps each Soda Cloud data source to the corresponding collection in data.world. Add one entry per data source.

{% code title="ds\_config.yml" %}

```yaml
data_sources:
  your_datasource_name:                     # must match the data source name in Soda Cloud
    context: <org-runner-id-of-source-org>
    databaseLocation: <host:port>
    orgRunnerId: <org-runner-id-of-target-org>
    collectionIri: <collection-iri>
    dw_api_token: ${env.DW_API_TOKEN}
```

{% endcode %}

The table below describes each field. All values except `dw_api_token` come from data.world.

| Field              | Description                                                                                                               |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------- |
| `context`          | The runner ID of the data.world org that originally cataloged the database objects (tables).                              |
| `databaseLocation` | The `host:port` of the source database, e.g. `myorg.snowflakecomputing.com:443` or `dbc-abc123.cloud.databricks.com:443`. |
| `orgRunnerId`      | The runner ID of the data.world org that owns the collection where check results will be stored.                          |
| `collectionIri`    | The IRI of the data.world collection where check results will be stored.                                                  |
| `dw_api_token`     | A data.world API token with write access to the target collection.                                                        |

> For more details on how data.world uses these identifiers, see the [data.world Add Check Runs API reference](https://developer.data.world/reference/addcheckruns).
> {% endstep %}

{% step %}

#### Verify your setup with a dry run

Before uploading anything to data.world, use `--dry-run` to write the payloads to local JSON files for inspection:

```bash
soda-dw-sync \
  --ds-config ./ds_config.yml \
  --sc-config ./sc_config.yml \
  --dry-run \
  --out-dir ./output
```

Inspect `output/check_runs_<datasource>.json` and `output/badges_<datasource>.json` to confirm the data looks correct, then run without `--dry-run` to push results to data.world.
{% endstep %}

{% step %}

#### Run the integration

```bash
soda-dw-sync --ds-config ./ds_config.yml --sc-config ./sc_config.yml
```

**Useful CLI options:**

| Option                           | Description                                                                      |
| -------------------------------- | -------------------------------------------------------------------------------- |
| `--only-dataset db.schema.table` | Limit the sync to one or more specific datasets. Repeatable.                     |
| `--dry-run`                      | Write JSON payloads to `--out-dir` instead of uploading to data.world.           |
| `--out-dir ./output`             | Directory for dry-run JSON output files.                                         |
| `--log-level DEBUG`              | Increase log verbosity. Accepts `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. |

{% hint style="info" %}
Run `soda-dw-sync --help` to see all options.
{% endhint %}
{% endstep %}
{% endstepper %}

## Schedule the integration

### Cron

After a local install, schedule with cron to keep data.world up to date automatically:

```bash
# Runs daily at 02:15
15 2 * * * /path/to/.venv/bin/soda-dw-sync \
  --ds-config /etc/soda-dw/ds_config.yml \
  --sc-config /etc/soda-dw/sc_config.yml \
  >> /var/log/soda-dw-sync.log 2>&1
```

Edit your crontab with `crontab -e` and remove the entry to stop the job.

***

### Docker

Build the image:

```bash
docker build -t soda-dw:local .
```

Run via cron using Docker:

```bash
15 2 * * * docker run --rm \
  -v /etc/soda-dw:/etc/soda-dw:ro \
  -e SODA_API_KEY_ID=$SODA_API_KEY_ID \
  -e SODA_API_KEY_SECRET=$SODA_API_KEY_SECRET \
  -e DW_API_TOKEN=$DW_API_TOKEN \
  soda-dw:local \
  --ds-config /etc/soda-dw/ds_config.yml \
  --sc-config /etc/soda-dw/sc_config.yml \
  >> /var/log/soda-dw-sync.log 2>&1
```

***

### Kubernetes

The `k8s/` directory in the repository contains ready-to-use manifest templates for running the sync as a Kubernetes CronJob.

Load your local image into the cluster:

```bash
minikube start -p minikube --driver=docker
minikube image load -p minikube soda-dw:local
```

Apply the manifests:

```bash
kubectl apply -f k8s/secret.example.yml
kubectl apply -f k8s/configmap.example.yml
kubectl apply -f k8s/cronjob.local-image.yml
kubectl get cronjobs
```

View logs from the most recent job run:

```bash
kubectl logs job/<job-name>
```

Delete the CronJob to stop scheduling:

```bash
kubectl delete cronjob soda-dw-sync
```

The ConfigMap mounts both `sc_config.yml` and `ds_config.yml` into the container. Store secrets (API keys) in a Kubernetes Secret and reference them via `envFrom` in the CronJob spec, using the `${env.VAR}` syntax in your YAML files.

***

## Use the integration

Once the sync runs, navigate to any cataloged table in data.world. Open the **Data Quality** tab to see:

* All Soda checks associated with the table, their latest result (Pass / Fail / Warn), and a direct link back to Soda Cloud for each check.
* A dataset-level quality badge. By default, the values are set to:
  * **Good** (≥ 90% passing)
  * **Moderate** (≥ 70% passing)
  * **Poor** (< 70% passing).

Click any check link to open the corresponding check page in Soda Cloud, where you can review diagnostics, historical results, and metric values.

***

## Limitations & edge cases

* **Metric Monitor checks are not synced.** Checks that use Soda's anomaly detection (Metric Monitors) are automatically skipped. Only threshold-based checks are pushed to data.world.
* **Datasets with no eligible checks are skipped.** If a dataset has no checks, or all of its checks are Metric Monitors, no check runs or badge are posted for that dataset.
* **Table identity depends on qualified name format.** Tables are matched in data.world using the fully-qualified name extracted from Soda Cloud (`database.schema.table` or `database.table`). If the name in Soda Cloud does not match what data.world expects, results will not surface on the correct table.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# GitHub

Learn how to integrate GitHub with Soda.

This page explains the GitHub Actions workflows that integrate with [Soda](https://soda.io) to manage data contracts automatically. It covers two key workflows:

1. **Publish Contracts on Merge to Main**
2. **Verify Contracts on Pull Request**

***

## Publish Contracts on Merge

### Overview

The **Publish Contracts on Merge** workflow automates the publishing of any updated or new Soda contracts when changes are pushed to the `main` branch.

This ensures that all contract changes are automatically deployed to Soda Cloud whenever they're merged into the production branch.

### Action

```yaml
name: Publish Updated Contracts on Merge

on:
  push:
    branches:
      - main

jobs:
  publish-contracts:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install soda-postgres
        run: pip install -i https://pypi.cloud.soda.io/simple "soda>=4.0.0.dev1" -U

      - name: Get all changed files
        id: changed-files
        uses: tj-actions/changed-files@v46

      - name: List all changed files
        env:
          ALL_CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
        run: |
          for file in ${ALL_CHANGED_FILES}; do
            echo "$file was changed"
          done

      - name: Debug environment variables
        env:
          SODA_CLOUD_API_KEY: ${{ secrets.SODA_CLOUD_API_KEY }}
          SODA_CLOUD_API_SECRET: ${{ secrets.SODA_CLOUD_API_SECRET }}
        run: |
          echo "Environment variables status:"
          echo "SODA_CLOUD_API_KEY: $(if [ -n "$SODA_CLOUD_API_KEY" ]; then echo "✅ Set (${#SODA_CLOUD_API_KEY} chars)"; else echo "❌ Not set"; fi)"
          echo "SODA_CLOUD_API_SECRET: $(if [ -n "$SODA_CLOUD_API_SECRET" ]; then echo "✅ Set (${#SODA_CLOUD_API_SECRET} chars)"; else echo "❌ Not set"; fi)"

      - name: Filter and publish contracts
        env:
          ALL_CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
          SODA_CLOUD_CONFIG_FILE_PATH: soda-cloud.yaml
          SODA_CLOUD_API_KEY: ${{ secrets.SODA_CLOUD_API_KEY }}
          SODA_CLOUD_API_SECRET: ${{ secrets.SODA_CLOUD_API_SECRET }}
        run: |
          for file in ${ALL_CHANGED_FILES}; do
            if [[ "$file" == contracts/*.yml || "$file" == contracts/*.yaml ]]; then
              echo "Publishing $file"
              echo "Executing: soda contract publish --contract \"$file\" --soda-cloud ${SODA_CLOUD_CONFIG_FILE_PATH}"
              soda contract publish --contract "$file" --soda-cloud ${SODA_CLOUD_CONFIG_FILE_PATH}
            else
              echo "Skipping $file (not a contract)"
            fi
          done

```

### What It Does

1. **Checks out the repo**
2. **Sets up Python**
3. **Installs the latest version of `soda`**
4. **Identifies changed files**
5. **Filters YAML files in the `contracts/` directory**
6. **Publishes valid contracts to Soda Cloud**

### Required GitHub Secrets

Make sure these are set in your repository’s **GitHub Secrets:**

* `SODA_CLOUD_API_KEY`
* `SODA_CLOUD_API_SECRET`

> Learn how to [generate API keys](https://docs.soda.io/apis/generate-api-keys).

### Customization Options

<table><thead><tr><th>Option</th><th>Description</th></tr></thead><tbody><tr><td><pre><code>on:
  push:
    branches:
      - main
</code></pre></td><td>Change the Action trigger.</td></tr><tr><td><code>pip install</code></td><td>Can specify a fixed version of <code>soda</code> for stability.</td></tr><tr><td><code>SODA_CLOUD_CONFIG_FILE_PATH</code></td><td>Path to your Soda Cloud config. Can be replaced if your setup uses a different config file name or location.</td></tr><tr><td><code>contracts/*.yml</code> or <code>contracts/*.yaml</code></td><td>Modify file pattern to match a different directory or naming convention.</td></tr></tbody></table>

### Example output

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FrpKff36fn4elktNO2uga%2FScreenshot%202025-07-30%20at%203.28.58%E2%80%AFPM.png?alt=media&amp;token=a13eba43-88b9-4ea3-b40c-ae434c87409d" alt=""><figcaption></figcaption></figure>

***

## Verify Contracts on Pull Request

### Overview

The **Verify Contracts on Pull Request** workflow ensures that contract changes in PRs are valid and do not break expectations before merging.

The workflow runs when a PR is opened, updated, or reopened.

### Action

```yaml
name: Verify Data Contracts on pull request

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  verify-contracts:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install soda-postgres
        run: pip install -i https://pypi.cloud.soda.io/simple -U soda-postgres

      - name: Get all changed files
        id: changed-files
        uses: tj-actions/changed-files@v46

      - name: List all changed files
        env:
          ALL_CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
        run: |
          for file in ${ALL_CHANGED_FILES}; do
            echo "$file was changed"
          done

      - name: Debug environment variables
        env:
          DATASOURCE_USERNAME: ${{ secrets.DATASOURCE_USERNAME }}
          DATASOURCE_PASSWORD: ${{ secrets.DATASOURCE_PASSWORD }}
        run: |
          echo "Environment variables status:"
          echo "DATASOURCE_USERNAME: $(if [ -n "$DATASOURCE_USERNAME" ]; then echo "✅ Set"; else echo "❌ Not set"; fi)"
          echo "DATASOURCE_PASSWORD: $(if [ -n "$DATASOURCE_PASSWORD" ]; then echo "✅ Set"; else echo "❌ Not set"; fi)"

      - name: Filter and verify contracts
        env:
          ALL_CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }}
          DATASOURCE_CONFIG_FILE_PATH: postgres.yaml
          DATASOURCE_USERNAME: ${{ secrets.DATASOURCE_USERNAME }}
          DATASOURCE_PASSWORD: ${{ secrets.DATASOURCE_PASSWORD }}
        run: |
          for file in ${ALL_CHANGED_FILES}; do
            if [[ "$file" == contracts/*.yml || "$file" == contracts/*.yaml ]]; then
              echo "Verifying $file"
              echo "Executing: soda contract verify --data-source ${DATASOURCE_CONFIG_FILE_PATH} --contract \"$file\""
              soda contract verify --data-source ${DATASOURCE_CONFIG_FILE_PATH} --contract "$file"
            else
              echo "Skipping $file (not a contract)"
            fi
          done

```

### What It Does

1. **Checks out the PR branch**
2. **Sets up Python**
3. **Installs latest `soda-postgres`**
4. **Identifies changed files**
5. **Filters contracts in the `contracts/` directory**
6. **Runs verification checks against a configured data source**

### Required Secrets

Make sure these are set in your repository’s **GitHub Secrets:**

* `DATASOURCE_USERNAME`
* `DATASOURCE_PASSWORD`

These secrets can be customized depending on the data source type and your needs.

### Customization Options

<table><thead><tr><th>Option</th><th>Description</th></tr></thead><tbody><tr><td><pre><code>on:
  pull_request:
    types: [opened, synchronize, reopened]
</code></pre></td><td>Change the Action trigger</td></tr><tr><td><code>pip install</code></td><td><p>Adapt the install command to install the necessary package for your data source.</p><p><a data-mention href="https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core">Data source reference for Soda Core</a></p><p>You can specify a fixed version of <code>soda</code> for stability.</p></td></tr><tr><td><code>contracts/*.yml</code> or <code>contracts/*.yaml</code></td><td>Change to match your directory structure.</td></tr><tr><td><code>DATASOURCE_CONFIG_FILE_PATH</code></td><td>Replace with the path to your data source configuration</td></tr><tr><td><code>DATASOURCE_USERNAME</code> and <code>DATASOURCE_PASSWORD</code></td><td>Adapt the secrets used to connect to your data source depending on the data source type and security requirements.</td></tr></tbody></table>

### Example output

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F4S5jf0H3SEvyjWLBZnhs%2FScreenshot%202025-07-30%20at%203.22.29%E2%80%AFPM.png?alt=media&amp;token=523cc9a7-6382-401d-899a-0b4bb2c47f9d" alt=""><figcaption></figcaption></figure>

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FcXa0NozGZmyXfQBCLcV2%2FScreenshot%202025-07-30%20at%203.22.37%E2%80%AFPM.png?alt=media&amp;token=60b5f093-67e5-4ef3-b3da-5ebe6824c99f" alt=""><figcaption></figcaption></figure>

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}


# Jira

Configure a [Webhook](https://docs.soda.io/soda-documentation/integrate/webhook) in Soda Cloud to connect to your Jira workspace.

In this guide, we will show how you can integrate Soda Cloud Incidents with Jira. After the integration is set up, then creating an incident in Soda will automatically trigger the creation of corresponding bug ticket in Jira. The Jira ticket will include information related to the incident created in Soda, including:

* The number and title of the Incident
* The description of the Incident
* The severity of the incident
* The status of the incident
* The user who reported the Incident
* A link to the Incident in Soda Cloud
* A link to the associated Check in Soda Cloud

A link to this Jira ticket will be sent back to Soda and displayed on the Incident page in the Integrations box. Any updates to the status of the Incident in Soda Cloud will trigger corresponding changes to the Status of the Jira ticket. Any updates to the status of the Jira ticket will trigger corresponding changes to the Status of the Incident in Soda Cloud.

In Jira, you can set up an Automation Rule that enables you to define what you want an incoming webhook to do, then provides you with a URL that you use in the URL field in the Soda Cloud integration setup.

This integration is built on two webhook events IncidentCreated and IncidentUpdated (Soda -> Jira; [Event payloads](https://docs.soda.io/apis/webhook-api#event-types-and-payloads)), as well as the Soda Cloud API endpoint for updating incidents (Jira -> Soda; [API](https://docs.soda.io/apis/rest-api)).

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FnFDn1tdRasbZZ7Og5q1c%2Fimage-1.avif?alt=media&amp;token=63577ff2-fea8-4f27-b19a-0dc2eb40845d" alt=""><figcaption></figcaption></figure>

### Create a Jira project for DQ tickets <a href="#create-a-jira-project-for-dq-tickets" id="create-a-jira-project-for-dq-tickets"></a>

In Jira, start by creating a new project dedicated to tracking data quality tickets. Navigate to the **Project settings** > **Work Items**, and make sure you have a bug type work item with the fields, as shown in the image below:

* Summary
* Description
* Assignee
* IncidentSeverity
* IncidentID
* IncidentURL
* CheckURL

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2F4Ajexvg0wzHRhTmGrbb2%2Fhttps___files.gitbook.com_v0_b_gitbook-x-prod.appspot.com_o_spaces_2FA2PmHkO5cBgeRPdiPPOG_2Fuploads_2FNUqKM4hhAXwIpeCJDDWX_2FScreenshot_202025-08-21_20at_203.10.27_E2_80_AFPM.avif?alt=media&amp;token=6eb1906d-1f1c-45e0-bffe-85d6c543f05a" alt=""><figcaption></figcaption></figure>

From the same page, next click the **Edit Workflow** button, and make sure your workflow includes the following statuses:

* Reported
* Investigating
* Fixing
* Resolved<br>

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FvTtp4fqUG1O5p7zglick%2Fimage.avif?alt=media&amp;token=73131433-ca9d-4289-81ce-a98398690466" alt=""><figcaption></figcaption></figure>

## Automation Rule (Inbound) <a href="#automation-rule-inbound" id="automation-rule-inbound"></a>

### Initialize the webhook-trigger <a href="#initialize-the-webhook-trigger" id="initialize-the-webhook-trigger"></a>

Here we will set up the automation in Jira so that when an Incident is created or updated in Soda, then a bug ticket will automatically be created or updated in Jira.

Navigate to `Project settings` > `Automation`, then click `Create rule` and, for the type of `New trigger`, select `Incoming webhook`.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FMOvkwoFoFHTF3iebtiyO%2Fimage-1.avif?alt=media&amp;token=f4b25f02-71cc-4195-abaa-f6ebf4749da1" alt=""><figcaption></figcaption></figure>

Under the `When: Incoming webhook trigger`, click `Add a component`, select `IF: Add a condition`, then smart values condition.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FbwBDwM5EYItrs12LlyOV%2Fimage.png?alt=media&amp;token=2515805b-cb84-4569-a228-df6b47133ed9" alt=""><figcaption></figcaption></figure>

What this means is that, **if** an incoming webhook has the `incidentCreated` event, then we will do something.

### Automatic creation of the Jira ticket <a href="#automatic-creation-of-the-jira-ticket" id="automatic-creation-of-the-jira-ticket"></a>

Next we will add another component: `THEN: Add an action`.

The action will be to **Create work item** and the **Issue Type** should be `Bug` and the **Project** should be our new project.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FS9PlKx7ChWfxcOevj7LU%2Fimage-1.png?alt=media&amp;token=993b6d7f-864e-4cad-8d8e-9bd18f6ef023" alt=""><figcaption></figcaption></figure>

Next we add some steps to fill out our ticket with extra information obtained from the webhook data.

We start by creating a branch rule to identify our ticket:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FBkebbzU4ZRpIUQyBukig%2Fimage.avif?alt=media&amp;token=6dcba744-a8da-4ffe-a2e7-867f07c15daf" alt=""><figcaption></figcaption></figure>

Then we Edit the ticket fields:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FUfg2C89DvypTdl3G0yyC%2Fimage-1.avif?alt=media&amp;token=737c6918-19ea-4ced-9aa4-7cc5a0680001" alt=""><figcaption></figcaption></figure>

Finally, the last step in our incident *creation* workflow is to send a post request back to Soda with a link to the issue in Jira:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2Fcu2Xegk1mcftW98cCqMn%2Fimage.avif?alt=media&amp;token=24a5bbf1-892d-43d3-a858-9b7694c30d38" alt=""><figcaption></figcaption></figure>

### Automatic updates to the Jira ticket <a href="#automatic-updates-to-the-jira-ticket" id="automatic-updates-to-the-jira-ticket"></a>

The remaining parts of this automation rule cover the scenarios where the status of the incident is updated in Soda, then we will detect this change and make the corresponding updates to the issue in Jira.

When the status changes to `Reported`:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FPDzB3KqDEuzJSLLRUU5U%2Fimage-1.avif?alt=media&amp;token=557f0753-2a4a-4bce-9cb7-fc9a8e016e5c" alt=""><figcaption></figcaption></figure>

The same logic is used for other status changes such as `Investigating` and `Fixing`.

In case the status changes to `Resolved`, our rule uses a similar logic, but with the additional step of adding resolution notes as a comment to the issue in Jira:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FvsOJlRR7WLAqwYLw0dia%2Fimage.avif?alt=media&amp;token=c432ac6b-7d8b-48d4-991a-61387a8cca8f" alt=""><figcaption></figcaption></figure>

Once you save/enable this new rule, then you can access a URL and secret that you will provide to Soda when setting up the new webhook integration.

After saving or enabling the rule, you can view details of the webhook trigger as shown below:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FHlGIxnZFb0F8A371JPZq%2Fimage.png?alt=media&amp;token=dd4578e1-16ed-40b0-8f52-6924e2d8082f" alt=""><figcaption></figcaption></figure>

### Define the Webhook integration in Soda <a href="#define-the-webhook-integration-in-soda" id="define-the-webhook-integration-in-soda"></a>

Next, you create a new webhook integration in Soda and provide the details from the webhook trigger above, as shown in the image below.

Paste the Webhook URL from Jira into the URL field in Soda and paste the Secret from Jira into a custom HTTP header called `X-Automation-Webhook-Token`.

Finally, in the `Define Scope` tab, make sure to select `Incidents - Triggered when users create or update incidents`.

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FGuj5Y6Y2OYKU63WpL9RT%2Fimage-1.png?alt=media&amp;token=4130d584-2b12-470d-a872-7985824333bb" alt=""><figcaption></figcaption></figure>

## Automation Rule (outbound) <a href="#automation-rule-outbound" id="automation-rule-outbound"></a>

We will set up a second automation rule in Jira so that when the status of the ticket changes in Jira, these changes are also reflected in Soda.

First, we set up the trigger for this automation to be when a Work item is transitioned:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FGT2RuTd4X1DPvcNGAVAZ%2Fimage.avif?alt=media&amp;token=49cf453d-dd3a-4f1e-916a-b91f18362abd" alt=""><figcaption></figcaption></figure>

Finally, we send a post request to the Soda Cloud API `incidents` endpoint, using information from our Jira ticket to update the severity and status of the corresponding incident in Soda:

<figure><img src="https://1123167021-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FA2PmHkO5cBgeRPdiPPOG%2Fuploads%2FuQb6MNBf9ivGitmLzkZ9%2Fimage.png?alt=media&amp;token=be6ab3c4-76c6-48e6-9bc8-5f7f73e2e4a0" alt=""><figcaption></figcaption></figure>

Note that the `Authorization` header value must be formatted like:

`Basic <base64_encoded_credentials>`. Base64-encoded credentials can be generated using Soda Cloud API keys in Python like so:

```bash
import base64
api_key_id = "your_api_key_id"
api_key_secret = "your_api_key_secret"

credentials = f"{api_key_id}:{api_key_secret}"
encoded_credentials = base64.b64encode(credentials.encode()).decode()
print(f"Basic {encoded_credentials}")
```

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/documentation-access-and-licensing).
{% endhint %}




---

[Next Page](https://docs.soda.io/llms-full.txt/1)

