Definition

The number of distinct non-NULL values in the monitored column. Highlights unexpected changes in cardinality (e.g., new user IDs, codes).

Source

data

Computation

COUNT(DISTINCT <column>

These monitors require executing queries against the data itself to surface usage and content recency patterns, for example:

• Most recent timestamp: the latest event or ingestion time across all rows

• Partition row count: the number of records within the current partition (e.g. today’s data)

Query-based monitors give you a window into data flow and freshness, helping detect lags in ingestion pipelines or staleness in source systems.

These monitors are derived directly from the data platform’s system metadata, without scanning row-level values. They surface structural signals, such as:

• : when the dataset was last updated

• : any alterations to the schema

• : the overall number of records in the dataset

Definition

The maximum value within a specific column.

Source

data (any datatype with an order defined for max/min)

Definition

The difference in total row count between the current scan and the immediately preceding scan (current_count – previous_count).

Source

Definition

The standard deviation of the values within a specific column.

Source

data (numeric)

Text metrics help catch formatting issues, truncated values, or unexpectedly long/free-form entries.

With Soda, it's possible to assess the character‐length properties of string columns:

Definition

Quartiles divide all non-NULL values in the column within the latest partition into four equal parts based on value: Q3 represents the 75th percentile.

Definition

The total number of rows in the dataset at scan time.

Source

metadata

Definition

Quartiles divide all non-NULL values in the column within the latest partition into four equal parts based on value: Q1 represents the 25th percentile.

Timestamp metrics highlight recency and time‐based anomalies, which is crucial for validating timeliness in event streams and incremental loads:

Definition

Quartiles divide all non-NULL values in the column within the latest partition into four equal parts based on value: Q2 represents the 50th percentile, that is, the median.

Definition

The minimum value within a specific column.

Source

data (any datatype with an order defined for max/min)

Definition

The number of rows in the most recent time partition at scan time (e.g. all rows where partition_col = current_partition).

Source

Definition

Interval between scan time and the maximum value in a date/time/time-stamp column (within the partition). Supported only on date/datetime/time columns.

Source

data (timestamp)

This section introduces the key features and workflows in Soda for managing data quality issues and reporting.

Learn how to find datasets and checks, navigate dashboards, and understand check results.

You’ll also learn how to set up notifications to stay informed and build custom dashboards using tools like Power BI or Tableau.

Numeric metrics capture central tendency and dispersion in numerical columns, such as:

• Mean ()

• &

Soda helps data teams make sure their data can be trusted. It makes it easy to find, understand, and fix problems in the data.

You can use Soda to:

• Monitor production data with automated, ML-powered observability that spots unexpected changes without needing to define every rule up front.

• Define data contracts, making expectations explicit and enabling producers and consumers to collaborate on reliable data at the source.

• Test data earlier in the pipeline, as part of CI/CD workflows or during development, to prevent bad data from reaching production.

Soda helps teams to start right and automatically detect anomalies in metrics that have already happened. And shift left to prevent issues from happening again with collaborative data contracts.

Soda v4 vs v3

The new version of Soda has transformed the software into a full data-quality platform by layering on:

• End-to-end data observability:

• Collaborative data contracts:

This marks the shift from a CLI-centric checks engine toward a unified, observability-driven data quality platform with a refined, three-tier Core + Agent + Cloud architecture, built-in contracts, orchestration, and deep integrations.

to learn more about Soda's capabilities.

What is data quality?

Data quality refers to how well a dataset meets the expectations of completeness, accuracy, timeliness, uniqueness, and consistency. Good data supports business goals, drives confident decision-making, and is the base for great data products.

Poor data quality causes failed pipelines, incorrect reports, and broken AI models. Managing data quality means proactively validating assumptions and reactively monitoring for drift or degradation.

Soda helps you answer questions like:

• Is the data fresh and complete?

• Are there unexpected values or duplicates?

• Did values shift outside of expected ranges?

• Are schema or contract changes causing breakage?

Key Concepts

Data Observability

Data observability is a reactive approach to monitoring data in production and catching unexpected issues as they emerge. It helps answer the question: What is happening with my data right now, and how is that changing over time?

Use data observability to:

• Detect anomalies in data quality metrics such as freshness, row counts, null values or custom ones

• Monitor metric trends and seasonality

• Identify late-arriving or missing records

• Get alerted when values deviate from historical norms

Data Testing

Data testing is a proactive approach that validates known expectations about your data during development, deployment, or transformation. It helps you catch issues before they reach production, break reports, or impact downstream systems.

Use data testing to:

• Align on what “good data” looks like through data contracts

• Verify that your data meets those expectations, including schema, values, and transformations

• Test data at every step of the pipeline to prevent bad data from reaching downstream systems

• Integrate with CI/CD workflows for continuous quality checks during development

Data Contracts

Data contracts define what a dataset should look like, including its schema, data types, value ranges, and other constraints. They establish a shared agreement between data producers and consumers about what’s expected and what must be upheld.

Both testing and observability play a role in upholding data contracts:

• Testing validates that data meets the contract during development, pipeline execution, and on schedule.

• Observability monitors contract adherence in production and detects unexpected issues.

Data Observability vs Data Testing

While data testing and observability are different in when and how they operate, they work best together as a unified strategy.

Together, they enable end-to-end data quality management: testing prevents problems, and observability detects those that escape prevention. At the same time, observability can help prioritize which issues to address and shift left to resolve them upstream.

Data quality at scale across the enterprise

Divide and conquer

Managing data quality across hundreds or thousands of datasets requires a scalable, federated approach. Soda enables this through:

• Metadata-driven observability that adapts checks to each dataset's structure and context.

• Role-based collaboration so teams can take ownership of the data they know best.

• An interface for both engineering and business users, enabling collaboration through code, UI, or APIs, depending on user preference and role.

• Integration with existing tools and workflows, such as data catalogs and incident management systems.

Data quality as a team sport

Reliable data depends on collaboration across roles:

• Data engineers embed tests and monitor pipelines to catch issues early.

• Data producers and consumers align on expectations through data contracts.

• Data consumers report issues and collaborate with producers to interpret metrics and resolve problems.

• Governance teams define and enforce data quality standards.

Soda Cloud acts as the shared workspace where these roles collaborate, triage incidents, and resolve issues.

Deployment options

Soda offers three deployment models, depending on your infrastructure and data privacy needs.

Read more about

Supported data sources and integrations

Soda integrates with the modern data stack:

• Data warehouses and databases: Databricks, Snowflake, BigQuery, Redshift, PostgreSQL, MySQL, Spark, Presto, DuckDB, and more.

• Orchestration platforms: Airflow, Dagster, Prefect, Azure Data Factory.

• Metadata tools: Atlan, Alation, Collibra, data.world, Zeenea.

• Cloud providers: AWS, Google Cloud, Azure.

What’s next?

• To get started with Soda, check out the end-to-end guide.

Community & Support

Need help or want to contribute?

• Join our Slack Community: ​

• Browse GitHub Discussions: ​

Still have questions? Use the search bar above or reach out through our community channels for additional help.

Metrics that support all data types are foundational checks that apply to any column regardless of its data type:

• Count of non-NULL values

• Duplicate percentage

• Minimum and maximum values

• of distinct entries

These metrics form the backbone of data completeness and consistency monitoring, ensuring every column meets basic quality expectations.

Definition

Arithmetic mean of all non-NULL values in the column, computed per partition.

Source

data (numeric)

Computation

Using the built in AVG(column_name) function of every database.

Definition

The maximum (maxLength) of non-NULL string values in the column.

Source

data (value length in characters)

Computation

For each value, Soda encapsulates the length in the aggregation metric: MAX(LENGTH(column))

Define, version, and test contracts as code

For teams that manage data like software, Git-managed data contracts offer a code-first way to define and enforce data quality expectations.

In this model, contracts are written in YAML and stored in your Git repository, right alongside your data models, transformation logic, and CI/CD workflows. You write, version, test, and promote contracts just like any other code artifact.

This approach gives engineers full control, reproducibility, and integration into development pipelines. And with the right setup, you can still collaborate with non-technical users via Soda Cloud and even sync UI-authored changes into Git using our future proposal workflow.

Why Git-managed?

• Full version control Track every change, roll back when needed, and manage contracts with the same discipline as application code.

• Code-first workflow Keep contracts close to your data models and transformations for better alignment, automation, and traceability.

• CI/CD integration Run contract verifications in your existing pipelines; on every commit, PR, or deployment.

• Team governance

If you're already managing your data infrastructure in Git, Git-managed contracts are the natural extension for bringing data quality under control without adding friction or silos.

In the next sections, we’ll walk you through how to set up, author, and run Git-managed contracts using the Soda CLI.

Cloud-managed Data Contracts let you define and manage expectations for your data directly in the Soda Cloud UI.

This approach is perfect for data analysts, product owners, and business stakeholders who know what “good data” looks like but prefer intuitive tools over code. It’s also ideal for teams that want to move fast, collaborate visually, and integrate seamlessly with engineering workflows when needed.

With Soda Cloud, you can browse datasets, add quality rules, test and publish contracts, and set up scheduled or on-demand verification. All from your browser.

Why Cloud-managed?

• Faster time to value – no setup required

• Accessible to everyone – empower domain experts, not just engineers

• Built for collaboration – share, comment, and propose changes in a shared UI

• Easily operationalized – schedule tests and trigger verifications programmatically

Cloud-managed contracts are a powerful way to bring your organization together around trusted data.

Prerequisites

Before creating contracts in Soda Cloud, make sure:

• You have a Soda Cloud account

• You have access to an organization in Soda Cloud

• You have connected at least one data source via a Soda Agent

Definition

Number of missing values relative to the number of rows in the partition, expressed as percentage → (number of null or missing values in column ÷ total rows in partition) × 100

Source

data

Computation

1 - (count(column) / count(*)) × 100

The Soda Agent allows you to securely scan your data sources for quality issues directly from Soda Cloud. It can be self-hosted or Soda-hosted, depending on your deployment preferences. The self-hosted option allows for a more custom and secure deployment, while the Soda-hosted agent is easier to start with. Learn more about Deployment options

You can deploy a self-hosted agent in the infrastructure of your choice:

• Kubernetes cluster

• Amazon EKS

• Azure AKS

• Google GKE

Definition

The average (avgLength) of non-NULL string values in the column.

Source

data (value length in characters)

Computation

For each value, Soda encapsulates the length in the aggregation metric: AVG(LENGTH(column))

Definition

The total number of non-NULL values in the monitored column. Useful for identifying unexpected drops or spikes in data completeness.

Source

data

Computation

COUNT(<column>)

When you access a monitor (from Metric Monitoring) or a check (from the Contract), Soda Cloud provides a time series view that shows how the monitored metric or check result evolves. This helps you explore the history of data quality issues, spot trends, and understand changes in your data.

Diagnostics

For certain types of checks and monitors, additional diagnostic information is also available for each monitor or check results to help you investigate issues in more detail.

For example:

• Schema Checks: View a side-by-side comparison of the actual vs. expected schema to identify differences.

• Missing, Duplicate, or Invalid Checks: See the percentage of failed rows vs. passing rows to understand the scale and impact of the issue.

This view helps you drill down into specific data issues, explore context, and take informed action.

Definition

Detects anomalies in the total (sum) of all non-NULL values in a given column over the latest partition. It flags unexpected increases or decreases in the aggregate amount.

Source

data (numeric)

Computation

SUM(<column>)

Definition

The variance of the values within a specific column.

Source

data (numeric)

Computation

Uses the SQL standard for all databases VAR_SAMP(), which is a sampling based method.

Definition

The interval between scan time and the maximum timestamp in the partition column (within the latest partition).

Source

data (time partition)

Computation

Via MAX(column) for any time related column.

No sampling used.

Definition

The minimum (minLength) of non-NULL string values in the column.

Source

data (value length in characters)

Computation

For each value, Soda encapsulates the length in the aggregation metric: AVG(LENGTH(column))

Definition

The count of schema alterations (column additions, removals, or data-type changes) detected since the previous scan. Any schema change is treated as an anomaly.

Source

metadata

Computation

No sampling used. The value is calculated through the difference of two full table definitions.

As soon as a data source is connected, the metric monitoring dashboard is available and will have historical information. Soda establishes a statistical baseline for each metric and continually compares new scan results against that baseline, flagging anomalies according to the sensitivity, exclusions, and threshold strategy you’ve configured.

What are Metric Monitors?

Metric monitors are the foundation of data observability in Soda. Monitors track data quality metrics over time and leverage historical values for analysis. Soda automatically collects these metrics and examines how they evolve over time through a to identify when metrics deviate from expected patterns and trigger alerts. These deviations are surfaced and recorded in the Metric Monitors

Definition

The elapsed interval between the scan time and the timestamp of the most recent change to the database. This includes any change to the data (inserts, updates, deletes) as well as any change to the schema.

Source

Oracle

• Historical backfilling: not possible.

• Row count: metadata row counts are calculated via count(*)

The Organization and Admin Settings in Soda Cloud provide a centralized interface for managing your organization’s configuration, roles, user access, and integrations. From setting your organization’s name to defining global roles, managing user groups, and enabling the Soda-hosted Agent, these settings help you tailor Soda Cloud to your team’s needs and governance policies.

To access the settings, click on your avatar on the top right, and then click Organization Settings.

Only users with the Manage Organization Settings global role can access and modify these settings.

Definition

Percentage of all duplicate non-NULL records in a column.

For example: duplicate_percent(id)

Before you can define, test, or verify Git-managed data contracts, you need to install the Soda CLI and configure your environment.

This setup gives you full control over your contracts, letting you version them in Git, execute them locally or remotely, and integrate them into your CI/CD pipelines.

Install the Soda Core Python Package

Install Soda Core using pip:

If you need authenticated access, follow the to set up your environment and install the necessary Soda extensions.

Connect to Soda Cloud (optional)

If you want to interact with Soda Cloud to publish contracts and view verification results, or to use Soda Agent, you’ll need to connect the CLI to your Soda Cloud account.

Don’t have an account? to get started.

Create the config file

This generates a basic Soda Cloud configuration file.

Add your API keys

Open sc.yml and fill in your API key and organization details.

Learn more about how to generate keys:

Test the connection

This ensures the CLI can authenticate and communicate with Soda Cloud.

Configure your data source

To verify a contract, Soda needs to know how to connect to your data source. You have two options:

Connect with Soda Core

If you prefer to define your own connection locally (or aren’t using a Soda Agent), you can create a data source config file for Soda Core.

Install the required package for your data source. For example, for PostgreSQL:

See the for supported packages and configurations.

Create the config file

Open ds.yml and provide the necessary credentials.

For example with PostgreSQL:

Refer to the for the configurations of each data source type.

Test the connection:

Use an existing Data Source via Soda Agent

If your data source is already connected to Soda Cloud using a Soda Agent (hosted or self-hosted), you can reuse that connection without managing credentials or configs locally.

You just need to ensure you have set up the connection with Soda Cloud.

Choose the method that best fits your setup:

Use Soda Agent for a centralized, cloud-managed connection, or local configuration if you want full control within your environment.

The Soda Agent is a Helm chart that you deploy on a Kubernetes cluster and connect to your Soda Cloud account using API keys.

To take advantage of new or improved features and functionality in the Soda Agent, including new features in the Soda Library, you can upgrade your agent when a new version becomes available in ArtifactHub.io.

Note that there is no downtime associated with the exercise of upgrading a self-hosted Soda Agent. Because Soda does not define the .spec.strategy in the deployment manifest of the Soda Agent Helm chart, Kubernetes uses the default RollingUpdate to upgrade; refer to Kubernetes documentation .

1. If you regularly access multiple clusters, you must ensure that are first accessing the cluster which contains your deployed Soda Agent. Use the following command to determine which cluster you are accessing.

If you must switch contexts to access a different cluster, copy the name of cluster you wish to use, then run the following command.

1. To upgrade the agent, you must know the values for:

• namespace - the namespace you created, and into which you deployed the Soda Agent

• release - the name of the instance of a helm chart that is running in your Kubernetes cluster

• API keys - the values Soda Cloud created which you used to run the agent application in the cluster Access the first two values by running the following command.

Output:

1. Access the API key values by running the following command, replacing the placeholder values with your own details.

From the output above, the command to use is:

1. Use the following command to search ArifactHub for the most recent version of the Soda Agent Helm chart.

1. Use the following command to upgrade the Helm repository.

1. Upgrade the Soda Agent Helm chart. The value for the chart argument can be a chart reference such as example/agent, a path to a chart directory, a packaged chart, or a URL. To upgrade the agent, Soda uses a chart reference: soda-agent/soda-agent.

From the output above, the command to use would be:

OR, if you use a values YAML file,

The Datasets page displays all datasets that have been onboarded into Soda Cloud—either through publishing a contract or via the onboarding process: Onboard datasets on Soda Cloud .

It provides a quick overview of each dataset’s health, showing at a glance if a dataset has issues, how many checks from its contract are failing, how many anomalies were detected through metric monitoring, and when the last scan was executed.

You can filter datasets by properties like data source, owners, arrival time, attributes, or flags such as failures or anomalies. Use the search bar to quickly find a specific dataset by name, and sort the list by name, creation time, or data quality status.

Learn more about custom attributes: Dataset Attributes & Responsibilities

You can also sort the datasets list by name, creation date, check failures, or anomalies to prioritize your focus.

Customize your datasets' view

You can tailor the Datasets view to focus on the areas that matter most to you:

1. Use the filter options to narrow down the view

2. Click the Save Dashboard button to store your current filter configuration as a collection.

1. Enter a name for the collection and click Save

1. Once saved, your collection will be available in the dropdown at the top right of the dashboard. Simply select it to switch views.

1. Use the context menu next to the collection name to:

• Delete the collection if it’s no longer needed.

• Share the collection with others in your organization.

Overview

Diagnostics Warehouse stores all Soda scan details, failed records, and historical data quality issues directly in your data warehouse of choice, safely and securely. Nothing is stored outside. This gives you the ability to run diagnostics, resolve issues, and see exactly why problems happen. You can go as deep as you need, from a single record to a full dataset.

Each time a Soda scan runs, Diagnostics Warehouse stores failed rows together with check and scan results, and related metadata attributes. With that information, data teams can quickly diagnose and resolve issues at both dataset and row level. Additionally, Soda's Diagnostics Warehouse makes it easier for teams to build on top of Soda's outputs to set up operational workflows, and connect to BI tools you already know and trust.

Features & capabilities

• Full diagnostic information in one place, including attributes.

• Transparency for all: replace black-box runs with auditable facts and keep an immutable, queryable history of what was checked, when, how long it took, what failed, and why.

• Faster root-cause analysis: jump from a failed check to the exact failed rows, affected datasets/columns, and prior history to see if it’s a one-off issue or a pattern.

Security & governance

• Data minimization: Diagnostics Warehouse stores metadata about runs and checks and, for row-level checks, it only stores failed rows when the option is enabled.

• Warehouse residency: Diagnostics are not stored in Soda. They live in your analytics warehouse, respecting your access controls, encryption, and audit trails.

Get started

1. Enable Diagnostics Warehouse in your Soda data source settings.

2. Grant the service identity permission to create and write to the Diagnostics Warehouse schema in your warehouse.

3. Run your checks; Diagnostics Warehouse tables populate automatically.

4. Query your warehouse and connect to your BI tools to start exploring.

Next: to enable Diagnostics Warehouse in your organization, reach out to Soda at .

The Checks page displays all checks defined in a data contract and tracked in Soda Cloud. It provides a quick overview of check health across datasets, allowing you to create custom groupings by applying filters such as data source, dataset, owners, or status (pass, fail, warning). This helps you focus on specific areas or teams that matter most.

You can also review key details like the check type, the dataset it belongs to, and the time of the last scan. Use the search bar to quickly find a specific check by name, and sort the list by name, last run time, or check status.

You can filter checks by properties such as data source, dataset, owners, attributes, or status (pass, fail, warning). Use the search bar to quickly find a specific check by name.

Learn more about custom attributes: Check and dataset attributes

You can also sort the list by name, last run time, or check status.

Customize your check view

You can tailor the Checks view to focus on the areas that matter most to you:

1. Use the filter options to narrow down the view

2. Click the Save Dashboard button to store your current filter configuration as a collection.

1. Enter a name for the collection and click Save

1. Once saved, your collection will be available in the dropdown at the top right of the dashboard. Simply select it to switch views.

1. Use the context menu next to the collection name to:

• Delete the collection if it’s no longer needed.

• Share the collection with others in your organization.

The Dataset Page provides a detailed view of each dataset’s health and monitoring information in Soda Cloud. It includes several tabs to help you explore and manage data quality at the dataset level.

Checks tab

Displays the results of all checks defined in the dataset’s contract. Checks are grouped by column, with column-level checks nested under their respective columns. Columns with failed checks are automatically expanded so you can spot issues quickly. You can filter the view to show only failed checks and search for specific checks or columns by name for faster navigation and troubleshooting.

Learn more about Data Testing

Metric Monitoring tab

Shows the metrics that are actively monitored for the dataset, helping you track trends and detect anomalies over time.

Learn more about

Profiling tab

Provides an overview of the dataset’s structure, including column names, data types, distinct counts, and other statistics. You can search for a specific column by name to quickly locate and review its profiling details.

Incidents tab

Lists incidents related to the dataset, helping you track issues and collaborate on resolution. You can filter incidents based on criteria such as user lead, status, or severity to focus on the most important or urgent cases.

Learn more about

The Organization Dashboard provides a high-level overview of your data quality across datasets and checks in Soda Cloud. It shows key trends over time, such as the number of checks that are passing, failing, or in a warning state, helping you identify issues early.

You’ll also find key metrics, including:

• Scans in failed mode: Datasets that are currently blocked due to failing checks.

• Checks currently failing: Active checks that need attention.

• Overall Health Score: The number of failing checks out of the total number of checks

These insights allow you to quickly identify where action is needed.

Customize your dashboard

You can tailor the Organization Dashboard to focus on the areas that matter most to you:

1. Apply filters based on attributes: Use the filter options to narrow down the view by attributes

1. Click the Save Dashboard button to store your current filter configuration as a collection.

1. Enter a name for the collection and click Save

1. Once saved, your collection will be available in the dropdown at the top right of the dashboard. Simply select it to switch views.

1. Use the context menu next to the collection name to:

• Delete the collection if it’s no longer needed.

• Share the collection with others in your organization.

Activity section

The Activity section offers insights into how Soda is being used across your organization. It tracks adoption metrics, such as active users, active checks, active datasets, and the number of alerts in the last 90 days.

Custom Dashboards

In addition to the built-in dashboards in Soda Cloud, you can build custom dashboards tailored to your organization’s specific needs. By leveraging the Soda REST API, you can programmatically retrieve data quality metrics, check results, and incident details, and integrate them into external dashboarding tools such as Power BI, Tableau, or Looker.

This enables you to create tailored views and reports that align with your business logic and audience, ensuring stakeholders get the right insights in the tools they already use.

Learn more on

The General Settings page allows you to configure foundational settings for your organization in Soda Cloud. These settings impact how your organization operates and how users interact with the platform.

Organization Name

Set the name of your organization. This name appears throughout Soda Cloud, such as in dashboards, reports and notifications.

Allow "Login As"

Enable the Login As feature to allow the Soda Support team to log in as an admin within your organization. This can be useful when troubleshooting issues or providing assistance.

Enable or Disable the Soda-hosted Agent

You can choose whether to use the Soda-hosted Agent by enabling or disabling it in the Organization Settings:

• Toggle the Soda-hosted Agent option to enable or disable the agent for your organization.

• Disabling the agent prevents Soda Cloud from running scans or checks via the managed agent. You’ll need to use a self-hosted agent or Soda Core in your environment instead.

Profiling Data Collection

By default, the Soda-hosted Agent collects profiling information (such as column-level statistics and schema details) to support features like dataset discovery and monitoring in Soda Cloud.

You can choose to disable profiling if you prefer not to send profiling data to Soda Cloud: Toggle the Profiling Data Collection option to disable profiling for your organization.

This ensures that no profiling information is collected or pushed to Soda Cloud. Only check results and metadata necessary for contract validation will be processed.

Enable Data Source Secrets

Manage secure storage of secrets such as API keys, credentials or connection details. Secrets can be used in data source configurations, checks, and other automated processes.

For more information, see the

Once your contract is authored and published (or available locally), you can verify whether the actual data complies with the defined expectations. Soda provides two execution options:

• Soda Core – run verifications locally, typically in CI/CD pipelines or dev environments.

• Soda Agent – run verifications remotely using an agent deployed in your environment, triggered via Soda Cloud.

Both approaches support variable overrides, publishing results to Soda Cloud, and integration into automated workflows.

Learn more about Deployment options

Using Soda Core

Soda Core runs the verification locally, connecting to your data source using the defined data source configuration file.

This command:

• Connects to your database using the local config

• Loads the contract

• Runs all checks and returns a pass/fail result

With variable overrides

You can pass variables defined in the contract using the --set flag:

Learn about variables in Data Contract:

Publish results to Soda Cloud

To send verification results to Soda Cloud for visibility and reporting.

Add the flag --publish to the command.

Learn how to connect the CLI to Soda Cloud:

This is recommended if you want stakeholders to see the outcomes in Soda Cloud or include them in dashboards and alerting.

Using Soda Agent

Soda Agent executes verifications using data sources configured in Soda Cloud.

This setup:

• Runs verifications through the Soda Agent connected to your data source

• Fetches the published contract from Soda Cloud

• Returns the result locally in the CLI

With variable overrides

You can pass variables defined in the contract using the --set flag:

Learn about variables in Data Contract:

Publish results to Soda Cloud

You can also push results to Soda Cloud from the agent-based run.

Add the flag --publish to the command.

This is recommended if you want stakeholders to see the outcomes in the Soda Cloud or include them in dashboards and alerting.

Soda’s notification system helps you stay informed when data issues occur—whether it’s a failed check or an anomaly detected through metric monitoring. Notifications are dynamically dispatched using notification rules, allowing you to target alerts based on specific properties, attributes, or datasets.

How Notification Rules Work

Notification rules define when and to whom a notification is sent. Rules can be configured to match specific checks or anomalies, ensuring the right people are notified at the right time.

Creating a Notification Rule

To create a new notification rule:

1. Click on your profile in Soda Cloud and select Notification Rules from the menu.

1. Click New Rule.

1. Provide a name for the rule.

1. Define the Rule Scope

Checks:

• All checks: The rule applies to every check in your organization.

• Specific checks: Build custom rules by filtering on check properties, dataset properties, or attributes.

• Anomalies from Metric Monitoring: Select specific datasets where the rule applies.

1. Define the recipients (users, groups, or integrations like Slack, Teams, or webhooks).

1. ...and choose the alert type (only applicable for checks, not anomalies):

• Only failures

• Failures and warnings

• All statuses

1. Save to create the notification rule

Pausing Notification Rules

You can pause a notification rule at any time to temporarily disable alerts without deleting the rule.

When you delete the Soda Agent Helm chart from your cluster, you also delete all the agent resources on your cluster. However, if you wish to redeploy the previously-registered agent (using the same name), you need to specify the agent ID in your override values in your values YAML file.

1. In Soda Cloud, navigate to your avatar > Agents.

2. Click to select the agent you wish to redeploy, then copy the agent ID of the previously-registered agent from the URL. For example, in the following URL, the agent ID is the long UUID at the end. https://cloud.soda.io/agents/842feab3-snip-87eb-06d2813a72c1. Alternatively, if you use the base64 CLI tool, you can run the following command to obtain the agentID.

 kubectl get secret/soda-agent-id -n soda-agent --template={{.data.SODA_AGENT_ID}} | base64 --decode

1. Open your values.yml file, then add the id key:value pair under agent, using the agent ID you copied from the URL as the value.

1. To redeploy the agent, you need to provide the values for the API keys the agent uses to connect to Soda Cloud in the values YAML file. Access the values by running the following command, replacing the soda-agent values with your own details, then paste the values into your values YAML file.

Alternatively, if you use the base64 CLI tool, you can run the following commands to obtain the API key and API secret, respectively.

1. In the same directory in which the values.yml file exists, use the following command to install the Soda Agent helm chart.

1. Validate the Soda Agent deployment by running the following command:

Group By monitors enable you to track data quality metrics across specific segments of your dataset. Instead of monitoring a metric for a column as a whole, you can break it down per category (for example, per region, per school year, per status).

This functionality is especially valuable when:

• You want to detect anomalies at a more granular level, within each segment or category.

• You need visibility into how data quality differs across categories.

Incidents help you track, investigate, and resolve data quality issues when they occur. An incident is created when a data issue, such as a failed or warning check, has been confirmed and assigned to someone for resolution.

To create or update an incident, the user has to have "Manage Incidents" on the related dataset.

Creating an Incident

With Git-managed contracts, you define your expectations as code using YAML. This gives you full control over how your data is validated, and allows you to manage contracts just like any other code artifact: versioned, tested, and deployed via Git.

To learn all about the structure and supported features, refer to the full specification in the

The contract structure includes:

• Dataset and column structure

• Available check types (missing, invalid

Starting from version 1.2.0 all images required for the Soda Agent are distributed using a Soda-hosted image registry.

For more information, see .

Set up authentication for the Soda image registry

Once a Data Contract is published, the next step is to verify that the actual data matches the expectations you’ve defined. Soda offers several flexible ways to execute contract verifications, depending on your needs and technical setup.

Manual execution (from the Dataset Page)

You can manually verify a contract at any time from the dataset page in Soda Cloud.

Simply open the dataset and click Verify Contract. This will:

Attributes allow you to add descriptive metadata to your datasets and checks. This metadata can then be:

• Used for filtering in Soda Cloud, making it easier to search and organize datasets and checks based on specific criteria (e.g., business domain, sensitivity, criticality).

• Leveraged in reporting, enabling you to group datasets, track ownership, and monitor data quality across different categories or dimensions.

Adding meaningful attributes enhances discoverability, governance, and collaboration within Soda and its integrations.

Connection configuration reference

Install the following package:

Data source YAML

This reference hub includes detailed documentation for Soda’s key interfaces and configuration options, as well as information on Soda's architecture and specifics, including:

• Data Contract Language Reference – Author, validate, and manage data contracts using YAML-based definitions.

• CLI Command and Python Reference – Use Soda’s command-line interface to configure, run, and automate verification workflows.

• REST API Reference – Interact with Soda Cloud programmatically to manage datasets, run verifications, and retrieve results.

What is Data Testing?

Data testing is the practice of validating that your data meets the expectations you’ve defined for it before it reaches stakeholders, dashboards, or downstream systems. Just like software testing ensures your code behaves as intended, data testing safeguards the quality and reliability of your data.

At Soda, we see data testing as the foundation of data trust. Whether you’re verifying row counts, checking for missing or invalid values, or enforcing schema integrity, the goal is the same: catch issues early, reduce incidents, and keep your data consumers confident.

What is a Data Contract?

A Data Contract is a formal agreement between data producers and data consumers that defines what “good data” looks like. It sets expectations about schema, freshness, quality rules, and more, and makes those expectations explicit and testable.

With a data contract in place, producers commit to delivering data that meets certain standards. Consumers, in turn, can rely on that contract to build reports, models, or pipelines without second-guessing the data.

At Soda, Data Contracts are testable artifacts that can be authored, versioned, verified, and monitored, whether in code or in the UI. They’re the connective tissue between producers and consumers, aligning teams and eliminating ambiguity.

What is Contract Verification?

Defining a contract is only the first step. Verifying that your data actually meets the expectations is where the value is realized. Contract verification is the process of testing whether the data in your datasets aligns with the rules, thresholds, and schema defined in the contract.

At Soda, contract verification is fully automated. Whether triggered manually, on a schedule, or as part of your CI/CD pipelines, each verification run checks that:

• The schema matches the contract definition (columns, data types, structure)

• The data complies with checks like missing, duplicate, invalid values, and custom rules

This helps you catch issues early, ensure data quality over time, and build trust across your organization.

Authoring and managing Data Contracts: choose your style

Soda supports two complementary ways to author and manage data contracts. They are designed to fit the way your team works.

Cloud-managed Contracts (Soda Cloud UI)

If you’re a data analyst, product owner, or stakeholder, who prefers intuitive interfaces over code, Soda Cloud is the ideal workspace.

With the Soda Cloud UI, you can:

• Browse datasets and view profiling insights

• Define a contract with a no-code Editor

• Schedule and monitor contract verifications

• Collaborate with your team and publish contracts with a click

There’s no setup or YAML required, just fast, visual workflows that enable domain experts to contribute directly to data quality.

Git-managed Contracts (Soda Core CLI)

If you live in your terminal and manage your data pipelines as code, you’ll want to use Soda Core and the Soda CLI.

With this setup, you can:

• Define contracts in YAML

• Run contract verifications in CI/CD

• Push the contract and verification results to Soda Cloud for visibility

• Use Git as the source of truth for version control, collaboration, and reviews

This path offers full control, transparency, and seamless integration into your dev tooling.

Combine UI and Git for cross-team collaboration

Soda gives you the flexibility to blend both approaches. For example, non-technical users can define or adjust contracts visually in Soda Cloud for the datasets they manage, while engineers can use Git-managed contracts for the datasets they own.

This hybrid model enables collaboration across teams:

• Business users bring domain expertise directly into the contract

• Engineers maintain quality, consistency, and governance

• Each dataset follows the authoring method that best suits the team responsible for it

You can mix and match—using the UI for some contracts, and code for others—depending on your team's structure and preferences.

And even if Data Contracts are managed in Git, you can still involve non-technical users who can propose changes to a contract in the UI. These approved changes can be embedded into engineering workflows and synced to Git, ensuring that every update follows your organization’s quality and deployment standards.

Choose the model, or combination of models, that works best for your organization.

Soda Agent vs. Soda Core for execution

Once a contract is published, you’ll want to verify that the actual data meets the contract’s expectations. This verification can be done in two ways:

• Soda Agent is our managed runner that lives in your environment and connects to Soda Cloud. It handles contract verification, scheduling, and execution securely, without exposing your data externally. It is great for teams who want central management without maintaining CLI infrastructure.

• Soda Core is our open-source engine you can run anywhere: locally, in CI, or data pipeline. It’s lightweight, customizable, and great for teams that prefer full control or have strict environment constraints.

Both approaches support the same Data Contract logic. Choose the one that best fits your deployment model.

What is RAD?

Ensuring data quality can be difficult, especially when you need broad coverage quickly. Checks and column monitors are great for enforcing specific rules, but they take time to set up and require a deep understanding of your data. Soda’s Record-level Anomaly Detection (RAD) helps you get started fast, providing instant coverage across all columns, rows, and segments, without any configuration.

The algorithm analyzes historical data to build a clear picture of what normal data is supposed to look like. When incoming rows show unusual patterns, unexpected values, inconsistencies, or errors, RAD automatically triggers an alert and runs a Root Cause Analysis to pinpoint the issue. This provides quick, actionable insights while you work toward more detailed control using checks and column monitors.

Why use RAD?

1. Instant, broad coverage Monitor all columns, rows, and segments at once, detecting both known and unknown issues.

2. No configuration needed Get started immediately: no metrics or checks need to be defined. RAD automatically determines which columns to use.

3. One metric to track and alert on The Record-level Drift Score provides a single, explainable metric to monitor data health.

When to use RAD?

Order of operations to achieve the best coverage in the most efficient way:

1. Firstly, : always begin with high‑level monitors to verify if the right amount of data arrived on time and in the correct format. These require no configuration. They just need to be enabled

2. Secondly, RAD: apply Record-level Anomaly Detection to validate the actual content of the data. This step also requires no configuration (only enablement) and provides broad coverage across all columns and segments.

3. Next, : apply column‑level monitoring for specific use cases where the potential data quality issue and metric are known but expected to change over time. These should be minimized as they are prone to generating false alerts.

RAD requirements

For a dataset to be monitored by RAD, the following conditions must be met:

• Time partition column: the dataset must include a column that (for example, created_at).

• Primary key: the dataset must have a primary key to uniquely identify rows.

• Diagnostics Warehouse setup: a must be configured to store the daily sample, consisting of either primary keys or, ideally, a full copy of the sampled rows.

Next: to enable Record-level Anomaly Detection in your organization, reach out to Soda at .

The Soda↔Collibra optimized integration synchronizes data quality checks from Soda to Collibra, creating a unified view of your data quality metrics. The implementation is optimized for performance, reliability, and maintainability, with support for bi-directional ownership sync and advanced diagnostic metrics.

Key features

• High Performance: 3-5x faster execution through caching, batching, and parallel processing

• Custom Attribute Syncing: Flexible mapping of Soda check attributes to Collibra attributes for rich business context

• Ownership Synchronization: Bi-directional ownership sync between Collibra and Soda

• Deletion Synchronization: Automatically removes obsolete check assets from Collibra when checks are deleted in Soda

• Multiple Dimensions Support: Link checks to multiple data quality dimensions simultaneously

• Monitor Exclusion: Option to exclude Soda monitors from synchronization, focusing only on data quality checks

• Diagnostic Metrics Processing: Automatic extraction of diagnostic metrics from any Soda check type with intelligent fallbacks

• Robust Error Handling: Comprehensive retry logic and graceful error recovery

• Advanced Monitoring: Real-time metrics, performance tracking, and detailed reporting

• CLI Interface: Flexible command-line options for different use cases

• Backward Compatibility: Legacy test methods preserved for smooth migration

Quickstart

For technical details on how to configure the bi-directional Collibra integration, head to .

Prerequisites

• Python 3.10+ required

• Valid Soda Cloud API credentials

• Valid Collibra API credentials

• Properly configured Collibra asset types and relations

Basic Usage

Advanced Usage

How It Works

1. Optimized Dataset Processing

• Smart Filtering: Only processes datasets marked for synchronization

• Parallel Processing: Handles multiple operations concurrently

• Caching: Reduces API calls through intelligent caching

• Batch Operations: Groups similar operations for efficiency

2. Enhanced Check Processing

For each check in a dataset:

Asset Management

• Bulk Creation/Updates: Processes multiple assets simultaneously

• Duplicate Handling: Intelligent naming to avoid conflicts

• Status Tracking: Monitors creation vs. update operations

Attribute Processing

• Standard Attributes: Evaluation status, timestamps, definitions

• Diagnostic Metrics: Automatically extracts and calculates diagnostic metrics from check results

• Custom Attributes: Flexible mappings for business context (see )

• Batch Updates: Groups attribute operations for performance

Relationship Management

• Dimension Relations: Links checks to data quality dimensions

• Table/Column Relations: Creates appropriate asset relationships

• Error Recovery: Graceful handling of missing or ambiguous assets

3. Ownership Synchronization

• Collibra to Soda Sync: Automatically syncs dataset owners from Collibra to Soda

• User Mapping: Maps Collibra users to Soda users by email address

• Error Handling: Tracks missing users and synchronization failures

• Metrics Tracking: Monitors successful ownership transfers

4. Advanced Error Handling

• Retry Logic: Exponential backoff for transient failures

• Rate Limiting: Intelligent throttling to avoid API limits

• Error Aggregation: Collects and reports all issues at the end

• Graceful Degradation: Continues processing despite individual failures

Head to to learn how to integrate Collibra.

Profiling helps you understand the shape, quality, and uniqueness of your data before creating checks or metric monitors.

With profiling, you can explore metadata about your dataset, such as column names, data types, distinct counts, null counts, and summary statistics. You can also quickly search for specific columns to focus on the attributes that matter most to your analysis.

Profiling is useful for:

• Business teams: Gain a fast understanding of what’s inside a dataset, its completeness, and potential anomalies.

• Data teams: Validate schema, data types, and distributions before writing quality tests or transformations.

• Data owners: Quickly identify unexpected values, nulls, or structural changes in a dataset.

Key features

• Dataset overview: Displays a structured view of all columns, their types, and counts.

• Interactive navigation: Scroll through the dataset structure or jump directly to a column of interest.

• Search and filter: Quickly locate a column by name to review its profiling details.

• Column-level insights:

Enable & configure Profiling

1. Enable Profiling

You can enable Profiling during .

If you want to enable Profiling on an existing dataset, follow the next steps:

1. Click on Datasets > The dataset of your choosing

2. Navigate to the Columns tab in the dataset view

3. Click on Update Profiling Configuration

2. Configure Profiling

Once Profiling has been enabled, you can configure it to adapt to your organization's needs.

1. Choose a Profiling schedule

Profiling happens every 24 hours. Choose a UTC time from the dropdown menu to pick a specific hour when the scan will be scheduled.

1. Choose a Profiling strategy

   • Use sampling: To perform Profiling, Soda will use a sample of up to 1 million rows from the dataset.

   • Use a time window: To perform Profiling, Soda will use data present in a 30-day time window, based on the dataset time-partition column.

1. Click on Finish

Now, Profiling will be scheduled.

Disable Profiling

Disable column profiling at the organization level

If you wish to disable column profiling at the organization level, you must possess Admin privileges in your Soda Cloud account. Once confirmed, follow these steps:

1. Navigate to your avatar.

2. Click on Organization settings.

3. Uncheck the box labeled Allow Soda to collect column profile information.

How it works

When you open Profiling for a dataset:

1. Soda runs a lightweight scan of the dataset’s metadata and a sample of the data (depending on configuration).

2. It calculates summary statistics for each column.

3. Results are displayed in the Profiling view for exploration.

Key considerations

• Soda can only profile columns that contain NUMBERS or TEXT type data; it cannot profile columns that contain TIMESTAMP data except to create a freshness check for the anomaly dashboard.

• Soda performs the Discover datasets and Profile datasets actions independently, relative to each other. If you define exclude or include rules in the Discover tab, the Profile configuration does not inherit the Discover rules. For example, if, for Discover, you exclude all datasets that begin with staging_, then configure Profile to include all datasets, Soda discovers and profiles all datasets.

Next Steps

After reviewing profiling results, you can:

• Create tests based on profiling insights (e.g., "column should not have nulls").

• Set up monitors to track data quality over time.

• Export profiling information to support documentation and governance processes.

Configure Soda Cloud to connect your account to Slack so that you can:

• Send Notificationsfor failed or warning check results to Slack channels

• Start conversations to track and resolve data quality Incidentswith Slack channels

Configure a Slack integration

1. In Soda Cloud, navigate to your avatar > Organization Settings, then navigate to the Integrations tab and click the + icon to add a new integration.

2. Choose Slack and proceed

1. Follow the guided steps to authorize Soda Cloud to connect to your Slack workspace. If necessary, contact your organization’s Slack Administrator to approve the integration with Soda Cloud.

Configuration tab: select the public channels to which Soda can post messages; Soda cannot post to private channels.

Note that Soda caches the response from the Slack API, refreshing it hourly. If you created a new public channel in Slack to use for your integration with Soda, be aware that the new channel may not appear in the Configuration tab in Soda until the hourly Slack API refresh is complete.

Scope tab: select the Soda features (alert notifications and/or incidents) that can access the Slack integration.

About integration scopes

Integration for alert notifications

You can use this integration to enable Soda Cloud to send alert notifications to a Slack channel to notify your team of warn and fail check results.

With such an integration, Soda Cloud enables users to select a Slack channel as the destination for an alert notification of an individual check or checks that form a part of an agreement, or multiple checks.

To send notifications that apply to multiple checks, see

Integration for Soda Cloud incidents

You can use this integration to notify your team when a new incident has been created in Soda Cloud. With such an integration, Soda Cloud displays an external link to an incident-specific Slack channel in the Incident Details.

Selecting the right scan time is essential for accurate data monitoring and reliable metric collection. Scans that occur too early may run before the data has been fully loaded into the database, leading to false positives or misleading results. This guide outlines how to determine the best scan time based on your data load patterns and operational needs.

Scan frequency

Scans can be scheduled to occur from hourly to weekly. The time jumps are meant to fit into a 24-hour cycle that matches hourly/daily seasonalities related to how humans organize their day. Metric Monitoring can happen every:

Test a contract on a sample

When testing a data contract, Soda allows you to run contract validation on a sample of your dataset instead of the full data. This feature helps you quickly and cost-efficiently verify that your contract runs correctly before executing full scans.

What has changed?

As of July 2025, the container images required for the self-hosted Soda agent will be distributed using private registries, hosted by Soda.

EU cloud customers will use the EU registry located at registry.cloud.soda.io. US cloud customers will use the US registry located at registry.us.soda.io.

The images currently distributed through Docker Hub will stay available there. New releases will only be available in the Soda-hosted registries.

Existing or new Soda cloud API keys can be used to authenticate to the Soda-hosted registries. Starting from version

Soda offers flexible deployment options to suit your team’s infrastructure, scale, and security needs. Whether you want to embed Soda directly into your pipelines, use a centrally managed deployment, or rely on Soda’s fully-hosted solution, there’s an option for you.

This guide provides an overview of the three main deployment options: Soda Python Libraries, Soda-hosted Soda Agent, and Self-hosted Soda Agent, to help you choose the right setup for your organization.

Overview of Deployment Options

Custom SQL Monitors enable you to define monitoring logic using your own SQL queries. This is ideal when built-in Soda metrics or anomaly checks don’t meet your needs; for example, when you must aggregate data across multiple tables, compute ratios, or detect anomalies in grouped datasets.

A Custom SQL Monitor runs your SQL query against your connected data source and evaluates its results on a schedule, just like any other Soda monitor.

This feature can be used to:

Connection configuration reference

Install the following package:

Data source YAML