Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Add Soda to a Databricks notebook

Verify a Soda data contract on a Spark DataFrame from a Databricks notebook, and publish the results to Soda Cloud.

This guide shows how to install Soda in a Databricks notebook and verify a data contract on a Spark DataFrame. Soda runs the checks with the notebook's own Spark session, so it reads the same data the notebook does and needs no separate connection to Databricks. The results appear in the notebook output and in Soda Cloud.

Use the example as a starting point and adapt the dataset and checks to your own data.

Example notebook

A data engineer loads AdventureWorks customer data from the Unity Catalog table main.adventureworks.dim_customer into a DataFrame. Before the notebook uses the data, it verifies a contract that checks customer keys, last names, email addresses, phone numbers, and how recent the data is.

The notebook and the Soda files sit in the same workspace folder:

customers/
├── verify_dim_customer
└── soda/
    ├── sc_config.yml
    └── contracts/
        └── dim_customer.yml

Prerequisites

  • A Databricks workspace with Unity Catalog, and a notebook attached to compute that runs Databricks Runtime 14.0 or later

  • The Databricks CLI, authenticated to the workspace, to create a secret scope

  • A Soda Cloud account and an API key, to publish results. See Generate API keys. To publish results for a new dataset, you need the global permission Create new datasets and data sources with Soda Library.

Store the API keys in a secret scope

Store the Soda Cloud API key in a Databricks secret scope, so the notebook doesn't contain it. Create a scope named soda and add the key ID and the key secret to it:

Each put-secret command opens an editor where you enter the value.

For secret scopes and access control, see Secret management in the Databricks documentation.

Add the Soda files

In the workspace folder of the notebook, create a soda folder. Add a Soda Cloud configuration file to it. Use cloud.us.soda.io or cloud.soda.io, depending on your Soda region. The values in ${env.…} are read from environment variables, which the notebook sets from the secret scope.

Add the contract in a contracts folder. The dataset name starts with the data source name that the notebook gives Soda, spark_df, followed by the name of the temporary view that holds the DataFrame, dim_customer.

Soda compares the column names and data types with the view, so the schema check fails if a column is added, removed, or changes type. The freshness check fails if the most recent date_first_purchase is 7 days old or more.

To verify a table instead of a DataFrame, name it with its catalog and schema in the dataset name, for example spark_df/main/adventureworks/dim_customer.

For every check type and threshold option, see the Contract Language reference.

Coming from Soda v3? The named format email has no contract equivalent, so this contract checks email addresses with a regular expression. For the rest of the syntax changes, see Migrate from v3 to v4.

Install Soda in the notebook

In the first cell of the notebook, install the Soda package for Spark DataFrames:

%pip installs the package for this notebook only. In the second cell, restart Python so the notebook can import the new packages:

For serverless compute, Databricks recommends a pinned version, such as "soda-sparkdf==4.27.0". See Notebook-scoped Python libraries in the Databricks documentation.

If the import fails with a validation error or cannot import name 'sql' from 'databricks', see Troubleshoot in the Spark DataFrame reference.

Verify the contract

In the third cell, add the code that loads the data and verifies the contract:

A few details in the cell:

  • dbutils.secrets.get reads the API key from the secret scope, and the cell passes it to Soda through the environment variables that sc_config.yml refers to. Databricks redacts secret values in the notebook output.

  • createOrReplaceTempView makes the DataFrame available to Soda under the name in the contract. Any DataFrame works, whether the notebook reads it from a table, from files, or builds it in a transformation.

  • SparkDataFrameDataSource.from_existing_session runs the checks with the notebook's spark session. Its name is the first part of the dataset name in the contract.

  • The file paths are relative to the notebook's folder, which is the working directory in Databricks Runtime 14.0 and later. See What is the default current working directory? in the Databricks documentation.

  • result.is_ok is False when a check fails or the verification errors. The cell then raises an error, so nothing that runs after it uses the data.

For the result object and its attributes, see Python API.

Run the notebook

Run all cells. The third cell prints a table with the result of each check. In this example all seven checks pass.

When a check fails, the cell raises an error. For example, a customer with the email address ruben35.adventure-works.com fails the check Email addresses are valid, and the cell stops with this error:

The results are also published to Soda Cloud. Open Datasets, select dim_customer, and open the Checks tab to see the latest result of each check and its result history across runs.

The Checks tab of the dim_customer dataset in Soda Cloud, with the check Email addresses are valid failed
The dim_customer results in Soda Cloud

To get notified when a check fails, see Notifications.

Next steps

Last updated

Was this helpful?