Add Soda to a Databricks notebook
Verify a Soda data contract on a Spark DataFrame from a Databricks notebook, and publish the results to Soda Cloud.
This guide shows how to install Soda in a Databricks notebook and verify a data contract on a Spark DataFrame. Soda runs the checks with the notebook's own Spark session, so it reads the same data the notebook does and needs no separate connection to Databricks. The results appear in the notebook output and in Soda Cloud.
Use the example as a starting point and adapt the dataset and checks to your own data.
Example notebook
A data engineer loads AdventureWorks customer data from the Unity Catalog table main.adventureworks.dim_customer into a DataFrame. Before the notebook uses the data, it verifies a contract that checks customer keys, last names, email addresses, phone numbers, and how recent the data is.
The notebook and the Soda files sit in the same workspace folder:
customers/
├── verify_dim_customer
└── soda/
├── sc_config.yml
└── contracts/
└── dim_customer.ymlPrerequisites
A Databricks workspace with Unity Catalog, and a notebook attached to compute that runs Databricks Runtime 14.0 or later
The Databricks CLI, authenticated to the workspace, to create a secret scope
A Soda Cloud account and an API key, to publish results. See Generate API keys. To publish results for a new dataset, you need the global permission Create new datasets and data sources with Soda Library.
Store the API keys in a secret scope
Store the Soda Cloud API key in a Databricks secret scope, so the notebook doesn't contain it. Create a scope named soda and add the key ID and the key secret to it:
Each put-secret command opens an editor where you enter the value.
For secret scopes and access control, see Secret management in the Databricks documentation.
Add the Soda files
In the workspace folder of the notebook, create a soda folder. Add a Soda Cloud configuration file to it. Use cloud.us.soda.io or cloud.soda.io, depending on your Soda region. The values in ${env.…} are read from environment variables, which the notebook sets from the secret scope.
Add the contract in a contracts folder. The dataset name starts with the data source name that the notebook gives Soda, spark_df, followed by the name of the temporary view that holds the DataFrame, dim_customer.
Soda compares the column names and data types with the view, so the schema check fails if a column is added, removed, or changes type. The freshness check fails if the most recent date_first_purchase is 7 days old or more.
To verify a table instead of a DataFrame, name it with its catalog and schema in the dataset name, for example spark_df/main/adventureworks/dim_customer.
For every check type and threshold option, see the Contract Language reference.
Install Soda in the notebook
In the first cell of the notebook, install the Soda package for Spark DataFrames:
%pip installs the package for this notebook only. In the second cell, restart Python so the notebook can import the new packages:
For serverless compute, Databricks recommends a pinned version, such as "soda-sparkdf==4.27.0". See Notebook-scoped Python libraries in the Databricks documentation.
If the import fails with a validation error or
cannot import name 'sql' from 'databricks', see Troubleshoot in the Spark DataFrame reference.
Verify the contract
In the third cell, add the code that loads the data and verifies the contract:
A few details in the cell:
dbutils.secrets.getreads the API key from the secret scope, and the cell passes it to Soda through the environment variables thatsc_config.ymlrefers to. Databricks redacts secret values in the notebook output.createOrReplaceTempViewmakes the DataFrame available to Soda under the name in the contract. Any DataFrame works, whether the notebook reads it from a table, from files, or builds it in a transformation.SparkDataFrameDataSource.from_existing_sessionruns the checks with the notebook'ssparksession. Itsnameis the first part of the dataset name in the contract.The file paths are relative to the notebook's folder, which is the working directory in Databricks Runtime 14.0 and later. See What is the default current working directory? in the Databricks documentation.
result.is_okisFalsewhen a check fails or the verification errors. The cell then raises an error, so nothing that runs after it uses the data.
For the result object and its attributes, see Python API.
Run the notebook
Run all cells. The third cell prints a table with the result of each check. In this example all seven checks pass.
When a check fails, the cell raises an error. For example, a customer with the email address ruben35.adventure-works.com fails the check Email addresses are valid, and the cell stops with this error:
The results are also published to Soda Cloud. Open Datasets, select dim_customer, and open the Checks tab to see the latest result of each check and its result history across runs.

dim_customer results in Soda CloudTo get notified when a check fails, see Notifications.
Next steps
Verify contracts at each stage of a pipeline with a Databricks job. See Test data quality in a Databricks pipeline.
Add Metric Monitoring to the dataset to detect anomalies over time.
To verify contracts from outside Databricks, connect Soda to a SQL warehouse. See Databricks SQL.
Last updated
Was this helpful?
