> For the complete documentation index, see [llms.txt](https://docs.soda.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soda.io/guides-and-tutorials/data-owner-producer/add-soda-to-a-databricks-notebook.md).

# Add Soda to a Databricks notebook

Verify a Soda data contract on a Spark DataFrame from a Databricks notebook, and publish the results to Soda Cloud.

This guide shows how to install Soda in a Databricks notebook and verify a data contract on a Spark DataFrame. Soda runs the checks with the notebook's own Spark session, so it reads the same data the notebook does and needs no separate connection to Databricks. The results appear in the notebook output and in Soda Cloud.

Use the example as a starting point and adapt the dataset and checks to your own data.

## Example notebook

A data engineer loads AdventureWorks customer data from the Unity Catalog table `main.adventureworks.dim_customer` into a DataFrame. Before the notebook uses the data, it verifies a contract that checks customer keys, last names, email addresses, phone numbers, and how recent the data is.

The notebook and the Soda files sit in the same workspace folder:

```
customers/
├── verify_dim_customer
└── soda/
    ├── sc_config.yml
    └── contracts/
        └── dim_customer.yml
```

## Prerequisites

* A Databricks workspace with Unity Catalog, and a notebook attached to compute that runs Databricks Runtime 14.0 or later
* The Databricks CLI, authenticated to the workspace, to create a secret scope
* A Soda Cloud account and an API key, to publish results. See [Generate API keys](https://docs.soda.io/apis/generate-api-keys.md). To publish results for a new dataset, you need the global permission **Create new datasets and data sources with Soda Library**.

## Store the API keys in a secret scope

Store the Soda Cloud API key in a Databricks secret scope, so the notebook doesn't contain it. Create a scope named `soda` and add the key ID and the key secret to it:

```bash
databricks secrets create-scope soda
databricks secrets put-secret soda api-key-id
databricks secrets put-secret soda api-key-secret
```

Each `put-secret` command opens an editor where you enter the value.

> For secret scopes and access control, see [Secret management](https://docs.databricks.com/aws/en/security/secrets/) in the Databricks documentation.

## Add the Soda files

In the workspace folder of the notebook, create a `soda` folder. Add a Soda Cloud configuration file to it. Use `cloud.us.soda.io` or `cloud.soda.io`, depending on your Soda region. The values in `${env.…}` are read from environment variables, which the notebook sets from the secret scope.

{% code title="soda/sc\_config.yml" %}

```yaml
soda_cloud:
  host: cloud.us.soda.io
  api_key_id: ${env.SODA_CLOUD_API_KEY_ID}
  api_key_secret: ${env.SODA_CLOUD_API_KEY_SECRET}
```

{% endcode %}

Add the contract in a `contracts` folder. The dataset name starts with the data source name that the notebook gives Soda, `spark_df`, followed by the name of the temporary view that holds the DataFrame, `dim_customer`.

{% code title="soda/contracts/dim\_customer.yml" %}

```yaml
dataset: spark_df/dim_customer

columns:
  - name: customer_key
    data_type: int
    checks:
      - missing:
          name: All customers have a key
      - duplicate:
          name: Customer keys are unique
  - name: last_name
    data_type: string
    checks:
      - missing:
          name: All customers have a last name
  - name: email_address
    data_type: string
    checks:
      - invalid:
          name: Email addresses are valid
          valid_format:
            name: Email address
            regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$
  - name: phone
    data_type: string
    checks:
      - duplicate:
          name: Phone numbers are unique
  - name: date_first_purchase
    data_type: date

checks:
  - schema:
      name: Columns have not been added, removed, or changed
  - freshness:
      name: Data is less than 7 days old
      column: date_first_purchase
      threshold:
        unit: day
        must_be_less_than: 7
```

{% endcode %}

Soda compares the column names and data types with the view, so the schema check fails if a column is added, removed, or changes type. The freshness check fails if the most recent `date_first_purchase` is 7 days old or more.

To verify a table instead of a DataFrame, name it with its catalog and schema in the dataset name, for example `spark_df/main/adventureworks/dim_customer`.

> For every check type and threshold option, see the [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference.md).

{% hint style="info" %}
Coming from Soda v3? The named format `email` has no contract equivalent, so this contract checks email addresses with a regular expression. For the rest of the syntax changes, see [Migrate from v3 to v4](https://docs.soda.io/soda-documentation/govern/migrate-from-v3-to-v4.md).
{% endhint %}

## Install Soda in the notebook

In the first cell of the notebook, install the Soda package for Spark DataFrames:

```python
%pip install -i https://pypi.cloud.soda.io/simple "soda-sparkdf>4"
```

`%pip` installs the package for this notebook only. In the second cell, restart Python so the notebook can import the new packages:

```python
dbutils.library.restartPython()
```

For serverless compute, Databricks recommends a pinned version, such as `"soda-sparkdf==4.27.0"`. See [Notebook-scoped Python libraries](https://docs.databricks.com/aws/en/libraries/notebooks-python-libraries) in the Databricks documentation.

> If the import fails with a validation error or `cannot import name 'sql' from 'databricks'`, see [Troubleshoot](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/spark-dataframe.md#troubleshoot) in the Spark DataFrame reference.

## Verify the contract

In the third cell, add the code that loads the data and verifies the contract:

{% code title="verify\_dim\_customer" %}

```python
import os

from soda_core import configure_logging
from soda_core.contracts import verify_contract_locally
from soda_sparkdf import SparkDataFrameDataSource

os.environ["SODA_CLOUD_API_KEY_ID"] = dbutils.secrets.get(scope="soda", key="api-key-id")
os.environ["SODA_CLOUD_API_KEY_SECRET"] = dbutils.secrets.get(scope="soda", key="api-key-secret")

customers = spark.table("main.adventureworks.dim_customer")
customers.createOrReplaceTempView("dim_customer")

configure_logging(verbose=False)
data_source = SparkDataFrameDataSource.from_existing_session(session=spark, name="spark_df")

result = verify_contract_locally(
    data_sources=[data_source],
    contract_file_path="soda/contracts/dim_customer.yml",
    soda_cloud_file_path="soda/sc_config.yml",
    publish=True,
)

if not result.is_ok:
    raise ValueError(
        f"Contract verification failed: {result.number_of_checks_failed} failed checks "
        f"{result.get_errors_str()}"
    )
```

{% endcode %}

A few details in the cell:

* `dbutils.secrets.get` reads the API key from the secret scope, and the cell passes it to Soda through the environment variables that `sc_config.yml` refers to. Databricks redacts secret values in the notebook output.
* `createOrReplaceTempView` makes the DataFrame available to Soda under the name in the contract. Any DataFrame works, whether the notebook reads it from a table, from files, or builds it in a transformation.
* `SparkDataFrameDataSource.from_existing_session` runs the checks with the notebook's `spark` session. Its `name` is the first part of the dataset name in the contract.
* The file paths are relative to the notebook's folder, which is the working directory in Databricks Runtime 14.0 and later. See [What is the default current working directory?](https://docs.databricks.com/aws/en/files/cwd-dbr-14) in the Databricks documentation.
* `result.is_ok` is `False` when a check fails or the verification errors. The cell then raises an error, so nothing that runs after it uses the data.

> For the result object and its attributes, see [Python API](https://docs.soda.io/apis/python-api.md).

## Run the notebook

Run all cells. The third cell prints a table with the result of each check. In this example all seven checks pass.

When a check fails, the cell raises an error. For example, a customer with the email address `ruben35.adventure-works.com` fails the check **Email addresses are valid**, and the cell stops with this error:

```
ValueError: Contract verification failed: 1 failed checks
```

The results are also published to Soda Cloud. Open **Datasets**, select `dim_customer`, and open the **Checks** tab to see the latest result of each check and its result history across runs.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-10374d0525e7335ea8d62e871f4d9239c5f37228%2Fadd-soda-to-a-databricks-notebook-databricks-dim-customer-checks.png?alt=media" alt="The Checks tab of the dim_customer dataset in Soda Cloud, with the check Email addresses are valid failed"><figcaption><p>The <code>dim_customer</code> results in Soda Cloud</p></figcaption></figure>

> To get notified when a check fails, see [Notifications](https://docs.soda.io/soda-documentation/fix/notifications.md).

## Next steps

* Verify contracts at each stage of a pipeline with a Databricks job. See [Test data quality in a Databricks pipeline](https://app.gitbook.com/s/JJOow5YZtrrxZJBNXV2D/data-owner/test-data-quality-in-a-databricks-pipeline).
* Add [Metric Monitoring](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard.md) to the dataset to detect anomalies over time.
* To verify contracts from outside Databricks, connect Soda to a SQL warehouse. See [Databricks SQL](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/databricks.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.soda.io/guides-and-tutorials/data-owner-producer/add-soda-to-a-databricks-notebook.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
