> For the complete documentation index, see [llms.txt](https://docs.soda.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soda.io/guides-and-tutorials/data-owner-producer/test-data-quality-in-a-databricks-pipeline.md).

# Test data quality in a Databricks pipeline

Verify data contracts as tasks in a Databricks job, and stop the pipeline before a machine learning model trains on data that fails a check.

This guide shows how to verify Soda data contracts at each stage of a Databricks job: after ingestion, after the data is transformed for a machine learning model, and after the model is trained. When a check fails, the task that verified it fails, and the tasks that depend on it don't run, so the model never trains on data that fails a check. Warnings are published to Soda Cloud without stopping the job. Every result is also written to a Delta table, so you can query and chart it in Databricks.

Use the example as a starting point and adapt the datasets and checks to your own pipeline.

## Example pipeline

A data engineer and a data scientist build a model that forecasts employee attrition from human resources data. The source data is in three Unity Catalog tables in `main.hr`: `employee_info`, `employee_survey`, and `manager_survey`. A transformation notebook joins them into `input_data_attrition_model`, and a training notebook writes the model's predictions to `attrition_model_output`. Both write a daily partition, identified by the `PartitionDate` column.

A Databricks job runs the pipeline every day. One notebook, `verify_contracts`, verifies a set of contracts after each stage. Its `stage` parameter selects the set:

| Task                    | Notebook                | `stage`     | What it does                                           |
| ----------------------- | ----------------------- | ----------- | ------------------------------------------------------ |
| `verify_ingest`         | `verify_contracts`      | `ingest`    | Verifies the contracts for the three source tables     |
| `prepare_training_data` | `prepare_training_data` |             | Builds `input_data_attrition_model`                    |
| `verify_transform`      | `verify_contracts`      | `transform` | Verifies the contract for `input_data_attrition_model` |
| `train_model`           | `train_model`           |             | Trains the model and writes `attrition_model_output`   |
| `verify_training`       | `verify_contracts`      | `training`  | Verifies the contract for `attrition_model_output`     |

The notebooks and the Soda files sit in the same workspace folder:

```
employee_attrition/
├── prepare_training_data
├── train_model
├── verify_contracts
└── soda/
    ├── sc_config.yml
    └── contracts/
        ├── ingest/
        │   ├── employee_info.yml
        │   ├── employee_survey.yml
        │   └── manager_survey.yml
        ├── transform/
        │   └── input_data_attrition_model.yml
        └── training/
            └── attrition_model_output.yml
```

This guide covers the Soda files, the `verify_contracts` notebook, and the job. The transformation and training notebooks are your own.

## Prerequisites

* A Databricks workspace with Unity Catalog, permission to create jobs, and compute that runs Databricks Runtime 14.0 or later
* The Databricks CLI, authenticated to the workspace, to create a secret scope
* A Soda Cloud account and an API key, to publish results. See [Generate API keys](https://docs.soda.io/apis/generate-api-keys.md). To publish results for new datasets, you need the global permission **Create new datasets and data sources with Soda Library**.
* The **Manage Attributes** permission in Soda Cloud, or someone who has it, to create the check attribute

## Store the API keys in a secret scope

Store the Soda Cloud API key in a Databricks secret scope, so the notebook doesn't contain it. Create a scope named `soda` and add the key ID and the key secret to it:

```bash
databricks secrets create-scope soda
databricks secrets put-secret soda api-key-id
databricks secrets put-secret soda api-key-secret
```

Each `put-secret` command opens an editor where you enter the value.

> For secret scopes and access control, see [Secret management](https://docs.databricks.com/aws/en/security/secrets/) in the Databricks documentation.

## Connect Soda to Soda Cloud

In the workspace folder of the notebooks, create a `soda` folder. Add a Soda Cloud configuration file to it. Use `cloud.us.soda.io` or `cloud.soda.io`, depending on your Soda region. The values in `${env.…}` are read from environment variables, which the notebook sets from the secret scope.

{% code title="soda/sc\_config.yml" %}

```yaml
soda_cloud:
  host: cloud.us.soda.io
  api_key_id: ${env.SODA_CLOUD_API_KEY_ID}
  api_key_secret: ${env.SODA_CLOUD_API_KEY_SECRET}
```

{% endcode %}

## Create the check attribute

Every contract in this example labels its checks with a `pipeline_stage` attribute, so you can filter the results of each stage in Soda Cloud. Soda Cloud accepts results only for attributes it knows, so create the attribute before the first run.

In Soda Cloud, click your profile icon, select **Attributes**, and click **New Attribute**. Enter the label `Pipeline stage`, select the resource type **Check** and the type **Single select**, and add the values `Ingest`, `Transform`, and `Training`. Click **Save**.

The contracts refer to the attribute by its name, which Soda Cloud generates from the label. If the name differs from `pipeline_stage`, use that name in the contracts.

> For attribute types and permissions, see [Check and dataset attributes](https://docs.soda.io/soda-documentation/govern/check-and-dataset-attributes.md).

## Write the contracts

Each contract defines the checks for one dataset. The notebook verifies every contract in a stage folder, so to add a dataset to a stage, add its contract to that folder.

The dataset names start with `employees`, the data source name that the notebook gives Soda, followed by the catalog, schema, and table. The top-level `check_attributes` apply the attribute to every check in the contract.

### Ingest contracts

The source tables come from HR systems that the data engineer can't fix directly. These contracts catch problems as soon as the data lands, so the engineer can report them to the data owners before the model uses the data.

{% code title="soda/contracts/ingest/employee\_info.yml" %}

```yaml
dataset: employees/main/hr/employee_info

check_attributes:
  pipeline_stage: Ingest

columns:
  - name: EmployeeID
    checks:
      - missing:
          name: All employees have an ID
      - duplicate:
          name: Employee IDs are unique
  - name: Age
    checks:
      - invalid:
          name: All employees are 18 or older
          valid_min: 18
  - name: Department
    checks:
      - invalid:
          name: Departments are valid
          valid_values: ['Sales', 'Research & Development', 'Human Resources']
  - name: Gender
    checks:
      - invalid:
          name: Gender values are valid
          valid_values: ['Female', 'Male', 'Non-binary']
  - name: MonthlyIncome
    checks:
      - missing:
          name: All employees have a monthly income
      - failed_rows:
          name: Monthly income meets the legal minimum
          expression: MonthlyIncome < 11000

checks:
  - schema:
      name: Columns have not been added, removed, or changed
```

{% endcode %}

The survey contracts check that every answer belongs to an employee in `employee_info`. An `invalid` check with `valid_reference_data` fails for each value that doesn't appear in the reference column.

{% code title="soda/contracts/ingest/employee\_survey.yml" %}

```yaml
dataset: employees/main/hr/employee_survey

check_attributes:
  pipeline_stage: Ingest

columns:
  - name: EmployeeID
    checks:
      - missing:
          name: All survey answers have an employee ID
      - duplicate:
          name: Each employee answers once
      - invalid:
          name: Survey answers belong to known employees
          valid_reference_data:
            dataset: employees/main/hr/employee_info
            column: EmployeeID
  - name: EnvironmentSatisfaction
    checks:
      - invalid:
          name: Environment satisfaction is between 1 and 5
          valid_min: 1
          valid_max: 5
  - name: WorkLifeBalance
    checks:
      - invalid:
          name: Work-life balance is between 1 and 5
          valid_min: 1
          valid_max: 5

checks:
  - schema:
      name: Columns have not been added, removed, or changed
```

{% endcode %}

{% code title="soda/contracts/ingest/manager\_survey.yml" %}

```yaml
dataset: employees/main/hr/manager_survey

check_attributes:
  pipeline_stage: Ingest

columns:
  - name: EmployeeID
    checks:
      - invalid:
          name: Manager ratings belong to known employees
          valid_reference_data:
            dataset: employees/main/hr/employee_info
            column: EmployeeID
  - name: PerformanceRating
    checks:
      - invalid:
          name: Performance rating is between 1 and 5
          valid_min: 1
          valid_max: 5

checks:
  - schema:
      name: Columns have not been added, removed, or changed
```

{% endcode %}

### Transform contract

The model trains on the latest data only, so the contract's `filter` limits every check to the partitions of today and yesterday. A few negative values in `TotalWorkingYears` are tolerated: the check warns as soon as one row is invalid, and fails when more than 10% of the rows are.

{% code title="soda/contracts/transform/input\_data\_attrition\_model.yml" %}

```yaml
dataset: employees/main/hr/input_data_attrition_model

filter: PartitionDate >= date_sub(current_date(), 1)

check_attributes:
  pipeline_stage: Transform

columns:
  - name: EmployeeID
    checks:
      - invalid:
          name: Training rows belong to known employees
          valid_reference_data:
            dataset: employees/main/hr/employee_info
            column: EmployeeID
  - name: Attrition
    checks:
      - missing:
          name: Target value is not missing
  - name: TotalWorkingYears
    checks:
      - invalid:
          name: Working years are not negative
          valid_min: 0
          threshold:
            metric: percent
            must_be_less_than_or_equal: 10
            additional:
              must_be: 0
              level: warn
  - name: WorkingMinutes
    checks:
      - invalid:
          name: No overtime above 750 minutes
          valid_max: 750
  - name: PartitionDate

checks:
  - row_count:
      name: Training data is present
  - freshness:
      name: Data is less than 2 days old
      column: PartitionDate
      threshold:
        unit: day
        must_be_less_than: 2
```

{% endcode %}

### Training contract

After training, the contract checks the model's output, with the same filter. The average accuracy fails below 0.6 and warns below 0.7.

{% code title="soda/contracts/training/attrition\_model\_output.yml" %}

```yaml
dataset: employees/main/hr/attrition_model_output

filter: PartitionDate >= date_sub(current_date(), 1)

check_attributes:
  pipeline_stage: Training

columns:
  - name: EmployeeID
  - name: Attrition
    checks:
      - missing:
          name: Every employee has a prediction
  - name: Accuracy
    checks:
      - aggregate:
          name: Accuracy is at least 60%
          function: avg
          threshold:
            must_be_greater_than_or_equal: 0.6
            additional:
              must_be_greater_than_or_equal: 0.7
              level: warn
  - name: PartitionDate

checks:
  - row_count:
      name: Model output is not empty
```

{% endcode %}

> For every check type and threshold option, see the [Contract Language reference](https://docs.soda.io/soda-documentation/developer-tools/contract-language-reference.md).

{% hint style="info" %}
Coming from Soda v3? `group evolution` checks have no contract equivalent. `group by` checks need the `soda-groupby` package from the private PyPI, with a Team or Enterprise license. Anomaly detection moves to [Metric Monitoring](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard.md), and profiling to [Profiling](https://docs.soda.io/soda-documentation/find/data-observability/profiling.md) in Soda Cloud. For the rest of the syntax changes, see [Migrate from v3 to v4](https://docs.soda.io/soda-documentation/govern/migrate-from-v3-to-v4.md).
{% endhint %}

## Create the verification notebook

Create a notebook named `verify_contracts` in the `employee_attrition` folder. In the first cell, install the Soda package for Spark DataFrames:

```python
%pip install -i https://pypi.cloud.soda.io/simple "soda-sparkdf>4"
```

In the second cell, restart Python so the notebook can import the new packages:

```python
dbutils.library.restartPython()
```

For serverless compute, Databricks recommends a pinned version, such as `"soda-sparkdf==4.27.0"`. See [Notebook-scoped Python libraries](https://docs.databricks.com/aws/en/libraries/notebooks-python-libraries) in the Databricks documentation.

In the third cell, add the code that verifies the contracts of a stage:

{% code title="verify\_contracts" %}

```python
import os
from pathlib import Path

from soda_core import configure_logging
from soda_core.contracts import verify_contract_locally
from soda_sparkdf import SparkDataFrameDataSource

RESULTS_TABLE = "main.hr.soda_check_results"

dbutils.widgets.text("stage", "ingest")
stage = dbutils.widgets.get("stage")

os.environ["SODA_CLOUD_API_KEY_ID"] = dbutils.secrets.get(scope="soda", key="api-key-id")
os.environ["SODA_CLOUD_API_KEY_SECRET"] = dbutils.secrets.get(scope="soda", key="api-key-secret")

data_source = SparkDataFrameDataSource.from_existing_session(session=spark, name="employees")

rows = []
failed = []
for contract in sorted(Path("soda/contracts", stage).glob("*.yml")):
    configure_logging(verbose=False)
    result = verify_contract_locally(
        data_sources=[data_source],
        contract_file_path=str(contract),
        soda_cloud_file_path="soda/sc_config.yml",
        publish=True,
    )
    for contract_result in result.contract_verification_results:
        for check_result in contract_result.check_results:
            rows.append((
                contract_result.started_timestamp,
                stage,
                contract_result.contract.dataset_name,
                check_result.check.name,
                check_result.outcome.name,
            ))
    if not result.is_ok:
        failed.append(
            f"{contract.name}: {result.number_of_checks_failed} failed checks "
            f"{result.get_errors_str()}"
        )

spark.createDataFrame(
    rows, "verified_at timestamp, stage string, dataset string, check_name string, outcome string"
).write.mode("append").saveAsTable(RESULTS_TABLE)

if failed:
    raise ValueError("Contract verification failed: " + "; ".join(failed))
```

{% endcode %}

A few details in the cell:

* `dbutils.widgets.get("stage")` reads the task parameter `stage`. When you run the notebook by hand, the widget defaults to `ingest`.
* `SparkDataFrameDataSource.from_existing_session` runs the checks with the notebook's `spark` session, which reads the Unity Catalog tables directly. Its `name` is the first part of the dataset names in the contracts.
* `configure_logging` is called before each verification in the loop. See [Python API](https://docs.soda.io/apis/python-api.md).
* `result.is_ok` is `False` when a check fails or the verification errors, and `True` when checks only warn. A warning therefore doesn't stop the job.
* The cell appends the result of every check to `main.hr.soda_check_results` before it raises an error, so failed checks are recorded too.
* The file paths are relative to the notebook's folder, which is the working directory in Databricks Runtime 14.0 and later. See [What is the default current working directory?](https://docs.databricks.com/aws/en/files/cwd-dbr-14) in the Databricks documentation.

> If the import fails with a validation error or `cannot import name 'sql' from 'databricks'`, see [Troubleshoot](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/spark-dataframe.md#troubleshoot) in the Spark DataFrame reference.

## Create the job

Create a job with the five tasks from the [example pipeline](#example-pipeline), each of type **Notebook**:

1. For each verification task, select the `verify_contracts` notebook, and add a parameter with the key `stage` and the stage as its value.
2. For `prepare_training_data` and `train_model`, select your own notebooks.
3. Make each task depend on the task before it. Keep the default **Run if** condition, **All succeeded**, so that a task runs only when the task before it succeeds.
4. Add a daily schedule to the job.

> For notebook tasks, parameters, and dependencies, see [Notebook task for jobs](https://docs.databricks.com/aws/en/jobs/notebook) and [Configure task dependencies](https://docs.databricks.com/aws/en/jobs/run-if) in the Databricks documentation.

## Run the job

Run the job once to test it. In this example, all five tasks succeed. `verify_transform` publishes a warning for **Working years are not negative**, because one of the 20 rows that the filter selects has a negative value. That is 5% of the rows, below the 10% that fails the check.

When a check fails, the task that verified it fails, and the tasks after it are marked **Upstream failed**. For example, a survey answer from an employee who isn't in `employee_info` fails the check **Survey answers belong to known employees**. `verify_ingest` then fails with this error, and the model doesn't train:

```
ValueError: Contract verification failed: employee_survey.yml: 1 failed checks
```

## Results in Soda Cloud

Each verification publishes its results to the dataset in Soda Cloud. To see the results of one stage, open **Checks**, filter by the **Pipeline stage** attribute, and click **Save** to keep the filter as a collection.

> For filters and collections, see [Browse checks](https://docs.soda.io/soda-documentation/understand/manage-issues/browse-checks.md). To get notified when a check fails, see [Notifications](https://docs.soda.io/soda-documentation/fix/notifications.md).

## Results in a Databricks table

Every run appends one row per check to `main.hr.soda_check_results`, with the time of the verification, the stage, the dataset, the check name, and its outcome: `PASSED`, `WARN`, `FAILED`, `NOT_EVALUATED`, or `EXCLUDED`. Query the table to find the checks that need attention:

```sql
SELECT verified_at, stage, dataset, check_name, outcome
FROM main.hr.soda_check_results
WHERE outcome IN ('FAILED', 'WARN')
ORDER BY verified_at DESC
```

To chart the results, build a dashboard on the table. See [Dashboards](https://docs.databricks.com/aws/en/dashboards/) in the Databricks documentation.

## Next steps

* Add [Metric Monitoring](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard.md) to the same datasets to detect anomalies over time.
* Verify a single DataFrame from a notebook. See [Add Soda to a Databricks notebook](https://app.gitbook.com/s/JJOow5YZtrrxZJBNXV2D/data-owner/add-soda-to-a-databricks-notebook).
* To verify contracts from outside Databricks, connect Soda to a SQL warehouse. See [Databricks SQL](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/databricks.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.soda.io/guides-and-tutorials/data-owner-producer/test-data-quality-in-a-databricks-pipeline.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
