Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Test data quality in a Databricks pipeline

Verify data contracts as tasks in a Databricks job, and stop the pipeline before a machine learning model trains on data that fails a check.

This guide shows how to verify Soda data contracts at each stage of a Databricks job: after ingestion, after the data is transformed for a machine learning model, and after the model is trained. When a check fails, the task that verified it fails, and the tasks that depend on it don't run, so the model never trains on data that fails a check. Warnings are published to Soda Cloud without stopping the job. Every result is also written to a Delta table, so you can query and chart it in Databricks.

Use the example as a starting point and adapt the datasets and checks to your own pipeline.

Example pipeline

A data engineer and a data scientist build a model that forecasts employee attrition from human resources data. The source data is in three Unity Catalog tables in main.hr: employee_info, employee_survey, and manager_survey. A transformation notebook joins them into input_data_attrition_model, and a training notebook writes the model's predictions to attrition_model_output. Both write a daily partition, identified by the PartitionDate column.

A Databricks job runs the pipeline every day. One notebook, verify_contracts, verifies a set of contracts after each stage. Its stage parameter selects the set:

Task

Notebook

stage

What it does

verify_ingest

verify_contracts

ingest

Verifies the contracts for the three source tables

prepare_training_data

prepare_training_data

Builds input_data_attrition_model

verify_transform

verify_contracts

transform

Verifies the contract for input_data_attrition_model

train_model

train_model

Trains the model and writes attrition_model_output

verify_training

verify_contracts

training

Verifies the contract for attrition_model_output

The notebooks and the Soda files sit in the same workspace folder:

employee_attrition/
├── prepare_training_data
├── train_model
├── verify_contracts
└── soda/
    ├── sc_config.yml
    └── contracts/
        ├── ingest/
        │   ├── employee_info.yml
        │   ├── employee_survey.yml
        │   └── manager_survey.yml
        ├── transform/
        │   └── input_data_attrition_model.yml
        └── training/
            └── attrition_model_output.yml

This guide covers the Soda files, the verify_contracts notebook, and the job. The transformation and training notebooks are your own.

Prerequisites

  • A Databricks workspace with Unity Catalog, permission to create jobs, and compute that runs Databricks Runtime 14.0 or later

  • The Databricks CLI, authenticated to the workspace, to create a secret scope

  • A Soda Cloud account and an API key, to publish results. See Generate API keys. To publish results for new datasets, you need the global permission Create new datasets and data sources with Soda Library.

  • The Manage Attributes permission in Soda Cloud, or someone who has it, to create the check attribute

Store the API keys in a secret scope

Store the Soda Cloud API key in a Databricks secret scope, so the notebook doesn't contain it. Create a scope named soda and add the key ID and the key secret to it:

Each put-secret command opens an editor where you enter the value.

For secret scopes and access control, see Secret management in the Databricks documentation.

Connect Soda to Soda Cloud

In the workspace folder of the notebooks, create a soda folder. Add a Soda Cloud configuration file to it. Use cloud.us.soda.io or cloud.soda.io, depending on your Soda region. The values in ${env.…} are read from environment variables, which the notebook sets from the secret scope.

Create the check attribute

Every contract in this example labels its checks with a pipeline_stage attribute, so you can filter the results of each stage in Soda Cloud. Soda Cloud accepts results only for attributes it knows, so create the attribute before the first run.

In Soda Cloud, click your profile icon, select Attributes, and click New Attribute. Enter the label Pipeline stage, select the resource type Check and the type Single select, and add the values Ingest, Transform, and Training. Click Save.

The contracts refer to the attribute by its name, which Soda Cloud generates from the label. If the name differs from pipeline_stage, use that name in the contracts.

For attribute types and permissions, see Check and dataset attributes.

Write the contracts

Each contract defines the checks for one dataset. The notebook verifies every contract in a stage folder, so to add a dataset to a stage, add its contract to that folder.

The dataset names start with employees, the data source name that the notebook gives Soda, followed by the catalog, schema, and table. The top-level check_attributes apply the attribute to every check in the contract.

Ingest contracts

The source tables come from HR systems that the data engineer can't fix directly. These contracts catch problems as soon as the data lands, so the engineer can report them to the data owners before the model uses the data.

The survey contracts check that every answer belongs to an employee in employee_info. An invalid check with valid_reference_data fails for each value that doesn't appear in the reference column.

Transform contract

The model trains on the latest data only, so the contract's filter limits every check to the partitions of today and yesterday. A few negative values in TotalWorkingYears are tolerated: the check warns as soon as one row is invalid, and fails when more than 10% of the rows are.

Training contract

After training, the contract checks the model's output, with the same filter. The average accuracy fails below 0.6 and warns below 0.7.

For every check type and threshold option, see the Contract Language reference.

Coming from Soda v3? group evolution checks have no contract equivalent. group by checks need the soda-groupby package from the private PyPI, with a Team or Enterprise license. Anomaly detection moves to Metric Monitoring, and profiling to Profiling in Soda Cloud. For the rest of the syntax changes, see Migrate from v3 to v4.

Create the verification notebook

Create a notebook named verify_contracts in the employee_attrition folder. In the first cell, install the Soda package for Spark DataFrames:

In the second cell, restart Python so the notebook can import the new packages:

For serverless compute, Databricks recommends a pinned version, such as "soda-sparkdf==4.27.0". See Notebook-scoped Python libraries in the Databricks documentation.

In the third cell, add the code that verifies the contracts of a stage:

A few details in the cell:

  • dbutils.widgets.get("stage") reads the task parameter stage. When you run the notebook by hand, the widget defaults to ingest.

  • SparkDataFrameDataSource.from_existing_session runs the checks with the notebook's spark session, which reads the Unity Catalog tables directly. Its name is the first part of the dataset names in the contracts.

  • configure_logging is called before each verification in the loop. See Python API.

  • result.is_ok is False when a check fails or the verification errors, and True when checks only warn. A warning therefore doesn't stop the job.

  • The cell appends the result of every check to main.hr.soda_check_results before it raises an error, so failed checks are recorded too.

  • The file paths are relative to the notebook's folder, which is the working directory in Databricks Runtime 14.0 and later. See What is the default current working directory? in the Databricks documentation.

If the import fails with a validation error or cannot import name 'sql' from 'databricks', see Troubleshoot in the Spark DataFrame reference.

Create the job

Create a job with the five tasks from the example pipeline, each of type Notebook:

  1. For each verification task, select the verify_contracts notebook, and add a parameter with the key stage and the stage as its value.

  2. For prepare_training_data and train_model, select your own notebooks.

  3. Make each task depend on the task before it. Keep the default Run if condition, All succeeded, so that a task runs only when the task before it succeeds.

  4. Add a daily schedule to the job.

For notebook tasks, parameters, and dependencies, see Notebook task for jobs and Configure task dependencies in the Databricks documentation.

Run the job

Run the job once to test it. In this example, all five tasks succeed. verify_transform publishes a warning for Working years are not negative, because one of the 20 rows that the filter selects has a negative value. That is 5% of the rows, below the 10% that fails the check.

When a check fails, the task that verified it fails, and the tasks after it are marked Upstream failed. For example, a survey answer from an employee who isn't in employee_info fails the check Survey answers belong to known employees. verify_ingest then fails with this error, and the model doesn't train:

Results in Soda Cloud

Each verification publishes its results to the dataset in Soda Cloud. To see the results of one stage, open Checks, filter by the Pipeline stage attribute, and click Save to keep the filter as a collection.

For filters and collections, see Browse checks. To get notified when a check fails, see Notifications.

Results in a Databricks table

Every run appends one row per check to main.hr.soda_check_results, with the time of the verification, the stage, the dataset, the check name, and its outcome: PASSED, WARN, FAILED, NOT_EVALUATED, or EXCLUDED. Query the table to find the checks that need attention:

To chart the results, build a dashboard on the table. See Dashboards in the Databricks documentation.

Next steps

Last updated

Was this helpful?