Test data quality in a Databricks pipeline
Verify data contracts as tasks in a Databricks job, and stop the pipeline before a machine learning model trains on data that fails a check.
This guide shows how to verify Soda data contracts at each stage of a Databricks job: after ingestion, after the data is transformed for a machine learning model, and after the model is trained. When a check fails, the task that verified it fails, and the tasks that depend on it don't run, so the model never trains on data that fails a check. Warnings are published to Soda Cloud without stopping the job. Every result is also written to a Delta table, so you can query and chart it in Databricks.
Use the example as a starting point and adapt the datasets and checks to your own pipeline.
Example pipeline
A data engineer and a data scientist build a model that forecasts employee attrition from human resources data. The source data is in three Unity Catalog tables in main.hr: employee_info, employee_survey, and manager_survey. A transformation notebook joins them into input_data_attrition_model, and a training notebook writes the model's predictions to attrition_model_output. Both write a daily partition, identified by the PartitionDate column.
A Databricks job runs the pipeline every day. One notebook, verify_contracts, verifies a set of contracts after each stage. Its stage parameter selects the set:
Task
Notebook
stage
What it does
verify_ingest
verify_contracts
ingest
Verifies the contracts for the three source tables
prepare_training_data
prepare_training_data
Builds input_data_attrition_model
verify_transform
verify_contracts
transform
Verifies the contract for input_data_attrition_model
train_model
train_model
Trains the model and writes attrition_model_output
verify_training
verify_contracts
training
Verifies the contract for attrition_model_output
The notebooks and the Soda files sit in the same workspace folder:
employee_attrition/
├── prepare_training_data
├── train_model
├── verify_contracts
└── soda/
├── sc_config.yml
└── contracts/
├── ingest/
│ ├── employee_info.yml
│ ├── employee_survey.yml
│ └── manager_survey.yml
├── transform/
│ └── input_data_attrition_model.yml
└── training/
└── attrition_model_output.ymlThis guide covers the Soda files, the verify_contracts notebook, and the job. The transformation and training notebooks are your own.
Prerequisites
A Databricks workspace with Unity Catalog, permission to create jobs, and compute that runs Databricks Runtime 14.0 or later
The Databricks CLI, authenticated to the workspace, to create a secret scope
A Soda Cloud account and an API key, to publish results. See Generate API keys. To publish results for new datasets, you need the global permission Create new datasets and data sources with Soda Library.
The Manage Attributes permission in Soda Cloud, or someone who has it, to create the check attribute
Store the API keys in a secret scope
Store the Soda Cloud API key in a Databricks secret scope, so the notebook doesn't contain it. Create a scope named soda and add the key ID and the key secret to it:
Each put-secret command opens an editor where you enter the value.
For secret scopes and access control, see Secret management in the Databricks documentation.
Connect Soda to Soda Cloud
In the workspace folder of the notebooks, create a soda folder. Add a Soda Cloud configuration file to it. Use cloud.us.soda.io or cloud.soda.io, depending on your Soda region. The values in ${env.…} are read from environment variables, which the notebook sets from the secret scope.
Create the check attribute
Every contract in this example labels its checks with a pipeline_stage attribute, so you can filter the results of each stage in Soda Cloud. Soda Cloud accepts results only for attributes it knows, so create the attribute before the first run.
In Soda Cloud, click your profile icon, select Attributes, and click New Attribute. Enter the label Pipeline stage, select the resource type Check and the type Single select, and add the values Ingest, Transform, and Training. Click Save.
The contracts refer to the attribute by its name, which Soda Cloud generates from the label. If the name differs from pipeline_stage, use that name in the contracts.
For attribute types and permissions, see Check and dataset attributes.
Write the contracts
Each contract defines the checks for one dataset. The notebook verifies every contract in a stage folder, so to add a dataset to a stage, add its contract to that folder.
The dataset names start with employees, the data source name that the notebook gives Soda, followed by the catalog, schema, and table. The top-level check_attributes apply the attribute to every check in the contract.
Ingest contracts
The source tables come from HR systems that the data engineer can't fix directly. These contracts catch problems as soon as the data lands, so the engineer can report them to the data owners before the model uses the data.
The survey contracts check that every answer belongs to an employee in employee_info. An invalid check with valid_reference_data fails for each value that doesn't appear in the reference column.
Transform contract
The model trains on the latest data only, so the contract's filter limits every check to the partitions of today and yesterday. A few negative values in TotalWorkingYears are tolerated: the check warns as soon as one row is invalid, and fails when more than 10% of the rows are.
Training contract
After training, the contract checks the model's output, with the same filter. The average accuracy fails below 0.6 and warns below 0.7.
For every check type and threshold option, see the Contract Language reference.
Create the verification notebook
Create a notebook named verify_contracts in the employee_attrition folder. In the first cell, install the Soda package for Spark DataFrames:
In the second cell, restart Python so the notebook can import the new packages:
For serverless compute, Databricks recommends a pinned version, such as "soda-sparkdf==4.27.0". See Notebook-scoped Python libraries in the Databricks documentation.
In the third cell, add the code that verifies the contracts of a stage:
A few details in the cell:
dbutils.widgets.get("stage")reads the task parameterstage. When you run the notebook by hand, the widget defaults toingest.SparkDataFrameDataSource.from_existing_sessionruns the checks with the notebook'ssparksession, which reads the Unity Catalog tables directly. Itsnameis the first part of the dataset names in the contracts.configure_loggingis called before each verification in the loop. See Python API.result.is_okisFalsewhen a check fails or the verification errors, andTruewhen checks only warn. A warning therefore doesn't stop the job.The cell appends the result of every check to
main.hr.soda_check_resultsbefore it raises an error, so failed checks are recorded too.The file paths are relative to the notebook's folder, which is the working directory in Databricks Runtime 14.0 and later. See What is the default current working directory? in the Databricks documentation.
If the import fails with a validation error or
cannot import name 'sql' from 'databricks', see Troubleshoot in the Spark DataFrame reference.
Create the job
Create a job with the five tasks from the example pipeline, each of type Notebook:
For each verification task, select the
verify_contractsnotebook, and add a parameter with the keystageand the stage as its value.For
prepare_training_dataandtrain_model, select your own notebooks.Make each task depend on the task before it. Keep the default Run if condition, All succeeded, so that a task runs only when the task before it succeeds.
Add a daily schedule to the job.
For notebook tasks, parameters, and dependencies, see Notebook task for jobs and Configure task dependencies in the Databricks documentation.
Run the job
Run the job once to test it. In this example, all five tasks succeed. verify_transform publishes a warning for Working years are not negative, because one of the 20 rows that the filter selects has a negative value. That is 5% of the rows, below the 10% that fails the check.
When a check fails, the task that verified it fails, and the tasks after it are marked Upstream failed. For example, a survey answer from an employee who isn't in employee_info fails the check Survey answers belong to known employees. verify_ingest then fails with this error, and the model doesn't train:
Results in Soda Cloud
Each verification publishes its results to the dataset in Soda Cloud. To see the results of one stage, open Checks, filter by the Pipeline stage attribute, and click Save to keep the filter as a collection.
For filters and collections, see Browse checks. To get notified when a check fails, see Notifications.
Results in a Databricks table
Every run appends one row per check to main.hr.soda_check_results, with the time of the verification, the stage, the dataset, the check name, and its outcome: PASSED, WARN, FAILED, NOT_EVALUATED, or EXCLUDED. Query the table to find the checks that need attention:
To chart the results, build a dashboard on the table. See Dashboards in the Databricks documentation.
Next steps
Add Metric Monitoring to the same datasets to detect anomalies over time.
Verify a single DataFrame from a notebook. See Add Soda to a Databricks notebook.
To verify contracts from outside Databricks, connect Soda to a SQL warehouse. See Databricks SQL.
Last updated
Was this helpful?
