Via Soda Cloud
Connect a data source to Soda Cloud and onboard its datasets, step by step, from the connection form to monitoring and profiling.
Onboard a new data source
Before you can define contracts, you need to connect Soda Cloud to your data source. This allows Soda to access your datasets for profiling, metric monitoring, and contract verification.
First, navigate to the Data Sources page from the left navigation bar in Soda Cloud. On this page, you'll see a list of connected sources and a New Data Source button.
You need the "Manage data sources" global permission to add a new data source.
Learn more about Global and Dataset Roles
Establish connection
On the Data Sources page, click on New Data Source
Select your data source provider
After selecting a source, you'll be presented with a configuration form.
Name your data source under Data Source Identifier
Enter a friendly, unique label. A unique identifier will be automatically generated from this label. This becomes the immutable ID of the data source and can also be used to reference the same connection in Soda Core.
Set a default runner
You’ll be asked to select a runner. This is the component that connects to your data source and runs scans.
You can choose from:
Soda-hosted runner – Quickest option, fully managed by Soda (recommended for getting started)
Self-hosted runner – For custom or secure deployments where you manage the runner yourself
[Preview feature] Bind a runner to a service account - If you need to scope the permissions of a runner to those of a service account.
Learn more about deployment options: Deployment options
[Optional] Set a default schedule for contracts
Toggle on Default schedule to pre-fill contract schedules for this data source. When off, contracts created against this data source default to midnight UTC.
Existing contracts are not affected. Contracts keep their own schedule once saved, so changes to the Default schedule only apply to contracts that have not been configured yet.
The default schedule is used by:
Contract Editor
When the default schedule is enabled, the time zone and cron pre-fill from the Default schedule.
Contract Autopilot
During dataset onboarding: when the default schedule is enabled, the Add schedule to generated contracts section pre-fills with the Default schedule.
When generating data contracts in bulk: you can choose Use the data source's default schedule so each contract picks up its own data source's default, or Manual schedule to apply one schedule to all selected datasets.
Secure your credentials with secrets
You’ll need to fill in the connection details. Soda uses the official Python packages for each supported data source, which means you can define any properties required by those libraries, flexibly and reliably.
This includes common fields like host, port, database name, username, and more, depending on the data source.
Using secrets for sensitive credentials
For sensitive values such as passwords, tokens, or keys, you should use Soda Secrets instead of entering them directly in the configuration.
Secrets are encrypted and securely stored in Soda Cloud.
They can be safely referenced in your data source configuration without exposing them in plain text.
To add secrets:
Navigate to the Data Sources tab in the top navigation.
Click the Secrets tab.
Define key-value pairs for your sensitive credentials.
You can then reference a secret in your data source configuration using this syntax:
This ensures your sensitive values stay secure while still being accessible to the runner at runtime.
Test and Connect
Once the form is complete:
Click Test Connection to validate that Soda can successfully connect to your data source.
If the test passes, click Connect to finalize the setup.
[Optional] Enable Diagnostics Warehouse
Enable Diagnostics Warehouse to store record-level diagnostics, such as failed rows, in a warehouse you own.
Choose the Diagnostics Warehouse connection
[Optional] Diagnostics Warehouse Advanced Configuration
Provide a schema configuration
This will be the default schema where Soda creates diagnostic tables. If not provided, Soda will create a new schema with the default name soda_diagnostics.
Provide table names
For your own organization, you can add prefixes/suffixes to the table names that Soda will create in your diagnostics warehouse.
[Optional] Allow collections of check results & scans
You can allow Soda to collect check results and scans (only for this data source) in order to store them securely on your diagnostics warehouse.
[Optional] Allow failed row collection
You can allow Soda to collect failed rows (only for this data source) in order to store them securely on your diagnostics warehouse.
Learn more about this options when setting up Diagnostics Warehouse.
Click Test connection, then click Save
The Diagnostics Warehouse connection must have write access in order to perform correctly.
Learn more about setting up Diagnostics Warehouse.
Onboard datasets
After connecting, Soda will perform an automated dataset discovery. Soda triggers a scan that analyzes the datasets and retrieves their metadata, including columns and column data types. This reduces manual setup efforts, ensures data coverage in your environment and keeps Soda's dataset inventory aligned with your data sources. This feature allows other Soda features to work seamlessly:
Contract generation
Automated discovery of time partition column
Automated discovery of Primary Keys for Diagnostics Warehouse
Choose a dataset selection strategy
Dataset selection can be manual or rules-based.
Manual dataset selection
Manual selection allows you to browse a directory view of all the datasets in your data source.
The Scope can range from the entire data source to a specific schema. Any element selected on the left panel becomes the scope of the dataset search.
The manual selection is made for scale; it can easily handle thousands of schemas and hundreds of thousands of datasets.
Datasets that have already been onboarded will not be visible in the manual dataset selection.
Rules-based dataset selection
Rules-based selection allows you to automate the dataset onboarding process, only selecting those which match specified rules.
Rules-based selection includes existing and future datasets that match the conditions.
Soda will run discovery scans once per day at a random time on the data source. When Soda discovers a new dataset that matches the conditions set in the rules, it will automatically onboard it.
You can choose the specific schemas where your datasets of interest are located.
You can add rules to include or exclude datasets that match certain conditions, such as "name contains" or "name starts with", or provide your own regex pattern.
To create a rule, choose a condition that datasets must match in order to be onboarded.
In the example below, only datasets whose name does not start with "
dwh" from thepublicschema will be onboarded.
Once you click on Validate rule, Soda will calculate how many datasets currently match the defined conditions:
Onboarding rules review
Once the onboarding process is finished (after Enabling Metric Monitoring), an overview of the Onboarding Rules will be provided. From this view, rules can be edited or deleted:
Rules will be executed in order of appearance on this view.
The order of the rules can be changed. As soon as a dataset matches a rule, it will be onboarded automatically; datasets can only be onboarded once.
Confirm onboarding
Click on Next to finish the process.
Once onboarded, datasets will appear in your Soda Cloud UI and become available for contract creation or metric monitoring.
Refresh dataset discovery: Soda runs discovery scans once per day at a random time to get the latest view of tables and schemas within a data source. By pressing on the icon on the top right of the page, you can run the scan on demand.
To configure the discovery scan schedule, go to Configure discovery scan.
Enable/disable Contract Autopilot
Enable Contract Autopilot to auto-generate contracts tailored to your data. This is a hands-off, AI-powered feature that allows you to have a first contract draft that is made to fit your dataset's columns.
[Optional] Schedule generated contracts
You can add a schedule to generated contracts to define when Soda will run them, including frequency and time. Default is daily at 00:00 UTC.
[Optional] Enable Metric Monitoring & Profiling
Through Metric Monitoring, you can enable built-in monitors to automatically track row counts, schema changes, freshness, and more across your datasets. This step is optional but recommended. This can be enabled in bulk when onboarding data sources and datasets.
Learn more about Metric Monitoring: Metric Monitoring dashboard
Toggle on Metric Monitoring
When metric monitoring is enabled it's possible to later add column monitors on dataset level or overwrite any of the settings.
Set a Monitoring Schedule
The monitoring schedule defines when Soda scans a dataset to capture and evaluate metrics. While scans may run slightly later due to system delays, Soda uses the actual execution time, not the scheduled time, when visualizing time-sensitive metadata metrics like insert lag or row count deltas. This ensures accuracy.
Data-based metrics like averages or null rates are not affected by small delays, as Soda only scans complete partitions, keeping these metrics stable and reliable.
Scans can be scheduled to occur from hourly to weekly, depending on your needs.
Learn more about how to pick a scan time.
Toggle on/off Historical Metric Collection
When Historical Metric Collection is enabled, Soda automatically calculates past data quality metrics through backfilling and applies the anomaly detection algorithm to that historical data through backtesting. This gives you immediate visibility into past data quality issues, even before monitoring was activated.
The historical data also helps train the anomaly detection algorithm, improving its accuracy from day one. You can specify a start date to control how far back the backfilling process should begin.
Suggest a Time Partition Column
Metrics that are not based on metadata require a time partition column to group data into daily intervals or 24-hour buckets, depending on the monitoring schedule. This column must be a timestamp field, ideally something like a created_at or last_updated column. It's important that this timestamp reflects when the data arrives in the database, rather than when the record was originally created.
Soda uses a list of suggested time partition columns to determine which column to apply. If multiple columns are suggested, Soda checks them in the order they are listed, starting with the first. It will try to match one by validating that the column is a proper timestamp and suitable for partitioning.
If none of the suggested columns match, Soda falls back to a heuristic approach. This heuristic looks at metadata, typical naming conventions, and column content to infer the most likely time partition column.
If the heuristic fails to find a suitable column or selects the wrong one, the time partition column can be manually configured after onboarding under dataset settings.
[Optional] Toggle on Advanced monitor configuration
Enable or disable each monitor to suit your data quality needs.
Click on Next
Define a dataset owner
Define a dataset owner by choosing a user from the drop-down menu.
[Optional] Enable check results & scans collection
From this onboarding view, you can also enable check results & scans collection if Diagnostics Warehouse is enabled for this data source.
[Optional] Enable failed row collection
From this onboarding view, you can also enable Failed row collection if Diagnostics Warehouse is enabled for this data source. When you enable Collect failed rows, select a storage and a detection strategy.
Profiling scheduled execution is set at 12:00AM UTC by default.
[Optional] Enable Profiling
Learn more about Profiling.
Click on Finish
If you used Rules-based selection to onboard datasets, an Active Onboarding Rule Pipeline view will appear now to confirm the conditions.
Access the datasets
Once onboarding is completed, your data source will appear in the Data Sources list. You can click the Onboarded Datasets button to access the connected datasets.
Soda Cloud will now spin up its Soda-hosted Runner and perform an initial Profiling & Historical Metric Collection scan, if configured. This usually takes only a few minutes.
🎉 Congrats! You've successfully onboarded your data source and datasets. You're now ready to create data contracts and start monitoring the quality of your data.
To generate a first contract for the datasets you just onboarded, refer to Generate a contract with Autopilot.
Onboard datasets from an existing data source
Note that you can repeat the datasets onboarding process at any time to add more datasets from the same data source. Datasets that previously have been onboarded will not re-appear in the data selection step.
Simply return to the data source page and click Onboard Datasets to update your selection. You will be prompted to follow the dataset onboarding steps.
You need the Manage data sources global permission to add a new data source. Learn about Global and Dataset Roles
Connect a data source onboarded with Soda Core to a Runner
When you create or push a new data source from Soda Core, it becomes available in Soda Cloud, but it is not automatically connected to a Soda Runner.
Connecting the data source to a Runner enables Soda Cloud features that require a Runner runtime, such as:
Metric Monitoring
Profiling
Running Data Contracts on the Soda Runner (via scheduling and the Soda Cloud interface)
Locate the partially onboarded data source in Soda Cloud
Navigate to Data Sources. Find the data source that was created or pushed from Soda Core.
Click on "⋮" > Edit connection.
Edit the connection and select a Runner
Fill in the connection form:
Select the Runner you want to use (Soda-hosted or self-hosted)
Provide the required connection details and credentials (use Soda Secrets for sensitive values)
Click Test Connection.
Click Connect to save the configuration.
Enable Runner-powered features on your datasets
Once the data source is connected to a Runner, you can enable Runner-powered features on your datasets, including:
Metric Monitoring
Profiling
Running Contracts on the Runner
Configure discovery scan
By default, Soda runs discovery scans once per day at a random time; if you need a different cadence, you can configure a custom schedule for each data source.
Navigate to Scans and click on the scan you want to configure.
Click on Edit Scan Definition (top right).
Edit the schedule.
You can choose to trigger the scan on schedule or externally using a Soda Library CLI command.
Click on Save.
You are not logged in to Soda and are viewing the default public documentation. Learn more about Documentation access & licensing.
If you do have a Soda license, make sure to log in to Soda Cloud in this same browser.
Last updated
Was this helpful?