Use a cross check to compare row counts between datasets within the same, or different, data sources.
checks for dim_customer: # Check row count between datasets in one data source - row_count same as dim_department_group # Check row count between datasets in different data sources - row_count same as retail_customers in aws_postgres_retail
In the context of SodaCL check types, cross checks are unique. This check employs the
row_count metric but is limited in its syntax variation, with only a few mutable parts to specify dataset and data source names.
The example check below compares the volume of rows in two datasets in the same data source. If the row count in the
dim_department_group is not the same as in
dim_customer, the check fails.
checks for dim_customer: - row_count same as dim_department_group
You can use cross checks to compare row counts between datasets in different data sources, as in the example below.
In the example,
retail_customers is the name of the other dataset, and
aws_postgres_retail is the name of the data source in which
checks for dim_customer: - row_count same as retail_customers in aws_postgres_retail
- If you wish to compare row counts of datasets in different data sources, you must have configured a connection to both data sources in your configuration YAML file. Soda needs access to both data sources in order to execute a cross check between data sources.
- The data sources do not need to be the same type; you can compare a dataset in a PostgreSQL data source to a dataset in a BigQuery data source.
|✓||Define a name for a cross check; see example.||Customize check names|
|Define alert configurations to specify warn and fail alert conditions.||-|
|Apply a filter to return results for a specific portion of the data in your dataset.||-|
|✓||Use quotes when identifying dataset or column names; see example||Use quotes in a check|
|Use wildcard characters ( % or * ) in values in the check.||-|
|Use for each to apply schema checks to multiple datasets in one scan.||-|
|Apply a dataset filter to partition data during a scan; see example.||-|
checks for dim_customer: - row_count same as retail_customers in aws_postgres_retail: name: Cross check customer datasets
checks for dim_customer: - row_count same as "dim_department_group"
- Learn more about SodaCL metrics and checks in general.
- Use a schema check to discover missing or forbidden columns in a dataset.
- Need help? Join the Soda community on Slack.
Last modified on 01-Jul-22
Was this documentation helpful?
Share feedback in the Soda community on Slack.
Help improve our docs!