For each

Last modified on 06-Dec-22

Use a for each configuration to execute checks against multiple datasets during a scan.

for each dataset T:
  datasets:
    - dim_products%
    - fact%
    - exclude fact_survey_response
  checks:
    - row_count > 0

Define a for each configuration
Limitations and specifics
Optional check configurations
For each results in Soda Cloud
Go further

Define a for each configuration

Add a for each section to your checks YAML file to specify a list of checks you wish to execute on multiple datasets.

Add a for each dataset T section header anywhere in your YAML file. The purpose of the T is only to ensure that every for each configuration has a unique name.
Nested under the section header, add two nested keys, one for datasets and one for checks.
Nested under datasets, add a list of datasets against which to run the checks. Refer to the example below that illustrates how to use include and exclude configurations and wildcard characters (%) .
Nested under checks, write the checks you wish to execute against all the datasets listed under datasets.

for each dataset T:
  datasets:
    # include the dataset 
    - dim_customers
    # include all datasets matching the wildcard expression
    - dim_products%
    # (optional) explicitly add the word include to make the list more readable
    - include dim_employee
    # exclude a specific dataset
    - exclude fact_survey_response
    # exclude any datasets matching the wildcard expression
    - exclude prospective_%
  checks:
    - row_count > 0

Limitations and specifics

For each is not compatible with dataset filters.
Soda Core dataset names matching is case insensitive.
If any of your checks specify column names as arguments, make sure the column exists in all datasets listed under the datasets heading.
To add multiple for each configurations in your checks YAML file, configure another for each section header with a different letter identifier, such as for each dataset R.

Optional check configurations

Supported	Configuration	Documentation
✓	Define a name for a for each check; see example.	Customize check names
✓	Add an identity to a check.	Add a check identity
✓	Define alert configurations to specify warn and fail alert conditions; see example.	Add alert configurations.
✓	Apply an in-check filter to return results for a specific portion of the data in your dataset; see example.
✓	Use quotes when identifying dataset or column names; see example. Note that the type of quotes you use must match that which your data source uses. For example, BigQuery uses a backtick (`) as a quotation mark.	Use quotes in a check
✓	Use wildcard characters ( % in values in the for each configuration; see example.	-
	Apply a dataset filter to partition data during a scan.	-

Example with check name

for each dataset T:
  datasets:
    - dim_employee

  checks:
    - max(vacation_hours) < 80:
        name: Too many vacation hours for US Sales

Example with alert configuration

for each dataset T:
  datasets:
    - dim_employee
    - dim_customer

  checks:
    - row_count:
        fail:
          when < 5
        warn:
          when > 10

Example with in-check filter

for each dataset T:
  datasets:
    - dim_employee

  checks:
    - max(vacation_hours) < 80:
        filter: sales_territory_key = 11

Example with quotes

for each dataset T:
  datasets:
    - dim_employee
    - "dim_customer"

  checks:
    - row_count > 1

Example with wildcard

for each dataset T:
  datasets:
    - dim_%

  checks:
    - row_count > 1

For each results in Soda Cloud

If you have connected Soda Core to a Soda Cloud account, Soda Core pushes the check results for each dataset to Soda Cloud as individual rows in the Check Results table. Filter the results by dataset to review dataset-specific results.

for each dataset T:
  datasets:
    - dim_employee
    - dim_customer

  checks:
    - row_count > 1

foreach-cloud

Go further

Need help? Join the Soda community on Slack.
Reference tips and best practices for SodaCL.

Was this documentation helpful?

What could we do to improve this page?

Suggest a docs change in GitHub.
Share feedback in the Soda community on Slack.

Last modified on 06-Dec-22