Author a Data Standard

Define the checks that make up a Data Standard.

A Data Standard is authored as a set of data quality checks, using the same check language as a Data Contract. If you can write contract checks, you can write a Data Standard: the columns: list, dataset-level checks:, thresholds, filters, , and variables all work exactly as they do in a contract.

You focus on the checks. Soda manages the rest, the standard's identity and the datasets it applies to, so those are not part of the check definition you write.

For the full catalogue of check types and configuration options, see the Contract Language reference.

Author a Data Standard

A Data Standard's definition is a columns: list and/or a dataset-level checks: block, just like a contract:

data-standard.yml

columns:
  - name: email
    checks:
      - missing:
      - invalid:
          valid_format:
            name: Email format
            regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$

The Data Standard editor: basic information and scope on the left, the standard's checks on the right.

---

Match columns by pattern or attribute

A Data Standard runs across many datasets, where the same kind of data can live under different column names. Rather than listing explicit column names, you can match columns dynamically, by name pattern or by attribute, using a match: block in place of name: on a column entry. The checks then apply to every column that matches, in every dataset the standard runs against.

A standard using a match block to apply its checks to columns selected by attribute.

Match by name pattern

Use pattern with a regular expression to match column names. The example below applies its checks to every column whose name contains email (case-insensitive, via the (?i) flag), and uses exclude to skip specific columns.

data-standard.yml

columns:
  - match:
      pattern: "(?i).*email.*"
    exclude:
      any_of: [email_archived]
    checks:
      - missing:
          qualifier: c1
      - invalid:
          qualifier: c2
          valid_format:
            name: Email Regex
            regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$

exclude.any_of takes a list of column names to omit from the match, even if they would otherwise match the pattern.

Match by attribute

Use attributes to match columns that carry a specific attribute value. The example below applies its checks to every column whose term attribute equals Email, regardless of the column's name.

data-standard.yml

columns:
  - match:
      attributes: { name: term, equals: Email }
    checks:
      - missing:
      - invalid:
          valid_format:
            name: Email Regex
            regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$

A single match can expand to several columns, and each expanded check needs a stable identity. Use a qualifier to distinguish checks of the same type so their results are tracked consistently over time.

Standards generated in Soda Cloud populate qualifiers for you. Learn more about Check qualifiers.

---

Targeting datasets

References to specific checks do not take place in the checks of a Data Standard. Instead, the datasets a standard applies to are determined by its scope, which you configure when you deploy the standard in Soda Cloud. Soda resolves the scope and runs the standard's checks against every matching dataset. See Deploy at scale.

---

Example: a fuller standard

This standard enforces several common quality rules. It can be deployed across every dataset that holds customer contact data.

data-standard.yml

columns:
  - name: email
    checks:
      - missing:
      - invalid:
          name: Email format
          valid_format:
            name: Email format
            regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$
  - name: phone
    checks:
      - invalid:
          valid_format:
            name: Phone format
            regex: ^\+?[0-9\-\s()]+$
  - name: customer_id
    checks:
      - missing:
      - duplicate:

checks:
  - row_count:

---

Supported fields

Data Standards support the same fields as Data Contracts:

Dataset-level checks

Like contracts, standards support dataset-level checks such as schema, row_count, freshness, and duplicate across multiple columns. A standard that defines only dataset-level checks still requires a columns: key; an empty list is enough.

data-standard.yml

checks:
  - schema:
      allow_extra_columns: false
  - row_count:

columns: []

Variables

Standards support the same ${var.VARIABLE_NAME} variable substitution as contracts, which is useful when the same rule needs slightly different parameters per environment or schedule. Declare variables at the top of the definition and pass values at verification time with --set.

Learn more about variables: Make contracts dynamic with variables.

---

Next steps

• Deploy the standard across your datasets: Deploy at scale
• Verify it from the CLI: Verify a Data Standard

---

{% if (visitor.claims.plan === 'datasetStandard') %}

You are logged in to Soda and seeing the Dataset Standard license documentation. Learn more about Documentation access & licensing.

{% if (visitor.claims.plan === 'enterprise') %}

You are logged in to Soda and seeing the Team license documentation. Learn more about Documentation access & licensing.

{% if (visitor.claims.plan === 'enterpriseUserBased') %}

You are logged in to Soda and seeing the Enterprise license documentation. Learn more about Documentation access & licensing.

{% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %}

You are not logged in to Soda and are viewing the default public documentation. Learn more about Documentation access & licensing.

If you do have a Soda license, make sure to log in to Soda Cloud in this same browser.