Author a Data Standard

Define the checks that make up a Data Standard.

A Data Standard is authored as a set of data quality checks, using the same check language as a Data Contract. If you can write contract checks, you can write a Data Standard: the columns: list, dataset-level checks:, thresholds, filters, Reconciliation checks, and variables all work exactly as they do in a contract.

You focus on the checks. Soda manages the rest, the standard's identity and the datasets it applies to, so those are not part of the check definition you write.

For the full catalogue of check types and configuration options, see the Contract Language reference.

Author a Data Standard

A Data Standard's definition is a columns: list and/or a dataset-level checks: block, just like a contract:

data-standard.yml

columns:
  - name: email
    checks:
      - missing:
      - invalid:
          valid_format:
            name: Email format
            regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$

The Data Standard editor: basic information and scope on the left, the standard's checks on the right.

---

Match columns by pattern or attribute

A Data Standard runs across many datasets, where the same kind of data can live under different column names. Rather than listing explicit column names, you can match columns dynamically using a match: block in place of name: on a column entry. The checks then apply to every column that matches, in every dataset the standard runs against.

A standard using a match block to apply its checks to columns selected by attribute.

match: is resolved per dataset. When a standard runs against a dataset, Soda expands each match: block into concrete columns using that dataset's real column names and attributes, so the same standard can apply to differently-named columns across your fleet.

A match: block selects columns by name and/or by attribute, and an optional exclude: block (same grammar) removes columns from the result. A column matches only if it satisfies both the name predicate and the attribute predicate present in the block.

Match by name

Use pattern (a regular expression), any_of (an explicit list of column names), or the keyword all (every column). pattern and any_of may be combined — a column matches if it satisfies either.

data-standard.yml

columns:
  # every column whose name contains "email" (case-insensitive), minus one
  - match:
      pattern: "(?i).*email.*"
    exclude:
      any_of: [email_archived]
    checks:
      - missing:
          qualifier: c1
      - invalid:
          qualifier: c2
          valid_format:
            name: Email Regex
            regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$

  # a fixed set of columns by name
  - match:
      any_of: [customer_id, account_id]
    checks:
      - missing:
          qualifier: id_missing

  # every column in the dataset
  - match: all
    checks:
      - missing:
          qualifier: no_nulls

exclude accepts the full match grammar (pattern, any_of, attributes), so you can exclude by pattern or attribute, not only by an explicit list.

Match by attribute

Use attributes to match columns that carry a specific attribute value. A leaf condition names an attribute and either equals a single value or matches any_of a list of values:

data-standard.yml

columns:
  # single value
  - match:
      attributes: { name: term, equals: Email }
    checks:
      - missing:

  # any of several values
  - match:
      attributes: { name: term, any_of: [Email, Contact] }
    checks:
      - missing:

Combine several conditions with all_of (every condition must hold) or any_of (at least one must hold). You can also combine a name predicate and an attribute predicate in one block — both must match:

data-standard.yml

columns:
  - match:
      pattern: "(?i).*email.*"
      attributes:
        all_of:
          - { name: term, equals: Email }
          - { name: pii, equals: true }
    checks:
      - missing:
      - invalid:
          valid_format:
            name: Email Regex
            regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$

A single match can expand to several columns, and each expanded check needs a stable identity. Use a qualifier to distinguish checks of the same type so their results are tracked consistently over time.

Standards generated in Soda Cloud populate qualifiers for you. Learn more about Check qualifiers.

---

Reference the current column

A match: block copies the same check text to every column it matched, so a column name written into a check would be wrong for all but one of them. Write ${soda.data-standards.CURRENT_COLUMN} instead, and Soda replaces it with the name of the column that entry resolved to.

data-standard.yml

columns:
  - match:
      pattern: "(?i).*_amount"
    checks:
      - failed_rows:
          name: "${soda.data-standards.CURRENT_COLUMN} must be non-negative"
          expression: "${soda.data-standards.CURRENT_COLUMN} >= 0"

Against a dataset with a net_amount and a gross_amount column, that single entry produces two checks: net_amount must be non-negative, testing net_amount >= 0, and gross_amount must be non-negative, testing gross_amount >= 0.

Soda replaces the token in every text value inside the column entry, however deeply it is nested: check names, expression, filter, query, entries in lists such as valid_values, and column attributes. A value can hold the token more than once and combine it with other text.

data-standard.yml

columns:
  - match:
      any_of: [score, rating]
    checks:
      - failed_rows:
          expression: "${soda.data-standards.CURRENT_COLUMN} > 0 AND ${soda.data-standards.CURRENT_COLUMN} < 100"

The token is resolved after match: expands into concrete columns, so every expanded column carries its own name. It also works in a column entry written with name:, where it saves you repeating the name.

The token resolves only inside a columns: entry. A dataset-level checks: block has no current column, so the token stays exactly as written and reaches the check engine as literal text. Nothing rejects a misplaced or misspelled token when you save the standard.

---

Targeting datasets

References to specific checks do not take place in the checks of a Data Standard. Instead, the datasets a standard applies to are determined by its scope, which you configure when you deploy the standard in Soda Cloud. Soda resolves the scope and runs the standard's checks against every matching dataset. See Deploy at scale.

---

Example: a fuller standard

This standard enforces several common quality rules. It can be deployed across every dataset that holds customer contact data.

data-standard.yml

columns:
  - name: email
    checks:
      - missing:
      - invalid:
          name: Email format
          valid_format:
            name: Email format
            regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$
  - name: phone
    checks:
      - invalid:
          valid_format:
            name: Phone format
            regex: ^\+?[0-9\-\s()]+$
  - name: customer_id
    checks:
      - missing:
      - duplicate:

checks:
  - row_count:

---

Supported fields

Data Standards support the same fields as Data Contracts:

Dataset-level checks

Like contracts, standards support dataset-level checks such as schema, row_count, freshness, and duplicate across multiple columns. A standard that defines only dataset-level checks still requires a columns: key; an empty list is enough.

data-standard.yml

checks:
  - schema:
      allow_extra_columns: false
  - row_count:

columns: []

Variables

Standards support the same ${var.VARIABLE_NAME} variable substitution as contracts, which is useful when the same rule needs slightly different parameters per environment or schedule. Declare variables at the top of the definition and pass values at verification time with --set. To substitute the name of the column a check runs on, use ${soda.data-standards.CURRENT_COLUMN} instead: see Reference the current column.

Learn more about variables: Make contracts dynamic with variables.

---

Next steps

• Deploy the standard across your datasets: Deploy at scale
• Verify it from the CLI: Verify a Data Standard

---

You are not logged in to Soda and are viewing the default public documentation. Learn more about Licensing & documentation access.

If you do have a Soda license, make sure to log in to Soda Cloud in this same browser.