> For the complete documentation index, see [llms.txt](https://docs.soda.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soda.io/data-testing/data-standards/author-a-data-standard.md).
# Author a Data Standard
A Data Standard is authored as a set of data quality checks, using the **same check language as a Data Contract**. If you can write contract checks, you can write a Data Standard: the `columns:` list, dataset-level `checks:`, thresholds, filters, [Broken mention](broken://pages/QkO7w20yeaFP3U9Ow5Kk), and variables all work exactly as they do in a contract.
**You focus on the checks. Soda manages the rest**, the standard's identity and the datasets it applies to, so those are not part of the check definition you write.
> For the full catalogue of check types and configuration options, see the [Contract Language reference](/reference/contract-language-reference.md).
## Author a Data Standard
A Data Standard's definition is a `columns:` list and/or a dataset-level `checks:` block, just like a contract:
{% code title="data-standard.yml" %}
```yaml
columns:
- name: email
checks:
- missing:
- invalid:
valid_format:
name: Email format
regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$
```
{% endcode %}
The Data Standard editor: basic information and scope on the left, the standard's checks on the right.
***
## Match columns by pattern or attribute
A Data Standard runs across many datasets, where the same kind of data can live under different column names. Rather than listing explicit column names, **you can match columns dynamically** using a `match:` block in place of `name:` on a column entry. The checks then **apply to every column that matches**, in every dataset the standard runs against.
A standard using a match block to apply its checks to columns selected by attribute.
{% hint style="info" %}
**`match:` is resolved per dataset.** When a standard runs against a dataset, Soda expands each `match:` block into concrete columns using that dataset's real column names and attributes, so the same standard can apply to differently-named columns across your fleet.
{% endhint %}
A `match:` block selects columns by **name** and/or by **attribute**, and an optional `exclude:` block (same grammar) removes columns from the result. A column matches only if it satisfies both the name predicate and the attribute predicate present in the block.
### Match by name
Use `pattern` (a regular expression), `any_of` (an explicit list of column names), or the keyword `all` (every column). `pattern` and `any_of` may be combined — a column matches if it satisfies **either**.
{% code title="data-standard.yml" %}
```yaml
columns:
# every column whose name contains "email" (case-insensitive), minus one
- match:
pattern: "(?i).*email.*"
exclude:
any_of: [email_archived]
checks:
- missing:
qualifier: c1
- invalid:
qualifier: c2
valid_format:
name: Email Regex
regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$
# a fixed set of columns by name
- match:
any_of: [customer_id, account_id]
checks:
- missing:
qualifier: id_missing
# every column in the dataset
- match: all
checks:
- missing:
qualifier: no_nulls
```
{% endcode %}
`exclude` accepts the full match grammar (`pattern`, `any_of`, `attributes`), so you can exclude by pattern or attribute, not only by an explicit list.
### Match by attribute
Use `attributes` to **match columns that carry a specific attribute value**. A leaf condition names an attribute and either `equals` a single value or matches `any_of` a list of values:
{% code title="data-standard.yml" %}
```yaml
columns:
# single value
- match:
attributes: { name: term, equals: Email }
checks:
- missing:
# any of several values
- match:
attributes: { name: term, any_of: [Email, Contact] }
checks:
- missing:
```
{% endcode %}
Combine several conditions with `all_of` (every condition must hold) or `any_of` (at least one must hold). You can also combine a name predicate and an attribute predicate in one block — both must match:
{% code title="data-standard.yml" %}
```yaml
columns:
- match:
pattern: "(?i).*email.*"
attributes:
all_of:
- { name: term, equals: Email }
- { name: pii, equals: true }
checks:
- missing:
- invalid:
valid_format:
name: Email Regex
regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$
```
{% endcode %}
{% hint style="info" %}
**A single `match` can expand to several columns**, and each expanded check needs a stable identity. Use a `qualifier` to distinguish checks of the same type so their results are tracked consistently over time.
Standards generated in Soda Cloud populate qualifiers for you. Learn more about [Contract Language reference](/reference/contract-language-reference.md#check-qualifiers).
{% endhint %}
***
## Targeting datasets
References to specific checks do not take place in the checks of a Data Standard. Instead, the datasets a standard applies to are determined by its **scope**, which you configure when you deploy the standard in Soda Cloud. Soda resolves the scope and runs the standard's checks against every matching dataset. See [Deploy at scale](/data-testing/data-standards/deploy-at-scale.md).
***
## Example: a fuller standard
This standard enforces several common quality rules. It can be deployed across every dataset that holds customer contact data.
{% code title="data-standard.yml" %}
```yaml
columns:
- name: email
checks:
- missing:
- invalid:
name: Email format
valid_format:
name: Email format
regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$
- name: phone
checks:
- invalid:
valid_format:
name: Phone format
regex: ^\+?[0-9\-\s()]+$
- name: customer_id
checks:
- missing:
- duplicate:
checks:
- row_count:
```
{% endcode %}
***
## Supported fields
Data Standards support the same fields as Data Contracts:
### Dataset-level checks
Like contracts, **standards support dataset-level checks** such as `schema`, `row_count`, `freshness`, and `duplicate` across multiple columns. A standard that defines **only** dataset-level checks still requires a `columns:` key; an empty list is enough.
### Variables
Standards support the same `${var.VARIABLE_NAME}` variable substitution as contracts, which is useful when the same rule needs slightly different parameters per environment or schedule. Declare variables at the top of the definition and pass values at verification time with `--set`.
> Learn more about variables: [Contract Language reference](/reference/contract-language-reference.md#make-contracts-dynamic-with-variables).
***
## Next steps
* Deploy the standard across your datasets: [Deploy at scale](/data-testing/data-standards/deploy-at-scale.md)
* Verify it from the CLI: [Verify a Data Standard](/data-testing/data-standards/verify-a-data-standard.md)
***
{% if visitor.claims.plan === 'datasetStandard' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}
{% if visitor.claims.plan === 'enterprise' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}
{% if visitor.claims.plan === 'enterpriseUserBased' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}
{% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %}
{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**.
{% endhint %}
{% endif %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.soda.io/data-testing/data-standards/author-a-data-standard.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.