> For the complete documentation index, see [llms.txt](https://docs.soda.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soda.io/guides-and-tutorials/data-governance/manage-sensitive-data.md).

# Manage sensitive data

Control who sees your data in Soda Cloud, where Soda runs its queries, and which data leaves your data warehouse.

This guide shows the Soda Cloud settings that limit the exposure of sensitive data. Soda reads your data to verify contracts, profile columns, and collect failed rows. Each of these leaves a trace in Soda Cloud or in your warehouse, and each has a setting that controls who can see it or turns it off.

Most of these settings are in **Settings** > **Organization** and require the **Manage organization settings** global permission.

{% hint style="info" %}
The self-hosted Soda Runner, Diagnostics Warehouse, and SCIM provisioning are available on license plans. Need access? [Contact us](mailto:support@soda.io).
{% endhint %}

## Data exposure and controls

| Exposure                       | What Soda Cloud holds or does                                                        | Control                                                     |
| ------------------------------ | ------------------------------------------------------------------------------------ | ----------------------------------------------------------- |
| Access to datasets and results | Datasets, checks, check results, and incidents, visible to users of the organization | [Roles and permissions](#roles-and-permissions)             |
| Access to Soda Cloud           | Sign-in from any network, with accounts managed in Soda Cloud                        | [Sign-in and network access](#sign-in-and-network-access)   |
| Queries and credentials        | The Soda Runner that queries your data, and the data source credentials it uses      | [Soda Runner and credentials](#soda-runner-and-credentials) |
| Datasets in scope              | The datasets Soda onboards and monitors                                              | [Dataset selection](#dataset-selection)                     |
| Column statistics              | Profiling results, including frequent and extreme values                             | [Profiling](#profiling)                                     |
| Failed rows                    | Rows that fail a check, stored in your warehouse or as samples in Soda Cloud         | [Failed rows](#failed-rows)                                 |
| AI features                    | Requests sent to a model provider                                                    | [Soda AI](#soda-ai)                                         |

## Roles and permissions

Soda Cloud grants access through roles at three levels:

* **Global roles** set what a user can do across the organization, such as create data sources or manage organization settings.
* **Data source roles** set who can see a data source and everything onboarded from it. A user without a role on a data source sees none of its datasets, checks, or results.
* **Dataset roles** set what a user can do on one dataset. Users get a dataset role through the dataset's responsibilities.

Manage roles in **Settings** > **Organization** > **Roles**, on the **Global Roles**, **Data Source Roles**, and **Dataset Roles** tabs.

Two dataset permissions control access to data itself:

| Permission                               | Grants access to                           |
| ---------------------------------------- | ------------------------------------------ |
| **Access dataset profiling and samples** | Profiling results of the dataset           |
| **Access failed row samples for checks** | Failed row samples of the dataset's checks |

The default **Viewer** dataset role includes both. For users who only need check results, create a custom dataset role without these two permissions with **Add Dataset Role**, and assign it on the dataset's **Settings** tab, under **Responsibilities**. The same tab lists your own permissions on the dataset under **My permissions**.

New datasets get the assignments in the **Responsibilities** panel of the **Dataset Roles** tab, under **For newly onboarded Datasets**. When the **Everyone** group has an assignment there, every user gets that role on each new dataset. To narrow this, give the **Everyone** group a role with fewer permissions, or remove the assignment.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-fd601677f43a73fc27ba5d5d73782790cbfecdf2%2Fmanage-sensitive-data-dataset-roles.png?alt=media" alt="The Dataset Roles tab in Soda Cloud settings, with the Editor, Manager and Viewer roles and the Responsibilities panel that assigns a role to the Everyone group for newly onboarded datasets"><figcaption><p>Dataset roles and the default responsibilities for new datasets</p></figcaption></figure>

{% hint style="info" %}
Data source roles are available to all new organizations.

If your existing organization doesn't have them yet, the **Data Source Roles** tab shows **Onboard Data Source Roles**. To opt in, [contact us](mailto:support@soda.io).
{% endhint %}

> For the permissions of each role and how to create custom roles, see [Global and dataset roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles.md).

## Sign-in and network access

* **Single sign-on (SSO):** Connect Soda Cloud to your SAML 2.0 identity provider, so users sign in with their company account and lose access when your IT team revokes it. See [User and user group management with SSO](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/user-and-user-group-management-with-sso.md).
* **SCIM provisioning:** Let Microsoft Entra ID create and remove Soda Cloud users and user groups. SCIM is an opt-in feature on Enterprise plans. See [User provisioning with SCIM](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/user-provisioning-with-scim.md).
* **IP allowlist:** Accept requests to Soda Cloud only from a list of IP addresses or CIDR ranges, such as your corporate network or VPN. Requests from other addresses are blocked before authentication. In **Settings** > **Organization** > **General**, under **IP allowlist**, check **Restrict access to specific IP addresses** and add the addresses. See [IP Allowlist](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/general-settings.md#ip-allowlist).

{% hint style="danger" %}
An IP allowlist takes effect as soon as you save it. Add your own IP address to the list before you save, or you lose access to Soda Cloud.
{% endhint %}

## Soda Runner and credentials

A Soda Runner connects to your data sources and runs the queries for contract verifications, profiling, and metric monitoring. Choose where it runs:

* **Soda-hosted Runner:** Soda runs it in Soda's infrastructure. To reach your data, it needs network access to your data source, through an allowlist of its IP address.
* **Self-hosted Runner:** You deploy it with Helm in your own Kubernetes cluster, so queries run inside your network. The runner reads data source credentials from Kubernetes Secrets, which you can back with an external secrets manager. See [Deploy a Soda Runner](broken://spaces/A2PmHkO5cBgeRPdiPPOG/pages/DVzqeL43wc3c9AsZ4TFi).

A self-hosted runner sends check names, metrics, and aggregated results to Soda Cloud. It sends no row-level data unless [failed row samples](#failed-row-samples) are on. See [Data flows between Soda & user](https://docs.soda.io/soda-documentation/govern/security-licensing-and-compliance/data-flows-between-soda-and-user.md).

Once a self-hosted runner is in place, turn off the Soda-hosted Runner so that no scan runs outside your network. In **Settings** > **Organization** > **General**, uncheck **Enable Soda-hosted Runner**. **Settings** > **Runners** lists the runners of your organization. See [Enable or disable the Soda-hosted Runner](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/general-settings.md#enable-or-disable-the-soda-hosted-runner).

For data sources that you connect in Soda Cloud, store passwords, tokens, and keys as secrets instead of typing them into the data source configuration. Soda Cloud stores secrets encrypted, and the configuration keeps only a `${secret.NAME}` placeholder. A credential typed as a literal value stays in the configuration, where users who can manage the data source settings can read it.

Add secrets with **New Secret** in **Settings** > **Secrets**, then reference them in the connection configuration. The list shows each secret's name and dates, never its value.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-c6eb04b73954939453515e173a3cba73c579145e%2Fmanage-sensitive-data-secrets.png?alt=media" alt="The Secrets page in Soda Cloud settings, listing secrets by name with their created and last updated dates"><figcaption><p>Secrets are listed by name only</p></figcaption></figure>

The **Secrets** section of **Settings** > **Organization** > **General** has an **Enable Secrets** option, which must be checked to use secrets.

```yaml
connection:
  host: <your_host>
  user: <your_user>
  password: ${secret.POSTGRES_PASSWORD}
```

> See [Secure your credentials with secrets](https://docs.soda.io/spaces/A2PmHkO5cBgeRPdiPPOG/pages/cHo9ixopiR0flOMCbyUF#id-2.3.-secure-your-credentials-with-secrets).

To keep data source connections out of Soda Cloud entirely, verify contracts with the Soda Python libraries in your own pipelines. Their data source connections are configured in your environment. See [Deployment options](https://docs.soda.io/soda-documentation/connect/deployment-options.md).

## Dataset selection

Soda checks, profiles, and monitors only the datasets you onboard. When you onboard datasets from a data source, choose one of two selection strategies:

* **Manual:** Pick the datasets one by one.
* **Rules-based:** Add rules that include or exclude datasets by name, for example to leave out every table whose name starts with `pii_`. The rules also apply to datasets created later.

Soda still runs a discovery scan on the data source once a day, which reads the names of its tables and schemas.

> See [Onboard datasets on Soda Cloud](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud.md).

## Profiling

Profiling computes statistics for each column of a dataset and shows them in Soda Cloud: distinct and missing counts, minimum, maximum, and mean, a histogram, frequent values, and extreme values. Frequent and extreme values come from the data itself, so on a column with personal data they show that data.

Turn profiling off at one of two levels:

* **Organization:** In **Settings** > **Organization** > **General**, under **Profiling**, uncheck **Allow Soda to collect column profile information**. Soda then collects no profiling information for any dataset. Check and monitor results and the dataset schema are still processed.
* **Dataset:** Open the dataset, go to the **Profiling** tab, click **Update Profiling Configuration**, and turn profiling off.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-a78f0b57e23c1b0446804414bf1b8bb4fd2f8393%2Fmanage-sensitive-data-dataset-profiling.png?alt=media" alt="The Profiling tab of a dataset in Soda Cloud, with the Update Profiling Configuration button and the distinct and missing counts of each column"><figcaption><p>The Profiling tab of a dataset</p></figcaption></figure>

Profiling settings apply to the whole dataset, not to single columns. To hide profiling results from some users instead, use a dataset role without **Access dataset profiling and samples**.

> See [Profiling](https://docs.soda.io/soda-documentation/find/data-observability/profiling.md).

## Failed rows

When a check fails, Soda can store the rows that failed it. Two settings decide where they go.

### Diagnostics Warehouse

Diagnostics Warehouse stores failed rows in a schema of your own data warehouse, so they don't leave it. Soda Cloud shows a message on the check's **Failed Rows** tab that says where the rows are, and can show the failed rows query and a button that links to your own tool.

Set it up per data source with **Edit Diagnostics Warehouse**. Then, for each dataset, open the dataset's **Settings** tab and, under **Diagnostics Warehouse Configuration**, turn on **Collect failed rows** and choose a **Storage strategy**:

* **Store full failed rows** keeps the whole rows.
* A unique keys strategy keeps only the keys of the failed rows. To see the rows, run the failed rows query from the check's **Failed Rows** tab in your warehouse.

Under **Guardrails**, **Protect against data floods** and **Limit failed row storage** cap the number of rows that one check can store. AI features have no access to failed rows in Diagnostics Warehouse.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-6dc35af89c5bdc10502c71a229adbce71c520eed%2Fmanage-sensitive-data-dataset-diagnostics-warehouse.png?alt=media" alt="The Diagnostics Warehouse Configuration section of a dataset&#x27;s Settings tab, with Collect failed rows on, the storage strategy Store full failed rows, a unique key column, and the guardrails"><figcaption><p>Failed row collection for one dataset</p></figcaption></figure>

> See [Configure Diagnostics Warehouse](broken://spaces/A2PmHkO5cBgeRPdiPPOG/pages/E1KTiEL4c6zytpedbEZg).

### Failed row samples

Failed row samples store a subset of the failed rows in Soda Cloud, to show them on the check's **Failed Rows** tab. They are off until the Soda support team enables them for your organization. Once enabled, allow them per data source in **Edit Diagnostics Warehouse** > **Failed Row Samples** with **Allow sending failed row samples to Soda Cloud**, and override the setting per dataset under **Diagnostics Warehouse Configuration** on the dataset's **Settings** tab. The default sample size is 100 rows.

Failed row settings apply to a data source or a dataset, not to single columns. Leave failed row samples off for datasets with sensitive columns, and use a dataset role without **Access failed row samples for checks** for users who don't need them.

> See [Failed row samples](broken://spaces/A2PmHkO5cBgeRPdiPPOG/pages/r1iLFOJA0mD4a674X3en).

## Soda AI

Soda AI features send requests to a model provider, currently OpenAI. They don't send row-level data, with one exception: Contract Autopilot can send a sample of up to 100 rows to improve the contracts it generates. Soda does not use your data, metadata, or prompts to train AI models.

Admins control AI features in **Settings** > **Organization** > **General**:

* On the **Organization** tab, under **Soda AI Features**, check or uncheck **Enable Soda AI Agent**, **Enable Contract Copilot powered by OpenAI**, and **Enable Contract Autopilot powered by OpenAI**.
* For Contract Autopilot, leave **Send row-level data to improve the quality of your auto-generated data contract** unchecked to send dataset metadata and computed profiling metrics only.
* On the **AI** tab, use **Add provider** to add your own OpenAI account (bring your own key), and choose it for each feature under **Feature Configuration**. Requests then go to your OpenAI project, under your retention settings.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-ed82b5a5679ab36eb09443183db57873cfacd901%2Fmanage-sensitive-data-organization-general.png?alt=media" alt="The General tab of the organization settings in Soda Cloud, with the options Enable Soda-hosted Runner, Restrict access to specific IP addresses, Allow Soda to collect column profile information, the Soda AI Features, and Enable Secrets"><figcaption><p>Organization settings for the Soda-hosted Runner, IP allowlist, profiling, Soda AI, and secrets</p></figcaption></figure>

> See [AI security, compliance & privacy](https://docs.soda.io/soda-documentation/soda-ai/ai-security-compliance-and-privacy.md) and [Bring your own key (BYOK)](https://docs.soda.io/soda-documentation/soda-ai/bring-your-own-key-byok.md).

{% hint style="info" %}
Coming from Soda v3? The `sample datasets`, `discover datasets`, and `profile columns` data source configurations, and custom samplers, don't apply to Soda v4. Choose datasets with dataset selection rules, turn profiling on or off per dataset, and keep failed rows in your warehouse with Diagnostics Warehouse.
{% endhint %}

## Next steps

* Review who holds which role on your most sensitive datasets in their responsibilities. See [Dataset attributes and responsibilities](https://docs.soda.io/soda-documentation/govern/dataset-attributes-and-responsibilities.md).
* Investigate failed rows inside your warehouse with [Diagnostics Warehouse](broken://spaces/A2PmHkO5cBgeRPdiPPOG/pages/UZCAHd8ZBgl01336NkZa).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.soda.io/guides-and-tutorials/data-governance/manage-sensitive-data.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
