> For the complete documentation index, see [llms.txt](https://docs.soda.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soda.io/guides-and-tutorials/data-platform/double-onboard-a-data-source.md).

# Double-onboard a data source

Publish a contract from Soda Core, then connect the data source to a Soda Runner in Soda Cloud to use Metric Monitoring, Profiling, and contract verification on the runner.

A data source reaches Soda Cloud in one of two ways. When you publish a contract or verification results from Soda Core, Soda Cloud creates the data source and the dataset for you, but does not connect the data source to a Soda Runner. When you add a data source in Soda Cloud, you connect it through a runner.

This guide does both for the same data source. A team starts with Soda Core, verifying contracts from the command line with a local configuration file. Later, they connect the same data source to a runner in Soda Cloud. The runner unlocks the features that run in Soda Cloud:

* Metric Monitoring
* Profiling
* Running contracts on the runner, from the Soda Cloud interface or on a schedule

Soda Core keeps working as before: the team can go on verifying contracts locally and publishing the results.

The examples use a PostgreSQL data source and its `fact_internet_sales` table.

## Prerequisites

* Python 3.10, 3.11, or 3.12, and a read-only user for the PostgreSQL database.
* A Soda Cloud account and an API key. See [Generate API keys](https://docs.soda.io/apis/generate-api-keys.md).
* To publish a contract for a new data source, the global permission **Create new datasets and data sources with Core/Library**. To connect the data source to a runner, permission to manage the data source's settings. See [Global and dataset roles](https://docs.soda.io/soda-documentation/govern/organization-and-admin-settings/global-and-dataset-roles.md).
* A runner that can reach the database: the Soda-hosted runner, or a self-hosted runner in your own environment. To use the Soda-hosted runner, allow its IP address in your network. See [Deployment options](https://docs.soda.io/soda-documentation/connect/deployment-options.md).

## Install Soda Core

Install the Soda Core package for PostgreSQL:

```bash
pip install -i https://pypi.cloud.soda.io/simple "soda-postgres>4"
```

> For other data sources and the private PyPI, see [Soda Python libraries](https://docs.soda.io/soda-documentation/connect/deployment-options/soda-python-libraries.md).

## Connect Soda Core

Create a data source configuration file. The `name` becomes the name of the data source in Soda Cloud. The values in `${env.…}` are read from environment variables.

{% code title="ds\_config.yml" %}

```yaml
type: postgres
name: sales_warehouse
connection:
  host: ${env.POSTGRES_HOST}
  port: ${env.POSTGRES_PORT}
  user: ${env.POSTGRES_USER}
  password: ${env.POSTGRES_PASSWORD}
  database: sales_dw
```

{% endcode %}

Create a Soda Cloud configuration file. Use `cloud.us.soda.io` or `cloud.soda.io`, depending on your Soda region.

{% code title="sc\_config.yml" %}

```yaml
soda_cloud:
  host: cloud.us.soda.io
  api_key_id: ${env.SODA_CLOUD_API_KEY_ID}
  api_key_secret: ${env.SODA_CLOUD_API_KEY_SECRET}
```

{% endcode %}

Test both connections:

```bash
soda data-source test -ds ds_config.yml
soda cloud test -sc sc_config.yml
```

> For every connection parameter, see [PostgreSQL](https://docs.soda.io/soda-documentation/developer-tools/data-source-reference-for-soda-core/postgresql.md).

## Publish a contract from Soda Core

Write a contract for the table. Its `dataset` starts with the data source `name` from `ds_config.yml`, followed by the database, the schema, and the table.

{% code title="contracts/fact\_internet\_sales.yml" %}

```yaml
dataset: sales_warehouse/sales_dw/public/fact_internet_sales

columns:
  - name: sales_order_number
    checks:
      - missing:
          name: All sales have an order number
      - duplicate:
          name: Order numbers are unique
  - name: product_key
    checks:
      - missing:
          name: All sales have a product
  - name: order_quantity
  - name: sales_amount
    checks:
      - failed_rows:
          name: All sales have a non-zero order amount
          expression: sales_amount <= 0

checks:
  - row_count:
      name: Sales are present
      threshold:
        must_be_greater_than: 0
```

{% endcode %}

Check that the contract is valid, then verify it and publish the contract and its results to Soda Cloud:

```bash
soda contract test -c contracts/fact_internet_sales.yml
soda contract verify -ds ds_config.yml -c contracts/fact_internet_sales.yml -sc sc_config.yml --publish
```

Soda Core runs the checks locally and sends the results to Soda Cloud, which creates the data source and the dataset. The data source has no connection details in Soda Cloud and no runner, so the features that need a runner are not available yet. On the dataset's **Metric Monitors** tab, Soda Cloud explains that the metric monitoring dashboard is only compatible with data sources connected through a Soda Runner.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-86170332e3d24d805d2bb4286621910adf5f70fc%2Fdouble-onboard-a-data-source-metric-monitors-before.png?alt=media" alt="The Metric Monitors tab of fact_internet_sales, with a message that the data source is connected to Soda Cloud via Soda Library and needs a Soda Runner"><figcaption><p>A dataset published from Soda Core, before the data source has a runner</p></figcaption></figure>

> For more on this flow, see [Onboard a data source via Soda Core](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/via-soda-core.md).

## Connect the data source to a runner

To complete the onboarding, give the data source a runner and connection details in Soda Cloud.

{% stepper %}
{% step %}

### Open the connection

In Soda Cloud, go to **Data Sources**. In the row of the data source that Soda Core created, click **⋮** > **Edit Connection**.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-3ac1a7c94cd382badd33bd23dd9bdef890326305%2Fdouble-onboard-a-data-source-edit-connection-menu.png?alt=media" alt="The Data Sources page with the row menu of the data source open, showing Edit Connection"><figcaption><p>The data source's row menu</p></figcaption></figure>
{% endstep %}

{% step %}

### Select a runner

The **Edit Connection** panel keeps the data source's name and owner. Under **Default Runner**, select the runner that connects to the data source in **Default Scan Runner**.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-6587dac2d7cad21586eff0b12f19bbf7028923bd%2Fdouble-onboard-a-data-source-edit-connection-runner.png?alt=media" alt="The Edit Connection panel with the Default Scan Runner list open, showing the Soda-hosted runner"><figcaption><p>Select the runner</p></figcaption></figure>
{% endstep %}

{% step %}

### Enter the connection details

Under **Connection Details**, fill in the same connection details that Soda Core uses: the host, port, user, password, and database.

For the password, use a Soda secret: click **Secret** next to the field. In the **Secrets** dialog, click **Use** next to an existing secret, or enter a **Name** and a **Value** and click **Create & Use**. Soda Cloud stores secrets encrypted, so the password does not appear in plain text in the connection configuration.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-29629ec507f3f8c06f2deb9ccb92df824c36d538%2Fdouble-onboard-a-data-source-edit-connection-details.png?alt=media" alt="The Connection Details section of the Edit Connection panel, with Host, Port, User, Password, and Database fields, each with a Secret button"><figcaption><p>Connection details</p></figcaption></figure>
{% endstep %}

{% step %}

### Test and save

Click **Test connection**. When the test passes, click **Save**.
{% endstep %}
{% endstepper %}

The data source keeps its name, so the contract, the dataset, and the check results that Soda Core published stay where they are. After you save, Soda runs a discovery scan on the runner to list the datasets in the data source. On the **Data Sources** page, **Onboard Datasets** is now available for the data source, to onboard the other datasets that discovery found.

On the dataset's **Metric Monitors** tab, the message about Soda Library is gone. Click **Update Metric Monitoring Configuration** to turn on Metric Monitoring for the dataset.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-a3c1cd2d42a49bcf39f133a78e7152fd74ea9b21%2Fdouble-onboard-a-data-source-metric-monitors-after.png?alt=media" alt="The Metric Monitors tab of fact_internet_sales after the data source is connected to a runner, with the button Update Metric Monitoring Configuration"><figcaption><p>Metric Monitoring is available once the data source has a runner</p></figcaption></figure>

> For every field of the connection form, see [Onboard datasets on Soda Cloud](https://docs.soda.io/soda-documentation/connect/onboard-data-sources-and-datasets/onboard-datasets-on-soda-cloud.md).

## Verify the contract on the runner

The runner can now verify the published contract. From the command line, add `--use-runner` and name the dataset instead of the contract file. The command needs only the Soda Cloud configuration, because the runner holds the connection details. Add `--publish` to record the results on the dataset; without it, the runner runs the contract as a test.

```bash
soda contract verify --use-runner -d sales_warehouse/sales_dw/public/fact_internet_sales -sc sc_config.yml --publish
```

The results appear on the dataset's **Checks** tab and its **Scans** tab, next to the results Soda Core published. To verify the contract from Soda Cloud instead, click **Verify all checks** on the **Checks** tab.

<figure><img src="https://1928210810-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FJJOow5YZtrrxZJBNXV2D%2Fuploads%2Fgit-blob-00e09d926d891f65d5beba9e3e117a09c7f04430%2Fdouble-onboard-a-data-source-dataset-checks.png?alt=media" alt="The Checks tab of fact_internet_sales, with the five checks of the contract passing"><figcaption><p>Contract results of <code>fact_internet_sales</code></p></figcaption></figure>

> To verify a contract from the dataset page or on a schedule, see [Verify a contract](https://docs.soda.io/soda-documentation/find/data-testing/cloud-managed-data-contracts/verify-a-contract.md).

## Next steps

* Turn on Metric Monitoring for the dataset. See [Metric monitoring dashboard](https://docs.soda.io/soda-documentation/find/data-observability/metric-monitoring-dashboard.md).
* Collect column statistics with Profiling. See [Profiling](https://docs.soda.io/soda-documentation/find/data-observability/profiling.md).
* Keep publishing contracts from Soda Core. See [Core CLI reference](https://docs.soda.io/soda-documentation/developer-tools/cli-reference.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.soda.io/guides-and-tutorials/data-platform/double-onboard-a-data-source.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
