Sigma

Learn how to integrate Sigma with Soda.

Connect Soda Cloud to Sigma to build live reporting dashboards on top of your data quality results. You can create dashboards that visualize Soda check results, track failed rows over time, and report on the overall health of your data assets.

The integration works by exposing Soda data, such as check results, scan time, or failed rows. Once Sigma is connected to Soda or a Soda data source, such as a Diagnostics Warehouse, you can build and share dashboards directly in Sigma without any additional export or scripting steps.

You can use the Soda Cloud API to extract check and dataset results programmatically and write them to a data warehouse of your choice, which Sigma can then query.

Prerequisites

• Python 3.8+

• Pip 21.0+

• A Sigma account with permission to create workbooks and connect data sources

• A Soda Cloud account with at least one data source configured and scans running

  • Permission in Soda Cloud to access dataset metadata; see Global and Dataset Roles

  • Soda Cloud API keys. See Generate API keys

---

Soda Cloud Reporting API dashboard

Set up a Python script

1

Install the required dependencies:

pip install pandas requests
pip install "snowflake-connector-python[pandas]"

2

In a new Python script, configure your Soda Cloud connection. Use cloud.us.soda.io for US accounts and cloud.soda.io for EU accounts.

See Generate API keys for instructions on obtaining your keys.

soda_cloud_url = 'https://cloud.us.soda.io'
soda_apikey = 'xxx'        # API key ID from Soda Cloud
soda_apikey_secret = 'xxx' # API key secret from Soda Cloud

3

Define the Snowflake table names in which to store the metadata. Use uppercase names to meet Snowflake's case sensitivity requirements.

datasets_table = 'DATASETS_REPORT'
checks_table   = 'CHECKS_REPORT'

4

Configure your Snowflake connection:

snowflake_details = snowflake.connector.connect(
    user=user,
    password=password,
    account=account,
    warehouse=warehouse,
    database=database,
    schema=schema,
)

Capture and store metadata

1

Make a GET request to the Soda Cloud API to retrieve dataset information. The script exits with an error if the request is unauthorised.

response_datasets = requests.get(
    soda_cloud_url + '/api/v1/datasets?page=0',
    auth=(soda_apikey, soda_apikey_secret)
)
if response_datasets.status_code in (401, 403):
    print("Unauthorized or Forbidden. Check your API keys and permissions.")
    sys.exit()

2

Once the connection is confirmed, iterate over all pages of results and load them into a Pandas DataFrame. The script handles API rate limiting automatically. On HTTP 429, it pauses 30 seconds and retries the same page.

if response_datasets.status_code == 200:
    dataset_pages = response_datasets.json().get('totalPages')
    i = 0
    while i < dataset_pages:
        dq_datasets = requests.get(
            soda_cloud_url + '/api/v1/datasets?page=' + str(i),
            auth=(soda_apikey, soda_apikey_secret))
        if dq_datasets.status_code == 200:
            datasets.extend(dq_datasets.json().get("content"))
            i += 1
        elif dq_datasets.status_code == 429:
            print("Rate limit reached. Pausing 30 seconds.")
            time.sleep(30)
df_datasets = pd.DataFrame(datasets)

3

Following the same pattern, extract all check information from the Checks endpoint. This retrieves each check's name, dataset, last evaluation time, result (pass/warn/fail), owner, and any custom attributes.

response_checks = requests.get(
    soda_cloud_url + '/api/v1/checks?size=100',
    auth=(soda_apikey, soda_apikey_secret))

if response_checks.status_code == 200:
    check_pages = response_checks.json().get('totalPages')
    i = 0
    while i < check_pages:
        dq_checks = requests.get(
            soda_cloud_url + '/api/v1/checks?size=100&page=' + str(i),
            auth=(soda_apikey, soda_apikey_secret))
        if dq_checks.status_code == 200:
            checks.extend(dq_checks.json().get("content"))
            i += 1
        elif dq_checks.status_code == 429:
            time.sleep(30)
df_checks = pd.DataFrame(checks)

Any custom Soda attributes defined on your checks are automatically expanded into individual columns. If the target tables already exist in Snowflake, the script detects new attribute columns and adds them with ALTER TABLE without overwriting existing data.

4

Write both DataFrames to your data source:

write_pandas(snowflake_details, df_checks,   checks_table,   auto_create_table=True)
write_pandas(snowflake_details, df_datasets, datasets_table, auto_create_table=True)

5

To track changes over time, schedule the script to run regularly and store results in incremental tables.

Build a dashboard in Sigma

1

Follow Sigma's documentation to connect to your data source, pointing to the database and schema where your CHECKS_REPORT and DATASETS_REPORT tables are stored.

2

Access the metadata either by modelling data from the database tables directly, or by creating a dataset using custom SQL.

3

Create a new workbook in Sigma and add your visualisations.

A typical data quality dashboard might include:

• KPI tiles — total datasets monitored, total checks executed, number of failing checks

• Weighted data quality score — using the Weight check attribute to calculate a custom health score, trended over time

• Breakdowns by check attribute — such as Data Quality Dimension (Completeness, Validity, Consistency, Accuracy, Timeliness, Uniqueness), Data Domain, Data Team, or Pipeline Stage

Sigma dashboard on top of Soda Cloud Reporting API results

Check attributes such as Data Quality Dimension, Data Domain, Data Team, and Pipeline Stage are particularly useful for filtering and segmenting results in your dashboard. The Weight attribute lets you assign a numerical importance level to each check, enabling a custom data health score.

See check and dataset attributes for more.

---

You are not logged in to Soda and are viewing the default public documentation. Learn more about Documentation access & licensing.