# What is Soda? **Soda is a data quality platform** that helps organizations make sure their data can be trusted. Soda gives teams a **shared workflow** to **monitor data quality**, **catch problems** early, **understand issues** at source, and **take action** quickly. You can use Soda to: * **Monitor production data** with automated, ML-powered observability that spots unexpected changes without needing to define every rule up front. * **Define data contracts,** making expectations explicit and enabling producers and consumers to collaborate on reliable data at the source. * **Test data earlier in the pipeline**, as part of CI/CD workflows or during development, to prevent bad data from reaching production. Soda helps teams to *start right* and automatically detect anomalies in metrics that have already happened. And *shift left* to prevent issues from happening again with collaborative data contracts. ### Soda v4 vs v3 {% if !(visitor.claims.plan === %} {% hint style="success" %} This is the documentation for Soda v4. If you are still using Soda v3, head to the [v3 documentation](https://docs.soda.io/soda-documentation/soda-v3/). {% endhint %} {% endif %} {% if visitor.claims.plan === %} {% hint style="success" %} This is the documentation for Soda v4. If you are still using Soda v3, head to the [v3 documentation](https://docs.soda.io/soda-documentation/soda-v3/) or, even better, [**migrate to Soda v4**](/reference/migrate-from-v3-to-v4). {% endhint %} {% endif %} The new version of Soda has transformed the software into a **full data-quality platform** by layering on: * **End-to-end data observability:** [Data Observability](/data-observability) * **Collaborative data contracts:** [Data Testing](/data-testing) This marks the shift from a CLI-centric checks engine toward a unified, observability-driven data quality platform with a refined, three-tier **Core + Runner + Cloud** architecture, built-in contracts, orchestration, and deep integrations. > Contact us to learn more about Soda's capabilities. ### What is data quality? **Data quality** refers to how well a dataset meets the expectations of completeness, accuracy, timeliness, uniqueness, and consistency. Good data supports business goals, drives confident decision-making, and is the base for great data products. Poor data quality causes failed pipelines, incorrect reports, and broken AI models. Managing data quality means proactively validating assumptions and reactively monitoring for drift or degradation. Soda helps you answer questions like: * Is the data fresh and complete? * Are there unexpected values or duplicates? * Did values shift outside of expected ranges? * Are schema or contract changes causing breakage? * Are data quality metrics changing over time? ### Key Concepts #### Data Observability Data observability is a reactive approach to monitoring data in production and catching unexpected issues as they emerge. It helps answer the question: What is happening with my data right now, and how is that changing over time? Use data observability to: * Detect anomalies in data quality metrics such as freshness, row counts, null values or custom ones * Monitor metric trends and seasonality * Identify late-arriving or missing records * Get alerted when values deviate from historical norms #### Data Testing Data testing is a proactive approach that validates known expectations about your data during development, deployment, or transformation. It helps you catch issues before they reach production, break reports, or impact downstream systems. Use data testing to: * Align on what “good data” looks like through data contracts * Verify that your data meets those expectations, including schema, values, and transformations * **Test data at every step of the pipeline** to prevent bad data from reaching downstream systems * Integrate with CI/CD workflows for continuous quality checks during development #### Data Contracts **Data contracts define what a dataset should look like, including its schema, data types, value ranges, and other constraints.**\ They establish a shared agreement between data producers and consumers about what’s expected and what must be upheld. Both **testing** and **observability** play a role in upholding data contracts: * **Testing** validates that data meets the contract during development, pipeline execution, and on schedule. * **Observability** monitors contract adherence in production and detects unexpected issues. ### Data Observability vs Data Testing While data testing and observability are different in when and how they operate, they work best together as a **unified strategy**.
ApproachTimingUse case
Data TestingProactive and preventative: Pre-production, during development or CI/CDPrevent breakages before they happen: Validate known rules and enforce contracts
Data ObservabilityReactive and adaptive: In production, runtime monitoringMonitor data behavior and changes over time with automated detection of anomalies, schema changes, and other unexpected issues.
Together, they enable end-to-end data quality management: testing prevents problems, and observability detects those that escape prevention. At the same time, observability can help prioritize which issues to address and shift left to resolve them upstream. ### Data quality at scale across the enterprise #### Divide and conquer Managing data quality across hundreds or thousands of datasets requires a **scalable, federated approach**. Soda enables this through: * **Metadata-driven observability** that adapts checks to each dataset's structure and context. * **Role-based collaboration** so teams can take ownership of the data they know best. * **An interface for both engineering and business users**, enabling collaboration through code, UI, or APIs, depending on user preference and role. * **Integration with existing tools and workflows**, such as data catalogs and incident management systems. * **Pipeline** **and** **CI/CD integration** to automate data quality checks. #### Data quality as a team sport Reliable data depends on collaboration across roles: * **Data engineers** embed tests and monitor pipelines to catch issues early. * **Data producers and consumers** align on expectations through data contracts. * **Data consumers** report issues and collaborate with producers to interpret metrics and resolve problems. * **Governance teams** define and enforce data quality standards. * **Platform teams** deploy, manage, and secure the underlying infrastructure. Soda Cloud acts as the shared workspace where these roles collaborate, triage incidents, and resolve issues. ### Deployment options Soda offers three deployment models, depending on your infrastructure and data privacy needs. | Deployment Model | Description | Ideal For | Key Features | Considerations | | ---------------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Soda Core** | Open-source Python library (with commercial extensions) and CLI for running Data Contracts in your pipelines. | Data engineers integrating Soda into custom workflows. | Full control over orchestration, in-memory data support, contract verification. |

No observability features.

Required for in-memory sources (e.g., Spark, DataFrames).

Data source connections managed at the environment level.

| | **Soda-hosted Runner** | Managed version of Soda that runs observability features, executes Data Contracts and scheduled them. | Teams seeking a simple, managed solution for data quality. | Centralized data source access, no setup required, observability features enabled. Enables users to create, test, execute, and schedule contracts and checks directly from the Soda Cloud UI. |

Required for observability features.

Cannot scan in-memory sources like Spark or DataFrames.

| | **Self-hosted Runner** | Same as Soda-hosted Runner, but deployed and managed in your own Kubernetes environment. | Teams needing full control over infrastructure and deployment. | Similar to Soda-hosted Runner, but deployed within the customer’s environment; data stays within your network. |

Required for observability features.

Cannot scan in-memory sources like Spark or DataFrames.


Kubernetes expertise required.

| Read more about [Deployment options](/deployment-options) ### Supported data sources and integrations Soda integrates with the modern data stack: * **Data warehouses and databases**: Databricks, Snowflake, BigQuery, Redshift, PostgreSQL, MySQL, Spark, Presto, DuckDB, and more. * **Orchestration platforms**: Airflow, Dagster, Prefect, Azure Data Factory. * **Metadata tools**: Atlan, Alation, Collibra, data.world, Zeenea. * **Cloud providers**: AWS, Google Cloud, Azure. * **BI tools**: Looker, Tableau, Power BI. * **Messaging and ticketing**: Slack, Microsoft Teams, Jira, PagerDuty, ServiceNow, Opsgenie. ### What’s next? * To get started with Soda, check out the end-to-end [Quickstart](/quickstart) guide. *** ## Community & Support Need help or want to contribute? * **Join our Slack Community**: [Soda Community](https://join.slack.com/t/soda-community/shared_invite/zt-3epazj3kw-00z15nnW4KEt4j_vk8lbdQ)​ * **Browse GitHub Discussions**: [Soda on GitHub](https://github.com/sodadata/soda-core)​ > **Still have questions?** Use the search bar above or reach out through our community channels for additional help. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Quickstart Learn how to get started with Soda. This quickstart shows how Soda **detects unexpected data issues** by leveraging AI powered Anomaly Detection and **prevents future problems** by using data contracts. ### Scenario A data engineer at a retail company needs to maintain the `regional_sales` dataset so their team can manage regional sales data from hundreds of stores across the country. The dataset feeds executive dashboards and downstream ML models for inventory planning. Accuracy and freshness are critical, so you need both: * **Automated anomaly detection** on key metrics (row counts, freshness, schema drift) * **Proactive enforcement** of business rules via data contracts ### Sign up Contact us at to get an account set up. After signing up, you can follow the steps below to set up a data source and start improving data quality. ### Add a Data Source Soda Cloud’s no-code UI lets you connect to any data source in minutes. {% if visitor.claims.plan === %} **Step 1/5 — Establish connection** {% stepper %} {% step %} In [cloud.soda.io or**, click on Data Sources → New Data Source.**](https://cloud.soda.io)
{% endstep %} {% step %} **Choose your data source provider**.
{% endstep %} {% step %} **Name your data source** under Data Source Identifier.
{% endstep %} {% step %} **Name a data source owner**
{% endstep %} {% step %} **Set a default runner**
{% endstep %} {% step %} \[Optional] **Set a default schedule** for contracts
{% endstep %} {% step %} Scroll down and fill in the following **credentials** from your data source:
{% endstep %} {% step %} Click **Connect** or **Test connection**. This will trigger the connection and move to the next step.
{% endstep %} {% endstepper %} **Step 2/5 — Diagnostics Warehouse** {% stepper %} {% step %} \[Optional] **Enable** [**Diagnostics Warehouse**](broken://pages/UZCAHd8ZBgl01336NkZa) 1. **Choose the Diagnostics Warehouse connection** 2. \[Optional] **Provide a schema configuration** and **Allow failed row collection** 3. **Test connection** {% hint style="warning" %} The Diagnostics Warehouse connection must have write access in order to perform correctly. {% endhint %} {% endstep %} {% step %} Click on **Save** {% endstep %} {% endstepper %} {% columns %} {% column %}

Enable Diagnostics Warehouse

{% endcolumn %} {% column %}

Choose the Diagnostics Warehouse connection

{% endcolumn %} {% column %}

Provide a schema configuration & allow failed row collection

{% endcolumn %} {% endcolumns %} > Learn more about [setting up Diagnostics Warehouse](broken://pages/E1KTiEL4c6zytpedbEZg#set-up-diagnostics-warehouse). **Step 3/5 — Dataset selection** {% stepper %} {% step %} **Select the datasets** you want to onboard on Soda Cloud
{% endstep %} {% step %} Click on **Next.**
{% endstep %} {% endstepper %} **Step 4/5 — Metric Monitoring** {% stepper %} {% step %} **Enable Metric Monitoring** By default, Metric Monitoring is enabled to automatically track key metrics on all the datasets you onboard and alert you when anomalies are detected. It is powered by built-in machine learning that compares current values against historical trends.
{% endstep %} {% step %} \[Optional] Enable **Advanced monitor configuration**.
{% endstep %} {% step %} Click on **Next**.
{% endstep %} {% endstepper %} **Step 5/5 — Confirmation and additional options** {% stepper %} {% step %} **Define a dataset owner** by choosing a user from the drop-down menu.
{% endstep %} {% step %} \[Optional] Enable **Collect failed rows**. Then, select a **storage** & **detection strategy**
{% endstep %} {% step %} \[Optional] **Enable Profiling** and configure it Profiling scheduled execution is set at 12:00AM UTC by default.
{% endstep %} {% step %} Click **Finish** to onboard your datasets {% endstep %} {% endstepper %} Soda Cloud will now spin up its **Soda-hosted Runner** and perform an initial **Profiling** & **Historical Metric Collection scan**, if configured. This usually takes only a few minutes. {% endif %} {% if !visitor.claims.plan === %} **Step 1/4 — Establish connection** {% stepper %} {% step %} In [cloud.soda.io](https://cloud.soda.io) or[ cloud.us.soda.io](https://cloud.us.soda.io/), click on **Data Sources** → **New Data Source**.
{% endstep %} {% step %} **Choose your data source provider**
{% endstep %} {% step %} **Name your data source** under Data Source Label.
{% endstep %} {% step %} Scroll down and fill in the following credentials from your data source:
{% endstep %} {% step %} Click **Connect** or **Test connection**. This will trigger the connection and move to the next step.
{% endstep %} {% endstepper %} **Step 2/4 — Dataset selection** After connecting, Soda will perform an **automated dataset discovery**. > Learn more about dataset onboarding in detail. {% stepper %} {% step %} **Choose a dataset selection strategy**.
{% endstep %} {% step %} **Select the datasets** you want to onboard on Soda Cloud. {% endstep %} {% step %} Click on **Next**.
{% endstep %} {% endstepper %} For the purposes of this Quickstart, **Soda recommends manual selection**. {% columns %} {% column %}

Manual selection allows you to browse a directory view of all the datasets in your data source.

{% endcolumn %} {% column %}

Rules-based selection allows you to automate the dataset onboarding process, only selecting tables that match specified rules.

{% endcolumn %} {% endcolumns %} > Learn more about [dataset selection strategies](broken://pages/cHo9ixopiR0flOMCbyUF#choose-a-dataset-selection-strategy). **Step 3/4 — Metric Monitoring** {% stepper %} {% step %} **Enable Metric Monitoring.** By default, Metric Monitoring is enabled to automatically track key metrics on all the datasets you onboard and alert you when anomalies are detected. It is powered by built-in machine learning that compares current values against historical trends.
{% endstep %} {% step %} \[Optional] Enable **Advanced Monitor Configuration**.
{% endstep %} {% step %} Click on **Next**.
{% endstep %} {% endstepper %} **Step 4/4 — Confirmation and additional options** {% stepper %} {% step %} **Define a dataset owner** by choosing a user from the drop-down menu.
{% endstep %} {% step %} **Enable Profiling** and configure it. Profiling scheduled execution is set at 12:00AM UTC by default.
{% endstep %} {% step %} Click **Finish** to onboard your datasets. {% endstep %} {% endstepper %} Soda Cloud will now spin up its **Soda-hosted Runner** and perform an initial **Profiling** & **Historical Metric Collection scan**, if configured. This usually takes only a few minutes. {% endif %} ### Part 1: Review Anomaly Detection results Congratulations, you’ve onboarded your first dataset! Now let’s make sure you always know what’s happening with it. That’s where Metric Monitoring comes in. It automatically tracks key metrics like volume, freshness, and schema changes, with no manual setup required. You’ll spot anomalies, detect trends, and catch unexpected shifts before they become problems. #### Step 1: Open the Metric Monitors dashboard {% stepper %} {% step %} Go to **Datasets** → select the dataset to inspect.
{% endstep %} {% step %} Navigate to the **Metric Monitors** tab to learn more about the metrics calculated. You'll immediately see that key metrics are automatically monitored by default, helping you detect pipeline issues, data delays, and unexpected structural changes as they happen.\ No setup needed, just visibility you can trust.
{% endstep %} {% endstepper %} #### Step 2: View anomalies in a specific monitor In this guide, we will focus on the **Most recent timestamp** monitor. The panel shows that it was expected to be in a range of **0 - 5m 31s**, but the recorded value at scan time was **56m 49s**. In order to take a closer look: {% stepper %} {% step %} Click the **Most recent timestamp** (or monitor of your choice) block.
{% endstep %} {% step %} In the monitor page you’ll see: * any red-dot anomalies flagged by the model, * measured value vs. expected range, * buttons to **Mark as expected**, **Create new incident**, etc. {% endstep %} {% step %} Flag an outlier as "expected" or investigate it further. {% endstep %} {% endstepper %} **Soda’s anomaly detection engine was built in-house** (no third-party libraries) and optimized for high precision. It continuously adapts to your data patterns, and it incorporates your feedback to reduce false alarms. Designed to minimize false positives and missed detections, it shows a 70% improvement in detecting anomalous data quality metrics compared to Facebook Prophet across hundreds of diverse, internally curated datasets containing known data quality issues. The Anomaly Detection Algorithm offers complete control and transparency in the modeling process to allow for interpretability and adaptations. It features high accuracy while leveraging historical data, delivering improvements over time. ### Part 2: Attack the Issues at Source (No-Code) Our automated anomaly detection has just done the heavy lifting for you, identifying unusual patterns and potential data issues without any setup required. But to prevent those issues from happening again, you must define exactly what your data should look like; every column, every rule, every expectation. That’s where **Data Contracts** come in. They let you proactively set the standards for your data, so problems like this are flagged or even prevented before they impact your business. #### Step 1: Create a Data Contract Create a new data contract to define and enforce data quality expectations. {% stepper %} {% step %} In your **Dataset Details** page, go to the **Checks** tab. {% endstep %} {% step %} Click **Create Contract**.
{% endstep %} {% step %} Build a **data contract**. When creating a data contract, Soda will connect to your dataset and build a data contract template based on the dataset schema. From this point, you can start adding both dataset-level checks and column-level checks, as well as defining a verification schedule or a partition.
{% endstep %} {% step %} Toggle **View Code** if you’d like to inspect the generated SodaCL/YAML. This gives you access to the full contract code.
{% endstep %} {% step %} You can copy the following full example, paste it into the editor and edit it as you wish. You can toggle back to no-code view to see and edit the checks in the no-code editor. ```yaml dataset: databricks_demo/unity_catalog/demo_sales_operations/regional_sales filter: | order_date >= ${var.start_timestamp} AND order_date < ${var.end_timestamp} variables: start_timestamp: default: DATE_TRUNC('week', CAST('${soda.NOW}' AS TIMESTAMP)) end_timestamp: default: DATE_TRUNC('week', CAST('${soda.NOW}' AS TIMESTAMP)) + INTERVAL '7 days' checks: - row_count: - schema: columns: - name: order_id data_type: INTEGER checks: - missing: name: Must not have null values - name: customer_id data_type: INTEGER checks: - missing: name: Must not have null values - name: order_date data_type: DATE checks: - missing: name: Must not have null values - failed_rows: name: Cannot be in the future expression: order_date > DATE_TRUNC('day', CAST('${soda.NOW} ' AS TIMESTAMP)) + INTERVAL '1 day' threshold: must_be: 0 - name: region data_type: VARCHAR checks: - invalid: valid_values: - North - South - East - West name: Valid values - name: product_category data_type: VARCHAR - name: quantity data_type: INTEGER checks: - missing: name: Must not have null values - invalid: valid_min: 0 name: Must be higher than 0 - name: price data_type: NUMERIC checks: - invalid: valid_min: 0 name: Must be higher than 0 - missing: name: Must not have null values - name: payment_method data_type: VARCHAR checks: - missing: name: Must not have null values - invalid: threshold: metric: count must_be: 0 filter: region <> 'north' valid_values: - PayPal - Bank Transfer - Cash - Credit Card name: Valid values in all regions except North - invalid: name: Valid values in North filter: region = 'north' valid_values: - PayPal - Bank Transfer - Credit Card qualifier: ABC124 ``` {% endstep %} {% endstepper %} That’s right: with Soda, you can edit a contract using either a **no-code interface** or **directly in code**. This ensures an optimal experience for all user personas while also providing a version-controlled code format that can be synced with a Git repository. #### Step 2: Publish & verify {% stepper %} {% step %} Click **Test** to verify the contract executes as expected {% endstep %} {% step %} When you are done with the contract, click **Publish**
{% endstep %} {% step %} Click **Verify**. Soda will evaluate your rules against the current data.
{% endstep %} {% endstepper %} #### Step 3: Review check results Review the outcomes of the contract checks to confirm whether the data meets expectations.\ You can drill into those failures in the **Checks** tab.
### Part 3: Attack the Issues at Source (Code) You can trigger contract verification programmatically as part of your pipeline, so your data gets tested every time it runs. We’ve prepared an example notebook to show you how it works: Open the following Notebook example: \ In your Python environment, first install the Soda Core library ```sh pip install -i https://pypi.cloud.soda.io/simple -U soda-core ``` Then, in the same environment, create a `sc_config.yml` file that contains your API keys, which are necessary to connect to Soda Cloud. You can get your API keys from your Profile: [Broken mention](broken://pages/H6Piqnv5lUFCAv1Cvm75) The `sc_config.yml` file should look like the following: {% code title="sc\_config.yml" %} ```yaml soda_cloud: host: cloud.soda.io ## Or cloud.us.soda.io api_key_id: YOUR_API_KEY_ID ## Replace with your actual key ID api_key_secret: YOUR_API_KEY_ID ## Replace with your actual key secret ``` {% endcode %} Now you are ready to trigger the verification of the contract. To do that just provide the identifier of your dataset as well as the path to the configuration file you just created in the previous step. This will trigger a verification using Soda Runner and return the logs. Create a `verify_contract.py` file in your environment with the code below (or run it from a Jupyter notebook/Python interpreter): ```python from soda_core import configure_logging from soda_core.contracts import verify_contracts_on_runner configure_logging(verbose=False) res = verify_contracts_on_runner( dataset_identifiers=["databricks_demo/unity_catalog/demo_sales_operations/regional_sales"], soda_cloud_file_path="sc_config.yml", ) print(res.get_logs()) ``` > You can learn more about the Python API here: [Python API](/reference/soda-apis/python-api) **You’ve completed the tutorial and are now ready to start catching data quality issues with Soda** ### What’s Next? * Explore [**Profiling**](broken://pages/p1ZnzQiAs79m2od7t60v) in the **Discover** tab to curate column selections for deeper analysis. * Set up [**Notification Rules**](broken://pages/GAK0Cdoq2hZmucVmrovV) (bell icon → Add Notification Rule) to push alerts to Slack, Jira, PagerDuty, etc. * Dive into [**Custom Monitors**](broken://pages/hbDvJBNrc1PAzUeC2KXB) via `scan.yml` or the UI for even more tailored metrics. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Soda AI Discover how Soda AI can help you improve and automate your data quality seamlessly **Soda AI gets your data AI-ready**. It includes several capabilities that serve one goal: trusted data that's ready for an AI-driven workflow. With Soda AI, your team and AI agents can turn data into actionable, understandable and trustworthy information. The following capabilities make this work:
FeatureWhat it does
Chat interfaceConversational entry point to Soda AI: performs actions in Soda, guides workflows, and answers questions, all in-product.
Contract AutopilotCreates a first version of a data contract.
Contract CopilotHelps edit and iterate on existing data contracts.
MCPConnect your own agents to trusted data.
APIIntegrate Soda with your own systems.
Soda CLIRun data quality programmatically in your existing pipelines.
## What is Soda AI? **Soda AI is the collection of AI-powered features** that make your data quality AI-ready. It isn't a single tool but the umbrella over everything in Soda that uses AI to define, maintain, and reason about data quality: the **in-product chat** **interface**, **Autopilot**, **Copilot**, and the **tools** that connect your contracts and data-quality status to **external agents**. Soda AI's features fall into a few groups:
Cover image
Chat interfaceThe chat interface is the in-product way of interacting with Soda AI in natural language. It answers questions about Soda and your data, guides you through platform workflows, and proposes/performs actions on your behalf. As a Soda AI interface, the chat reaches for the tools below to carry out tasks./files/p8HLZZXJXHZp4DOqYhVW#chat-interface
Contract AutopilotGenerate fully populated contracts with recommended checks, drawn from your data's own profile. Data teams can reach broad coverage in an afternoon instead of authoring contracts by hand for weeks./files/SeXLJXXWta7IkyaRBqSq/pages/SIhP0wH6BwPidEAnAUIS
Contract CopilotEdit and iterate on contracts in plain English (adding a freshness check, tightening a rule, handling a nested field) without hand-writing code lines. Contract Copilot will show you exactly what changed./files/lrjqKlsT85mjIKPm0Axj/pages/PZNeraYaFXY51diHXyWO
Agent-facing interfacesConnect agents to trusted data. MCP, Soda CLI and API put your contracts and data-quality status in front of the tools your team and your agents already use. Engineers can run quality checks inside their existing pipelines, and agents act on verified data./files/irbgCrGkjfVTMHTTq99z/pages/c9RIRkDSFHZNEE44x1zB
Across all of these, **Soda AI is designed to**: * **Reduce manual effort** in defining and maintaining data quality expectations * **Help** users **reason about data** and contracts faster * **Integrate with existing workflows**, enabling AI agents to interact with Soda Cloud in your own pipelines. * **Keep humans in control**: every proposed action is shown and approved by a person before it is applied * **Operate safely** within strict privacy, security, and governance boundaries ### Limitations * Soda AI features may not always produce complete or perfectly accurate answers. * Users remain responsible for reviewing any proposed action before it is applied. * The chat interface, Contract Autopilot, and Contract Copilot do not operate outside the context of your Soda Cloud workspace. {% hint style="success" %} **The chat interface, Contract Autopilot, and Contract Copilot are assistive, not autonomous**. These features are designed as a control surface for human-in-the-loop data quality, scoped to the same operations you could perform yourself, subject to your role and permissions. {% endhint %} ## Chat interface The chat interface is **the in-product, conversational face of Soda AI**. It lives alongside your work in Soda Cloud, and incorporates AI-powered, assistive features that help **you understand data**, **author data contracts**, and **accelerate onboarding** without replacing human decision-making. You can request it to perform actions on your behalf, get step-by-step guidance on platform workflows, or ask it questions about Soda, all in **natural language, without leaving the page you’re on**. As an agentic assistant within Soda Cloud, **the chat interface can access the tooling** necessary to perform actions on your behalf, such as Autopilot or MCP.
### Chat interface capabilities Soda AI's actions are scoped to the **same operations you can perform** yourself in Soda Cloud, subject to your role and permissions. **The chat interface can:** * **Perform** safe, reviewable **actions in Soda Cloud** on your behalf. For example: * **Generate data contracts.** Using [Contract Autopilot](/soda-ai/contract-autopilot), you can point Soda AI at your datasets and generate fully populated contracts with recommended checks, drawn from your data's own profile. Data teams can get full coverage in an afternoon, instead of manually creating contracts for weeks. * **Edit and iterate over data contracts** in plain English. Add a freshness check, tighten a rule, handle a nested field, all without hand-writing code. Within the chat, you will see exactly what changed, so you keep full agency over the editing process. * **Guide** **you through platform workflows** (configuring datasets, authoring contracts, investigating incidents) * **Answer product and docs questions.** Ask what a check, monitor, or incident means, or tell it to make a change. It proposes the action, scoped to what your role already allows, and waits for you to approve. * **Agentic Navigation.** Ask Soda AI Chat to take you in another place in the Web UI and it will take you there. * **Context Awareness.** Soda AI chat knows basic information about the page you are visiting to be better able to answer related questions. {% hint style="success" %} Every change is proposed, shown, and approved by a human user. Soda AI is an assistant and control surface, and is designed to perform **human-in-the-loop processes**. {% endhint %} {% if visitor.claims.plan === %} ### Conversations and history Each chat is saved as a **conversation** in the Soda AI chat panel. You can resume a previous conversation or start a new one from any page on Soda Cloud. {% hint style="info" %} **Your prompts and conversation history are stored in your browser local storage**. If you change browsers or open Soda Cloud in a different device, you will not have access to your conversation history. {% endhint %} {% endif %} {% if visitor.claims.plan === %} *** ## Enable/Disable AI features AI features in Soda can be configured from the **Organization Settings**. {% stepper %} {% step %} Navigate to **Settings** > **Organization Settings**. {% endstep %} {% step %} Under the **Organization** tab **> Soda AI features:** Check/Uncheck the feature checkbox to enable/disable it.
{% endstep %} {% step %} Click **Save** on the top right corner. {% endstep %} {% endstepper %} {% endif %} *** ## Soda AI security & privacy measures Soda's features are built with **privacy and security as first-class design principles**. The following safeguards apply to all AI/ML-powered features in Soda. ### Privacy and data usage * Soda uses **third-party foundation models** (GPT-5 family). * **API request data** sent to the model provider is **not retained for training**. * **Soda does not use customer data**, metadata, or results from your environment to **train** or **fine-tune** external AI, LLM, or ML models. * **Source data**, such as row-level data and/or PII, **is never sent by default** to Soda’s AI/ML engine or to any external model provider. **AI features that require the use of source data are** **opt-in**. * To answer questions and propose/perform actions, **Soda AI analyzes** the user's **natural-language request**, Soda **documentation content**, **dataset schemas**, and **non-identifiable column-level metadata** when needed. {% if visitor.claims.plan === %} * Soda AI features can be pointed at an OpenAI account you own by [**bringing your own key (BYOK)**](broken://pages/cS46FGE9oVlrTS48iaBE)**.** Soda only stores a reference to the key, never the key itself. {% endif %} {% if !(visitor.claims.plan === %} * Soda AI features can be pointed at an OpenAI account you own by bringing your own key (BYOK)**.** Soda only stores a reference to the key, never the key itself. {% endif %} {% hint style="info" %} **The data sent to each model provider depends on the feature** and is limited to what is strictly required by the individual tool being used. See more details on each specific tool page. {% endhint %} #### Diagnostics data remains local If record-level diagnostics (such as `Failed Rows`) are enabled: {% if visitor.claims.plan === %} * They are stored **only in your own data warehouse(s)** via [Soda’s Diagnostics Warehouse](broken://pages/UZCAHd8ZBgl01336NkZa) {% endif %} {% if !(visitor.claims.plan === %} * They are stored **only in your own data warehouse(s)** via [Soda’s Diagnostics Warehouse](/diagnostics-warehouse) {% endif %} * They are **not** stored in Soda Cloud * They are **not accessible** to Soda’s AI/ML processing ### Security and compliance All AI/ML processing is performed securely and in alignment with **Soda’s SOC 2 Type 2 compliance**: * All processes and development align with information security, application security, and Software Development Life Cycle (SDLC) policies * **Data in transit is encrypted** using industry standards * **Data at rest is encrypted** using industry standards * **Access** to AI features is governed by **existing Soda roles and permissions** * Third-party providers (such as OpenAI) are **evaluated for compliance before adoption** {% hint style="success" %} Soda continuously monitors AI features via metrics, quality indicators and internal testing to ensure reliable operation at scale. {% endhint %} #### Confidentiality & access control * Soda AI features **accessed through the chat interface** work within an **authenticated Soda Cloud session** only. * Soda AI's actions are subject to your **existing** **Soda roles and permissions**. None of the features can perform anything you could not perform yourself. * Requests are **not used for model training**. * Soda **does not store prompts or conversations**. Your prompts and conversation history are stored in your browser local storage.

If you change browsers or open Soda Cloud in a different device, you will not have access to your conversation history.

### AI guardrails and ethics * Foundation models include built-in safety guardrails from the provider * Additional **system-level guardrails** restrict responses to the intended Soda use cases * Soda AI features do not generate harmful, discriminatory, or user-targeted content Because Soda AI features do not evaluate users or make decisions, traditional bias and fairness risks are minimal. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Contract Autopilot Learn more about Contract Autopilot, a Soda AI-powered feature to create data contracts tailored to your data. {% hint style="success" icon="database" %} Contract Autopilot is currently available as a **preview** feature. Contact for early access. {% endhint %} **Contract Autopilot** automatically generates initial data contracts for datasets with **recommended data quality checks** to accelerate onboarding in Soda Cloud. It is designed to help teams achieve **fast time to value** and iterate later. {% hint style="success" %} **New: run Autopilot with Soda AI.** Point Soda AI at your datasets to draft fully populated contracts with recommended checks from your data's own profile. You approve what gets enforced. Read more about [Soda AI](/soda-ai)'s capabilities. {% endhint %} **Contract Autopilot** automatically generates initial data contracts for datasets with **recommended data quality checks** to accelerate onboarding in Soda Cloud. It is designed to help teams achieve **fast time to value** and iterate later. {% if !(visitor.claims.plan === %}

Generate contracts in bulk and ensure data quality with just one click.

{% endif %} {% if visitor.claims.plan === %} #### :camcorder: Watch Contract Autopilot in action {% embed url="" %} #### :bulb: Contract Autopilot vs. Create Contract {% columns %} {% column %} **Contract Autopilot** analyzes sampled data and profiling metadata to automatically generate a **fully populated contract with recommended checks**. {% endcolumn %} {% column %} When you click **Create Contract** (or use the CLI `generate` command), Soda creates a **contract skeleton** based on the dataset schema. It includes columns only and **no checks are added**. {% endcolumn %} {% endcolumns %} ## Generate a contract with Autopilot {% hint style="warning" %} This feature requires the **Manage contract** permission on the select datasets. Learn more about permissions here: [Dataset Attributes & Responsibilities](/dataset-attributes-and-responsibilities) {% endhint %} {% tabs %} {% tab title="Via Soda AI" %} You can generate a contract with natural language while chatting with Soda AI: {% stepper %} {% step %} #### Open Soda AI From any page, navigate to the left panel and click on **Soda AI** on the bottom left corner. The Soda AI chat interface will open on the same screen.
{% endstep %} {% step %} #### Request contract generation When you request an action in Soda AI, you will be prompted to confirm the changes:
{% endstep %} {% step %} #### Confirm contract generation
{% endstep %} {% step %} #### Review changes After generation, Soda AI will begin the process in the background. You can continue using Soda Cloud while contracts are being generated, and check later the status of the operation on the **Scans** page or by asking Soda AI.
{% endstep %} {% endstepper %} :octagon-check: You can now **view the generated contracts** to review, edit or suggest changes. {% endtab %} {% tab title="Via a dataset" %} You can generate a contract manually by selecting the datasets you want to monitor: {% stepper %} {% step %} ### Select datasets On the **Datasets** page, select the datasets for which you want to generate a contract. Then, click on **Generate Contracts** (top right).
{% endstep %} {% step %} #### \[Optional] Set a schedule for contracts You can optionally define a schedule to run contract verification at a time of your choosing. * **Use the data source's default schedule** if your data source has a default schedule enabled. * **Manual schedule** if your data source does not have a default schedule, or if you wish to manually set a schedule for the generated contracts.
{% endstep %} {% step %} #### Generate After selecting the datasets, Soda will provide a snapshot of which contracts will be generated, and a time estimate for generation. You can continue using Soda Cloud while contracts are being generated. After reviewing what will be created, click on **Generate `[num]` Contracts**.
{% endstep %} {% endstepper %} :octagon-check: You can now **view the generated contracts** to review, edit or suggest changes. {% endtab %} {% endtabs %} This process will create a scan for each selected dataset. You can cancel the scan or review the execution logs in the **Scans** tab on the dataset, as well as on the general **Scans** page. {% columns %} {% column width="50%" %} :check: **Successful scan** * **The generated contract is published.** * A **purple checkmark** icon indicates that it is an automatically generated contract. * The icon will change to a **blue checkmark** icon after there is a **user edit**, to indicate it is a user-generated contract. * **A** [**data contract history**](broken://pages/gc7k6Fk5zaG6vZozUHWM#contract-history) **entry is published**, indicating that Autopilot published a first version of the contract. * \[If configured] A "published contract" Webhook will be published (for example, to allow sync to Git). {% endcolumn %} {% column width="50%" %} :x: **Failed scan** * The dataset is marked with the message "**Contract generation has failed**". * **Scan logs** can be reviewed to understand the underlying cause. * The contract generation can be **re-triggered** after inspecting the root cause. {% endcolumn %} {% endcolumns %} *** ## Requirements & limitations {% hint style="info" %} Autopilot is **only available upon request**. [Contact us](mailto:support@soda.io) to enable it in your organization. {% endhint %} **Contract Generation Scope** Autopilot can only generate data contracts for datasets that do **not** already have an existing data contract. Datasets with an existing contract are not eligible for contract generation via Autopilot. **Autopilot is currently a Soda Cloud feature:** * A **Soda Runner** is required * CLI or Python API are not available * APIs are not available **Data types** Autopilot works on the following data types: * text * numeric * timestamps Columns with other data types, such as JSON or structured array, for example, are not currently supported. **Data sources** Autopilot is supported on: * Snowflake * Databricks * Postgres * Athena * Țrino * Dremio * Oracle * SQLServer * Synapse * Fabric * Redshift {% hint style="success" %} **Contract Autopilot is assistive** and does not autonomously modify data or execute destructive operations. Users can review generated contracts before execution. {% endhint %} *** ## Privacy & security Contract Autopilot is designed to minimize data exposure while generating meaningful contract recommendations. The following data is used locally by Soda: * A sample of approximately **10,000 rows** to compute profiling metrics The following data is sent to Soda’s managed OpenAI API: * Computed profiling metrics * Data source type * Dataset schema * A sample of up to 100 rows, if enabled in the [organization settings](broken://pages/Dt8rj970uYl7OfKTepgr#send-row-level-data-to-soda). {% hint style="info" %} Access to Contract Autopilot is **restricted to users with appropriate** dataset **permissions**. {% endhint %} > Learn more about [Soda's AI security & privacy measures](/soda-ai#sodas-ai-security-and-privacy-measures). *** ## Next steps Manage the [organization settings](broken://pages/Dt8rj970uYl7OfKTepgr#enable-contract-autopilot-powered-by-openai) to: * enable/disable Contract Autopilot * allow Soda to send a small sample of row data to OpenAI After contract generation, you can: * Review generated checks * Validate assumptions with domain experts * Adjust schedules and thresholds * Iterate using [Contract Copilot](/soda-ai/contract-copilot) {% endif %} *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Contract Copilot Learn more about Contract Copilot, a Soda AI-powered feature to edit data contracts. {% hint style="success" %} **Do this with Soda AI.** Describe the change you want in plain English and Soda AI will update your contract from schema and metadata. You approve the diff. Read more about [Soda AI](/soda-ai)'s capabilities. {% endhint %} **Contract Copilot** is an AI assistant that helps teams **understand**, **edit and evolve existing data contracts** using natural language. It acts as an intelligent guide throughout the contract lifecycle, making it easier to define expectations, detect gaps, and keep contracts aligned with real-world data behavior, **without ever accessing or processing your raw data**. It allows users to: * Add, modify, or remove checks * Update contracts faster without manually writing checks * Understand how a contract was changed {% if !(visitor.claims.plan === %}
{% endif %} {% if visitor.claims.plan === %} #### :camcorder: Watch Contract Copilot in action {% embed url="" %} Contract Copilot is designed to support both: * **Business and analytics users**, who need assistance drafting data contracts and establishing data expectations, and * **Data engineers and platform teams**, who need precise, maintainable, and scalable contracts. ## How does Contract Copilot work? {% hint style="warning" %} This feature requires the **Manage contract** permission on the dataset. Learn more about permissions here: [Dataset Attributes & Responsibilities](/dataset-attributes-and-responsibilities) {% endhint %} {% stepper %} {% step %} **You provide a natural-language request** (e.g. “Add freshness checks”) {% endstep %} {% step %} Soda retrieves relevant **dataset metadata**, such as: * Column names and types * Available check attributes * Data source type {% endstep %} {% step %} This metadata, together with the request, is sent to the model {% endstep %} {% step %} **The model proposes contract updates** directly on the existing contract {% endstep %} {% step %} The user reviews and applies the changes {% endstep %} {% endstepper %}
{% hint style="info" %} **Traceability** Contract Copilot allows you to see what changed directly in the Soda Cloud UI. Requests and outputs can be logged for review and troubleshooting, allowing for cross-team traceability and transparency. {% endhint %} *** ## Enable/Disable Contract Copilot Soda Contract Copilot can be configured from the **Organization Settings**. {% stepper %} {% step %} Click on the **profile icon** on the top right corner → Navigate to **Organization Settings.**
{% endstep %} {% step %} Under the **Organization** tab **> Soda AI:** Check/Uncheck the "Enable Contract Copilot powered by OpenAI" checkbox to enable/disable the feature.
{% endstep %} {% step %} Click **Save** on the top right corner. {% endstep %} {% endstepper %} *** ## Privacy & security {% columns %} {% column %} :square-check: Contract Copilot uses AI/ML techniques to **help you write data contracts**, analyzing: * Dataset schemas * Non-identifiable column-level metadata * Statistical summaries * Existing contract definitions and checks {% endcolumn %} {% column %} :xmark: Contract Copilot **does not** analyze or inspect: * Raw table data * Individual records * Row-level values * PII or sensitive fields {% endcolumn %} {% endcolumns %} ### Confidentiality & access control * Metadata is retrieved from Soda Cloud on a **need-to-know basis** * Only users with **permission to edit or propose contract changes** can access Contract Copilot * All users must authenticate > Learn more about [Soda's AI security & privacy measures](/soda-ai#sodas-ai-security-and-privacy-measures). {% hint style="info" %} Contract Copilot can generate contracts that include [`column_expression`](broken://pages/cCsset8WuU1Ds6xFe7RD) when detecting: * Nested fields * Type mismatches * Casting requirements {% endhint %} ## Limitations * Copilot does not guarantee correctness of checks * Users remain responsible for validating contract logic * Copilot does not operate outside the context of an existing contract {% hint style="success" %} Soda Contract Copilot focuses on **assisting decision-making**, not replacing human ownership of contracts. The output is a set of **suggestions, explanations, or guidance** that you can review and apply. {% endhint %} {% endif %} *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Soda MCP Use Soda from your preferred AI agent. Soda MCP allows your team to **fully manage your data** **from inside the AI agents your team already works in**, such as Claude, Cursor, Codex, or GitHub Copilot. `soda-mcp` is a Model Context Protocol (MCP) server for Soda Cloud. **It lets MCP-capable AI clients use Soda Cloud's public API** to inspect and manage data quality resources such as datasets, checks, contracts, scans, monitors, incidents, attributes, data sources, users, and user groups. Reach for Soda MCP **when you want an AI agent to help with data workflows**, for example: * **List datasets** and **inspect** their health status. * **Retrieve** checks, contracts, scans, and incidents. * **Create** or **update** contracts and monitors. * **Trigger** discovery, contract verification, historical metric collection, or other supported Soda Cloud actions. * **Manage** **Soda Cloud resources** such as attributes, roles, users, and user groups. * **Build complex workflows** across your entire AI layer, connecting to other agents and MCPs {% if !(visitor.claims.plan === %} > Soda MCP is available for Soda Cloud customers. [Request a free account](https://soda.io/pricing) or [book a call](https://soda.io/pricing) to learn more. > {% endif %} {% if visitor.claims.plan === %} > To view the full list of MCP tools, navigate to the [Tools documentation.](broken://pages/jCHplp9NkfS0uNHxgBgL) > {% endif %} #### MCP vs. Soda AI ​[Soda AI](https://app.gitbook.com/o/ig9n9VJPAFRSFLtZKVB2/s/A2PmHkO5cBgeRPdiPPOG/~/diff/~/changes/673/soda-mcp) and Soda MCP **both let you work with Soda Cloud through natural language**, and they overlap on what they can do, but **they suit different jobs**. * **Soda MCP** is the choice for **automated, scheduled, parallel, and background** workflows. Because it runs inside your own tooling you can build cron, parallelism, and completion notifications around it. * **Soda AI**, on the other hand, is a **conversational tool for one-off work**, like drafting a contract, or asking what an incident means, while you're on the loop. | Workflow | Soda MCP | Soda AI | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | **Scheduled runs** | Wire MCP-driven tasks into a cron job or pipeline so they run unattended on a schedule. | Soda AI is designed to run only when you're in a session with it, in order to confirm actions. | | **Parallel work** | MCP tasks can fan out across many datasets or datasources at once, driven by your own orchestration. | Soda AI works through one interactive flow and does not orchestrate tasks, it keeps you in the loop. | | **Background execution with notifications** | A long-running MCP job can run in the background and ping you through whatever tooling you've wired up (Slack, email, your pipeline's own alerts) when it finishes. | Soda AI completes the API call but won't notify you when a background-style job is done. | In short: **use Soda AI for interactive, human-in-the-loop work**, and **Soda MCP** when you want the same kinds of actions **to** **run on a schedule, in parallel, or in the background** as part of an automated workflow. {% if visitor.claims.plan === %} ### Installation instructions The server runs as a local stdio MCP process. Your AI client starts `soda-mcp`, passes Soda Cloud credentials through environment variables, and then calls Soda tools through the MCP protocol. On startup, `soda-mcp` authenticates with Soda Cloud and loads organization context so the client can use the exposed tools against the correct tenant. Follow the instructions in this section to install the Soda MCP server for your host client. #### Requirements * Python 3.10+ * [`uv`](https://docs.astral.sh/uv/) python package on `PATH` * A Soda Cloud API key ID and secret * A Soda Cloud host URL If you don't have API keys already you can create them following the instructions [here](broken://pages/H6Piqnv5lUFCAv1Cvm75#generate-api-keys-for-use-with-soda-core-or-a-soda-cloud-api). Set the runtime environment variables before starting the server: ```bash export SODA_CLOUD_HOST="cloud.soda.io" export SODA_API_KEY_ID="your-key-id" export SODA_API_KEY_SECRET="your-key-secret" ``` :exclamation: **Note:** If you are using Windows without WSL you will need to modify the commands to work with either the Windows command prompt: ```shellscript set SODA_CLOUD_HOST=cloud.soda.io set SODA_API_KEY_ID=your-key-id set SODA_API_KEY_SECRET=your-key-secret ``` or Windows Powershell: ```powershell $env:SODA_CLOUD_HOST = "cloud.soda.io" $env:SODA_API_KEY_ID = "your-key-id" $env:SODA_API_KEY_SECRET = "your-key-secret" ``` You will also need to modify the commands of the following sections as is appropriate. #### Install from Soda's private PyPI Stable releases are published to Soda's private PyPI indexes for Team and Enterprise customers. Choose the private index that matches your license and region, then set `UV_INDEX` using the same Soda API key credentials:
LicenseRegionIndex URL
TeamEUhttps://team.pypi.cloud.soda.io
TeamUShttps://team.pypi.us.soda.io
EnterpriseEUhttps://enterprise.pypi.cloud.soda.io
EnterpriseUShttps://enterprise.pypi.us.soda.io
Pick the appropriate command below according to your plan and region. Note that this command depends on the Soda API key environment variables mentioned above in order to execute successfully. Pick the proper one according to your plan and region: {% tabs %} {% tab title="Team EU" %} {% code overflow="wrap" %} ```bash export UV_INDEX="https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@team.pypi.cloud.soda.io" ``` {% endcode %} {% endtab %} {% tab title="Team US" %} {% code overflow="wrap" %} ```bash export UV_INDEX="https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@team.pypi.us.soda.io" ``` {% endcode %} {% endtab %} {% tab title="Enterprise EU" %} {% code overflow="wrap" %} ```bash export UV_INDEX="https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.cloud.soda.io" ``` {% endcode %} {% endtab %} {% tab title="Enterprise US" %} {% code overflow="wrap" %} ```bash export UV_INDEX="https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.us.soda.io" ``` {% endcode %} {% endtab %} {% endtabs %} Smoke-test that `uvx` can resolve and start the package: 
timeout 10s uvx -qq --no-progress soda-mcp@latest
If everything is set up correctly you will see the following output: ``` Authenticated as FirstName LastName (userEmail@soda.io) in org Soda TestOrg Starting soda-mcp server (base_url=https://cloud.soda.io) ``` #### Configure an MCP client Configure your MCP client to start `soda-mcp` over stdio and pass the required environment variables to the child process. For **Claude Code:** ```bash claude mcp add soda-mcp \ --transport stdio \ --scope user \ -e SODA_CLOUD_HOST="$SODA_CLOUD_HOST" \ -e SODA_API_KEY_ID="$SODA_API_KEY_ID" \ -e SODA_API_KEY_SECRET="$SODA_API_KEY_SECRET" \ -e UV_INDEX="$UV_INDEX" \ -- uvx -qq --no-progress soda-mcp@latest ``` For **OpenCode**, add a local MCP server entry to your user-scope `opencode.jsonc`: ```jsonc { "mcp": { "soda-mcp": { "type": "local", "command": [ "env", "SODA_CLOUD_HOST=", "SODA_API_KEY_ID=", "SODA_API_KEY_SECRET=", "UV_INDEX=", "uvx", "-qq", "--no-progress", "soda-mcp@latest" ] } } } ``` For a generic stdio MCP client: ```json { "mcpServers": { "soda-mcp": { "command": "uvx", "args": ["-qq", "--no-progress", "soda-mcp@latest"], "env": { "SODA_CLOUD_HOST": "", "SODA_API_KEY_ID": "", "SODA_API_KEY_SECRET": "", "UV_INDEX": "" } } } } ``` If you installed from a local wheel, replace the `uvx -qq --no-progress soda-mcp@latest` command with `soda-mcp` in your client configuration. #### Verify the server After configuring the client, confirm that `soda-mcp` appears in the client's MCP server list and is connected. Then invoke a low-cost read tool such as `list_attributes`. A successful response confirms that: * The MCP client can start the server. * Soda Cloud credentials are reaching the server. * Authentication succeeds against the configured Soda Cloud tenant. * The client can see and call the registered Soda tools. {% endif %} {% if visitor.claims.plan === %} ### Usage instructions #### Use the tools Once connected, ask your AI client to perform Soda Cloud tasks naturally. Examples: ``` List the datasets in my Soda Cloud organization and summarize unhealthy datasets. ``` ``` Can you show me the latest failing checks? ``` ``` Verify the contract for the dataset datasource/database/schema/table ``` ``` Use Contract Autopoilot to craete a contract for the dataset datasource/database/schema/table ``` Some tools modify Soda Cloud resources or trigger scans. Review the client's proposed tool call before approving write, delete, or trigger actions. #### Usage notes While you can create and update secrets through the MCP client it is not recommended to do so. For long lived secrets prefer to create them from the Web User Interface. MCP servers expose tools and instructions on how to user them. Some MCP clients, like Codex and OpenCode, do not load the instructions, only the tools. This may affect performance. Claude Code loads both tools and instructions. #### Troubleshooting * If `uvx` cannot resolve `soda-mcp`, confirm `UV_INDEX` is set in the MCP client's environment and points at the correct private index. * If private PyPI returns `401` or `403`, confirm the API key ID and secret are valid for the selected Team or Enterprise index. * If startup fails with missing credential errors, confirm the MCP client passes `SODA_API_KEY_ID`, `SODA_API_KEY_SECRET`, and `SODA_CLOUD_HOST` . * If Soda Cloud authentication fails, confirm the API key belongs to the same tenant as the configured host or URL. * If the client does not show any tools, refresh or restart the MCP client so it reconnects and reloads tool descriptions. {% endif %} *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Data Observability An overview of Soda's key observability features and how they help catch data issues early. ## What is Data Observability? Data observability is the ongoing process of **monitoring and assessing** the health of your data throughout its lifecycle. It focuses on analyzing metadata, metrics, and logs to detect issues as they arise, helping teams maintain trust in their data. At the core of data observability are **monitors** that track key data quality metrics over time. When a metric behaves unexpectedly, anomaly detection algorithms analyze historical patterns to determine whether an alert should be triggered. Typical data quality metrics to monitor are: * **Schema changes** to surface structural modifications * **Row counts** to detect unexpected changes in data volume * **Most recent timestamps** to detect data freshness, missing or delayed data * **Missing values** to track data completeness * **Averages** to observe shifts in distributions
## Soda’s practical approach to Data Quality Soda embraces pragmatism over purity: practical outcomes and effectiveness are more important than ideal, unidimensional approaches. **Effective data quality comes from combining data observability and data testing**. Each serves a different purpose. Observability is about speed and broad coverage. Testing is about precision, enforcement, and prevention. **Benefits of Data Observability** * Enables broad coverage quickly, even across large data sources * Surfaces unknown issues without needing to define every rule * Requires minimal configuration to get started * Leverages existing metadata for fast and efficient monitoring * Provides early signals when something might be wrong **Limitations of Data Observability** * Serves only as a signal. An anomaly suggests an issue but doesn’t confirm it * Can generate false alerts, since detection is driven by algorithms * Requires further investigation to validate and resolve alerts * Does not prevent issues. It flags them after they’ve happened * May result in extra work to follow up and interpret alerts **Start with Observability, but rely on Testing** **Observability is a fast and efficient way to get initial coverage**. It helps surface unknown issues with minimal setup and delivers immediate value across many datasets. However, **for lasting reliability and trust in your data, testing is more important**. Testing requires more effort up front. It involves **defining explicit expectations and rules** for your data. But that investment pays off. When a test fails, you know there is a real data quality issue, no guesswork, no false alerts. When an anomaly is detected, it doesn't necessarily mean there is an underlying data quality issue, and more investigation effort is required. **For long-term reliability, testing is essential**. It adds rigor by enforcing defined standards and helps prevent bad data going into production. Start with your most critical datasets, then expand gradually using a collaborative approach, where business users help by proposing checks. This creates a **scalable framework** that grows with your organization while ensuring **lasting data quality**. ## What makes Soda’s Data Observability so useful? Soda’s data observability allows teams to **monitor data health** across large environments **without manual setup**. All anomalies are surfaced in a single, easy-to-navigate dashboard, making it simple to spot issues and investigate patterns. Behind the scenes, a **proprietary anomaly detection algorithm** ensures high precision by minimizing false positives and focusing on meaningful deviations. Notifications are opt-in and alerts are only triggered when they matter, helping teams stay focused without being overwhelmed by noise.

Metric Monitoring Dashboard

### Effortlessly monitor thousands of tables Soda enables **large-scale observability** with ease. Instead of configuring each table manually, monitoring is applied at the data source level and automatically extends to all datasets underneath. This allows teams to activate observability across **hundreds or even thousands of tables** in minutes. By leveraging metadata such as row counts, schema evolution, and insert activity, **Soda delivers lightweight and efficient monitoring**. There is no need to scan entire datasets or write custom logic for each table. You can do that if needed, but it is not required. Observability starts working immediately and is built to handle even the largest data platforms. ### Start today—look one year back Observability is not just about what happens next. With built-in backfilling and backtesting, **Soda instantly analyzes historical metadata and metric trends**. From the moment observability is enabled, teams gain visibility into past data quality metrics and can detect potential anomalies that may have gone unnoticed. This historical context is essential. It helps determine whether a current anomaly is truly new or part of an ongoing pattern. It also allows the anomaly detection algorithm to establish baselines immediately, which improves the quality of alerts from the very beginning. ### High precision alerts with fewer false positives Soda’s **proprietary anomaly detection algorithm** is specifically designed for data quality monitoring. Every component has been developed entirely in-house without relying on third-party frameworks. This gives Soda full control over the modeling stack and ensures **transparency**, **customization**, and **explainability**. These attributes are especially important in production environments where trust in alerts is essential. The algorithm is built on a **proprietary evaluation framework** that rigorously tests its performance using hundreds of internally curated datasets with known data quality issues. This framework enables structured, repeatable experimentation and continuous benchmarking of new techniques. It prioritizes reducing false positives to ensure alerts are accurate, meaningful, and reliable. In benchmark testing, Soda’s algorithm demonstrated a **70 percent improvement in anomaly detection accuracy compared to Facebook Prophet**. Unlike generic forecasting tools that rely on rigid assumptions, Soda’s model is tailored to the real-world challenges of monitoring data quality at scale. The system is flexible and adapts to different team needs. It can run autonomously with smart defaults or be fine-tuned through a user-in-the-loop approach. Teams can improve detection by providing feedback and adjusting sensitivity. This flexibility ensures that alerts remain focused, useful, and aligned with the needs of each organization. ## Metric Monitoring Soda’s **Metric Monitoring feature** is the foundation of Data Observability, allowing users to automatically **track key dataset and column-level statistics** over time, detect deviations, and get alerted before data issues impact downstream analytics. While quality checks also keep track of measurements over time, metric monitors use that history of measurements to learn from them and automatically adjust thresholds to inform about expected values or alert about anomalies. ### Implement Metric Monitoring at scale Metric Monitoring is developed to be a hassle-free feature. You can unlock organization‐wide observability through Soda Cloud’s [no-code dataset onboarding](/onboard-data-sources-and-datasets): * **Manual** dataset onboarding: allows you to browse and select specific databases, schemas or datasets to onboard them in bulk. * **Rules-based** dataset onboarding: allows you to set rules and automate the dataset onboarding process, not only for current datasets, but also for future ones. This instantly provides automated metric monitoring across hundreds of thousands of tables by simply selecting all the datasets you care about and defining a shared schedule in one step. No more configuring each table by hand: stay ahead of pipeline failures, data delivery delays, and structural changes with consistent, centralized monitoring that grows as fast as your data. > Learn more about [dataset onboarding](broken://pages/cHo9ixopiR0flOMCbyUF).
*** > Learn more about how roles and permissions affect Metric Monitoring capabilities via the [Organization and Admin settings](/organization-and-admin-settings). *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Record-level Anomaly Detection (RAD) This page explains Record-level Anomaly Detection (RAD) and Soda's anomaly detection capabilities through RAD. {% hint style="success" %} **Coming soon!** RAD functionalities will be available soon for [Enterprise plan users](https://www.soda.io/pricing). {% endhint %} ## What is RAD? Ensuring data quality can be difficult, especially when you need broad coverage quickly. Checks and column monitors are great for enforcing specific rules, but they take time to set up and require a deep understanding of your data. **Soda’s Record-level Anomaly Detection (RAD) helps you get started fast, providing instant coverage across all columns, rows, and segments, without any configuration.**
The algorithm analyzes historical data to build a clear picture of what normal data is supposed to look like. When incoming rows show unusual patterns, unexpected values, inconsistencies, or errors, RAD automatically triggers an alert and runs a Root Cause Analysis to pinpoint the issue. This provides quick, actionable insights while you work toward more detailed control using checks and column monitors. ## Why use RAD? 1. **Instant, broad coverage**\ Monitor all columns, rows, and segments at once, detecting both known and unknown issues. 2. **No configuration needed**\ Get started immediately: no metrics or checks need to be defined. RAD automatically determines which columns to use. 3. **One metric to track and alert on**\ The Record-level Drift Score provides a single, explainable metric to monitor data health. * **Assess the impact of data quality issues**: easily determine how many rows in your dataset are affected by data quality problems. * **Prioritize what matters**: use the Record-level Drift Score consistently across datasets and data sources to rank and focus on the most critical data quality issues. * **Reduce false alerts**: traditional column-level monitoring increases the risk of false positives with every additional monitor. With RAD, you only need one anomaly detection monitor per dataset, minimizing noise. * **Optimize compute usage**: monitoring a single metric per dataset lowers computational overhead. Additionally, RAD can work with sampled data, further reducing processing demands. 4. **Built‑in root cause analysis**\ Quickly understand what changed and why. 5. **Native support for backfilling and back‑testing**\ Automatically generate and assess historical Record-level Drift Scores to review past data quality trends. ## When to use RAD? Order of operations to achieve the best coverage in the most efficient way: 1. Firstly, [**dataset‑level metadata metrics**](broken://pages/D1Orsy7Y854tCyYgpmyY): always begin with high‑level monitors to verify if the right amount of data arrived on time and in the correct format. These require no configuration. They just need to be enabled 2. Secondly, **RAD**: apply Record-level Anomaly Detection to validate the actual content of the data. This step also requires no configuration (only enablement) and provides broad coverage across all columns and segments. 3. Next, [**column monitors**](broken://pages/R45r6jTiEnu77LchZMI4): apply column‑level monitoring for specific use cases where the potential data quality issue and metric are known but expected to change over time. These should be minimized as they are prone to generating false alerts. 4. Lastly, [**checks**](broken://pages/22bc4928a6b701b7433fc0695b1b51ab37c12afa): use checks for critical tables where expectations are clearly defined. For example
Data quality toolColumnMetricFailureConfiguration
ChecksKnownKnownKnownMissing values in Amount < 5%
Column monitorsKnownKnownUnknownAnomaly detection on Amount for missing values
RAD monitorUnknownUnknownUnknownRAD on all columns
### RAD requirements For a dataset to be monitored by RAD, the following conditions must be met: * **Time partition column**: the dataset must include a column that [partitions data by time](broken://pages/D1Orsy7Y854tCyYgpmyY#time-partition-column) (for example, `created_at`). * **Primary key**: the dataset must have a primary key to uniquely identify rows. * **Diagnostics Warehouse setup**: a [Diagnostics Warehouse](/diagnostics-warehouse#get-started) must be configured to store the daily sample, consisting of either primary keys or, ideally, a full copy of the sampled rows. > **Next**: to enable Record-level Anomaly Detection in your organization, **reach out to Soda at** [**sales@soda.io**](mailto:sales@soda.io). *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Data Testing ## What is Data Testing? Data testing is the practice of validating that your data meets the expectations you’ve defined for it before it reaches stakeholders, dashboards, or downstream systems. Just like software testing ensures your code behaves as intended, data testing safeguards the quality and reliability of your data. At Soda, we see data testing as **the foundation of data trust**. Whether you’re verifying row counts, checking for missing or invalid values, or enforcing schema integrity, the goal is the same: catch issues early, reduce incidents, and keep your data consumers confident. ## What is a Data Contract? A **Data Contract** is a formal agreement between data producers and data consumers that defines what “good data” looks like. It sets expectations about schema, freshness, quality rules, and more, and makes those expectations explicit and testable. With a data contract in place, producers commit to delivering data that meets certain standards. Consumers, in turn, can rely on that contract to build reports, models, or pipelines without second-guessing the data. At Soda, Data Contracts are **testable artifacts** that can be authored, versioned, verified, and monitored, whether in code or in the UI. They’re the connective tissue between producers and consumers, aligning teams and eliminating ambiguity. ### What is Contract Verification? Defining a contract is only the first step. Verifying that your data actually **meets** the expectations is where the value is realized. Contract verification is the process of testing whether the data in your datasets aligns with the rules, thresholds, and schema defined in the contract. At Soda, contract verification is **fully automated**. Whether triggered manually, on a schedule, or as part of your CI/CD pipelines, each verification run checks that: * The schema matches the contract definition (columns, data types, structure) * The data complies with checks like missing, duplicate, invalid values, and custom rules This helps you catch issues early, ensure data quality over time, and build trust across your organization. ## Authoring and managing Data Contracts: choose your style Soda supports two complementary ways to author and manage data contracts. They are designed to fit the way your team works. ### Cloud-managed Contracts (Soda Cloud UI) If you’re a data analyst, product owner, or stakeholder, who prefers intuitive interfaces over code, **Soda Cloud** is the ideal workspace. With the Soda Cloud UI, you can: * Browse datasets and view profiling insights * Define a contract with a no-code Editor * Schedule and monitor contract verifications * Collaborate with your team and publish contracts with a click There’s no setup or YAML required, just fast, visual workflows that enable domain experts to contribute directly to data quality.
### Git-managed Contracts (Soda Core CLI) If you live in your terminal and manage your data pipelines as code, you’ll want to use **Soda Core** and the **Soda CLI**. With this setup, you can: * Define contracts in YAML * Run contract verifications in CI/CD * Push the contract and verification results to Soda Cloud for visibility * Use Git as the source of truth for version control, collaboration, and reviews * Collaborate with non-technical users that use Soda Cloud and integrate with engineering workflows via Git This path offers full control, transparency, and seamless integration into your dev tooling.
*** ### Combine UI and Git for cross-team collaboration Soda gives you the flexibility to blend both approaches. For example, non-technical users can define or adjust contracts visually in Soda Cloud for the datasets they manage, while engineers can use Git-managed contracts for the datasets they own.
This hybrid model enables collaboration across teams: * Business users bring domain expertise directly into the contract * Engineers maintain quality, consistency, and governance * Each dataset follows the authoring method that best suits the team responsible for it You can mix and match—using the UI for some contracts, and code for others—depending on your team's structure and preferences. And even if Data Contracts are managed in Git, you can still involve non-technical users who can propose changes to a contract in the UI. These approved changes can be embedded into engineering workflows and synced to Git, ensuring that every update follows your organization’s quality and deployment standards. Choose the model, or combination of models, that works best for your organization. *** ## Soda Runner vs. Soda Core for execution Once a contract is published, you’ll want to verify that the actual data meets the contract’s expectations. This verification can be done in two ways: * **Soda Runner** is our managed runner that lives in your environment and connects to Soda Cloud. It handles contract verification, scheduling, and execution securely, without exposing your data externally. It is great for teams who want central management without maintaining CLI infrastructure. * **Soda Core** is our open-source engine you can run anywhere: locally, in CI, or data pipeline. It’s lightweight, customizable, and great for teams that prefer full control or have strict environment constraints. Both approaches support the same Data Contract logic. Choose the one that best fits your deployment model. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Data Standards Write data quality checks once and deploy them across many datasets at scale with Data Standards. {% hint style="success" icon="database" %} Data Standards are currently available as a **preview** feature. Contact for early access. {% endhint %} ## What is a Data Standard? A **Data Standard** is a **reusable set of data quality checks** that you write **once** and deploy across **many datasets** at the same time. Where a Data Contract describes the expectations for a *single* dataset, a Data Standard expresses an expectation that should hold *everywhere* a rule applies, for example: "every dataset that contains an `email` column must have valid, non-missing email addresses," or "every table in the `finance` schema must have a passing `schema` and `row_count` check." Data Standards are how teams enforce **organization-wide data quality governance** without authoring and maintaining a separate contract for every dataset. You define the rule centrally, target the datasets it should apply to, and Soda keeps it enforced as your data landscape grows.

The Data Standards overview, listing each standard with its owner, matched datasets, check results, compliance, and status.

{% hint style="info" %} **Data Standards reuse the same check engine and YAML check language as Data Contracts.** If you already know how to write contract checks, you already know how to write a Data Standard: the checks (`missing`, `invalid`, `duplicate`, `schema`, `row_count`, `freshness`, and so on) are identical. {% endhint %} ## Data Standards vs. Data Contracts Data Standards and Data Contracts are complementary. They share the same check language but solve different problems. | Data Standard | Data Contract | | --------------------------------------------------------------------- | -------------------------------------------------------- | | One expectation that should hold in all datasets it applies | A tailored agreement for the expectations in one dataset | | Written once, deployed across a fleet | Written per dataset | | Scope: many datasets at once, selected by a rule | Scope: a single dataset | | When onboarding, matching datasets inherit the standard automatically | Suitable for new datasets that need their own contract | | Best for organization-wide governance enforced consistently at scale | Best for a dataset that deserves its own bespoke rules |
AspectData StandardData Contract
ScopeMany datasets at once, selected by a scope ruleA single, specific dataset
IntentEnforce a quality rule in all datasets it appliesDefine what "good data" means for this dataset
Authored byData governance / platform teams setting org-wide policyThe team that owns the dataset
IdentityNamed rule that fans out to every matching datasetBound to one dataset's fully qualified name
Typical use
  • "All PII columns must be valid", "every table must pass schema"
  • Matching datasets inherit a standard automatically
  • Producer/consumer agreement on a key table
  • A new dataset needs its own tailored contract
Use a **Data Contract** when a dataset deserves its own tailored agreement. Use a **Data Standard** when you want the *same* rule applied consistently across a fleet of datasets, whether it is a whole contract, or a check for a specific field. ## How to implement a Data Standard {% hint style="warning" %} Creating and managing Data Standards requires the **Manage data standards** permission. Data Standards must also be enabled for your organization. Learn more about [Broken mention](broken://pages/n914KXqJNBoZ19Ay5W0P) {% endhint %} {% stepper %} {% step %} ### Write the standard Define the data quality checks you want to enforce, using the same check language as a Data Contract. > See [Author a Data Standard](/data-testing/data-standards/author-a-data-standard). > {% endstep %} {% step %} ### Deploy it at scale Create the standard in Soda Cloud and give it a **scope**, a rule that selects which datasets it applies to. Soda links the standard to every matching dataset and re-evaluates the scope as datasets change. > See [Deploy at scale](/data-testing/data-standards/deploy-at-scale). > {% endstep %} {% step %} ### Verify the data Soda runs the standard's checks against every linked dataset, on a schedule or on demand. You can also trigger a verification for a dataset from the CLI. > See [Verify a Data Standard](/data-testing/data-standards/verify-a-data-standard). > {% endstep %} {% step %} ### Review the results Inspect pass/fail results, history, and diagnostics for each check across every dataset the standard touches. > See [Review results](/data-testing/data-standards/review-results). > {% endstep %} > {% endstepper %} ## Lifecycle A Data Standard moves through a small set of states: * **Draft** – created but not yet enforced. No datasets are linked and no scans run. * **Active** – enforced. The standard is linked to every dataset that matches its scope, and its checks run on the configured schedule. * **Paused** – temporarily suspended. Existing links are kept, but no new scans run until you resume it. ## Prerequisites Before working with Data Standards, make sure: * You have a **Soda Cloud account** with the **Dataset Standard**, **Team**, or **Enterprise** license. * You have connected at least one **data source** and onboarded datasets. See [how to onboard data sources](broken://pages/cHo9ixopiR0flOMCbyUF). * To create or manage standards, your user needs the **Manage Data Standards** permission. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Author a Data Standard Define the checks that make up a Data Standard. A Data Standard is authored as a set of data quality checks, using the **same check language as a Data Contract**. If you can write contract checks, you can write a Data Standard: the `columns:` list, dataset-level `checks:`, thresholds, filters, [Broken mention](broken://pages/QkO7w20yeaFP3U9Ow5Kk), and variables all work exactly as they do in a contract. **You focus on the checks. Soda manages the rest**, the standard's identity and the datasets it applies to, so those are not part of the check definition you write. > For the full catalogue of check types and configuration options, see the [Contract Language reference](/reference/contract-language-reference). ## Author a Data Standard A Data Standard's definition is a `columns:` list and/or a dataset-level `checks:` block, just like a contract: {% code title="data-standard.yml" %} ```yaml columns: - name: email checks: - missing: - invalid: valid_format: name: Email format regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$ ``` {% endcode %}

The Data Standard editor: basic information and scope on the left, the standard's checks on the right.

*** ## Match columns by pattern or attribute A Data Standard runs across many datasets, where the same kind of data can live under different column names. Rather than listing explicit column names, **you can match columns dynamically**, by name pattern or by attribute, using a `match:` block in place of `name:` on a column entry. The checks then **apply to every column that matches**, in every dataset the standard runs against.

A standard using a match block to apply its checks to columns selected by attribute.

### Match by name pattern Use `pattern` with a regular expression to **match column names**. The example below applies its checks to every column whose name contains `email` (case-insensitive, via the `(?i)` flag), and uses `exclude` to skip specific columns. {% code title="data-standard.yml" %} ```yaml columns: - match: pattern: "(?i).*email.*" exclude: any_of: [email_archived] checks: - missing: qualifier: c1 - invalid: qualifier: c2 valid_format: name: Email Regex regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$ ``` {% endcode %} `exclude.any_of` takes a list of column names to omit from the match, even if they would otherwise match the pattern. ### Match by attribute Use `attributes` to **match columns that carry a specific attribute value**. The example below applies its checks to every column whose `term` attribute equals `Email`, regardless of the column's name. {% code title="data-standard.yml" %} ```yaml columns: - match: attributes: { name: term, equals: Email } checks: - missing: - invalid: valid_format: name: Email Regex regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$ ``` {% endcode %} {% hint style="info" %} **A single `match` can expand to several columns**, and each expanded check needs a stable identity. Use a `qualifier` to distinguish checks of the same type so their results are tracked consistently over time. Standards generated in Soda Cloud populate qualifiers for you. Learn more about [Contract Language reference](/reference/contract-language-reference#check-qualifiers). {% endhint %} *** ## Targeting datasets References to specific checks do not take place in the checks of a Data Standard. Instead, the datasets a standard applies to are determined by its **scope**, which you configure when you deploy the standard in Soda Cloud. Soda resolves the scope and runs the standard's checks against every matching dataset. See [Deploy at scale](/data-testing/data-standards/deploy-at-scale). *** ## Example: a fuller standard This standard enforces several common quality rules. It can be deployed across every dataset that holds customer contact data. {% code title="data-standard.yml" %} ```yaml columns: - name: email checks: - missing: - invalid: name: Email format valid_format: name: Email format regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$ - name: phone checks: - invalid: valid_format: name: Phone format regex: ^\+?[0-9\-\s()]+$ - name: customer_id checks: - missing: - duplicate: checks: - row_count: ``` {% endcode %} *** ## Supported fields Data Standards support the same fields as Data Contracts: ### Dataset-level checks Like contracts, **standards support dataset-level checks** such as `schema`, `row_count`, `freshness`, and `duplicate` across multiple columns. A standard that defines **only** dataset-level checks still requires a `columns:` key; an empty list is enough. 
checks:
  - schema:
      allow_extra_columns: false
  - row_count:

columns: []
### Variables Standards support the same `${var.VARIABLE_NAME}` variable substitution as contracts, which is useful when the same rule needs slightly different parameters per environment or schedule. Declare variables at the top of the definition and pass values at verification time with `--set`. > Learn more about variables: [Contract Language reference](/reference/contract-language-reference#make-contracts-dynamic-with-variables). *** ## Next steps * Deploy the standard across your datasets: [Deploy at scale](/data-testing/data-standards/deploy-at-scale) * Verify it from the CLI: [Verify a Data Standard](/data-testing/data-standards/verify-a-data-standard) *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Deploy at scale Set up a Data Standard with a scope to enforce it across many datasets at once. Writing a Data Standard defines a rule. **Deploying** it is what **makes that rule enforceable** across your data landscape. When you create a Data Standard in Soda Cloud, you give it a **scope**, and Soda automatically links the standard to every dataset the scope selects. Then, Soda keeps that set of datasets up to date as your data landscape changes. That is the core value of Data Standards: **you maintain one rule**, and **it applies to every dataset** that should follow it, including datasets onboarded after the standard was created. {% hint style="warning" %} Creating and managing Data Standards requires the **Manage data standards** permission. Data Standards must also be enabled for your organization. {% endhint %} ## Elements of a Data Standard A Data Standard is made up of the following:
FieldDescription
NameA unique name that identifies the standard within your organization.
LabelA human-readable display name shown in Soda Cloud.
DescriptionOptional context explaining what the standard enforces and why.
ContentsThe Data Standard YAML, including the checks to run.
ScopeThe rule that selects which datasets the standard applies to (see below).
StatusThe lifecycle state: draft, active, or paused.
OwnersThe users or groups responsible for the standard. At least one owner is required.
## Scope: selecting datasets The **scope** determines **which datasets** a standard is enforced on. You define it in the editor using one of two modes: * **Rules** – match datasets by pattern. Each rule targets a **data source**, with an optional **schema pattern** and **dataset pattern**. Patterns support wildcards (for example, `customer_data_*`), so a single rule can match many datasets, and you can add several rules to cover datasets across different data sources or schemas. * **Datasets** – select specific datasets explicitly.

Defining scope with rules: each rule matches datasets by data source, schema pattern, and dataset pattern.

Because rules match by pattern rather than a fixed list, **Soda re-evaluates the scope against your catalog** and links the standard to every matching dataset. You can re-run this at any time with **Refresh scope** to confirm exactly which datasets the standard targets. {% hint style="info" %} **Scope is evaluated continuously**, not just once. When a new dataset is onboarded that matches an active standard's scope, the standard is automatically linked to it, no manual step required. This is what keeps a standard enforced "at scale" as your data grows. {% endhint %} ## Status and lifecycle A standard's **status** controls whether and how it is enforced: {% stepper %} {% step %} ### Draft **The standard is saved but not enforced.** Use this state to author and review a standard before it affects any data. No datasets are linked and no scans run. {% endstep %} {% step %} ### Active **The standard is enforced.** Soda links it to every dataset matching its scope and runs its checks on the configured schedule. New matching datasets are picked up automatically. {% endstep %} {% step %} ### Paused **Enforcement is temporarily suspended.** Existing dataset links are preserved, but no new scans run until you set the standard back to "active". Use this to pause a noisy or in-progress standard without losing its configuration. {% endstep %} {% endstepper %}

Switch a standard's status from its page in Soda Cloud.

## Ownership Every Data Standard has one or more **owners** (users or user groups). Owners are responsible for the standard and are who Soda surfaces when a standard needs attention. Assign owners that reflect accountability for the rule, typically a data governance or platform team. ## Scheduling Active standards run their checks on a schedule, the same cron-based scheduling model used for contract verification. This means a standard, once active, keeps validating every linked dataset over time without manual intervention. > Learn how scheduling works: [Broken mention](broken://pages/adn2iQZu7t1aWGfkGzSJ#scheduled-execution-from-the-contract-editor). ## How results stay organized Because a standard runs across many datasets, Soda tracks its results **per dataset** while keeping them grouped under the standard. This lets you answer both questions that matter at scale: * *Is this dataset compliant with all the standards that apply to it?* * *Is this standard being met across every dataset it targets?* > See [Review results](/data-testing/data-standards/review-results). ## Next steps * Run a standard from the CLI: [Verify a Data Standard](/data-testing/data-standards/verify-a-data-standard) * Review results across datasets: [Review results](/data-testing/data-standards/review-results) *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Verify a Data Standard Run a dataset's Data Standard checks on demand with the Soda CLI. Verifying a Data Standard **runs its checks against real data** and **reports** whether the data meets the rule. Active standards are verified **automatically on their schedule** in Soda Cloud. You can also trigger a verification **on demand** from the **Soda CLI**, which is useful for CI/CD pipelines, custom orchestration, or reproducing a scheduled run locally. ## Verify a dataset's standards Use `soda data-standard verify` with `--dataset` (`-d`) to **fetch and run all active standards** that apply to a specific dataset, as resolved by their scopes in Soda Cloud. ```bash soda data-standard verify \ --dataset datasource/db/schema/customers \ --data-source ds_config.yml \ --soda-cloud sc_config.yml ``` This command: * Fetches the active standards for the dataset from Soda Cloud, already resolved with the dataset and its active columns * Connects to your data source using the configuration file * Runs the checks and returns a pass/fail result Because the standards are resolved by Soda Cloud, this run mirrors exactly what the scheduled cloud-managed verification does. {% hint style="info" %} `--dataset` requires `--soda-cloud`, since the active standards are fetched from Soda Cloud. {% endhint %} ## Publish results to Soda Cloud Add `--publish` (`-p`) to **send verification results to Soda Cloud** for visibility, history, and alerting. ```bash soda data-standard verify \ --dataset datasource/db/schema/customers \ --data-source ds_config.yml \ --soda-cloud sc_config.yml \ --publish ``` > Learn how to connect the CLI to Soda Cloud: [Core CLI reference](/reference/cli-reference#connect-to-soda-cloud). ## Override variables If a standard declares variables, pass values at verification time with `--set`: ```bash soda data-standard verify -d datasource/db/schema/customers -ds ds_config.yml -sc sc_config.yml --set ENV=prod ``` ## Run a subset of checks Use `--check-paths` (`-cp`) to run **only specific checks**. This is useful for targeted debugging of a single check. A dataset can have several standards active at the same time, so a check path names **which standard** the check belongs to as well as where it sits inside that standard: ``` data-standard.: ``` ```bash soda data-standard verify -d datasource/db/schema/customers -ds ds_config.yml -sc sc_config.yml \ -cp data-standard.pii_rules:columns.email.checks.missing ``` Pass several paths to run a subset that spans standards: ```bash soda data-standard verify -d datasource/db/schema/customers -ds ds_config.yml -sc sc_config.yml \ -cp data-standard.pii_rules:columns.email.checks.missing \ data-standard.freshness_rules:columns.updated_at.checks.freshness ``` {% hint style="info" %} **On Soda Cloud**, the check path of a given check is on the check page, in the check info panel. {% endhint %} Because `.` and `:` separate the parts of a check path, a Data Standard's **name** cannot contain either character. ## Command options
OptionDescription
--dataset, -dA Soda Cloud dataset identifier. Fetches and verifies that dataset's active standards. Required.
--data-source, -dsOne or more data source configuration file paths. The first is treated as primary. Required.
--soda-cloud, -scPath to a Soda Cloud configuration file. Required.
--publish, -pSend the verification results to Soda Cloud.
--setSet a variable value, in the form --set NAME=VALUE. Repeatable.
--check-paths, -cpOne or more check paths to run a subset of the checks. Each path names its standard: data-standard.<name>:<path>.
--diagnostics-warehouse, -dwPath to a diagnostics warehouse configuration file.
--verbose, -vDisplay detailed logs during execution.
## Next steps * Review verification results across datasets: [Review results](/data-testing/data-standards/review-results) *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Review results Review Data Standard check results across every dataset they apply to. Once a Data Standard runs, whether on its schedule or from the CLI, its **results are available in Soda Cloud**. Because a standard applies to many datasets, you can read its results from two complementary angles: **per dataset** (is this dataset meeting the standards that apply to it?) and **per standard** (is this rule being met everywhere it's enforced?). ## Check outcomes Each check produced by a standard reports one of the following outcomes for its most recent run: * **Passed** – the data met the check's expectation. * **Warned** – the data crossed a threshold configured to raise a warning rather than a failure (using `level: warn`). * **Failed** – the data violated the check. * **Error** – the check could not be evaluated due to a configuration, connectivity, or runtime issue. Soda tracks each check's results under its standard, so the same check is correlated consistently every time the standard runs and across every dataset it touches. This is what lets you trust a check's history even as a standard fans out across your data. ## Per-dataset view From a dataset's page in Soda Cloud, you can see the checks contributed by every standard that applies to that dataset, alongside any contract checks. This answers: *does this dataset comply with all the rules that target it?*

Checks contributed by Data Standards appear on the dataset, with a Linked standards panel listing each applied standard and its check count.

A **Linked standards** panel lists every standard applied to the dataset and how many checks each one contributes, so you can see at a glance which rules are in force. Selecting a check opens its detailed view, with the same tabs used for contract checks: * **Historical plot** – how the check has behaved across runs over time, helping you spot recurring issues and confirm fixes. * **Result** – a summary of the most recent run: status, measured value, time of execution, and any linked incident. * **Diagnostics** – execution metadata explaining how the result was computed (rows tested, failed rows count, and so on). * **Failed rows** – when failed row storage is configured, the specific records that violated the rule. > These tabs work the same way as for contract checks. See [Broken mention](broken://pages/hqOaUlrcYskMagIB0QNj) for a detailed walkthrough. ## Per-standard view From a standard, you can see how it is performing across **every dataset** it is linked to: the number of matched datasets, the overall passing rate, and a **Matched datasets** list with each dataset's check results and compliance. This answers: *is this rule being met across the fleet?*, and helps you quickly identify which datasets are out of compliance with an organization-wide policy.

A standard's page: matched datasets, overall passing rate, scope, and per-dataset compliance.

## Store failed rows and results in the Diagnostics Warehouse Failed rows and check results from Data Standard verifications can also be stored in your [Diagnostics Warehouse](/diagnostics-warehouse), a location in your own data source that Soda writes diagnostic data to. This keeps the underlying records and results in your environment, where you can query them directly, join them with other data, and investigate failures at scale without the data leaving your infrastructure. > Learn how to set this up in [Broken mention](broken://pages/E1KTiEL4c6zytpedbEZg). ## Notifications and incidents Because Data Standard results flow into the same results model as the rest of Soda Cloud, you can route failures through **notifications** and turn them into **incidents** for triage and follow-up, using check attributes to control who gets alerted. > Learn more about [Broken mention](broken://pages/GAK0Cdoq2hZmucVmrovV) and [Broken mention](broken://pages/SbOLXiiSRxTpmSKQeFyW). ## Closing the loop **Reviewing results** is what turns a Data Standard from a written policy into an **enforced** one. The standard defines what "good data" means across your organization; these views show, for every dataset it touches, whether the data currently meets that bar, and give you the detail needed to investigate and resolve any violation. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Failed Rows check {% hint style="info" %} In the Soda Cloud no-code builder, Failed Rows Checks are labelled **SQL Expression**, **SQL Query – Failed Rows**, and **SQL Query – Failed Keys** {% endhint %} ## `expression` vs `query` vs `keys_query` A `failed_rows` check flags individual rows that break a rule and sends them to the Diagnostics Warehouse (if it's set up). * You must provide **exactly one** of `expression`, `query`, or `keys_query`. * The parser rejects zero or more than one (`failed_rows_check_yaml.py`). * All three support `metric: count` (default, threshold `= 0`) or `metric: percent`.
| | **SQL Expression** (`expression`) | **SQL Query – Failed Rows** (`query`) | **SQL Query – Failed Keys** (`keys_query`) | | -------------------------------- | -------------------------------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | **What you write** | A **boolean predicate** that is TRUE for a failing row | A **full `SELECT`** returning the failing rows | A **full `SELECT`** returning only the key column(s) of the failing rows, which should be mapped to the Diagnostics Warehouse key column(s) | | **How Soda counts failures** | `COUNT(CASE WHEN THEN 1)` — Soda builds the query | Counts rows returned by your query | Counts rows returned by your query (identical to `query`) | | **Where failed rows are stored** | **Shared** diagnostics-warehouse tables (`fk_`/`fr_`) | **Per-check** table (`frq_`) | **Shared** diagnostics-warehouse tables (`fk_`/`fr_`) | #### When to use * **Boolean condition on one dataset?** → **`expression`**. Simplest; Soda applies filters, derives passing rows, and computes rows-tested for you. * **Need custom SQL (joins/aggregations) and want the failing rows themselves, stored per-check?** → **`query`**. * **Same, but you want only key identifiers stored (hashed, in the shared diagnostics warehouse) for privacy/correlation, or your rule is key-based?** → **`keys_query`**. ### 1. SQL Expression (`expression`) A **boolean SQL expression that evaluates to TRUE for a failing row**. It functions similar to a `WHERE`-clause predicate. Soda counts a row as failing only when the expression is TRUE, via `COUNT(CASE WHEN THEN 1)`; rows where it's FALSE **or NULL/UNKNOWN** pass (`failed_rows.py`, `failed_rows_check.py`). The expression is transparent to Soda, which means this is the most capable form: * Soda can **derive the passing (complement) set** of rows. * Soda **applies the dataset filter and check-level `filter:`** at scan time. * Soda **computes `check_rows_tested` automatically**, so `metric: percent` needs no extra query. {% code overflow="wrap" %} ```yaml checks: - failed_rows: expression: "discount > list_price" # rows where this is TRUE are failures ``` {% endcode %} {% hint style="success" %} **Use it when** the rule is a row-level boolean condition on columns of the *same* dataset (ranges, allowed values, cross-column comparisons, null logic). It's the simplest and the only form where Soda understands and can scope the query. {% endhint %} ### 2. SQL Query – Failed Rows (`query`) A **verbatim, full `SELECT`** you supply; every row it returns is a failed row. Soda treats it as opaque SQL, which means it cannot compute the complement and **cannot express it as a dataset+filter AST**, so the **dataset filter is not applied** (`failed_rows.py`). The failing rows land in a **per-check diagnostics table** (`frq_`). {% code overflow="wrap" %} ```yaml checks: - failed_rows: query: | SELECT o.* FROM orders o LEFT JOIN customers c ON o.customer_id = c.id WHERE c.id IS NULL -- orphaned orders metric: percent rows_tested_query: "SELECT COUNT(*) FROM orders" # required for percent ``` {% endcode %} {% hint style="success" %} **Use it when** the failing logic needs joins, aggregations, CTEs, or cross-dataset lookups **and you want the actual failing rows** captured/displayed. For `metric: percent` you must supply `rows_tested_query` for the denominator (the parser errors otherwise). {% endhint %} ### 3. SQL Query – Failed Keys (`keys_query`) Also a **verbatim, full `SELECT`**, but it returns only the **key/identifier column(s)** of the failing rows. **Counting is identical to `query`**. The returned keys are **hashed by Soda and routed into the *****shared***** diagnostics-warehouse tables (`fk_`/`fr_`)** rather than a per-check `frq_` table (`failed_rows_check.py`, `failed_rows_check_yaml.py`). It additionally supports **`key_column_mapping`**, which maps each diagnostics-warehouse key-column name to the output column of your query that supplies it (identity mapping when omitted).
{% code overflow="wrap" %} ```yaml checks: - failed_rows: keys_query: | SELECT order_id, line_no -- composite key of failing rows FROM order_lines GROUP BY order_id, line_no HAVING COUNT(*) > 1 -- duplicates key_column_mapping: dwh_order_id: order_id dwh_line_no: line_no metric: count ``` {% endcode %} {% hint style="success" %} **Use it when** you want **key-based diagnostics stored centrally.** The keys are hashed and share a warehouse, so this suits **not persisting full row payloads** (privacy/sensitive tables), and identifier-centric rules (dedup, orphans, cross-source mismatches). Same limitations as `query`: opaque to Soda, dataset filter not applied, `rows_tested_query` required for `percent`. {% endhint %} ## Shared behavior & limitations * Diagnostics carry three SQL fields end-to-end: `failedRowsSourceQuery`, `failedRowsDataQuery`, and `failedRowsKeysQuery` (`soda_cloud_failed_rows_diagnostic.py`). * **Sample collection must be enabled** organization wide (Organization Settings → *Allow Soda to collect sample data and failed row samples*); default sample volume is **100 rows**, adjustable via the samples limit (set `0` to disable). * **Dataset filters do not apply** to `query`/`keys_query`. Scope with a `WHERE` clause inside the query yourself. Only `expression` respects the dataset filter and check `filter:`. * `rows_tested_query` is **only** used with `query`/`keys_query` * Supplying it with `expression` just logs a warning. * `key_column_mapping` is only used with `keys_query`. # Diagnostics Warehouse Diagnostics Warehouse provides a clear, detailed view of the state of data checks while allowing access to failed rows in order to take a closer look and resolve data quality issues. {% if visitor.claims.plan === %} {% hint style="info" %} Can't find what you're looking for? *You are seeing the **Free license** documentation.* **Get access to Diagnostics Warehouse private documentation** with an **Enterprise license**. If you are in the wrong organization: 1. **Log out and back into Soda** with an organization under **Enterprise license** 2. **Refresh this page** (Soda and docs must be open in the same browser) Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% endif %} {% if visitor.claims.plan === %} {% hint style="info" %} Can't find what you're looking for? *You are seeing the **Teams license** documentation.* **Get access to Diagnostics Warehouse private documentation** with an **Enterprise license**. If you are in the wrong organization: 1. **Log out and back into Soda** with an organization under **Enterprise license** 2. **Refresh this page** (Soda and docs must be open in the same browser) Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% endif %} {% if !(visitor.claims.plan === %} {% hint style="info" %} Can't find what you're looking for? **Get access to Diagnostics Warehouse private documentation**: 1. **Log into Soda** 2. **Refresh this page** (Soda and docs must be open in the same browser) Note: Your organization must be under an **Enterprise license** to view private docs. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% endif %} ## Overview ### What is Diagnostics Warehouse? The **Diagnostics Warehouse is a place inside your data warehouse** (in one or multiple schemas) **reserved for Soda to store data quality information**. *** With Diagnostics Warehouse, Soda stores all scans, failed records, and historical data quality issues directly in your own data warehouse. **Nothing is stored outside.** This gives you the ability to run diagnostics, resolve issues, and see exactly why problems happen. You can go as deep as you need, from a single record to a full dataset. Each time a Soda scan runs, Diagnostics Warehouse stores failed rows together with **check**, **scan**, and **dataset attributes & metadata**. With that information, data teams can quickly **diagnose and resolve issues at both row and dataset level**. Additionally, Soda's Diagnostics Warehouse makes it easier for teams to build on top of Soda's outcomes to set up operation workflows, and connect to BI tools you already know and trust. ### Features & capabilities * **Full diagnostic information in one place**, including attributes. * **Faster root-cause analysis:** jump from a failed check to the exact failed rows, affected datasets/columns, and prior history to see if it’s a one-off issue or a pattern. * **Open & portable features:** it’s just tables in your warehouse. Query with SQL, power dashboards, join with lineage, incident, or cost data, and automate workflows. * **Privacy and transparency:** Diagnostics Warehouse stores tables in your own warehouse, giving you full control over security, retention and access. ### Security & governance * **Data minimization:** Diagnostics Warehouse stores metadata about runs and checks and, **for row-level checks**, it only stores failed rows when the option is enabled. * **Warehouse residency:** Diagnostics are **not stored in Soda**. They live in your analytics warehouse, respecting your access controls, encryption, and audit trails. * **No destructive access**: Soda does not require permissions to delete data. It only **evaluates** checks, **stores** metadata, and **moves** failed rows into a designated location.\ Cleanup and garbage-collection queries are provided, but Soda never drops tables itself. ### Supported data sources Diagnostics Warehouse integrates with a range of data sources so that you can centralize, analyze, and act on diagnostics data across your organization. | Data source | Support status | Notes | | -------------------------- | ----------------- | ----------------------------------- | | **Athena** | ✅ Fully supported | Full functionality and scalability. | | **BigQuery** | ✅ Fully supported | Full functionality and scalability. | | **Databricks**1 | ✅ Fully supported | Full functionality and scalability. | | **Dremio** | ✅ Fully supported | Views are not supported in Dremio. | | **OracleDB** | ✅ Fully supported | Full functionality and scalability. | | **Postgres** | ✅ Fully supported | Full functionality and scalability. | | **Snowflake** | ✅ Fully supported | Full functionality and scalability. | | **SQL Server** | ✅ Fully supported | Full functionality and scalability. | | **Redshift** | ✅ Fully supported | Full functionality and scalability. | | **MS Fabric** | ✅ Fully supported | Full functionality and scalability. | | **Synapse** | ✅ Fully supported | Full functionality and scalability. | 1 Supports Spark DataFrame scans in Databricks when Diagnostics Warehouse is configured on Databricks/Unity Catalog. *** ### Get started 1. **Enable Diagnostics Warehouse** in your Soda data source settings. 2. Grant the service **identity permission** to create and write to the Diagnostics Warehouse schema in your warehouse. 3. **Run your checks**; Diagnostics Warehouse tables populate automatically. 4. Query your warehouse and connect to your BI tools to **start exploring**. > **Next**: to enable Diagnostics Warehouse in your organization, **reach out to Soda at** [**sales@soda.io**](mailto:sales@soda.io). *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Onboard data sources & datasets Before you can monitor, test, or enforce data contracts in Soda, you need to **connect your data source and onboard your datasets**. **Onboarding** is the process of: * **Connecting Soda to your data source** (e.g., Snowflake, BigQuery, Databricks, PostgreSQL, etc.) * **Discovering** available datasets (tables or views) * **Selecting** which datasets to monitor and/or validate * **Configuring** observability, testing, and scheduling behavior How you onboard datasets depends on how you deploy Soda. *** ### Choose your onboarding path Soda supports two main ways to onboard datasets: {% if visitor.claims.plan === %} {% columns %} {% column %} Via Soda Cloud This approach is ideal if you want: * No-code dataset onboarding * Automated dataset discovery * Built-in metric monitoring * Data contracts created in the UI * Centralized scheduling and alerting {% endcolumn %} {% column %} Via Soda Core This approach is ideal if you want: * Full control in code * Git-based workflows * CI/CD integration * Contract verification inside pipelines * Support for in-memory datasets {% endcolumn %} {% endcolumns %} {% endif %} {% if !(visitor.claims.plan === %} {% columns %} {% column %} #### Via Soda Cloud This approach is ideal if you want: * No-code dataset onboarding * Automated dataset discovery * Built-in metric monitoring * Data contracts created in the UI * Centralized scheduling and alerting {% endcolumn %} {% column %} #### Via Soda Core This approach is ideal if you want: * Full control in code * Git-based workflows * CI/CD integration * Contract verification inside pipelines * Support for in-memory datasets {% endcolumn %} {% endcolumns %} {% endif %} You can also combine both approaches: use Soda Cloud for **centralized observability and governance**, and Soda Core for **pipeline-level validation**. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Manage data quality issues Learn about the key features and workflows in Soda for managing data quality issues and reporting. **Detecting a data quality issue is only the start**. The harder part is the work that follows: knowing **which** issues matter, **who** should look at them, **what** the business impact is, and **how** to drive them to resolution before downstream consumers are affected. **Soda Cloud** gives data and platform teams **a single place** to make that work manageable **at scale**, turning thousands of individual check results into a coherent operational picture of data health across your organization. ## Soda for issue management Most data quality tooling stops at "the check failed." That leaves the rest of the workflow—triage, ownership, communication, and resolution—scattered across Slack threads, tickets, and tribal knowledge. **Soda closes that gap**. {% columns %} {% column %} #### See the whole estate at a glance Track the health of every onboarded dataset and every active check from one view. Spot trends, identify the datasets most at risk, and measure how data quality is evolving over time across teams, domains, and data sources. {% endcolumn %} {% column %} #### Focus on what matters to your team Slice the picture by domain, owner, criticality, environment, or any custom dimension that reflects how your organization actually works. Save the views your team relies on and share them with stakeholders. {% endcolumn %} {% endcolumns %} {% columns %} {% column %} #### Investigate without leaving the platform When something fails, drill from a high-level health score down to the dataset, the column, the check, and the historical trend behind the failure, with the diagnostic context needed to understand scope and impact. {% endcolumn %} {% column %} #### Get the right people involved, fast Route alerts to the teams and channels that own the affected data, open incidents that track resolution to closure, and keep a record of what was broken, what was done, and why. {% endcolumn %} {% endcolumns %} ## Soda Cloud capabilities Soda Cloud is the layer that sits on top of your data quality checks, whether they're authored as data contracts, monitored as metrics, or run from [Soda Core](https://github.com/sodadata/soda-core), and turns them into an **operational workflow**. At a high level, teams use it to: * **Monitor data health across the organization** with executive-level dashboards covering scan activity, failing checks, and overall health scores, filterable by the dimensions that matter to your business. * **Browse datasets and checks** with rich filtering, search, and saveable views, so each team can carve out the slice of the estate they're responsible for. * **Tag datasets and checks with custom metadata** (domain, criticality, data layer, owning team, sensitivity) and use that metadata to drive filtering, reporting, and alert routing consistently across the platform. * **Analyze check and monitor results over time** with historical trend views and per-check-type diagnostics that surface the *why* behind a failure, not just the *what*. * **Send the right alert to the right place** through a notification system that targets recipients based on the properties of the affected data, rather than blasting every failure to a single channel. * **Run investigations as incidents** with assigned ownership, status tracking, resolution notes, and integrations into the tools your on-call and operations teams already use. ## Collaboration across roles Issue management in Soda is designed around the reality that **data quality is a team sport**. Data engineers, platform owners, analysts, and business stakeholders all need different views of the same underlying signal. {% hint style="info" %} Saved views, custom attributes, and scoped notification rules let each persona work from a **tailored slice of the same source of truth**, without forking dashboards or duplicating logic. {% endhint %} ### Integrate Soda with your existing tools The built-in dashboards cover the most common needs, but every organization eventually wants to roll data quality metrics into its own reporting layer. **Soda exposes its data through a public REST API**, so you can pull health scores, check results, and incident data into **Power B**I, **Tableau**, **Looker**, or any internal tool your business already trusts. ## Soda Core vs Soda Cloud **Soda Core** gives you a powerful, scriptable way to run checks against your data and get pass/fail output wherever you invoke it: perfect for **CI pipelines**, **orchestrators**, and **local validation**. **Soda Cloud provides the layer above**: cross-team dashboards, historical trends, routing of alerts to the right recipients, incident workflow, and shared metadata model. Those are the problems Soda Cloud solves once a single team's checks become an organization's data quality program. If you've outgrown the "check ran, here's the result" loop and need to manage data quality as an ongoing operational discipline across many teams and many datasets, the Soda Cloud features are what makes that step possible. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Dataset Attributes & Responsibilities Dataset settings allow you to define key metadata, ownership, and business context for your datasets. This information helps ensure data governance, accountability, and seamless integration with other tools like your data catalog. ## Dataset Owner #### Purpose of the Dataset Owner Each dataset should have a designated **dataset owner**: a person or team responsible for the dataset's quality, availability and usage. **Typically, the role of a Dataset Owner includes:** * Defining and maintaining the dataset's purpose and documentation. * Ensuring the dataset meets data quality standards and contract requirements. * Responding to issues, such as failed checks or data quality alerts. * Reviewing and approving changes to the dataset schema or contract. {% if visitor.claims.plan === %} #### Updating the Dataset Owner {% hint style="warning" %} Only users with the **Configure Dataset** permission on the dataset can update the dataset owners. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles) {% endhint %} To assign a **Dataset Owner**: {% stepper %} {% step %} Open the **dataset page**. {% endstep %} {% step %} Click the **context menu** (⋮) in the top-right corner and select **Edit Dataset**.
{% endstep %} {% step %} In the **Owned by** section, select one or more users and/or user groups.
{% endstep %} {% step %} Fill in the optional information and click **Save** (top right) to apply the changes. {% endstep %} {% endstepper %} {% endif %} ## Responsibilities #### What are Responsibilities? **Responsibilities** allow you to assign permissions to users or user groups, ensuring they have the access they need to work with a dataset. A **Responsibility** is a combination of: * A **User** or **User Group**. * A **Dataset Role**, which is a predefined collection of permissions (such as the ability to edit contracts, view checks, or manage settings). By assigning Responsibilities, you define **who can do what** for each dataset, supporting clear ownership, governance, and collaboration. {% if visitor.claims.plan === %} > Learn about defining custom roles: [Global and Dataset Roles](broken://pages/n914KXqJNBoZ19Ay5W0P). #### How to Add Responsibilities Managing responsibilities requires the following dataset role: "Manage dataset responsibilities" To assign a Responsibility to a user or group: {% stepper %} {% step %} Open the **dataset page**. {% endstep %} {% step %} Click the **context menu** (⋮) in the top-right corner and select **Edit Responsibilities**.
{% endstep %} {% step %} Add the desired **users** or **user groups**.
{% endstep %} {% step %} Select the appropriate **Dataset Role** for each.
{% endstep %} {% step %} Click **Save** to apply the changes.
{% endstep %} {% endstepper %} {% endif %} ### Default Dataset Owner role Every dataset has a **default Dataset Owner role**, automatically assigned to the designated Dataset Owner(s). * This role provides essential permissions to manage and maintain the dataset. * The Dataset Owner role **cannot be removed**, but it can be combined with other roles for additional permissions. {% if visitor.claims.plan === %}
{% hint style="info" %} The **default permissions** granted to the Dataset Owner role are customizable at the organization level. For more details on configuring the default Dataset Owner role and other roles, see the [Broken mention](broken://pages/n914KXqJNBoZ19Ay5W0P#define-default-responsibilities) {% endhint %} {% endif %} ## Data source responsibilities Responsibilities can also be assigned at the **data source** level, mapping a user or user group to a **data source role**. A user must have a data source responsibility to see the datasets and checks onboarded from that data source. {% if visitor.claims.plan === %} {% hint style="warning" %} Only users with the **Manage responsibilities** permission on the data source can manage data source responsibilities. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles) {% endhint %} To assign a responsibility on a data source: {% stepper %} {% step %} Open the **data source page**. {% endstep %} {% step %} Click the **context menu** (⋮) in the top-right corner and select **Edit Responsibilities**. {% endstep %} {% step %} Add the desired **users** or **user groups** and select a **data source role** for each. {% endstep %} {% step %} Click **Save** to apply the changes. {% endstep %} {% endstepper %} {% endif %} ## Dataset Attributes {% hint style="info" %} Only users with the **Configure Dataset** permission on the dataset can update the attributes. [Global and Dataset Roles](https://docs.soda.io/soda-v4/organization-and-admin-settings/global-and-dataset-roles) {% endhint %} #### Purpose of Attributes Dataset attributes allow you to add descriptive **metadata** to your datasets. This metadata can then be: * **Used for filtering** in Soda Cloud, making it easier to search and organize datasets and checks based on specific criteria (e.g., business domain, sensitivity, criticality). * **Leveraged in reporting**, enabling you to group datasets, track ownership, and monitor data quality across different categories or dimensions. Adding meaningful attributes enhances discoverability, governance, and collaboration within Soda and its integrations. Learn how to define attribute types: [Broken mention](broken://pages/20sCqrhqLvNa1cFSthLv) {% if visitor.claims.plan === %} #### Adding Dataset Attributes You can add or modify dataset attributes in the Dataset Settings page: {% stepper %} {% step %} Click the **context menu** (⋮) in the top-right corner and select **Edit Dataset**.
{% endstep %} {% step %} Set a value for the existing attribute type. They are all optional.
{% endstep %} {% step %} **Save** your changes.
{% endstep %} {% endstepper %} {% endif %} {% if visitor.claims.plan === %} ## Bulk edit of Attributes and Responsibilities When managing multiple datasets, you can save time by applying changes in bulk using the **Bulk Edit** feature. #### How to bulk edit Datasets {% stepper %} {% step %} Go to the **Datasets** page. {% endstep %} {% step %} Select the datasets you want to edit using the checkboxes. Click **Edit** in the action menu.
{% endstep %} {% step %} Define attributes you want to add or modify across the selected datasets.
{% endstep %} {% step %} Define responsibilities you want to add or modify across the selected datasets. Choose whether to **update existing** responsibilities (add new without removing existing) or **reset** (replace all existing responsibilities with the new definition).
{% endstep %} {% step %} Click **Continue** to review your changes.
{% endstep %} {% endstepper %} {% endif %} ## Integrate with data catalog You can automate the management of **dataset attributes** and **responsibilities** in Soda Cloud using our **REST API**. This allows you to: * Programmatically **set or update attributes** for multiple datasets. * Assign **responsibilities** (users, groups, and roles) to datasets at scale. * Keep your Soda Cloud configuration in sync with your data catalog or external metadata management systems. This automation ensures that your metadata stays up-to-date and consistent across your ecosystem, supporting seamless governance and discoverability. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Deployment options Soda offers flexible deployment options to suit your team’s infrastructure, scale, and security needs. Whether you want to embed Soda directly into your pipelines, use a centrally managed deployment, or rely on Soda’s fully-hosted solution, there’s an option for you. This guide provides an overview of the three main deployment options: **Soda Python Libraries**, **Soda-hosted Soda Runner, and Self-hosted Soda Runner**, to help you choose the right setup for your organization. *** ## Overview of Deployment Options
Deployment ModelDescriptionIdeal ForKey FeaturesConsiderationsPlans
Soda-hosted Soda RunnerFully-managed Soda Runner, hosted by Soda.Teams seeking a simple, managed solution for data quality.
  • Centralized data source access
  • No setup required
  • Observability features enabled

Enables users to create, test, execute, and schedule contracts and checks directly from the Soda Cloud UI.

Required for observability features.

Cannot scan in-memory sources like Spark or DataFrames.		Available for Free, Team and Enterprise Plans.
Self-hosted Soda RunnerSame as Soda-hosted Soda Runner, but deployed and managed in your own Kubernetes environment.Teams needing full control over infrastructure and deployment.

Similar to Soda-hosted Runner, but deployed within the customer’s environment; data stays within your network.

  • Full control over deployment
  • Integration with secret managers
  • Customization to meet your organization’s specific requirements
Required for observability features.

Cannot scan in-memory sources like Spark or DataFrames.
Kubernetes expertise required.		Available for Enterprise Plan. Contact us at support@soda.io
Soda Python LibrariesOpen-source Python library (with commercial extensions) for programmatic configuration and enforcement of data contracts in your pipelines.Data engineers integrating Soda into custom workflows.
  • Full control over orchestration
  • In-memory data support
  • Contract verification
No observability features.

Required for in-memory sources (e.g., Spark, DataFrames).

Data source connections managed at the environment level.		Open Source. Available for Free, Team and Enterprise Plans.
## Deployment options in detail ### Soda Runner #### Soda-hosted The Soda-hosted Soda Runner is a fully-managed deployment of the Soda Runner, hosted by Soda in our infrastructure. It allows you to connect to your data sources and manage data quality directly from the Soda Cloud UI without any infrastructure setup on your end. You need only whitelist the IP address of the Soda-hosted runner so that it can connect to your data. Key points: * **No setup or management required**. Soda handles deployment and scaling. * **Data source connections are centralized** in Soda Cloud, and users can leverage the Soda Runner to execute scans across those data sources. * Enable **observability features** in Soda Cloud, such as profiling, metric monitoring, and anomaly detection. * Enables users to create, test, execute, and schedule **contracts** and checks directly from the **Soda Cloud UI**.
*** > Onboard your datasets in Soda Cloud with Soda-hosted runner: [Broken mention](broken://pages/cHo9ixopiR0flOMCbyUF) *** #### Self-hosted Runner The Self-hosted Runner offers the same capabilities as the Soda-hosted Runner, but it is deployed and managed by your team within your own Kubernetes environment (e.g., AWS, GCP, Azure). This model provides full control over deployment, infrastructure, and security, while enabling the same centralized data source access and Soda Cloud integration for scans, contract execution, and observability features.
> Learn how to deploy the Self-hosted Soda Runner: [Broken mention](broken://pages/DVzqeL43wc3c9AsZ4TFi). > Onboard your datasets in Soda Cloud with self-hosted runner: [Broken mention](broken://pages/cHo9ixopiR0flOMCbyUF). ### Soda Python Libraries Soda Core is an **open-source Python library and CLI** that allows you to **embed Soda directly in your data pipelines**. You can orchestrate scans using your preferred orchestration tools or pipelines, and execute them within your own infrastructure. Additional commercial extensions are available via extensions packages, such as `soda-groupby` , `soda-reconciliation`, etc. > See detailed installation instructions here: [Install Soda Python Libraries](/deployment-options/soda-python-libraries) Key points: * Ideal for teams who want **full control** over scan orchestration and execution. * Data source connections are configured and managed **at the environment level**. * Required for working with **in-memory data sources** like Spark and Pandas DataFrames.
*** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Soda Runner This page describes a Soda Runner and its characteristics. {% hint style="danger" %} **Nomenclature change: "Agent" is being updated to "Runner"** The Soda Agent is being renamed to [**Soda Runner**](/reference/soda-runner-basic-concepts) across all Soda products and interfaces. **This change will soon affect deployment terminology**, and includes: * Soda Cloud UI labels * API permission names: `MANAGE_DATASOURCES_AND_AGENTS` → `MANAGE_DATASOURCES_AND_RUNNERS` * CLI flags: `--use-agent` → `--use-runner` * Python API methods: `verify_contract_on_agent` → `verify_contract_on_runner` Deployment options still reference "agent", but will shortly be changed to "runner" in the upcoming weeks. {% endhint %} The **Soda Runner** is a tool that empowers Soda Cloud users to **securely access data sources** to scan for data quality. For a self-hosted runner, **create a Kubernetes cluster** in a cloud services provider environment, then use Helm to deploy a Soda Runner in the cluster. This setup enables Soda Cloud users to securely connect to data sources (Snowflake, Amazon Athena, etc.) **from within the Soda Cloud web application**. Any user in your Soda Cloud account can add a new data source via the runner, then write their own no-code checks to check for data quality in the new data source. When you deploy an runner, you also deploy **two types of workloads** in your Kubernetes cluster from a Docker image: * a **Soda Runner Orchestrator** which creates Kubernetes Jobs to trigger scheduled and on-demand scans of data * a **Soda Runner Scan Launcher** which wraps around the Soda Python Libraries, which implement the scans. * a **Soda Runner Contract Launcher** which wraps around the Soda Core Python libraries, providing data contract functionality.
### How does Soda integrate with Kubernetes? **Kubernetes** is a system for orchestrating containerized applications; a **Kubernetes cluster** is a set of resources that supports an application deployment. You need a Kubernetes cluster in which to deploy the containerized applications that make up the **Soda Runner**. Kubernetes uses the concept of [**Secrets**](https://kubernetes.io/docs/concepts/configuration/secret/) that the Soda Runner Helm chart employs to store connection secrets that you specify as values during the Helm release of the Soda Runner. Depending on your cloud provider, you can arrange to store these Secrets in specialized storage such as [Azure Key Vault](https://learn.microsoft.com/en-us/azure/key-vault/general/basic-concepts), [AWS Key Management Service (KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html), or [AWS Cloud Secrets Management (CSM)](https://aws.amazon.com/secrets-manager/). > Learn more about [using external secrets](broken://pages/0Cs2lWTZr1mIVYrDs9Y0#using-existing-external-secrets). The Jobs that the runner creates access these Secrets when they execute. > Learn more about [Kubernetes concepts](https://www.youtube.com/watch?v=BOj1sgWVXko). ### Where can a Soda Runner be deployed? Within a cloud services provider environment is *where* you create your Kubernetes cluster. You can deploy a Soda Runner in **any environment in which you can create Kubernetes clusters**, such as: * Amazon Elastic Kubernetes Service (EKS) * Microsoft Azure Kubernetes Service (AKS) * Google Kubernetes Engine (GKE) * Any Kubernetes cluster version 1.21 or greater which uses standard Kubernetes * Locally, for testing purposes, using tools like [Minikube](https://minikube.sigs.k8s.io/docs/), [microk8s](https://microk8s.io/docs), [kind](https://kind.sigs.k8s.io/), [k3s](https://docs.k3s.io/), or [Docker Desktop](https://www.docker.com/products/docker-desktop/) with Kubernetes support. ### What is Helm? **Helm** is a **package manager for Kubernetes** which bundles YAML files together for storage in a public or private repository. This bundle of YAML files is referred to as a **Helm chart**. The Soda Runner is a Helm chart. Anyone with access to the Helm chart’s repo can deploy the chart to make use of YAML files in it. > Learn more about [Helm concepts](https://www.youtube.com/watch?v=-ykwb1d0DXU). The Soda Runner Helm chart is stored on a [public repository](https://helm.soda.io/soda-agent/) and published on [ArtifactHub.io](https://artifacthub.io/packages/helm/soda-agent/soda-agent). Anyone can use Helm to find and deploy the Soda Runner Helm chart in their Kubernetes cluster. ### Why Kubernetes? Kubernetes is the most **powerful** and **future-proof** platform for running the Soda Runner because it delivers the best of both worlds: the **flexibility of raw compute** without the operational burden, and the **scalability of managed services** without their restrictions. * Kubernetes goes far beyond raw compute like EC2 or traditional Virtual Machines (VMs) by **abstracting away the heavy lifting of networking, deployments, and scaling**, while still giving teams precise control when needed. Practically, this makes it easy for Soda’s customers to [deploy, manage, and upgrade Soda Runners](broken://pages/DVzqeL43wc3c9AsZ4TFi) using [Kubernetes](#how-does-soda-integrate-with-kubernetes) and [Helm](#what-is-helm), always staying up to date with the latest releases. * Unlike fully managed options such as AWS Lambda, **Kubernetes has no execution time limits** and is built to handle long-running, stateful, and highly scalable workloads. This means **Soda is not limited to lightweight samples** but can perform complete, row-level operations—powering advanced capabilities like the Diagnostics Warehouse, which securely stores the exact failing records inside your own infrastructure, and [Reconciliation Checks](broken://pages/QkO7w20yeaFP3U9Ow5Kk), which compare data at row-level across sources. Whether running in the cloud or on-premises, Kubernetes ensures resilience, portability, and cost-efficient resource use, making it the clear choice for complex, enterprise-grade data quality workloads. # Soda Python Libraries This page describes how to install the Soda Python packages, which are required for running Soda scans via the CLI or Python API. ## Installation #### Requirements To use Soda, you must have installed the following on your system. * **Python 3.10, 3.11, or 3.12**.\ To check your existing version, use the CLI command: `python --version` or `python3 --version`. If you have not already installed Python, consider using `pyenv` to manage multiple versions of Python in your environment. {% hint style="info" %} While Python 3.12 is the highest officially supported version, there are no known constraints preventing use of Python 3.13+. {% endhint %} * **Pip 21.0 or greater.**\ To check your existing version, use the CLI command: `pip --version` * **A Soda Cloud account**; see how to [sign up](/quickstart#sign-up). Best practice dictates that you **install the Soda CLI using a virtual environment**. If you haven't yet, in your command-line interface tool, create a virtual environment in the `.venv` directory using the commands below. Soda recommends using [`uv`](https://github.com/astral-sh/uv): ```bash uv venv .venv source .venv/bin/activate ``` If you prefer the standard Python tooling (not using `uv`), depending on your version of Python, you may need to replace `python` with `python3` in the first command (or with the Python version you want your virtual environment to run, e.g. `py -3.11`): ```sh python -m venv .venv source .venv/bin/activate ``` #### Choose an installation flow Before you install the Soda CLI, decide which installation flow applies to your environment and license type. The two flows available serve different purposes: {% if visitor.claims.plan === %}
Use CaseInstallation FlowDescription
Executing data contracts with basic data quality checks on enterprise data sources.Public PyPI

Use this installation method if you’re just getting started.

The Public PyPI index hosts Soda Core packages for all supported data sources.

Same as above, plus: group by checks, reconciliation checks, migrating checks from v3 to v4, running checks on Oracle data, and capturing failed rows with the Diagnostics Warehouse.Private PyPIPrivate PyPI repositories are region-specific and require authentication using your API key credentials. This method ensures secure access to licensed components, enterprise-only extensions, and region-compliant hosting.
{% endif %} {% hint style="info" %} Different installations will support **different packages**. Learn more about which packages are supported in [public](#supported-packages) and [private](#supported-packages2) PyPI. {% endhint %} {% hint style="success" %} **How to differentiate between free open-source Soda, and paid licensed Soda?** **Soda V3**: package names included `core` if the package was free open-source. E.g.: * `soda-core-postgres` (free open-source) * `soda-postgres` (paid licensed Soda). **Soda V4**: no differentiation using `core` in package names. Differentiation will be **based on the installation flows** listed above. {% endhint %} *** ### Public PyPI installation flow To use the open source Soda Core python packages, you must install them from the public Soda PyPi registry: `https://pypi.cloud.soda.io/simple` . 1. **Install the Soda Core package** for your data source. This gives you access to all the basic CLI functionality for working with contracts. ```bash uv pip install -i https://pypi.cloud.soda.io/simple --pre -U "soda-postgres>4" ``` Replace `soda-postgres` with the appropriate package for your data source. See the [Data source reference for Soda Core](/reference/data-source-reference-for-soda-core) for supported packages and configurations. :octagon-check: Now you can connect to Soda Cloud. #### Supported packages * `soda-core`: "umbrella" package (does not include Diagnostics Warehouse) * Data-source-specific packages: naming pattern is `soda-` (e.g. `soda-postgres`, `soda-bigquery`, `soda-sparkdf`, etc.) *** ### Private PyPI installation flow {% if !(visitor.claims.plan === %} If you wish to use commercial extensions to the Soda Core python package, you must install them from one of the private Soda PyPi registries. The private PyPI installation process adds an authentication layer and region-based repositories for license-based access control of **Team** and **Enterprise** customers. > To get access to the private PyPi registries, Soda Cloud, Diagnostics Warehouse, and more Soda licensed features, contact us at . > {% endif %} {% if (visitor.claims.plan === %} If you wish to use commercial extensions to the Soda Core python package, you must install them from one of the private Soda PyPi registries below. The private PyPI installation process adds an authentication layer and region-based repositories for license-based access control of **Team** and **Enterprise** customers. 1. \[If not using `uv`] **Upgrade `pip`** inside your new virtual environment. ```sh pip install --upgrade pip ``` 2. **Choose the correct repository** based on your license and region.
LicenseSoda RegionRepository URL
Team1EUteam.pypi.cloud.soda.io/
TeamUSteam.pypi.us.soda.io
Enterprise2EUenterprise.pypi.cloud.soda.io
EnterpriseUSenterprise.pypi.us.soda.io/
1 Team: Any license except "Trial" or "Enterprise" (see below)\ 2 Enterprise: one of `enterprise` , `enterprise_user_based` , `dataset_standard` , `premier` licenses. 3. **Set your credentials**. ```bash export SODA_API_KEY_ID="your_key_id" export SODA_API_KEY_SECRET="your_key_secret" ``` > See how to generate your own [API key values](broken://pages/H6Piqnv5lUFCAv1Cvm75). 4. Based on your license and region, **choose and execute one of the following commands**, **replacing** `soda>=4` with the package that you need to install. {% tabs %} {% tab title="Team EU" %} {% code overflow="wrap" %} ```bash uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@team.pypi.cloud.soda.io" --extra-index-url=https://pypi.cloud.soda.io ``` {% endcode %} {% endtab %} {% tab title="Team US" %} {% code overflow="wrap" %} ```bash uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@team.pypi.us.soda.io" --extra-index-url=https://pypi.cloud.soda.io ``` {% endcode %} {% endtab %} {% tab title="Enterprise EU" %} {% code overflow="wrap" %} ```bash uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.cloud.soda.io" --extra-index-url=https://pypi.cloud.soda.io ``` {% endcode %} {% endtab %} {% tab title="Enterprise US" %} {% code overflow="wrap" %} ```bash uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.us.soda.io" --extra-index-url=https://pypi.cloud.soda.io ``` {% endcode %} {% endtab %} {% endtabs %} * `soda`: required for the contract generator (includes Diagnostics Warehouse) 
uv pip install "soda>=4" --pre -i "https://${SODA_API_KEY_ID}:${SODA_API_KEY_SECRET}@enterprise.pypi.soda.io" --extra-index-url=https://pypi.cloud.soda.io
* `soda-groupby` * `soda-migration` * `soda-reconciliation` * `soda-oracle` {% endif %} ## Available extensions The extensions you need to install depend on the **features** you want to use and your Soda subscription plan. The table below lists the supported features, the **required extension** for each one, and which plans include access. {% if !visitor.claims.plan === %} | Feature | Required extension | Available in plan | | ----------------------------------------------------------------- | --------------------- | ------------------- | | [Create contract](/reference/cli-reference#create-a-contract) | `soda` | Team and Enterprise | | [Group by](/reference/contract-language-reference#group-by-check) | `soda-groupby` | Team and Enterprise | | [Diagnostics Warehouse](/diagnostics-warehouse) | `soda` | Team and Enterprise | | [Reconciliation](broken://pages/QkO7w20yeaFP3U9Ow5Kk) | `soda-reconciliation` | Enterprise only | | Oracle connector | `soda-oracle` | Enterprise only | | Dremio connector | `soda-dremio` | Enterprise only | | {% endif %} | | | {% if visitor.claims.plan === %} | Feature | Required extension | Available in plan | | ----------------------------------------------------------------- | --------------------- | ------------------- | | [Create contract](/reference/cli-reference#create-a-contract) | `soda` | Team and Enterprise | | [Group by](/reference/contract-language-reference#group-by-check) | `soda-groupby` | Team and Enterprise | | [Diagnostics Warehouse](broken://pages/UZCAHd8ZBgl01336NkZa) | `soda` | Team and Enterprise | | [Reconciliation](broken://pages/QkO7w20yeaFP3U9Ow5Kk) | `soda-reconciliation` | Enterprise only | | Oracle connector | `soda-oracle` | Enterprise only | | Dremio connector | `soda-dremio` | Enterprise only | | {% endif %} | | | *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Organization and Admin settings Configure Soda Cloud to match how your organization governs access, identity, and automation, with enterprise-grade controls and network security. The **Organization and Admin Settings** in Soda Cloud provide a centralized interface for managing your organization’s configuration, roles, user access, and integrations. From setting your organization’s name to defining global roles, managing user groups, and enabling the Soda-hosted Runner, these settings help you tailor Soda Cloud to your team’s needs and governance policies. {% if visitor.claims.plan === %} To access the settings, click on your avatar on the top right, and then click Organization Settings.
{% hint style="warning" %} Only users with the **Manage Organization Settings** global role can access and modify organization settings. {% endhint %} ### Add multiple organizations You may find it useful to set up multiple organizations in Soda Cloud so that each corresponds with a different environment in your network infrastructure, such as production, staging, and development. Such a setup makes it easy for you and your team to access multiple, independent Soda Cloud organizations using the same profile, or login credentials. Note that Soda Cloud associates any [API keys](broken://pages/H6Piqnv5lUFCAv1Cvm75) that you generate within an organization with both your profile *and* the organization in which you generated the keys. **API keys are not interchangeable between organizations.** > Contact to request multiple organizations for Soda Cloud. > {% endif %} {% if !(visitor.claims.plan === %} *** ## Built for enterprise governance Adopting a data quality platform across an organization raises the same questions as adopting any other system of record: Who can log in? How is their identity managed? What can each role actually do? How do we let CI/CD pipelines act without handing out a person's credentials? Soda Cloud is designed to answer all of those out of the box. {% columns %} {% column %} #### Identity, your way Federate authentication through your existing SAML 2.0 identity provider (including Azure AD, Okta, and Google Workspace) and let your IdP remain the source of truth for who works at your company and which teams they belong to. {% endcolumn %} {% column %} #### Permissions that match your org chart Define what each role can do across the organization and per dataset, with a fully customizable role model. Grant exactly the access each team needs without forcing your structure into a rigid template. {% endcolumn %} {% endcolumns %} {% columns %} {% column %} #### Network-level access controls Restrict access to Soda Cloud to your corporate network, VPN, or any defined set of IP ranges. Requests from anywhere else are blocked before they ever reach authentication. {% endcolumn %} {% column %} #### Automation-ready by design Run scans, verify contracts, and call the API from CI/CD pipelines using service accounts: machine identities that are separate from any individual user and revocable in one click. {% endcolumn %} {% endcolumns %} ### Customizable platform Soda Cloud allows organizations to have full control over each user and group permissions: * **Organization-level configuration:** set your organization's identity in the platform, control which Soda capabilities are enabled (including AI-powered assistants and dataset profiling), and manage the storage of data source secrets used by your scans. * **User management:** invite, deactivate, and organize users into groups, with group-based permission assignment so access scales as the team grows. * **Single Sign-On (SSO):** connect Soda Cloud to your identity provider, with optional one-way sync of user groups directly from the IdP. * **Global and dataset roles (RBAC):** design the role model that fits your organization, with separate scopes for what users can do across the platform versus what they can do on a specific dataset. * **Service accounts:** issue API-only identities for pipelines and integrations, with their own permissions and lifecycle independent of any human user. * **Network controls:** restrict platform access by IP address or CIDR range, enforced before authentication. * **Multiple environments:** operate separate organizations for production, staging, and development from the same user profile, so each environment has its own data, roles, and credentials. ### Security and access control at a glance Soda Cloud treats identity as the security perimeter. Every request is authenticated, scoped to the user's organization, and evaluated through role-based access control before any data is returned. The result is a platform that supports both **least-privilege access** for sensitive datasets and **broad data democratization** for the rest — without forcing one approach on the entire organization. > Contact to learn more about Soda Cloud and what it can do for your organization. > {% endif %} *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Integrations Soda offers seamless integrations with many tools across your data stack. Whether you're aligning data governance efforts, collaborating across teams, or triggering workflows, you can enhance Soda’s observability capabilities with the following connections: ## Supported integrations ### Messaging and collaboration {% if visitor.claims.plan === %} * [Broken mention](broken://pages/FGtUrTJIXyPOscNE4OOZ) * [Broken mention](broken://pages/WaXQIin4SfDtevIpvjEH) * [Broken mention](broken://pages/Nc5tlgvZtKCPAtVEOYEs) * [Broken mention](broken://pages/iVQcFCGjN0403SqITjee) * [Broken mention](broken://pages/aXoNhHFByZzkyWiAiouz) > For more details on notification rules, see the [Notification rules documentation](broken://pages/GAK0Cdoq2hZmucVmrovV). > {% endif %} {% if !(visitor.claims.plan === %} * MS Teams * Slack * Jira * ServiceNow * Webhook {% endif %} ### Catalogs and governance tools {% if visitor.claims.plan === %} * [Broken mention](broken://pages/EWewnaGMWlq3NVKKsqxa) * [Broken mention](broken://pages/LMdfap2DOppG3Cjz1UXY) * [Broken mention](broken://pages/0KvKvx3xHbjxohh3AN5o) * [Broken mention](broken://pages/CjPXGBtlUjlS7PaZy07X) * [Broken mention](broken://pages/EfXes0ukV6Zcyhz41XXh) * [Broken mention](broken://pages/UFfB50RiaiykEQajllf2) {% endif %} {% if !(visitor.claims.plan === %} * Alation * Atlan * Collibra * data.world * Metaphor * Purview {% endif %} ### BI tools {% if visitor.claims.plan === %} * [Broken mention](broken://pages/MuQKFasv9rqrJV9lZI8M) {% endif %} {% if !(visitor.claims.plan === %} * Sigma {% endif %} ### Data transformation and code repositories {% if visitor.claims.plan === %} * [Broken mention](broken://pages/bXqO5zEK57sLVoMrTP50) {% endif %} {% if !(visitor.claims.plan === %} * GitHub {% endif %} {% if visitor.claims.plan === %} *** ## Create an integration To create an integration: 1. Go to the **Integrations** section in **Settings**. 2. Click the **+** button to add a new integration. 3. Select the integration type (**Slack**, **Microsoft Teams**, or **Webhook**). 4. Follow the setup steps for the chosen integration 5. Click **Save** to activate the integration. *** ## Edit an integration You can update existing integrations if connection details or configurations change. To edit an integration: 1. Go to the **Integrations** section in **Settings**. 2. Find the integration you want to update. 3. Click the context menu and select **Edit Integration Settings**. 4. Update the configuration as needed. 5. Click **Save** to apply the changes. ## Pause an integration You can temporarily pause an integration if you want to stop sending notifications and incident updates without fully deleting the configuration. The integration will no longer be available in notification rules. To pause an integration: 1. Go to the **Integrations** section in **Settings**. 2. Locate the integration you want to pause. 3. Change the status to "Paused" in the table 4. Select **Pause**. While paused, the integration will no longer send any notifications. You can resume it at any time by following the same steps and selecting **Active**. {% endif %} *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Reference This reference hub includes detailed documentation for Soda’s key interfaces and configuration options, as well as information on Soda's architecture and specifics, including: * Data Contract Language Reference – Author, validate, and manage data contracts using YAML-based definitions. * CLI Command and Python Reference – Use Soda’s command-line interface to configure, run, and automate verification workflows. * REST API Reference – Interact with Soda Cloud programmatically to manage datasets, run verifications, and retrieve results. * Data flow and data source reference – Understand how Soda interacts with other systems and manage exceptions. Each section includes practical, example-based documentation structured to help data engineers, analysts, and platform teams apply Soda in real-world use cases. {% hint style="danger" %} **Nomenclature change: "Agent" is now "Runner"** The Soda Agent has been renamed to [**Soda Runner**](/reference/soda-runner-basic-concepts) across all Soda products and interfaces. This change affects terminology throughout the platform, including: * Soda Cloud UI labels * API permission names: `MANAGE_DATASOURCES_AND_AGENTS` → `MANAGE_DATASOURCES_AND_RUNNERS` * CLI flags: `--use-agent` → `--use-runner` * Python API methods: `verify_contract_on_agent` → `verify_contract_on_runner` The underlying functionality is unchanged, and **both runner/agent nomenclatures are backward compatible** at present. If you are following older documentation or have existing scripts and automation that reference "agent", we recommend to gradually replace every occurrence of "agent" / "Agent" with "runner" / "Runner". {% endhint %} *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Data source reference for Soda Core This page lists the supported data source types and their required connection parameters for use with Soda Core. Soda uses the official Python drivers for each supported data source. The configuration examples below include the default required fields, but you can extend them with any additional parameters supported by the underlying driver. Each data source configuration must be written in a YAML file and passed as an argument using the CLI or Python API. ## General Guidelines * Each configuration must include `type`, `name`, and a `connection` block. * Use the exact structure required by the underlying Python driver. * Test the connection before using the configuration in a contract. ```bash soda data-source test -ds ds_config.yml ``` *** #### Connect to a data source already onboarded in Soda Cloud If you have already [onboarded a data source](broken://pages/cHo9ixopiR0flOMCbyUF) in **Soda Cloud**, make sure to use the **exact same data source name** in your Soda Core data source configuration. This ensures that datasets are correctly identified and mapped to the existing data source in Soda Cloud, whether you run verifications locally with Soda Core or remotely via a Soda Runner. *** #### Onboard a data source programmatically It is possible to onboard a data source to Soda Cloud (and to a Soda Runner) after onboarding it using Soda Core. > To learn how, head to [Broken mention](broken://pages/cHo9ixopiR0flOMCbyUF). #### Using Environment Variables You can reference environment variables in your data source configuration. This is useful for securely managing sensitive values (like credentials) or dynamically setting parameters based on your environment (e.g., dev, staging, prod). **Example:** 
type: postgres
name: postgres
connection:
  host: host_name
  port: 5432
  user: user_name
  password: ${env.SODA_DEMO_POSTGRES_PW}
  database: db_name
Environment variables must be available in the runtime environment where Soda is executed (e.g., your terminal, CI/CD runner, or Docker container). > For Soda to run quality scans on your data, you must configure it to connect to your data source.\ > To learn how to set up Soda from scratch and configure it to connect to your data sources, see Soda's [Quickstart](/quickstart). *** ## Supported data sources
Data sourceOnboardingMetadataMetadata historyQuerying data
/pages/ctP9k7n5AxdoMIBmXxD2✅✅—✅
/pages/xlK0ugO80HSregboFQUM✅✅✅✅
/pages/jggdwLvBUVJjJlkdmGUe✅✅✅✅
/pages/qQQSBTgs2jlSAdkxg3mF✅Upcoming—✅
/pages/dESWjWE03JmceyspJhlb (Pandas, Polars, etc.)Partial/Python only——✅
/pages/eCdm3TE8crsXRy3LKZpO✅✅—✅
MySQLUpcoming——✅
/pages/yymUzQ59nkr9L3YddiK0✅✅—✅
/pages/xqU5pcfJVCubI3BL8MAH✅✅—✅
/pages/AUAQ8kj87Q98kRNninUy✅✅✅✅
/pages/KxwupsjjHWfxegxEzjwSPartial/Python only——✅
/pages/jmuttQi4HNCqZoA6ehA6✅✅—✅
/pages/hnAwM7zDdGkq4RkDqhBt✅✅—✅
/pages/GzT8YXwoPxHYCfTPHap4✅✅—✅
/pages/2aL0YaZUaMgSp9hJMghh✅——✅
*** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Athena Access configuration details to connect Soda to an Amazon Athena data source. ### Connection configuration reference Install the following package: ```bash pip install soda-athena>4 ``` #### Data source YAML **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` The data source configuration YAML should look like the following: {% code title="ds\_config.yml" %} ```yaml type: athena name: my_athena connection: catalog: ${env.ATHENA_CATALOG} access_key_id: ${env.ATHENA_ACCESS_KEY_ID} secret_access_key: ${env.ATHENA_SECRET_ACCESS_KEY} staging_dir: ${env.ATHENA_STAGING_DIR} ## e.g., s3://my-company-athena-staging/query-results/ region_name: ${env.ATHENA_REGION} work_group: ${env.ATHENA_WOKRGROUP} # role_arn: # profile_name: # session_token: ``` {% endcode %} #### Connection test Test the data source connection: ```bash soda data-source test -ds ds_config.yml ``` *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # BigQuery Access configuration details to connect Soda to a Google Cloud BigQuery data source. ### Connection configuration reference Install the following package: ```bash pip install soda-bigquery ``` #### Data source YAML Provide **exactly one** of the following: * `account_info_json` – inline BigQuery service account JSON * `account_info_json_path` – path to a BigQuery service account JSON file {% hint style="info" %} If `use_context_auth: true`, Soda uses **Application Default Credentials (ADC)** and neither option is required. {% endhint %} **Provide `account_info_json`:** 
type: bigquery
name: my_bigquery
connection:
    account_info_json: '{
    "type": "service_account",
    "project_id": "dbt-quickstart-44203",
    "private_key_id": "fe0a60e9cb7d4369f73f7b5691ce397d1e",
    "private_key": "-----BEGIN PRIVATE KEY-----<insert-private-key>-----END PRIVATE KEY-----\n",
    "client_email": "dbt-user@dbt-quickstart-448203.iam.gserviceaccount.com",
    "client_id": "114963712293161062",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/dbt-user%40dbt-quickstart-44803.iam.gserviceaccount.com",
    "universe_domain": "googleapis.com"
  }' # example service account JSON string, exported from BQ
  dataset: ${env.BQ_DATASET_NAME}
  # optional
  project_id: ${env.BQ_PROJECT_ID}        # Defaults to the one embedded in the account JSON
  storage_project_id: ${env.BQ_STORAGE_PROJECT_ID}
  location: ${env.BQ_LOCATION}            # Defaults to the specified project's location
  auth_scopes:
    - https://www.googleapis.com/auth/bigquery
    - https://www.googleapis.com/auth/cloud-platform
    - https://www.googleapis.com/auth/drive
  client_options: <options-dict-for-bq-client>
  labels: <labels-dict-for-bq-client>
  impersonation_account: <name-of-impersonation-account>
  delegates: <list-of-delegates-names>
  ##use_context_auth: false     # whether to use Application Default Credentials
**Provide `account_info_json_path`:** **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` The data source configuration YAML should look like the following: 
type: bigquery
name: my_bigquery
connection:
  account_info_json_path: /path/to/service-account.json
  dataset: ${env.BQ_DATASET_NAME}
    # optional
  project_id: ${env.BQ_PROJECT_ID}        # Defaults to the one embedded in the account JSON
  storage_project_id: ${env.BQ_STORAGE_PROJECT_ID}
  location: ${env.BQ_LOCATION}            # Defaults to the specified project's location
  auth_scopes:
    - https://www.googleapis.com/auth/bigquery
    - https://www.googleapis.com/auth/cloud-platform
    - https://www.googleapis.com/auth/drive
  client_options: <options-dict-for-bq-client>
  labels: <labels-dict-for-bq-client>
  impersonation_account: <name-of-impersonation-account>
  delegates: <list-of-delegates-names>
  ##use_context_auth: false                # whether to use Application Default Credentials
{% hint style="info" %} **Note:** Google uses the term "dataset" differently than Soda: * In the context of Soda, a [dataset](/soda-v3/learning-resources/glossary#dataset) is a representation of a tabular data structure with rows and columns, such as a table, view, or data frame. * In the context of BigQuery, a [dataset](https://cloud.google.com/bigquery/docs/datasets-intro) is “a top-level container that is used to organize and control access to your tables and views. A table or view must belong to a dataset…” Instances of "dataset" in Soda documentation always reference the former. {% endhint %} > * See [Google BigQuery Integration parameters](https://cloud.google.com/chronicle/docs/soar/marketplace-integrations/google-big-query#integration_parameters) > * See[ BigQuery's locations documentation](https://cloud.google.com/bigquery/docs/locations) to learn more about `location`. #### Connection test Test the data source connection: ```bash soda data-source test -ds ds_config.yml ``` *** ## Limitations ### **`DATE` vs `TIMESTAMP` partition columns** * Soda **expects a `TIMESTAMP`-based partition column** for full compatibility with partition-based monitoring and freshness calculations in BigQuery. * If your dataset uses a `DATE` column as the partition key, you must **explicitly cast** the column to `TIMESTAMP` to ensure valid SQL execution. #### Recommended approaches: * **Preferred**: Use a `TIMESTAMP` column as the time partition column when onboarding BigQuery datasets. * **Alternative**: If a `DATE` column must be used, apply an explicit cast to `TIMESTAMP` in filters or expressions. ### Sharded tables Soda Cloud treats each date-sharded table in BigQuery as an independent dataset. In practice, tables named with date suffixes (for example, `events_20240617`, `events_20240618`) will: * **Count separately** in the UI, appearing as distinct entries in the Datasets list. * **Count separately** toward Soda pricing, since each shard is billed as its own dataset. * **Maintain independent schemas**, so any structural differences between shards must be managed manually (for example, by aligning column names and types before onboarding). There is no built-in metadata or grouping mechanism in Soda Cloud to unify shards under a “main” logical table. #### Recommended migration to native partitions BigQuery now recommends using **time-partitioned tables** instead of legacy sharding. Partitioned tables offer better performance, lower cost, and a single logical table for both queries and metadata. To migrate: 1. Follow BigQuery’s guide to [convert date-sharded tables into a partitioned table](https://cloud.google.com/bigquery/docs/creating-partitioned-tables#convert-date-sharded-tables). 2. In Soda Cloud, onboard the new partitioned table once. Soda will treat it as a single dataset. #### Short-term workarounds If migration to partitioned tables is not yet feasible for you: * **Table-name patterns:** Onboard your shards as separate datasets, but use consistent naming conventions and Dashboard filters to group them visually. * **External catalog consolidation:** If you use a metadata layer, such as Atlan, to virtualize shards into a single logical asset, you can point Soda at that consolidated view. Keep in mind Soda will still count it as one dataset only if it surfaces as a single table name in BigQuery. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Databricks SQL Access configuration details to connect Soda to a Databricks SQL data source. > For information about how to connect to Databricks Spark Dataframes, see: [Spark Dataframe](/reference/data-source-reference-for-soda-core/spark-dataframe#example-contract-spark-databricks) ### Connection configuration reference Install the following package: ```bash pip install soda-databricks ``` #### Data source YAML **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` The data source configuration YAML should look like the following: {% code title="ds\_config.yml" %} ```yaml type: databricks name: my_databricks connection: host: ${env.DATABRICKS_HOST} http_path: ${env.DATABRICKS_SQL_HTTP_PATH} catalog: ${env.DATABRICKS_CATALOG} access_token: ${env.DATABRICKS_SQL_ACCESS_TOKEN} ## optional # session_configuration: ``` {% endcode %} {% hint style="info" %} **Hive Metastore is supported**. When the `catalog` is set to `hive_metastore`, Soda will use Hive-compatible queries. {% endhint %} #### Connection test Test the data source connection: ```bash soda data-source test -ds ds_config.yml ``` *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Dremio Access configuration details to connect Soda to a Dremio data source. ### Connection configuration reference {% hint style="info" %} Make sure you have installed the **latest version of** [**`soda-core`**](/release-notes-legacy/soda-core-release-notes-legacy). {% endhint %} Install the following package: {% code overflow="wrap" %} ```bash pip install "soda-dremio" -i "https://${SODA_CLOUD_API_KEY_ID}:${SODA_CLOUD_API_KEY_SECRET}@enterprise.pypi.soda.io/simple" ``` {% endcode %} * `${SODA_CLOUD_API_KEY_ID}` and `${SODA_CLOUD_API_KEY_SECRET}` are environment variables that you should store with your API Key ID and API Key Secret, respectively. {% hint style="info" %} Learn how to [generate API keys](broken://pages/H6Piqnv5lUFCAv1Cvm75). {% endhint %} * For US host users, use `@enterprise.pypi.us.soda.io`\ For EU host users, use `@enterprise.pypi.soda.io` #### Data source YAML {% code title="ds\_config.yml" %} ```yaml type: dremio name: my_dremio connection: # Required host: ${env.DREMIO_HOST} port: ${env.DREMIO_PORT} username: ${env.DREMIO_USERNAME} password: ${env.DREMIO_PASSWORD} # Optional # schema: ${env.DREMIO_SCHEMA} # schema / space that contains your datasets # use_encryption: "true" # "false" by default; enable TLS by setting "true" # disable_certificate_verification: "true" # "false" by default; set "true" to skip TLS cert checks ``` {% endcode %} #### Connection test Test the data source connection: ```bash soda data-source test -ds ds_config.yml ``` ### Limitations & edge cases * Dremio does not fully support schema evolution in the same way other data sources do it. Tables are still versioned, but Soda cannot create views that point to the latest data. * Soda does not support Dremio schemas with `.` in their name. {% if %} * Diagnostics Warehouse: When enabling **Diagnostics Warehouse advanced configuration > Allow failed row collection** during [Diagnostics Warehouse setup](broken://pages/E1KTiEL4c6zytpedbEZg): * A **schema configuration** must be provided. Dremio requires a top level ID of a configured writable data source (e.g. `s3_instance.folder1.soda_diagnostics`, `gcp_instance.soda_diagnostics`). This field defaults to `soda_diagnostics` and will fail if left with that value.

The examples above (s3_instance.folder1.soda_diagnostics, gcp_instance.soda_diagnostics) look like object storage paths because, generally, databases are not writable from Dremio, and object storage is.

{% endif %} *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # DuckDB Access configuration details to connect Soda to a DuckDB data source. Soda supports DuckDB as a flexible, lightweight SQL engine that can be used with native `.duckdb` files, in-memory data, or external dataframes such as Pandas and Polars. ### Connection configuration reference Install the following package: ```bash pip install soda-duckdb ``` #### Data source YAML **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` The data source configuration YAML should look like the following: {% code title="ds\_config.yml" %} ```yaml type: duckdb name: my_duckdb connection: database: "adventureworks.duckdb" # or a supported file path like "dim_employee.parquet" ``` {% endcode %} #### Contract YAML {% code title="contract.yml" %} ```yaml dataset: datasource/main/adventureworks columns: - name: id checks: - missing: - name: name checks: - missing: threshold: metric: percent must_be_less_than: 10 - name: size checks: - invalid: valid_values: ['S', 'M', 'L'] checks: - schema: - row_count: ``` {% endcode %} *** DuckDB also supports registering **in-memory data frames** from **Pandas** or **Polars,** and creating **temporary tables for contract testing**. You can run Soda contracts against these datasets by passing the live DuckDB cursor to `DuckDBDataSource.from_existing_cursor` as described in the following page: > Learn more: [DuckDB advanced usage](/reference/data-source-reference-for-soda-core/duckdb/duckdb-advanced-usage) ### Connecting to MotherDuck You can also connect Soda to **MotherDuck** using the same DuckDB package. MotherDuck is a managed cloud service for DuckDB that provides persistent storage and database sharing while preserving DuckDB’s execution model. To connect, use the `md:` connection string and provide a MotherDuck service token via an environment variable. ```yaml name: my_duckdb connection: database: "md:my_db?motherduck_token=${env.MDTOKEN}" ``` Soda uses DuckDB’s native MotherDuck integration, so no additional drivers or configuration are required. The specified database is created automatically if it does not already exist. Ensure the `MDTOKEN` environment variable is set before running Soda. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # DuckDB advanced usage Soda supports DuckDB as a flexible, lightweight SQL engine that can be used with files and in-memory data frames such as Pandas and Polars. Install the following package: ```bash pip install -i https://pypi.cloud.soda.io/simple --pre -U "soda-duckdb>4" ``` ### From Pandas DataFrame ```python import pandas as pd import duckdb from soda_core.contracts import verify_contract_locally from soda_duckdb import DuckDBDataSource df = pd.read_parquet("adventureworks.parquet") conn = duckdb.connect(database=":memory:") cursor = conn.cursor() cursor.register(view_name="adventureworks", python_object=df) result = verify_contract_locally( data_sources=[DuckDBDataSource.from_existing_cursor(cursor, name="duckdb")], contract_file_path="adventureworks.yml" ) ``` *** ### From Polars DataFrame ```python import polars as pl import duckdb from soda_core.contracts import verify_contract_locally from soda_duckdb import DuckDBDataSource df = pl.read_parquet("adventureworks.parquet") conn = duckdb.connect(database=":memory:") cursor = conn.cursor() cursor.register(view_name="adventureworks", python_object=df) result = verify_contract_locally( data_sources=[DuckDBDataSource.from_existing_cursor(cursor, name="duckdb")], contract_file_path="adventureworks.yml" ) ``` *** ### In-Memory with DuckDB SQL 
import duckdb
from soda_core.contracts import verify_contract_locally
from soda_duckdb import DuckDBDataSource

db_connection = duckdb.connect(database=":memory:")
cursor = db_connection.cursor()

cursor.execute("CREATE SCHEMA analytics")
cursor.execute("CREATE TABLE analytics.adventureworks AS SELECT * FROM read_parquet('adventureworks.parquet')")

result = verify_contract_locally(
    data_sources=[DuckDBDataSource.from_existing_cursor(cursor, name="duckdb")],
    contract_file_path="adventureworks.yml"
)
*** ### Data from Parquet File You can point directly to a `.parquet` file as a DuckDB "database": {% code title="ds\_config.yml" %} ```yaml type: duckdb name: duckdb connection: database: "adventureworks.parquet" ``` {% endcode %} Then you can verify a contract on this database using the CLI: ```bash soda contract verify -ds ds_config.yml -c adventureworks.yml ``` Or Python API: ```python from soda_core.contracts import verify_contract_locally result = verify_contract_locally( data_source_file_path="ds_config.yml", contract_file_path="adventureworks.yml" ) ``` *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Fabric Access configuration details to connect Soda to a Microsoft Fabric data source. ### Connection configuration reference Install the following package: ```bash pip install soda-fabric ``` #### Data source YAML **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` The data source configuration YAML should look like the following: {% code title="ds\_config.yml" %} ```yaml type: fabric name: my_fabric connection: host: ${env.FABRIC_HOST} port: 1433 database: ${env.FABRIC_DB} user: ${env.FABRIC_USER} # SEE NOTE password: ${env.FABRIC_PW} # SEE NOTE authentication: sql # activedirectoryserviceprincipal | activedirectoryinteractive | activedirectorypassword # optional client_id: ${env.FABRIC_SERVICE_CLIENT_ID} # SEE NOTE client_secret: ${env.FABRIC_SERVICE_CLIENT_SECRET} # SEE NOTE driver: ODBC Driver 18 for SQL Server trusted_connection: false encrypt: false trust_server_certificate: false ``` {% endcode %} {% hint style="info" %} **Note:** depending on the authentication method that is used, `user` and `password` may not be required (e.g. `activedirectoryserviceprincipal` requires `client_id` and `client_secret`). {% endhint %} #### Connection test Test the data source connection: ```bash soda data-source test -ds ds_config.yml ``` ### Regex patterns in Fabric Fabric databases **do not offer universal regex support**. When using regex-based checks in these environments, Soda adapts pattern-matching queries. Instead of regex functions, **Soda translates patterns into SQL queries using the `PATINDEX` function**. #### How Soda translates regex patterns If the user specifies a regex `pattern` which matches a `column expression`, it will be expanded in SQL as: ```sql PATINDEX('%{pattern}%', {column expression} COLLATE SQL_Latin1_General_Cp1_CS_AS) > 0 ``` **Key considerations** * **Collation** A collation (a rule set used for determining string matches) must be specified. Soda uses the `SQL_Latin1_General_Cp1_CS_AS`, which is **case-sensitive** and **accent-sensitive**, so `a` and `A` are not equal, and accented characters (such as `é`) are treated distinctly. * **Pattern handling** Regex patterns are wrapped in `%…%`, so any substring match within the column value will return `true`. Example: `^abc` becomes `%abc%`, which will match anywhere in the string. * **Character ranges**\ In SQL Server and related databases, like Fabric, character range expansion for alpha characters is handled differently than in most regex engines. * Soda **auto-expands** the most common ranges for you: * `[a-z]` → `[abcdefghijklmnopqrstuvwxyz]` * `[A-Z]` → `[ABCDEFGHIJKLMNOPQRSTUVWXYZ]` * **Other ranges are not expanded automatically** and you may need to manually expand the full set when defining your pattern. * If you need `[0-9]`, `[α-ω]`, or similar, you may have to manually expand them. * Example: instead of `[0-9]`, write `[0123456789]`. ## Troubleshoot :square-xmark: **Problem:** Contract Verification Post-Processing fails with {% code overflow="wrap" %} ``` pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]The external policy action 'Microsoft.Sql/Sqlservers/Databases/Schemas/Tables/Create' was denied on the requested resource. (368) ``` {% endcode %} :square-check: **Solution:** Ensure you are connecting to Soda via a **Fabric Warehouse**, and not via Fabric Lakehouse. Soda's Diagnostics Warehouse requires Data Definition Language (DDL) e.g. `CREATE TABLE` , which **Fabric Warehouse** supports but Fabric Lakehouse does not. > Refer to the [official Fabric documentation for a full comparison and details](https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-lakehouse-warehouse). *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Oracle Access configuration details to connect Soda to an Oracle data source. ### Connection configuration reference {% hint style="warning" %} Connecting to Oracle requires having installed the `soda-oracle` package using the [Private PyPi installation flow](/deployment-options/soda-python-libraries#private-pypi-installation-flow) available wtih Entreprise license, unless you are using a Runner. Need access to the private PyPI? Please [contact us](mailto:support@soda.io). {% endhint %} Install the following package: {% code overflow="wrap" %} ```bash pip install --pre -U "soda-oracle>4" -i "https://${SODA_CLOUD_API_KEY_ID}:${SODA_CLOUD_API_KEY_SECRET}@enterprise.pypi.us.soda.io/simple ``` {% endcode %} The package must be installed from Soda's private enterprise pypi server using [Soda Cloud API keys](broken://pages/H6Piqnv5lUFCAv1Cvm75) from an organization with an Enterprise license. #### Data source YAML **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` The data source configuration YAML should look like the following: {% code title="ds\_config.yml" %} ```yaml type: oracle name: my_oracle connection: user: system password: ${env.ORACLE_PASSWORD} host: oracle.databases port: 1521 service_name: SODA ``` {% endcode %} {% hint style="info" %} **Note: Oracle Client Libraries must be available to connect to Oracle via Soda Core. You should install the appropriate Oracle Instant Client and make sure that the libraries are accessible, e.g.,:** ``` export PATH=$PATH:/Users/adkinsty/Downloads/instantclient_23_3 export DYLD_FALLBACK_LIBRARY_PATH="/usr/local/lib:~/Downloads/instantclient_23_3:~/lib" ``` {% endhint %} #### Connection test Test the data source connection: ```bash soda data-source test -ds ds_config.yml ``` *** {% hint style="warning" %} **Text truncation in Diagnostics Warehouse** When using **Oracle** as your Diagnostics Warehouse target, **large text values may be truncated** (4,000 char limit) **or produce hashing collisions** when stored in Soda result tables. > To learn more, go to [limitations & edge cases](broken://pages/dWoYeagXP1e77Kf8TS5K#limitations-and-edge-cases) in Diagnostics Warehouse data sources. > {% endhint %} *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Pandas Access configuration details to connect Soda to a Pandas data source. You can run contracts on Pandas dataframes by registering them in DuckDB as in-memory tables, as described [here](/reference/data-source-reference-for-soda-core/duckdb/duckdb-advanced-usage#from-pandas-dataframe). > Learn more: [DuckDB advanced usage](/reference/data-source-reference-for-soda-core/duckdb/duckdb-advanced-usage) *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Polars Access configuration details to connect Soda to a Polars data source. You can run contracts on Polars dataframes by registering them in DuckDB as in-memory tables, as described [here](/reference/data-source-reference-for-soda-core/duckdb/duckdb-advanced-usage#from-polars-dataframe). > Learn more: [DuckDB advanced usage](/reference/data-source-reference-for-soda-core/duckdb/duckdb-advanced-usage) *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # PostgreSQL Access configuration details to connect Soda to a PostgreSQL data source. ### Connection configuration reference Install the package `soda-postgres`: ```bash pip install soda-postgres ``` #### Data source YAML **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` The data source configuration YAML should look like the following: {% code title="ds\_config.yml" %} ```yaml type: postgres name: my_postgres connection: user: ${env.POSTGRES_USER} host: ${env.POSTGRES_HOST} port: 5432 password: ${env.POSTGRES_PW} database: ${env.POSTGRES_DB} ``` {% endcode %} #### Connection test Test the data source connection: ```bash soda data-source test -ds ds_config.yml ``` *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Redshift Access configuration details to connect Soda to an Amazon Redshift data source. ### Connection configuration reference Install the following package: ```bash pip install soda-redshift ``` #### Data source YAML **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` The data source configuration YAML should look like the following: {% code title="ds\_config.yml" %} ```yaml type: redshift name: my_redshift connection: host: ${env.REDSHIFT_HOST} port: 5439 database: ${env.REDSHIFT_DB} user: ${env.REDSHIFT_USER} # optional password: ${env.REDSHIFT_PW} access_key_id: ${env.REDSHIFT_AWS_ACCESS_KEY_ID} secret_access_key: ${env.REDSHIFT_AWS_SECRET_ACCESS_KEY} session_token: ${env.REDSHIFT_AWS_SESSION_TOKEN} role_arn: ${env.REDSHIFT_ROLE_ARN} # e.g., arn:aws:iam::123456789012:role/MyRole region: ${env.REDSHIFT_REGION} # e.g., us-east-1 profile_name: ${env.REDSHIFT_AWS_PROFILE} cluster_identifier: ${env.REDSHIFT_CLUSTER_ID} ``` {% endcode %} #### Connection test Test the data source connection: ```bash soda data-source test -ds ds_config.yml ``` *** ### Case sensitivity {% hint style="warning" %} In [Soda v3](https://docs.soda.io/soda-documentation/soda-v3/), Soda only supports case-insensitive identifiers for this data source. **Users migrating from v3 to v4 might have to account for case** since **Soda v4 enforces case-sensitive identifiers**. {% endhint %} Redshift is usually case-insensitive and interprets all identifiers (e.g. table and column names) as lower case regardless of the original input. However, **Soda enforces case-sensitive identifiers for its sessions**. Starting with **Soda v4**, all queries executed against Redshift are now run with **`enable_case_sensitive_identifier = True`**, regardless of your database’s default setting. This ensures consistent handling of table and column names, including those containing uppercase characters. If you previously relied on case-insensitive behavior (the default in Soda v3), you may need to **review and update any custom SQL**, filters, or references to identifiers to ensure they match the exact case used in your Redshift schema. This change prevents onboarding issues for mixed-case identifiers, but it also means **all SQL executed by Soda is now case sensitive**. {% hint style="warning" %} **Text truncation in Diagnostics Warehouse** When using **Redshift** as your Diagnostics Warehouse target, **large text values may be truncated** (65,535 char limit) **or produce hashing collisions** when stored in Soda result tables. > To learn more, go to [limitations & edge cases](broken://pages/dWoYeagXP1e77Kf8TS5K#limitations-and-edge-cases) in Diagnostics Warehouse data sources. > {% endhint %} *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Spark Dataframe Access configuration details to connect Soda to a Spark Dataframe data source. ## Connection configuration reference Soda supports **Apache Spark** as a scalable distributed SQL engine that can be used with in-memory data frames and existing Spark sessions. Install the following package: ```bash pip install soda-sparkdf ``` *** ### From existing spark session If you already have a running Spark session, you can initialize a Soda Spark DataFrame data source directly from it. ```python from pyspark.sql import SparkSession from soda_core.contracts import verify_contract_locally from soda_sparkdf import SparkDataFrameDataSource spark = ( SparkSession.builder.master("local[*]") .appName("soda_sparkdf") .getOrCreate() ) # Create a database (schema) for organization spark.sql("CREATE DATABASE IF NOT EXISTS my_schema") spark.sql("USE my_schema") # Create the DataFrame and save it as a table in the schema df = spark.createDataFrame([(1,), (2,), (3,)], ["id"]) df.write.mode("overwrite").saveAsTable("my_table") spark_data_source = SparkDataFrameDataSource.from_existing_session( session=spark, name="my_sparkdf" ) result = verify_contract_locally( data_sources=[spark_data_source], contract_file_path="./my_table.yaml", soda_cloud_file_path="../soda-cloud.yaml", publish=True ) if result.is_ok: print("✅ Contract verification passed.") else: print("❌ Contract verification failed:") print(result.get_errors_str()) ``` > Learn more about [Python API](/reference/soda-apis/python-api). #### Example contract Here’s a minimal example of a **Soda contract** that validates the `my_table` dataset in Spark: {% code title="contract.yml" %} ```yaml dataset: my_sparkdf/my_schema/my_table columns: - name: id data_type: integer checks: - missing: checks: - row_count: threshold: must_be: 3 ``` {% endcode %} ### ## Using Spark DataFrames on Databricks ### Example contract: Spark - Databricks Below is an example of a contract scan that runs on a Databricks table with a Spark connector: ```python from soda_core.contracts import verify_contract_locally from soda_sparkdf import SparkDataFrameDataSource from soda_core import configure_logging # Enable or disable verbose logging configure_logging(verbose=True) # unity catalog tables are available in the spark session # dataset DQN in the contract should include the full path to the table, e.g., # dataset: soda_databricks_example/unity_catalog/tyler/obs_test_data_seasoned spark_data_source = SparkDataFrameDataSource.from_existing_session( session=spark, name="soda_databricks_example" ) result = verify_contract_locally( data_sources=[spark_data_source], contract_file_path="obs_test_data_seasoned.yml", soda_cloud_file_path="sc_config.yml", publish=False ) if result.is_ok: print("✅ Contract verification passed.") else: print("❌ Contract verification failed:") print(result.get_errors_str()) ``` ### Databricks Connect compatibility The **Soda Data Contract Spark package** is compatible with the following **Databricks Connect** versions: * **17.3.x:** **17.3.1** and above (within the same minor version) * **17.2.x: 17.2.4** and above (within the same minor version) * **17.1.x:** **17.1.7** and above (within the same minor version) * **17.0.x:** **17.0.10** and above (within the same minor version) * **16.x**: **16.4.9** and above (within the same major version) Versions below these minimums are **not supported**. ### Troubleshoot :square-xmark: **Problem:** In a Databricks Notebook, running Soda results in validation or assertion errors, such as: ``` ValidationError AssertionError: Expected a list, got Unexpected or missing fields during Soda initialization ``` **Solution:**\ Databricks runtimes ship with a preinstalled version of **`pydantic`** that may be incompatible with the version required by Soda. Before invoking Soda, upgrade `pydantic` in your Databricks Notebook: ```python !pip install -U pydantic dbutils.library.restartPython() # for pip installation to take effect ``` :square-xmark: **Problem:** In Databricks Notebook, running ```python from soda_sparkdf import SparkDataFrameDataSource ``` results in error {% code overflow="wrap" %} ``` ImportError: cannot import name 'sql' from 'databricks' (/databricks/spark/python/databricks/__init__.py) ``` {% endcode %} :square-check: **Solution:** Run these in your Databricks Notebook: ```python !pip install databricks-sql-connector dbutils.library.restartPython() # for pip installation to take effect ``` *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Snowflake Access configuration details to connect Soda to a Snowflake data source. ## Connection configuration reference Install the following package: ```bash pip install soda-snowflake ``` #### Core connection: **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` The data source configuration YAML should look like the following: {% code title="ds\_config.yml" %} ```yaml type: snowflake name: my_datasource_name connection: # Core account: ${env.SNOWFLAKE_ACCOUNT} # ., e.g.,ET23172.eu-west-1 database: warehouse: role: # optional, strongly recommended connection_timeout: 240 # optional client_session_keep_alive: false # optional ``` {% endcode %} {% hint style="info" %} Though optional, best practice dictates that you provide a value for `role`. If you do not provide a role, and Snowflake has not assigned a [Snowflake System-Defined Role](https://docs.snowflake.com/en/user-guide/security-access-control-overview.html#system-defined-roles) to the user account, Snowflake may, confusingly, deny access to the data source. {% endhint %}
Core connection properties
PropertyRequiredTypeDescription
typeyesstringIdentify the type of data source for Soda. In this case, must be snowflake.
accountyesstringProvide the unique value that identifies your account. Consider using system variables to retrieve this value securely using, for example, ${SNOWFLAKE_ACCOUNT}. Note: Account sometimes needs to take the form of <account_identifier>-<account_name> or <account_identifier>.<region>.
databaseyesstringProvide an identifier for your database.
warehouseyesstringProvide an identifier for the cluster of resources that is a Snowflake virtual warehouse. See Overview of Warehouses.
role1recommendedstringSpecify a Snowflake role that has permission to access the database and schema of your data source.
connection_timeoutnointegerSet the timeout period in seconds for an inactive login session.
client_session_keep_alivenobooleanUse this parameter to keep the session active, even with no user activity.
Default value: false.
session_parametersnoobjectPass-through Snowflake session params (e.g., QUERY_TAG, QUOTED_IDENTIFIERS_IGNORE_CASE).
proxy_httpnostringHTTP proxy (runner environments); see Troubleshoot section.
proxy_httpsnostringHTTPS proxy (runner environments); see Troubleshoot section.
1 Though optional, best practice dictates that you provide a value for `role`. If you do not provide a role, and Snowflake has not assigned a [Snowflake System-Defined Role](https://docs.snowflake.com/en/user-guide/security-access-control-overview.html#system-defined-roles) to the user account, Snowflake may, confusingly, deny access to the data source. **Authenticator selector**
PropertyRequiredValues / ExampleDescription
authenticator2noexternalbrowser | OAUTH_CLIENT_CREDENTIALS

Add an authenticator paramater with value externalbrowser to authenticate the connection to your Snowflake data source using any SAML 2.0-compliant identity provider (IdP) such as

Okta or OneLogin.

If omitted, Soda uses user/password auth (default)

2 Use this parameter when adding Snowflake connection configurations to a `configuration.yml` file. However, if you are adding connection configuration details directly in Soda Cloud (connecting to your Snowflake data source via a Soda Runner) to authenticate using Okta, you must follow the instructions documented by Snowflake for [Native SSO - Okta Only](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-use.html#native-sso-okta-only).
Choose exactly **one authentication approach** of the four options below. :octagon-check: **Supported by Soda** * Default (`SnowflakePasswordAuth`) * External browser (`SnowflakeSSOAuth`) * key\_pair (`SnowflakeKeyPairAuth`) * OAuth (`SnowflakeOAuthAuth`) * OAuth client credentials (`SnowflakeClientCredentialsOAuthAuth`) :octagon-xmark: **Not supported by Soda** * OAuth **Authorization Code** grant * User/password **MFA** flows * **Workload identity** authenticator * **Programmatic access token** (PAT) * Programmatic access token **with external session** {% hint style="info" %} **Notes**: * Use **one** authentication method per connection. * While `role` is technically optional, providing it avoids confusing access errors. * Private key auth can use inline `private_key` (PEM) or `private_key_path`. * `externalbrowser` SSO uses your SAML 2.0 IdP (e.g., Okta/OneLogin). * Proxy parameters are supported when connecting via a runner behind a proxy. {% endhint %} ### 1. User + password ```yaml # 1) User + password user: ${env.SNOWFLAKE_USER} password: ${env.SNOWFLAKE_PASSWORD} ```
User/password properties (when no authenticator is set)
PropertyRequiredDescription
useryesSnowflake user login.
Consider using system variables to retrieve this value securely using, for example, ${SNOWFLAKE_USER}.
passwordyesPassword for the user.
Consider using system variables to retrieve this value securely using, for example, ${SNOWFLAKE_PASSWORD}.
### 2. External browser SSO ```yaml # 2) External browser SSO (SAML 2.0 IdP such as Okta/OneLogin) authenticator: externalbrowser ```
External browser SSO properties (when authenticator: externalbrowser)
PropertyRequiredDescription
usernoDepending on IdP settings, may be required.
Consider using system variables to retrieve this value securely using, for example, ${SNOWFLAKE_USER}.
authenticatoryesMust be externalbrowser to enable SAML 2.0 browser-based SSO.
### 3. Key pair (JWT) authentication You can use the `private_key` and `private_key_passphrase` parameters to specify for key pair authentication. In you configuration YML file, add the parameters as per the following example. ```yaml # 3) Key pair (JWT) authentication authenticator: SNOWFLAKE_JWT user: private_key: | -----BEGIN ENCRYPTED PRIVATE KEY----- -----END ENCRYPTED PRIVATE KEY----- private_key_passphrase: ${env.SNOWFLAKE_PASSPHRASE} # or use a file path instead of inline key: private_key_path: /path/to/private-key.pk8 ```
Key pair / JWT properties (when authenticator: SNOWFLAKE_JWT)
PropertyRequiredDescription
useryesSnowflake user that owns the key pair.
private_keyyes*Inline PEM private key. Use either this or private_key_path.
private_key_pathyes*Path to private key file (.pk8). Use either this or private_key.
private_key_passphrasenoPassphrase for encrypted private key.
{% hint style="info" %} Do **not** include `password` when `authenticator: SNOWFLAKE_JWT` is used. {% endhint %}
### 4. OAuth 2.0 Client Credentials ```yaml # 4) OAuth 2.0 Client Credentials authenticator: OAUTH_CLIENT_CREDENTIALS oauth_client_id: oauth_client_secret: oauth_token_request_url: https:///oauth/token oauth_scope: "scope1 scope2" # space-delimited; optional if derived by role ```
OAuth 2.0 client properties (when authenticator: OAUTH_CLIENT_CREDENTIALS)
PropertyRequiredDescription
oauth_client_idyesClient ID from the IdP for the Snowflake security integration.
oauth_client_secretyesClient secret from the IdP for the Snowflake security integration.
oauth_token_request_urlyesIdP token endpoint that issues access tokens to the driver. (With Snowflake as IdP, derive from server/account parameters.)
oauth_scopenoSpace-delimited, case-sensitive scopes. Defaults may be derived from role; specify explicitly for multiple/custom scopes.
{% hint style="info" %} Only the **Client Credentials** grant is supported. The **Authorization Code** grant is **not** supported. {% endhint %} *** ### Other parameters ```yaml # Optional Snowflake session parameters (passed through) session_parameters: QUERY_TAG: soda-queries QUOTED_IDENTIFIERS_IGNORE_CASE: false # Optional proxies (for runners behind proxy) proxy_http: http://host:port proxy_https: https://host:port ```
Other parameters properties | Property | Required | Notes | | -------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | other params | optional | You can pass any other Snowflake parameters you wish by adding the key:value pairs to your Snowflake connection configuration. See [Snowflake Python Connector API documentation](https://docs.snowflake.com/en/user-guide/python-connector-api.html#connect) for a list of passable parameters. | | QUERY\_TAG | optional | See [QUERY\_TAG](https://docs.snowflake.com/en/sql-reference/parameters.html#query-tag) in Snowflake documentation. | |

QUOTED\_IDENTIFIERS\_
IGNORE\_CASE

| optional | See [QUOTED\_IDENTIFIERS\_IGNORE\_CASE](https://docs.snowflake.com/en/sql-reference/parameters.html#quoted-identifiers-ignore-case) in Snowflake documentation. |
*** ## Use a values file to store private key authentication values If you use a private key authentication with Snowflake and have deployed a [Soda Runner](/deployment-options/soda-runner), you can provide the required private key values in a `values.yml` file when you deploy or redeploy the runner. You can also use the `values.yml` file to store other environment variables for the Soda Runner to use, such as `SNOWFLAKE_USER`, `SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_PASSPHRASE`, etc. 1. First, run the following command to create a local path to the Snowflake private key. Replace the `local path to the Snowflake private key` with your own value. 
kubectl create secret generic -n <soda-runner-namespace> snowflake-private-key --from-file=snowflake-private-key.pk8=<local path to the Snowflake private key>
2. Then, add the following to your `values.yml` file, adjusting the values to your own specific details. 
soda:
     scanLauncher:
       volumeMounts:
         - name: snowflake-private-key
           mountPath: /opt/soda/etc
       volumes:
         - name: snowflake-private-key
           secret:
             secretName: snowflake-private-key
             items:
               - key: snowflake-private-key.pk8
                 path: snowflake-private-key.pk8
     contractLauncher:
       volumeMounts:
         - name: snowflake-private-key
           mountPath: /opt/soda/etc
3. Adjust the configuration file to include the new path in the connection details, as in the following example. ```yaml data_source ltsnowflakecustomer: type: snowflake account: ${env.SNOWFLAKE_ACCOUNT} database: PUBLISH_DEV warehouse: ${env.SNOWFLAKE_WAREHOUSE} role: ${env.SNOWFLAKE_ROLE} user: ${env.SNOWFLAKE_USER} authenticator: SNOWFLAKE_JWT private_key_passphrase: ${env.SNOWFLAKE_PASSPHRASE} private_key_path: /opt/soda/etc/snowflake-private-key.pk8 client_session_keep_alive: true session_parameters: QUERY_TAG: soda-queries QUOTED_IDENTIFIERS_IGNORE_CASE: false ``` 4. Deploy, or redeploy, the runner for the changes to take effect. ## Connection test Test the data source connection: ```bash soda data-source test -ds ds_config.yml ``` ## Supported data types | Category | Data type | | -------- | --------------------------------------------------------------------------------------------------------------- | | text | CHAR, VARCHAR, CHARACTER, STRING, TEXT | | number | NUMBER, INT, INTEGER, BIGINT, SMALLINT, TINYINT, BYTEINT, FLOAT, FLOAT4, FLOAT8, DOUBLE, DOUBLE PRECISION, REAL | | time | DATE, DATETIME, TIME, TIMESTAMP, TIMESTAMPT\_LTZ, TIMESTAMP\_NTZ, TIMESTAMP\_TZ | ## Troubleshoot :square-xmark: **Problem:** When testing the connection to your Snowflake data source, Snowflake returns an error message about using the `use database` command. :square-check: **Solution:** Use the `role` parameter to specify a Snowflake role that has permission to access the `database` and `schema` of your data source.\ Though optional, best practice dictates that you provide a value for `role`. If you do not provide a role, and Snowflake has not assigned a [Snowflake System-Defined Role](https://docs.snowflake.com/en/user-guide/security-access-control-overview.html#system-defined-roles) to the user account, Snowflake may, confusingly, deny access to the data source.
:square-xmark: **Problem:** When Soda attempts to connect to your Snowflake data source, it produces a connectivity error that includes something like `RuntimeError: Command failed with exit code 2: ..... ocsp_response_validation_cash.lock`. :square-check: **Solution:**\ Use [Snowflake's troubleshooting guide](https://community.snowflake.com/s/article/How-to-Triage-OCSP-Related-Connectivity-Problems) to triage OCSP-related connectivity issues.
:square-xmark: **Problem:** You have defined a Group By check and the scan that executes the check yields an error. :square-check: **Solution:** Be aware that, by default, Snowflake returns columns names in uppercase. Therefore, when defining a Group By check, you must specify the custom name of the check in uppercase as in the following example. ```yaml checks for VOLUME: - group by: group_limit: 50 query: | SELECT TRADER, count(*) as trader_row_count FROM TRADEVOLUME WHERE TRADE_DATE = '2023-01-01' GROUP BY TRADER fields: - TRADER checks: - TRADER_ROW_COUNT > 40: name: Trader row count ``` :square-xmark: **Problem:** While attempting to connect Soda to a Snowflake data source using proxy parameters, you encounter an error that reads something similar to `Could not connect to data source "name_db": 250001 (08001): Failed to connect to DB: mydb.eu-west-1.snowflakecomputing.com:443. Incoming request with IP/Token xx.xxx.xx.xxx is not allowed to access Snowflake.` ```yaml data_source my_data_source: type: snowflake ... session_parameters: QUERY_TAG: soda-test QUOTED_IDENTIFIERS_IGNORE_CASE: false proxy_http: http://a-proxy-o-dd-dddd-net:8000 proxy_https: https://a-proxy-o-dd-dddd-net:8000 ``` :square-check: **Solution:** When connecting to a Snowflake data source by proxy, be sure to set the new proxy environment variables from the command-line using export statements, as in the following example. ```sh export HTTP_PROXY=http://a-proxy-o-dd-dddd-net:8000 export HTTPS_PROXY=https://a-proxy-o-dd-dddd-net:8000 ``` *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # SQL Server Access configuration details to connect Soda to a Microsoft SQL Server data source. ### Requirements * Install the [Microsoft ODBC driver for SQL Server](https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver17) before proceeding with the connection configuration. ### Connection configuration reference Install the following package: ```bash pip install soda-sqlserver ``` #### Data source YAML **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` The data source configuration YAML should look like the following: {% code title="ds\_config.yml" %} ```yaml type: sqlserver name: my_sqlserver connection: host: ${env.SQLSERVER_HOST} port: 1433 database: ${env.SQLSERVER_DB} user: ${env.SQLSERVER_USER} # SEE NOTE password: ${env.SQLSERVER_PW} # SEE NOTE authentication: sql # activedirectoryserviceprincipal | activedirectoryinteractive | activedirectorypassword # optional client_id: ${env.SQLSERVER_SERVICE_CLIENT_ID} # SEE NOTE client_secret: ${env.SQLSERVER_SERVICE_CLIENT_SECRET} # SEE NOTE driver: ODBC Driver 18 for SQL Server trusted_connection: false encrypt: false trust_server_certificate: false connection_parameters: application_intent: 'ReadOnly' ``` {% endcode %} {% hint style="info" %} **Note:** depending on the authentication method that is used, `user` and `password` may not be required (e.g. `activedirectoryserviceprincipal` requires `client_id` and `client_secret`). {% endhint %} #### Connection test Test the data source connection: ```bash soda data-source test -ds ds_config.yml ``` ### Regex patterns in SQL Server SQL Server and related databases **do not offer universal regex support**. When using regex-based checks in these environments, Soda adapts pattern-matching queries. Instead of regex functions, **Soda translates patterns into SQL queries using the `PATINDEX` function**. #### How Soda translates regex patterns If the user specifies a regex `pattern` which matches a `column expression`, it will be expanded in SQL as: {% code overflow="wrap" %} ```sql PATINDEX('%{pattern}%', {column expression} COLLATE SQL_Latin1_General_Cp1_CS_AS) > 0 ``` {% endcode %} **Key considerations** * **Collation** A collation (a rule set used for determining string matches) must be specified. Soda uses the `SQL_Latin1_General_Cp1_CS_AS`, which is **case-sensitive** and **accent-sensitive**, so `a` and `A` are not equal, and accented characters (such as `é`) are treated distinctly. * **Pattern handling** Regex patterns are wrapped in `%…%`, so any substring match within the column value will return `true`. Example: `^abc` becomes `%abc%`, which will match anywhere in the string. * **Character ranges**\ In SQL Server and related databases, character range expansion for alpha characters is handled differently than in most regex engines. * Soda **auto-expands** the most common ranges for you: * `[a-z]` → `[abcdefghijklmnopqrstuvwxyz]` * `[A-Z]` → `[ABCDEFGHIJKLMNOPQRSTUVWXYZ]` * **Other ranges are not expanded automatically** and you may need to manually expand the full set when defining your pattern. * If you need `[0-9]`, `[α-ω]`, or similar, you may have to manually expand them. * Example: instead of `[0-9]`, write `[0123456789]`. * **`connection_parameters`** You can define key-value pairs that will be passed directly to the connection string. For example: `application_intent: 'ReadOnly'` → `application_intent=ReadOnly` in the final connection string. {% if visitor.claims.plan === %} ### Troubleshoot :square-xmark: **Problem:** Metrics Monitoring Scan fails with error: {% code overflow="wrap" %} ``` Query execution error in ...: ('42000', "[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]VIEW SERVER PERFORMANCE STATE permission was denied on object 'server', database 'master'. (300) (SQLExecDirectW)") SELECT SCHEMA_NAME(O.schema_id), O.name, SUM(P.rows), MAX(O.modify_date) , MAX(S.last_user_update) FROM sys.objects AS O INNER JOIN sys.partitions P ON P.object_id = O.object_id LEFT JOIN sys.dm_db_index_usage_stats S ON S.object_id = O.object_id WHERE type='U' AND ((SCHEMA_NAME(O.schema_id) = 'tab' AND O.name IN (''))) GROUP BY O.schema_id, O.name | ('42000', "[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]VIEW SERVER PERFORMANCE STATE permission was denied on object 'server', database 'master'. (300) (SQLExecDirectW)") ``` {% endcode %} :square-check: **Solution:** Ensure the user login name used to connect to Soda (e.g. `soda_user` ) has VIEW permissions on `sys.dm_db_index_usage_stats` (which is one of the built-in SQL Server Dynamic Management Views DMVs, prefixed with sys.dm\_\*). For example, run this SQL query (e.g. as sysadmin) to `GRANT` the necessary permissions to the user: ```sql USE master; GO GRANT VIEW SERVER PERFORMANCE STATE TO soda_user; --- alternatively, VIEW SERVER STATE GO ``` {% endif %} *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Synapse Access configuration details to connect Soda to an Azure Synapse Analytics data source. ### Requirements * Install the [Microsoft ODBC driver for SQL Server](https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver17) before proceeding with the connection configuration. ### Connection configuration reference Install the following package: ```bash pip install soda-synapse ``` #### Data source YAML **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` The data source configuration YAML should look like the following: {% code title="ds\_config.yml" %} ```yaml type: synapse name: my_synapse connection: host: port: 1433 database: ${env.SYNAPSE_DB} username: ${env.SYNAPSE_USER} # SEE NOTE password: ${env.SYNAPSE_PW} # SEE NOTE authentication: sql # activedirectoryserviceprincipal | activedirectoryinteractive | activedirectorypassword # optional client_id: ${env.SYNAPSE_SERVICE_CLIENT_ID} # SEE NOTE client_secret: ${env.SYNAPSE_SERVICE_CLIENT_SECRET} # SEE NOTE driver: ODBC Driver 18 for SQL Server trusted_connection: false encrypt: false trust_server_certificate: false ``` {% endcode %} {% hint style="info" %} **Note:** depending on the authentication method that is used, `user` and `password` may not be required (e.g. `activedirectoryserviceprincipal` requires `client_id` and `client_secret`). {% endhint %} #### Connection test Test the data source connection: ```bash soda data-source test -ds ds_config.yml ``` ## Limitations & edge cases ### Regex patterns in Synapse Synapse databases **do not offer universal regex support**. When using regex-based checks in these environments, Soda adapts pattern-matching queries. Instead of regex functions, **Soda translates patterns into SQL queries using the `PATINDEX` function**. #### How Soda translates regex patterns If the user specifies a regex `pattern` which matches a `column expression`, it will be expanded in SQL as: ```sql PATINDEX('%{pattern}%', {column expression} COLLATE SQL_Latin1_General_Cp1_CS_AS) > 0 ``` **Key considerations** * **Collation** A collation (a rule set used for determining string matches) must be specified. Soda uses the `SQL_Latin1_General_Cp1_CS_AS`, which is **case-sensitive** and **accent-sensitive**, so `a` and `A` are not equal, and accented characters (such as `é`) are treated distinctly. * **Pattern handling** Regex patterns are wrapped in `%…%`, so any substring match within the column value will return `true`. Example: `^abc` becomes `%abc%`, which will match anywhere in the string. * **Character ranges**\ In SQL Server and related databases, like Synapse, character range expansion for alpha characters is handled differently than in most regex engines. * Soda **auto-expands** the most common ranges for you: * `[a-z]` → `[abcdefghijklmnopqrstuvwxyz]` * `[A-Z]` → `[ABCDEFGHIJKLMNOPQRSTUVWXYZ]` * **Other ranges are not expanded automatically** and you may need to manually expand the full set when defining your pattern. * If you need `[0-9]`, `[α-ω]`, or similar, you may have to manually expand them. * Example: instead of `[0-9]`, write `[0123456789]`. ### Storage structure Soda uses a **Heap** table storage structure in Synapse. Unlike indexed structures, **Heap stores rows without predefined order or compression**, which is required for compatibility with certain data types used in the warehouse (VARCHAR(MAX), TEXT, and XML) without character limits. As a side effect, **tables in the diagnostics warehouse may consume more storage** than equivalent tables using columnar compression. This is expected behavior and a known trade-off of the current architecture. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Trino Access configuration details to connect Soda to a Trino data source. {% hint style="info" %} Soda uses the official [Trino Python client](https://github.com/trinodb/trino-python-client) internally to establish connections and execute queries. {% endhint %} ## Connection configuration reference Install the following package: ```bash pip install soda-trino ``` ### Data source YAML #### Core connection **Create the config file:** ```shellscript soda data-source create -f ds_config.yml ``` **Configure connection** To connect to Trino, add a `data_source` block to your `ds_config.yml` file.
Authentication methodSoda Cloud supportSoda Core support
Username/password✅✅
JWT (existing token)✅
JWT (OAuth – client credentials flow)✅✅
No authentication✅
**Core connection parameters** | Parameter | Required | Default | Description | | --------- | -------- | ------- | ------------------------------------------------ | | `type` | Yes | — | Must be `trino`. | | `host` | Yes | — | Hostname or IP address of the Trino coordinator. | | `catalog` | Yes | — | Trino catalog name. | | `verify` | No | `True` | SSL verification. Supported in Soda Core only. | #### 1. Username/password Supported in Soda Cloud and Soda Core. {% code title="ds\_config.yml" %} ```yaml type: trino name: my_trino connection: host: trino.example.com catalog: hive # Optional connection settings port: 443 http_scheme: https source: soda-core client_tags: ["dq", "prod"] # Authentication # auth_type: BasicAuthentication # Optional user: ${TRINO_USER} password: ${TRINO_PASSWORD} ``` {% endcode %} | Parameter | Required | Default | Description | | ------------- | ----------------- | ----------- | ---------------------------------------- | | `port` | No | `443` | Trino port. | | `http_scheme` | No | `https` | One of `http`, `https`. | | `source` | No | `soda-core` | Trino-internal label for the connection. | | `client_tags` | No | — | Trino-internal tags for this connection. | | `auth_type` | Depends | — | Authentication type. See modes above. | | `user` | Username/password | — | Trino username. | | `password` | Username/password | — | Trino password. | *** #### 2. JWT (existing token) Supported in Soda Core only. {% code title="ds\_config.yml" %} ```yaml type: trino name: my_trino connection: host: trino.example.com catalog: hive auth_type: JWTAuthentication access_token: ${TRINO_JWT_ACCESS_TOKEN} ``` {% endcode %} | Parameter | Required | Default | Description | | -------------- | -------------------- | ------- | ------------------------------------- | | `auth_type` | Depends | — | Authentication type. See modes above. | | `access_token` | JWT (existing token) | — | JWT access token. | *** #### 3. JWT (OAuth – client credentials) Supported in Soda Cloud and Soda Core. ```yaml type: trino name: my_trino connection: host: trino.example.com catalog: hive auth_type: OAuth2ClientCredentialsAuthentication oauth: token_url: https://auth.example.com/oauth2/token client_id: ${TRINO_OAUTH_CLIENT_ID} client_secret: ${TRINO_OAUTH_CLIENT_SECRET} scope: "scope1 scope2" # Optional grant_type: client_credentials ``` | Parameter | Required | Default | Description | | --------------------- | ----------- | -------------------- | ------------------------------------- | | `auth_type` | Depends | — | Authentication type. See modes above. | | `oauth` | JWT (OAuth) | — | OAuth configuration block. | | `oauth.token_url` | JWT (OAuth) | — | OAuth token endpoint. | | `oauth.client_id` | JWT (OAuth) | — | OAuth client ID. | | `oauth.client_secret` | JWT (OAuth) | — | OAuth client secret. | | `oauth.scope` | No | `null` | Space-separated scopes. | | `oauth.grant_type` | No | `client_credentials` | OAuth grant type. | *** #### 4. No authentication Supported in Soda Core only. ```yaml type: trino name: my_trino connection: host: trino.example.com catalog: hive port: 8080 http_scheme: http auth_type: NoAuthentication ``` *** ### Test the connection To test your connection configuration: ```bash soda data-source test -ds ds_config.yml ``` *** ## Limitations & edge cases Using Trino as a Diagnostics Warehouse destination through between-source copying is available but **not fully supported**. There is a known issue when the Trino catalog is Iceberg-based. You are advised to: * Use within-source copying, or * Configure Soda to connect directly to the destination data source instead of routing through the Trino catalog connection. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Soda CLI reference Manage your entire data quality lifecycle from the terminal or your pipeline. Soda CLI (`sodacli`) is a **single command-line tool for managing Soda Cloud.** From your terminal or a CI/CD pipeline, you can connect data sources, onboard datasets, generate and verify contracts, configure monitors, review results, and manage runners, secrets, and access. All through one interface. Soda CLI is a single Go binary with **no dependencies**. Every command is built to work the same way for a person typing in a terminal and for an AI agent or automation calling it programmatically: flags, exit codes and outputs are all machine- and human-readable. > **Core CLI and Soda CLI are not the same.** > > Looking for **local contract authoring and verification with Soda Core**? See the [Core CLI reference](/reference/cli-reference) for the `soda` command. > >
> > Soda CLI vs Core CLI > >
> **Which CLI should I use?** > > Soda has two command-line tools: > > * [**Core CLI**](/reference/cli-reference) **(`soda`)** — a Python tool focused on **data contracts**, run locally. Create, test, publish, and verify contracts, with fine-grained check selection. > * **Soda CLI (`sodacli`)** — a single Go binary that manages your entire Soda Cloud platform: datasources, datasets, contracts, monitors, results, incidents, IAM, runners, and secrets. Authenticate once and drive it from your terminal, CI/CD, or an AI agent. > > Use **`sodacli`** to manage the full platform; use **`soda`** for local, contract-only work. They work together:`sodacli contract verify --local` runs `soda` under the hood. > | | **Soda CLI (`sodacli`)** | **Soda Core CLI (`soda`)** | | ----------------------- | ---------------------------------------------------------------------------------------------------- | --------------------------------------------- | | **Language** | Go single binary, no dependencies | Python (pip + data source connectors) | | **Execution** | Cloud via Soda Runner, or local with `--local` | Local only | | **Scope** | Full platform: datasources, datasets, contracts, monitors, results, runners, secrets, IAM, incidents | Contract verification and data source testing | | **Contract generation** | `contract create --mode copilot` (AI) or `skeleton` | Manual authoring | | **CI/CD** | `--no-interactive`, `--output json`, structured exit codes | Basic exit codes | ## Install {% tabs %} {% tab title=" macOS/Linux" %} #### Homebrew ```bash brew tap sodadata/tap brew install sodacli ``` #### Install script {% code overflow="wrap" %} ```bash curl -sSL https://raw.githubusercontent.com/sodadata/soda-cli/main/install.sh | sh ``` {% endcode %} {% endtab %} {% tab title="Windows" %} Download the latest `.zip` for your architecture from [GitHub Releases](https://github.com/sodadata/soda-cli/releases), extract `sodacli.exe`, and add it to your `PATH`. {% endtab %} {% tab title="From source" %} * Requires Go 1.22+. ```bash git clone https://github.com/sodadata/soda-cli.git cd soda-cli/go go build -o sodacli . sudo mv sodacli /usr/local/bin/ ``` {% endtab %} {% endtabs %} Verify the installation: ```bash sodacli version sodacli --help ``` ## Quickstart {% stepper %} {% step %} ### Authenticate ```bash # Interactive: prompts for host, API key ID, and secret sodacli auth login # Confirm the connection sodacli auth status ``` [Generate API keys](broken://pages/H6Piqnv5lUFCAv1Cvm75) in Soda Cloud. {% endstep %} {% step %} ### Onboard a datasource ```bash # Create the datasource, discover datasets, and enable monitoring, # profiling, contract generation, and verification in one command sodacli datasource onboard warehouse.yml --monitoring --profiling --contracts copilot ``` {% endstep %} {% step %} ### Verify a contract ```bash # Run checks via a Soda Cloud Runner sodacli contract verify orders.yml # Or run locally via Soda Core sodacli contract verify orders.yml --local --datasource datasource.yml # Review results sodacli results list --status failing sodacli job logs ``` {% endstep %} {% endstepper %} ## Global flags These flags work on every command.
FlagShortDescription
--output table|json|csv-oOutput format. Auto-detects: a table in an interactive terminal, JSON when piped.
--profile <name>Override the active auth profile.
--no-colorDisable color output.
--quiet-qSuppress non-essential output.
--verbose-vShow detailed output.
--no-interactiveNever prompt; fail with a clear error if required input is missing. Use in CI/CD and for AI agents.
## Output formats The CLI picks a format automatically: a human-readable, colored table in an interactive terminal, and JSON when output is piped. Override the format on any command with `--output`. ```bash sodacli dataset list # colored table sodacli dataset list --output json # JSON sodacli dataset list --output csv # CSV sodacli dataset list | jq '.[] | .id' # auto-JSON when piped ``` ## Essential Commands {% hint style="info" %} This section covers the most common commands. For the complete command surface, see the `soda-cli` repository, or run `sodacli --help`. {% embed url="" %} Soda CLI command tree on GitHub {% endembed %} {% endhint %} ### Authentication Credentials are stored in `~/.soda/credentials`. `login` runs an interactive wizard unless `--no-interactive` is set, in which case `--api-key-id` and `--api-key-secret` are required. | Command | Description | | ------------------------------- | ------------------------------------------------------------------------------ | | `sodacli auth login` | Authenticate with Soda Cloud. Add `--host cloud.us.soda.io` for the US region. | | `sodacli auth status` | Check connection health for the active profile. | | `sodacli auth switch ` | Switch between saved profiles. | ### Datasources Manage warehouse connections. Aliased to `ds`. | Command | Description | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `sodacli datasource list` | List datasources. | | `sodacli datasource get ` | Show a datasource's configuration. | | `sodacli datasource create ` | Register a datasource from a YAML config; returns its ID. | | `sodacli datasource onboard ` | Guided end-to-end setup. Supports `--monitoring`, `--profiling`, and `--contracts copilot\|skeleton\|none`. | | `sodacli datasource update ` | Update label, runner, or connection. | | `sodacli datasource test-connection ` | Async connection test via a Runner. | | `sodacli datasource diagnostics ` | View or configure the diagnostics warehouse. | | `sodacli datasource delete ` | Delete a datasource and its resources. | ### Datasets Manage datasets, their configuration, and access. | Command | Description | | ----------------------------------------- | -------------------------------------------------------------------- | | `sodacli dataset list` | List datasets. Supports `--datasource`, `--status`, `--limit`. | | `sodacli dataset get ` | Show dataset details. | | `sodacli dataset update ` | Update tags or owner. | | `sodacli dataset attributes ` | List a dataset's attributes. | | `sodacli dataset profiling ` | View or configure profiling, e.g. `--enable --schedule "0 6 * * *"`. | | `sodacli dataset time-partition ` | Set or view the time-partition column. | | `sodacli dataset diagnostics ` | View or configure dataset-level diagnostics. | | `sodacli dataset permissions list ` | List role assignments on a dataset. | | `sodacli dataset permissions assign ` | Assign a role to a user or group. | ### Contracts Generate, validate, publish, and verify data contracts. | Command | Description | | ----------------------------------------- | ------------------------------------------------------------------------------- | | `sodacli contract list` | List all contracts. | | `sodacli contract create --dataset ` | Generate a contract from a dataset schema. `--mode skeleton\|copilot`. | | `sodacli contract pull ` | Download a contract from Soda Cloud. | | `sodacli contract push ` | Upload a local contract to Soda Cloud. | | `sodacli contract diff ` | Show a local-vs-cloud diff. | | `sodacli contract lint ` | Validate contract syntax offline. Supports globs. | | `sodacli contract verify ` | Run checks via a cloud Runner, or locally with `--local --datasource `. | ### Monitors Configure ML-based metric monitors (anomaly detection). | Command | Description | | ------------------------------------- | --------------------------------------------------------------------------------------------------------- | | `sodacli monitor list --dataset ` | List a dataset's monitors. | | `sodacli monitor config ` | View or set the monitoring schedule, e.g. `--enable --schedule "0 */6 * * *"`. | | `sodacli monitor add --dataset ` | Add a monitor: `--type column` (with `--column`, `--metric`) or `--type custom` (with `--name`, `--sql`). | | `sodacli monitor update ` | Update a monitor. | | `sodacli monitor delete ` | Delete a monitor. | ### Results | Command | Description | | ---------------------- | ------------------------------------------------------------------------------- | | `sodacli results list` | List check results. Supports `--dataset-name`, `--status`, `--from`, `--limit`. | ### Jobs Observe scans. Aliased to `scan`. | Command | Description | | ------------------------------ | -------------------------------------- | | `sodacli job status ` | Show a scan's state and check summary. | | `sodacli job logs ` | Show a scan's logs. | ### IAM Manage users, groups, and roles. | Command | Description | | ----------------------------------------- | ----------------------------------------------- | | `sodacli iam user list` | List users. | | `sodacli iam user invite --email ` | Invite users (repeatable, up to 10). | | `sodacli iam group create --name ` | Create a group; add members with `--member`. | | `sodacli iam group update ` | Update membership, e.g. `--add-member `. | | `sodacli iam role list` | List roles. Supports `--scope global\|dataset`. | ### Runners | Command | Description | | ------------------------------------- | --------------------------------------- | | `sodacli runner list` | List registered runners. | | `sodacli runner get ` | Show a runner's details. | | `sodacli runner create --name ` | Create runner credentials (shown once). | | `sodacli runner delete ` | Delete a runner. | ### Secrets Encrypted credentials referenced from datasource configs as `${secret.NAME}`. Values are encrypted client-side before they leave your machine. | Command | Description | | ------------------------------------- | ------------------------------------------------------- | | `sodacli secret list` | List secrets. | | `sodacli secret get ` | Show a secret's details. | | `sodacli secret create --name ` | Create a secret via masked prompt, `--value`, or stdin. | | `sodacli secret update ` | Update a secret's value. | | `sodacli secret delete ` | Delete a secret. | ## CI/CD integration Every command runs non-interactively with `--no-interactive`, emits machine-readable output with `--output json`, and returns structured exit codes. ```bash sodacli auth login \ --host cloud.soda.io \ --api-key-id "$SODA_API_KEY_ID" \ --api-key-secret "$SODA_API_KEY_SECRET" \ --no-interactive sodacli contract verify contracts/orders.yml --no-interactive --output json ``` `contract verify` exit codes:
CodeMeaning
0All checks passed.
1One or more checks failed — fail the pipeline.
2Execution error — retry or alert.
3Authentication error — check credentials.
### GitHub Actions example ```yaml - name: Verify data contracts run: | sodacli auth login --host cloud.soda.io \ --api-key-id ${{ secrets.SODA_KEY_ID }} \ --api-key-secret ${{ secrets.SODA_KEY_SECRET }} \ --no-interactive sodacli contract verify contracts/orders.yml --no-interactive ``` ## Telemetry Soda CLI collects anonymous usage data (command names, exit codes, duration, OS/arch) to help improve the tool. No personal information, API keys, file contents, or query data is collected. Opt out by setting an environment variable: ```bash export SODACLI_TELEMETRY=false ``` *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Core CLI reference This guide documents the Soda Core CLI commands for working with Soda Data Contracts. You can use the CLI to generate, test, publish, and verify contracts using either Soda Core (local execution) or Soda Runner (remote execution). > **Core CLI and Soda CLI are not the same.** > > Looking to **manage your full Soda Cloud platform** (datasources, datasets, monitors, runners, and more) from one tool? See the [Soda CLI reference](/reference/soda-cli-reference) for the `sodacli` command. > >
> > Soda CLI vs Core CLI > >
> **Which CLI should I use?** > > Soda has two command-line tools: > > * **Core CLI (`soda`)** — a Python tool focused on **data contracts**, run locally. Create, test, publish, and verify contracts, with fine-grained check selection. > * [**Soda CLI** ](/reference/soda-cli-reference)**(`sodacli`)** — a single Go binary that manages your entire Soda Cloud platform: datasources, datasets, contracts, monitors, results, incidents, IAM, runners, and secrets. Authenticate once and drive it from your terminal, CI/CD, or an AI agent. > > Use **`sodacli`** to manage the full platform; use **`soda`** for local, contract-only work. They work together:`sodacli contract verify --local` runs `soda` under the hood. > | | **Soda CLI (`sodacli`)** | **Soda Core CLI (`soda`)** | | ----------------------- | ---------------------------------------------------------------------------------------------------- | --------------------------------------------- | | **Language** | Go single binary, no dependencies | Python (pip + data source connectors) | | **Execution** | Cloud via Soda Runner, or local with `--local` | Local only | | **Scope** | Full platform: datasources, datasets, contracts, monitors, results, runners, secrets, IAM, incidents | Contract verification and data source testing | | **Contract generation** | `contract create --mode copilot` (AI) or `skeleton` | Manual authoring | | **CI/CD** | `--no-interactive`, `--output json`, structured exit codes | Basic exit codes | {% hint style="danger" %} **Nomenclature change: "Agent" is now "Runner"** The Soda Agent has been renamed to [**Soda Runner**](/reference/soda-runner-basic-concepts) across all Soda products and interfaces. This change affects terminology throughout the platform, including: * Soda Cloud UI labels * API permission names: `MANAGE_DATASOURCES_AND_AGENTS` → `MANAGE_DATASOURCES_AND_RUNNERS` * CLI flags: `--use-agent` → `--use-runner` * Python API methods: `verify_contract_on_agent` → `verify_contract_on_runner` The underlying functionality is unchanged, and **both runner/agent nomenclatures are backward compatible** at present. If you are following older documentation or have existing scripts and automation that reference "agent", we recommend to gradually replace every occurrence of "agent" / "Agent" with "runner" / "Runner". {% endhint %} > For full language reference, see: [Contract Language reference](/reference/contract-language-reference) > > For supported data source configurations, see: [Data source reference for Soda Core](/reference/data-source-reference-for-soda-core) *** ## Installation {% content-ref url="/pages/xbHoaBtcgajPRYZQ1UoX" %} [Soda Python Libraries](/deployment-options/soda-python-libraries) {% endcontent-ref %} Make sure you install the following packages: * `soda` * `soda-`: the package relevant to the data source(s) you will use. E.g. `soda-postgres`, or `soda-bigquery`. > Learn more about [supported data sources](/reference/data-source-reference-for-soda-core). *** ## Connect to Soda Cloud The connection to Soda Cloud is made via a config file (`sc_config.yml`), used to create and test your Soda Cloud configuration. This is required for publishing contracts or pushing verification results. > Don’t have an account? [Book a demo](https://soda.io/demo?utm_source=documentation\&utm_medium=cloud_docs\&utm_campaign=demo_request\&utm_content=connect_soda_cloud) to get started. ### Create Soda Cloud config To connect the CLI or Python API to Soda Cloud, **you must provide an `sc_config.yml` file for authentication**, which will be necessary to run scans that send results to Soda Cloud, to publish or verify contracts via CLI, and to use the Python API. ```bash soda cloud create -f sc_config.yml ``` #### Parameters
ParameterRequiredDescription
--file, -fYesPath to a Soda Cloud YAML configuration file.
--verbose, -vNoDisplay detailed logs during execution.
The above command will create the `sc_config.yml` file, which will look like the following: {% code title="sc\_config.yml" %} ```yaml soda_cloud: host: cloud.soda.io # Use cloud.us.soda.io for US region api_key_id: YOUR_API_KEY_ID api_key_secret: YOUR_API_KEY_SECRET ``` {% endcode %} **Required fields** | Field | Required | Description | | ---------------- | -------- | -------------------------------------------------------------- | | `host` | Yes | Soda Cloud region endpoint | | `api_key_id` | Yes | [Generated in Soda Cloud](broken://pages/H6Piqnv5lUFCAv1Cvm75) | | `api_key_secret` | Yes | [Generated in Soda Cloud](broken://pages/H6Piqnv5lUFCAv1Cvm75) | You can reference environment variables instead of hardcoding values: {% code title="sc\_config.yml" %} ```yaml soda_cloud: host: cloud.soda.io api_key_id: ${env.SODA_API_KEY_ID} api_key_secret: ${env.SODA_API_KEY_SECRET} ``` {% endcode %} Set them in your environment: ```bash export SODA_API_KEY_ID=your_key_id export SODA_API_KEY_SECRET=your_key_secret ``` *** {% columns %} {% column %} **CLI usage** Pass the `sc_config.yml` file when running CLI commands: {% code overflow="wrap" %} ```shellscript soda scan -d my_datasource -c configuration.yml --soda-cloud sc_config.yml ``` {% endcode %} The CLI uses this file to authenticate and send results to Soda Cloud. {% endcolumn %} {% column %} **Python API usage** Provide the file path when calling Soda Cloud operations: 
verify_contracts_on_runner(
  dataset_identifiers=["my_dataset"],
  soda_cloud_file_path="sc_config.yml",
)
The Python API reads the file to authenticate and connect to Soda Cloud. {% endcolumn %} {% endcolumns %} {% hint style="warning" %} Without a valid `sc_config.yml`, the CLI and Python API cannot authenticate with Soda Cloud. {% endhint %} *** ### Test connection ```bash soda cloud test -sc sc_config.yml ```
ParameterRequiredDescription
--soda-cloud, -scYesPath to a Soda Cloud YAML configuration file
--verbose, -vNoDisplay detailed logs during execution.
*** ## Configure a data source These commands help you define a local configuration for your data source (used by Soda Core) and validate the connection. ### Create data source config ```bash soda data-source create -f ds_config.yml ``` | Parameter | Required | Description | | --------------- | -------- | ------------------------------------------------------------- | | `--file, -f` | Yes | Output file path for the data source YAML configuration file. | | `--verbose, -v` | No | Display detailed logs during execution. | {% hint style="info" %} By default, the YAML file generated as `ds_config.yml` is a template for **PostgreSQL connections**. To see how to modify it to connect to other data sources, head to [Data source reference for Soda Core](/reference/data-source-reference-for-soda-core). {% endhint %} ### Test data source connection ```bash soda data-source test -ds ds_config.yml ``` | Parameter | Required | Description | | -------------------- | -------- | ---------------------------------------------- | | `--data-source, -ds` | Yes | Path to a data source YAML configuration file. | | `--verbose, -v` | No | Display detailed logs during execution. | *** ## Create a contract {% hint style="warning" %} Creating a contract requires having installed the `soda` package using **the** [**Private PyPi installation flow**](/deployment-options/soda-python-libraries#private-pypi-installation-flow) available both for Team and Entreprise license, unless you are using a Runner. Need access to the private PyPI? Please [contact us](mailto:support@soda.io). {% endhint %} Bootstraps a contract file from an existing dataset's schema. Runs either locally or on a Soda Runner; output is written to a local file, to Soda Cloud as a draft, or both. **Local execution:** {% code overflow="wrap" %} ```bash soda contract create --dataset datasource/db/schema/table --data-source ds_config.yml --file contract.yaml ``` {% endcode %} **Runner execution:** {% code overflow="wrap" %} ```bash soda contract create --dataset datasource/db/schema/table --soda-cloud sc_config.yml --file contract.yaml --use-runner ``` {% endcode %} | Parameter | Required | Description | | ---------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------- | | `--dataset, -d` | Yes | Fully qualified dataset name (data\_source\_name/database\_name/schema\_name/table\_name) aka Soda Cloud dataset identifier. | | `--file, -f` | Conditional | Path to a contract YAML file to be created. Directories will be created if needed. Required unless `--soda-cloud` is set. | | `--data-source, -ds` | Conditional | Path to a data source YAML config file. Required for local execution; ignored when `--use-runner` is set. | | `--soda-cloud, -sc` | Conditional | Path to Soda Cloud YAML config file. Required when `--use-runner` is set, or when `--file` is omitted. | | `--use-runner, -a` | No | Run introspection on Soda Runner instead of locally. Default: `false`. | | `--no-type-parameters, -ntp` | No | Omit data type parameters (precision, scale, length) from generated columns. | | `--verbose, -v` | No | Display detailed logs during execution. | {% if visitor.claims.plan === %} > For a full walkthrough including what the generated skeleton contains and common errors, see [Broken mention](broken://pages/TqEASvClRiNF5wnQVu6e). > {% endif %} *** ## Test a contract (dry run) Checks that a contract is syntactically valid before publishing or running a verification. ```bash soda contract test --contract contract.yaml ``` | Parameter | Required | Description | | ---------------- | -------- | --------------------------------------- | | `--contract, -c` | Yes | Path to a contract YAML file | | `--verbose, -v` | No | Display detailed logs during execution. | ## Publish a contract {% hint style="warning" %} Publishing a new version of a contract to Soda Cloud requires additional permissions in Soda Cloud. * If you are publishing a new version for an **existing dataset**, you must have the **Manage contract** permission for that dataset. * If you are publishing a **new dataset**, you must have the global permission **Create new datasets and data sources with Soda Library**. Learn more about permissions in Soda Cloud: [Global and Dataset Roles](broken://pages/n914KXqJNBoZ19Ay5W0P). {% endhint %} Publishes a local contract to Soda Cloud, making it the source of truth for verification. ```bash soda contract publish --contract contract.yaml --soda-cloud sc_config.yml ``` | Parameter | Required | Description | | ------------------- | -------- | --------------------------------------- | | `--contract, -c` | Yes | Path to a contract YAML file. | | `--soda-cloud, -sc` | Yes | Path to Soda Cloud YAML config file. | | `--verbose, -v` | No | Display detailed logs during execution. | This action requires the "Manage contract" permission on the dataset; the user is identified based on the API key provided in the Soda Cloud configuration. {% if visitor.claims.plan === %} > Learn more about permissions here: [Dataset Attributes & Responsibilities](/dataset-attributes-and-responsibilities) > {% endif %} ## Fetch a contract from Soda Cloud {% hint style="warning" %} To fetch a contract from Soda Cloud, you must have the **View** permission for that dataset. Learn more about permissions in Soda Cloud [Broken mention](broken://pages/n914KXqJNBoZ19Ay5W0P) {% endhint %} You can fetch a contract from Soda Cloud to output it in a local file. ```bash soda contract fetch --dataset datasource/db/schema/table --file contract.yaml --soda-cloud sc_config.yml ``` | Parameter | Required | Description | | ------------------- | -------- | ----------------------------------------------------- | | `--dataset, -d` | Yes | Soda Cloud dataset identifier. | | `--file, -f` | Yes | The path to a contract YAML file to update or create. | | `--soda-cloud, -sc` | Yes | Path to Soda Cloud YAML config file. | | `--verbose, -v` | no | Display detailed logs during execution. | ## Verify a contract {% hint style="warning" %} Publishing verification results to Soda Cloud with `--publish` requires additional permissions in Soda Cloud. * If you are publishing to an **existing dataset**, you must have the **Manage contract** permission for that dataset. * If you are publishing a **new dataset**, you must have the global permission **Create new datasets and data sources with Soda Library**. Learn more about permissions in Soda Cloud [Broken mention](broken://pages/n914KXqJNBoZ19Ay5W0P) {% endhint %} Executes a contract verification to check if the dataset complies with its expectations. You can verify a local contract file or a Soda Cloud contract either locally (in your Python environment) or remotely with a Soda Runner. ```bash soda contract verify --data-source ds_config.yml --contract contract.yaml ```
ParameterRequiredDescription
--use-runner, -aNoUse Soda Runner for execution
--publishNoPublish results and contract to Soda Cloud. This action requires the "Manage contract" permission on the dataset; the user is identified based on the API key provided in the Soda Cloud configuration. Learn more about permissions here: /pages/TGapvVUqrfNHhnb53GBI
--data-source, -dswithout --use-runnerPath to a data source YAML config file
--contract, -cwithout --use-runnerPath to the contract YAML file
--soda-cloud, -scwith --use-runner or with --publishPath to a Soda Cloud YAML config file
--dataset, -dwith --use-runnerSoda Cloud dataset identifier
--setNoOverride contract variables at runtime (can be used multiple times)
--check-filter, -cfNoFilter checks by field. Format: key=value. Can be repeated. Supported keys: type, name, column, relative_path (alias path), check_path, qualifier, attributes.<key>. Wildcards (*, ?) supported. See #run-a-subset-of-checks
--check-paths, -cpNoOne or more check paths to run. See #check-paths
--verbose, -vNoDisplay detailed logs during execution
*** ### Override variables Use the `--set` option to define or override variables in the contract when running a verification. ```bash soda contract verify --data-source ds_config.yml --contract contract.yaml --set country=BE --set threshold=5 ``` | Parameter | Required | Description | | --------- | -------- | ------------------------------------------------ | | `--set` | No | Define variable key-value pairs for substitution | *** ## Contract Collaboration Soda enables collaboration on data contracts through **requests** and **proposals**. Data consumers can request changes, propose updates, and iterate with dataset owners until alignment is reached. From the CLI, you can fetch, review, and publish proposals, ensuring contract changes are tracked and versioned in Git. {% if visitor.claims.plan === %} > Read more about [Broken mention](broken://pages/0H6xN2Unwo0hqiAG1BgW) > {% endif %} ### Fetch a proposal Fetches the content of a proposal from Soda Cloud and saves it as a contract file, which can then be published to Git. This allows dataset owners to incorporate approved changes into version-controlled data contracts. ```bash soda request fetch -r 7 -p 1 -sc soda-cloud.yaml --f ./contracts/ecommerce_orders.yaml ``` | Parameter | Required | Description | | ------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-r` | Yes | The request number. Identifies the request to fetch. Request numbers can be found when reviewing a proposal. See screenshot below. | | `-p` | No | The proposal number. Defaults to the latest proposal if not specified. Proposal numbers are shown as the decimal part when reviewing a proposal. See screenshot below. | | `--soda-cloud, -sc` | Yes | Path to the Soda Cloud config file (e.g., `soda-cloud.yaml`). | | `--f` | Yes | Path to the output file where the contract will be written. | ### Push a proposal to Soda Cloud Uploads a contract file to Soda Cloud as a proposal for a specific request. This allows dataset consumers or owners to share updates directly from the CLI and provide context with an accompanying message. ```bash soda request push -sc soda-cloud.yaml -f ./contracts/ecommerce_orders.yaml -r 7 -m "Added new field for order_status" ``` | Parameter | Required | Description | | ------------------- | -------- | -------------------------------------------------------------- | | `-sc, --soda-cloud` | Yes | Path to the Soda Cloud configuration file. | | `-f, --file` | Yes | Path to the contract file to be pushed to Soda Cloud. | | `-r, --request` | Yes | The **Contract Request number** to which the proposal belongs. | | `-m, --message` | No | A descriptive message about the changes being proposed. | *** ### Transition a request Updates the status of a contract request in Soda Cloud. This is useful for marking a request as open, resolved, or closed when no action will be taken. ```bash soda request transition -sc soda-cloud.yaml -r 7 -s done ``` | Parameter | Required | Description | | ------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-sc, --soda-cloud` | Yes | Path to the Soda Cloud configuration file. | | `-r, --request` | Yes | The **Contract Request number** to be transitioned. | | `-s, --status` | Yes |

• open → Reopen or keep the request active
• done → Mark the request as completed.
• wontDo → Mark the request as closed without changes (“won’t do”).

| *** ## Run a subset of checks You can choose to run only specific checks from your contract using **check filters** (`--check-filter`) or **check paths** (`--check-paths`). Both options can be combined. {% hint style="warning" %} Check filters and check paths are **only available when running Soda Core locally**.\ They are **not supported with Soda Runner** and **cannot be used together with the `--use-runner` option**. {% endhint %} ### Check filters Use `--check-filter` (or `-cf`) to select checks by their properties. Each filter is a `key=value` expression. ```bash soda contract verify \ --data-source ds_config.yml \ --contract contract.yaml \ -cf type=missing ``` #### Supported fields | Field | Description | Example | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | `type` | Check type | `type=missing`, `type=schema` | | `name` | Check name (custom or default) | `name=Email format` | | `column` | Column name (column-level checks only) | `column=customer_email` | | `relative_path` | YAML path to the check within its file. `path` is an alias. | `relative_path=columns.id.checks.missing` | | `check_path` | Full check path, including the collection it comes from where the check is not from a contract (see [#check-paths](#check-paths)) | `check_path=data-standard.pii_rules:columns.id.checks.missing` | | `qualifier` | Check qualifier | `qualifier=c2` | | `attributes.` | Custom attribute value (supports [list-valued attributes](#list-valued-attributes)) | `attributes.severity=critical` | > Learn more about check names, qualifiers, and attributes in the [Contract Language reference](/reference/contract-language-reference). {% if visitor.claims.plan === %} Every check belongs to a **check collection** — the file that defines it. A contract is one kind of check collection, a Data Standard is another. When a run covers more than one collection, two more fields select by collection: | Field | Description | Example | | ------------------------ | ------------------------------------------------------------ | ---------------------- | | `collection`, `standard` | Name of the check collection the check comes from | `collection=pii_rules` | | `source` | Kind of check collection: `soda-contract` or `data-standard` | `source=data-standard` | ```bash # Run every check that comes from one collection soda data-standard verify -d datasource/db/schema/customers -ds ds_config.yml -sc sc_config.yml \ -cf collection=pii_rules ``` {% endif %} #### Combining multiple filters Repeat `-cf` to add multiple filters. Filters follow AND/OR grouping logic: * **Same field** (OR): at least one value must match. * **Different fields** (AND): all field groups must match. ```bash # OR within same field — type is missing OR invalid soda contract verify -ds ds_config.yml -c contract.yaml \ -cf type=missing -cf type=invalid # AND across fields — type is missing AND column is id soda contract verify -ds ds_config.yml -c contract.yaml \ -cf type=missing -cf column=id # Combined — (type is missing OR invalid) AND column is id soda contract verify -ds ds_config.yml -c contract.yaml \ -cf type=missing -cf type=invalid -cf column=id # Filter by custom attribute soda contract verify -ds ds_config.yml -c contract.yaml \ -cf attributes.severity=critical ``` #### Wildcards Filter values support wildcards using `*` (matches any sequence of characters) and `?` (matches any single character). Matching is case-sensitive. ```bash # All check types starting with "miss" soda contract verify -ds ds_config.yml -c contract.yaml \ -cf type=miss* # All columns starting with "user_" soda contract verify -ds ds_config.yml -c contract.yaml \ -cf column=user_* ``` #### List-valued attributes When filtering by an attribute that contains a list (e.g., `tags: [prod, critical]`), check filters support two matching modes: **Member match** — use a plain value to match if *any* element in the list matches. Wildcards are supported. ```bash # Matches any check where tags contains "prod" soda contract verify -ds ds_config.yml -c contract.yaml \ -cf attributes.tags=prod # Wildcard member match soda contract verify -ds ds_config.yml -c contract.yaml \ -cf "attributes.tags=prod*" ``` **Full list match** — use `[a,b]` syntax to match only if the list equals that exact set of values. Order does not matter. Wildcards are **not** supported in full list match mode. ```bash # Matches only checks where tags is exactly [prod, critical] (in any order) soda contract verify -ds ds_config.yml -c contract.yaml \ -cf "attributes.tags=[prod,critical]" ``` For list elements that contain commas or spaces, use double quotes inside the brackets: ```bash -cf "attributes.tags=[\"a,b\",hello world,unquoted]" ``` {% hint style="info" %} Values containing shell special characters (`[`, `]`, `*`, `?`) must be quoted on the command line. {% endhint %} *** ### Check paths Use `--check-paths` (or `-cp`) to select checks by their hierarchical location in the contract YAML. A **check path** is the hierarchical location of a check within your YAML contract file. It follows the contract's structure and identifies the exact check(s) to run. ```bash soda contract verify \ --data-source ds_config.yml \ --contract contract.yaml \ --check-paths checks.freshness columns.Order_Date.checks.invalid_format ``` A check path is composed of the keys that define where the check is located in the YAML.\ It can reference: * a **check at the dataset level** (e.g. `checks.freshness`), * a **column-level check** (e.g. `columns.last_updated_on.checks.invalid`) * a check with a qualifier (see [Contract Language reference](/reference/contract-language-reference#check-qualifiers)) (e.g. `columns.last_updated_on.checks.invalid.5bb8b1`or `checks.row_count.5bb8b1` ) . Each segment of the path corresponds to a level in the YAML hierarchy, separated by dots (`.`). **Example**: ```yaml dataset: datasource/schema/dataset checks: # Check path: checks.freshness - freshness: column: last_updated_on threshold: must_be_less_than: 1 unit: day # Check path: checks.row_count - row_count: columns: - name: last_updated_on checks: # Check path: columns.last_updated_on.checks.invalid - invalid: invalid_format: regex: ^(0[1-9]|1[0-2])/(0[1-9]|[12]\d|3[01])/\d{4}$ name: MM/DD/YYYY # Check path: columns.last_updated_on.checks.invalid.5bb8b1 - invalid: qualifier: 5bb8b1 invalid_format: regex: ^(0[1-9]|[12]\d|3[01])/(0[1-9]|1[0-2])/\d{4}$ name: DD/MM/YYYY filter: region = 'EU' ``` **On Soda Cloud**, the check path of a given check can be found on the check page in the check info panel.
{% if visitor.claims.plan === %} #### Check paths outside a contract Everything above describes contract check paths. A dataset can have other **check collections** applied to it at the same time, so a check from one of those is identified by a path that also names the collection it came from: ``` .: ``` Data Standards are the only other kind today, so their checks look like this: ```bash soda data-standard verify -d datasource/db/schema/customers -ds ds_config.yml -sc sc_config.yml \ -cp data-standard.pii_rules:columns.email.checks.missing ``` Because `.` and `:` separate the parts of a check path, a collection's name cannot contain either character. **Contract check paths are unaffected** — they stay exactly as shown above. {% hint style="info" %} Internally, `--check-paths` values become `check_path=` check filters, so `--check-filter check_path=checks.freshness` is equivalent. To match a check's location while ignoring which collection it came from, filter on `relative_path=` instead. {% endhint %} {% endif %} *** ### Combining check filters and check paths You can use `--check-filter` and `--check-paths` together. The path selectors from `--check-paths` are combined with any other check filters using AND logic across different fields. ```bash # Run only the missing check on the id column that has severity=critical soda contract verify -ds ds_config.yml -c contract.yaml \ -cp columns.id.checks.missing \ -cf attributes.severity=critical ``` *** ### Behavior and results The behavior below applies to both `--check-filter` and `--check-paths`.
ScenarioBehavior
One or more valid filters or paths providedOnly matching checks are executed.

Empty --check-paths

e.g. ... --check-paths --verbose ...

Ignored; all checks execute.

No checks match the filters

e.g. ... -cf type=nonexistent

No checks execute.

Partial match

e.g. ... -cf type=freshness -cf type=nonexistent

Only matching checks execute (in this case, only the freshness check).
#### Results in Soda Cloud When using `--check-filter` or `--check-paths`, only the checks matching the selection are executed. * Checks **not matching** the filters are **not executed** and therefore **remain unchanged** in **Soda Cloud**. * Their status, metrics, and results are **not updated;** they are effectively **untouched** during this run. However, in the **Reporting API**, the response will include **all checks defined in the contract**, representing, for each scan, the **outcomes of all checks**.\ Checks that were **skipped** (because they did not match the filters) are still present in the list of check results, but are **marked with** `CheckResultsData.level: excluded`. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Contract Language reference A Soda data contract is a YAML document that contains data quality checks. The checks in a Soda data contract are similar to unit tests for testing software. Verifying a contract means evaluating the checks against the actual data. If contract verification fails, notifications can be sent out or the new data can be stopped from being released to users. This page documents the necessary contract structure and the supported check types, filters, and configuration options available in soda's data contract language. ## Contract YAML document structure ### Example A Soda data contract YAML must include the following blocks: * A top-level `dataset:` key * One of the following: * A `checks:` block, if you’re defining dataset-wide checks * A `columns:` list {% code title="contract.yml" %} ```yaml dataset: datasource/db/schema/dataset #mandatory dataset fully qualified name checks: #dataset level checks - schema: - row_count: columns: #columns list - name: id checks: # column level checks (optional) - missing: - name: name checks: - missing: threshold: metric: percent must_be_less_than: 10 - name: size checks: - invalid: valid_values: ['S', 'M', 'L'] ``` {% endcode %} ### Dataset fully qualified name Every contract must have a dataset fully qualified name that follows the structure below: ```yaml dataset: datasource/db/schema/dataset ``` Casing has to match the casing in the data source. A dataset fully qualified name is composed of 3 parts, all slash-separated: * The **data source name**: This is the name of the data source in which the dataset exists. The data source name is configured in a data source YAML configuration file or in Soda Cloud. * A **list of prefixes**: Prefixes represent the full list of hierarchical elements of the data source, like the **database name** and the **schema name**.\ Postgres, for example, has a database name and a schema name as prefixes. Databricks has catalog and schema as prefixes. Prefix names are also slash-separated. * The **dataset name**: The name of the table or view. ### Columns Every contract must include a list of columns. If a schema check is configured, this list can be used to validate the **presence, data type, and order** of columns. Each column entry includes: * `name`: the column name (required) * `data_type` (optional): used by the schema check * `character_maximum_length` (optional): used by the schema check to verify the maximum character length of a string column (e.g. `VARCHAR(100)`) * `numeric_precision` (optional): used by the schema check to verify the precision (total number of digits) of a numeric column (e.g. `NUMERIC(10,2)` has precision 10) * `numeric_scale` (optional): used by the schema check to verify the scale (digits after the decimal point) of a numeric column (e.g. `NUMERIC(10,2)` has scale 2) * `datetime_precision` (optional): used by the schema check to verify the fractional seconds precision of a datetime column (e.g. `TIMESTAMP(6)` has precision 6) * `checks` (optional): a list of checks that apply only to that column * `type` * Threshold keys (for any check that *needs* a threshold, such as `must_be`, `must_be_between`, etc.) Example: {% code title="contract.yml" %} ```yaml # Example of fully operational contract with only column-level checks dataset: datasource/db/schema/dataset columns: - name: id data_type: varchar checks: # Column-level checks definition - missing: - duplicate: - name: name - name: code ``` {% endcode %} {% hint style="warning" %} A contract that exclusively defines **dataset-level checks** that do not run against any specific column **still requires the `columns` list**. In this case, providing an empty list is enough. {% endhint %} 
# Example of fully operational contract with only dataset-level checks

dataset: datasource/db/schema/dataset

checks:
  - schema:
  - row_count:
  
columns: []
### Checks The `checks` section defines the quality rules and expectations for your dataset. Checks can be written at two levels: * **Dataset-level**: Rules that apply to the dataset as a whole (e.g., row count, schema, freshness, duplicates across multiple columns). * **Column-level**: Rules that apply to a specific column (e.g., missing, invalid, duplicate on a single column). Each check entry includes: * `type` * `column:` (not required to compile) * Threshold keys (for any check that *needs* a threshold, such as `must_be`, `must_be_between`, etc.) **Example of a dataset-level `checks` block:** ```yaml checks: - schema: - row_count: ``` *** **Example of column-level `checks` block:** 
columns:
  - name: email
    checks:
      - missing:
      - invalid:
          invalid_format:
            name: Email format
            regex: ^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$
Each check defines what “good data” looks like, ensuring that your dataset meets the expected standards. You can customize each check by adding filters, thresholds, or variable references depending on your specific use case. Learn more about the different check types on this page. *** ## Schema check A Schema check verifies that the structure of a dataset matches the expectations defined in the contract—such as required columns, data types, and optional constraints. It ensures that changes to the dataset schema (like missing columns, unexpected data types, or reordered fields) are detected early, helping maintain stability and trust in your data. **Example:** 
dataset: datasource/db/schema/dataset

columns:
  - name: id
  - name: last_name
  - name: address_line1

checks:
  - schema:
      allow_extra_columns: false
      allow_other_column_order: false
**Example with data type parameters:** You can further constrain a column's type by declaring `character_maximum_length`, `numeric_precision`, `numeric_scale`, or `datetime_precision`. The schema check fails if the data source reports a different value. 
dataset: datasource/db/schema/dataset

columns:
  - name: label
    data_type: varchar
    character_maximum_length: 100
  - name: total_amount
    data_type: numeric
    numeric_precision: 10
    numeric_scale: 2
  - name: created_at
    data_type: timestamp
    datetime_precision: 6

checks:
  - schema:
**Column keys used by the schema check:** The schema check uses the columns as specified in the contract and compares them with the measured dataset columns as as the expected columns.
KeyDescriptionOptional
nameThe name of the column. The name is case sensitive.No
data_typeThe data type of the column as is returned by the metadata from the data source. Data types are compared case insensitive.Yes
character_maximum_lengthThe character maximum length. This represents the maximum length for data types like VARCHAR(255).Yes
numeric_precisionThe numeric precision (total number of digits). For example, NUMERIC(10,2) has precision 10.Yes
numeric_scaleThe numeric scale (number of digits after the decimal point). For example, NUMERIC(10,2) has scale 2.Yes
datetime_precisionThe fractional seconds precision for a datetime column. For example, TIMESTAMP(6) has precision 6.Yes
**Check configuration keys:**
KeyDescriptionOptional
allow_extra_columnsSpecifies if columns other than the ones listed in the contract are allowed in the dataset. Values are true or false. Default is false, which means that all columns in the dataset must be specified in the contract.Yes
allow_other_column_orderSpecifies if column ordering must be exact the same as in the contract YAML document. Values are true or false. Default is false, which means that columns must appear in the order as in the contract.Yes
name#check-namesYes
qualifier#check-qualifiersYes
attributes#check-attributesYes
*** ## Row count check A Row Count check verifies that the dataset contains the expected number of rows. It ensures that the data source has at least a minimum number of records and no unexpected gaps. This is a fundamental check to confirm the presence and completeness of data in a dataset. **Example**: 
dataset: datasource/db/schema/dataset

checks:
  - row_count:

columns: []
**Check configuration keys**
KeyDescriptionOptional
name#check-namesYes
thresholdThe default row count threshold verifies there is at least 1 row present. #thresholdsYes
filter#configure-a-check-filterYes
qualifier#check-qualifiersYes
attributes#check-attributesYes
*** ## Freshness check A freshness check verifies if data is not too old by measuring the period between the current time and the most recent timestamp present in a given column. *** **Example:** 
dataset: datasource/db/schema/dataset

checks:
  - freshness:
      column: created_at
      threshold:
        unit: hour
        must_be_less_than: 24

columns: []
**Check configuration keys**
KeyDescriptionOptional
columnSpecifies a timestamp column that represents the time of the row orNo
name#check-namesYes
threshold

It defines the acceptable data age limit. It sets the maximum time allowed between the current time and the newest timestamp found in the target column.

  • unit : The time unit used for the threshold, e.g., minute, hour, day.
  • #thresholds
No
filter#configure-a-check-filterYes
qualifier#check-qualifiersYes
attributes#check-attributesYes
*** ## Missing check A Missing check verifies that a specific column does not contain null or empty values beyond an acceptable threshold. It ensures that critical fields are populated and helps catch incomplete or corrupted data that could impact downstream processes. **Example:** 
dataset: datasource/db/schema/dataset

columns:
  - name: id
    checks:
      - missing:
*** **Check configuration keys**
KeyDescriptionOptional
missing_valuesThe list of values is considered missing. NULL is always considered a missing value and doesn't need to be includedNo*
missing_formatA SQL regex that matches with missing values. (Advanced)No*
name#check-namesYes
thresholdThe default missing check threshold verifies there are no missing values in the column. #thresholdsYes
filter#configure-a-check-filterYes
qualifier#check-qualifiersYes
attributes#check-attributesYes
\* there are several configuration keys to configure the missing values. At least one missing configuration is required. *** **Configure extra missing values in a list** Configure a list of values that, apart from NULL, must also be considered as missing values. Typical examples are `'-'`, `'No value'`, `'N/A'`, `'None'`, `'null'`, `-1`, `999` 
dataset: datasource/db/schema/dataset

columns:
  - name: id
    checks:
      - missing:
          missing_values: ['N/A', '-']
{% hint style="info" %} The `missing_values` configuration is only supported on **string** and **numeric** data types. {% endhint %} *** **Configure extra missing values with a regular expression** Configure a SQL regular expression that, apart from NULL, also matches values that are considered as missing values. Only supported for columns having a text data type 
dataset: datasource/db/schema/dataset

columns:
  - name: id
    checks:
      - missing:
          missing_format:
            name: All dashes
            regex: ^[-]+$
The `regex` syntax must match the SQL engine of the data source. The `name` is a human-readable description of what the regex does. It helps explain the purpose of the pattern, making it easier for others to understand the intent of the check. *** ## Invalid check An Invalid check verifies that the values in a column conform to a set of allowed formats, values, or constraints. It helps catch data that does not meet the expected standards, such as incorrect formats, out-of-range values, or entries that violate business rules. **Example with valid values**: 
dataset: datasource/db/schema/dataset

columns:
  - name: size
    checks:
      - invalid:
          valid_values: ['S', 'M', 'L']
**Check configuration keys**
KeyDescriptionOptional
valid_valuesA list of valid valuesNo*
valid_formatA SQL regular expression that matches with valid valuesNo*
valid_reference_dataReferences a dataset and column that contains valid valuesNo*
valid_minValid values must be greater or equal than the configured valueNo*
valid_maxValid values must be less or equal than the configured valueNo*
valid_lengthThe fixed length of valid valuesNo*
valid_min_lengthThe minimum length of valid valuesNo*
valid_max_lengthThe maximum length of valid valuesNo*
invalid_valuesA list of invalid valuesNo*
invalid_formatA SQL regular expression that matches with invalid valuesNo*
name#check-namesYes
thresholdThe default invalid check threshold ensures there must not be any invalid values. #thresholdsYes
filter#configure-a-check-filterYes
qualifier#check-qualifiersYes
attributes#check-attributesYes
\* there are several configuration keys to configure the invalid values. At least one validity configuration is required. Multiple validity configurations can be combined. **Example with a regular expression** Format is only supported for string data type. 
dataset: datasource/db/schema/dataset

columns:
  - name: code
    checks:
      - invalid:
          valid_format:
            name: Alpha Code
            regex: \^[A-Z]{3}$\
The `regex` syntax must match the SQL engine of the data source. The `name` is a human-readable description of what the regex does. It helps explain the purpose of the pattern, making it easier for others to understand the intent of the check. *** **Example minimum and maximum values:** 
dataset: datasource/db/schema/dataset

columns:
  - name: code
    checks:
      - invalid:
          valid_min: 1
          valid_max: 100
`valid_min` and `valid_max` can be used independently. It's supported on columns with numerical data types. *** **Example minimum and maximum length:** 
dataset: datasource/db/schema/dataset

columns:
  - name: code
    checks:
      - invalid:
          valid_length: 2
          valid_min_length: 1
          valid_max_length: 5
`valid_length` , `valid_min_length` and `valid_max_length` can be used independently. *** #### **Example with a reference dataset** 
dataset: datasource/db/schema/dataset

columns:
  - name: code
    checks:
      - invalid:
          valid_reference_data:
            dataset: data_source/db/schema/reference_dataset
            column: valid_code
`valid_reference_data` has 2 nested configurations: * `dataset`: fully qualified dataset name. Limitation: The reference dataset must be in the same data source. * `column`: the name of the column containing the valid values Commonly referred to as a reference check or referential integrity check. *** ## Duplicate check A Duplicate check ensures that values in a column or combination of columns are unique, helping prevent data integrity issues. It can be used at the **column level** or **dataset level**. ### Single column duplicate check 
dataset: datasource/db/schema/dataset

columns:
  - name: id
    checks:
      - duplicate:
**Configuration keys:**
KeyDescriptionOptional
name#check-namesYes
thresholdThe default duplicate check threshold requires that all values are unique. #thresholdsYes
filterCheck filterYes
qualifier#check-qualifiersYes
attributes#check-attributesYes
### Multi-column duplicate check 
dataset: datasource/db/schema/dataset

checks:
  - duplicate:
      columns: ['col1', 'col2']
      
columns: []
**Configuration keys:**
KeyDescriptionOptional
columnsList of column namesNo
name#check-namesYes
thresholdThe default duplicate check threshold requires that all values are unique. Data Contract Language Reference ✅Yes
filter#configure-a-check-filterYes
qualifier#check-qualifiersYes
attributes#check-attributesYes
*** ## Aggregate check An Aggregate check verifies that the result of an aggregate function, such as `avg`, `sum`, `min`, `max`, or `count`—meets the expected thresholds. It helps ensure that summary statistics over a column remain within acceptable ranges and that key metrics derived from your data stay accurate and consistent. **Example:** 
dataset: datasource/db/schema/dataset

columns:
  - name: age
    checks:
      - aggregate:
          function: avg
          threshold:
            must_be_between:
              greater_than: 20
              less_than: 50
**Check configuration keys**
KeyDescriptionOptional
functionSupported by all data sources: avg, avg_length, max, min, max_length, min_length, sumNo
name#check-namesYes
threshold#thresholdsNo
filter#configure-a-check-filterYes
qualifier#check-qualifiersYes
attributes#check-attributesYes
*** ## Metric check A Metric check validates the result of a custom SQL expression or query against a threshold. It supports complex business logic, calculated metrics, or cross-column relationships. Metric checks can be defined at the dataset level (across multiple columns) or column level (single column). For better performance, we recommend SQL expressions over full SQL queries—they’re simpler to write and can be combined with other checks in a single query. ### Verify a SQL expression value against a threshold **Example at the dataset level:** 
dataset: datasource/db/schema/dataset

checks:
  - metric:
      expression: AVG("end" - "start")
      threshold:
        must_be_between:
          greater_than: 3
          less_than: 8

columns: []
**Example at the column level:** 
dataset: datasource/db/schema/dataset

columns:
  - name: end
    checks:
      - metric:
          expression: AVG("end" - "start")
          threshold:
            must_be_between:
              greater_than: 3
              less_than: 8
*** **Check configuration keys**
KeyDescriptionOptional
expressionA SQL expression that produces the numeric metric value that will be compared with the threshold.No
name#check-namesYes
threshold#thresholdsNo
qualifier#check-qualifiersYes
attributes#check-attributesYes
### Verify a SQL query value against a threshold **Example at the dataset level:** 
dataset: datasource/db/schema/dataset

checks:
  - metric:
      query: |
        SELECT AVG("end" - "start")
        FROM datasource.db.schema.table
      threshold:
        must_be_greater_than: 3

columns: []
**Example at the column level:** 
dataset: datasource/db/schema/dataset

columns:
  - name: end
    checks:
      - metric:
          query: |
            SELECT AVG("end" - "start")
            FROM datasource.db.schema.table
          threshold:
            must_be_greater_than: 3
KeyDescriptionOptional
queryA SQL query that produces a single numeric metric value that will be compared with the threshold.No
name#check-namesYes
threshold#thresholdsNo
qualifier#check-qualifiersYes
attributes#check-attributesYes
*** ## Failed rows check A Failed Rows check identifies rows that violate a specific condition, such as a filter or SQL expression. It helps pinpoint problematic data like outliers or rule violations and can save references for further review. Failed Rows checks can apply at the dataset level (testing conditions across multiple columns) or at the column level (validating individual column values). We recommend using SQL expressions over full SQL queries for simplicity and efficiency—they allow combining multiple checks into a single query for better performance. ### Verify failed rows with a SQL expression **Example at the dataset level:** 
dataset: datasource/db/schema/dataset

checks:
  - failed_rows:
      expression: ("end" - "start") > 5
      
columns: []
**Example at the column level:** 
dataset: datasource/db/schema/dataset

columns:
  - name: end
    data_type: DATE
    checks:
      - failed_rows:
          expression: ("end" - "start") > 0
**Check configuration keys**
KeyDescriptionOptional
ExpressionA SQL expression, used as the WHERE clause, that produces any tabular data indicating failures with the data. An expression returning zero rows indicates there is no problem with the data.No
name#check-namesYes
thresholdThe default is that there should be no failed rows.#thresholdsYes
qualifier#check-qualifiersYes
attributes#check-attributesYes
### Verify and visualize failed rows with a SQL query **Example at the dataset level:** 
dataset: datasource/db/schema/dataset

checks:
  - failed_rows:
      query: |
        SELECT * FROM datasource.db.schema.table WHERE ("end" - "start") > 5
        
columns: []
**Example at the column level:** 
dataset: datasource/db/schema/dataset

columns:
  - name: end
    data_type: DATE
    checks:
      - failed_rows:
        query: |
          SELECT * FROM datasource.db.schema.table WHERE ("end" - "start") > 0
**Check configuration keys**
KeyDescriptionOptional
queryA SQL query that produces any tabular data indicating failures with the data. An expression returning zero rows indicates there is no problem with the data.No
rows_tested_queryA SQL query that returns a single numeric value representing the total number of rows tested by the check. When provided, Soda stores the result as check_rows_tested on the check result.Yes
name#check-namesYes
thresholdThe default is that there should be no failed rows. The threshold can only be count-based for non-licensed organizations. #thresholdsYes
qualifier#check-qualifiersYes
attributes#check-attributesYes
#### Report the percentage of failing rows {% if !(visitor.claims.plan === %} Soda can also report the percentage of failing rows on a `failed_rows` check by computing total rows tested at scan time, so you can express thresholds like "fewer than 1% of rows may fail" instead of absolute counts. This requires a Teams or Enterprise license, and running checks via Soda Runner. {% endif %} {% if (visitor.claims.plan === %} Configure `rows_tested_query` alongside `query` in a Data Contract to calculate the total number of rows that were tested.
This allows you to see percentage of failing rows in a given check:
The `rows_tested_query` must return a single numeric value representing the total number of tested rows. **Example at the dataset level:** 
dataset: datasource/db/schema/retail_orders

checks:
  - failed_rows:
      name: Quantity must be positive
      query: |-
        SELECT *
        FROM retail_orders
        WHERE quantity <= 0
      rows_tested_query: |-
        SELECT COUNT(*)
        FROM retail_orders
      threshold:
        metric: percent
        must_be_less_than: 1

columns: []
{% hint style="warning" %} `rows_tested_query` on failed rows checks requires `soda-runner 1.4.7` or later. Earlier runner versions will ignore the key. {% endhint %} {% endif %} *** ## Reconciliation checks {% if !(visitor.claims.plan === %} Reconciliation checks validate that a **target dataset** matches one or more **source datasets**, ensuring that data remains consistent after migrations, in pipelines, or during synchronizations. They can be used for both **metric-level (aggregate) validation** and **row-level (record-by-record) validation**. {% endif %} {% if (visitor.claims.plan === %} > Documentation for reconciliation checks is available in [Broken mention](broken://pages/QkO7w20yeaFP3U9Ow5Kk) > {% endif %} ## Group By check A `group by` check allows you to run validations on **aggregated metrics per group of records** based on one or more fields. You can then apply thresholds to metrics like counts, sums, or averages for each group. {% hint style="warning" %} Group By checks require having installed **the `soda-groupby` package using the** [**private PyPI**](#private-pypi-installation-flow) (Team or Enterprise license), unless you are using a runner. Need access to the private PyPI? Please [contact us](mailto:support@soda.io). {% endhint %} {% if visitor.claims.plan === %} > **Follow the** [**private PyPi installation flow**](/deployment-options/soda-python-libraries#private-pypi-installation-flow) to set up your environment and install the necessary Soda extensions. ### Verify a SQL query value against a threshold **Example at the dataset level:** 
dataset: datasource/db/schema/dataset

checks:
  - group_by:
      name: ""
      query: |-
        SELECT
            warehouse_id,
            product_id,
            COUNT(*) AS record_count,
            SUM(qty_on_hand) AS total_qty_on_hand,
        FROM soda_demo.retail_stock_levels
        GROUP BY warehouse_id, product_id
      fields:
        - warehouse_id
        - product_id
      metric_name: total_qty_on_hand
      group_tested_rows: <tested_rows_metric_alias>
      threshold:
        metric: percent
        must_be_less_than: 10
    
columns: []
**Check configuration keys**
KeyDescriptionOptional
querySQL query returning grouping fields and metricsNo
fieldsList of column(s) used to group resultsNo
metric_nameMetric (from query alias) used for threshold evaluation
name#check-namesYes
thresholdThe threshold to compare each of the group metrics with.#thresholdsNo
qualifier#check-qualifiersYes
attributes#check-attributesYes
group_tested_rowsQuery alias containing the number of rows tested per groupYes
#### Percentage thresholds for Group By checks To use a percentage threshold on a `group_by` check, the query must return both: * the metric value to evaluate, referenced by `metric_name` * the number of rows tested for each group, referenced by `group_tested_rows` Soda evaluates the percentage threshold independently for each group. `group_tested_rows` is required when using `threshold.metric: percent`. {% endif %} *** ## Common check configurations ### Check filter Most checks support a `filter` to limit the rows on which the check applies: 
dataset: datasource/db/schema/dataset

columns:
  - name: id
    checks:
      - missing:
          filter: "country = 'USA'"
Note: If a dataset filter is configured, both the dataset and the check filter will be applied. ### Column expression **All column-level checks** (except `failed_rows` check type) as well as the dataset-level **Freshness check,** **support a `column_expression` configuration** that allows you to modify the column value before the check is evaluated. You can optionally define: ```yaml column_expression: ``` This SQL expression: * Uses the SQL dialect of your connected data source. * Is applied during query generation. * Modifies the column value before the check is evaluated Soda then runs the check against the result of that expression instead of the raw column. {% hint style="info" %} Performance will depend on the complexity of the SQL expression. All expressions must be valid in the underlying warehouse dialect; **Soda does not validate SQL syntax before execution**. {% endhint %} You can use `column_expression` to: * Access and validate **nested properties** in structured columns (for example, JSON, STRUCT, or OBJECT fields). * **Cast data types** before applying checks (e.g., convert a string to a date for freshness validation). * Reuse standard Soda checks (freshness, missing, validity, etc.) without writing custom SQL. #### Define a column expression at the column level You can define `column_expression` at the column level so that it applies to all checks on that column. 
columns:
  - name: column_name
    column_expression: <SQL expression>
    checks:
      - missing:
      - duplicate:
      - invalid:
          valid_min: 0
#### Override at check level You can also define a `column_expression` inside an individual check: 
columns:
  - name: employee
    checks:
      - invalid:
          valid_min: 100
          column_expression: <SQL expression>
If both column-level and check-level `column_expression` are defined, the check-level expression overrides the column-level expression. #### Use Cases
Validate nested fields in structured columns If a column "employee" contains structured data such as JSON or STRUCT: ```yaml { "data": { "id": 1234, "value": 250 } } ``` You can validate nested properties directly using standard checks. **Example (Postgres syntax)** ```yaml columns: - name: employee data_type: struct column_expression: employee.data->>'id' checks: - invalid: valid_min: 1000 column_expression: employee.data->>'id' - invalid: valid_min: 200 column_expression: employee.data->>'value' ``` Soda applies built-in validation logic to the extracted nested value. In this example, `employee.data->>'id'` is evaluated before applying the `invalid` check.
Cast data types for compatibility with checks Some checks require specific data types. For example: * `freshness` requires a date/timestamp * `valid format` (regex) works on strings * Numeric validations require numeric types If your column type does not match the check’s requirement, you can cast it. **Example: Cast string to date for freshness (Postgres syntax)** ```yaml checks: - freshness: column: hire_date column_expression: hire_date::date threshold: must_be_less_than: "7 days" ``` If `hire_date` is stored as a string, casting it to `date` using `::date` allows freshness evaluation, since the check requires a valid date-formatted value. **Example: Cast numeric to string for regex validation (Postgres syntax)** ```yaml columns: - name: employee_id type: checks: - invalid: valid_format: "^[A-Z]{2}[0-9]{6}$" column_expression: employee_id::text ``` This enables regex validation on numeric IDs, since in some data sources, regular expressions can only be executed on text-type columns.
{% hint style="warning" %} **Limitations & considerations** * Expressions must use the **data source dialect** (Snowflake, BigQuery, Databricks, etc.). * Soda does not normalize SQL syntax across data sources. * `column_expression` is not supported for: * `failed_rows` * `row_count` * `duplicate` (dataset level) * `group_by` * `schema` * Custom metric checks {% endhint %} ### Thresholds Every check (except the schema check) has a metric value that is evaluated and produces a check outcome. The threshold is the check part that specifies which numeric values make the check pass or fail. An example threshold is: "The missing count metric value must be less than 5." 
dataset: datasource/db/schema/dataset

columns:
  - name: status
    checks:
      - missing:
          threshold:
            must_be_less_than: 0
Use one of the following threshold configuration keys to specify an open range. | Key | Description | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `must_be` | The metric value must be equal to the configured value | | `must_not_be` | The metric value must be different from the configured value | | `must_be_greater_than` | The metric value must be greater than the configured value | | `must_be_greater_than_or_equal` | The metric value must be greater than or equal to the configured value | | `must_be_less_than` | The metric value must be less than the configured value | | `must_be_less_than_or_equal` | The metric value must be less than or equal to the configured value | | `must_be_between` | The metric value must be between the acceptable bounds for a metric. The bounds must be expressed as either a **strict** range (`greater_than` / `less_than`) or an **inclusive** range (`greater_than_or_equal` / `less_than_or_equal`). The bounds are specified as **nested properties** under `must_be_between`. | | `must_be_not_between` | The metric value must not be between the acceptable bounds for a metric. The bounds must be expressed as either a **strict** range (`greater_than` / `less_than`) or an **inclusive** range (`greater_than_or_equal` / `less_than_or_equal`). The bounds are specified as **nested properties** under `must_not_be_between`. | Use a closed range to ensure metric values are between 2 boundaries. **Example:** 
dataset: datasource/db/schema/dataset

columns:
  - name: success_ratio
    checks:
      - missing:
          threshold:
            must_be_between:
              greater_than: 0
              less_than: 100
* Use `greater_than` or `greater_than_or_equal` to specify the lower bound * Use `less_than` or `less_than_or_equal` to specify the upper bound Use an open range to ensure metric values are outside 2 boundaries. **Example:** 
dataset: datasource/db/schema/dataset

columns:
  - name: success_ratio
    checks:
      - missing:
          threshold:
            must_be_not_between:
              less_than: 0
              greater_than: 100
* Use `less_than` or `less_than_or_equal` to specify the lower bound * Use `greater_than` or `greater_than_or_equal` to specify the upper bound **Set a threshold as a percentage** To specify a threshold as a percentage of the total, add `metric: percent` to the threshold. The `metric` The property has 2 possible values: `count` is the default and `percentage` can be configured. Percentage-based thresholds are only available on checks where the metric value can be expressed as a percentage of a total value. These are: * Missing checks * Invalid checks * Duplicate checks * Failed rows query checks (only when `rows_tested_query` is configured on the check) Note that the total value refers to the total number of rows checked. In the case of a dataset filter or a check filter, the total refers to the number of rows passing the filters. **Example:** 
dataset: datasource/db/schema/table

columns:
  - name: status
    checks:
      - missing:
          threshold:
            metric: percent
            must_be_less_than: 5
#### Set a threshold level To change the default "fail" threshold level, add `level: warn` to the threshold. This will cause a given check to emit warning instead of check failure. **Example:** 
dataset: datasource/db/schema/dataset

columns:
  - name: status
    checks:
      - missing:
          threshold:
            level: warn
### Check names The `name` property provides a human-readable description for a check. It helps explain the purpose of the check in plain language, making it easier for users to understand the intent behind each rule, especially when reviewing results in Soda Cloud. For example, instead of relying on technical details like a regex pattern or threshold, a well-written `name` offers a simple summary, such as: 
dataset: datasource/db/schema/dataset

columns:
  - name: status
    checks:
      - missing:
          name:  The missing count metric value must be less than 5%
          threshold:
            metric: percent
            must_be_less_than: 5
In this example, the `name` ("Email format") clarifies the purpose of the regex. This is particularly helpful for non-technical users reviewing the contract or verification results. **Best Practices for Check Names:** * Write check names in plain language. * Focus on the intent or business rule the check enforces. * Avoid repeating technical details already visible in the check configuration. The `name` Property is optional but highly recommended for improving clarity and collaboration. By default, the check name is the check type. ### Check Attributes Use attributes to **label**, **sort**, and **route** your checks in Soda Cloud. Attributes help you organize checks by properties such as domain, priority, location, and sensitivity (e.g., PII). > Learn how to leverage attributes with [Broken mention](broken://pages/GAK0Cdoq2hZmucVmrovV) and [Broken mention](broken://pages/wfc7eM74x6b5WmYmxh3p). **Apply Attributes to Checks** You can add attributes directly to individual checks. For example: 
dataset: datasource/db/schema/dataset

columns:
  - name: customer_email
    data_type: VARCHAR
    checks:
      - missing:
          attributes:
            domain: Sales
            pii: True
            dimension: completeness
Attributes can also hold **list values**: 
columns:
  - name: customer_email
    checks:
      - missing:
          attributes:
            tags: [prod, critical]
            domain: Sales
**Set Default Attributes at the Top Level** You can also define default attributes at the dataset level. These attributes apply to **all checks**, unless overridden at the individual check level. 
dataset: datasource/db/schema/dataset

check_attributes:
  domain: Sales
  
columns:
  - name: customer_email
    data_type: VARCHAR
    checks:
      - missing:
          attributes:
            pii: True
            dimension: completeness
#### **Filter checks by attributes at runtime** Attributes can also be used to select which checks to run during contract verification using the `--check-filter` CLI option. For example, `-cf attributes.severity=critical` runs only checks with `severity: critical`. For list-valued attributes (e.g., `tags: [prod, critical]`), use `-cf attributes.tags=prod` to match any check where the list contains `prod`, or `-cf "attributes.tags=[prod,critical]"` to match the exact list. > Learn more in the [CLI reference](/reference/cli-reference#check-filters). #### **Attribute Validation in Soda Cloud** When publishing contract results to Soda Cloud, **all check attributes must be pre-defined in Soda Cloud**. If any attribute used in a contract is not registered in your Soda Cloud environment, the results will **not be published**, and the data contract scan will be **marked as failed**. > Learn how to configure attributes in Soda Cloud: [Broken mention](broken://pages/20sCqrhqLvNa1cFSthLv). ### Check qualifiers When sending results to Soda Cloud, check qualifiers ensure that the checks in the source YAML document can be correlated with the checks in Soda Cloud. By default, **every check in a contract has a qualifier**. Contracts generated through Soda Cloud (via the contract editor, Contract Copilot, Contract Autopilot, or the skeleton generator) **populate a qualifier on every check**, including the first check of each type. The example below shows the form you'll see in generated contracts: 
dataset: datasource/db/schema/table

columns:
  - name: status
    checks:
      - invalid:
          qualifier: c1
          valid_values: ["C", "A", "D"]
      - invalid:
          qualifier: c2
          valid_format:
            regex: ^[CAD]$
            name: Characters format
However, in hand-written contracts, the qualifier is only strictly required when a `checks` section contains multiple checks of the same type: Soda needs distinct qualifiers to give each check a unique identity. **The first check of a given type can omit the qualifier**; the absence of one is treated as a distinct identity from any qualifier value. Both of the following are valid: 
columns:
  - name: status
    checks:
      - invalid:                         # no qualifier — valid for the first check of this type
          valid_values: ["C", "A", "D"]
      - invalid:
          qualifier: c2
          valid_format:
            regex: ^[CAD]$
            name: Characters format
Any text or numeric value is allowed as the qualifier. {% hint style="warning" %} If you change the qualifier on an existing check, the **historical results of that check in Soda Cloud are lost**; Soda treats it as a new check identity. If two checks resolve to the same identity, contract verification fails with a `Duplicate identity` error. {% endhint %} *** ## Configure a filter for all checks A dataset filter ensures that all checks in the contract are only applied to a subset of the rows in the dataset. This is most commonly used to apply all checks to the latest time partition of the data. Each time new data is appended to an incremental dataset, the contract verification should evaluate the checks only on the rows in the latest time partition. 
dataset: datasource / db / schema / dataset;

filter: "created_at >= CURRENT_DATE - INTERVAL '1 day'";
    
columns: []
## Make contracts dynamic with variables Variables allow dynamic substitution of values in contracts. They help you: * **Parameterize** values that differ across environments, datasets, or schedules. * **Reuse values** in multiple places within the same contract to reduce duplication and improve maintainability. Variables can be used in filters, checks, thresholds, and more. Declare variables at the top of the contract: 
dataset: datasource/db/schema/dataset

variables:
  COUNTRY:
    default: France
    
columns: []
The default value is optional. Variables without a `default` are required when verifying a contract (see below). *If you don't specify a* `default` *value, don't forget the colon (*`:`*) after the variable name.* Use variables in other places in the contract with syntax: `${var.VARIABLE_NAME}` {% code title="contract.yml" %} ```yaml dataset: datasource/db/schema/dataset filter: "created_at >= '${var.START_DATE}'" columns: - name: status checks: - missing: ``` {% endcode %} {% hint style="info" %} **Variables are case sensitive**. We recommend always using upper case and underscores for variable names. {% endhint %} You can specify variables values when verifying a contract: ```bash soda contract verify --set START_DATE=2024-05-19T14:30:00Z ``` {% hint style="danger" %} Setting a variable at verification time overrides the default value configured in the contract. {% endhint %} #### Allowed variable type Soda currently supports only **string** and **numeric** variables, meaning that variables must be represented as either **text values** or **numbers**. Other variable types (such as boolean, date/time, arrays, or complex objects) are not supported at this time, so **any inputs should be formatted or converted into one of these two supported formats** before being used in Soda. #### Soda variables `${soda.NOW}` is a built-in variable that provides the **current timestamp at the moment of scan execution**, allowing you to create dynamic filters relative to the time of scan execution. For example, to check for records in the last 24 hours. *** ## Schedule a contract verification Soda Cloud can execute the contract verification on a time schedule. First, configure the time schedule in the contract. {% code title="" %} ```yaml dataset: datasource/db/schema/dataset soda_runner: checks_schedule: cron: 0 0 * * * timezone: UTC columns: [] ``` {% endcode %} * This defines a cron-based schedule to trigger contract **verification** * Requires the data source to be configured in Soda Cloud with Soda Runner Second, publish the contract to Soda Cloud. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Soda APIs # Python API This reference explains how to use the Soda Core Python API to generate, test, publish, and verify data contracts using local execution (Soda Core) or remote execution (Soda Runner). ## Installation {% content-ref url="/pages/xbHoaBtcgajPRYZQ1UoX" %} [Soda Python Libraries](/deployment-options/soda-python-libraries) {% endcontent-ref %} ## Publish a local Contract to Soda Cloud {% hint style="warning" %} Publishing a new version of a contract to Soda Cloud requires additional permissions in Soda Cloud. * If you are publishing a new version for an **existing dataset**, you must have the **Managed contract** permission for that dataset. * If you are publishing a **new dataset**, you must have the global permission **Create new datasets and data sources with Soda Library**. Learn more about permissions in Soda Cloud [Broken mention](broken://pages/n914KXqJNBoZ19Ay5W0P) {% endhint %} Use `publish_contract` to publish a contract to Soda Cloud: ```python from soda_core.contracts import publish_contract result = publish_contract( contract_file_path="contract1.yaml", soda_cloud_file_path="path/to/soda_cloud.yml" ) ```
ParameterRequiredTypeDescription
contract_file_pathYesstrPath to local contract file
soda_cloud_file_pathyesstrPath to Soda Cloud config file
### **Configure Logging** To enable logging for contract operations, use: ```python from soda_core import configure_logging # Enable or disable verbose logging configure_logging(verbose=True) ``` To access logs **after** a verification run: ```python from soda_core.contracts import publish_contract result = publish_contract( contract_file_path="contract1.yaml", soda_cloud_file_path="path/to/soda_cloud.yml" ) result.get_logs() ``` ### Publish Result Object #### Publish Verification Session Result Represents the overall publication session (may include multiple contracts). **Attributes** * `ContractPublicationResultList: list[ContractPublicationResult]` : all per-contract publication results. See [#contractpublicationresult](#contractpublicationresult "mention") **Methods** * `get_logs() -> list[str]` : publication logs across all contracts. * `get_errors() -> list[str]` : publication errors across all contracts. *** #### **ContractPublicationResult** Immutable outcome for a **single** contract publishing. **Attributes** * `contract: Optional[Contract]` : See [#contract](#contract "mention") *** #### Contract Describes the published contract (dataset identity and source info). **Attributes** * `data_source_name str` : name of the data source (as configured in Soda). * `dataset_prefix: str[]` : hierarchical prefix (e.g., db, schema). * `dataset_name: str` : name of the dataset (table/view). * `soda_qualified_dataset_name: str` : full qualified dataset identifier. * `source: YamlFileContentInfo` : reference to YAML source definition. See [#yamlfilecontentinfo](#yamlfilecontentinfo "mention") * `dataset_id: Optional[str]`: the unique identifier of the dataset (only filled in when results are sent to Soda Cloud). #### YamlFileContentInfo Describes the published YAML. **Attributes** * `source_content_str: Optional[str]` : YAML provided inline. * `local_path: Optional[str]` : local file path to the YAML. ## Verify a Contract {% hint style="info" %} Soda Library can verify contracts either locally (Soda Core) or remotely (Soda Runner). The returned session result objects share a common structure, but log/diagnostics availability differs by execution context. {% endhint %} {% hint style="warning" %} Publishing verification results to Soda Cloud with `publish=True` requires additional permissions in Soda Cloud. * If you are publishing to an **existing dataset**, you must have the **Managed contract** permission for that dataset. * If you are publishing a **new dataset**, you must have the global permission **Create new datasets and data sources with Soda Library**. Learn more about permissions in Soda Cloud [Broken mention](broken://pages/n914KXqJNBoZ19Ay5W0P) {% endhint %} ### With Soda Core Use `verify_contract_locally` to run the verification using **local configuration and execution**: ```python from soda_core.contracts import verify_contract_locally result = verify_contract_locally( data_source_file_path="path/to/data_source.yml", contract_file_path="contract1.yaml", ## OR dataset_identifier="datasource/db/schema/dataset" soda_cloud_file_path="path/to/soda_cloud.yml", variables={"START_DATE": "2024-01-01"}, publish=False, ) ```
ParameterRequiredTypeDescription
data_source_file_pathYes (or data_source_file_paths is required)strPath to data source file /pages/Ovp9NbJpqBhQz07AImSa
data_source_file_pathsYes (or data_source_file_path is required)[str]

Paths to data source files /pages/Ovp9NbJpqBhQz07AImSa .

Required when using reconciliation checks to compare with a source dataset located in another data source.

contract_file_pathYes (or dataset_identifier is required)strPath to local contract file
dataset_identifierYes (or contract_file_path is required)strDataset identifier to fetch contracts from Soda Cloud /pages/yB0XMIIob1nyTOjyZraz#dataset-fully-qualified-name
variablesNodictVariables to override in contract
publishNoboolPush results to Soda Cloud.
soda_cloud_file_pathNo (unless publish is True)strPath to Soda Cloud config file
### With Soda Runner Use `verify_contract_on_runner` to run verification **remotely via Soda Runner**. ```python from soda_core.contracts import verify_contract_on_runner result = verify_contract_on_runner( contract_file_path="path/to/contract.yaml", ## OR dataset_identifier="datasource/db/schema/dataset" soda_cloud_file_path="path/to/soda_cloud.yml", variables={"COUNTRY": "US"}, publish=True, ) ```
ParameterRequiredTypeDescription
contract_file_pathYes (or dataset_identifier is required)strPath to local contract file
dataset_identifierYes (or contract_file_path is required)strDataset identifier to fetch contract from Soda Cloud /pages/yB0XMIIob1nyTOjyZraz#dataset-fully-qualified-name
variablesNodictVariables to override in contract
publishNoboolPush results to Soda Cloud.
soda_cloud_file_pathYesstrPath to Soda Cloud config file
blocking_timeout_in_minutesNointHow long to wait for the runner to return results (default: 60 minutes)
*** ### **Configure Logging** To enable logging for contract operations, use: ```python from soda_core import configure_logging # Enable or disable verbose logging configure_logging(verbose=True) ``` To access logs **after** a verification run: ```python from soda_core.contracts import verify_contract_on_runner result = verify_contract_on_runner( contract_file_path="path/to/contract.yaml", soda_cloud_file_path="path/to/soda_cloud.yml", ) result.get_logs() ``` {% hint style="warning" %} If you call `verify_contract_locally(...)` or `verify_contract_on_runner(...)` in a loop, you should invoke `configure_logging(...)` **inside the loop before each verification call**. The logging setup is not automatically reset between iterations, so placing `configure_logging(...)` only once outside the loop may lead to missing or inconsistent log output for subsequent runs. {% endhint %} ### Verification Result Object Running a verification returns a **`ContractVerificationSessionResult`** that aggregates results for one or more contracts verified in a single session. Use it to read logs and errors across contracts and to compute aggregate pass/fail counts. Each contract gets a **`ContractVerificationResult`** per-check **`CheckResult`** entries and derived counters. *** #### Contract Verification Session Result Represents the overall verification session (may include multiple contracts). **Attributes available for Soda Core and Soda Runner scans** * `contract_verification_results: list[ContractVerificationResult]` : all per-contract results. See [#contractverificationresult](#contractverificationresult "mention") * `has_errors: bool` : true if any contract has execution errors. * `has_excluded: bool` : true if any contract has any excluded checks. * `is_failed: bool` : true if any contract has failed checks. Ignores execution errors. * `is_passed: bool` : true if all contracts have no failed checks. Ignores execution errors. * `is_warned: bool` : true if any contract has warned checks. Ignores execution errors. * `is_ok: bool` : true if all contracts are both not failed and have no errors. **Attributes only available for Soda Core scans** * `number_of_checks: int` : sum across contract. * `number_of_checks_excluded`: sum across contract. * `number_of_checks_passed: int` : sum across contract. * `number_of_checks_failed: int` : sum across contract. * `started_timestamp: timestamp` : verification start timestamp * `data_timestamp: timestamp` : Effective timestamp of the contract. In the future, It will be possible to overwrite the value for backfilling use case. * `ended_timestamp: timestamp` : verification end timestamp **Methods** * `get_logs() -> list[str]` : concatenated logs from all contracts. * `get_logs_str() -> str` : newline-joined logs. * `get_errors() -> list[str]` / `get_errors_str() -> str` : aggregated error lines from logs. *** #### ContractVerificationResult Immutable outcome for a **single** contract: status, timing, measurements, and per-check results. **Attributes available for Soda Core and Soda Runner scans** * `contract: Contract` : See [#contract](#contract "mention") * `status: ContractVerificationStatus` : `FAILED`, `PASSED`, `WARNED`, `ERROR`, or `UNKNOWN`. See [#contractverificationstatus](#contractverificationstatus "mention") * `sending_results_to_soda_cloud_failed: bool` : whether publishing failed. * `log_records: Optional[list[logging.LogRecord]]` : detailed logs. * `has_errors: bool` : true if status is `ERROR`. * `is_failed: bool` : true if status is `FAILED`. (Only checks, not execution errors.) * `is_warned: bool` : true if status is `WARNED`. (Ignores execution errors.) * `is_passed: bool` : true if status is `PASSED`. (Ignores execution errors.) * `is_ok: bool` : not failed and not in error. * `number_of_checks: int` / `number_of_checks_passed: int` / `number_of_checks_failed: int` : counts derived from `check_results`. **Attributes only available for Soda Core scans** * `data_source: DataSource` : See [#datasource](#datasource "mention") * `data_timestamp: Optional[datetime]` : data time context (if any). * `started_timestamp: datetime` / `ended_timestamp: datetime` : execution window. * `measurements: list[Measurement]` — See [#measurement](#measurement "mention") * `check_results: list[CheckResult]` — See [#checkresult](#checkresult "mention") * `scan_id: Optional[str]`: the id of the scan (filled in when results are sent to soda-cloud) **Methods** * `get_logs() -> list[str]` / `get_logs_str() -> str` : pull message text from log records. * `get_errors() -> list[str]` / `get_errors_str() -> str` : log lines at error level or higher. *** #### CheckResult Represents the outcome of a single check in the contract. **Attributes** * `check: Check` : See [#check](#check "mention") * `outcome: CheckOutcome` : `PASSED`, `WARNED`, `FAILED`, or `NOT_EVALUATED`. See [#checkoutcome](#checkoutcome "mention") * `threshold_value: Optional[float|int]` : threshold applied. * `diagnostic_metric_values: Optional[dict[str, float|int|str]]` : diagnostic metrics. * `is_passed: bool` / `is_failed: bool` / `is_not_evaluated: bool` : convenience flags. * `outcome_emoticon: str` : Unicode for quick visual status. **Methods** * `log_table_row() -> dict` : compact, user-friendly log row (includes diagnostics). * `log_table_row_diagnostics(verbose: bool=True) -> str` : formatted diagnostic string; uses `_log_console_format(...)` for short form. *** #### CheckOutcome `PASSED`, `WARNED`, `FAILED`, `NOT_EVALUATED`. *** #### Measurement Metric container captured during verification. **Attributes** * `metric_id: str` : unique identifier of the metric. * `metric_name: Optional[str]` : name of the metric. * `value: Any` : captured value of the metric. *** #### ContractVerificationStatus Execution status of a contract. Drives `is_failed`, `is_passed`, and `has_errors`. **Values** * `UNKNOWN` : status not determined. * `FAILED` : contract executed and one or more checks failed. * `WARNED`: contract executed and one or more checks warned. * `PASSED` : contract executed and all checks passed. * `ERROR` : execution error occurred. *** #### Check Represents the executed check definition (identity, type, threshold, etc.). **Attributes** * `type: str` : check type (e.g., `missing`, `row_count`). * `name: Optional[str]` : ame of the check. * `qualifier: Optional[str]` : string to distinguish multiple checks of same type. * `identity: str` : unique identifier of the check, **used to correlate results in Soda Cloud**. It is **automatically generated based on the check path** in the contract. * `column_name: Optional[str]` : column the check applies to (if applicable). * `threshold: Optional[Threshold]` : threshold configuration for this check. * `attributes: dict[str, Any]` : metadata attributes (for Soda Cloud, routing, etc.). * `location: str` : where in the contract YAML the check is defined. *** #### Threshold Configured numeric boundaries for a check outcome. **Attributes** * `must_be: Optional[float|int]` : value must equal the configured number. * `must_not_be: Optional[float|int]` : value must not equal the configured number. * `must_be_greater_than: Optional[float|int]` : value must be strictly greater than. * `must_be_greater_than_or_equal: Optional[float|int]` : value must be greater than or equal. * `must_be_less_than: Optional[float|int]` : value must be strictly less than. * `must_be_less_than_or_equal: Optional[float|int]` : value must be less than or equal. * `must_be_between: Optional[dict]` : value must be within the specified range. * `must_be_not_between: Optional[dict]` : value must not be within the specified range. *** #### Contract Describes the verified contract (dataset identity and source info). **Attributes** * `data_source_name str` : name of the data source (as configured in Soda). * `dataset_prefix: str[]` : hierarchical prefix (e.g., db, schema). * `dataset_name: str` : name of the dataset (table/view). * `source: YamlFileContentInfo` : reference to YAML source definition. See [#yamlfilecontentinfo](#yamlfilecontentinfo "mention") * `dataset_id: Optional[str]`: the unique identifier of the dataset (only filled in when results are sent to Soda Cloud). *** #### YamlFileContentInfo Describes where the YAML was loaded from. **Attributes** * `source_content_str: Optional[str]` : YAML provided inline. * `local_path: Optional[str]` : local file path to the YAML. *** #### DataSource Represents the data source used for verification. **Attributes** * `name: str` : logical name of the data source. * `type: str` : type of the data source (e.g., Postgres, Snowflake, BigQuery). *** *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # REST API The Soda Cloud API enables you to trigger actions and query data in your Soda Cloud account. | **Cloud EU:** | ​​ | | ------------- | ---------------------------- | | **Cloud US:** | ​​ | ### Connect to the Soda Cloud API The following example demonstrates how to connect to the Soda Cloud API using **Python**. #### Requirements * A **Soda Cloud account** * **Python 3+** * **Recommended:** A `.env` file containing `API_KEY_ID` and `API_KEY_SECRET`\ Example of `.env`: ``` ## Replace the following with your credentials API_KEY_ID=333c3c33-3c33-3c3c-cccc-3c3c3333c333 API_KEY_SECRET=CcC3cCC3C_CCCc33cCC3cCcCC333c_C3Cc3C333CC33cCCC--33cCcC ``` > Learn how to [Broken mention](broken://pages/H6Piqnv5lUFCAv1Cvm75). #### Example connection You can use this example as a starting point for integrating Soda into your workflows, ensuring that you **replace placeholders** with your own credentials and project-specific details. In one `.py` file, you can: 1. Create a working session 2. Send a POST request 3. Send a GET request ```python ## 1. Create a working session from dotenv import load_dotenv import os import json import requests import base64 # (Recommended) Load environment variables from a .env file load_dotenv() api_key_id = os.getenv("API_KEY_ID") api_key_secret = os.getenv("API_KEY_SECRET") # Create token from your credentials TOKEN = base64.b64encode(f"{api_key_id}:{api_key_secret}".encode()).decode() HOST = "cloud.us.soda.io" # Or "cloud.soda.io" HEADERS = {"Authorization": f"Basic {TOKEN}", "Content-Type": "application/json"} datasetId = "" # Replace with your details data = {"key": "value", "key2": "value2"} # Replace with your details ``` ```python ## 2. Send a POST request to create or update data for the specified dataset response = requests.post( f"https://{HOST}/api/v1/datasets/{datasetId}", headers=HEADERS, json=data ) print(response.json()) ``` ```python ## 3. Send a GET request to retrieve check results or details response = requests.get( f"https://{HOST}/api/v1/checks", headers=HEADERS, params={"checkIds": ["abcd"]} ## (optional) Replace with your details ) print(response.json()) ``` *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Attributes Soda Cloud API Attribute Endpoints ## List attributes > This endpoint enables you to list the attributes available in your organization's Soda Cloud account.\ > \ > Optionally filter by resource type using the \`resourceType\` query parameter.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 1000, inclusive. The default value is 1000.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \- \`resourceType\`: Optional. Filter by resource type (\`dataset\`, \`check\`, or \`column\`).\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Any Soda Cloud user in your organization may execute this query.\ > \ > \## Tags\ > \ > \`Attributes\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Attribute Endpoints","name":"Attributes"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"AttributeResourceTypeDTO":{"type":"string","enum":["dataset","check","column"]},"PublicApiListAttributesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/AttributeContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"AttributeContentDTO":{"type":"object","properties":{"allowedValues":{"type":"array","items":{"type":"string"}},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"resourceType":{"type":"object","$ref":"#/components/schemas/AttributeResourceTypeDTO"},"type":{"type":"object","$ref":"#/components/schemas/AttributeTypeDTO"}},"required":["id","label","name","resourceType","type"]},"AttributeTypeDTO":{"type":"string","enum":["text","number","checkbox","datetime","singleSelect","multiSelect"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/attributes":{"get":{"description":"This endpoint enables you to list the attributes available in your organization's Soda Cloud account.\n\nOptionally filter by resource type using the `resourceType` query parameter.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 1000.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `resourceType`: Optional. Filter by resource type (`dataset`, `check`, or `column`).\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Attributes`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/attributes","parameters":[{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"resourceType","schema":{"type":"object","$ref":"#/components/schemas/AttributeResourceTypeDTO"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiListAttributesResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List attributes","tags":["Attributes"]}}}} ``` ## Create an attribute > This endpoint enables you to create a new attribute in your organization's Soda Cloud account.\ > \ > Attributes are scoped to a resource type (dataset, check, or column). Once an attribute is created, it can be set on the associated resource type. Attributes can be used to configure notification rules, dashboards, filters, etc. For example sending email alerts when checks marked as critical fail.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ATTRIBUTES to execute this call.\ > \ > \## Tags\ > \ > \`Attributes\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Attribute Endpoints","name":"Attributes"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiCreateAttributeRequestDTO":{"type":"object","properties":{"allowedValues":{"type":"array","items":{"type":"string"}},"description":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"resourceType":{"type":"object","$ref":"#/components/schemas/AttributeResourceTypeDTO"},"type":{"type":"object","$ref":"#/components/schemas/AttributeTypeDTO"}},"required":["label","name","resourceType","type"]},"AttributeResourceTypeDTO":{"type":"string","enum":["dataset","check","column"]},"AttributeTypeDTO":{"type":"string","enum":["text","number","checkbox","datetime","singleSelect","multiSelect"]},"PublicApiCreateAttributeResponse":{"type":"object","properties":{"attribute":{"type":"object","$ref":"#/components/schemas/AttributeContentDTO"}},"required":["attribute"]},"AttributeContentDTO":{"type":"object","properties":{"allowedValues":{"type":"array","items":{"type":"string"}},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"resourceType":{"type":"object","$ref":"#/components/schemas/AttributeResourceTypeDTO"},"type":{"type":"object","$ref":"#/components/schemas/AttributeTypeDTO"}},"required":["id","label","name","resourceType","type"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/attributes":{"post":{"description":"This endpoint enables you to create a new attribute in your organization's Soda Cloud account.\n\nAttributes are scoped to a resource type (dataset, check, or column). Once an attribute is created, it can be set on the associated resource type. Attributes can be used to configure notification rules, dashboards, filters, etc. For example sending email alerts when checks marked as critical fail.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ATTRIBUTES to execute this call.\n\n## Tags\n\n`Attributes`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/attributes","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateAttributeRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateAttributeResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create an attribute","tags":["Attributes"]}}}} ``` ## Update an attribute > This endpoint enables you to update an existing attribute in your organization's Soda Cloud account.\ > \ > Only label, description, and allowedValues (for select types) can be updated.\ > \ > When updating allowedValues, existing attribute values on resources are left in place, even if they are no longer in the allowed set.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ATTRIBUTES to execute this call.\ > \ > \## Tags\ > \ > \`Attributes\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Attribute Endpoints","name":"Attributes"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateAttributeRequestDTO":{"type":"object","properties":{"allowedValues":{"type":"array","items":{"type":"string"}},"description":{"type":"string"},"label":{"type":"string"}}},"PublicApiUpdateAttributeResponse":{"type":"object","properties":{"attribute":{"type":"object","$ref":"#/components/schemas/AttributeContentDTO"}},"required":["attribute"]},"AttributeContentDTO":{"type":"object","properties":{"allowedValues":{"type":"array","items":{"type":"string"}},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"resourceType":{"type":"object","$ref":"#/components/schemas/AttributeResourceTypeDTO"},"type":{"type":"object","$ref":"#/components/schemas/AttributeTypeDTO"}},"required":["id","label","name","resourceType","type"]},"AttributeResourceTypeDTO":{"type":"string","enum":["dataset","check","column"]},"AttributeTypeDTO":{"type":"string","enum":["text","number","checkbox","datetime","singleSelect","multiSelect"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/attributes/{attributeId}":{"post":{"description":"This endpoint enables you to update an existing attribute in your organization's Soda Cloud account.\n\nOnly label, description, and allowedValues (for select types) can be updated.\n\nWhen updating allowedValues, existing attribute values on resources are left in place, even if they are no longer in the allowed set.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ATTRIBUTES to execute this call.\n\n## Tags\n\n`Attributes`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/attributes/{attributeId}","parameters":[{"in":"path","name":"attributeId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateAttributeRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateAttributeResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update an attribute","tags":["Attributes"]}}}} ``` ## Delete an attribute > This endpoint enables you to delete an existing attribute from your organization's Soda Cloud account.\ > \ > The attribute will be removed from any resources where it is set.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ATTRIBUTES to execute this call.\ > \ > \## Tags\ > \ > \`Attributes\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Attribute Endpoints","name":"Attributes"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteAttributeResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/attributes/{attributeId}":{"delete":{"description":"This endpoint enables you to delete an existing attribute from your organization's Soda Cloud account.\n\nThe attribute will be removed from any resources where it is set.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ATTRIBUTES to execute this call.\n\n## Tags\n\n`Attributes`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"DELETE/api/v1/attributes/{attributeId}","parameters":[{"in":"path","name":"attributeId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteAttributeResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete an attribute","tags":["Attributes"]}}}} ``` # Checks Soda Cloud API Check Endpoints ## Check information > This endpoint enables you to gather information about the checks that in exist in your organization's Soda Cloud account, including information about the datasets and agreements with which they are associated and the incidents to which they are linked. .\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \- \`datasetId\`: Optionally, use this parameter to collect only the information for checks associated with a specific dataset. Find the \`datasetId\` in the URL of the dataset page in Soda Cloud.\ > \ > \- \`dataStandardId\`: Optionally, use this parameter to collect only the information for checks belonging to a specific data standard. Find the \`dataStandardId\` in the URL of the data standard page in Soda Cloud.\ > \ > \- \`checkIds\`: Optionally, use this parameter to collect only the information for a specific batch of checks. Provide multiple values as a comma-separated list of check IDs, which you can find on the individual check pages in Soda Cloud. This query parameter is limited to 1000 check IDs and it can't be combined with other query parameters like \`page\`, \`datasetId\`, \`size\`.\ > \ > If not specified, the query gathers information for all checks in the account and sorts the results by check name in ascending order.\ > \ > \## Authorization\ > \ > Soda only returns the checks linked to datasets to which the user has \*\*View dataset\*\* permissions. Soda Cloud Admins have access to all checks.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Checks\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Check Endpoints","name":"Checks"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiChecksResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ChecksContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"ChecksContentDTO":{"type":"object","properties":{"agreements":{"type":"array","deprecated":true,"items":{"type":"object","$ref":"#/components/schemas/AgreementSlimDTO"}},"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringToString"},"checkType":{"type":"string"},"cloudUrl":{"type":"string"},"column":{"type":"string"},"createdAt":{"type":"string","format":"date-time"},"datasets":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetSlimDTO"}},"definition":{"type":"string"},"description":{"type":"string"},"evaluationStatus":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"group":{"type":"object","$ref":"#/components/schemas/ChecksContentDTO_Group"},"id":{"type":"string"},"incidents":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/IncidentSlimDTO"}},"lastCheckResultValue":{"type":"object","$ref":"#/components/schemas/CheckResultValueDTO"},"lastCheckRunTime":{"type":"string","format":"date-time","deprecated":true},"lastUpdated":{"type":"string","format":"date-time","deprecated":true},"metricType":{"type":"string"},"name":{"type":"string"},"owner":{"type":"object","$ref":"#/components/schemas/OwnerDTO","deprecated":true}},"required":["createdAt","datasets","evaluationStatus","id","incidents","name"]},"AgreementSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"name":{"type":"string"}}},"MapOfStringToString":{"type":"object","additionalProperties":true},"DatasetSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"qualifiedName":{"type":"string"}}},"EvaluationStatusDTO":{"type":"string","enum":["pass","warn","fail","notEvaluated","excluded"]},"ChecksContentDTO_Group":{"type":"object","properties":{"groupType":{"type":"string"},"identity":{"type":"string"},"name":{"type":"string"}}},"IncidentSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"number":{"type":"integer","format":"int32"},"status":{"type":"object","$ref":"#/components/schemas/IncidentStatusDTO"}},"required":["cloudUrl","id","name","number","status"]},"IncidentStatusDTO":{"type":"string","enum":["reported","investigating","fixing","resolved"]},"CheckResultValueDTO":{"type":"object","properties":{"anomalyDetectionDetails":{"type":"object","$ref":"#/components/schemas/CheckResultAnomalyDetectionDetailsDTO"},"dataTimestamp":{"type":"string","format":"date-time"},"diagnostics":{"type":"object","$ref":"#/components/schemas/CheckResultDiagnosticsV4"},"id":{"type":"string"},"measure":{"type":"object","$ref":"#/components/schemas/MeasureDTO"},"value":{"type":"number"},"valueLabel":{"type":"string"},"valueSeries":{"type":"object","$ref":"#/components/schemas/CheckResultValueSeriesDTO"}},"required":["id"]},"CheckResultAnomalyDetectionDetailsDTO":{"type":"object","properties":{"predictedValue":{"type":"number"}}},"CheckResultDiagnosticsV4":{"type":"object","properties":{"aggregate":{"type":"object","$ref":"#/components/schemas/AggregateDiagnosticsV4"},"aggregate_diff":{"type":"object","$ref":"#/components/schemas/AggregateDiffDiagnosticsV4"},"duplicate":{"type":"object","$ref":"#/components/schemas/DuplicateDiagnosticsV4"},"duplicate_diff":{"type":"object","$ref":"#/components/schemas/DuplicateDiffDiagnosticsV4"},"failed_rows":{"type":"object","$ref":"#/components/schemas/FailedRowsDiagnosticsV4"},"freshness":{"type":"object","$ref":"#/components/schemas/FreshnessDiagnosticsV4"},"freshness_diff":{"type":"object","$ref":"#/components/schemas/FreshnessDiffDiagnosticsV4"},"fuzzy_duplicate":{"type":"object","$ref":"#/components/schemas/FuzzyDuplicateDiagnosticsV4"},"group_by":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4"},"hierarchy":{"type":"object","$ref":"#/components/schemas/HierarchyDiagnosticsV4"},"invalid":{"type":"object","$ref":"#/components/schemas/InvalidDiagnosticsV4"},"llm":{"type":"object","$ref":"#/components/schemas/LlmDiagnosticsV4"},"metric":{"type":"object","$ref":"#/components/schemas/MetricDiagnosticsV4"},"metric_diff":{"type":"object","$ref":"#/components/schemas/MetricDiffDiagnosticsV4"},"missing":{"type":"object","$ref":"#/components/schemas/MissingDiagnosticsV4"},"reference_diff":{"type":"object","$ref":"#/components/schemas/ReferenceDiffDiagnosticsV4"},"row_count":{"type":"object","$ref":"#/components/schemas/RowCountDiagnosticsV4"},"row_count_diff":{"type":"object","$ref":"#/components/schemas/RowCountDiffDiagnosticsV4"},"rows_diff":{"type":"object","$ref":"#/components/schemas/RowsDiffDiagnosticsV4"},"schema":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4"}}},"AggregateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}},"AggregateDiffDiagnosticsV4":{"type":"object","properties":{"diff":{"type":"number"},"diffPercent":{"type":"number"},"function":{"type":"string"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceValue":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetValue":{"type":"number"}}},"DuplicateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}},"DuplicateDiffDiagnosticsV4":{"type":"object","properties":{"duplicateCountDiff":{"type":"integer","format":"int32"},"duplicateDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceDuplicateCount":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetDuplicateCount":{"type":"integer","format":"int32"}}},"FailedRowsDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}},"FreshnessDiagnosticsV4":{"type":"object","properties":{"actualTimestamp":{"type":"string","format":"date-time"},"actualTimestampUtc":{"type":"string","format":"date-time"},"datasetRowsTested":{"type":"integer","format":"int32"},"expectedTimestamp":{"type":"string","format":"date-time"},"expectedTimestampUtc":{"type":"string","format":"date-time"}}},"FreshnessDiffDiagnosticsV4":{"type":"object","properties":{"freshnessDiff":{"type":"integer","format":"int32"},"freshnessDiffUnit":{"type":"object","$ref":"#/components/schemas/FreshnessDiffDiagnosticsV4_UnitOfTimeDTO"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMaxTimestamp":{"type":"string","format":"date-time"},"sourceMaxTimestampUtc":{"type":"string","format":"date-time"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMaxTimestamp":{"type":"string","format":"date-time"},"targetMaxTimestampUtc":{"type":"string","format":"date-time"}}},"FreshnessDiffDiagnosticsV4_UnitOfTimeDTO":{"type":"string","enum":["minutes","hours","days"]},"FuzzyDuplicateDiagnosticsV4":{"type":"object","properties":{"candidateRowsScored":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"clusterCount":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"fuzzyDuplicateCount":{"type":"integer","format":"int32"},"fuzzyDuplicatePercent":{"type":"number"},"meanConfidence":{"type":"number"}}},"GroupByDiagnosticsV4":{"type":"object","properties":{"diagnosticsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"groups":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_Group"}}}},"GroupByDiagnosticsV4_Group":{"type":"object","properties":{"group":{"type":"string"},"rowsTested":{"type":"number"},"status":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_GroupByOutcomeDTO"},"value":{"type":"number"}}},"GroupByDiagnosticsV4_GroupByOutcomeDTO":{"type":"string","enum":["pass","warn","fail"]},"HierarchyDiagnosticsV4":{"type":"object","properties":{"brokenPathCount":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"invalidValueCount":{"type":"integer","format":"int32"},"missingValueCount":{"type":"integer","format":"int32"}}},"InvalidDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}},"LlmDiagnosticsV4":{"type":"object","properties":{"completionTokens":{"type":"number"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"model":{"type":"string"},"promptTokens":{"type":"number"},"rowsChecked":{"type":"integer","format":"int32"},"rowsFailed":{"type":"integer","format":"int32"},"rowsFailedPercent":{"type":"number"},"tokensSpent":{"type":"integer","format":"int32"}}},"MetricDiagnosticsV4":{"type":"object","properties":{"datasetRowsTested":{"type":"integer","format":"int32"}}},"MetricDiffDiagnosticsV4":{"type":"object","properties":{"metricDiff":{"type":"number"},"metricDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMetric":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMetric":{"type":"number"}}},"MissingDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}},"ReferenceDiffDiagnosticsV4":{"type":"object","properties":{"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"RowCountDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}},"RowCountDiffDiagnosticsV4":{"type":"object","properties":{"rowCountDiff":{"type":"integer","format":"int32"},"rowCountDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"RowsDiffDiagnosticsV4":{"type":"object","properties":{"changed":{"type":"integer","format":"int32"},"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"SchemaDiagnosticsV4":{"type":"object","properties":{"actual":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}},"expected":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}}}},"SchemaDiagnosticsV4_Column":{"type":"object","properties":{"name":{"type":"string"},"type":{"type":"string"}}},"MeasureDTO":{"type":"string","enum":["time"]},"CheckResultValueSeriesDTO":{"type":"object","properties":{"values":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/CheckResultValueSeriesDTO_ValueSeriesValue"}}}},"CheckResultValueSeriesDTO_ValueSeriesValue":{"type":"object","properties":{"label":{"type":"string"},"level":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"value":{"type":"number"}}},"OwnerDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/checks":{"get":{"description":"This endpoint enables you to gather information about the checks that in exist in your organization's Soda Cloud account, including information about the datasets and agreements with which they are associated and the incidents to which they are linked. .\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `datasetId`: Optionally, use this parameter to collect only the information for checks associated with a specific dataset. Find the `datasetId` in the URL of the dataset page in Soda Cloud.\n\n- `dataStandardId`: Optionally, use this parameter to collect only the information for checks belonging to a specific data standard. Find the `dataStandardId` in the URL of the data standard page in Soda Cloud.\n\n- `checkIds`: Optionally, use this parameter to collect only the information for a specific batch of checks. Provide multiple values as a comma-separated list of check IDs, which you can find on the individual check pages in Soda Cloud. This query parameter is limited to 1000 check IDs and it can't be combined with other query parameters like `page`, `datasetId`, `size`.\n\nIf not specified, the query gathers information for all checks in the account and sorts the results by check name in ascending order.\n\n## Authorization\n\nSoda only returns the checks linked to datasets to which the user has **View dataset** permissions. Soda Cloud Admins have access to all checks.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Checks`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/checks","parameters":[{"in":"query","name":"checkIds","schema":{"type":"string"}},{"in":"query","name":"dataStandardId","schema":{"type":"string"}},{"in":"query","name":"datasetId","schema":{"type":"string"}},{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiChecksResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Check information","tags":["Checks"]}}}} ``` ## Delete a check > This endpoint allows you to delete a specific check in your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage checks\*\* permission (V3 datasets) or \*\*Manage contracts\*\* permission (V4 datasets) can delete checks. Soda Cloud Admins can delete checks for all datasets. \*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Checks\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Check Endpoints","name":"Checks"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/checks/{checkId}":{"delete":{"description":"This endpoint allows you to delete a specific check in your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **Manage checks** permission (V3 datasets) or **Manage contracts** permission (V4 datasets) can delete checks. Soda Cloud Admins can delete checks for all datasets. **See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Checks`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"DELETE/api/v1/checks/{checkId}","parameters":[{"in":"path","name":"checkId","required":true,"schema":{"type":"string"}}],"responses":{"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete a check","tags":["Checks"]}}}} ``` # Contracts Soda Cloud API Contract Endpoints ## List contracts > This endpoint allows you to list contracts in your organization.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \- \`datasetId\`: Optionally, use this parameter to list only the contracts for a specific dataset.\ > \ > \- \`datasetQualifiedName\`: Optionally, use this parameter instead of \`datasetId\` to identify the dataset by its qualified name. Do not supply both \`datasetId\` and \`datasetQualifiedName\`.\ > \ > \## Authorization\ > \ > When filtering by dataset, only users with \*\*View dataset\*\* permission on the specified dataset can list its contracts. When no dataset filter is supplied, all contracts in the organization are returned.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Contracts\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Contract Endpoints","name":"Contracts"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiListContractsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ContractContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"ContractContentDTO":{"type":"object","properties":{"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/contracts":{"get":{"description":"This endpoint allows you to list contracts in your organization.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `datasetId`: Optionally, use this parameter to list only the contracts for a specific dataset.\n\n- `datasetQualifiedName`: Optionally, use this parameter instead of `datasetId` to identify the dataset by its qualified name. Do not supply both `datasetId` and `datasetQualifiedName`.\n\n## Authorization\n\nWhen filtering by dataset, only users with **View dataset** permission on the specified dataset can list its contracts. When no dataset filter is supplied, all contracts in the organization are returned.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Contracts`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/contracts","parameters":[{"in":"query","name":"datasetId","schema":{"type":"string"}},{"in":"query","name":"datasetQualifiedName","schema":{"type":"string"}},{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiListContractsResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List contracts","tags":["Contracts"]}}}} ``` ## Create a contract > This endpoint allows you to create a new contract on a dataset. The contract is initialized with a template containing the dataset identifier and an empty columns list. Use the publish endpoint to replace this with your full contract YAML.\ > \ > Provide either \`datasetId\` or \`datasetQualifiedName\` in the request body to identify the target dataset. Exactly one must be supplied. If both are supplied, the request is rejected with a 400 error.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage contracts\*\* permission can create contracts.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Contracts\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Contract Endpoints","name":"Contracts"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiCreateContractRequestDTO":{"type":"object","properties":{"contents":{"type":"string"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"}},"required":["contents"]},"PublicApiCreateContractResponse":{"type":"object","properties":{"contract":{"type":"object","$ref":"#/components/schemas/ContractContentDTO"}}},"ContractContentDTO":{"type":"object","properties":{"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/contracts":{"post":{"description":"This endpoint allows you to create a new contract on a dataset. The contract is initialized with a template containing the dataset identifier and an empty columns list. Use the publish endpoint to replace this with your full contract YAML.\n\nProvide either `datasetId` or `datasetQualifiedName` in the request body to identify the target dataset. Exactly one must be supplied. If both are supplied, the request is rejected with a 400 error.\n\n## Authorization\n\nOnly users with **Manage contracts** permission can create contracts.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Contracts`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/contracts","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateContractRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateContractResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create a contract","tags":["Contracts"]}}}} ``` ## Generate skeleton contract > Triggers skeleton contract generation for a single dataset based on its schema.\ > \ > The dataset must be connected to a Soda Runner and must not already have a published contract.\ > \ > Provide either \`datasetId\` (UUID) or \`datasetQualifiedName\` (e.g. \`my\_datasource/my\_schema/my\_table\`). Exactly one must be supplied.\ > \ > Returns \`202 Accepted\` with a \`Location\` header pointing to the status endpoint. Use the status endpoint to monitor the progress of the skeleton generation.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Contracts\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Contract Endpoints","name":"Contracts"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiCreateSkeletonContractRequestDTO":{"type":"object","properties":{"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/contracts/actions/createSkeleton":{"post":{"description":"Triggers skeleton contract generation for a single dataset based on its schema.\n\nThe dataset must be connected to a Soda Runner and must not already have a published contract.\n\nProvide either `datasetId` (UUID) or `datasetQualifiedName` (e.g. `my_datasource/my_schema/my_table`). Exactly one must be supplied.\n\nReturns `202 Accepted` with a `Location` header pointing to the status endpoint. Use the status endpoint to monitor the progress of the skeleton generation.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Contracts`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/contracts/actions/createSkeleton","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateSkeletonContractRequestDTO"}}},"required":true},"responses":{"202":{"description":"Accepted","headers":{"Location":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Generate skeleton contract","tags":["Contracts"]}}}} ``` ## Get skeleton contract generation status > Returns the status of a skeleton contract generation operation that was triggered via the \*\*Generate skeleton contract\*\* endpoint.\ > \ > The \`operationId\` is the last path segment of the \`Location\` header returned by the \*\*Generate skeleton contract\*\* response.\ > \ > The response includes the operation state, dataset information, and creation timestamp.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Contracts\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Contract Endpoints","name":"Contracts"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiCreateSkeletonContractStatusResponse":{"type":"object","properties":{"created":{"type":"string","format":"date-time"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"operationId":{"type":"string"},"state":{"type":"object","$ref":"#/components/schemas/SkeletonContractOperationStateDTO"}}},"SkeletonContractOperationStateDTO":{"type":"string","enum":["ongoing","completed","failed","canceled"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/contracts/actions/createSkeleton/{operationId}":{"get":{"description":"Returns the status of a skeleton contract generation operation that was triggered via the **Generate skeleton contract** endpoint.\n\nThe `operationId` is the last path segment of the `Location` header returned by the **Generate skeleton contract** response.\n\nThe response includes the operation state, dataset information, and creation timestamp.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Contracts`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/contracts/actions/createSkeleton/{operationId}","parameters":[{"in":"path","name":"operationId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateSkeletonContractStatusResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get skeleton contract generation status","tags":["Contracts"]}}}} ``` ## Generate contracts > Triggers AI-powered contract generation for one or more datasets. The contract generation runs asynchronously.\ > \ > Each dataset must be connected to a Soda Runner and must not already have a contract. If any dataset is not eligible, the request is rejected with a \`400\` response listing the ineligible datasets and reasons.\ > \ > Provide \`datasetIds\` (list of UUIDs) and/or \`datasetQualifiedNames\` (list of qualified names, e.g. \`my\_datasource/my\_schema/my\_table\`). Both lists are additive. At least one must be non-empty.\ > \ > Returns \`202 Accepted\` with a \`Location\` header pointing to the status endpoint. Use the status endpoint to monitor the progress of the contract generation.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Contracts\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Contract Endpoints","name":"Contracts"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiGenerateContractRequestDTO":{"type":"object","properties":{"datasetIds":{"type":"array","items":{"type":"string"}},"datasetQualifiedNames":{"type":"array","items":{"type":"string"}}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/contracts/actions/generate":{"post":{"description":"Triggers AI-powered contract generation for one or more datasets. The contract generation runs asynchronously.\n\nEach dataset must be connected to a Soda Runner and must not already have a contract. If any dataset is not eligible, the request is rejected with a `400` response listing the ineligible datasets and reasons.\n\nProvide `datasetIds` (list of UUIDs) and/or `datasetQualifiedNames` (list of qualified names, e.g. `my_datasource/my_schema/my_table`). Both lists are additive. At least one must be non-empty.\n\nReturns `202 Accepted` with a `Location` header pointing to the status endpoint. Use the status endpoint to monitor the progress of the contract generation.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Contracts`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/contracts/actions/generate","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiGenerateContractRequestDTO"}}},"required":true},"responses":{"202":{"description":"Accepted","headers":{"Location":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Generate contracts","tags":["Contracts"]}}}} ``` ## Get contract generation status > Returns the status of a contract generation operation that was triggered via the \*\*Generate contracts\*\* endpoint.\ > \ > The \`operationId\` is the last path segment of the \`Location\` header returned by the \*\*Generate contracts\*\* response.\ > \ > While the operation is ongoing, only the operation state is returned (\`datasets\` will be \`null\`). Once the operation reaches a terminal state (completed, failed, or canceled), the full per-dataset scan details are included.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Contracts\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Contract Endpoints","name":"Contracts"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiGenerateContractStatusResponse":{"type":"object","properties":{"created":{"type":"string","format":"date-time"},"datasets":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiGenerateContractStatusResponse_DatasetGenerationStatus"}},"operationId":{"type":"string"},"state":{"type":"object","$ref":"#/components/schemas/GenerateContractOperationStateDTO"}}},"PublicApiGenerateContractStatusResponse_DatasetGenerationStatus":{"type":"object","properties":{"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"scanCloudUrl":{"type":"string"},"scanId":{"type":"string"},"scanState":{"type":"object","$ref":"#/components/schemas/ScanStateDTO"}}},"ScanStateDTO":{"type":"string","enum":["queuing","executing","started","cancelationRequested","timeOutRequested","canceled","timedOut","failed","completedWithErrors","completedWithFailures","completedWithWarnings","completed"]},"GenerateContractOperationStateDTO":{"type":"string","enum":["ongoing","completed","failed","canceled"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/contracts/actions/generate/{operationId}":{"get":{"description":"Returns the status of a contract generation operation that was triggered via the **Generate contracts** endpoint.\n\nThe `operationId` is the last path segment of the `Location` header returned by the **Generate contracts** response.\n\nWhile the operation is ongoing, only the operation state is returned (`datasets` will be `null`). Once the operation reaches a terminal state (completed, failed, or canceled), the full per-dataset scan details are included.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Contracts`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/contracts/actions/generate/{operationId}","parameters":[{"in":"path","name":"operationId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiGenerateContractStatusResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get contract generation status","tags":["Contracts"]}}}} ``` ## Get a contract > This endpoint allows you to retrieve a specific contract, including its YAML content.\ > \ > \## Authorization\ > \ > Only users with \*\*View dataset\*\* permission can view contracts.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Contracts\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Contract Endpoints","name":"Contracts"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiGetContractResponse":{"type":"object","properties":{"contract":{"type":"object","$ref":"#/components/schemas/ContractContentDTO"}}},"ContractContentDTO":{"type":"object","properties":{"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/contracts/{contractId}":{"get":{"description":"This endpoint allows you to retrieve a specific contract, including its YAML content.\n\n## Authorization\n\nOnly users with **View dataset** permission can view contracts.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Contracts`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"GET/api/v1/contracts/{contractId}","parameters":[{"in":"path","name":"contractId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiGetContractResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get a contract","tags":["Contracts"]}}}} ``` ## Publish a contract > This endpoint allows you to publish new YAML content for an existing contract.\ > \ > The \`contents\` field must contain valid Soda contract YAML. If the YAML is malformed or does not conform to the contract schema, the request is rejected with a 400 error.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage contracts\*\* permission can publish contracts.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Contracts\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Contract Endpoints","name":"Contracts"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiPublishContractRequestDTO":{"type":"object","properties":{"contents":{"type":"string"}},"required":["contents"]},"PublicApiPublishContractResponse":{"type":"object","properties":{"contract":{"type":"object","$ref":"#/components/schemas/ContractContentDTO"}}},"ContractContentDTO":{"type":"object","properties":{"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/contracts/{contractId}":{"post":{"description":"This endpoint allows you to publish new YAML content for an existing contract.\n\nThe `contents` field must contain valid Soda contract YAML. If the YAML is malformed or does not conform to the contract schema, the request is rejected with a 400 error.\n\n## Authorization\n\nOnly users with **Manage contracts** permission can publish contracts.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Contracts`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/contracts/{contractId}","parameters":[{"in":"path","name":"contractId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiPublishContractRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiPublishContractResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Publish a contract","tags":["Contracts"]}}}} ``` ## Verify a contract > This endpoint triggers a contract verification scan for the specified contract.\ > \ > The response, when successful, is \`201\` and contains headers \`Location\` and \`X-Soda-Scan-Id\` which identify the triggered scan. Use the value of \`X-Soda-Scan-Id\` in the \`scanId\` parameter of the \*\*Get scan status\*\*, \*\*Get scan logs\*\*, and \*\*Cancel a scan\*\* endpoints.\ > \ > Optionally, you can provide \`checkPaths\` to limit verification to specific checks within the contract. Note: \`checkPaths\` may not be available for all organizations.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage contracts\*\* permission can verify contracts.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Contracts\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Contract Endpoints","name":"Contracts"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiVerifyContractRequestDTO":{"type":"object","properties":{"checkPaths":{"type":"array","items":{"type":"string"}}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/contracts/{contractId}/verify":{"post":{"description":"This endpoint triggers a contract verification scan for the specified contract.\n\nThe response, when successful, is `201` and contains headers `Location` and `X-Soda-Scan-Id` which identify the triggered scan. Use the value of `X-Soda-Scan-Id` in the `scanId` parameter of the **Get scan status**, **Get scan logs**, and **Cancel a scan** endpoints.\n\nOptionally, you can provide `checkPaths` to limit verification to specific checks within the contract. Note: `checkPaths` may not be available for all organizations.\n\n## Authorization\n\nOnly users with **Manage contracts** permission can verify contracts.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Contracts`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/contracts/{contractId}/verify","parameters":[{"in":"path","name":"contractId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiVerifyContractRequestDTO"}}},"required":true},"responses":{"201":{"description":"Created","headers":{"Location":{"schema":{"type":"string"}},"X-Soda-Scan-Id":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Verify a contract","tags":["Contracts"]}}}} ``` ## List contract versions > This endpoint allows you to list versions of a specific contract.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \- \`createdBefore\`: Optionally, supply a UTC datetime value (ISO 8601, e.g. \`2024-01-15T10:30:00\`) to only return versions created strictly before this timestamp. Use this for consistent pagination.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Contracts\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Contract Endpoints","name":"Contracts"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiListContractVersionsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ContractVersionDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"ContractVersionDTO":{"type":"object","properties":{"checksum":{"type":"string"},"contents":{"type":"string"},"contractId":{"type":"string"},"created":{"type":"string","format":"date-time"},"creatorId":{"type":"string"},"id":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/contracts/{contractId}/versions":{"get":{"description":"This endpoint allows you to list versions of a specific contract.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `createdBefore`: Optionally, supply a UTC datetime value (ISO 8601, e.g. `2024-01-15T10:30:00`) to only return versions created strictly before this timestamp. Use this for consistent pagination.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Contracts`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/contracts/{contractId}/versions","parameters":[{"in":"path","name":"contractId","required":true,"schema":{"type":"string"}},{"in":"query","name":"createdBefore","schema":{"type":"string","format":"date-time"}},{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiListContractVersionsResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List contract versions","tags":["Contracts"]}}}} ``` # Data Standards Soda Cloud API Data Standard Endpoints ## List data standards > This endpoint allows you to list the data standards in your organization.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 1 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \- \`search\`: Optionally, use this parameter to perform a fuzzy search on the data standard name and label.\ > \ > \- \`status\`: Optionally, use this parameter to list only data standards with a given status (\`draft\`, \`active\`, \`paused\`, or \`invalid\`).\ > \ > \- \`sort\`: Optionally, sort the results by \`name\` (default) or \`label\`.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"DataStandardSortFieldDTO":{"type":"string","enum":["name","label"]},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]},"PublicApiListDataStandardsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DataStandardContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"compliance":{"type":"number"},"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetCount":{"type":"integer","format":"int32"},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["id"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards":{"get":{"description":"This endpoint allows you to list the data standards in your organization.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 1 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `search`: Optionally, use this parameter to perform a fuzzy search on the data standard name and label.\n\n- `status`: Optionally, use this parameter to list only data standards with a given status (`draft`, `active`, `paused`, or `invalid`).\n\n- `sort`: Optionally, sort the results by `name` (default) or `label`.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/dataStandards","parameters":[{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"search","schema":{"type":"string"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"sort","schema":{"type":"object","$ref":"#/components/schemas/DataStandardSortFieldDTO"}},{"in":"query","name":"status","schema":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiListDataStandardsResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List data standards","tags":["Data Standards"]}}}} ``` ## Create a data standard > This endpoint allows you to create a new data standard from YAML contents, a structured scope, status, owners and an optional schedule.\ > \ > The \`contents\` field must contain valid data standard YAML. If the YAML is malformed, the request is rejected with a 400 error.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage data standards\*\* permission can create data standards.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATA\_STANDARDS to execute this call.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDataStandardRequestDTO":{"type":"object","properties":{"contents":{"type":"string"},"description":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardOwnerRequestDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["contents","name","owners","status"]},"DataStandardOwnerRequestDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]},"PublicApiCreateDataStandardResponse":{"type":"object","properties":{"dataStandard":{"type":"object","$ref":"#/components/schemas/DataStandardContentDTO"}}},"DataStandardContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"compliance":{"type":"number"},"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetCount":{"type":"integer","format":"int32"},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["id"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards":{"post":{"description":"This endpoint allows you to create a new data standard from YAML contents, a structured scope, status, owners and an optional schedule.\n\nThe `contents` field must contain valid data standard YAML. If the YAML is malformed, the request is rejected with a 400 error.\n\n## Authorization\n\nOnly users with **Manage data standards** permission can create data standards.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATA_STANDARDS to execute this call.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/dataStandards","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDataStandardRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateDataStandardResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create a data standard","tags":["Data Standards"]}}}} ``` ## Execute data standards for a dataset > This endpoint triggers a scan that runs the active data standards linked to the specified dataset.\ > \ > The response, when successful, is \`201\` and contains the header \`X-Soda-Scan-Id\` and a \`Location\` header which identify the triggered scan. Use the value of \`X-Soda-Scan-Id\` in the \`scanId\` parameter of the \*\*Get scan status\*\*, \*\*Get scan logs\*\*, and \*\*Cancel a scan\*\* endpoints.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage data standards\*\* permission can execute data standards.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATA\_STANDARDS to execute this call.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiExecuteDataStandardsRequestDTO":{"type":"object","properties":{"datasetId":{"type":"string"}},"required":["datasetId"]},"PublicApiExecuteDataStandardsResponse":{"type":"object","properties":{"dataStandardIds":{"type":"array","items":{"type":"string"}},"scanId":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards/actions/execute":{"post":{"description":"This endpoint triggers a scan that runs the active data standards linked to the specified dataset.\n\nThe response, when successful, is `201` and contains the header `X-Soda-Scan-Id` and a `Location` header which identify the triggered scan. Use the value of `X-Soda-Scan-Id` in the `scanId` parameter of the **Get scan status**, **Get scan logs**, and **Cancel a scan** endpoints.\n\n## Authorization\n\nOnly users with **Manage data standards** permission can execute data standards.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATA_STANDARDS to execute this call.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/dataStandards/actions/execute","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiExecuteDataStandardsRequestDTO"}}},"required":true},"responses":{"201":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiExecuteDataStandardsResponse"}}},"description":"Created"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Execute data standards for a dataset","tags":["Data Standards"]}}}} ``` ## Preview the datasets matching a data standard scope > This endpoint evaluates a candidate scope and returns the datasets that would match it, without creating or modifying any data standard.\ > \ > The \`scope\` uses the same structured rule grammar as the create-data-standard endpoint. An omitted or empty scope matches \*\*all\*\* datasets, mirroring how a data standard without a scope applies to every dataset. Results are paged with \`page\` (0-based, default 0) and \`size\` (between 1 and 1000, default 10).\ > \ > \## Authorization\ > \ > Only users with \*\*Manage data standards\*\* permission can preview a data standard scope.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATA\_STANDARDS to execute this call.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiPreviewDataStandardScopeRequestDTO":{"type":"object","properties":{"page":{"type":"integer","format":"int32"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"size":{"type":"integer","format":"int32"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"PublicApiPreviewDataStandardScopeResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeMatchingDatasetContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DataStandardScopeMatchingDatasetContentDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"qualifiedName":{"type":"string"}},"required":["datasourceId","id","name"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards/actions/previewScope":{"post":{"description":"This endpoint evaluates a candidate scope and returns the datasets that would match it, without creating or modifying any data standard.\n\nThe `scope` uses the same structured rule grammar as the create-data-standard endpoint. An omitted or empty scope matches **all** datasets, mirroring how a data standard without a scope applies to every dataset. Results are paged with `page` (0-based, default 0) and `size` (between 1 and 1000, default 10).\n\n## Authorization\n\nOnly users with **Manage data standards** permission can preview a data standard scope.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATA_STANDARDS to execute this call.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/dataStandards/actions/previewScope","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiPreviewDataStandardScopeRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiPreviewDataStandardScopeResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Preview the datasets matching a data standard scope","tags":["Data Standards"]}}}} ``` ## Test a data standard against a dataset > This endpoint runs the checks of an unsaved data standard against one dataset's live data, without creating or modifying any data standard. The tested standard does not appear in your data standards and no checks or check results are persisted; the outcome is only visible on the triggered scan.\ > \ > The \`contents\` field must contain valid data standard YAML. If the YAML is malformed, the request is rejected with a 400 error.\ > \ > The response, when successful, is \`201\` and contains the header \`X-Soda-Scan-Id\` and a \`Location\` header which identify the triggered scan. Use the value of \`X-Soda-Scan-Id\` in the \`scanId\` parameter of the \*\*Get scan status\*\*, \*\*Get scan logs\*\*, and \*\*Cancel a scan\*\* endpoints.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage data standards\*\* permission who can view the dataset's datasource can test a data standard.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATA\_STANDARDS to execute this call.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiTestDataStandardRequestDTO":{"type":"object","properties":{"contents":{"type":"string"},"datasetId":{"type":"string"}},"required":["contents","datasetId"]},"PublicApiTestDataStandardResponse":{"type":"object","properties":{"scanId":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards/actions/test":{"post":{"description":"This endpoint runs the checks of an unsaved data standard against one dataset's live data, without creating or modifying any data standard. The tested standard does not appear in your data standards and no checks or check results are persisted; the outcome is only visible on the triggered scan.\n\nThe `contents` field must contain valid data standard YAML. If the YAML is malformed, the request is rejected with a 400 error.\n\nThe response, when successful, is `201` and contains the header `X-Soda-Scan-Id` and a `Location` header which identify the triggered scan. Use the value of `X-Soda-Scan-Id` in the `scanId` parameter of the **Get scan status**, **Get scan logs**, and **Cancel a scan** endpoints.\n\n## Authorization\n\nOnly users with **Manage data standards** permission who can view the dataset's datasource can test a data standard.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATA_STANDARDS to execute this call.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/dataStandards/actions/test","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiTestDataStandardRequestDTO"}}},"required":true},"responses":{"201":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiTestDataStandardResponse"}}},"description":"Created"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Test a data standard against a dataset","tags":["Data Standards"]}}}} ``` ## Data standards activity > This endpoint returns an organization-wide rollup of data standards activity, including counts of active and total standards, matched datasets, and check results.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDataStandardsActivityResponse":{"type":"object","properties":{"activity":{"type":"object","$ref":"#/components/schemas/DataStandardActivityContentDTO"}}},"DataStandardActivityContentDTO":{"type":"object","properties":{"activeCount":{"type":"integer","format":"int32"},"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"datasetCount":{"type":"integer","format":"int32"},"totalCount":{"type":"integer","format":"int32"}},"required":["activeCount","checkResults","datasetCount","totalCount"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards/activity":{"get":{"description":"This endpoint returns an organization-wide rollup of data standards activity, including counts of active and total standards, matched datasets, and check results.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/dataStandards/activity","responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDataStandardsActivityResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Data standards activity","tags":["Data Standards"]}}}} ``` ## Get a data standard > This endpoint allows you to retrieve a specific data standard, including its YAML contents and scope.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiGetDataStandardResponse":{"type":"object","properties":{"dataStandard":{"type":"object","$ref":"#/components/schemas/DataStandardContentDTO"}}},"DataStandardContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"compliance":{"type":"number"},"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetCount":{"type":"integer","format":"int32"},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["id"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards/{dataStandardId}":{"get":{"description":"This endpoint allows you to retrieve a specific data standard, including its YAML contents and scope.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"GET/api/v1/dataStandards/{dataStandardId}","parameters":[{"in":"path","name":"dataStandardId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiGetDataStandardResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get a data standard","tags":["Data Standards"]}}}} ``` ## Update a data standard > This endpoint allows you to replace an existing data standard's YAML contents, scope, status, owners and schedule.\ > \ > The \`contents\` field must contain valid data standard YAML. If the YAML is malformed, the request is rejected with a 400 error.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage data standards\*\* permission can update data standards.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATA\_STANDARDS to execute this call.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDataStandardRequestDTO":{"type":"object","properties":{"contents":{"type":"string"},"description":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardOwnerRequestDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["contents","name","owners","status"]},"DataStandardOwnerRequestDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]},"PublicApiUpdateDataStandardResponse":{"type":"object","properties":{"dataStandard":{"type":"object","$ref":"#/components/schemas/DataStandardContentDTO"}}},"DataStandardContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"compliance":{"type":"number"},"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetCount":{"type":"integer","format":"int32"},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["id"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards/{dataStandardId}":{"post":{"description":"This endpoint allows you to replace an existing data standard's YAML contents, scope, status, owners and schedule.\n\nThe `contents` field must contain valid data standard YAML. If the YAML is malformed, the request is rejected with a 400 error.\n\n## Authorization\n\nOnly users with **Manage data standards** permission can update data standards.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATA_STANDARDS to execute this call.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/dataStandards/{dataStandardId}","parameters":[{"in":"path","name":"dataStandardId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDataStandardRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDataStandardResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update a data standard","tags":["Data Standards"]}}}} ``` ## Delete a data standard > Schedules a data standard for deletion. Deletion is processed asynchronously: the standard and its dataset associations, owners and YAML file are removed in the background.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage data standards\*\* permission can delete data standards.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATA\_STANDARDS to execute this call.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteDataStandardResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards/{dataStandardId}":{"delete":{"description":"Schedules a data standard for deletion. Deletion is processed asynchronously: the standard and its dataset associations, owners and YAML file are removed in the background.\n\n## Authorization\n\nOnly users with **Manage data standards** permission can delete data standards.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATA_STANDARDS to execute this call.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"DELETE/api/v1/dataStandards/{dataStandardId}","parameters":[{"in":"path","name":"dataStandardId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteDataStandardResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete a data standard","tags":["Data Standards"]}}}} ``` ## Evaluate a data standard's scope > This endpoint re-evaluates a data standard's scope and returns the ids of the datasets that currently match it.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage data standards\*\* permission can evaluate a data standard's scope.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATA\_STANDARDS to execute this call.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiEvaluateDataStandardScopeResponse":{"type":"object","properties":{"dataStandardId":{"type":"string"},"datasetIds":{"type":"array","items":{"type":"string"}}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards/{dataStandardId}/actions/evaluateScope":{"post":{"description":"This endpoint re-evaluates a data standard's scope and returns the ids of the datasets that currently match it.\n\n## Authorization\n\nOnly users with **Manage data standards** permission can evaluate a data standard's scope.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATA_STANDARDS to execute this call.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/dataStandards/{dataStandardId}/actions/evaluateScope","parameters":[{"in":"path","name":"dataStandardId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiEvaluateDataStandardScopeResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Evaluate a data standard's scope","tags":["Data Standards"]}}}} ``` ## Update a data standard's status > This endpoint transitions a data standard to a new status without modifying its contents, scope, owners or schedule.\ > \ > Allowed transitions are validated; an invalid transition is rejected with a 400 error.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage data standards\*\* permission can update a data standard's status.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATA\_STANDARDS to execute this call.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateDataStandardStatusRequestDTO":{"type":"object","properties":{"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["status"]},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]},"PublicApiUpdateDataStandardStatusResponse":{"type":"object","properties":{"dataStandard":{"type":"object","$ref":"#/components/schemas/DataStandardContentDTO"}}},"DataStandardContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"compliance":{"type":"number"},"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetCount":{"type":"integer","format":"int32"},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["id"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards/{dataStandardId}/actions/status":{"post":{"description":"This endpoint transitions a data standard to a new status without modifying its contents, scope, owners or schedule.\n\nAllowed transitions are validated; an invalid transition is rejected with a 400 error.\n\n## Authorization\n\nOnly users with **Manage data standards** permission can update a data standard's status.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATA_STANDARDS to execute this call.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/dataStandards/{dataStandardId}/actions/status","parameters":[{"in":"path","name":"dataStandardId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDataStandardStatusRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDataStandardStatusResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update a data standard's status","tags":["Data Standards"]}}}} ``` ## List data standard checks > This endpoint allows you to list the aggregated checks of a specific data standard.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 1 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiListDataStandardChecksResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardCheckContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DataStandardCheckContentDTO":{"type":"object","properties":{"checkType":{"type":"object","$ref":"#/components/schemas/CheckTypeDTO"},"compliance":{"type":"number"},"datasetCount":{"type":"integer","format":"int32"},"failingCount":{"type":"integer","format":"int32"},"lastScanTime":{"type":"string","format":"date-time"},"name":{"type":"string"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"},"worstLevel":{"type":"object","$ref":"#/components/schemas/CheckResultLevelDTO"}}},"CheckTypeDTO":{"type":"string","enum":["missing","invalid","row_count","aggregate","schema","duplicate","freshness","failed_rows","metric","group_by","row_count_diff","aggregate_diff","freshness_diff","metric_diff","rows_diff","duplicate_diff","llm","hierarchy","fuzzy_duplicate","reference_diff"]},"CheckResultLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated","excluded"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards/{dataStandardId}/checks":{"get":{"description":"This endpoint allows you to list the aggregated checks of a specific data standard.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 1 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/dataStandards/{dataStandardId}/checks","parameters":[{"in":"path","name":"dataStandardId","required":true,"schema":{"type":"string"}},{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiListDataStandardChecksResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List data standard checks","tags":["Data Standards"]}}}} ``` ## List data standard datasets > This endpoint allows you to list the datasets matched by a specific data standard.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 1 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Data Standards\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Data Standard Endpoints","name":"Data Standards"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiListDataStandardDatasetsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardDatasetContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DataStandardDatasetContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanId":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"qualifiedName":{"type":"string"}},"required":["id","name"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/dataStandards/{dataStandardId}/datasets":{"get":{"description":"This endpoint allows you to list the datasets matched by a specific data standard.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 1 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Data Standards`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/dataStandards/{dataStandardId}/datasets","parameters":[{"in":"path","name":"dataStandardId","required":true,"schema":{"type":"string"}},{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiListDataStandardDatasetsResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List data standard datasets","tags":["Data Standards"]}}}} ``` # Datasets Soda Cloud API Dataset Endpoints ## Dataset information > This endpoint enables you to gather information about the datasets that in exist in your organization's Soda Cloud account, including information about the Soda Cloud resources with which it is associated, such as data source, incidents, attributes and health status.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \- \`from\`: Optionally, use this parameter to collect only the information for datasets that have been updated after a particular point in time. Supply an ISO8601 timestamp value. Example: \`2023-12-31T10:15:30+01:00\`\ > \ > \- \`datasourceName\`: Optionally, use this parameter to collect only the information for datasets associated with a specific datasource.\ > \ > \- \`search\`: Optionally, use this parameter to perform a fuzzy search on the dataset name.\ > \ > If not specified, the query gathers information for all datasets in the account and sorts the results first by dataset name and then by datasource name in ascending order.\ > \ > \## Authorization\ > \ > Soda only returns the datasets to which the user has \*\*View dataset\*\* permissions. Soda Cloud Admins have access to all datasets.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 30 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasetsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetsContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DatasetsContentDTO":{"type":"object","properties":{"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"},"checks":{"type":"number"},"cloudUrl":{"type":"string"},"dataQualityStatus":{"type":"object","$ref":"#/components/schemas/DataQualityStatusDTO"},"datasetQualifiedName":{"type":"string"},"datasource":{"type":"object","$ref":"#/components/schemas/DatasourcePropertiesDTO"},"healthStatus":{"type":"integer","format":"int32"},"id":{"type":"string"},"incidents":{"type":"number"},"label":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO","deprecated":true},"qualifiedName":{"type":"string"},"tags":{"type":"array","items":{"type":"string"}},"timePartition":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionDTO"}},"required":["checks","cloudUrl","healthStatus","id","incidents","name"]},"MapOfStringTo_object":{"type":"object","additionalProperties":true},"DataQualityStatusDTO":{"type":"string","enum":["pass","warn","fail"]},"DatasourcePropertiesDTO":{"type":"object","properties":{"label":{"type":"string"},"name":{"type":"string"},"prefix":{"type":"string"},"type":{"type":"string"},"version":{"type":"object","$ref":"#/components/schemas/ModelVersionDTO"}}},"ModelVersionDTO":{"type":"string","enum":["3","4"]},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]},"DatasetTimePartitionDTO":{"type":"object","properties":{"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO"},"sqlExpression":{"type":"string"},"timePartitionType":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionTypeDTO"}}},"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets":{"get":{"description":"This endpoint enables you to gather information about the datasets that in exist in your organization's Soda Cloud account, including information about the Soda Cloud resources with which it is associated, such as data source, incidents, attributes and health status.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `from`: Optionally, use this parameter to collect only the information for datasets that have been updated after a particular point in time. Supply an ISO8601 timestamp value. Example: `2023-12-31T10:15:30+01:00`\n\n- `datasourceName`: Optionally, use this parameter to collect only the information for datasets associated with a specific datasource.\n\n- `search`: Optionally, use this parameter to perform a fuzzy search on the dataset name.\n\nIf not specified, the query gathers information for all datasets in the account and sorts the results first by dataset name and then by datasource name in ascending order.\n\n## Authorization\n\nSoda only returns the datasets to which the user has **View dataset** permissions. Soda Cloud Admins have access to all datasets.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n30 requests/60 seconds","operationId":"GET/api/v1/datasets","parameters":[{"in":"query","name":"datasourceName","schema":{"type":"string"}},{"in":"query","name":"from","schema":{"type":"string","format":"date-time"}},{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"search","schema":{"type":"string"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetsResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Dataset information","tags":["Datasets"]}}}} ``` ## Get dataset by qualified name > This endpoint enables you to retrieve a single dataset by its qualified name (contractIdentifier) from your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*View dataset\*\* permission can view a dataset. Soda Cloud Admins have permission to view any dataset. \*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"DatasetsContentDTO":{"type":"object","properties":{"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"},"checks":{"type":"number"},"cloudUrl":{"type":"string"},"dataQualityStatus":{"type":"object","$ref":"#/components/schemas/DataQualityStatusDTO"},"datasetQualifiedName":{"type":"string"},"datasource":{"type":"object","$ref":"#/components/schemas/DatasourcePropertiesDTO"},"healthStatus":{"type":"integer","format":"int32"},"id":{"type":"string"},"incidents":{"type":"number"},"label":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO","deprecated":true},"qualifiedName":{"type":"string"},"tags":{"type":"array","items":{"type":"string"}},"timePartition":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionDTO"}},"required":["checks","cloudUrl","healthStatus","id","incidents","name"]},"MapOfStringTo_object":{"type":"object","additionalProperties":true},"DataQualityStatusDTO":{"type":"string","enum":["pass","warn","fail"]},"DatasourcePropertiesDTO":{"type":"object","properties":{"label":{"type":"string"},"name":{"type":"string"},"prefix":{"type":"string"},"type":{"type":"string"},"version":{"type":"object","$ref":"#/components/schemas/ModelVersionDTO"}}},"ModelVersionDTO":{"type":"string","enum":["3","4"]},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]},"DatasetTimePartitionDTO":{"type":"object","properties":{"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO"},"sqlExpression":{"type":"string"},"timePartitionType":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionTypeDTO"}}},"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/byDatasetQualifiedName/{datasetQualifiedName}":{"get":{"description":"This endpoint enables you to retrieve a single dataset by its qualified name (contractIdentifier) from your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **View dataset** permission can view a dataset. Soda Cloud Admins have permission to view any dataset. **See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"GET/api/v1/datasets/byDatasetQualifiedName/{datasetQualifiedName}","parameters":[{"in":"path","name":"datasetQualifiedName","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/DatasetsContentDTO"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get dataset by qualified name","tags":["Datasets"]}}}} ``` ## List dataset roles > This endpoint enables you to gather information about the dataset roles available in your organization's Soda Cloud account. Use the dataset roles to manage access to individual datasets.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 100, inclusive. The default value is 100.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Any Soda Cloud user in your organization may execute this query.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasetRolesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetRoleContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DatasetRoleContentDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"executeContracts":{"type":"boolean","deprecated":true},"id":{"type":"string"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"publishContracts":{"type":"boolean","deprecated":true},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["id","name"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/roles":{"get":{"description":"This endpoint enables you to gather information about the dataset roles available in your organization's Soda Cloud account. Use the dataset roles to manage access to individual datasets.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 100, inclusive. The default value is 100.\n\n- `page`: Supply an integer value. The default value is 0.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/datasets/roles","parameters":[{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetRolesResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List dataset roles","tags":["Datasets"]}}}} ``` ## Create a dataset role > This endpoint allows you to create a new custom dataset role in your organization's Soda Cloud account. Dataset roles define a named set of permissions that control what actions users can perform on individual datasets, such as managing checks, viewing profiling data, configuring the dataset, or managing incidents.\ > \ > Once created, the role can be assigned to users or user groups via the dataset responsibilities endpoints.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ORGANISATION\_SETTINGS to execute this call.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasetRoleRequestDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["name"]},"PublicApiCreateDatasetRoleResponse":{"type":"object","properties":{"datasetRole":{"type":"object","$ref":"#/components/schemas/DatasetRoleContentDTO"}},"required":["datasetRole"]},"DatasetRoleContentDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"executeContracts":{"type":"boolean","deprecated":true},"id":{"type":"string"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"publishContracts":{"type":"boolean","deprecated":true},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["id","name"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/roles":{"post":{"description":"This endpoint allows you to create a new custom dataset role in your organization's Soda Cloud account. Dataset roles define a named set of permissions that control what actions users can perform on individual datasets, such as managing checks, viewing profiling data, configuring the dataset, or managing incidents.\n\nOnce created, the role can be assigned to users or user groups via the dataset responsibilities endpoints.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ORGANISATION_SETTINGS to execute this call.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"POST/api/v1/datasets/roles","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetRoleRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateDatasetRoleResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create a dataset role","tags":["Datasets"]}}}} ``` ## Update a dataset role > This endpoint allows you to update the name or permissions of an existing custom dataset role. Dataset roles define a named set of permissions that control what actions users can perform on individual datasets, such as managing checks, viewing profiling data, configuring the dataset, or managing incidents.\ > \ > Changes apply immediately to all users and user groups assigned this role.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ORGANISATION\_SETTINGS to execute this call.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasetRoleRequestDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["name"]},"PublicApiUpdateDatasetRoleResponse":{"type":"object","properties":{"datasetRole":{"type":"object","$ref":"#/components/schemas/DatasetRoleContentDTO"}},"required":["datasetRole"]},"DatasetRoleContentDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"executeContracts":{"type":"boolean","deprecated":true},"id":{"type":"string"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"publishContracts":{"type":"boolean","deprecated":true},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["id","name"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/roles/{roleId}":{"post":{"description":"This endpoint allows you to update the name or permissions of an existing custom dataset role. Dataset roles define a named set of permissions that control what actions users can perform on individual datasets, such as managing checks, viewing profiling data, configuring the dataset, or managing incidents.\n\nChanges apply immediately to all users and user groups assigned this role.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ORGANISATION_SETTINGS to execute this call.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"POST/api/v1/datasets/roles/{roleId}","parameters":[{"in":"path","name":"roleId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetRoleRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasetRoleResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update a dataset role","tags":["Datasets"]}}}} ``` ## Delete a dataset role > This endpoint allows you to delete a custom dataset role from your organization's Soda Cloud account. Dataset roles define a named set of permissions that control what actions users can perform on individual datasets, such as managing checks, viewing profiling data, configuring the dataset, or managing incidents.\ > \ > Deletion is performed asynchronously and may take some time to complete.\ > \ > Deletion will fail if the role is still assigned to any dataset permissions or used in responsibility settings.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ORGANISATION\_SETTINGS to execute this call.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteDatasetRoleResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/roles/{roleId}":{"delete":{"description":"This endpoint allows you to delete a custom dataset role from your organization's Soda Cloud account. Dataset roles define a named set of permissions that control what actions users can perform on individual datasets, such as managing checks, viewing profiling data, configuring the dataset, or managing incidents.\n\nDeletion is performed asynchronously and may take some time to complete.\n\nDeletion will fail if the role is still assigned to any dataset permissions or used in responsibility settings.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ORGANISATION_SETTINGS to execute this call.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"DELETE/api/v1/datasets/roles/{roleId}","parameters":[{"in":"path","name":"roleId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteDatasetRoleResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete a dataset role","tags":["Datasets"]}}}} ``` ## Get dataset > This endpoint enables you to retrieve a single dataset by its ID from your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*View dataset\*\* permission can view a dataset. Soda Cloud Admins have permission to view any dataset. \*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"DatasetsContentDTO":{"type":"object","properties":{"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"},"checks":{"type":"number"},"cloudUrl":{"type":"string"},"dataQualityStatus":{"type":"object","$ref":"#/components/schemas/DataQualityStatusDTO"},"datasetQualifiedName":{"type":"string"},"datasource":{"type":"object","$ref":"#/components/schemas/DatasourcePropertiesDTO"},"healthStatus":{"type":"integer","format":"int32"},"id":{"type":"string"},"incidents":{"type":"number"},"label":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO","deprecated":true},"qualifiedName":{"type":"string"},"tags":{"type":"array","items":{"type":"string"}},"timePartition":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionDTO"}},"required":["checks","cloudUrl","healthStatus","id","incidents","name"]},"MapOfStringTo_object":{"type":"object","additionalProperties":true},"DataQualityStatusDTO":{"type":"string","enum":["pass","warn","fail"]},"DatasourcePropertiesDTO":{"type":"object","properties":{"label":{"type":"string"},"name":{"type":"string"},"prefix":{"type":"string"},"type":{"type":"string"},"version":{"type":"object","$ref":"#/components/schemas/ModelVersionDTO"}}},"ModelVersionDTO":{"type":"string","enum":["3","4"]},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]},"DatasetTimePartitionDTO":{"type":"object","properties":{"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO"},"sqlExpression":{"type":"string"},"timePartitionType":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionTypeDTO"}}},"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}":{"get":{"description":"This endpoint enables you to retrieve a single dataset by its ID from your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **View dataset** permission can view a dataset. Soda Cloud Admins have permission to view any dataset. **See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"GET/api/v1/datasets/{datasetId}","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/DatasetsContentDTO"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get dataset","tags":["Datasets"]}}}} ``` ## Update dataset > This endpoint enables you to update an existing dataset properties in your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*Configure dataset\*\* permission can update dataset properties. Soda Cloud Admins have permission to update all datasets.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateDatasetDTO":{"type":"object","properties":{"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"},"computeWarehouseOverride":{"type":"object","$ref":"#/components/schemas/UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO"},"diagnosticsWarehouse":{"type":"object","$ref":"#/components/schemas/UpdateDatasetDiagnosticsWarehouseConfigurationRequestDTO"},"label":{"type":"string"},"metricMonitoring":{"type":"object","$ref":"#/components/schemas/UpdateDatasetMetricMonitoringConfigurationRequestDTO"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasetOwnerRequestDTO"}},"partitionColumn":{"type":"string","deprecated":true},"profiling":{"type":"object","$ref":"#/components/schemas/UpdateDatasetProfilingConfigurationRequestDTO"},"tags":{"type":"array","items":{"type":"string"}},"timePartition":{"type":"object","$ref":"#/components/schemas/UpdateDatasetTimePartitionRequestDTO"}}},"MapOfStringTo_object":{"type":"object","additionalProperties":true},"UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO":{"type":"object","properties":{"computeWarehouseConfiguration":{"type":"object","$ref":"#/components/schemas/UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO_WarehouseConfigurationDTO"},"enabled":{"type":"boolean"}}},"UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO_WarehouseConfigurationDTO":{"type":"object","properties":{"name":{"type":"string"}},"required":["name"]},"UpdateDatasetDiagnosticsWarehouseConfigurationRequestDTO":{"type":"object","properties":{"failedRowsConfiguration":{"type":"object","$ref":"#/components/schemas/DatasetDiagnosticsWarehouseFailedRowsConfigurationRequest"},"scanAndResultsConfiguration":{"type":"object","$ref":"#/components/schemas/DatasetDiagnosticsWarehouseScanAndResultsConfigurationRequest"},"schema":{"type":"string"}}},"DatasetDiagnosticsWarehouseFailedRowsConfigurationRequest":{"type":"object","properties":{"enabled":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"maxRowCount":{"type":"integer","format":"int32"},"schema":{"type":"string","deprecated":true},"storageStrategy":{"type":"object","$ref":"#/components/schemas/DiagnosticsWarehouseFailedRowsStorageStrategyDTO"},"uniqueKeyColumnNames":{"type":"array","items":{"type":"string"}}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]},"DiagnosticsWarehouseFailedRowsStorageStrategyDTO":{"type":"string","enum":["fullFailedRows","uniqueKeys","scansAndResultsOnly"]},"DatasetDiagnosticsWarehouseScanAndResultsConfigurationRequest":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"UpdateDatasetMetricMonitoringConfigurationRequestDTO":{"type":"object","properties":{"datasetMetricMonitorsConfiguration":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetMetricMonitorConfigurationDTO"}},"enabled":{"type":"boolean"},"historicalMetricCollectionScanStartDate":{"type":"string","format":"date-time"},"scanSchedule":{"type":"object","$ref":"#/components/schemas/ZonedCronDTO"}}},"DatasetMetricMonitorConfigurationDTO":{"type":"object","properties":{"configuration":{"type":"object","$ref":"#/components/schemas/DatasetMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/DatasetMonitorMetricTypeDTO"}},"required":["configuration","metricType"]},"DatasetMonitorConfigurationDTO":{"type":"object","properties":{"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"DatasetMonitorMetricTypeDTO":{"type":"string","enum":["rowCount","freshness","schema","rowsInserted","totalRowCountChange","timeliness"]},"ZonedCronDTO":{"type":"object","properties":{"cronExpression":{"type":"string"},"timezone":{"type":"string"}},"required":["cronExpression","timezone"]},"PublicApiUpdateDatasetOwnerRequestDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UpdateDatasetProfilingConfigurationRequestDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"profilingSamplingStrategy":{"type":"object","$ref":"#/components/schemas/ProfilingSamplingStrategyConfigurationDTO"},"scanSchedule":{"type":"object","$ref":"#/components/schemas/ZonedCronDTO"}}},"ProfilingSamplingStrategyConfigurationDTO":{"type":"object","properties":{"numberOfRows":{"type":"integer","format":"int32"},"numberOfUnits":{"type":"integer","format":"int32"},"unitOfTime":{"type":"object","$ref":"#/components/schemas/UnitOfTimeDTO"}}},"UnitOfTimeDTO":{"type":"string","enum":["hours","days","weeks"]},"UpdateDatasetTimePartitionRequestDTO":{"type":"object","properties":{"partitionColumn":{"type":"string"},"sqlExpression":{"type":"string"}}},"DatasetsContentDTO":{"type":"object","properties":{"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"},"checks":{"type":"number"},"cloudUrl":{"type":"string"},"dataQualityStatus":{"type":"object","$ref":"#/components/schemas/DataQualityStatusDTO"},"datasetQualifiedName":{"type":"string"},"datasource":{"type":"object","$ref":"#/components/schemas/DatasourcePropertiesDTO"},"healthStatus":{"type":"integer","format":"int32"},"id":{"type":"string"},"incidents":{"type":"number"},"label":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO","deprecated":true},"qualifiedName":{"type":"string"},"tags":{"type":"array","items":{"type":"string"}},"timePartition":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionDTO"}},"required":["checks","cloudUrl","healthStatus","id","incidents","name"]},"DataQualityStatusDTO":{"type":"string","enum":["pass","warn","fail"]},"DatasourcePropertiesDTO":{"type":"object","properties":{"label":{"type":"string"},"name":{"type":"string"},"prefix":{"type":"string"},"type":{"type":"string"},"version":{"type":"object","$ref":"#/components/schemas/ModelVersionDTO"}}},"ModelVersionDTO":{"type":"string","enum":["3","4"]},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]},"DatasetTimePartitionDTO":{"type":"object","properties":{"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO"},"sqlExpression":{"type":"string"},"timePartitionType":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionTypeDTO"}}},"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}":{"post":{"description":"This endpoint enables you to update an existing dataset properties in your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **Configure dataset** permission can update dataset properties. Soda Cloud Admins have permission to update all datasets.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/datasets/{datasetId}","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasetDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/DatasetsContentDTO"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update dataset","tags":["Datasets"]}}}} ``` ## Delete dataset > This endpoint enables you to delete an existing dataset in your organization's Soda Cloud account. \ > \ > Deletion is not immediate and can take some time to complete, depending on the size of your dataset. You can check the status of the deletion by calling the \*\*Get dataset\*\* endpoint.\ > \ > \## Authorization\ > \ > Only users with \*\*Delete dataset\*\* permission can delete a dataset. Soda Cloud Admins have permission to delete any dataset.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteDatasetResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}":{"delete":{"description":"This endpoint enables you to delete an existing dataset in your organization's Soda Cloud account. \n\nDeletion is not immediate and can take some time to complete, depending on the size of your dataset. You can check the status of the deletion by calling the **Get dataset** endpoint.\n\n## Authorization\n\nOnly users with **Delete dataset** permission can delete a dataset. Soda Cloud Admins have permission to delete any dataset.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"DELETE/api/v1/datasets/{datasetId}","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteDatasetResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete dataset","tags":["Datasets"]}}}} ``` ## Set column attribute values in bulk > Upsert attribute values on the columns of a dataset.\ > \ > The request body maps a column name to a map of attribute-name to value. Setting one attribute on a column does not affect other attribute values on the same column. If any column or attribute name is unknown, or any value has the wrong type, the entire request is rejected and no changes are persisted.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have dataset role permission CONFIGURE\_DATASET to execute this call.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiSetColumnAttributesRequestDTO":{"type":"object","properties":{"columnAttributes":{"type":"object","$ref":"#/components/schemas/MapOfStringToMapOfStringTo_object"}},"required":["columnAttributes"]},"MapOfStringToMapOfStringTo_object":{"type":"object","additionalProperties":true},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/columnAttributes":{"post":{"description":"Upsert attribute values on the columns of a dataset.\n\nThe request body maps a column name to a map of attribute-name to value. Setting one attribute on a column does not affect other attribute values on the same column. If any column or attribute name is unknown, or any value has the wrong type, the entire request is rejected and no changes are persisted.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have dataset role permission CONFIGURE_DATASET to execute this call.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"POST/api/v1/datasets/{datasetId}/columnAttributes","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiSetColumnAttributesRequestDTO"}}},"required":true},"responses":{"200":{"description":"Successful response","headers":{"Location":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Set column attribute values in bulk","tags":["Datasets"]}}}} ``` ## Create a column metric monitor > This endpoint allows you to create a new column metric monitor for a specific dataset in your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage checks\*\* permission (V3 datasets) or \*\*Manage contracts\*\* permission (V4 datasets) can create column metric monitors. Soda Cloud Admins can create column metric monitors for all datasets.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiCreateColumnMetricMonitorRequestDTO":{"type":"object","properties":{"columnMetricMonitorConfiguration":{"type":"object","$ref":"#/components/schemas/ColumnMetricMonitorConfigurationDTO"},"columnName":{"type":"string"}},"required":["columnMetricMonitorConfiguration","columnName"]},"ColumnMetricMonitorConfigurationDTO":{"type":"object","properties":{"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/ColumnMonitorMetricTypeDTO"}},"required":["configuration","metricType"]},"ColumnMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByColumnConfigurationDTO"}},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"ColumnMonitorMetricTypeDTO":{"type":"string","enum":["missingPercentage","duplicatePercentage","average","columnFreshness","sum","distinctValuesCount","count","minimumValue","maximumValue","duplicateValuesPercentage","minimumLength","maximumLength","averageLength","standardDeviation","variance","q1","median","q3"]},"PublicApiCreateColumnMetricMonitorResponse":{"type":"object","properties":{"columnMetricMonitor":{"type":"object","$ref":"#/components/schemas/ColumnMetricMonitorDTO"}},"required":["columnMetricMonitor"]},"ColumnMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/ColumnMonitorMetricTypeDTO"}},"required":["checkId","columnName","configuration","metricType"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/columnMetricMonitors":{"post":{"description":"This endpoint allows you to create a new column metric monitor for a specific dataset in your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **Manage checks** permission (V3 datasets) or **Manage contracts** permission (V4 datasets) can create column metric monitors. Soda Cloud Admins can create column metric monitors for all datasets.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/datasets/{datasetId}/columnMetricMonitors","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateColumnMetricMonitorRequestDTO"}}},"required":true},"responses":{"201":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateColumnMetricMonitorResponse"}}},"description":"Created"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create a column metric monitor","tags":["Datasets"]}}}} ``` ## Update a column metric monitor > This endpoint allows you to update an existing column metric monitor for a specific dataset in your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage checks\*\* permission (V3 datasets) or \*\*Manage contracts\*\* permission (V4 datasets) can update column metric monitors. Soda Cloud Admins can update column metric monitors for all datasets. \*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateColumnMetricMonitorRequestDTO":{"type":"object","properties":{"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"}},"required":["configuration"]},"ColumnMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByColumnConfigurationDTO"}},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"PublicApiUpdateColumnMetricMonitorResponse":{"type":"object","properties":{"columnMetricMonitor":{"type":"object","$ref":"#/components/schemas/ColumnMetricMonitorDTO"}},"required":["columnMetricMonitor"]},"ColumnMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/ColumnMonitorMetricTypeDTO"}},"required":["checkId","columnName","configuration","metricType"]},"ColumnMonitorMetricTypeDTO":{"type":"string","enum":["missingPercentage","duplicatePercentage","average","columnFreshness","sum","distinctValuesCount","count","minimumValue","maximumValue","duplicateValuesPercentage","minimumLength","maximumLength","averageLength","standardDeviation","variance","q1","median","q3"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/columnMetricMonitors/{monitorId}":{"post":{"description":"This endpoint allows you to update an existing column metric monitor for a specific dataset in your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **Manage checks** permission (V3 datasets) or **Manage contracts** permission (V4 datasets) can update column metric monitors. Soda Cloud Admins can update column metric monitors for all datasets. **See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/datasets/{datasetId}/columnMetricMonitors/{monitorId}","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}},{"in":"path","name":"monitorId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateColumnMetricMonitorRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateColumnMetricMonitorResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update a column metric monitor","tags":["Datasets"]}}}} ``` ## Delete a column metric monitor > This endpoint allows you to delete a column metric monitor for a specific dataset in your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage checks\*\* permission (V3 datasets) or \*\*Manage contracts\*\* permission (V4 datasets) can delete column metric monitors. Soda Cloud Admins can delete column metric monitors for all datasets. \*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteColumnMetricMonitorResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/columnMetricMonitors/{monitorId}":{"delete":{"description":"This endpoint allows you to delete a column metric monitor for a specific dataset in your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **Manage checks** permission (V3 datasets) or **Manage contracts** permission (V4 datasets) can delete column metric monitors. Soda Cloud Admins can delete column metric monitors for all datasets. **See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"DELETE/api/v1/datasets/{datasetId}/columnMetricMonitors/{monitorId}","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}},{"in":"path","name":"monitorId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteColumnMetricMonitorResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete a column metric monitor","tags":["Datasets"]}}}} ``` ## List the active columns of a dataset with their attribute values > Returns the active columns of a dataset, each with the attribute values currently assigned to it.\ > \ > Columns are ordered by name ascending. Columns with no assigned attribute values are returned with an empty \`attributes\` map.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 1000, inclusive. The default value is 1000.\ > \ > \- \`page\`: Supply a non-negative integer. The default value is 0.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have dataset role permission VIEW to execute this call.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiListDatasetColumnsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetColumnContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"PublicApiDatasetColumnContentDTO":{"type":"object","properties":{"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"},"columnId":{"type":"string"},"columnName":{"type":"string"}},"required":["attributes","columnId","columnName"]},"MapOfStringTo_object":{"type":"object","additionalProperties":true},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/columns":{"get":{"description":"Returns the active columns of a dataset, each with the attribute values currently assigned to it.\n\nColumns are ordered by name ascending. Columns with no assigned attribute values are returned with an empty `attributes` map.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 1000.\n\n- `page`: Supply a non-negative integer. The default value is 0.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have dataset role permission VIEW to execute this call.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"GET/api/v1/datasets/{datasetId}/columns","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}},{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiListDatasetColumnsResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List the active columns of a dataset with their attribute values","tags":["Datasets"]}}}} ``` ## Get dataset compute warehouse configuration > This endpoint retrieves the compute warehouse configuration for a specific dataset in your organization's Soda Cloud account. When the compute warehouse configuration is enabled, the warehouse in the configuration will be used for compute-heavy tasks like data contract execution, metric monitoring, and profiling.\ > \ > \## Authorization\ > \ > Only users with \*\*View dataset\*\* permission can retrieve dataset compute warehouse configuration. Soda Cloud Admins have permission to view all datasets.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"DatasetComputeWarehouseConfigurationDTO":{"type":"object","properties":{"availableWarehouses":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetComputeWarehouseConfigurationDTO_WarehouseConfigurationDTO"}},"currentWarehouse":{"type":"object","$ref":"#/components/schemas/DatasetComputeWarehouseConfigurationDTO_WarehouseConfigurationDTO"},"datasetWarehouseOverride":{"type":"object","$ref":"#/components/schemas/DatasetComputeWarehouseConfigurationDTO_WarehouseConfigurationDTO"},"defaultWarehouse":{"type":"object","$ref":"#/components/schemas/DatasetComputeWarehouseConfigurationDTO_WarehouseConfigurationDTO"},"enabled":{"type":"boolean"}}},"DatasetComputeWarehouseConfigurationDTO_WarehouseConfigurationDTO":{"type":"object","properties":{"name":{"type":"string"}},"required":["name"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/computeWarehouse":{"get":{"description":"This endpoint retrieves the compute warehouse configuration for a specific dataset in your organization's Soda Cloud account. When the compute warehouse configuration is enabled, the warehouse in the configuration will be used for compute-heavy tasks like data contract execution, metric monitoring, and profiling.\n\n## Authorization\n\nOnly users with **View dataset** permission can retrieve dataset compute warehouse configuration. Soda Cloud Admins have permission to view all datasets.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"GET/api/v1/datasets/{datasetId}/computeWarehouse","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/DatasetComputeWarehouseConfigurationDTO"}}},"description":"Successful response"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get dataset compute warehouse configuration","tags":["Datasets"]}}}} ``` ## Create a custom SQL monitor > This endpoint allows you to create a new custom SQL monitor for a specific dataset in your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage contracts\*\* permission can create custom SQL monitors. Soda Cloud Admins can create custom SQL monitors for all datasets.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiCreateCustomSqlMonitorRequestDTO":{"type":"object","properties":{"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/CustomSqlMonitorConfigurationDTO"},"monitorName":{"type":"string"}},"required":["configuration","monitorName"]},"CustomSqlMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}},"required":["resultMetric","sqlQuery"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"PublicApiCreateCustomSqlMonitorResponse":{"type":"object","properties":{"customSqlMonitor":{"type":"object","$ref":"#/components/schemas/CustomMetricMonitorDTO"}},"required":["customSqlMonitor"]},"CustomMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/CustomSqlMonitorConfigurationDTO"},"monitorName":{"type":"string"}},"required":["checkId","configuration","monitorName"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/customSqlMonitors":{"post":{"description":"This endpoint allows you to create a new custom SQL monitor for a specific dataset in your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **Manage contracts** permission can create custom SQL monitors. Soda Cloud Admins can create custom SQL monitors for all datasets.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/datasets/{datasetId}/customSqlMonitors","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateCustomSqlMonitorRequestDTO"}}},"required":true},"responses":{"201":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateCustomSqlMonitorResponse"}}},"description":"Created"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create a custom SQL monitor","tags":["Datasets"]}}}} ``` ## Update a custom SQL monitor > This endpoint allows you to update an existing custom SQL monitor for a specific dataset in your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage contracts\*\* permission can update custom SQL monitors. Soda Cloud Admins can update custom SQL monitors for all datasets. \*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateCustomSqlMonitorRequestDTO":{"type":"object","properties":{"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/CustomSqlMonitorConfigurationUpdateDTO"},"name":{"type":"string"}}},"CustomSqlMonitorConfigurationUpdateDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"PublicApiUpdateCustomSqlMonitorResponse":{"type":"object","properties":{"customSqlMetricMonitor":{"type":"object","$ref":"#/components/schemas/CustomMetricMonitorDTO"}},"required":["customSqlMetricMonitor"]},"CustomMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/CustomSqlMonitorConfigurationDTO"},"monitorName":{"type":"string"}},"required":["checkId","configuration","monitorName"]},"CustomSqlMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}},"required":["resultMetric","sqlQuery"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/customSqlMonitors/{monitorId}":{"post":{"description":"This endpoint allows you to update an existing custom SQL monitor for a specific dataset in your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **Manage contracts** permission can update custom SQL monitors. Soda Cloud Admins can update custom SQL monitors for all datasets. **See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/datasets/{datasetId}/customSqlMonitors/{monitorId}","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}},{"in":"path","name":"monitorId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateCustomSqlMonitorRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateCustomSqlMonitorResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update a custom SQL monitor","tags":["Datasets"]}}}} ``` ## Delete a custom SQL monitor > This endpoint allows you to delete a custom SQL monitor for a specific dataset in your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage contracts\*\* permission can delete custom SQL monitors. Soda Cloud Admins can delete custom SQL monitors for all datasets. \*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteCustomSqlMonitorResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/customSqlMonitors/{monitorId}":{"delete":{"description":"This endpoint allows you to delete a custom SQL monitor for a specific dataset in your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **Manage contracts** permission can delete custom SQL monitors. Soda Cloud Admins can delete custom SQL monitors for all datasets. **See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"DELETE/api/v1/datasets/{datasetId}/customSqlMonitors/{monitorId}","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}},{"in":"path","name":"monitorId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteCustomSqlMonitorResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete a custom SQL monitor","tags":["Datasets"]}}}} ``` ## Get dataset diagnostics warehouse information > This endpoint allows you to retrieve detailed diagnostics warehouse information for a specific dataset in your organization's Soda Cloud account. The diagnostics warehouse collects scan-related data and securely forwards it to the customer's warehouse for storage and analysis.\ > \ > \## Authorization\ > \ > Only users with \*\*View dataset\*\* permission can view dataset diagnostics warehouse. Soda Cloud Admins have access to diagnostics warehouse information for all datasets.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasetDiagnosticsWarehouseResponse":{"type":"object","properties":{"failedRowsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetDiagnosticsWarehouseResponse_DiagnosticsWarehouseFailedRowsConfigurationDTO"},"scanAndResultsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetDiagnosticsWarehouseResponse_DiagnosticsWarehouseScanAndResultsConfigurationDTO"}}},"PublicApiDatasetDiagnosticsWarehouseResponse_DiagnosticsWarehouseFailedRowsConfigurationDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"maxRowCount":{"type":"integer","format":"int32"},"state":{"type":"object","$ref":"#/components/schemas/FailedRowsConfigurationStateDTO"},"storageStrategy":{"type":"object","$ref":"#/components/schemas/DiagnosticsWarehouseFailedRowsStorageStrategyDTO"},"uniqueKeyColumnNames":{"type":"array","items":{"type":"string"}}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]},"FailedRowsConfigurationStateDTO":{"type":"string","enum":["ready","doNotDetectAutomatically","awaitingPrimaryKeysInformationFromAgent","awaitingPrimaryKeysInformationFromRunner","errorMissingUniqueKeys"]},"DiagnosticsWarehouseFailedRowsStorageStrategyDTO":{"type":"string","enum":["fullFailedRows","uniqueKeys","scansAndResultsOnly"]},"PublicApiDatasetDiagnosticsWarehouseResponse_DiagnosticsWarehouseScanAndResultsConfigurationDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/diagnosticsWarehouse":{"get":{"description":"This endpoint allows you to retrieve detailed diagnostics warehouse information for a specific dataset in your organization's Soda Cloud account. The diagnostics warehouse collects scan-related data and securely forwards it to the customer's warehouse for storage and analysis.\n\n## Authorization\n\nOnly users with **View dataset** permission can view dataset diagnostics warehouse. Soda Cloud Admins have access to diagnostics warehouse information for all datasets.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"GET/api/v1/datasets/{datasetId}/diagnosticsWarehouse","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetDiagnosticsWarehouseResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get dataset diagnostics warehouse information","tags":["Datasets"]}}}} ``` ## Get dataset metric monitoring configuration information > This endpoint allows you to retrieve detailed metric monitoring configuration information for a specific dataset in your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*View dataset\*\* permission can view dataset metric monitoring configuration. Soda Cloud Admins have access to metric monitoring configuration information for all datasets.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasetMetricMonitoringResponse":{"type":"object","properties":{"columnMetricMonitors":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ColumnMetricMonitorDTO"}},"customSqlMetricMonitors":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/CustomMetricMonitorDTO"}},"datasetId":{"type":"string"},"datasetMetricMonitorsConfiguration":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetMetricMonitorConfigurationDTO"}},"enabled":{"type":"boolean"},"historicalMetricCollectionScanStartDate":{"type":"string","format":"date-time"},"partitionColumn":{"type":"string","deprecated":true},"scanSchedule":{"type":"object","$ref":"#/components/schemas/ZonedCronDTO"},"timePartition":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionDTO"}}},"ColumnMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/ColumnMonitorMetricTypeDTO"}},"required":["checkId","columnName","configuration","metricType"]},"ColumnMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByColumnConfigurationDTO"}},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"ColumnMonitorMetricTypeDTO":{"type":"string","enum":["missingPercentage","duplicatePercentage","average","columnFreshness","sum","distinctValuesCount","count","minimumValue","maximumValue","duplicateValuesPercentage","minimumLength","maximumLength","averageLength","standardDeviation","variance","q1","median","q3"]},"CustomMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/CustomSqlMonitorConfigurationDTO"},"monitorName":{"type":"string"}},"required":["checkId","configuration","monitorName"]},"CustomSqlMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}},"required":["resultMetric","sqlQuery"]},"DatasetMetricMonitorConfigurationDTO":{"type":"object","properties":{"configuration":{"type":"object","$ref":"#/components/schemas/DatasetMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/DatasetMonitorMetricTypeDTO"}},"required":["configuration","metricType"]},"DatasetMonitorConfigurationDTO":{"type":"object","properties":{"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"DatasetMonitorMetricTypeDTO":{"type":"string","enum":["rowCount","freshness","schema","rowsInserted","totalRowCountChange","timeliness"]},"ZonedCronDTO":{"type":"object","properties":{"cronExpression":{"type":"string"},"timezone":{"type":"string"}},"required":["cronExpression","timezone"]},"DatasetTimePartitionDTO":{"type":"object","properties":{"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO"},"sqlExpression":{"type":"string"},"timePartitionType":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionTypeDTO"}}},"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]},"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/metricMonitoring":{"get":{"description":"This endpoint allows you to retrieve detailed metric monitoring configuration information for a specific dataset in your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **View dataset** permission can view dataset metric monitoring configuration. Soda Cloud Admins have access to metric monitoring configuration information for all datasets.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"GET/api/v1/datasets/{datasetId}/metricMonitoring","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetMetricMonitoringResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get dataset metric monitoring configuration information","tags":["Datasets"]}}}} ``` ## Run historical metric collection for a dataset > This endpoint triggers a historical metric collection scan for a specific dataset in your organization's Soda Cloud account.\ > \ > The dataset must have metric monitoring enabled and a historical collection scan definition configured.\ > \ > The response includes the scan ID and a Location header pointing to the scan status endpoint.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage contracts\*\* permission can run historical metric collection. Soda Cloud Admins have access to run historical metric collection for all datasets. \*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/metricMonitoring/historicalMetricCollection":{"post":{"description":"This endpoint triggers a historical metric collection scan for a specific dataset in your organization's Soda Cloud account.\n\nThe dataset must have metric monitoring enabled and a historical collection scan definition configured.\n\nThe response includes the scan ID and a Location header pointing to the scan status endpoint.\n\n## Authorization\n\nOnly users with **Manage contracts** permission can run historical metric collection. Soda Cloud Admins have access to run historical metric collection for all datasets. **See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/datasets/{datasetId}/metricMonitoring/historicalMetricCollection","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"responses":{"201":{"description":"Created","headers":{"Location":{"schema":{"type":"string"}},"X-Soda-Scan-Id":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Run historical metric collection for a dataset","tags":["Datasets"]}}}} ``` ## Get dataset profiling information > This endpoint allows you to retrieve detailed profiling information for a specific dataset in your organization's Soda Cloud account. Profiling information provides insights into the dataset's structure, statistical summaries, and data characteristics.\ > \ > \## Authorization\ > \ > Only users with \*\*Access dataset profiling and sampling\*\* permission can view dataset profiling. Soda Cloud Admins have access to profiling information for all datasets.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasetProfilingResponse":{"type":"object","properties":{"columnCount":{"type":"number"},"columns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetColumnProfilingContentDTO"}},"datasetId":{"type":"string"},"enabled":{"type":"boolean"},"partitionColumn":{"type":"string","deprecated":true},"profilingTime":{"type":"string","format":"date-time"},"rowCount":{"type":"number"},"samplingStrategyConfiguration":{"type":"object","$ref":"#/components/schemas/ProfilingSamplingStrategyConfigurationDTO"},"scanSchedule":{"type":"object","$ref":"#/components/schemas/ZonedCronDTO"},"timePartition":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionDTO"}},"required":["columns","datasetId"]},"DatasetColumnProfilingContentDTO":{"type":"object","properties":{"id":{"type":"string"},"metrics":{"type":"object","$ref":"#/components/schemas/ColumnMetricContentDTO"},"name":{"type":"string"},"type":{"type":"string"}},"required":["id","name","type"]},"ColumnMetricContentDTO":{"type":"object","properties":{"average":{"type":"number"},"averageLength":{"type":"number"},"distinctCount":{"type":"number"},"extremeValues":{"type":"object","$ref":"#/components/schemas/ExtremeValuesContentDTO"},"frequentValues":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ValueCountContentDTO"}},"histogram":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/HistogramBinContentDTO"}},"maximum":{"type":"object","$ref":"#/components/schemas/_object"},"maximumLength":{"type":"integer","format":"int32"},"maximumTimestamp":{"type":"string","format":"date-time"},"median":{"type":"number"},"minimum":{"type":"object","$ref":"#/components/schemas/_object"},"minimumLength":{"type":"integer","format":"int32"},"minimumTimestamp":{"type":"string","format":"date-time"},"missingCount":{"type":"number"},"standardDeviation":{"type":"number"},"sum":{"type":"number"},"variance":{"type":"number"}}},"ExtremeValuesContentDTO":{"type":"object","properties":{"maximum":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/_object"}},"minimum":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/_object"}}}},"_object":{"type":"object"},"ValueCountContentDTO":{"type":"object","properties":{"count":{"type":"number"},"value":{"type":"object","$ref":"#/components/schemas/_object"}}},"HistogramBinContentDTO":{"type":"object","properties":{"count":{"type":"number"},"lowerBound":{"type":"number"},"upperBound":{"type":"number"}},"required":["count","lowerBound","upperBound"]},"ProfilingSamplingStrategyConfigurationDTO":{"type":"object","properties":{"numberOfRows":{"type":"integer","format":"int32"},"numberOfUnits":{"type":"integer","format":"int32"},"unitOfTime":{"type":"object","$ref":"#/components/schemas/UnitOfTimeDTO"}}},"UnitOfTimeDTO":{"type":"string","enum":["hours","days","weeks"]},"ZonedCronDTO":{"type":"object","properties":{"cronExpression":{"type":"string"},"timezone":{"type":"string"}},"required":["cronExpression","timezone"]},"DatasetTimePartitionDTO":{"type":"object","properties":{"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO"},"sqlExpression":{"type":"string"},"timePartitionType":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionTypeDTO"}}},"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]},"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/profiling":{"get":{"description":"This endpoint allows you to retrieve detailed profiling information for a specific dataset in your organization's Soda Cloud account. Profiling information provides insights into the dataset's structure, statistical summaries, and data characteristics.\n\n## Authorization\n\nOnly users with **Access dataset profiling and sampling** permission can view dataset profiling. Soda Cloud Admins have access to profiling information for all datasets.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"GET/api/v1/datasets/{datasetId}/profiling","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetProfilingResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get dataset profiling information","tags":["Datasets"]}}}} ``` ## List dataset responsibilities > This endpoint enables you to gather information about the user and user groups permissions assigned to a dataset, and their associated roles in your organization's Soda Cloud account.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 1 and 100, inclusive. The default value is 100.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \## Authorization\ > \ > Soda only returns the dataset responsibilities to which the user has \*\*View dataset\*\* permissions. Soda Cloud Admins have access to all dataset responsibilities.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasetResponsibilitiesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetResponsibilityContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DatasetResponsibilityContentDTO":{"type":"object","properties":{"managed":{"type":"boolean"},"role":{"type":"object","$ref":"#/components/schemas/DatasetRoleContentDTO"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["managed","role","type"]},"DatasetRoleContentDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"executeContracts":{"type":"boolean","deprecated":true},"id":{"type":"string"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"publishContracts":{"type":"boolean","deprecated":true},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["id","name"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/responsibilities":{"get":{"description":"This endpoint enables you to gather information about the user and user groups permissions assigned to a dataset, and their associated roles in your organization's Soda Cloud account.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 1 and 100, inclusive. The default value is 100.\n\n- `page`: Supply an integer value. The default value is 0.\n\n## Authorization\n\nSoda only returns the dataset responsibilities to which the user has **View dataset** permissions. Soda Cloud Admins have access to all dataset responsibilities.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"GET/api/v1/datasets/{datasetId}/responsibilities","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}},{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetResponsibilitiesResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List dataset responsibilities","tags":["Datasets"]}}}} ``` ## Update dataset responsibilities > This endpoint enables you to update the user and user groups permissions and their associated roles for a dataset in your organization's Soda Cloud account.\ > \ > \*\*Note\*\*: This API fully replaces the existing responsibilities. The entire list of responsibilities must be provided in the request, including both new and existing entries, as any omissions will result in their removal.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage dataset responsibilities\*\* permission can update dataset responsibilities. Soda Cloud Admins have permission to update all datasets responsibilities.\*\*See \[Manage dataset roles]\() for more information.\*\*The Response of this call, when successful, is \`201\` and contains header \`Location\` which identify the URL where the responsibilities will eventually become available.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasets\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Dataset Endpoints","name":"Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateDatasetResponsibilitiesRequestDTO":{"type":"object","properties":{"responsibilities":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetResponsibilityContentRequestDTO"}}},"required":["responsibilities"]},"DatasetResponsibilityContentRequestDTO":{"type":"object","properties":{"roleId":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["roleId","type"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasets/{datasetId}/responsibilities":{"post":{"description":"This endpoint enables you to update the user and user groups permissions and their associated roles for a dataset in your organization's Soda Cloud account.\n\n**Note**: This API fully replaces the existing responsibilities. The entire list of responsibilities must be provided in the request, including both new and existing entries, as any omissions will result in their removal.\n\n## Authorization\n\nOnly users with **Manage dataset responsibilities** permission can update dataset responsibilities. Soda Cloud Admins have permission to update all datasets responsibilities.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**The Response of this call, when successful, is `201` and contains header `Location` which identify the URL where the responsibilities will eventually become available.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasets`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/datasets/{datasetId}/responsibilities","parameters":[{"in":"path","name":"datasetId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasetResponsibilitiesRequestDTO"}}},"required":true},"responses":{"201":{"description":"Created","headers":{"Location":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update dataset responsibilities","tags":["Datasets"]}}}} ``` # Datasources Soda Cloud API Datasource Endpoints ## List datasources > This endpoint allows you to list datasources in your organization.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \- \`search\`: Supply a string value to filter datasources by name (case-insensitive partial match).\ > \ > \## Authorization\ > \ > Soda only returns datasources for which the user has the \*\*View\*\* permission. Soda Cloud Admins have access to all datasources.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiListDatasourcesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"PublicApiDatasourceDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceTypeDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"PublicApiDatasourceTypeDTO":{"type":"string","enum":["bigquery","mysql","postgres","snowflake","dask","spark","sparkdf","redshift","duckdb","presto","oracle","athena","trino","dremio","vertica","denodo","db2","db2z","sqlserver","databricks","synapse","fabric"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources":{"get":{"description":"This endpoint allows you to list datasources in your organization.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `search`: Supply a string value to filter datasources by name (case-insensitive partial match).\n\n## Authorization\n\nSoda only returns datasources for which the user has the **View** permission. Soda Cloud Admins have access to all datasources.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/datasources","parameters":[{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"search","schema":{"type":"string"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiListDatasourcesResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List datasources","tags":["Datasources"]}}}} ``` ## Create a datasource > Creates a new V4 datasource with the provided configuration.\ > \ > The datasource type is extracted from the YAML configuration file contents.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATASOURCES\_AND\_AGENTS to execute this call.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiCreateDatasourceRequestDTO":{"type":"object","properties":{"configurationFileContents":{"type":"string"},"label":{"type":"string"},"runnerId":{"type":"string"}},"required":["configurationFileContents"]},"PublicApiCreateDatasourceResponse":{"type":"object","properties":{"datasource":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDTO"},"discoveryScanId":{"type":"string"}}},"PublicApiDatasourceDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceTypeDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"PublicApiDatasourceTypeDTO":{"type":"string","enum":["bigquery","mysql","postgres","snowflake","dask","spark","sparkdf","redshift","duckdb","presto","oracle","athena","trino","dremio","vertica","denodo","db2","db2z","sqlserver","databricks","synapse","fabric"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources":{"post":{"description":"Creates a new V4 datasource with the provided configuration.\n\nThe datasource type is extracted from the YAML configuration file contents.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATASOURCES_AND_AGENTS to execute this call.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/datasources","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateDatasourceRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateDatasourceResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create a datasource","tags":["Datasources"]}}}} ``` ## Test datasource connection > This endpoint triggers an asynchronous connection test for a datasource configuration.\ > \ > The response includes an operation ID that you can use to poll the status of the connection test.\ > \ > \## Authorization\ > \ > Any authenticated Soda Cloud user in your organization may test a datasource connection. The test runs against a configuration provided in the request, not against an existing datasource.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiTestConnectionRequestDTO":{"type":"object","properties":{"configurationFileContents":{"type":"string"},"runnerId":{"type":"string"}},"required":["configurationFileContents","runnerId"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/actions/testConnection":{"post":{"description":"This endpoint triggers an asynchronous connection test for a datasource configuration.\n\nThe response includes an operation ID that you can use to poll the status of the connection test.\n\n## Authorization\n\nAny authenticated Soda Cloud user in your organization may test a datasource connection. The test runs against a configuration provided in the request, not against an existing datasource.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/datasources/actions/testConnection","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiTestConnectionRequestDTO"}}},"required":true},"responses":{"202":{"description":"Accepted","headers":{"Location":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Test datasource connection","tags":["Datasources"]}}}} ``` ## Get status for async connection test > This endpoint allows you to check the status of an asynchronous connection test operation.\ > \ > Use the operation ID returned by the test connection endpoint to poll for its current state.\ > \ > \## Authorization\ > \ > Any authenticated Soda Cloud user in your organization may poll the status of a connection test operation.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiTestConnectionStatusResponse":{"type":"object","properties":{"id":{"type":"string"},"message":{"type":"string"},"started":{"type":"string","format":"date-time"},"state":{"type":"object","$ref":"#/components/schemas/PublicApiTestConnectionStateDTO"}},"required":["id","state"]},"PublicApiTestConnectionStateDTO":{"type":"string","enum":["queued","processing","completed","failed","cancelled"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/actions/testConnection/{operationId}":{"get":{"description":"This endpoint allows you to check the status of an asynchronous connection test operation.\n\nUse the operation ID returned by the test connection endpoint to poll for its current state.\n\n## Authorization\n\nAny authenticated Soda Cloud user in your organization may poll the status of a connection test operation.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/datasources/actions/testConnection/{operationId}","parameters":[{"in":"path","name":"operationId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiTestConnectionStatusResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get status for async connection test","tags":["Datasources"]}}}} ``` ## List datasource roles > This endpoint enables you to gather information about the datasource roles available in your organization's Soda Cloud account. Use the datasource roles to manage access to individual datasources.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 100, inclusive. The default value is 100.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Any Soda Cloud user in your organization may execute this query.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasourceRolesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasourceRoleContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DatasourceRoleContentDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"id":{"type":"string"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["id","name"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/roles":{"get":{"description":"This endpoint enables you to gather information about the datasource roles available in your organization's Soda Cloud account. Use the datasource roles to manage access to individual datasources.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 100, inclusive. The default value is 100.\n\n- `page`: Supply an integer value. The default value is 0.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/datasources/roles","parameters":[{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceRolesResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List datasource roles","tags":["Datasources"]}}}} ``` ## Create a datasource role > This endpoint allows you to create a new custom datasource role in your organization's Soda Cloud account. Datasource roles define a named set of permissions that control what actions users can perform on individual datasources, such as creating datasets, managing datasource settings, managing permissions, or deleting the datasource.\ > \ > Once created, the role can be assigned to users or user groups via the datasource responsibilities endpoints.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ORGANISATION\_SETTINGS to execute this call.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasourceRoleRequestDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["name"]},"PublicApiCreateDatasourceRoleResponse":{"type":"object","properties":{"datasourceRole":{"type":"object","$ref":"#/components/schemas/DatasourceRoleContentDTO"}},"required":["datasourceRole"]},"DatasourceRoleContentDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"id":{"type":"string"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["id","name"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/roles":{"post":{"description":"This endpoint allows you to create a new custom datasource role in your organization's Soda Cloud account. Datasource roles define a named set of permissions that control what actions users can perform on individual datasources, such as creating datasets, managing datasource settings, managing permissions, or deleting the datasource.\n\nOnce created, the role can be assigned to users or user groups via the datasource responsibilities endpoints.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ORGANISATION_SETTINGS to execute this call.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"POST/api/v1/datasources/roles","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceRoleRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateDatasourceRoleResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create a datasource role","tags":["Datasources"]}}}} ``` ## Update a datasource role > This endpoint allows you to update the name or permissions of an existing custom datasource role. Datasource roles define a named set of permissions that control what actions users can perform on individual datasources, such as creating datasets, managing datasource settings, managing permissions, or deleting the datasource.\ > \ > Changes apply immediately to all users and user groups assigned this role.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ORGANISATION\_SETTINGS to execute this call.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasourceRoleRequestDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["name"]},"PublicApiUpdateDatasourceRoleResponse":{"type":"object","properties":{"datasourceRole":{"type":"object","$ref":"#/components/schemas/DatasourceRoleContentDTO"}},"required":["datasourceRole"]},"DatasourceRoleContentDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"id":{"type":"string"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["id","name"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/roles/{roleId}":{"post":{"description":"This endpoint allows you to update the name or permissions of an existing custom datasource role. Datasource roles define a named set of permissions that control what actions users can perform on individual datasources, such as creating datasets, managing datasource settings, managing permissions, or deleting the datasource.\n\nChanges apply immediately to all users and user groups assigned this role.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ORGANISATION_SETTINGS to execute this call.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"POST/api/v1/datasources/roles/{roleId}","parameters":[{"in":"path","name":"roleId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceRoleRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceRoleResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update a datasource role","tags":["Datasources"]}}}} ``` ## Delete a datasource role > This endpoint allows you to delete a custom datasource role from your organization's Soda Cloud account. Datasource roles define a named set of permissions that control what actions users can perform on individual datasources, such as creating datasets, managing datasource settings, managing permissions, or deleting the datasource.\ > \ > Deletion is performed asynchronously and may take some time to complete.\ > \ > Deletion will fail if the role is still assigned to any datasource permissions or used in responsibility settings.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ORGANISATION\_SETTINGS to execute this call.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteDatasourceRoleResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/roles/{roleId}":{"delete":{"description":"This endpoint allows you to delete a custom datasource role from your organization's Soda Cloud account. Datasource roles define a named set of permissions that control what actions users can perform on individual datasources, such as creating datasets, managing datasource settings, managing permissions, or deleting the datasource.\n\nDeletion is performed asynchronously and may take some time to complete.\n\nDeletion will fail if the role is still assigned to any datasource permissions or used in responsibility settings.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ORGANISATION_SETTINGS to execute this call.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"DELETE/api/v1/datasources/roles/{roleId}","parameters":[{"in":"path","name":"roleId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteDatasourceRoleResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete a datasource role","tags":["Datasources"]}}}} ``` ## Get a datasource > This endpoint allows you to retrieve a specific datasource by its ID.\ > \ > \## Authorization\ > \ > Users with the \*\*View\*\* permission on the datasource can retrieve its details.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiGetDatasourceResponse":{"type":"object","properties":{"datasource":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDTO"}}},"PublicApiDatasourceDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceTypeDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"PublicApiDatasourceTypeDTO":{"type":"string","enum":["bigquery","mysql","postgres","snowflake","dask","spark","sparkdf","redshift","duckdb","presto","oracle","athena","trino","dremio","vertica","denodo","db2","db2z","sqlserver","databricks","synapse","fabric"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/{datasourceId}":{"get":{"description":"This endpoint allows you to retrieve a specific datasource by its ID.\n\n## Authorization\n\nUsers with the **View** permission on the datasource can retrieve its details.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"GET/api/v1/datasources/{datasourceId}","parameters":[{"in":"path","name":"datasourceId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiGetDatasourceResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get a datasource","tags":["Datasources"]}}}} ``` ## Update a datasource > Updates the configuration of an existing contract (V4) datasource.\ > \ > Only the fields provided in the request body will be updated.\ > \ > \## Authorization\ > \ > Users with the \*\*Manage datasource settings\*\* permission on the datasource, or the organization-level \*\*Manage datasources and runners\*\* permission, can update its configuration.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateDatasourceRequestDTO":{"type":"object","properties":{"configurationFileContents":{"type":"string"},"label":{"type":"string"},"runnerId":{"type":"string"}}},"PublicApiUpdateDatasourceResponse":{"type":"object","properties":{"datasource":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDTO"}}},"PublicApiDatasourceDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceTypeDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"PublicApiDatasourceTypeDTO":{"type":"string","enum":["bigquery","mysql","postgres","snowflake","dask","spark","sparkdf","redshift","duckdb","presto","oracle","athena","trino","dremio","vertica","denodo","db2","db2z","sqlserver","databricks","synapse","fabric"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/{datasourceId}":{"post":{"description":"Updates the configuration of an existing contract (V4) datasource.\n\nOnly the fields provided in the request body will be updated.\n\n## Authorization\n\nUsers with the **Manage datasource settings** permission on the datasource, or the organization-level **Manage datasources and runners** permission, can update its configuration.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/datasources/{datasourceId}","parameters":[{"in":"path","name":"datasourceId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update a datasource","tags":["Datasources"]}}}} ``` ## Delete a datasource > Deletes an existing datasource and all its associated resources.\ > \ > The deletion is performed asynchronously.\ > \ > \## Authorization\ > \ > Users with the \*\*Delete\*\* permission on the datasource, or the organization-level \*\*Manage datasources and runners\*\* permission, can delete it.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteDatasourceResponse":{"type":"object","properties":{"message":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/{datasourceId}":{"delete":{"description":"Deletes an existing datasource and all its associated resources.\n\nThe deletion is performed asynchronously.\n\n## Authorization\n\nUsers with the **Delete** permission on the datasource, or the organization-level **Manage datasources and runners** permission, can delete it.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"DELETE/api/v1/datasources/{datasourceId}","parameters":[{"in":"path","name":"datasourceId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteDatasourceResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete a datasource","tags":["Datasources"]}}}} ``` ## Get datasource diagnostics warehouse configuration > This endpoint allows you to retrieve the diagnostics warehouse configuration for a specific datasource in your organization's Soda Cloud account. The diagnostics warehouse collects scan-related data and securely forwards it to the customer's warehouse for storage and analysis.\ > \ > \## Authorization\ > \ > Users with the \*\*View\*\* permission on the datasource can retrieve its diagnostics warehouse configuration.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasourceDiagnosticsWarehouseResponse":{"type":"object","properties":{"enabled":{"type":"boolean"},"failedRowsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse_FailedRowsConfigurationDTO"},"reuseDatasource":{"type":"boolean"},"scanAndResultsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse_ScanAndResultsConfigurationDTO"},"tableNameTemplate":{"type":"string"}}},"PublicApiDatasourceDiagnosticsWarehouseResponse_FailedRowsConfigurationDTO":{"type":"object","properties":{"actionButton":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse_ActionButtonDTO"},"enabled":{"type":"boolean"},"exposeQueries":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"locationMessage":{"type":"string"},"maxRowCount":{"type":"integer","format":"int32"}}},"PublicApiDatasourceDiagnosticsWarehouseResponse_ActionButtonDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"title":{"type":"string"},"url":{"type":"string"}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]},"PublicApiDatasourceDiagnosticsWarehouseResponse_ScanAndResultsConfigurationDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/{datasourceId}/diagnosticsWarehouse":{"get":{"description":"This endpoint allows you to retrieve the diagnostics warehouse configuration for a specific datasource in your organization's Soda Cloud account. The diagnostics warehouse collects scan-related data and securely forwards it to the customer's warehouse for storage and analysis.\n\n## Authorization\n\nUsers with the **View** permission on the datasource can retrieve its diagnostics warehouse configuration.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"GET/api/v1/datasources/{datasourceId}/diagnosticsWarehouse","parameters":[{"in":"path","name":"datasourceId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get datasource diagnostics warehouse configuration","tags":["Datasources"]}}}} ``` ## Update datasource diagnostics warehouse configuration > This endpoint allows you to update the diagnostics warehouse configuration for a specific datasource in your organization's Soda Cloud account. The diagnostics warehouse collects scan-related data and securely stores it in your warehouse for storage and analysis.\ > \ > \## Table name template\ > \ > The \`tableNameTemplate\` field controls how diagnostic tables are named in the warehouse. Use the \`{dataset\_name}\` placeholder to include the source dataset name. For example, \`dwh\_{dataset\_name}\_failed\` produces table names like \`dwh\_orders\_failed\`.\ > \ > \## Authorization\ > \ > Users with the \*\*Manage datasource settings\*\* permission on the datasource, or the organization-level \*\*Manage datasources and runners\*\* permission, can update its diagnostics warehouse configuration.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO":{"type":"object","properties":{"configurationFileContents":{"type":"string"},"enabled":{"type":"boolean"},"failedRowsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_FailedRowsConfigurationRequestDTO"},"reuseDatasource":{"type":"boolean"},"scanAndResultsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ScanAndResultsConfigurationRequestDTO"},"tableNameTemplate":{"type":"string"}}},"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_FailedRowsConfigurationRequestDTO":{"type":"object","properties":{"actionButton":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ActionButtonRequestDTO"},"enabled":{"type":"boolean"},"exposeQueries":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"locationMessage":{"type":"string"},"maxRowCount":{"type":"integer","format":"int32"}}},"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ActionButtonRequestDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"title":{"type":"string"},"url":{"type":"string"}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]},"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ScanAndResultsConfigurationRequestDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"PublicApiDatasourceDiagnosticsWarehouseResponse":{"type":"object","properties":{"enabled":{"type":"boolean"},"failedRowsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse_FailedRowsConfigurationDTO"},"reuseDatasource":{"type":"boolean"},"scanAndResultsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse_ScanAndResultsConfigurationDTO"},"tableNameTemplate":{"type":"string"}}},"PublicApiDatasourceDiagnosticsWarehouseResponse_FailedRowsConfigurationDTO":{"type":"object","properties":{"actionButton":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse_ActionButtonDTO"},"enabled":{"type":"boolean"},"exposeQueries":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"locationMessage":{"type":"string"},"maxRowCount":{"type":"integer","format":"int32"}}},"PublicApiDatasourceDiagnosticsWarehouseResponse_ActionButtonDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"title":{"type":"string"},"url":{"type":"string"}}},"PublicApiDatasourceDiagnosticsWarehouseResponse_ScanAndResultsConfigurationDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/{datasourceId}/diagnosticsWarehouse":{"post":{"description":"This endpoint allows you to update the diagnostics warehouse configuration for a specific datasource in your organization's Soda Cloud account. The diagnostics warehouse collects scan-related data and securely stores it in your warehouse for storage and analysis.\n\n## Table name template\n\nThe `tableNameTemplate` field controls how diagnostic tables are named in the warehouse. Use the `{dataset_name}` placeholder to include the source dataset name. For example, `dwh_{dataset_name}_failed` produces table names like `dwh_orders_failed`.\n\n## Authorization\n\nUsers with the **Manage datasource settings** permission on the datasource, or the organization-level **Manage datasources and runners** permission, can update its diagnostics warehouse configuration.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/datasources/{datasourceId}/diagnosticsWarehouse","parameters":[{"in":"path","name":"datasourceId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update datasource diagnostics warehouse configuration","tags":["Datasources"]}}}} ``` ## Trigger datasource discovery > This endpoint triggers a discovery scan on the specified datasource.\ > \ > Discovery scans detect new tables and schemas in the datasource. Use this to run discovery on-demand instead of waiting for the scheduled cron.\ > \ > The response includes a \`Location\` header and \`X-Soda-Scan-Id\` header identifying the triggered scan. Use the scan ID with the \*\*Get scan status\*\* endpoint to poll for completion.\ > \ > \## Authorization\ > \ > Users with the \*\*Manage datasource settings\*\* permission on the datasource, or the organization-level \*\*Manage datasources and runners\*\* permission, can trigger discovery on the datasource.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/{datasourceId}/discover":{"post":{"description":"This endpoint triggers a discovery scan on the specified datasource.\n\nDiscovery scans detect new tables and schemas in the datasource. Use this to run discovery on-demand instead of waiting for the scheduled cron.\n\nThe response includes a `Location` header and `X-Soda-Scan-Id` header identifying the triggered scan. Use the scan ID with the **Get scan status** endpoint to poll for completion.\n\n## Authorization\n\nUsers with the **Manage datasource settings** permission on the datasource, or the organization-level **Manage datasources and runners** permission, can trigger discovery on the datasource.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/datasources/{datasourceId}/discover","parameters":[{"in":"path","name":"datasourceId","required":true,"schema":{"type":"string"}}],"responses":{"201":{"description":"Created","headers":{"Location":{"schema":{"type":"string"}},"X-Soda-Scan-Id":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Trigger datasource discovery","tags":["Datasources"]}}}} ``` ## Onboard discovered datasets > This endpoint triggers the asynchronous onboarding of discovered datasets for a datasource.\ > \ > Supply a list of discovered dataset IDs in the request body to onboard them as datasets.\ > \ > The response includes an onboarding ID that you can use to poll the status of the onboarding process.\ > \ > \## Authorization\ > \ > Users with the \*\*Create datasets\*\* permission on the datasource, or the organization-level \*\*Create new datasources and datasets\*\* permission, can onboard discovered datasets.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiOnboardDatasetsRequestDTO":{"type":"object","properties":{"discoveredDatasetIds":{"type":"array","items":{"type":"string"}}},"required":["discoveredDatasetIds"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/{datasourceId}/onboardDatasets":{"post":{"description":"This endpoint triggers the asynchronous onboarding of discovered datasets for a datasource.\n\nSupply a list of discovered dataset IDs in the request body to onboard them as datasets.\n\nThe response includes an onboarding ID that you can use to poll the status of the onboarding process.\n\n## Authorization\n\nUsers with the **Create datasets** permission on the datasource, or the organization-level **Create new datasources and datasets** permission, can onboard discovered datasets.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/datasources/{datasourceId}/onboardDatasets","parameters":[{"in":"path","name":"datasourceId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiOnboardDatasetsRequestDTO"}}},"required":true},"responses":{"202":{"description":"Accepted","headers":{"Location":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Onboard discovered datasets","tags":["Datasources"]}}}} ``` ## Get status for async dataset onboarding process > This endpoint allows you to check the status of an asynchronous dataset onboarding operation.\ > \ > Use the operation ID returned by the onboard datasets endpoint to poll for its current state.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiOnboardDatasetsStatusResponse":{"type":"object","properties":{"ended":{"type":"string","format":"date-time"},"id":{"type":"string"},"message":{"type":"string"},"started":{"type":"string","format":"date-time"},"state":{"type":"object","$ref":"#/components/schemas/PublicApiOnboardingStateDTO"}},"required":["id","state"]},"PublicApiOnboardingStateDTO":{"type":"string","enum":["queued","processing","completed","failed","cancelled"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/{datasourceId}/onboardDatasets/{operationId}":{"get":{"description":"This endpoint allows you to check the status of an asynchronous dataset onboarding operation.\n\nUse the operation ID returned by the onboard datasets endpoint to poll for its current state.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/datasources/{datasourceId}/onboardDatasets/{operationId}","parameters":[{"in":"path","name":"datasourceId","required":true,"schema":{"type":"string"}},{"in":"path","name":"operationId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiOnboardDatasetsStatusResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get status for async dataset onboarding process","tags":["Datasources"]}}}} ``` ## List datasource responsibilities > This endpoint enables you to gather information about the user and user groups permissions assigned to a datasource, and their associated roles in your organization's Soda Cloud account.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 1 and 100, inclusive. The default value is 100.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \## Authorization\ > \ > Soda only returns the datasource responsibilities to which the user has \*\*View datasource\*\* permissions. Soda Cloud Admins have access to all datasource responsibilities.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDatasourceResponsibilitiesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasourceResponsibilityContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DatasourceResponsibilityContentDTO":{"type":"object","properties":{"managed":{"type":"boolean"},"role":{"type":"object","$ref":"#/components/schemas/DatasourceRoleContentDTO"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["managed","role","type"]},"DatasourceRoleContentDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"id":{"type":"string"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["id","name"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/{datasourceId}/responsibilities":{"get":{"description":"This endpoint enables you to gather information about the user and user groups permissions assigned to a datasource, and their associated roles in your organization's Soda Cloud account.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 1 and 100, inclusive. The default value is 100.\n\n- `page`: Supply an integer value. The default value is 0.\n\n## Authorization\n\nSoda only returns the datasource responsibilities to which the user has **View datasource** permissions. Soda Cloud Admins have access to all datasource responsibilities.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"GET/api/v1/datasources/{datasourceId}/responsibilities","parameters":[{"in":"path","name":"datasourceId","required":true,"schema":{"type":"string"}},{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceResponsibilitiesResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List datasource responsibilities","tags":["Datasources"]}}}} ``` ## Update datasource responsibilities > This endpoint enables you to update the user and user groups permissions and their associated roles for a datasource in your organization's Soda Cloud account.\ > \ > \*\*Note\*\*: This API fully replaces the existing responsibilities. The entire list of responsibilities must be provided in the request, including both new and existing entries, as any omissions will result in their removal.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage datasource responsibilities\*\* permission can update datasource responsibilities. Soda Cloud Admins have permission to update all datasources responsibilities.The Response of this call, when successful, is \`201\` and contains header \`Location\` which identify the URL where the responsibilities will eventually become available.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Datasources\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Datasource Endpoints","name":"Datasources"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateDatasourceResponsibilitiesRequestDTO":{"type":"object","properties":{"responsibilities":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasourceResponsibilityContentRequestDTO"}}},"required":["responsibilities"]},"DatasourceResponsibilityContentRequestDTO":{"type":"object","properties":{"roleId":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["roleId","type"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/datasources/{datasourceId}/responsibilities":{"post":{"description":"This endpoint enables you to update the user and user groups permissions and their associated roles for a datasource in your organization's Soda Cloud account.\n\n**Note**: This API fully replaces the existing responsibilities. The entire list of responsibilities must be provided in the request, including both new and existing entries, as any omissions will result in their removal.\n\n## Authorization\n\nOnly users with **Manage datasource responsibilities** permission can update datasource responsibilities. Soda Cloud Admins have permission to update all datasources responsibilities.The Response of this call, when successful, is `201` and contains header `Location` which identify the URL where the responsibilities will eventually become available.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Datasources`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/datasources/{datasourceId}/responsibilities","parameters":[{"in":"path","name":"datasourceId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceResponsibilitiesRequestDTO"}}},"required":true},"responses":{"201":{"description":"Created","headers":{"Location":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update datasource responsibilities","tags":["Datasources"]}}}} ``` # Discovered Datasets Soda Cloud API Discovered Dataset Endpoints ## Discovered datasets > This endpoint enables you to list datasets that Soda has discovered in your data sources. Discovered datasets are datasets that have been found during a discovery scan but may not yet be onboarded.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 1 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \- \`datasourceId\`: Optionally, use this parameter to collect only the discovered datasets associated with a specific data source.\ > \ > \- \`search\`: Optionally, use this parameter to perform a search on the dataset name.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Any Soda Cloud user in your organization may execute this query.\ > \ > \## Tags\ > \ > \`Discovered Datasets\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Discovered Dataset Endpoints","name":"Discovered Datasets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDiscoveredDatasetsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DiscoveredDatasetContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DiscoveredDatasetContentDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"datasourceId":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"onboarded":{"type":"boolean"},"qualifiedName":{"type":"string"}},"required":["datasourceId","id","name","onboarded","qualifiedName"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/discoveredDatasets":{"get":{"description":"This endpoint enables you to list datasets that Soda has discovered in your data sources. Discovered datasets are datasets that have been found during a discovery scan but may not yet be onboarded.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 1 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `datasourceId`: Optionally, use this parameter to collect only the discovered datasets associated with a specific data source.\n\n- `search`: Optionally, use this parameter to perform a search on the dataset name.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Discovered Datasets`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"GET/api/v1/discoveredDatasets","parameters":[{"in":"query","name":"datasourceId","schema":{"type":"string"}},{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"search","schema":{"type":"string"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDiscoveredDatasetsResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Discovered datasets","tags":["Discovered Datasets"]}}}} ``` # Incidents Soda Cloud API Incidents Endpoints ## Update incident > This endpoint enables you to update an existing incident title, severity, status, or description in your organization's Soda Cloud account.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage incident\*\* permission can update incidents. Soda Cloud Admins have permission to update all incidents.\*\*See \[Manage dataset roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Incidents\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Incidents Endpoints","name":"Incidents"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateIncidentDTO":{"type":"object","properties":{"description":{"type":"string"},"resolutionNotes":{"type":"string"},"severity":{"type":"object","$ref":"#/components/schemas/IncidentSeverityDTO"},"status":{"type":"object","$ref":"#/components/schemas/IncidentStatusDTO"},"title":{"type":"string"}}},"IncidentSeverityDTO":{"type":"string","enum":["minor","major","critical"]},"IncidentStatusDTO":{"type":"string","enum":["reported","investigating","fixing","resolved"]},"IncidentContentDTO":{"type":"object","properties":{"description":{"type":"string"},"resolutionNotes":{"type":"string"},"severity":{"type":"object","$ref":"#/components/schemas/IncidentSeverityDTO"},"cloudUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"number":{"type":"integer","format":"int32"},"status":{"type":"object","$ref":"#/components/schemas/IncidentStatusDTO"}},"required":["description","severity"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/incidents/{incidentId}":{"post":{"description":"This endpoint enables you to update an existing incident title, severity, status, or description in your organization's Soda Cloud account.\n\n## Authorization\n\nOnly users with **Manage incident** permission can update incidents. Soda Cloud Admins have permission to update all incidents.**See [Manage dataset roles](https://go.soda.io/roles-dataset) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Incidents`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/incidents/{incidentId}","parameters":[{"in":"path","name":"incidentId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateIncidentDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/IncidentContentDTO"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update incident","tags":["Incidents"]}}}} ``` # Notification Rules Soda Cloud API Notification Rule Endpoints ## List notification rules > This endpoint allows you to list the notification rules in your organization.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Notification Rules\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Notification Rule Endpoints","name":"Notification Rules"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiListNotificationRulesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiNotificationRuleDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"PublicApiNotificationRuleDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"customization":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationDTO"},"filter":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"id":{"type":"string"},"isDefault":{"type":"boolean"},"lastTriggeredAt":{"type":"string","format":"date-time"},"name":{"type":"string"},"ownerId":{"type":"string"},"recipients":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientDTO"}},"ruleType":{"type":"object","$ref":"#/components/schemas/PublicApiRuleTypeDTO"},"status":{"type":"object","$ref":"#/components/schemas/NotificationRuleStatusDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"NotificationCustomizationDTO":{"type":"object","properties":{"message":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationTypeDTO"}},"required":["type"]},"NotificationCustomizationTypeDTO":{"type":"string","enum":["rich","minimal"]},"NotificationFilterDTO":{"type":"object","properties":{"expression":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"expressions":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"}},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationFilterTypeDTO"},"value":{"type":"string"},"values":{"type":"array","items":{"type":"string"}}},"required":["type"]},"NotificationFilterFieldDTO":{"type":"object","properties":{"attributeName":{"type":"string"},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldTypeDTO"}},"required":["field"]},"NotificationFilterFieldTypeDTO":{"type":"string","enum":["datasourceId","datasourceLabel","datasetId","datasetLabel","datasetOwnerId","datasetTags","checkId","checkName","checkOwnerId","columnId","metricType","datasetResourceAttribute","checkResourceAttribute"]},"NotificationFilterTypeDTO":{"type":"string","enum":["equals","contains","in","and","or","not"]},"NotificationRecipientDTO":{"type":"object","properties":{"customRoleId":{"type":"string"},"email":{"type":"string"},"integrationChannel":{"type":"string"},"integrationId":{"type":"string"},"role":{"type":"object","$ref":"#/components/schemas/NotificationRecipientRoleDTO"},"triggers":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTriggerDTO"}},"type":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["triggers","type"]},"NotificationRecipientRoleDTO":{"type":"string","enum":["datasetOwner","monitorOwner"]},"NotificationRecipientTriggerDTO":{"type":"object","properties":{"level":{"type":"object","$ref":"#/components/schemas/NotificationLevelDTO"},"scanState":{"type":"object","$ref":"#/components/schemas/NotificationScanStateDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationTriggerTypeDTO"}},"required":["type"]},"NotificationLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated"]},"NotificationScanStateDTO":{"type":"string","enum":["failed","timedOut","canceled","completedWithErrors"]},"NotificationTriggerTypeDTO":{"type":"string","enum":["testResultLevel","scanState"]},"NotificationRecipientTypeDTO":{"type":"string","enum":["email","user","userGroup","integration","resourceRole","customRole"]},"PublicApiRuleTypeDTO":{"type":"string","enum":["check","monitor"]},"NotificationRuleStatusDTO":{"type":"string","enum":["active","paused"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/notificationRules":{"get":{"description":"This endpoint allows you to list the notification rules in your organization.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Notification Rules`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/notificationRules","parameters":[{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiListNotificationRulesResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List notification rules","tags":["Notification Rules"]}}}} ``` ## Create a notification rule > Creates a notification rule with the provided filter and recipients.\ > \ > The filter is a simplified expression tree (and/or/not/equals/contains/in) over check and monitor attributes; recipients can be users, user groups, emails, integrations, or roles.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_NOTIFICATION\_RULES to execute this call.\ > \ > \## Tags\ > \ > \`Notification Rules\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Notification Rule Endpoints","name":"Notification Rules"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiNotificationRuleRequestDTO":{"type":"object","properties":{"customization":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationDTO"},"filter":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"name":{"type":"string"},"recipients":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientDTO"}},"ruleType":{"type":"object","$ref":"#/components/schemas/PublicApiRuleTypeDTO"},"status":{"type":"object","$ref":"#/components/schemas/NotificationRuleStatusDTO"}},"required":["name","recipients","ruleType"]},"NotificationCustomizationDTO":{"type":"object","properties":{"message":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationTypeDTO"}},"required":["type"]},"NotificationCustomizationTypeDTO":{"type":"string","enum":["rich","minimal"]},"NotificationFilterDTO":{"type":"object","properties":{"expression":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"expressions":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"}},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationFilterTypeDTO"},"value":{"type":"string"},"values":{"type":"array","items":{"type":"string"}}},"required":["type"]},"NotificationFilterFieldDTO":{"type":"object","properties":{"attributeName":{"type":"string"},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldTypeDTO"}},"required":["field"]},"NotificationFilterFieldTypeDTO":{"type":"string","enum":["datasourceId","datasourceLabel","datasetId","datasetLabel","datasetOwnerId","datasetTags","checkId","checkName","checkOwnerId","columnId","metricType","datasetResourceAttribute","checkResourceAttribute"]},"NotificationFilterTypeDTO":{"type":"string","enum":["equals","contains","in","and","or","not"]},"NotificationRecipientDTO":{"type":"object","properties":{"customRoleId":{"type":"string"},"email":{"type":"string"},"integrationChannel":{"type":"string"},"integrationId":{"type":"string"},"role":{"type":"object","$ref":"#/components/schemas/NotificationRecipientRoleDTO"},"triggers":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTriggerDTO"}},"type":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["triggers","type"]},"NotificationRecipientRoleDTO":{"type":"string","enum":["datasetOwner","monitorOwner"]},"NotificationRecipientTriggerDTO":{"type":"object","properties":{"level":{"type":"object","$ref":"#/components/schemas/NotificationLevelDTO"},"scanState":{"type":"object","$ref":"#/components/schemas/NotificationScanStateDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationTriggerTypeDTO"}},"required":["type"]},"NotificationLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated"]},"NotificationScanStateDTO":{"type":"string","enum":["failed","timedOut","canceled","completedWithErrors"]},"NotificationTriggerTypeDTO":{"type":"string","enum":["testResultLevel","scanState"]},"NotificationRecipientTypeDTO":{"type":"string","enum":["email","user","userGroup","integration","resourceRole","customRole"]},"PublicApiRuleTypeDTO":{"type":"string","enum":["check","monitor"]},"NotificationRuleStatusDTO":{"type":"string","enum":["active","paused"]},"PublicApiNotificationRuleDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"customization":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationDTO"},"filter":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"id":{"type":"string"},"isDefault":{"type":"boolean"},"lastTriggeredAt":{"type":"string","format":"date-time"},"name":{"type":"string"},"ownerId":{"type":"string"},"recipients":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientDTO"}},"ruleType":{"type":"object","$ref":"#/components/schemas/PublicApiRuleTypeDTO"},"status":{"type":"object","$ref":"#/components/schemas/NotificationRuleStatusDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/notificationRules":{"post":{"description":"Creates a notification rule with the provided filter and recipients.\n\nThe filter is a simplified expression tree (and/or/not/equals/contains/in) over check and monitor attributes; recipients can be users, user groups, emails, integrations, or roles.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_NOTIFICATION_RULES to execute this call.\n\n## Tags\n\n`Notification Rules`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/notificationRules","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiNotificationRuleRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiNotificationRuleDTO"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create a notification rule","tags":["Notification Rules"]}}}} ``` ## Update a notification rule > Updates a notification rule. This is a full replacement: the rule's name, type, filter, recipients, status, and customization are all set from the request body.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_NOTIFICATION\_RULES to execute this call.\ > \ > \## Tags\ > \ > \`Notification Rules\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Notification Rule Endpoints","name":"Notification Rules"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiNotificationRuleRequestDTO":{"type":"object","properties":{"customization":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationDTO"},"filter":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"name":{"type":"string"},"recipients":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientDTO"}},"ruleType":{"type":"object","$ref":"#/components/schemas/PublicApiRuleTypeDTO"},"status":{"type":"object","$ref":"#/components/schemas/NotificationRuleStatusDTO"}},"required":["name","recipients","ruleType"]},"NotificationCustomizationDTO":{"type":"object","properties":{"message":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationTypeDTO"}},"required":["type"]},"NotificationCustomizationTypeDTO":{"type":"string","enum":["rich","minimal"]},"NotificationFilterDTO":{"type":"object","properties":{"expression":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"expressions":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"}},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationFilterTypeDTO"},"value":{"type":"string"},"values":{"type":"array","items":{"type":"string"}}},"required":["type"]},"NotificationFilterFieldDTO":{"type":"object","properties":{"attributeName":{"type":"string"},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldTypeDTO"}},"required":["field"]},"NotificationFilterFieldTypeDTO":{"type":"string","enum":["datasourceId","datasourceLabel","datasetId","datasetLabel","datasetOwnerId","datasetTags","checkId","checkName","checkOwnerId","columnId","metricType","datasetResourceAttribute","checkResourceAttribute"]},"NotificationFilterTypeDTO":{"type":"string","enum":["equals","contains","in","and","or","not"]},"NotificationRecipientDTO":{"type":"object","properties":{"customRoleId":{"type":"string"},"email":{"type":"string"},"integrationChannel":{"type":"string"},"integrationId":{"type":"string"},"role":{"type":"object","$ref":"#/components/schemas/NotificationRecipientRoleDTO"},"triggers":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTriggerDTO"}},"type":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["triggers","type"]},"NotificationRecipientRoleDTO":{"type":"string","enum":["datasetOwner","monitorOwner"]},"NotificationRecipientTriggerDTO":{"type":"object","properties":{"level":{"type":"object","$ref":"#/components/schemas/NotificationLevelDTO"},"scanState":{"type":"object","$ref":"#/components/schemas/NotificationScanStateDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationTriggerTypeDTO"}},"required":["type"]},"NotificationLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated"]},"NotificationScanStateDTO":{"type":"string","enum":["failed","timedOut","canceled","completedWithErrors"]},"NotificationTriggerTypeDTO":{"type":"string","enum":["testResultLevel","scanState"]},"NotificationRecipientTypeDTO":{"type":"string","enum":["email","user","userGroup","integration","resourceRole","customRole"]},"PublicApiRuleTypeDTO":{"type":"string","enum":["check","monitor"]},"NotificationRuleStatusDTO":{"type":"string","enum":["active","paused"]},"PublicApiNotificationRuleDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"customization":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationDTO"},"filter":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"id":{"type":"string"},"isDefault":{"type":"boolean"},"lastTriggeredAt":{"type":"string","format":"date-time"},"name":{"type":"string"},"ownerId":{"type":"string"},"recipients":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientDTO"}},"ruleType":{"type":"object","$ref":"#/components/schemas/PublicApiRuleTypeDTO"},"status":{"type":"object","$ref":"#/components/schemas/NotificationRuleStatusDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/notificationRules/{notificationRuleId}":{"post":{"description":"Updates a notification rule. This is a full replacement: the rule's name, type, filter, recipients, status, and customization are all set from the request body.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_NOTIFICATION_RULES to execute this call.\n\n## Tags\n\n`Notification Rules`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/notificationRules/{notificationRuleId}","parameters":[{"in":"path","name":"notificationRuleId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiNotificationRuleRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiNotificationRuleDTO"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update a notification rule","tags":["Notification Rules"]}}}} ``` ## Delete a notification rule > Schedules a notification rule for deletion. Deletion is processed asynchronously.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_NOTIFICATION\_RULES to execute this call.\ > \ > \## Tags\ > \ > \`Notification Rules\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Notification Rule Endpoints","name":"Notification Rules"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteNotificationRuleResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/notificationRules/{notificationRuleId}":{"delete":{"description":"Schedules a notification rule for deletion. Deletion is processed asynchronously.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_NOTIFICATION_RULES to execute this call.\n\n## Tags\n\n`Notification Rules`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"DELETE/api/v1/notificationRules/{notificationRuleId}","parameters":[{"in":"path","name":"notificationRuleId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteNotificationRuleResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete a notification rule","tags":["Notification Rules"]}}}} ``` # Runners Soda Cloud API Runner Endpoints ## List runners > This endpoint allows you to list runners in your organization.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Runners\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Runner Endpoints","name":"Runners"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiListRunnersResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"PublicApiRunnerDTO":{"type":"object","properties":{"id":{"type":"string"},"isOnline":{"type":"boolean"},"label":{"type":"string"},"lastSeenTimestamp":{"type":"string","format":"date-time"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerTypeDTO"},"versions":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerVersionsDTO"}}},"PublicApiRunnerTypeDTO":{"type":"string","enum":["SELF_HOSTED","SODA_HOSTED"]},"PublicApiRunnerVersionsDTO":{"type":"object","properties":{"agent":{"type":"string","deprecated":true},"library":{"type":"string"},"runner":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/runners":{"get":{"description":"This endpoint allows you to list runners in your organization.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Runners`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/runners","parameters":[{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiListRunnersResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List runners","tags":["Runners"]}}}} ``` ## Create a soda runner > Creates an API key for a new self-hosted runner deployment.\ > \ > Returns the API key credentials (id and secret) to configure the runner. The credentials are only returned at creation time and cannot be retrieved later, so make sure to store them securely.\ > \ > After creation the runner does not immediately become visible in Soda Cloud. The runner needs to registers itself using the API key credentials before it becomes available.\ > \ > See \[deployment instructions]\() for setting up the runner.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATASOURCES\_AND\_AGENTS to execute this call.\ > \ > \## Tags\ > \ > \`Runners\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Runner Endpoints","name":"Runners"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiCreateRunnerResponse":{"type":"object","properties":{"apiKeyId":{"type":"string"},"apiKeySecret":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/runners":{"post":{"description":"Creates an API key for a new self-hosted runner deployment.\n\nReturns the API key credentials (id and secret) to configure the runner. The credentials are only returned at creation time and cannot be retrieved later, so make sure to store them securely.\n\nAfter creation the runner does not immediately become visible in Soda Cloud. The runner needs to registers itself using the API key credentials before it becomes available.\n\nSee [deployment instructions](https://go.soda.io/agent) for setting up the runner.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATASOURCES_AND_AGENTS to execute this call.\n\n## Tags\n\n`Runners`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/runners","responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateRunnerResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create a soda runner","tags":["Runners"]}}}} ``` ## Get a runner > This endpoint allows you to retrieve a specific runner by its ID.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Tags\ > \ > \`Runners\`\ > \ > \## Rate limiting\ > \ > 1000 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Runner Endpoints","name":"Runners"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiGetRunnerResponse":{"type":"object","properties":{"runner":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerDTO"}}},"PublicApiRunnerDTO":{"type":"object","properties":{"id":{"type":"string"},"isOnline":{"type":"boolean"},"label":{"type":"string"},"lastSeenTimestamp":{"type":"string","format":"date-time"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerTypeDTO"},"versions":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerVersionsDTO"}}},"PublicApiRunnerTypeDTO":{"type":"string","enum":["SELF_HOSTED","SODA_HOSTED"]},"PublicApiRunnerVersionsDTO":{"type":"object","properties":{"agent":{"type":"string","deprecated":true},"library":{"type":"string"},"runner":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/runners/{runnerId}":{"get":{"description":"This endpoint allows you to retrieve a specific runner by its ID.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Tags\n\n`Runners`\n\n## Rate limiting\n\n1000 requests/60 seconds","operationId":"GET/api/v1/runners/{runnerId}","parameters":[{"in":"path","name":"runnerId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiGetRunnerResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get a runner","tags":["Runners"]}}}} ``` ## Delete a runner > This endpoint allows you to delete a specific self-hosted runner by its ID. If the runner is still in use, the request will fail with a \`400 Bad Request\` response. \ > \ > Soda-hosted and virtual runners cannot be deleted through this endpoint.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATASOURCES\_AND\_AGENTS to execute this call.\ > \ > \## Tags\ > \ > \`Runners\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Runner Endpoints","name":"Runners"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteRunnerResponse":{"type":"object"},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/runners/{runnerId}":{"delete":{"description":"This endpoint allows you to delete a specific self-hosted runner by its ID. If the runner is still in use, the request will fail with a `400 Bad Request` response. \n\nSoda-hosted and virtual runners cannot be deleted through this endpoint.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATASOURCES_AND_AGENTS to execute this call.\n\n## Tags\n\n`Runners`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"DELETE/api/v1/runners/{runnerId}","parameters":[{"in":"path","name":"runnerId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteRunnerResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete a runner","tags":["Runners"]}}}} ``` # Scans Soda Cloud API Scan Endpoints ## Get scan status > This endpoint enables you to check on the state of a scan that you executed using the \*\*Trigger a scan\*\* endpoint. Call this endpoint to monitor the status of a scan during its execution.\ > \ > If you wish to access the logs of a completed scan, use the \*\*Get scan logs\*\* endpoint.\ > \ > This POST uses the following parameter to provide specific details:\ > \ > \- \`scanId\`: Use the value of \`X-Soda-Scan-Id\` returned as part of the \`201\` response when you called the \*\*Trigger a scan\*\* endpoint.\ > \ > As a scan executes, you can call this endpoint to progressively collect values based on the state of the scan. Refer to the list below for the states that calls to this endpoint return.\ > \ > \- \`queuing\`: The scan is in the queue for execution, awaiting a pick-up from a Soda Runner.\ > \ > \- \`executing\`: A Soda Runner has picked up the scan and is executing.\ > \ > \- \`cancelationRequested\`: An entity requested cancelation of this scan and the request is awaiting pick-up from the Soda Runner responsible for the scan.\ > \ > \- \`timeOutRequested\`: A time out has been detected, and an automatic request to stop the scan execution is awaiting pick-up from the Soda Runner responsible for the scan.\ > \ > \- \`canceled\`: A Soda Runner confirmed that the scan has been cancelled. This is the final state of the scan.\ > \ > \- \`timedOut\`: A Soda Runner confirmed that the scan has timed-out. This is the final state of the scan.\ > \ > \- \`failed\`: The scan did not start, or it did not successfully complete because of an unexpected cause. This is the final state of the scan.\ > \ > \- \`completedWithErrors\`: The scan completed successfully, but there were errors involving some of the checks in the scan. This is the final state of the scan.\ > \ > \- \`completedWithFailures\`: The scan completed successfully and reveals failed results for some checks. This is the final state of the scan.\ > \ > \- \`completedWithWarnings\`: The scan completed successfully and reveals warning results for some checks. This is the final state of the scan.\ > \ > \- \`completed\`: The scan completed successfully and reveals passing results for all checks. This is a final state of a scan\ > \ > To get the logs of the completed scan, please use API \`/api/v1/scans/{scanId}/logs\`.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Any Soda Cloud user in your organization may execute this query.\ > \ > \## Tags\ > \ > \`Scans\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Scan Endpoints","name":"Scans"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiScanStatusResponse":{"type":"object","properties":{"agentId":{"type":"string","deprecated":true},"checks":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiCheckSlimDTO"}},"cloudUrl":{"type":"string"},"contractDatasetCloudUrl":{"type":"string"},"created":{"type":"string","format":"date-time"},"ended":{"type":"string","format":"date-time"},"errors":{"type":"integer","format":"int32"},"failures":{"type":"integer","format":"int32"},"id":{"type":"string"},"postProcessingStages":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiPostProcessingStageDTO"}},"runnerId":{"type":"string"},"scanDefinition":{"type":"object","$ref":"#/components/schemas/PublicApiScanDefinitionSlim"},"scanTime":{"type":"string","format":"date-time"},"started":{"type":"string","format":"date-time"},"state":{"type":"object","$ref":"#/components/schemas/ScanStateDTO"},"submitted":{"type":"string","format":"date-time"},"warnings":{"type":"integer","format":"int32"}},"required":["cloudUrl","created","id","state"]},"PublicApiCheckSlimDTO":{"type":"object","properties":{"evaluationStatus":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"id":{"type":"string"}}},"EvaluationStatusDTO":{"type":"string","enum":["pass","warn","fail","notEvaluated","excluded"]},"PublicApiPostProcessingStageDTO":{"type":"object","properties":{"ended":{"type":"string","format":"date-time"},"name":{"type":"object","$ref":"#/components/schemas/PublicApiPostProcessingStageTypeDTO"},"started":{"type":"string","format":"date-time"},"state":{"type":"object","$ref":"#/components/schemas/PublicApiPostProcessingStageStateDTO"}},"required":["name"]},"PublicApiPostProcessingStageTypeDTO":{"type":"string","enum":["diagnosticWarehouse"]},"PublicApiPostProcessingStageStateDTO":{"type":"string","enum":["ongoing","completed","failed"]},"PublicApiScanDefinitionSlim":{"type":"object","properties":{"id":{"type":"string"},"name":{"type":"string"}},"required":["id","name"]},"ScanStateDTO":{"type":"string","enum":["queuing","executing","started","cancelationRequested","timeOutRequested","canceled","timedOut","failed","completedWithErrors","completedWithFailures","completedWithWarnings","completed"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/scans/{scanId}":{"get":{"description":"This endpoint enables you to check on the state of a scan that you executed using the **Trigger a scan** endpoint. Call this endpoint to monitor the status of a scan during its execution.\n\nIf you wish to access the logs of a completed scan, use the **Get scan logs** endpoint.\n\nThis POST uses the following parameter to provide specific details:\n\n- `scanId`: Use the value of `X-Soda-Scan-Id` returned as part of the `201` response when you called the **Trigger a scan** endpoint.\n\nAs a scan executes, you can call this endpoint to progressively collect values based on the state of the scan. Refer to the list below for the states that calls to this endpoint return.\n\n- `queuing`: The scan is in the queue for execution, awaiting a pick-up from a Soda Runner.\n\n- `executing`: A Soda Runner has picked up the scan and is executing.\n\n- `cancelationRequested`: An entity requested cancelation of this scan and the request is awaiting pick-up from the Soda Runner responsible for the scan.\n\n- `timeOutRequested`: A time out has been detected, and an automatic request to stop the scan execution is awaiting pick-up from the Soda Runner responsible for the scan.\n\n- `canceled`: A Soda Runner confirmed that the scan has been cancelled. This is the final state of the scan.\n\n- `timedOut`: A Soda Runner confirmed that the scan has timed-out. This is the final state of the scan.\n\n- `failed`: The scan did not start, or it did not successfully complete because of an unexpected cause. This is the final state of the scan.\n\n- `completedWithErrors`: The scan completed successfully, but there were errors involving some of the checks in the scan. This is the final state of the scan.\n\n- `completedWithFailures`: The scan completed successfully and reveals failed results for some checks. This is the final state of the scan.\n\n- `completedWithWarnings`: The scan completed successfully and reveals warning results for some checks. This is the final state of the scan.\n\n- `completed`: The scan completed successfully and reveals passing results for all checks. This is a final state of a scan\n\nTo get the logs of the completed scan, please use API `/api/v1/scans/{scanId}/logs`.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Scans`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/scans/{scanId}","parameters":[{"in":"path","name":"scanId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiScanStatusResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get scan status","tags":["Scans"]}}}} ``` ## Cancel a scan > This endpoint enables you to cancel a scan.\ > \ > Depending on the state of the scan when you call this endpoint, the response returns one of the following:\ > \ > \- Where the state is \`pending\`, Soda immediately changes the state to \`canceled\`.\ > \ > \- Where the state is \`submitted\`, Soda immediately changes the state to \`cancelationRequested\`. - Where the scan is in any other state, the endpoint returns a \`400\` (Bad request) response.\ > \ > This DELETE uses the following parameters to provide specific details:\ > \ > \- \`scanId\`: Use the value of \`X-Soda-Scan-Id\` returned as part of the \`201\` response when you called the \*\*Trigger a scan\*\* endpoint.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Any Soda Cloud user in your organization may execute this query.\ > \ > \## Tags\ > \ > \`Scans\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Scan Endpoints","name":"Scans"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/scans/{scanId}":{"delete":{"description":"This endpoint enables you to cancel a scan.\n\nDepending on the state of the scan when you call this endpoint, the response returns one of the following:\n\n- Where the state is `pending`, Soda immediately changes the state to `canceled`.\n\n- Where the state is `submitted`, Soda immediately changes the state to `cancelationRequested`. - Where the scan is in any other state, the endpoint returns a `400` (Bad request) response.\n\nThis DELETE uses the following parameters to provide specific details:\n\n- `scanId`: Use the value of `X-Soda-Scan-Id` returned as part of the `201` response when you called the **Trigger a scan** endpoint.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Scans`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"DELETE/api/v1/scans/{scanId}","parameters":[{"in":"path","name":"scanId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"description":"Successful response","headers":{"Location":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Cancel a scan","tags":["Scans"]}}}} ``` ## Get scan logs > This endpoint enables you to gather log details about the final state of a scan you executed using the \*\*Trigger a scan\*\* endpoint. Use this endpoint to study scan logs to investigate issues with its execution.\ > \ > If you wish to access the state of a scan in progress, use the \*\*Get scan status\*\* endpoint.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`scanId\`: Use the value of \`X-Soda-Scan-Id\` returned as part of the \`201\` response when you called the \*\*Trigger a scan\*\* endpoint.\ > \ > \- \`size\`: Supply an integer value between 100 and 1000, inclusive. The default value is 1000.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > The response sorts the the log information by creation timestamp in ascending order.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Any Soda Cloud user in your organization may execute this query.\ > \ > \## Tags\ > \ > \`Scans\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Scan Endpoints","name":"Scans"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiScanLogsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/LogsContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"LogsContentDTO":{"type":"object","properties":{"index":{"type":"integer","format":"int32"},"level":{"type":"object","$ref":"#/components/schemas/LogLevelDTO"},"message":{"type":"string"},"timestamp":{"type":"string","format":"date-time"}},"required":["index","level","message","timestamp"]},"LogLevelDTO":{"type":"string","enum":["debug","info","warning","error"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/scans/{scanId}/logs":{"get":{"description":"This endpoint enables you to gather log details about the final state of a scan you executed using the **Trigger a scan** endpoint. Use this endpoint to study scan logs to investigate issues with its execution.\n\nIf you wish to access the state of a scan in progress, use the **Get scan status** endpoint.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `scanId`: Use the value of `X-Soda-Scan-Id` returned as part of the `201` response when you called the **Trigger a scan** endpoint.\n\n- `size`: Supply an integer value between 100 and 1000, inclusive. The default value is 1000.\n\n- `page`: Supply an integer value. The default value is 0.\n\nThe response sorts the the log information by creation timestamp in ascending order.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Scans`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/scans/{scanId}/logs","parameters":[{"in":"path","name":"scanId","required":true,"schema":{"type":"string"}},{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiScanLogsResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get scan logs","tags":["Scans"]}}}} ``` # Secrets Soda Cloud API Secret Endpoints ## List secrets > This endpoint allows you to list secrets in your organization.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \- \`search\`: Supply a string value to filter secrets by name (case-insensitive partial match).\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATASOURCES\_AND\_AGENTS to execute this call.\ > \ > \## Tags\ > \ > \`Secrets\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Secret Endpoints","name":"Secrets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiListSecretsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiSecretDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"PublicApiSecretDTO":{"type":"object","properties":{"created":{"type":"string","format":"date-time"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/secrets":{"get":{"description":"This endpoint allows you to list secrets in your organization.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 10.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `search`: Supply a string value to filter secrets by name (case-insensitive partial match).\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATASOURCES_AND_AGENTS to execute this call.\n\n## Tags\n\n`Secrets`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/secrets","parameters":[{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"search","schema":{"type":"string"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiListSecretsResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"List secrets","tags":["Secrets"]}}}} ``` ## Create a secret > Creates a new secret in your organization. Secrets store encrypted credentials that can be referenced in datasource configurations using \`${secret.NAME}\` placeholders, keeping sensitive values out of plain text.\ > \ > \## Encryption\ > \ > Secret values must be encrypted \*\*client-side\*\* before sending them to this endpoint. This enables a zero-trust design, where Soda never decrypts the secret. Decryption happens only during scan execution, within the runner.\ > \ > Use the \`GET /api/v1/secretsPublicKey\` endpoint to obtain the server's RSA public key. The encryption uses a two-layer scheme:\ > \ > 1\. Fetch the RSA public key from \`GET /api/v1/secretsPublicKey\`.\ > \ > 2\. Generate a random AES-256-GCM key (32 bytes) and initialization vector (IV, 12 bytes).\ > \ > 3\. Encrypt the secret value using AES-256-GCM with the generated key and IV. Base64-encode the ciphertext and prefix it with \`encrypted\_\`. This becomes the \`encryptedValue\` field.\ > \ > 4\. Export the raw AES key and IV as base64 strings and concatenate them as \`{base64\_key}:::{base64\_iv}\`. Encrypt this string using RSA-OAEP (SHA-256) with the public key from step 1. Base64-encode the result. This becomes the \`encryptionKey\` field.\ > \ > \## Constraints\ > \ > \- The secret \`name\` must not contain whitespace and must be unique within the organization.\ > \ > \- Both \`encryptionKey\` and \`encryptedValue\` are required.\ > \ > \- The organization must have the contracts feature enabled and the secure store must not be disabled.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATASOURCES\_AND\_AGENTS to execute this call.\ > \ > \## Tags\ > \ > \`Secrets\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Secret Endpoints","name":"Secrets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiCreateSecretRequestDTO":{"type":"object","properties":{"encryptedValue":{"type":"string"},"encryptionKey":{"type":"string"},"name":{"type":"string"}},"required":["encryptedValue","encryptionKey","name"]},"PublicApiCreateSecretResponse":{"type":"object","properties":{"secret":{"type":"object","$ref":"#/components/schemas/PublicApiSecretDTO"}}},"PublicApiSecretDTO":{"type":"object","properties":{"created":{"type":"string","format":"date-time"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/secrets":{"post":{"description":"Creates a new secret in your organization. Secrets store encrypted credentials that can be referenced in datasource configurations using `${secret.NAME}` placeholders, keeping sensitive values out of plain text.\n\n## Encryption\n\nSecret values must be encrypted **client-side** before sending them to this endpoint. This enables a zero-trust design, where Soda never decrypts the secret. Decryption happens only during scan execution, within the runner.\n\nUse the `GET /api/v1/secretsPublicKey` endpoint to obtain the server's RSA public key. The encryption uses a two-layer scheme:\n\n1. Fetch the RSA public key from `GET /api/v1/secretsPublicKey`.\n\n2. Generate a random AES-256-GCM key (32 bytes) and initialization vector (IV, 12 bytes).\n\n3. Encrypt the secret value using AES-256-GCM with the generated key and IV. Base64-encode the ciphertext and prefix it with `encrypted_`. This becomes the `encryptedValue` field.\n\n4. Export the raw AES key and IV as base64 strings and concatenate them as `{base64_key}:::{base64_iv}`. Encrypt this string using RSA-OAEP (SHA-256) with the public key from step 1. Base64-encode the result. This becomes the `encryptionKey` field.\n\n## Constraints\n\n- The secret `name` must not contain whitespace and must be unique within the organization.\n\n- Both `encryptionKey` and `encryptedValue` are required.\n\n- The organization must have the contracts feature enabled and the secure store must not be disabled.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATASOURCES_AND_AGENTS to execute this call.\n\n## Tags\n\n`Secrets`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/secrets","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateSecretRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateSecretResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create a secret","tags":["Secrets"]}}}} ``` ## Update a secret > Updates the encrypted value of an existing secret. The secret name cannot be changed.\ > \ > Both \`encryptionKey\` and \`encryptedValue\` must be provided together — you cannot update one without the other.\ > \ > \## Encryption\ > \ > The new secret value must be encrypted \*\*client-side\*\* using the same two-layer encryption scheme as creation. See the \`POST /api/v1/secrets\` (Create a secret) endpoint for the full encryption workflow.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATASOURCES\_AND\_AGENTS to execute this call.\ > \ > \## Tags\ > \ > \`Secrets\`\ > \ > \## Rate limiting\ > \ > 100 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Secret Endpoints","name":"Secrets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateSecretRequestDTO":{"type":"object","properties":{"encryptedValue":{"type":"string"},"encryptionKey":{"type":"string"}},"required":["encryptedValue","encryptionKey"]},"PublicApiUpdateSecretResponse":{"type":"object","properties":{"secret":{"type":"object","$ref":"#/components/schemas/PublicApiSecretDTO"}}},"PublicApiSecretDTO":{"type":"object","properties":{"created":{"type":"string","format":"date-time"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/secrets/{secretId}":{"post":{"description":"Updates the encrypted value of an existing secret. The secret name cannot be changed.\n\nBoth `encryptionKey` and `encryptedValue` must be provided together — you cannot update one without the other.\n\n## Encryption\n\nThe new secret value must be encrypted **client-side** using the same two-layer encryption scheme as creation. See the `POST /api/v1/secrets` (Create a secret) endpoint for the full encryption workflow.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATASOURCES_AND_AGENTS to execute this call.\n\n## Tags\n\n`Secrets`\n\n## Rate limiting\n\n100 requests/60 seconds","operationId":"POST/api/v1/secrets/{secretId}","parameters":[{"in":"path","name":"secretId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateSecretRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateSecretResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update a secret","tags":["Secrets"]}}}} ``` ## Delete a secret > Deletes an existing secret.\ > \ > The secret cannot be deleted if it is referenced by a datasource configuration.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATASOURCES\_AND\_AGENTS to execute this call.\ > \ > \## Tags\ > \ > \`Secrets\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Secret Endpoints","name":"Secrets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteSecretResponse":{"type":"object","properties":{"message":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/secrets/{secretId}":{"delete":{"description":"Deletes an existing secret.\n\nThe secret cannot be deleted if it is referenced by a datasource configuration.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATASOURCES_AND_AGENTS to execute this call.\n\n## Tags\n\n`Secrets`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"DELETE/api/v1/secrets/{secretId}","parameters":[{"in":"path","name":"secretId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteSecretResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete a secret","tags":["Secrets"]}}}} ``` ## Get the encryption public key > Returns the RSA public key in JWK (JSON Web Key) format, used for client-side encryption of secret values.\ > \ > \## Usage\ > \ > When creating or updating a secret, the secret value must be encrypted client-side before sending it to the API. This endpoint provides the RSA public key needed for that encryption.\ > \ > \## Encryption workflow\ > \ > 1\. Fetch this public key.\ > \ > 2\. Generate a random AES-256-GCM key (32 bytes) and initialization vector (IV, 12 bytes).\ > \ > 3\. Encrypt the secret value using AES-256-GCM with the generated key and IV. Base64-encode the ciphertext and prefix it with \`encrypted\_\`. This becomes the \`encryptedValue\` field.\ > \ > 4\. Export the raw AES key and IV as base64 strings and concatenate them as \`{base64\_key}:::{base64\_iv}\`. Encrypt this string using RSA-OAEP (SHA-256) with this public key. Base64-encode the result. This becomes the \`encryptionKey\` field.\ > \ > 5\. Use the \`encryptedValue\` and \`encryptionKey\` in the create or update secret request.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_DATASOURCES\_AND\_AGENTS to execute this call.\ > \ > \## Tags\ > \ > \`Secrets\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Secret Endpoints","name":"Secrets"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiGetEncryptionKeyResponse":{"type":"object","properties":{"encryptionKey":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"}}},"MapOfStringTo_object":{"type":"object","additionalProperties":true},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/secretsPublicKey":{"get":{"description":"Returns the RSA public key in JWK (JSON Web Key) format, used for client-side encryption of secret values.\n\n## Usage\n\nWhen creating or updating a secret, the secret value must be encrypted client-side before sending it to the API. This endpoint provides the RSA public key needed for that encryption.\n\n## Encryption workflow\n\n1. Fetch this public key.\n\n2. Generate a random AES-256-GCM key (32 bytes) and initialization vector (IV, 12 bytes).\n\n3. Encrypt the secret value using AES-256-GCM with the generated key and IV. Base64-encode the ciphertext and prefix it with `encrypted_`. This becomes the `encryptedValue` field.\n\n4. Export the raw AES key and IV as base64 strings and concatenate them as `{base64_key}:::{base64_iv}`. Encrypt this string using RSA-OAEP (SHA-256) with this public key. Base64-encode the result. This becomes the `encryptionKey` field.\n\n5. Use the `encryptedValue` and `encryptionKey` in the create or update secret request.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_DATASOURCES_AND_AGENTS to execute this call.\n\n## Tags\n\n`Secrets`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/secretsPublicKey","responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiGetEncryptionKeyResponse"}}},"description":"Successful response"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get the encryption public key","tags":["Secrets"]}}}} ``` # Users Soda Cloud API Users Endpoints ## Get user groups > This endpoint enables you to gather information about the user groups in your organization's Soda Cloud account, including lists of members.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 10 and 1000, inclusive. The default value is 1000.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \- \`search\`: Optionally, use this parameter to perform a fuzzy search on the user group name. \ > \ > If not specified, the query gathers information for all user groups in the account and sorts the results by user group name in ascending order.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Any Soda Cloud user in your organization may execute this query.\ > \ > \## Tags\ > \ > \`Users\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Users Endpoints","name":"Users"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUserGroupsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/UserGroupContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"UserGroupContentDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"},"users":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/UserContentDTO"}}},"required":["name","userGroupId","users"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/userGroups":{"get":{"description":"This endpoint enables you to gather information about the user groups in your organization's Soda Cloud account, including lists of members.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 10 and 1000, inclusive. The default value is 1000.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `search`: Optionally, use this parameter to perform a fuzzy search on the user group name. \n\nIf not specified, the query gathers information for all user groups in the account and sorts the results by user group name in ascending order.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Users`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/userGroups","parameters":[{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"search","schema":{"type":"string"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUserGroupsResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get user groups","tags":["Users"]}}}} ``` ## Create user group > This endpoint enables you to create a new user group in your organization's Soda Cloud account.\ > \ > The Response of this call, when successful, is \`201\` and contains headers \`Location\` which is a reference to the newly created User Group.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage organization settings\*\* permission can update user groups.\*\*See \[Manage global roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ORGANISATION\_SETTINGS to execute this call.\ > \ > \## Tags\ > \ > \`Users\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Users Endpoints","name":"Users"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiCreateUserGroupRequestDTO":{"type":"object","properties":{"name":{"type":"string"},"userIds":{"type":"array","items":{"type":"string"}}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/userGroups":{"post":{"description":"This endpoint enables you to create a new user group in your organization's Soda Cloud account.\n\nThe Response of this call, when successful, is `201` and contains headers `Location` which is a reference to the newly created User Group.\n\n## Authorization\n\nOnly users with **Manage organization settings** permission can update user groups.**See [Manage global roles](https://go.soda.io/roles-global) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ORGANISATION_SETTINGS to execute this call.\n\n## Tags\n\n`Users`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"POST/api/v1/userGroups","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiCreateUserGroupRequestDTO"}}},"required":true},"responses":{"200":{"description":"Successful response","headers":{"Location":{"schema":{"type":"string"}}}},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Create user group","tags":["Users"]}}}} ``` ## Get user group > This endpoint enables you to gather information about a single user group in your organization's Soda Cloud account, including lists of members.\ > \ > This GET is a non-paginated API that returns a single organization based on its ID.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Any Soda Cloud user in your organization may execute this query.\ > \ > \## Tags\ > \ > \`Users\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Users Endpoints","name":"Users"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"UserGroupContentDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"},"users":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/UserContentDTO"}}},"required":["name","userGroupId","users"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/userGroups/{userGroupId}":{"get":{"description":"This endpoint enables you to gather information about a single user group in your organization's Soda Cloud account, including lists of members.\n\nThis GET is a non-paginated API that returns a single organization based on its ID.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Users`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/userGroups/{userGroupId}","parameters":[{"in":"path","name":"userGroupId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/UserGroupContentDTO"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get user group","tags":["Users"]}}}} ``` ## Update user group > This endpoint enables you to update an existing user group in your organization's Soda Cloud account.\ > \ > The Response of this call, when successful, is \`200\` and contains updated User Group details.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage organization settings\*\* permission can update user groups.\*\*See \[Manage global roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ORGANISATION\_SETTINGS to execute this call.\ > \ > \## Tags\ > \ > \`Users\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Users Endpoints","name":"Users"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUpdateUserGroupRequestDTO":{"type":"object","properties":{"userIds":{"type":"array","items":{"type":"string"}}}},"UserGroupContentDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"},"users":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/UserContentDTO"}}},"required":["name","userGroupId","users"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/userGroups/{userGroupId}":{"post":{"description":"This endpoint enables you to update an existing user group in your organization's Soda Cloud account.\n\nThe Response of this call, when successful, is `200` and contains updated User Group details.\n\n## Authorization\n\nOnly users with **Manage organization settings** permission can update user groups.**See [Manage global roles](https://go.soda.io/roles-global) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ORGANISATION_SETTINGS to execute this call.\n\n## Tags\n\n`Users`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"POST/api/v1/userGroups/{userGroupId}","parameters":[{"in":"path","name":"userGroupId","required":true,"schema":{"type":"string"}}],"requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateUserGroupRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/UserGroupContentDTO"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Update user group","tags":["Users"]}}}} ``` ## Delete user group > This endpoint enables you to delete an existing user group from your organization's Soda Cloud account.\ > \ > In case of success, the response returns a \`200\` (OK) response.\ > \ > In case user group is not found, the response returns a \`404\` (Not found) response.\ > \ > \## Authorization\ > \ > Only users with \*\*Manage organization settings\*\* permission can update user groups.\*\*See \[Manage global roles]\() for more information.\*\*\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ORGANISATION\_SETTINGS to execute this call.\ > \ > \## Tags\ > \ > \`Users\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Users Endpoints","name":"Users"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDeleteUserGroupResponse":{"type":"object"},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/userGroups/{userGroupId}":{"delete":{"description":"This endpoint enables you to delete an existing user group from your organization's Soda Cloud account.\n\nIn case of success, the response returns a `200` (OK) response.\n\nIn case user group is not found, the response returns a `404` (Not found) response.\n\n## Authorization\n\nOnly users with **Manage organization settings** permission can update user groups.**See [Manage global roles](https://go.soda.io/roles-global) for more information.**\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ORGANISATION_SETTINGS to execute this call.\n\n## Tags\n\n`Users`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"DELETE/api/v1/userGroups/{userGroupId}","parameters":[{"in":"path","name":"userGroupId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDeleteUserGroupResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Delete user group","tags":["Users"]}}}} ``` ## Get users > This endpoint enables you to gather information about the users in your organization's Soda Cloud account.\ > \ > This GET is a paginated API that uses the following parameters to request specific details:\ > \ > \- \`size\`: Supply an integer value between 1 and 1000, inclusive. The default value is 1000.\ > \ > \- \`page\`: Supply an integer value. The default value is 0.\ > \ > \- \`search\`: Optionally, use this parameter to perform a fuzzy search on the user first name, last name, or email address. \ > \ > If not specified, the query gathers information for all users in the account and sorts the results by user name in ascending order.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Any Soda Cloud user in your organization may execute this query.\ > \ > \## Tags\ > \ > \`Users\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Users Endpoints","name":"Users"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiUsersResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/UserContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/users":{"get":{"description":"This endpoint enables you to gather information about the users in your organization's Soda Cloud account.\n\nThis GET is a paginated API that uses the following parameters to request specific details:\n\n- `size`: Supply an integer value between 1 and 1000, inclusive. The default value is 1000.\n\n- `page`: Supply an integer value. The default value is 0.\n\n- `search`: Optionally, use this parameter to perform a fuzzy search on the user first name, last name, or email address. \n\nIf not specified, the query gathers information for all users in the account and sorts the results by user name in ascending order.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Users`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"GET/api/v1/users","parameters":[{"in":"query","name":"page","schema":{"type":"integer","format":"int32"}},{"in":"query","name":"search","schema":{"type":"string"}},{"in":"query","name":"size","schema":{"type":"integer","format":"int32"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiUsersResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Get users","tags":["Users"]}}}} ``` ## Invite users > This endpoint enables you to invite users to your organization's Soda Cloud account.\ > \ > Invited users will receive an email with a link to set their password and join your organization in Soda Cloud. Once they complete the setup, they will have access to Soda Cloud based on the roles and permissions you assign.\ > \ > If your organization is using SSO, invited users will receive an invitation email with a link to Soda Cloud. When they log in using SSO, they are immediately added to the Soda Cloud organization.\ > \ > You can invite up to 10 users per request. The response contains lists of successful and failed invitations.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ORGANISATION\_SETTINGS to execute this call.\ > \ > \## Tags\ > \ > \`Users\`\ > \ > \## Rate limiting\ > \ > 10 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Users Endpoints","name":"Users"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiInviteUserRequestDTO":{"type":"object","properties":{"emails":{"type":"array","items":{"type":"string"}}},"required":["emails"]},"PublicApiInviteUserResponse":{"type":"object","properties":{"failedInvitations":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/FailedInvitationDTO"}},"validInvitations":{"type":"array","items":{"type":"string"}}}},"FailedInvitationDTO":{"type":"object","properties":{"email":{"type":"string"},"error":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/users":{"post":{"description":"This endpoint enables you to invite users to your organization's Soda Cloud account.\n\nInvited users will receive an email with a link to set their password and join your organization in Soda Cloud. Once they complete the setup, they will have access to Soda Cloud based on the roles and permissions you assign.\n\nIf your organization is using SSO, invited users will receive an invitation email with a link to Soda Cloud. When they log in using SSO, they are immediately added to the Soda Cloud organization.\n\nYou can invite up to 10 users per request. The response contains lists of successful and failed invitations.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ORGANISATION_SETTINGS to execute this call.\n\n## Tags\n\n`Users`\n\n## Rate limiting\n\n10 requests/60 seconds","operationId":"POST/api/v1/users","requestBody":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiInviteUserRequestDTO"}}},"required":true},"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiInviteUserResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Invite users","tags":["Users"]}}}} ``` ## Disable user > This endpoint disables a user in your organization's Soda Cloud account.\ > \ > Disabling a user deactivates their membership in the organization. The user is not deleted and can be reactivated later.\ > \ > Users in SSO-enabled organizations cannot be disabled via this endpoint, as they are managed by the identity provider.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Users must have global role permission MANAGE\_ORGANISATION\_SETTINGS to execute this call.\ > \ > \## Tags\ > \ > \`Users\`\ > \ > \## Rate limiting\ > \ > 60 requests/60 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Users Endpoints","name":"Users"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiDisableUserResponse":{"type":"object"},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/users/{userId}/disable":{"post":{"description":"This endpoint disables a user in your organization's Soda Cloud account.\n\nDisabling a user deactivates their membership in the organization. The user is not deleted and can be reactivated later.\n\nUsers in SSO-enabled organizations cannot be disabled via this endpoint, as they are managed by the identity provider.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nUsers must have global role permission MANAGE_ORGANISATION_SETTINGS to execute this call.\n\n## Tags\n\n`Users`\n\n## Rate limiting\n\n60 requests/60 seconds","operationId":"POST/api/v1/users/{userId}/disable","parameters":[{"in":"path","name":"userId","required":true,"schema":{"type":"string"}}],"responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiDisableUserResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Disable user","tags":["Users"]}}}} ``` # Utility Soda Cloud API Utility Endpoints ## Test connection > This endpoint enables you to test the API connection to your organization's Soda Cloud account. Use this endpoint to verify that the authentication and authorization details are accurate and the API is ready to accept calls.\ > \ > \## Authentication\ > \ > User authentication required: \`true\`\ > \ > This endpoint accepts authentication via API keys in the \`Basic\` authentication header, or a pre-authenticated token in HTTP cookie \`token\`. Cookie sessions extend automatically on each request.\ > \ > \## Authorization\ > \ > Any Soda Cloud user in your organization may execute this query.\ > \ > \## Tags\ > \ > \`Utility\`\ > \ > \## Rate limiting\ > \ > 10 requests/10 seconds ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"tags":[{"description":"Soda Cloud API Utility Endpoints","name":"Utility"}],"servers":[{"description":"Cloud EU","url":"https://cloud.soda.io"},{"description":"Cloud US","url":"https://cloud.us.soda.io"}],"security":[{"basicAuthApiKey":[]},{"cookieToken":[]}],"components":{"securitySchemes":{"basicAuthApiKey":{"scheme":"basic","type":"http"},"cookieToken":{"in":"cookie","name":"token","type":"apiKey"}},"schemas":{"PublicApiTestLoginResponse":{"type":"object","properties":{"organisationName":{"type":"string"}}},"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}},"paths":{"/api/v1/test-login":{"get":{"description":"This endpoint enables you to test the API connection to your organization's Soda Cloud account. Use this endpoint to verify that the authentication and authorization details are accurate and the API is ready to accept calls.\n\n## Authentication\n\nUser authentication required: `true`\n\nThis endpoint accepts authentication via API keys in the `Basic` authentication header, or a pre-authenticated token in HTTP cookie `token`. Cookie sessions extend automatically on each request.\n\n## Authorization\n\nAny Soda Cloud user in your organization may execute this query.\n\n## Tags\n\n`Utility`\n\n## Rate limiting\n\n10 requests/10 seconds","operationId":"GET/api/v1/test-login","responses":{"200":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/PublicApiTestLoginResponse"}}},"description":"Successful response"},"400":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Bad request"},"401":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Unauthorized"},"403":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Forbidden"},"404":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Not found"},"429":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Too many requests"},"500":{"content":{"application/json":{"schema":{"type":"object","$ref":"#/components/schemas/ErrorResponse"}}},"description":"Internal server error"}},"summary":"Test connection","tags":["Utility"]}}}} ``` # Models ## The AggregateDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"AggregateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}}}}} ``` ## The AggregateDiffDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"AggregateDiffDiagnosticsV4":{"type":"object","properties":{"diff":{"type":"number"},"diffPercent":{"type":"number"},"function":{"type":"string"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceValue":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetValue":{"type":"number"}}}}}} ``` ## The AgreementSlimDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"AgreementSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"name":{"type":"string"}}}}}} ``` ## The AttributeContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"AttributeContentDTO":{"type":"object","properties":{"allowedValues":{"type":"array","items":{"type":"string"}},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"resourceType":{"type":"object","$ref":"#/components/schemas/AttributeResourceTypeDTO"},"type":{"type":"object","$ref":"#/components/schemas/AttributeTypeDTO"}},"required":["id","label","name","resourceType","type"]},"AttributeResourceTypeDTO":{"type":"string","enum":["dataset","check","column"]},"AttributeTypeDTO":{"type":"string","enum":["text","number","checkbox","datetime","singleSelect","multiSelect"]}}}} ``` ## The AttributeResourceTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"AttributeResourceTypeDTO":{"type":"string","enum":["dataset","check","column"]}}}} ``` ## The AttributeTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"AttributeTypeDTO":{"type":"string","enum":["text","number","checkbox","datetime","singleSelect","multiSelect"]}}}} ``` ## The CheckResultAnomalyDetectionDetailsDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"CheckResultAnomalyDetectionDetailsDTO":{"type":"object","properties":{"predictedValue":{"type":"number"}}}}}} ``` ## The CheckResultDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"CheckResultDiagnosticsV4":{"type":"object","properties":{"aggregate":{"type":"object","$ref":"#/components/schemas/AggregateDiagnosticsV4"},"aggregate_diff":{"type":"object","$ref":"#/components/schemas/AggregateDiffDiagnosticsV4"},"duplicate":{"type":"object","$ref":"#/components/schemas/DuplicateDiagnosticsV4"},"duplicate_diff":{"type":"object","$ref":"#/components/schemas/DuplicateDiffDiagnosticsV4"},"failed_rows":{"type":"object","$ref":"#/components/schemas/FailedRowsDiagnosticsV4"},"freshness":{"type":"object","$ref":"#/components/schemas/FreshnessDiagnosticsV4"},"freshness_diff":{"type":"object","$ref":"#/components/schemas/FreshnessDiffDiagnosticsV4"},"fuzzy_duplicate":{"type":"object","$ref":"#/components/schemas/FuzzyDuplicateDiagnosticsV4"},"group_by":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4"},"hierarchy":{"type":"object","$ref":"#/components/schemas/HierarchyDiagnosticsV4"},"invalid":{"type":"object","$ref":"#/components/schemas/InvalidDiagnosticsV4"},"llm":{"type":"object","$ref":"#/components/schemas/LlmDiagnosticsV4"},"metric":{"type":"object","$ref":"#/components/schemas/MetricDiagnosticsV4"},"metric_diff":{"type":"object","$ref":"#/components/schemas/MetricDiffDiagnosticsV4"},"missing":{"type":"object","$ref":"#/components/schemas/MissingDiagnosticsV4"},"reference_diff":{"type":"object","$ref":"#/components/schemas/ReferenceDiffDiagnosticsV4"},"row_count":{"type":"object","$ref":"#/components/schemas/RowCountDiagnosticsV4"},"row_count_diff":{"type":"object","$ref":"#/components/schemas/RowCountDiffDiagnosticsV4"},"rows_diff":{"type":"object","$ref":"#/components/schemas/RowsDiffDiagnosticsV4"},"schema":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4"}}},"AggregateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}},"AggregateDiffDiagnosticsV4":{"type":"object","properties":{"diff":{"type":"number"},"diffPercent":{"type":"number"},"function":{"type":"string"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceValue":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetValue":{"type":"number"}}},"DuplicateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}},"DuplicateDiffDiagnosticsV4":{"type":"object","properties":{"duplicateCountDiff":{"type":"integer","format":"int32"},"duplicateDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceDuplicateCount":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetDuplicateCount":{"type":"integer","format":"int32"}}},"FailedRowsDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}},"FreshnessDiagnosticsV4":{"type":"object","properties":{"actualTimestamp":{"type":"string","format":"date-time"},"actualTimestampUtc":{"type":"string","format":"date-time"},"datasetRowsTested":{"type":"integer","format":"int32"},"expectedTimestamp":{"type":"string","format":"date-time"},"expectedTimestampUtc":{"type":"string","format":"date-time"}}},"FreshnessDiffDiagnosticsV4":{"type":"object","properties":{"freshnessDiff":{"type":"integer","format":"int32"},"freshnessDiffUnit":{"type":"object","$ref":"#/components/schemas/FreshnessDiffDiagnosticsV4_UnitOfTimeDTO"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMaxTimestamp":{"type":"string","format":"date-time"},"sourceMaxTimestampUtc":{"type":"string","format":"date-time"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMaxTimestamp":{"type":"string","format":"date-time"},"targetMaxTimestampUtc":{"type":"string","format":"date-time"}}},"FreshnessDiffDiagnosticsV4_UnitOfTimeDTO":{"type":"string","enum":["minutes","hours","days"]},"FuzzyDuplicateDiagnosticsV4":{"type":"object","properties":{"candidateRowsScored":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"clusterCount":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"fuzzyDuplicateCount":{"type":"integer","format":"int32"},"fuzzyDuplicatePercent":{"type":"number"},"meanConfidence":{"type":"number"}}},"GroupByDiagnosticsV4":{"type":"object","properties":{"diagnosticsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"groups":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_Group"}}}},"GroupByDiagnosticsV4_Group":{"type":"object","properties":{"group":{"type":"string"},"rowsTested":{"type":"number"},"status":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_GroupByOutcomeDTO"},"value":{"type":"number"}}},"GroupByDiagnosticsV4_GroupByOutcomeDTO":{"type":"string","enum":["pass","warn","fail"]},"HierarchyDiagnosticsV4":{"type":"object","properties":{"brokenPathCount":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"invalidValueCount":{"type":"integer","format":"int32"},"missingValueCount":{"type":"integer","format":"int32"}}},"InvalidDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}},"LlmDiagnosticsV4":{"type":"object","properties":{"completionTokens":{"type":"number"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"model":{"type":"string"},"promptTokens":{"type":"number"},"rowsChecked":{"type":"integer","format":"int32"},"rowsFailed":{"type":"integer","format":"int32"},"rowsFailedPercent":{"type":"number"},"tokensSpent":{"type":"integer","format":"int32"}}},"MetricDiagnosticsV4":{"type":"object","properties":{"datasetRowsTested":{"type":"integer","format":"int32"}}},"MetricDiffDiagnosticsV4":{"type":"object","properties":{"metricDiff":{"type":"number"},"metricDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMetric":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMetric":{"type":"number"}}},"MissingDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}},"ReferenceDiffDiagnosticsV4":{"type":"object","properties":{"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"RowCountDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}},"RowCountDiffDiagnosticsV4":{"type":"object","properties":{"rowCountDiff":{"type":"integer","format":"int32"},"rowCountDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"RowsDiffDiagnosticsV4":{"type":"object","properties":{"changed":{"type":"integer","format":"int32"},"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"SchemaDiagnosticsV4":{"type":"object","properties":{"actual":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}},"expected":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}}}},"SchemaDiagnosticsV4_Column":{"type":"object","properties":{"name":{"type":"string"},"type":{"type":"string"}}}}}} ``` ## The CheckResultLevelDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"CheckResultLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated","excluded"]}}}} ``` ## The CheckResultValueDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"CheckResultValueDTO":{"type":"object","properties":{"anomalyDetectionDetails":{"type":"object","$ref":"#/components/schemas/CheckResultAnomalyDetectionDetailsDTO"},"dataTimestamp":{"type":"string","format":"date-time"},"diagnostics":{"type":"object","$ref":"#/components/schemas/CheckResultDiagnosticsV4"},"id":{"type":"string"},"measure":{"type":"object","$ref":"#/components/schemas/MeasureDTO"},"value":{"type":"number"},"valueLabel":{"type":"string"},"valueSeries":{"type":"object","$ref":"#/components/schemas/CheckResultValueSeriesDTO"}},"required":["id"]},"CheckResultAnomalyDetectionDetailsDTO":{"type":"object","properties":{"predictedValue":{"type":"number"}}},"CheckResultDiagnosticsV4":{"type":"object","properties":{"aggregate":{"type":"object","$ref":"#/components/schemas/AggregateDiagnosticsV4"},"aggregate_diff":{"type":"object","$ref":"#/components/schemas/AggregateDiffDiagnosticsV4"},"duplicate":{"type":"object","$ref":"#/components/schemas/DuplicateDiagnosticsV4"},"duplicate_diff":{"type":"object","$ref":"#/components/schemas/DuplicateDiffDiagnosticsV4"},"failed_rows":{"type":"object","$ref":"#/components/schemas/FailedRowsDiagnosticsV4"},"freshness":{"type":"object","$ref":"#/components/schemas/FreshnessDiagnosticsV4"},"freshness_diff":{"type":"object","$ref":"#/components/schemas/FreshnessDiffDiagnosticsV4"},"fuzzy_duplicate":{"type":"object","$ref":"#/components/schemas/FuzzyDuplicateDiagnosticsV4"},"group_by":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4"},"hierarchy":{"type":"object","$ref":"#/components/schemas/HierarchyDiagnosticsV4"},"invalid":{"type":"object","$ref":"#/components/schemas/InvalidDiagnosticsV4"},"llm":{"type":"object","$ref":"#/components/schemas/LlmDiagnosticsV4"},"metric":{"type":"object","$ref":"#/components/schemas/MetricDiagnosticsV4"},"metric_diff":{"type":"object","$ref":"#/components/schemas/MetricDiffDiagnosticsV4"},"missing":{"type":"object","$ref":"#/components/schemas/MissingDiagnosticsV4"},"reference_diff":{"type":"object","$ref":"#/components/schemas/ReferenceDiffDiagnosticsV4"},"row_count":{"type":"object","$ref":"#/components/schemas/RowCountDiagnosticsV4"},"row_count_diff":{"type":"object","$ref":"#/components/schemas/RowCountDiffDiagnosticsV4"},"rows_diff":{"type":"object","$ref":"#/components/schemas/RowsDiffDiagnosticsV4"},"schema":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4"}}},"AggregateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}},"AggregateDiffDiagnosticsV4":{"type":"object","properties":{"diff":{"type":"number"},"diffPercent":{"type":"number"},"function":{"type":"string"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceValue":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetValue":{"type":"number"}}},"DuplicateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}},"DuplicateDiffDiagnosticsV4":{"type":"object","properties":{"duplicateCountDiff":{"type":"integer","format":"int32"},"duplicateDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceDuplicateCount":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetDuplicateCount":{"type":"integer","format":"int32"}}},"FailedRowsDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}},"FreshnessDiagnosticsV4":{"type":"object","properties":{"actualTimestamp":{"type":"string","format":"date-time"},"actualTimestampUtc":{"type":"string","format":"date-time"},"datasetRowsTested":{"type":"integer","format":"int32"},"expectedTimestamp":{"type":"string","format":"date-time"},"expectedTimestampUtc":{"type":"string","format":"date-time"}}},"FreshnessDiffDiagnosticsV4":{"type":"object","properties":{"freshnessDiff":{"type":"integer","format":"int32"},"freshnessDiffUnit":{"type":"object","$ref":"#/components/schemas/FreshnessDiffDiagnosticsV4_UnitOfTimeDTO"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMaxTimestamp":{"type":"string","format":"date-time"},"sourceMaxTimestampUtc":{"type":"string","format":"date-time"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMaxTimestamp":{"type":"string","format":"date-time"},"targetMaxTimestampUtc":{"type":"string","format":"date-time"}}},"FreshnessDiffDiagnosticsV4_UnitOfTimeDTO":{"type":"string","enum":["minutes","hours","days"]},"FuzzyDuplicateDiagnosticsV4":{"type":"object","properties":{"candidateRowsScored":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"clusterCount":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"fuzzyDuplicateCount":{"type":"integer","format":"int32"},"fuzzyDuplicatePercent":{"type":"number"},"meanConfidence":{"type":"number"}}},"GroupByDiagnosticsV4":{"type":"object","properties":{"diagnosticsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"groups":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_Group"}}}},"GroupByDiagnosticsV4_Group":{"type":"object","properties":{"group":{"type":"string"},"rowsTested":{"type":"number"},"status":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_GroupByOutcomeDTO"},"value":{"type":"number"}}},"GroupByDiagnosticsV4_GroupByOutcomeDTO":{"type":"string","enum":["pass","warn","fail"]},"HierarchyDiagnosticsV4":{"type":"object","properties":{"brokenPathCount":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"invalidValueCount":{"type":"integer","format":"int32"},"missingValueCount":{"type":"integer","format":"int32"}}},"InvalidDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}},"LlmDiagnosticsV4":{"type":"object","properties":{"completionTokens":{"type":"number"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"model":{"type":"string"},"promptTokens":{"type":"number"},"rowsChecked":{"type":"integer","format":"int32"},"rowsFailed":{"type":"integer","format":"int32"},"rowsFailedPercent":{"type":"number"},"tokensSpent":{"type":"integer","format":"int32"}}},"MetricDiagnosticsV4":{"type":"object","properties":{"datasetRowsTested":{"type":"integer","format":"int32"}}},"MetricDiffDiagnosticsV4":{"type":"object","properties":{"metricDiff":{"type":"number"},"metricDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMetric":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMetric":{"type":"number"}}},"MissingDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}},"ReferenceDiffDiagnosticsV4":{"type":"object","properties":{"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"RowCountDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}},"RowCountDiffDiagnosticsV4":{"type":"object","properties":{"rowCountDiff":{"type":"integer","format":"int32"},"rowCountDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"RowsDiffDiagnosticsV4":{"type":"object","properties":{"changed":{"type":"integer","format":"int32"},"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"SchemaDiagnosticsV4":{"type":"object","properties":{"actual":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}},"expected":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}}}},"SchemaDiagnosticsV4_Column":{"type":"object","properties":{"name":{"type":"string"},"type":{"type":"string"}}},"MeasureDTO":{"type":"string","enum":["time"]},"CheckResultValueSeriesDTO":{"type":"object","properties":{"values":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/CheckResultValueSeriesDTO_ValueSeriesValue"}}}},"CheckResultValueSeriesDTO_ValueSeriesValue":{"type":"object","properties":{"label":{"type":"string"},"level":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"value":{"type":"number"}}},"EvaluationStatusDTO":{"type":"string","enum":["pass","warn","fail","notEvaluated","excluded"]}}}} ``` ## The CheckResultValueSeriesDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"CheckResultValueSeriesDTO":{"type":"object","properties":{"values":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/CheckResultValueSeriesDTO_ValueSeriesValue"}}}},"CheckResultValueSeriesDTO_ValueSeriesValue":{"type":"object","properties":{"label":{"type":"string"},"level":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"value":{"type":"number"}}},"EvaluationStatusDTO":{"type":"string","enum":["pass","warn","fail","notEvaluated","excluded"]}}}} ``` ## The CheckResultValueSeriesDTO\_ValueSeriesValue object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"CheckResultValueSeriesDTO_ValueSeriesValue":{"type":"object","properties":{"label":{"type":"string"},"level":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"value":{"type":"number"}}},"EvaluationStatusDTO":{"type":"string","enum":["pass","warn","fail","notEvaluated","excluded"]}}}} ``` ## The CheckTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"CheckTypeDTO":{"type":"string","enum":["missing","invalid","row_count","aggregate","schema","duplicate","freshness","failed_rows","metric","group_by","row_count_diff","aggregate_diff","freshness_diff","metric_diff","rows_diff","duplicate_diff","llm","hierarchy","fuzzy_duplicate","reference_diff"]}}}} ``` ## The ChecksContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ChecksContentDTO":{"type":"object","properties":{"agreements":{"type":"array","deprecated":true,"items":{"type":"object","$ref":"#/components/schemas/AgreementSlimDTO"}},"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringToString"},"checkType":{"type":"string"},"cloudUrl":{"type":"string"},"column":{"type":"string"},"createdAt":{"type":"string","format":"date-time"},"datasets":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetSlimDTO"}},"definition":{"type":"string"},"description":{"type":"string"},"evaluationStatus":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"group":{"type":"object","$ref":"#/components/schemas/ChecksContentDTO_Group"},"id":{"type":"string"},"incidents":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/IncidentSlimDTO"}},"lastCheckResultValue":{"type":"object","$ref":"#/components/schemas/CheckResultValueDTO"},"lastCheckRunTime":{"type":"string","format":"date-time","deprecated":true},"lastUpdated":{"type":"string","format":"date-time","deprecated":true},"metricType":{"type":"string"},"name":{"type":"string"},"owner":{"type":"object","$ref":"#/components/schemas/OwnerDTO","deprecated":true}},"required":["createdAt","datasets","evaluationStatus","id","incidents","name"]},"AgreementSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"name":{"type":"string"}}},"MapOfStringToString":{"type":"object","additionalProperties":true},"DatasetSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"qualifiedName":{"type":"string"}}},"EvaluationStatusDTO":{"type":"string","enum":["pass","warn","fail","notEvaluated","excluded"]},"ChecksContentDTO_Group":{"type":"object","properties":{"groupType":{"type":"string"},"identity":{"type":"string"},"name":{"type":"string"}}},"IncidentSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"number":{"type":"integer","format":"int32"},"status":{"type":"object","$ref":"#/components/schemas/IncidentStatusDTO"}},"required":["cloudUrl","id","name","number","status"]},"IncidentStatusDTO":{"type":"string","enum":["reported","investigating","fixing","resolved"]},"CheckResultValueDTO":{"type":"object","properties":{"anomalyDetectionDetails":{"type":"object","$ref":"#/components/schemas/CheckResultAnomalyDetectionDetailsDTO"},"dataTimestamp":{"type":"string","format":"date-time"},"diagnostics":{"type":"object","$ref":"#/components/schemas/CheckResultDiagnosticsV4"},"id":{"type":"string"},"measure":{"type":"object","$ref":"#/components/schemas/MeasureDTO"},"value":{"type":"number"},"valueLabel":{"type":"string"},"valueSeries":{"type":"object","$ref":"#/components/schemas/CheckResultValueSeriesDTO"}},"required":["id"]},"CheckResultAnomalyDetectionDetailsDTO":{"type":"object","properties":{"predictedValue":{"type":"number"}}},"CheckResultDiagnosticsV4":{"type":"object","properties":{"aggregate":{"type":"object","$ref":"#/components/schemas/AggregateDiagnosticsV4"},"aggregate_diff":{"type":"object","$ref":"#/components/schemas/AggregateDiffDiagnosticsV4"},"duplicate":{"type":"object","$ref":"#/components/schemas/DuplicateDiagnosticsV4"},"duplicate_diff":{"type":"object","$ref":"#/components/schemas/DuplicateDiffDiagnosticsV4"},"failed_rows":{"type":"object","$ref":"#/components/schemas/FailedRowsDiagnosticsV4"},"freshness":{"type":"object","$ref":"#/components/schemas/FreshnessDiagnosticsV4"},"freshness_diff":{"type":"object","$ref":"#/components/schemas/FreshnessDiffDiagnosticsV4"},"fuzzy_duplicate":{"type":"object","$ref":"#/components/schemas/FuzzyDuplicateDiagnosticsV4"},"group_by":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4"},"hierarchy":{"type":"object","$ref":"#/components/schemas/HierarchyDiagnosticsV4"},"invalid":{"type":"object","$ref":"#/components/schemas/InvalidDiagnosticsV4"},"llm":{"type":"object","$ref":"#/components/schemas/LlmDiagnosticsV4"},"metric":{"type":"object","$ref":"#/components/schemas/MetricDiagnosticsV4"},"metric_diff":{"type":"object","$ref":"#/components/schemas/MetricDiffDiagnosticsV4"},"missing":{"type":"object","$ref":"#/components/schemas/MissingDiagnosticsV4"},"reference_diff":{"type":"object","$ref":"#/components/schemas/ReferenceDiffDiagnosticsV4"},"row_count":{"type":"object","$ref":"#/components/schemas/RowCountDiagnosticsV4"},"row_count_diff":{"type":"object","$ref":"#/components/schemas/RowCountDiffDiagnosticsV4"},"rows_diff":{"type":"object","$ref":"#/components/schemas/RowsDiffDiagnosticsV4"},"schema":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4"}}},"AggregateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}},"AggregateDiffDiagnosticsV4":{"type":"object","properties":{"diff":{"type":"number"},"diffPercent":{"type":"number"},"function":{"type":"string"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceValue":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetValue":{"type":"number"}}},"DuplicateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}},"DuplicateDiffDiagnosticsV4":{"type":"object","properties":{"duplicateCountDiff":{"type":"integer","format":"int32"},"duplicateDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceDuplicateCount":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetDuplicateCount":{"type":"integer","format":"int32"}}},"FailedRowsDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}},"FreshnessDiagnosticsV4":{"type":"object","properties":{"actualTimestamp":{"type":"string","format":"date-time"},"actualTimestampUtc":{"type":"string","format":"date-time"},"datasetRowsTested":{"type":"integer","format":"int32"},"expectedTimestamp":{"type":"string","format":"date-time"},"expectedTimestampUtc":{"type":"string","format":"date-time"}}},"FreshnessDiffDiagnosticsV4":{"type":"object","properties":{"freshnessDiff":{"type":"integer","format":"int32"},"freshnessDiffUnit":{"type":"object","$ref":"#/components/schemas/FreshnessDiffDiagnosticsV4_UnitOfTimeDTO"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMaxTimestamp":{"type":"string","format":"date-time"},"sourceMaxTimestampUtc":{"type":"string","format":"date-time"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMaxTimestamp":{"type":"string","format":"date-time"},"targetMaxTimestampUtc":{"type":"string","format":"date-time"}}},"FreshnessDiffDiagnosticsV4_UnitOfTimeDTO":{"type":"string","enum":["minutes","hours","days"]},"FuzzyDuplicateDiagnosticsV4":{"type":"object","properties":{"candidateRowsScored":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"clusterCount":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"fuzzyDuplicateCount":{"type":"integer","format":"int32"},"fuzzyDuplicatePercent":{"type":"number"},"meanConfidence":{"type":"number"}}},"GroupByDiagnosticsV4":{"type":"object","properties":{"diagnosticsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"groups":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_Group"}}}},"GroupByDiagnosticsV4_Group":{"type":"object","properties":{"group":{"type":"string"},"rowsTested":{"type":"number"},"status":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_GroupByOutcomeDTO"},"value":{"type":"number"}}},"GroupByDiagnosticsV4_GroupByOutcomeDTO":{"type":"string","enum":["pass","warn","fail"]},"HierarchyDiagnosticsV4":{"type":"object","properties":{"brokenPathCount":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"invalidValueCount":{"type":"integer","format":"int32"},"missingValueCount":{"type":"integer","format":"int32"}}},"InvalidDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}},"LlmDiagnosticsV4":{"type":"object","properties":{"completionTokens":{"type":"number"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"model":{"type":"string"},"promptTokens":{"type":"number"},"rowsChecked":{"type":"integer","format":"int32"},"rowsFailed":{"type":"integer","format":"int32"},"rowsFailedPercent":{"type":"number"},"tokensSpent":{"type":"integer","format":"int32"}}},"MetricDiagnosticsV4":{"type":"object","properties":{"datasetRowsTested":{"type":"integer","format":"int32"}}},"MetricDiffDiagnosticsV4":{"type":"object","properties":{"metricDiff":{"type":"number"},"metricDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMetric":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMetric":{"type":"number"}}},"MissingDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}},"ReferenceDiffDiagnosticsV4":{"type":"object","properties":{"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"RowCountDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}},"RowCountDiffDiagnosticsV4":{"type":"object","properties":{"rowCountDiff":{"type":"integer","format":"int32"},"rowCountDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"RowsDiffDiagnosticsV4":{"type":"object","properties":{"changed":{"type":"integer","format":"int32"},"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"SchemaDiagnosticsV4":{"type":"object","properties":{"actual":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}},"expected":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}}}},"SchemaDiagnosticsV4_Column":{"type":"object","properties":{"name":{"type":"string"},"type":{"type":"string"}}},"MeasureDTO":{"type":"string","enum":["time"]},"CheckResultValueSeriesDTO":{"type":"object","properties":{"values":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/CheckResultValueSeriesDTO_ValueSeriesValue"}}}},"CheckResultValueSeriesDTO_ValueSeriesValue":{"type":"object","properties":{"label":{"type":"string"},"level":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"value":{"type":"number"}}},"OwnerDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"}}}}}} ``` ## The ChecksContentDTO\_Group object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ChecksContentDTO_Group":{"type":"object","properties":{"groupType":{"type":"string"},"identity":{"type":"string"},"name":{"type":"string"}}}}}} ``` ## The ColumnMetricContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ColumnMetricContentDTO":{"type":"object","properties":{"average":{"type":"number"},"averageLength":{"type":"number"},"distinctCount":{"type":"number"},"extremeValues":{"type":"object","$ref":"#/components/schemas/ExtremeValuesContentDTO"},"frequentValues":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ValueCountContentDTO"}},"histogram":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/HistogramBinContentDTO"}},"maximum":{"type":"object","$ref":"#/components/schemas/_object"},"maximumLength":{"type":"integer","format":"int32"},"maximumTimestamp":{"type":"string","format":"date-time"},"median":{"type":"number"},"minimum":{"type":"object","$ref":"#/components/schemas/_object"},"minimumLength":{"type":"integer","format":"int32"},"minimumTimestamp":{"type":"string","format":"date-time"},"missingCount":{"type":"number"},"standardDeviation":{"type":"number"},"sum":{"type":"number"},"variance":{"type":"number"}}},"ExtremeValuesContentDTO":{"type":"object","properties":{"maximum":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/_object"}},"minimum":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/_object"}}}},"_object":{"type":"object"},"ValueCountContentDTO":{"type":"object","properties":{"count":{"type":"number"},"value":{"type":"object","$ref":"#/components/schemas/_object"}}},"HistogramBinContentDTO":{"type":"object","properties":{"count":{"type":"number"},"lowerBound":{"type":"number"},"upperBound":{"type":"number"}},"required":["count","lowerBound","upperBound"]}}}} ``` ## The ColumnMetricMonitorConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ColumnMetricMonitorConfigurationDTO":{"type":"object","properties":{"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/ColumnMonitorMetricTypeDTO"}},"required":["configuration","metricType"]},"ColumnMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByColumnConfigurationDTO"}},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"ColumnMonitorMetricTypeDTO":{"type":"string","enum":["missingPercentage","duplicatePercentage","average","columnFreshness","sum","distinctValuesCount","count","minimumValue","maximumValue","duplicateValuesPercentage","minimumLength","maximumLength","averageLength","standardDeviation","variance","q1","median","q3"]}}}} ``` ## The ColumnMetricMonitorDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ColumnMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/ColumnMonitorMetricTypeDTO"}},"required":["checkId","columnName","configuration","metricType"]},"ColumnMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByColumnConfigurationDTO"}},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"ColumnMonitorMetricTypeDTO":{"type":"string","enum":["missingPercentage","duplicatePercentage","average","columnFreshness","sum","distinctValuesCount","count","minimumValue","maximumValue","duplicateValuesPercentage","minimumLength","maximumLength","averageLength","standardDeviation","variance","q1","median","q3"]}}}} ``` ## The ColumnMonitorConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ColumnMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByColumnConfigurationDTO"}},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}}}}} ``` ## The ColumnMonitorMetricTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ColumnMonitorMetricTypeDTO":{"type":"string","enum":["missingPercentage","duplicatePercentage","average","columnFreshness","sum","distinctValuesCount","count","minimumValue","maximumValue","duplicateValuesPercentage","minimumLength","maximumLength","averageLength","standardDeviation","variance","q1","median","q3"]}}}} ``` ## The ColumnTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]}}}} ``` ## The ContractContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ContractContentDTO":{"type":"object","properties":{"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"}}}}}} ``` ## The ContractVersionDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ContractVersionDTO":{"type":"object","properties":{"checksum":{"type":"string"},"contents":{"type":"string"},"contractId":{"type":"string"},"created":{"type":"string","format":"date-time"},"creatorId":{"type":"string"},"id":{"type":"string"}}}}}} ``` ## The CustomMetricMonitorDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"CustomMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/CustomSqlMonitorConfigurationDTO"},"monitorName":{"type":"string"}},"required":["checkId","configuration","monitorName"]},"CustomSqlMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}},"required":["resultMetric","sqlQuery"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}}}}} ``` ## The CustomSqlMonitorConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"CustomSqlMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}},"required":["resultMetric","sqlQuery"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}}}}} ``` ## The CustomSqlMonitorConfigurationUpdateDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"CustomSqlMonitorConfigurationUpdateDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}}}}} ``` ## The DataQualityStatusDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataQualityStatusDTO":{"type":"string","enum":["pass","warn","fail"]}}}} ``` ## The DataStandardActivityContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardActivityContentDTO":{"type":"object","properties":{"activeCount":{"type":"integer","format":"int32"},"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"datasetCount":{"type":"integer","format":"int32"},"totalCount":{"type":"integer","format":"int32"}},"required":["activeCount","checkResults","datasetCount","totalCount"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}}}}} ``` ## The DataStandardCheckContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardCheckContentDTO":{"type":"object","properties":{"checkType":{"type":"object","$ref":"#/components/schemas/CheckTypeDTO"},"compliance":{"type":"number"},"datasetCount":{"type":"integer","format":"int32"},"failingCount":{"type":"integer","format":"int32"},"lastScanTime":{"type":"string","format":"date-time"},"name":{"type":"string"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"},"worstLevel":{"type":"object","$ref":"#/components/schemas/CheckResultLevelDTO"}}},"CheckTypeDTO":{"type":"string","enum":["missing","invalid","row_count","aggregate","schema","duplicate","freshness","failed_rows","metric","group_by","row_count_diff","aggregate_diff","freshness_diff","metric_diff","rows_diff","duplicate_diff","llm","hierarchy","fuzzy_duplicate","reference_diff"]},"CheckResultLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated","excluded"]}}}} ``` ## The DataStandardCheckResultsDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}}}}} ``` ## The DataStandardContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"compliance":{"type":"number"},"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetCount":{"type":"integer","format":"int32"},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["id"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]}}}} ``` ## The DataStandardDatasetContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardDatasetContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanId":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"qualifiedName":{"type":"string"}},"required":["id","name"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]}}}} ``` ## The DataStandardOwnerRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardOwnerRequestDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The DataStandardScheduleDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}}}}} ``` ## The DataStandardScopeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}}}}} ``` ## The DataStandardScopeMatchingDatasetContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardScopeMatchingDatasetContentDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"qualifiedName":{"type":"string"}},"required":["datasourceId","id","name"]}}}} ``` ## The DataStandardScopeRuleDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}}}}} ``` ## The DataStandardSortFieldDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardSortFieldDTO":{"type":"string","enum":["name","label"]}}}} ``` ## The DataStandardStatusDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]}}}} ``` ## The DatasetColumnProfilingContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetColumnProfilingContentDTO":{"type":"object","properties":{"id":{"type":"string"},"metrics":{"type":"object","$ref":"#/components/schemas/ColumnMetricContentDTO"},"name":{"type":"string"},"type":{"type":"string"}},"required":["id","name","type"]},"ColumnMetricContentDTO":{"type":"object","properties":{"average":{"type":"number"},"averageLength":{"type":"number"},"distinctCount":{"type":"number"},"extremeValues":{"type":"object","$ref":"#/components/schemas/ExtremeValuesContentDTO"},"frequentValues":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ValueCountContentDTO"}},"histogram":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/HistogramBinContentDTO"}},"maximum":{"type":"object","$ref":"#/components/schemas/_object"},"maximumLength":{"type":"integer","format":"int32"},"maximumTimestamp":{"type":"string","format":"date-time"},"median":{"type":"number"},"minimum":{"type":"object","$ref":"#/components/schemas/_object"},"minimumLength":{"type":"integer","format":"int32"},"minimumTimestamp":{"type":"string","format":"date-time"},"missingCount":{"type":"number"},"standardDeviation":{"type":"number"},"sum":{"type":"number"},"variance":{"type":"number"}}},"ExtremeValuesContentDTO":{"type":"object","properties":{"maximum":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/_object"}},"minimum":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/_object"}}}},"_object":{"type":"object"},"ValueCountContentDTO":{"type":"object","properties":{"count":{"type":"number"},"value":{"type":"object","$ref":"#/components/schemas/_object"}}},"HistogramBinContentDTO":{"type":"object","properties":{"count":{"type":"number"},"lowerBound":{"type":"number"},"upperBound":{"type":"number"}},"required":["count","lowerBound","upperBound"]}}}} ``` ## The DatasetComputeWarehouseConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetComputeWarehouseConfigurationDTO":{"type":"object","properties":{"availableWarehouses":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetComputeWarehouseConfigurationDTO_WarehouseConfigurationDTO"}},"currentWarehouse":{"type":"object","$ref":"#/components/schemas/DatasetComputeWarehouseConfigurationDTO_WarehouseConfigurationDTO"},"datasetWarehouseOverride":{"type":"object","$ref":"#/components/schemas/DatasetComputeWarehouseConfigurationDTO_WarehouseConfigurationDTO"},"defaultWarehouse":{"type":"object","$ref":"#/components/schemas/DatasetComputeWarehouseConfigurationDTO_WarehouseConfigurationDTO"},"enabled":{"type":"boolean"}}},"DatasetComputeWarehouseConfigurationDTO_WarehouseConfigurationDTO":{"type":"object","properties":{"name":{"type":"string"}},"required":["name"]}}}} ``` ## The DatasetComputeWarehouseConfigurationDTO\_WarehouseConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetComputeWarehouseConfigurationDTO_WarehouseConfigurationDTO":{"type":"object","properties":{"name":{"type":"string"}},"required":["name"]}}}} ``` ## The DatasetDiagnosticsWarehouseFailedRowsConfigurationRequest object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetDiagnosticsWarehouseFailedRowsConfigurationRequest":{"type":"object","properties":{"enabled":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"maxRowCount":{"type":"integer","format":"int32"},"schema":{"type":"string","deprecated":true},"storageStrategy":{"type":"object","$ref":"#/components/schemas/DiagnosticsWarehouseFailedRowsStorageStrategyDTO"},"uniqueKeyColumnNames":{"type":"array","items":{"type":"string"}}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]},"DiagnosticsWarehouseFailedRowsStorageStrategyDTO":{"type":"string","enum":["fullFailedRows","uniqueKeys","scansAndResultsOnly"]}}}} ``` ## The DatasetDiagnosticsWarehouseScanAndResultsConfigurationRequest object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetDiagnosticsWarehouseScanAndResultsConfigurationRequest":{"type":"object","properties":{"enabled":{"type":"boolean"}}}}}} ``` ## The DatasetMetricMonitorConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetMetricMonitorConfigurationDTO":{"type":"object","properties":{"configuration":{"type":"object","$ref":"#/components/schemas/DatasetMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/DatasetMonitorMetricTypeDTO"}},"required":["configuration","metricType"]},"DatasetMonitorConfigurationDTO":{"type":"object","properties":{"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"DatasetMonitorMetricTypeDTO":{"type":"string","enum":["rowCount","freshness","schema","rowsInserted","totalRowCountChange","timeliness"]}}}} ``` ## The DatasetMonitorConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetMonitorConfigurationDTO":{"type":"object","properties":{"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}}}}} ``` ## The DatasetMonitorMetricTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetMonitorMetricTypeDTO":{"type":"string","enum":["rowCount","freshness","schema","rowsInserted","totalRowCountChange","timeliness"]}}}} ``` ## The DatasetOwnerDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]}}}} ``` ## The DatasetOwnerTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The DatasetPartitionColumnContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]}}}} ``` ## The DatasetResponsibilityContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetResponsibilityContentDTO":{"type":"object","properties":{"managed":{"type":"boolean"},"role":{"type":"object","$ref":"#/components/schemas/DatasetRoleContentDTO"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["managed","role","type"]},"DatasetRoleContentDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"executeContracts":{"type":"boolean","deprecated":true},"id":{"type":"string"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"publishContracts":{"type":"boolean","deprecated":true},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["id","name"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The DatasetResponsibilityContentRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetResponsibilityContentRequestDTO":{"type":"object","properties":{"roleId":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["roleId","type"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The DatasetRoleContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetRoleContentDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"executeContracts":{"type":"boolean","deprecated":true},"id":{"type":"string"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"publishContracts":{"type":"boolean","deprecated":true},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["id","name"]}}}} ``` ## The DatasetSlimDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"qualifiedName":{"type":"string"}}}}}} ``` ## The DatasetTimePartitionDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetTimePartitionDTO":{"type":"object","properties":{"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO"},"sqlExpression":{"type":"string"},"timePartitionType":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionTypeDTO"}}},"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]},"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]}}}} ``` ## The DatasetTimePartitionTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]}}}} ``` ## The DatasetsContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasetsContentDTO":{"type":"object","properties":{"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"},"checks":{"type":"number"},"cloudUrl":{"type":"string"},"dataQualityStatus":{"type":"object","$ref":"#/components/schemas/DataQualityStatusDTO"},"datasetQualifiedName":{"type":"string"},"datasource":{"type":"object","$ref":"#/components/schemas/DatasourcePropertiesDTO"},"healthStatus":{"type":"integer","format":"int32"},"id":{"type":"string"},"incidents":{"type":"number"},"label":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO","deprecated":true},"qualifiedName":{"type":"string"},"tags":{"type":"array","items":{"type":"string"}},"timePartition":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionDTO"}},"required":["checks","cloudUrl","healthStatus","id","incidents","name"]},"MapOfStringTo_object":{"type":"object","additionalProperties":true},"DataQualityStatusDTO":{"type":"string","enum":["pass","warn","fail"]},"DatasourcePropertiesDTO":{"type":"object","properties":{"label":{"type":"string"},"name":{"type":"string"},"prefix":{"type":"string"},"type":{"type":"string"},"version":{"type":"object","$ref":"#/components/schemas/ModelVersionDTO"}}},"ModelVersionDTO":{"type":"string","enum":["3","4"]},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]},"DatasetTimePartitionDTO":{"type":"object","properties":{"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO"},"sqlExpression":{"type":"string"},"timePartitionType":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionTypeDTO"}}},"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]}}}} ``` ## The DatasourcePropertiesDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasourcePropertiesDTO":{"type":"object","properties":{"label":{"type":"string"},"name":{"type":"string"},"prefix":{"type":"string"},"type":{"type":"string"},"version":{"type":"object","$ref":"#/components/schemas/ModelVersionDTO"}}},"ModelVersionDTO":{"type":"string","enum":["3","4"]}}}} ``` ## The DatasourceResponsibilityContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasourceResponsibilityContentDTO":{"type":"object","properties":{"managed":{"type":"boolean"},"role":{"type":"object","$ref":"#/components/schemas/DatasourceRoleContentDTO"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["managed","role","type"]},"DatasourceRoleContentDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"id":{"type":"string"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["id","name"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The DatasourceResponsibilityContentRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasourceResponsibilityContentRequestDTO":{"type":"object","properties":{"roleId":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["roleId","type"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The DatasourceRoleContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DatasourceRoleContentDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"id":{"type":"string"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["id","name"]}}}} ``` ## The DiagnosticsWarehouseFailedRowsStorageStrategyDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DiagnosticsWarehouseFailedRowsStorageStrategyDTO":{"type":"string","enum":["fullFailedRows","uniqueKeys","scansAndResultsOnly"]}}}} ``` ## The DiscoveredDatasetContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DiscoveredDatasetContentDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"datasourceId":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"onboarded":{"type":"boolean"},"qualifiedName":{"type":"string"}},"required":["datasourceId","id","name","onboarded","qualifiedName"]}}}} ``` ## The DuplicateDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DuplicateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}}}}} ``` ## The DuplicateDiffDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"DuplicateDiffDiagnosticsV4":{"type":"object","properties":{"duplicateCountDiff":{"type":"integer","format":"int32"},"duplicateDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceDuplicateCount":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetDuplicateCount":{"type":"integer","format":"int32"}}}}}} ``` ## The ErrorResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ErrorResponse":{"type":"object","properties":{"code":{"type":"string"},"message":{"type":"string"}}}}}} ``` ## The EvaluationStatusDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"EvaluationStatusDTO":{"type":"string","enum":["pass","warn","fail","notEvaluated","excluded"]}}}} ``` ## The ExclusionZoneDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]}}}} ``` ## The ExtremeValuesContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ExtremeValuesContentDTO":{"type":"object","properties":{"maximum":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/_object"}},"minimum":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/_object"}}}},"_object":{"type":"object"}}}} ``` ## The FailedInvitationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"FailedInvitationDTO":{"type":"object","properties":{"email":{"type":"string"},"error":{"type":"string"}}}}}} ``` ## The FailedRowSamplesDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}}}}} ``` ## The FailedRowsCollectionStrategyDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]}}}} ``` ## The FailedRowsCollectionStrategyTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]}}}} ``` ## The FailedRowsConfigurationStateDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"FailedRowsConfigurationStateDTO":{"type":"string","enum":["ready","doNotDetectAutomatically","awaitingPrimaryKeysInformationFromAgent","awaitingPrimaryKeysInformationFromRunner","errorMissingUniqueKeys"]}}}} ``` ## The FailedRowsDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"FailedRowsDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}}}}} ``` ## The FreshnessDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"FreshnessDiagnosticsV4":{"type":"object","properties":{"actualTimestamp":{"type":"string","format":"date-time"},"actualTimestampUtc":{"type":"string","format":"date-time"},"datasetRowsTested":{"type":"integer","format":"int32"},"expectedTimestamp":{"type":"string","format":"date-time"},"expectedTimestampUtc":{"type":"string","format":"date-time"}}}}}} ``` ## The FreshnessDiffDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"FreshnessDiffDiagnosticsV4":{"type":"object","properties":{"freshnessDiff":{"type":"integer","format":"int32"},"freshnessDiffUnit":{"type":"object","$ref":"#/components/schemas/FreshnessDiffDiagnosticsV4_UnitOfTimeDTO"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMaxTimestamp":{"type":"string","format":"date-time"},"sourceMaxTimestampUtc":{"type":"string","format":"date-time"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMaxTimestamp":{"type":"string","format":"date-time"},"targetMaxTimestampUtc":{"type":"string","format":"date-time"}}},"FreshnessDiffDiagnosticsV4_UnitOfTimeDTO":{"type":"string","enum":["minutes","hours","days"]}}}} ``` ## The FreshnessDiffDiagnosticsV4\_UnitOfTimeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"FreshnessDiffDiagnosticsV4_UnitOfTimeDTO":{"type":"string","enum":["minutes","hours","days"]}}}} ``` ## The FuzzyDuplicateDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"FuzzyDuplicateDiagnosticsV4":{"type":"object","properties":{"candidateRowsScored":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"clusterCount":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"fuzzyDuplicateCount":{"type":"integer","format":"int32"},"fuzzyDuplicatePercent":{"type":"number"},"meanConfidence":{"type":"number"}}}}}} ``` ## The GenerateContractOperationStateDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"GenerateContractOperationStateDTO":{"type":"string","enum":["ongoing","completed","failed","canceled"]}}}} ``` ## The GroupByColumnConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]}}}} ``` ## The GroupByDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"GroupByDiagnosticsV4":{"type":"object","properties":{"diagnosticsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"groups":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_Group"}}}},"GroupByDiagnosticsV4_Group":{"type":"object","properties":{"group":{"type":"string"},"rowsTested":{"type":"number"},"status":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_GroupByOutcomeDTO"},"value":{"type":"number"}}},"GroupByDiagnosticsV4_GroupByOutcomeDTO":{"type":"string","enum":["pass","warn","fail"]}}}} ``` ## The GroupByDiagnosticsV4\_Group object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"GroupByDiagnosticsV4_Group":{"type":"object","properties":{"group":{"type":"string"},"rowsTested":{"type":"number"},"status":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_GroupByOutcomeDTO"},"value":{"type":"number"}}},"GroupByDiagnosticsV4_GroupByOutcomeDTO":{"type":"string","enum":["pass","warn","fail"]}}}} ``` ## The GroupByDiagnosticsV4\_GroupByOutcomeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"GroupByDiagnosticsV4_GroupByOutcomeDTO":{"type":"string","enum":["pass","warn","fail"]}}}} ``` ## The HierarchyDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"HierarchyDiagnosticsV4":{"type":"object","properties":{"brokenPathCount":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"invalidValueCount":{"type":"integer","format":"int32"},"missingValueCount":{"type":"integer","format":"int32"}}}}}} ``` ## The HistogramBinContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"HistogramBinContentDTO":{"type":"object","properties":{"count":{"type":"number"},"lowerBound":{"type":"number"},"upperBound":{"type":"number"}},"required":["count","lowerBound","upperBound"]}}}} ``` ## The IncidentContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"IncidentContentDTO":{"type":"object","properties":{"description":{"type":"string"},"resolutionNotes":{"type":"string"},"severity":{"type":"object","$ref":"#/components/schemas/IncidentSeverityDTO"},"cloudUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"number":{"type":"integer","format":"int32"},"status":{"type":"object","$ref":"#/components/schemas/IncidentStatusDTO"}},"required":["description","severity"]},"IncidentSeverityDTO":{"type":"string","enum":["minor","major","critical"]},"IncidentStatusDTO":{"type":"string","enum":["reported","investigating","fixing","resolved"]}}}} ``` ## The IncidentSeverityDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"IncidentSeverityDTO":{"type":"string","enum":["minor","major","critical"]}}}} ``` ## The IncidentSlimDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"IncidentSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"number":{"type":"integer","format":"int32"},"status":{"type":"object","$ref":"#/components/schemas/IncidentStatusDTO"}},"required":["cloudUrl","id","name","number","status"]},"IncidentStatusDTO":{"type":"string","enum":["reported","investigating","fixing","resolved"]}}}} ``` ## The IncidentStatusDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"IncidentStatusDTO":{"type":"string","enum":["reported","investigating","fixing","resolved"]}}}} ``` ## The InvalidDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"InvalidDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}}}}} ``` ## The LlmDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"LlmDiagnosticsV4":{"type":"object","properties":{"completionTokens":{"type":"number"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"model":{"type":"string"},"promptTokens":{"type":"number"},"rowsChecked":{"type":"integer","format":"int32"},"rowsFailed":{"type":"integer","format":"int32"},"rowsFailedPercent":{"type":"number"},"tokensSpent":{"type":"integer","format":"int32"}}}}}} ``` ## The LogLevelDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"LogLevelDTO":{"type":"string","enum":["debug","info","warning","error"]}}}} ``` ## The LogsContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"LogsContentDTO":{"type":"object","properties":{"index":{"type":"integer","format":"int32"},"level":{"type":"object","$ref":"#/components/schemas/LogLevelDTO"},"message":{"type":"string"},"timestamp":{"type":"string","format":"date-time"}},"required":["index","level","message","timestamp"]},"LogLevelDTO":{"type":"string","enum":["debug","info","warning","error"]}}}} ``` ## The MapOfStringToMapOfStringTo\_object object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"MapOfStringToMapOfStringTo_object":{"type":"object","additionalProperties":true}}}} ``` ## The MapOfStringToString object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"MapOfStringToString":{"type":"object","additionalProperties":true}}}} ``` ## The MapOfStringTo\_object object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"MapOfStringTo_object":{"type":"object","additionalProperties":true}}}} ``` ## The MeasureDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"MeasureDTO":{"type":"string","enum":["time"]}}}} ``` ## The MetricDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"MetricDiagnosticsV4":{"type":"object","properties":{"datasetRowsTested":{"type":"integer","format":"int32"}}}}}} ``` ## The MetricDiffDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"MetricDiffDiagnosticsV4":{"type":"object","properties":{"metricDiff":{"type":"number"},"metricDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMetric":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMetric":{"type":"number"}}}}}} ``` ## The MissingDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"MissingDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}}}}} ``` ## The ModelVersionDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ModelVersionDTO":{"type":"string","enum":["3","4"]}}}} ``` ## The NotificationCustomizationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationCustomizationDTO":{"type":"object","properties":{"message":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationTypeDTO"}},"required":["type"]},"NotificationCustomizationTypeDTO":{"type":"string","enum":["rich","minimal"]}}}} ``` ## The NotificationCustomizationTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationCustomizationTypeDTO":{"type":"string","enum":["rich","minimal"]}}}} ``` ## The NotificationFilterDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationFilterDTO":{"type":"object","properties":{"expression":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"expressions":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"}},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationFilterTypeDTO"},"value":{"type":"string"},"values":{"type":"array","items":{"type":"string"}}},"required":["type"]},"NotificationFilterFieldDTO":{"type":"object","properties":{"attributeName":{"type":"string"},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldTypeDTO"}},"required":["field"]},"NotificationFilterFieldTypeDTO":{"type":"string","enum":["datasourceId","datasourceLabel","datasetId","datasetLabel","datasetOwnerId","datasetTags","checkId","checkName","checkOwnerId","columnId","metricType","datasetResourceAttribute","checkResourceAttribute"]},"NotificationFilterTypeDTO":{"type":"string","enum":["equals","contains","in","and","or","not"]}}}} ``` ## The NotificationFilterFieldDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationFilterFieldDTO":{"type":"object","properties":{"attributeName":{"type":"string"},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldTypeDTO"}},"required":["field"]},"NotificationFilterFieldTypeDTO":{"type":"string","enum":["datasourceId","datasourceLabel","datasetId","datasetLabel","datasetOwnerId","datasetTags","checkId","checkName","checkOwnerId","columnId","metricType","datasetResourceAttribute","checkResourceAttribute"]}}}} ``` ## The NotificationFilterFieldTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationFilterFieldTypeDTO":{"type":"string","enum":["datasourceId","datasourceLabel","datasetId","datasetLabel","datasetOwnerId","datasetTags","checkId","checkName","checkOwnerId","columnId","metricType","datasetResourceAttribute","checkResourceAttribute"]}}}} ``` ## The NotificationFilterTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationFilterTypeDTO":{"type":"string","enum":["equals","contains","in","and","or","not"]}}}} ``` ## The NotificationLevelDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated"]}}}} ``` ## The NotificationRecipientDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationRecipientDTO":{"type":"object","properties":{"customRoleId":{"type":"string"},"email":{"type":"string"},"integrationChannel":{"type":"string"},"integrationId":{"type":"string"},"role":{"type":"object","$ref":"#/components/schemas/NotificationRecipientRoleDTO"},"triggers":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTriggerDTO"}},"type":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["triggers","type"]},"NotificationRecipientRoleDTO":{"type":"string","enum":["datasetOwner","monitorOwner"]},"NotificationRecipientTriggerDTO":{"type":"object","properties":{"level":{"type":"object","$ref":"#/components/schemas/NotificationLevelDTO"},"scanState":{"type":"object","$ref":"#/components/schemas/NotificationScanStateDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationTriggerTypeDTO"}},"required":["type"]},"NotificationLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated"]},"NotificationScanStateDTO":{"type":"string","enum":["failed","timedOut","canceled","completedWithErrors"]},"NotificationTriggerTypeDTO":{"type":"string","enum":["testResultLevel","scanState"]},"NotificationRecipientTypeDTO":{"type":"string","enum":["email","user","userGroup","integration","resourceRole","customRole"]}}}} ``` ## The NotificationRecipientRoleDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationRecipientRoleDTO":{"type":"string","enum":["datasetOwner","monitorOwner"]}}}} ``` ## The NotificationRecipientTriggerDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationRecipientTriggerDTO":{"type":"object","properties":{"level":{"type":"object","$ref":"#/components/schemas/NotificationLevelDTO"},"scanState":{"type":"object","$ref":"#/components/schemas/NotificationScanStateDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationTriggerTypeDTO"}},"required":["type"]},"NotificationLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated"]},"NotificationScanStateDTO":{"type":"string","enum":["failed","timedOut","canceled","completedWithErrors"]},"NotificationTriggerTypeDTO":{"type":"string","enum":["testResultLevel","scanState"]}}}} ``` ## The NotificationRecipientTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationRecipientTypeDTO":{"type":"string","enum":["email","user","userGroup","integration","resourceRole","customRole"]}}}} ``` ## The NotificationRuleStatusDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationRuleStatusDTO":{"type":"string","enum":["active","paused"]}}}} ``` ## The NotificationScanStateDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationScanStateDTO":{"type":"string","enum":["failed","timedOut","canceled","completedWithErrors"]}}}} ``` ## The NotificationTriggerTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"NotificationTriggerTypeDTO":{"type":"string","enum":["testResultLevel","scanState"]}}}} ``` ## The OwnerDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"OwnerDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"}}}}}} ``` ## The ProfilingSamplingStrategyConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ProfilingSamplingStrategyConfigurationDTO":{"type":"object","properties":{"numberOfRows":{"type":"integer","format":"int32"},"numberOfUnits":{"type":"integer","format":"int32"},"unitOfTime":{"type":"object","$ref":"#/components/schemas/UnitOfTimeDTO"}}},"UnitOfTimeDTO":{"type":"string","enum":["hours","days","weeks"]}}}} ``` ## The PublicApiCheckSlimDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCheckSlimDTO":{"type":"object","properties":{"evaluationStatus":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"id":{"type":"string"}}},"EvaluationStatusDTO":{"type":"string","enum":["pass","warn","fail","notEvaluated","excluded"]}}}} ``` ## The PublicApiChecksResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiChecksResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ChecksContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"ChecksContentDTO":{"type":"object","properties":{"agreements":{"type":"array","deprecated":true,"items":{"type":"object","$ref":"#/components/schemas/AgreementSlimDTO"}},"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringToString"},"checkType":{"type":"string"},"cloudUrl":{"type":"string"},"column":{"type":"string"},"createdAt":{"type":"string","format":"date-time"},"datasets":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetSlimDTO"}},"definition":{"type":"string"},"description":{"type":"string"},"evaluationStatus":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"group":{"type":"object","$ref":"#/components/schemas/ChecksContentDTO_Group"},"id":{"type":"string"},"incidents":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/IncidentSlimDTO"}},"lastCheckResultValue":{"type":"object","$ref":"#/components/schemas/CheckResultValueDTO"},"lastCheckRunTime":{"type":"string","format":"date-time","deprecated":true},"lastUpdated":{"type":"string","format":"date-time","deprecated":true},"metricType":{"type":"string"},"name":{"type":"string"},"owner":{"type":"object","$ref":"#/components/schemas/OwnerDTO","deprecated":true}},"required":["createdAt","datasets","evaluationStatus","id","incidents","name"]},"AgreementSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"name":{"type":"string"}}},"MapOfStringToString":{"type":"object","additionalProperties":true},"DatasetSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"qualifiedName":{"type":"string"}}},"EvaluationStatusDTO":{"type":"string","enum":["pass","warn","fail","notEvaluated","excluded"]},"ChecksContentDTO_Group":{"type":"object","properties":{"groupType":{"type":"string"},"identity":{"type":"string"},"name":{"type":"string"}}},"IncidentSlimDTO":{"type":"object","properties":{"cloudUrl":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"number":{"type":"integer","format":"int32"},"status":{"type":"object","$ref":"#/components/schemas/IncidentStatusDTO"}},"required":["cloudUrl","id","name","number","status"]},"IncidentStatusDTO":{"type":"string","enum":["reported","investigating","fixing","resolved"]},"CheckResultValueDTO":{"type":"object","properties":{"anomalyDetectionDetails":{"type":"object","$ref":"#/components/schemas/CheckResultAnomalyDetectionDetailsDTO"},"dataTimestamp":{"type":"string","format":"date-time"},"diagnostics":{"type":"object","$ref":"#/components/schemas/CheckResultDiagnosticsV4"},"id":{"type":"string"},"measure":{"type":"object","$ref":"#/components/schemas/MeasureDTO"},"value":{"type":"number"},"valueLabel":{"type":"string"},"valueSeries":{"type":"object","$ref":"#/components/schemas/CheckResultValueSeriesDTO"}},"required":["id"]},"CheckResultAnomalyDetectionDetailsDTO":{"type":"object","properties":{"predictedValue":{"type":"number"}}},"CheckResultDiagnosticsV4":{"type":"object","properties":{"aggregate":{"type":"object","$ref":"#/components/schemas/AggregateDiagnosticsV4"},"aggregate_diff":{"type":"object","$ref":"#/components/schemas/AggregateDiffDiagnosticsV4"},"duplicate":{"type":"object","$ref":"#/components/schemas/DuplicateDiagnosticsV4"},"duplicate_diff":{"type":"object","$ref":"#/components/schemas/DuplicateDiffDiagnosticsV4"},"failed_rows":{"type":"object","$ref":"#/components/schemas/FailedRowsDiagnosticsV4"},"freshness":{"type":"object","$ref":"#/components/schemas/FreshnessDiagnosticsV4"},"freshness_diff":{"type":"object","$ref":"#/components/schemas/FreshnessDiffDiagnosticsV4"},"fuzzy_duplicate":{"type":"object","$ref":"#/components/schemas/FuzzyDuplicateDiagnosticsV4"},"group_by":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4"},"hierarchy":{"type":"object","$ref":"#/components/schemas/HierarchyDiagnosticsV4"},"invalid":{"type":"object","$ref":"#/components/schemas/InvalidDiagnosticsV4"},"llm":{"type":"object","$ref":"#/components/schemas/LlmDiagnosticsV4"},"metric":{"type":"object","$ref":"#/components/schemas/MetricDiagnosticsV4"},"metric_diff":{"type":"object","$ref":"#/components/schemas/MetricDiffDiagnosticsV4"},"missing":{"type":"object","$ref":"#/components/schemas/MissingDiagnosticsV4"},"reference_diff":{"type":"object","$ref":"#/components/schemas/ReferenceDiffDiagnosticsV4"},"row_count":{"type":"object","$ref":"#/components/schemas/RowCountDiagnosticsV4"},"row_count_diff":{"type":"object","$ref":"#/components/schemas/RowCountDiffDiagnosticsV4"},"rows_diff":{"type":"object","$ref":"#/components/schemas/RowsDiffDiagnosticsV4"},"schema":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4"}}},"AggregateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}},"AggregateDiffDiagnosticsV4":{"type":"object","properties":{"diff":{"type":"number"},"diffPercent":{"type":"number"},"function":{"type":"string"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceValue":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetValue":{"type":"number"}}},"DuplicateDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}},"DuplicateDiffDiagnosticsV4":{"type":"object","properties":{"duplicateCountDiff":{"type":"integer","format":"int32"},"duplicateDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceDuplicateCount":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetDuplicateCount":{"type":"integer","format":"int32"}}},"FailedRowsDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}},"FreshnessDiagnosticsV4":{"type":"object","properties":{"actualTimestamp":{"type":"string","format":"date-time"},"actualTimestampUtc":{"type":"string","format":"date-time"},"datasetRowsTested":{"type":"integer","format":"int32"},"expectedTimestamp":{"type":"string","format":"date-time"},"expectedTimestampUtc":{"type":"string","format":"date-time"}}},"FreshnessDiffDiagnosticsV4":{"type":"object","properties":{"freshnessDiff":{"type":"integer","format":"int32"},"freshnessDiffUnit":{"type":"object","$ref":"#/components/schemas/FreshnessDiffDiagnosticsV4_UnitOfTimeDTO"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMaxTimestamp":{"type":"string","format":"date-time"},"sourceMaxTimestampUtc":{"type":"string","format":"date-time"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMaxTimestamp":{"type":"string","format":"date-time"},"targetMaxTimestampUtc":{"type":"string","format":"date-time"}}},"FreshnessDiffDiagnosticsV4_UnitOfTimeDTO":{"type":"string","enum":["minutes","hours","days"]},"FuzzyDuplicateDiagnosticsV4":{"type":"object","properties":{"candidateRowsScored":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"clusterCount":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"fuzzyDuplicateCount":{"type":"integer","format":"int32"},"fuzzyDuplicatePercent":{"type":"number"},"meanConfidence":{"type":"number"}}},"GroupByDiagnosticsV4":{"type":"object","properties":{"diagnosticsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"groups":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_Group"}}}},"GroupByDiagnosticsV4_Group":{"type":"object","properties":{"group":{"type":"string"},"rowsTested":{"type":"number"},"status":{"type":"object","$ref":"#/components/schemas/GroupByDiagnosticsV4_GroupByOutcomeDTO"},"value":{"type":"number"}}},"GroupByDiagnosticsV4_GroupByOutcomeDTO":{"type":"string","enum":["pass","warn","fail"]},"HierarchyDiagnosticsV4":{"type":"object","properties":{"brokenPathCount":{"type":"integer","format":"int32"},"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"invalidValueCount":{"type":"integer","format":"int32"},"missingValueCount":{"type":"integer","format":"int32"}}},"InvalidDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"},"missingCount":{"type":"integer","format":"int32"}}},"LlmDiagnosticsV4":{"type":"object","properties":{"completionTokens":{"type":"number"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsSourceQuery":{"type":"string"},"model":{"type":"string"},"promptTokens":{"type":"number"},"rowsChecked":{"type":"integer","format":"int32"},"rowsFailed":{"type":"integer","format":"int32"},"rowsFailedPercent":{"type":"number"},"tokensSpent":{"type":"integer","format":"int32"}}},"MetricDiagnosticsV4":{"type":"object","properties":{"datasetRowsTested":{"type":"integer","format":"int32"}}},"MetricDiffDiagnosticsV4":{"type":"object","properties":{"metricDiff":{"type":"number"},"metricDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"sourceMetric":{"type":"number"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"},"targetMetric":{"type":"number"}}},"MissingDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"},"failedRowsCount":{"type":"integer","format":"int32"},"failedRowsDataQuery":{"type":"string"},"failedRowsKeysQuery":{"type":"string"},"failedRowsPercent":{"type":"number"},"failedRowsSourceQuery":{"type":"string"}}},"ReferenceDiffDiagnosticsV4":{"type":"object","properties":{"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"RowCountDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}},"RowCountDiffDiagnosticsV4":{"type":"object","properties":{"rowCountDiff":{"type":"integer","format":"int32"},"rowCountDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"RowsDiffDiagnosticsV4":{"type":"object","properties":{"changed":{"type":"integer","format":"int32"},"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}},"SchemaDiagnosticsV4":{"type":"object","properties":{"actual":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}},"expected":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}}}},"SchemaDiagnosticsV4_Column":{"type":"object","properties":{"name":{"type":"string"},"type":{"type":"string"}}},"MeasureDTO":{"type":"string","enum":["time"]},"CheckResultValueSeriesDTO":{"type":"object","properties":{"values":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/CheckResultValueSeriesDTO_ValueSeriesValue"}}}},"CheckResultValueSeriesDTO_ValueSeriesValue":{"type":"object","properties":{"label":{"type":"string"},"level":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"value":{"type":"number"}}},"OwnerDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"}}}}}} ``` ## The PublicApiCreateAttributeRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateAttributeRequestDTO":{"type":"object","properties":{"allowedValues":{"type":"array","items":{"type":"string"}},"description":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"resourceType":{"type":"object","$ref":"#/components/schemas/AttributeResourceTypeDTO"},"type":{"type":"object","$ref":"#/components/schemas/AttributeTypeDTO"}},"required":["label","name","resourceType","type"]},"AttributeResourceTypeDTO":{"type":"string","enum":["dataset","check","column"]},"AttributeTypeDTO":{"type":"string","enum":["text","number","checkbox","datetime","singleSelect","multiSelect"]}}}} ``` ## The PublicApiCreateAttributeResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateAttributeResponse":{"type":"object","properties":{"attribute":{"type":"object","$ref":"#/components/schemas/AttributeContentDTO"}},"required":["attribute"]},"AttributeContentDTO":{"type":"object","properties":{"allowedValues":{"type":"array","items":{"type":"string"}},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"resourceType":{"type":"object","$ref":"#/components/schemas/AttributeResourceTypeDTO"},"type":{"type":"object","$ref":"#/components/schemas/AttributeTypeDTO"}},"required":["id","label","name","resourceType","type"]},"AttributeResourceTypeDTO":{"type":"string","enum":["dataset","check","column"]},"AttributeTypeDTO":{"type":"string","enum":["text","number","checkbox","datetime","singleSelect","multiSelect"]}}}} ``` ## The PublicApiCreateColumnMetricMonitorRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateColumnMetricMonitorRequestDTO":{"type":"object","properties":{"columnMetricMonitorConfiguration":{"type":"object","$ref":"#/components/schemas/ColumnMetricMonitorConfigurationDTO"},"columnName":{"type":"string"}},"required":["columnMetricMonitorConfiguration","columnName"]},"ColumnMetricMonitorConfigurationDTO":{"type":"object","properties":{"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/ColumnMonitorMetricTypeDTO"}},"required":["configuration","metricType"]},"ColumnMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByColumnConfigurationDTO"}},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"ColumnMonitorMetricTypeDTO":{"type":"string","enum":["missingPercentage","duplicatePercentage","average","columnFreshness","sum","distinctValuesCount","count","minimumValue","maximumValue","duplicateValuesPercentage","minimumLength","maximumLength","averageLength","standardDeviation","variance","q1","median","q3"]}}}} ``` ## The PublicApiCreateColumnMetricMonitorResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateColumnMetricMonitorResponse":{"type":"object","properties":{"columnMetricMonitor":{"type":"object","$ref":"#/components/schemas/ColumnMetricMonitorDTO"}},"required":["columnMetricMonitor"]},"ColumnMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/ColumnMonitorMetricTypeDTO"}},"required":["checkId","columnName","configuration","metricType"]},"ColumnMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByColumnConfigurationDTO"}},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"ColumnMonitorMetricTypeDTO":{"type":"string","enum":["missingPercentage","duplicatePercentage","average","columnFreshness","sum","distinctValuesCount","count","minimumValue","maximumValue","duplicateValuesPercentage","minimumLength","maximumLength","averageLength","standardDeviation","variance","q1","median","q3"]}}}} ``` ## The PublicApiCreateContractRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateContractRequestDTO":{"type":"object","properties":{"contents":{"type":"string"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"}},"required":["contents"]}}}} ``` ## The PublicApiCreateContractResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateContractResponse":{"type":"object","properties":{"contract":{"type":"object","$ref":"#/components/schemas/ContractContentDTO"}}},"ContractContentDTO":{"type":"object","properties":{"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"}}}}}} ``` ## The PublicApiCreateCustomSqlMonitorRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateCustomSqlMonitorRequestDTO":{"type":"object","properties":{"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/CustomSqlMonitorConfigurationDTO"},"monitorName":{"type":"string"}},"required":["configuration","monitorName"]},"CustomSqlMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}},"required":["resultMetric","sqlQuery"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}}}}} ``` ## The PublicApiCreateCustomSqlMonitorResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateCustomSqlMonitorResponse":{"type":"object","properties":{"customSqlMonitor":{"type":"object","$ref":"#/components/schemas/CustomMetricMonitorDTO"}},"required":["customSqlMonitor"]},"CustomMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/CustomSqlMonitorConfigurationDTO"},"monitorName":{"type":"string"}},"required":["checkId","configuration","monitorName"]},"CustomSqlMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}},"required":["resultMetric","sqlQuery"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}}}}} ``` ## The PublicApiCreateDataStandardResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateDataStandardResponse":{"type":"object","properties":{"dataStandard":{"type":"object","$ref":"#/components/schemas/DataStandardContentDTO"}}},"DataStandardContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"compliance":{"type":"number"},"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetCount":{"type":"integer","format":"int32"},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["id"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]}}}} ``` ## The PublicApiCreateDatasetRoleResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateDatasetRoleResponse":{"type":"object","properties":{"datasetRole":{"type":"object","$ref":"#/components/schemas/DatasetRoleContentDTO"}},"required":["datasetRole"]},"DatasetRoleContentDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"executeContracts":{"type":"boolean","deprecated":true},"id":{"type":"string"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"publishContracts":{"type":"boolean","deprecated":true},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["id","name"]}}}} ``` ## The PublicApiCreateDatasourceRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateDatasourceRequestDTO":{"type":"object","properties":{"configurationFileContents":{"type":"string"},"label":{"type":"string"},"runnerId":{"type":"string"}},"required":["configurationFileContents"]}}}} ``` ## The PublicApiCreateDatasourceResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateDatasourceResponse":{"type":"object","properties":{"datasource":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDTO"},"discoveryScanId":{"type":"string"}}},"PublicApiDatasourceDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceTypeDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"PublicApiDatasourceTypeDTO":{"type":"string","enum":["bigquery","mysql","postgres","snowflake","dask","spark","sparkdf","redshift","duckdb","presto","oracle","athena","trino","dremio","vertica","denodo","db2","db2z","sqlserver","databricks","synapse","fabric"]}}}} ``` ## The PublicApiCreateDatasourceRoleResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateDatasourceRoleResponse":{"type":"object","properties":{"datasourceRole":{"type":"object","$ref":"#/components/schemas/DatasourceRoleContentDTO"}},"required":["datasourceRole"]},"DatasourceRoleContentDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"id":{"type":"string"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["id","name"]}}}} ``` ## The PublicApiCreateRunnerResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateRunnerResponse":{"type":"object","properties":{"apiKeyId":{"type":"string"},"apiKeySecret":{"type":"string"}}}}}} ``` ## The PublicApiCreateSecretRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateSecretRequestDTO":{"type":"object","properties":{"encryptedValue":{"type":"string"},"encryptionKey":{"type":"string"},"name":{"type":"string"}},"required":["encryptedValue","encryptionKey","name"]}}}} ``` ## The PublicApiCreateSecretResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateSecretResponse":{"type":"object","properties":{"secret":{"type":"object","$ref":"#/components/schemas/PublicApiSecretDTO"}}},"PublicApiSecretDTO":{"type":"object","properties":{"created":{"type":"string","format":"date-time"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"}}}}}} ``` ## The PublicApiCreateSkeletonContractRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateSkeletonContractRequestDTO":{"type":"object","properties":{"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"}}}}}} ``` ## The PublicApiCreateSkeletonContractStatusResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateSkeletonContractStatusResponse":{"type":"object","properties":{"created":{"type":"string","format":"date-time"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"operationId":{"type":"string"},"state":{"type":"object","$ref":"#/components/schemas/SkeletonContractOperationStateDTO"}}},"SkeletonContractOperationStateDTO":{"type":"string","enum":["ongoing","completed","failed","canceled"]}}}} ``` ## The PublicApiCreateUserGroupRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiCreateUserGroupRequestDTO":{"type":"object","properties":{"name":{"type":"string"},"userIds":{"type":"array","items":{"type":"string"}}}}}}} ``` ## The PublicApiDataStandardRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDataStandardRequestDTO":{"type":"object","properties":{"contents":{"type":"string"},"description":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardOwnerRequestDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["contents","name","owners","status"]},"DataStandardOwnerRequestDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]}}}} ``` ## The PublicApiDataStandardsActivityResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDataStandardsActivityResponse":{"type":"object","properties":{"activity":{"type":"object","$ref":"#/components/schemas/DataStandardActivityContentDTO"}}},"DataStandardActivityContentDTO":{"type":"object","properties":{"activeCount":{"type":"integer","format":"int32"},"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"datasetCount":{"type":"integer","format":"int32"},"totalCount":{"type":"integer","format":"int32"}},"required":["activeCount","checkResults","datasetCount","totalCount"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}}}}} ``` ## The PublicApiDatasetColumnContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasetColumnContentDTO":{"type":"object","properties":{"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"},"columnId":{"type":"string"},"columnName":{"type":"string"}},"required":["attributes","columnId","columnName"]},"MapOfStringTo_object":{"type":"object","additionalProperties":true}}}} ``` ## The PublicApiDatasetDiagnosticsWarehouseResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasetDiagnosticsWarehouseResponse":{"type":"object","properties":{"failedRowsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetDiagnosticsWarehouseResponse_DiagnosticsWarehouseFailedRowsConfigurationDTO"},"scanAndResultsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetDiagnosticsWarehouseResponse_DiagnosticsWarehouseScanAndResultsConfigurationDTO"}}},"PublicApiDatasetDiagnosticsWarehouseResponse_DiagnosticsWarehouseFailedRowsConfigurationDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"maxRowCount":{"type":"integer","format":"int32"},"state":{"type":"object","$ref":"#/components/schemas/FailedRowsConfigurationStateDTO"},"storageStrategy":{"type":"object","$ref":"#/components/schemas/DiagnosticsWarehouseFailedRowsStorageStrategyDTO"},"uniqueKeyColumnNames":{"type":"array","items":{"type":"string"}}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]},"FailedRowsConfigurationStateDTO":{"type":"string","enum":["ready","doNotDetectAutomatically","awaitingPrimaryKeysInformationFromAgent","awaitingPrimaryKeysInformationFromRunner","errorMissingUniqueKeys"]},"DiagnosticsWarehouseFailedRowsStorageStrategyDTO":{"type":"string","enum":["fullFailedRows","uniqueKeys","scansAndResultsOnly"]},"PublicApiDatasetDiagnosticsWarehouseResponse_DiagnosticsWarehouseScanAndResultsConfigurationDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}}}}} ``` ## The PublicApiDatasetDiagnosticsWarehouseResponse\_DiagnosticsWarehouseFailedRowsConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasetDiagnosticsWarehouseResponse_DiagnosticsWarehouseFailedRowsConfigurationDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"maxRowCount":{"type":"integer","format":"int32"},"state":{"type":"object","$ref":"#/components/schemas/FailedRowsConfigurationStateDTO"},"storageStrategy":{"type":"object","$ref":"#/components/schemas/DiagnosticsWarehouseFailedRowsStorageStrategyDTO"},"uniqueKeyColumnNames":{"type":"array","items":{"type":"string"}}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]},"FailedRowsConfigurationStateDTO":{"type":"string","enum":["ready","doNotDetectAutomatically","awaitingPrimaryKeysInformationFromAgent","awaitingPrimaryKeysInformationFromRunner","errorMissingUniqueKeys"]},"DiagnosticsWarehouseFailedRowsStorageStrategyDTO":{"type":"string","enum":["fullFailedRows","uniqueKeys","scansAndResultsOnly"]}}}} ``` ## The PublicApiDatasetDiagnosticsWarehouseResponse\_DiagnosticsWarehouseScanAndResultsConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasetDiagnosticsWarehouseResponse_DiagnosticsWarehouseScanAndResultsConfigurationDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}}}}} ``` ## The PublicApiDatasetMetricMonitoringResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasetMetricMonitoringResponse":{"type":"object","properties":{"columnMetricMonitors":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ColumnMetricMonitorDTO"}},"customSqlMetricMonitors":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/CustomMetricMonitorDTO"}},"datasetId":{"type":"string"},"datasetMetricMonitorsConfiguration":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetMetricMonitorConfigurationDTO"}},"enabled":{"type":"boolean"},"historicalMetricCollectionScanStartDate":{"type":"string","format":"date-time"},"partitionColumn":{"type":"string","deprecated":true},"scanSchedule":{"type":"object","$ref":"#/components/schemas/ZonedCronDTO"},"timePartition":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionDTO"}}},"ColumnMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/ColumnMonitorMetricTypeDTO"}},"required":["checkId","columnName","configuration","metricType"]},"ColumnMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByColumnConfigurationDTO"}},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"ColumnMonitorMetricTypeDTO":{"type":"string","enum":["missingPercentage","duplicatePercentage","average","columnFreshness","sum","distinctValuesCount","count","minimumValue","maximumValue","duplicateValuesPercentage","minimumLength","maximumLength","averageLength","standardDeviation","variance","q1","median","q3"]},"CustomMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/CustomSqlMonitorConfigurationDTO"},"monitorName":{"type":"string"}},"required":["checkId","configuration","monitorName"]},"CustomSqlMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}},"required":["resultMetric","sqlQuery"]},"DatasetMetricMonitorConfigurationDTO":{"type":"object","properties":{"configuration":{"type":"object","$ref":"#/components/schemas/DatasetMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/DatasetMonitorMetricTypeDTO"}},"required":["configuration","metricType"]},"DatasetMonitorConfigurationDTO":{"type":"object","properties":{"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"DatasetMonitorMetricTypeDTO":{"type":"string","enum":["rowCount","freshness","schema","rowsInserted","totalRowCountChange","timeliness"]},"ZonedCronDTO":{"type":"object","properties":{"cronExpression":{"type":"string"},"timezone":{"type":"string"}},"required":["cronExpression","timezone"]},"DatasetTimePartitionDTO":{"type":"object","properties":{"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO"},"sqlExpression":{"type":"string"},"timePartitionType":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionTypeDTO"}}},"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]},"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]}}}} ``` ## The PublicApiDatasetProfilingResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasetProfilingResponse":{"type":"object","properties":{"columnCount":{"type":"number"},"columns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetColumnProfilingContentDTO"}},"datasetId":{"type":"string"},"enabled":{"type":"boolean"},"partitionColumn":{"type":"string","deprecated":true},"profilingTime":{"type":"string","format":"date-time"},"rowCount":{"type":"number"},"samplingStrategyConfiguration":{"type":"object","$ref":"#/components/schemas/ProfilingSamplingStrategyConfigurationDTO"},"scanSchedule":{"type":"object","$ref":"#/components/schemas/ZonedCronDTO"},"timePartition":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionDTO"}},"required":["columns","datasetId"]},"DatasetColumnProfilingContentDTO":{"type":"object","properties":{"id":{"type":"string"},"metrics":{"type":"object","$ref":"#/components/schemas/ColumnMetricContentDTO"},"name":{"type":"string"},"type":{"type":"string"}},"required":["id","name","type"]},"ColumnMetricContentDTO":{"type":"object","properties":{"average":{"type":"number"},"averageLength":{"type":"number"},"distinctCount":{"type":"number"},"extremeValues":{"type":"object","$ref":"#/components/schemas/ExtremeValuesContentDTO"},"frequentValues":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ValueCountContentDTO"}},"histogram":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/HistogramBinContentDTO"}},"maximum":{"type":"object","$ref":"#/components/schemas/_object"},"maximumLength":{"type":"integer","format":"int32"},"maximumTimestamp":{"type":"string","format":"date-time"},"median":{"type":"number"},"minimum":{"type":"object","$ref":"#/components/schemas/_object"},"minimumLength":{"type":"integer","format":"int32"},"minimumTimestamp":{"type":"string","format":"date-time"},"missingCount":{"type":"number"},"standardDeviation":{"type":"number"},"sum":{"type":"number"},"variance":{"type":"number"}}},"ExtremeValuesContentDTO":{"type":"object","properties":{"maximum":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/_object"}},"minimum":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/_object"}}}},"_object":{"type":"object"},"ValueCountContentDTO":{"type":"object","properties":{"count":{"type":"number"},"value":{"type":"object","$ref":"#/components/schemas/_object"}}},"HistogramBinContentDTO":{"type":"object","properties":{"count":{"type":"number"},"lowerBound":{"type":"number"},"upperBound":{"type":"number"}},"required":["count","lowerBound","upperBound"]},"ProfilingSamplingStrategyConfigurationDTO":{"type":"object","properties":{"numberOfRows":{"type":"integer","format":"int32"},"numberOfUnits":{"type":"integer","format":"int32"},"unitOfTime":{"type":"object","$ref":"#/components/schemas/UnitOfTimeDTO"}}},"UnitOfTimeDTO":{"type":"string","enum":["hours","days","weeks"]},"ZonedCronDTO":{"type":"object","properties":{"cronExpression":{"type":"string"},"timezone":{"type":"string"}},"required":["cronExpression","timezone"]},"DatasetTimePartitionDTO":{"type":"object","properties":{"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO"},"sqlExpression":{"type":"string"},"timePartitionType":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionTypeDTO"}}},"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]},"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]}}}} ``` ## The PublicApiDatasetResponsibilitiesResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasetResponsibilitiesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetResponsibilityContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DatasetResponsibilityContentDTO":{"type":"object","properties":{"managed":{"type":"boolean"},"role":{"type":"object","$ref":"#/components/schemas/DatasetRoleContentDTO"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["managed","role","type"]},"DatasetRoleContentDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"executeContracts":{"type":"boolean","deprecated":true},"id":{"type":"string"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"publishContracts":{"type":"boolean","deprecated":true},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["id","name"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The PublicApiDatasetRoleRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasetRoleRequestDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["name"]}}}} ``` ## The PublicApiDatasetRolesResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasetRolesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetRoleContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DatasetRoleContentDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"executeContracts":{"type":"boolean","deprecated":true},"id":{"type":"string"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"publishContracts":{"type":"boolean","deprecated":true},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["id","name"]}}}} ``` ## The PublicApiDatasetsResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasetsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetsContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DatasetsContentDTO":{"type":"object","properties":{"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"},"checks":{"type":"number"},"cloudUrl":{"type":"string"},"dataQualityStatus":{"type":"object","$ref":"#/components/schemas/DataQualityStatusDTO"},"datasetQualifiedName":{"type":"string"},"datasource":{"type":"object","$ref":"#/components/schemas/DatasourcePropertiesDTO"},"healthStatus":{"type":"integer","format":"int32"},"id":{"type":"string"},"incidents":{"type":"number"},"label":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO","deprecated":true},"qualifiedName":{"type":"string"},"tags":{"type":"array","items":{"type":"string"}},"timePartition":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionDTO"}},"required":["checks","cloudUrl","healthStatus","id","incidents","name"]},"MapOfStringTo_object":{"type":"object","additionalProperties":true},"DataQualityStatusDTO":{"type":"string","enum":["pass","warn","fail"]},"DatasourcePropertiesDTO":{"type":"object","properties":{"label":{"type":"string"},"name":{"type":"string"},"prefix":{"type":"string"},"type":{"type":"string"},"version":{"type":"object","$ref":"#/components/schemas/ModelVersionDTO"}}},"ModelVersionDTO":{"type":"string","enum":["3","4"]},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DatasetPartitionColumnContentDTO":{"type":"object","properties":{"columnType":{"type":"object","$ref":"#/components/schemas/ColumnTypeDTO"},"name":{"type":"string"}},"required":["columnType","name"]},"ColumnTypeDTO":{"type":"string","enum":["unknown","string","integer","float","time","boolean","date","dateTime"]},"DatasetTimePartitionDTO":{"type":"object","properties":{"partitionColumn":{"type":"object","$ref":"#/components/schemas/DatasetPartitionColumnContentDTO"},"sqlExpression":{"type":"string"},"timePartitionType":{"type":"object","$ref":"#/components/schemas/DatasetTimePartitionTypeDTO"}}},"DatasetTimePartitionTypeDTO":{"type":"string","enum":["disabled","partitionColumn","sqlExpression"]}}}} ``` ## The PublicApiDatasourceDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasourceDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceTypeDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"PublicApiDatasourceTypeDTO":{"type":"string","enum":["bigquery","mysql","postgres","snowflake","dask","spark","sparkdf","redshift","duckdb","presto","oracle","athena","trino","dremio","vertica","denodo","db2","db2z","sqlserver","databricks","synapse","fabric"]}}}} ``` ## The PublicApiDatasourceDiagnosticsWarehouseResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasourceDiagnosticsWarehouseResponse":{"type":"object","properties":{"enabled":{"type":"boolean"},"failedRowsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse_FailedRowsConfigurationDTO"},"reuseDatasource":{"type":"boolean"},"scanAndResultsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse_ScanAndResultsConfigurationDTO"},"tableNameTemplate":{"type":"string"}}},"PublicApiDatasourceDiagnosticsWarehouseResponse_FailedRowsConfigurationDTO":{"type":"object","properties":{"actionButton":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse_ActionButtonDTO"},"enabled":{"type":"boolean"},"exposeQueries":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"locationMessage":{"type":"string"},"maxRowCount":{"type":"integer","format":"int32"}}},"PublicApiDatasourceDiagnosticsWarehouseResponse_ActionButtonDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"title":{"type":"string"},"url":{"type":"string"}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]},"PublicApiDatasourceDiagnosticsWarehouseResponse_ScanAndResultsConfigurationDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}}}}} ``` ## The PublicApiDatasourceDiagnosticsWarehouseResponse\_ActionButtonDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasourceDiagnosticsWarehouseResponse_ActionButtonDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"title":{"type":"string"},"url":{"type":"string"}}}}}} ``` ## The PublicApiDatasourceDiagnosticsWarehouseResponse\_FailedRowsConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasourceDiagnosticsWarehouseResponse_FailedRowsConfigurationDTO":{"type":"object","properties":{"actionButton":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDiagnosticsWarehouseResponse_ActionButtonDTO"},"enabled":{"type":"boolean"},"exposeQueries":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"locationMessage":{"type":"string"},"maxRowCount":{"type":"integer","format":"int32"}}},"PublicApiDatasourceDiagnosticsWarehouseResponse_ActionButtonDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"title":{"type":"string"},"url":{"type":"string"}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]}}}} ``` ## The PublicApiDatasourceDiagnosticsWarehouseResponse\_ScanAndResultsConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasourceDiagnosticsWarehouseResponse_ScanAndResultsConfigurationDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}}}}} ``` ## The PublicApiDatasourceResponsibilitiesResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasourceResponsibilitiesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasourceResponsibilityContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DatasourceResponsibilityContentDTO":{"type":"object","properties":{"managed":{"type":"boolean"},"role":{"type":"object","$ref":"#/components/schemas/DatasourceRoleContentDTO"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["managed","role","type"]},"DatasourceRoleContentDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"id":{"type":"string"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["id","name"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The PublicApiDatasourceRoleRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasourceRoleRequestDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["name"]}}}} ``` ## The PublicApiDatasourceRolesResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasourceRolesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasourceRoleContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DatasourceRoleContentDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"id":{"type":"string"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["id","name"]}}}} ``` ## The PublicApiDatasourceTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDatasourceTypeDTO":{"type":"string","enum":["bigquery","mysql","postgres","snowflake","dask","spark","sparkdf","redshift","duckdb","presto","oracle","athena","trino","dremio","vertica","denodo","db2","db2z","sqlserver","databricks","synapse","fabric"]}}}} ``` ## The PublicApiDeleteAttributeResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteAttributeResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]}}}} ``` ## The PublicApiDeleteColumnMetricMonitorResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteColumnMetricMonitorResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]}}}} ``` ## The PublicApiDeleteCustomSqlMonitorResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteCustomSqlMonitorResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]}}}} ``` ## The PublicApiDeleteDataStandardResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteDataStandardResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]}}}} ``` ## The PublicApiDeleteDatasetResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteDatasetResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]}}}} ``` ## The PublicApiDeleteDatasetRoleResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteDatasetRoleResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]}}}} ``` ## The PublicApiDeleteDatasourceResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteDatasourceResponse":{"type":"object","properties":{"message":{"type":"string"}}}}}} ``` ## The PublicApiDeleteDatasourceRoleResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteDatasourceRoleResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]}}}} ``` ## The PublicApiDeleteNotificationRuleResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteNotificationRuleResponse":{"type":"object","properties":{"message":{"type":"string"}},"required":["message"]}}}} ``` ## The PublicApiDeleteRunnerResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteRunnerResponse":{"type":"object"}}}} ``` ## The PublicApiDeleteSecretResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteSecretResponse":{"type":"object","properties":{"message":{"type":"string"}}}}}} ``` ## The PublicApiDeleteUserGroupResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDeleteUserGroupResponse":{"type":"object"}}}} ``` ## The PublicApiDisableUserResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDisableUserResponse":{"type":"object"}}}} ``` ## The PublicApiDiscoveredDatasetsResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiDiscoveredDatasetsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DiscoveredDatasetContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DiscoveredDatasetContentDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"datasourceId":{"type":"string"},"id":{"type":"string"},"name":{"type":"string"},"onboarded":{"type":"boolean"},"qualifiedName":{"type":"string"}},"required":["datasourceId","id","name","onboarded","qualifiedName"]}}}} ``` ## The PublicApiEvaluateDataStandardScopeResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiEvaluateDataStandardScopeResponse":{"type":"object","properties":{"dataStandardId":{"type":"string"},"datasetIds":{"type":"array","items":{"type":"string"}}}}}}} ``` ## The PublicApiExecuteDataStandardsRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiExecuteDataStandardsRequestDTO":{"type":"object","properties":{"datasetId":{"type":"string"}},"required":["datasetId"]}}}} ``` ## The PublicApiExecuteDataStandardsResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiExecuteDataStandardsResponse":{"type":"object","properties":{"dataStandardIds":{"type":"array","items":{"type":"string"}},"scanId":{"type":"string"}}}}}} ``` ## The PublicApiGenerateContractRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiGenerateContractRequestDTO":{"type":"object","properties":{"datasetIds":{"type":"array","items":{"type":"string"}},"datasetQualifiedNames":{"type":"array","items":{"type":"string"}}}}}}} ``` ## The PublicApiGenerateContractStatusResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiGenerateContractStatusResponse":{"type":"object","properties":{"created":{"type":"string","format":"date-time"},"datasets":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiGenerateContractStatusResponse_DatasetGenerationStatus"}},"operationId":{"type":"string"},"state":{"type":"object","$ref":"#/components/schemas/GenerateContractOperationStateDTO"}}},"PublicApiGenerateContractStatusResponse_DatasetGenerationStatus":{"type":"object","properties":{"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"scanCloudUrl":{"type":"string"},"scanId":{"type":"string"},"scanState":{"type":"object","$ref":"#/components/schemas/ScanStateDTO"}}},"ScanStateDTO":{"type":"string","enum":["queuing","executing","started","cancelationRequested","timeOutRequested","canceled","timedOut","failed","completedWithErrors","completedWithFailures","completedWithWarnings","completed"]},"GenerateContractOperationStateDTO":{"type":"string","enum":["ongoing","completed","failed","canceled"]}}}} ``` ## The PublicApiGenerateContractStatusResponse\_DatasetGenerationStatus object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiGenerateContractStatusResponse_DatasetGenerationStatus":{"type":"object","properties":{"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"scanCloudUrl":{"type":"string"},"scanId":{"type":"string"},"scanState":{"type":"object","$ref":"#/components/schemas/ScanStateDTO"}}},"ScanStateDTO":{"type":"string","enum":["queuing","executing","started","cancelationRequested","timeOutRequested","canceled","timedOut","failed","completedWithErrors","completedWithFailures","completedWithWarnings","completed"]}}}} ``` ## The PublicApiGetContractResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiGetContractResponse":{"type":"object","properties":{"contract":{"type":"object","$ref":"#/components/schemas/ContractContentDTO"}}},"ContractContentDTO":{"type":"object","properties":{"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"}}}}}} ``` ## The PublicApiGetDataStandardResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiGetDataStandardResponse":{"type":"object","properties":{"dataStandard":{"type":"object","$ref":"#/components/schemas/DataStandardContentDTO"}}},"DataStandardContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"compliance":{"type":"number"},"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetCount":{"type":"integer","format":"int32"},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["id"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]}}}} ``` ## The PublicApiGetDatasourceResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiGetDatasourceResponse":{"type":"object","properties":{"datasource":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDTO"}}},"PublicApiDatasourceDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceTypeDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"PublicApiDatasourceTypeDTO":{"type":"string","enum":["bigquery","mysql","postgres","snowflake","dask","spark","sparkdf","redshift","duckdb","presto","oracle","athena","trino","dremio","vertica","denodo","db2","db2z","sqlserver","databricks","synapse","fabric"]}}}} ``` ## The PublicApiGetEncryptionKeyResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiGetEncryptionKeyResponse":{"type":"object","properties":{"encryptionKey":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"}}},"MapOfStringTo_object":{"type":"object","additionalProperties":true}}}} ``` ## The PublicApiGetRunnerResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiGetRunnerResponse":{"type":"object","properties":{"runner":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerDTO"}}},"PublicApiRunnerDTO":{"type":"object","properties":{"id":{"type":"string"},"isOnline":{"type":"boolean"},"label":{"type":"string"},"lastSeenTimestamp":{"type":"string","format":"date-time"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerTypeDTO"},"versions":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerVersionsDTO"}}},"PublicApiRunnerTypeDTO":{"type":"string","enum":["SELF_HOSTED","SODA_HOSTED"]},"PublicApiRunnerVersionsDTO":{"type":"object","properties":{"agent":{"type":"string","deprecated":true},"library":{"type":"string"},"runner":{"type":"string"}}}}}} ``` ## The PublicApiInviteUserRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiInviteUserRequestDTO":{"type":"object","properties":{"emails":{"type":"array","items":{"type":"string"}}},"required":["emails"]}}}} ``` ## The PublicApiInviteUserResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiInviteUserResponse":{"type":"object","properties":{"failedInvitations":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/FailedInvitationDTO"}},"validInvitations":{"type":"array","items":{"type":"string"}}}},"FailedInvitationDTO":{"type":"object","properties":{"email":{"type":"string"},"error":{"type":"string"}}}}}} ``` ## The PublicApiListAttributesResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiListAttributesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/AttributeContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"AttributeContentDTO":{"type":"object","properties":{"allowedValues":{"type":"array","items":{"type":"string"}},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"resourceType":{"type":"object","$ref":"#/components/schemas/AttributeResourceTypeDTO"},"type":{"type":"object","$ref":"#/components/schemas/AttributeTypeDTO"}},"required":["id","label","name","resourceType","type"]},"AttributeResourceTypeDTO":{"type":"string","enum":["dataset","check","column"]},"AttributeTypeDTO":{"type":"string","enum":["text","number","checkbox","datetime","singleSelect","multiSelect"]}}}} ``` ## The PublicApiListContractVersionsResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiListContractVersionsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ContractVersionDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"ContractVersionDTO":{"type":"object","properties":{"checksum":{"type":"string"},"contents":{"type":"string"},"contractId":{"type":"string"},"created":{"type":"string","format":"date-time"},"creatorId":{"type":"string"},"id":{"type":"string"}}}}}} ``` ## The PublicApiListContractsResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiListContractsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ContractContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"ContractContentDTO":{"type":"object","properties":{"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"}}}}}} ``` ## The PublicApiListDataStandardChecksResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiListDataStandardChecksResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardCheckContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DataStandardCheckContentDTO":{"type":"object","properties":{"checkType":{"type":"object","$ref":"#/components/schemas/CheckTypeDTO"},"compliance":{"type":"number"},"datasetCount":{"type":"integer","format":"int32"},"failingCount":{"type":"integer","format":"int32"},"lastScanTime":{"type":"string","format":"date-time"},"name":{"type":"string"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"},"worstLevel":{"type":"object","$ref":"#/components/schemas/CheckResultLevelDTO"}}},"CheckTypeDTO":{"type":"string","enum":["missing","invalid","row_count","aggregate","schema","duplicate","freshness","failed_rows","metric","group_by","row_count_diff","aggregate_diff","freshness_diff","metric_diff","rows_diff","duplicate_diff","llm","hierarchy","fuzzy_duplicate","reference_diff"]},"CheckResultLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated","excluded"]}}}} ``` ## The PublicApiListDataStandardDatasetsResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiListDataStandardDatasetsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardDatasetContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DataStandardDatasetContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanId":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"qualifiedName":{"type":"string"}},"required":["id","name"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]}}}} ``` ## The PublicApiListDataStandardsResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiListDataStandardsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DataStandardContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"compliance":{"type":"number"},"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetCount":{"type":"integer","format":"int32"},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["id"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]}}}} ``` ## The PublicApiListDatasetColumnsResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiListDatasetColumnsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiDatasetColumnContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"PublicApiDatasetColumnContentDTO":{"type":"object","properties":{"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"},"columnId":{"type":"string"},"columnName":{"type":"string"}},"required":["attributes","columnId","columnName"]},"MapOfStringTo_object":{"type":"object","additionalProperties":true}}}} ``` ## The PublicApiListDatasourcesResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiListDatasourcesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"PublicApiDatasourceDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceTypeDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"PublicApiDatasourceTypeDTO":{"type":"string","enum":["bigquery","mysql","postgres","snowflake","dask","spark","sparkdf","redshift","duckdb","presto","oracle","athena","trino","dremio","vertica","denodo","db2","db2z","sqlserver","databricks","synapse","fabric"]}}}} ``` ## The PublicApiListNotificationRulesResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiListNotificationRulesResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiNotificationRuleDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"PublicApiNotificationRuleDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"customization":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationDTO"},"filter":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"id":{"type":"string"},"isDefault":{"type":"boolean"},"lastTriggeredAt":{"type":"string","format":"date-time"},"name":{"type":"string"},"ownerId":{"type":"string"},"recipients":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientDTO"}},"ruleType":{"type":"object","$ref":"#/components/schemas/PublicApiRuleTypeDTO"},"status":{"type":"object","$ref":"#/components/schemas/NotificationRuleStatusDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"NotificationCustomizationDTO":{"type":"object","properties":{"message":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationTypeDTO"}},"required":["type"]},"NotificationCustomizationTypeDTO":{"type":"string","enum":["rich","minimal"]},"NotificationFilterDTO":{"type":"object","properties":{"expression":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"expressions":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"}},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationFilterTypeDTO"},"value":{"type":"string"},"values":{"type":"array","items":{"type":"string"}}},"required":["type"]},"NotificationFilterFieldDTO":{"type":"object","properties":{"attributeName":{"type":"string"},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldTypeDTO"}},"required":["field"]},"NotificationFilterFieldTypeDTO":{"type":"string","enum":["datasourceId","datasourceLabel","datasetId","datasetLabel","datasetOwnerId","datasetTags","checkId","checkName","checkOwnerId","columnId","metricType","datasetResourceAttribute","checkResourceAttribute"]},"NotificationFilterTypeDTO":{"type":"string","enum":["equals","contains","in","and","or","not"]},"NotificationRecipientDTO":{"type":"object","properties":{"customRoleId":{"type":"string"},"email":{"type":"string"},"integrationChannel":{"type":"string"},"integrationId":{"type":"string"},"role":{"type":"object","$ref":"#/components/schemas/NotificationRecipientRoleDTO"},"triggers":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTriggerDTO"}},"type":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["triggers","type"]},"NotificationRecipientRoleDTO":{"type":"string","enum":["datasetOwner","monitorOwner"]},"NotificationRecipientTriggerDTO":{"type":"object","properties":{"level":{"type":"object","$ref":"#/components/schemas/NotificationLevelDTO"},"scanState":{"type":"object","$ref":"#/components/schemas/NotificationScanStateDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationTriggerTypeDTO"}},"required":["type"]},"NotificationLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated"]},"NotificationScanStateDTO":{"type":"string","enum":["failed","timedOut","canceled","completedWithErrors"]},"NotificationTriggerTypeDTO":{"type":"string","enum":["testResultLevel","scanState"]},"NotificationRecipientTypeDTO":{"type":"string","enum":["email","user","userGroup","integration","resourceRole","customRole"]},"PublicApiRuleTypeDTO":{"type":"string","enum":["check","monitor"]},"NotificationRuleStatusDTO":{"type":"string","enum":["active","paused"]}}}} ``` ## The PublicApiListRunnersResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiListRunnersResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"PublicApiRunnerDTO":{"type":"object","properties":{"id":{"type":"string"},"isOnline":{"type":"boolean"},"label":{"type":"string"},"lastSeenTimestamp":{"type":"string","format":"date-time"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerTypeDTO"},"versions":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerVersionsDTO"}}},"PublicApiRunnerTypeDTO":{"type":"string","enum":["SELF_HOSTED","SODA_HOSTED"]},"PublicApiRunnerVersionsDTO":{"type":"object","properties":{"agent":{"type":"string","deprecated":true},"library":{"type":"string"},"runner":{"type":"string"}}}}}} ``` ## The PublicApiListSecretsResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiListSecretsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiSecretDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"PublicApiSecretDTO":{"type":"object","properties":{"created":{"type":"string","format":"date-time"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"}}}}}} ``` ## The PublicApiNotificationRuleDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiNotificationRuleDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"customization":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationDTO"},"filter":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"id":{"type":"string"},"isDefault":{"type":"boolean"},"lastTriggeredAt":{"type":"string","format":"date-time"},"name":{"type":"string"},"ownerId":{"type":"string"},"recipients":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientDTO"}},"ruleType":{"type":"object","$ref":"#/components/schemas/PublicApiRuleTypeDTO"},"status":{"type":"object","$ref":"#/components/schemas/NotificationRuleStatusDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"NotificationCustomizationDTO":{"type":"object","properties":{"message":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationTypeDTO"}},"required":["type"]},"NotificationCustomizationTypeDTO":{"type":"string","enum":["rich","minimal"]},"NotificationFilterDTO":{"type":"object","properties":{"expression":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"expressions":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"}},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationFilterTypeDTO"},"value":{"type":"string"},"values":{"type":"array","items":{"type":"string"}}},"required":["type"]},"NotificationFilterFieldDTO":{"type":"object","properties":{"attributeName":{"type":"string"},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldTypeDTO"}},"required":["field"]},"NotificationFilterFieldTypeDTO":{"type":"string","enum":["datasourceId","datasourceLabel","datasetId","datasetLabel","datasetOwnerId","datasetTags","checkId","checkName","checkOwnerId","columnId","metricType","datasetResourceAttribute","checkResourceAttribute"]},"NotificationFilterTypeDTO":{"type":"string","enum":["equals","contains","in","and","or","not"]},"NotificationRecipientDTO":{"type":"object","properties":{"customRoleId":{"type":"string"},"email":{"type":"string"},"integrationChannel":{"type":"string"},"integrationId":{"type":"string"},"role":{"type":"object","$ref":"#/components/schemas/NotificationRecipientRoleDTO"},"triggers":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTriggerDTO"}},"type":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["triggers","type"]},"NotificationRecipientRoleDTO":{"type":"string","enum":["datasetOwner","monitorOwner"]},"NotificationRecipientTriggerDTO":{"type":"object","properties":{"level":{"type":"object","$ref":"#/components/schemas/NotificationLevelDTO"},"scanState":{"type":"object","$ref":"#/components/schemas/NotificationScanStateDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationTriggerTypeDTO"}},"required":["type"]},"NotificationLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated"]},"NotificationScanStateDTO":{"type":"string","enum":["failed","timedOut","canceled","completedWithErrors"]},"NotificationTriggerTypeDTO":{"type":"string","enum":["testResultLevel","scanState"]},"NotificationRecipientTypeDTO":{"type":"string","enum":["email","user","userGroup","integration","resourceRole","customRole"]},"PublicApiRuleTypeDTO":{"type":"string","enum":["check","monitor"]},"NotificationRuleStatusDTO":{"type":"string","enum":["active","paused"]}}}} ``` ## The PublicApiNotificationRuleRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiNotificationRuleRequestDTO":{"type":"object","properties":{"customization":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationDTO"},"filter":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"name":{"type":"string"},"recipients":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientDTO"}},"ruleType":{"type":"object","$ref":"#/components/schemas/PublicApiRuleTypeDTO"},"status":{"type":"object","$ref":"#/components/schemas/NotificationRuleStatusDTO"}},"required":["name","recipients","ruleType"]},"NotificationCustomizationDTO":{"type":"object","properties":{"message":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/NotificationCustomizationTypeDTO"}},"required":["type"]},"NotificationCustomizationTypeDTO":{"type":"string","enum":["rich","minimal"]},"NotificationFilterDTO":{"type":"object","properties":{"expression":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"},"expressions":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationFilterDTO"}},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationFilterTypeDTO"},"value":{"type":"string"},"values":{"type":"array","items":{"type":"string"}}},"required":["type"]},"NotificationFilterFieldDTO":{"type":"object","properties":{"attributeName":{"type":"string"},"field":{"type":"object","$ref":"#/components/schemas/NotificationFilterFieldTypeDTO"}},"required":["field"]},"NotificationFilterFieldTypeDTO":{"type":"string","enum":["datasourceId","datasourceLabel","datasetId","datasetLabel","datasetOwnerId","datasetTags","checkId","checkName","checkOwnerId","columnId","metricType","datasetResourceAttribute","checkResourceAttribute"]},"NotificationFilterTypeDTO":{"type":"string","enum":["equals","contains","in","and","or","not"]},"NotificationRecipientDTO":{"type":"object","properties":{"customRoleId":{"type":"string"},"email":{"type":"string"},"integrationChannel":{"type":"string"},"integrationId":{"type":"string"},"role":{"type":"object","$ref":"#/components/schemas/NotificationRecipientRoleDTO"},"triggers":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTriggerDTO"}},"type":{"type":"object","$ref":"#/components/schemas/NotificationRecipientTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["triggers","type"]},"NotificationRecipientRoleDTO":{"type":"string","enum":["datasetOwner","monitorOwner"]},"NotificationRecipientTriggerDTO":{"type":"object","properties":{"level":{"type":"object","$ref":"#/components/schemas/NotificationLevelDTO"},"scanState":{"type":"object","$ref":"#/components/schemas/NotificationScanStateDTO"},"type":{"type":"object","$ref":"#/components/schemas/NotificationTriggerTypeDTO"}},"required":["type"]},"NotificationLevelDTO":{"type":"string","enum":["info","warning","critical","not_evaluated"]},"NotificationScanStateDTO":{"type":"string","enum":["failed","timedOut","canceled","completedWithErrors"]},"NotificationTriggerTypeDTO":{"type":"string","enum":["testResultLevel","scanState"]},"NotificationRecipientTypeDTO":{"type":"string","enum":["email","user","userGroup","integration","resourceRole","customRole"]},"PublicApiRuleTypeDTO":{"type":"string","enum":["check","monitor"]},"NotificationRuleStatusDTO":{"type":"string","enum":["active","paused"]}}}} ``` ## The PublicApiOnboardDatasetsRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiOnboardDatasetsRequestDTO":{"type":"object","properties":{"discoveredDatasetIds":{"type":"array","items":{"type":"string"}}},"required":["discoveredDatasetIds"]}}}} ``` ## The PublicApiOnboardDatasetsStatusResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiOnboardDatasetsStatusResponse":{"type":"object","properties":{"ended":{"type":"string","format":"date-time"},"id":{"type":"string"},"message":{"type":"string"},"started":{"type":"string","format":"date-time"},"state":{"type":"object","$ref":"#/components/schemas/PublicApiOnboardingStateDTO"}},"required":["id","state"]},"PublicApiOnboardingStateDTO":{"type":"string","enum":["queued","processing","completed","failed","cancelled"]}}}} ``` ## The PublicApiOnboardingStateDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiOnboardingStateDTO":{"type":"string","enum":["queued","processing","completed","failed","cancelled"]}}}} ``` ## The PublicApiPostProcessingStageDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiPostProcessingStageDTO":{"type":"object","properties":{"ended":{"type":"string","format":"date-time"},"name":{"type":"object","$ref":"#/components/schemas/PublicApiPostProcessingStageTypeDTO"},"started":{"type":"string","format":"date-time"},"state":{"type":"object","$ref":"#/components/schemas/PublicApiPostProcessingStageStateDTO"}},"required":["name"]},"PublicApiPostProcessingStageTypeDTO":{"type":"string","enum":["diagnosticWarehouse"]},"PublicApiPostProcessingStageStateDTO":{"type":"string","enum":["ongoing","completed","failed"]}}}} ``` ## The PublicApiPostProcessingStageStateDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiPostProcessingStageStateDTO":{"type":"string","enum":["ongoing","completed","failed"]}}}} ``` ## The PublicApiPostProcessingStageTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiPostProcessingStageTypeDTO":{"type":"string","enum":["diagnosticWarehouse"]}}}} ``` ## The PublicApiPreviewDataStandardScopeRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiPreviewDataStandardScopeRequestDTO":{"type":"object","properties":{"page":{"type":"integer","format":"int32"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"size":{"type":"integer","format":"int32"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}}}}} ``` ## The PublicApiPreviewDataStandardScopeResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiPreviewDataStandardScopeResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeMatchingDatasetContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"DataStandardScopeMatchingDatasetContentDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"qualifiedName":{"type":"string"}},"required":["datasourceId","id","name"]}}}} ``` ## The PublicApiPublishContractRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiPublishContractRequestDTO":{"type":"object","properties":{"contents":{"type":"string"}},"required":["contents"]}}}} ``` ## The PublicApiPublishContractResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiPublishContractResponse":{"type":"object","properties":{"contract":{"type":"object","$ref":"#/components/schemas/ContractContentDTO"}}},"ContractContentDTO":{"type":"object","properties":{"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetId":{"type":"string"},"datasetQualifiedName":{"type":"string"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"}}}}}} ``` ## The PublicApiRuleTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiRuleTypeDTO":{"type":"string","enum":["check","monitor"]}}}} ``` ## The PublicApiRunnerDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiRunnerDTO":{"type":"object","properties":{"id":{"type":"string"},"isOnline":{"type":"boolean"},"label":{"type":"string"},"lastSeenTimestamp":{"type":"string","format":"date-time"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerTypeDTO"},"versions":{"type":"object","$ref":"#/components/schemas/PublicApiRunnerVersionsDTO"}}},"PublicApiRunnerTypeDTO":{"type":"string","enum":["SELF_HOSTED","SODA_HOSTED"]},"PublicApiRunnerVersionsDTO":{"type":"object","properties":{"agent":{"type":"string","deprecated":true},"library":{"type":"string"},"runner":{"type":"string"}}}}}} ``` ## The PublicApiRunnerTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiRunnerTypeDTO":{"type":"string","enum":["SELF_HOSTED","SODA_HOSTED"]}}}} ``` ## The PublicApiRunnerVersionsDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiRunnerVersionsDTO":{"type":"object","properties":{"agent":{"type":"string","deprecated":true},"library":{"type":"string"},"runner":{"type":"string"}}}}}} ``` ## The PublicApiScanDefinitionSlim object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiScanDefinitionSlim":{"type":"object","properties":{"id":{"type":"string"},"name":{"type":"string"}},"required":["id","name"]}}}} ``` ## The PublicApiScanLogsResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiScanLogsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/LogsContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"LogsContentDTO":{"type":"object","properties":{"index":{"type":"integer","format":"int32"},"level":{"type":"object","$ref":"#/components/schemas/LogLevelDTO"},"message":{"type":"string"},"timestamp":{"type":"string","format":"date-time"}},"required":["index","level","message","timestamp"]},"LogLevelDTO":{"type":"string","enum":["debug","info","warning","error"]}}}} ``` ## The PublicApiScanStatusResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiScanStatusResponse":{"type":"object","properties":{"agentId":{"type":"string","deprecated":true},"checks":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiCheckSlimDTO"}},"cloudUrl":{"type":"string"},"contractDatasetCloudUrl":{"type":"string"},"created":{"type":"string","format":"date-time"},"ended":{"type":"string","format":"date-time"},"errors":{"type":"integer","format":"int32"},"failures":{"type":"integer","format":"int32"},"id":{"type":"string"},"postProcessingStages":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiPostProcessingStageDTO"}},"runnerId":{"type":"string"},"scanDefinition":{"type":"object","$ref":"#/components/schemas/PublicApiScanDefinitionSlim"},"scanTime":{"type":"string","format":"date-time"},"started":{"type":"string","format":"date-time"},"state":{"type":"object","$ref":"#/components/schemas/ScanStateDTO"},"submitted":{"type":"string","format":"date-time"},"warnings":{"type":"integer","format":"int32"}},"required":["cloudUrl","created","id","state"]},"PublicApiCheckSlimDTO":{"type":"object","properties":{"evaluationStatus":{"type":"object","$ref":"#/components/schemas/EvaluationStatusDTO"},"id":{"type":"string"}}},"EvaluationStatusDTO":{"type":"string","enum":["pass","warn","fail","notEvaluated","excluded"]},"PublicApiPostProcessingStageDTO":{"type":"object","properties":{"ended":{"type":"string","format":"date-time"},"name":{"type":"object","$ref":"#/components/schemas/PublicApiPostProcessingStageTypeDTO"},"started":{"type":"string","format":"date-time"},"state":{"type":"object","$ref":"#/components/schemas/PublicApiPostProcessingStageStateDTO"}},"required":["name"]},"PublicApiPostProcessingStageTypeDTO":{"type":"string","enum":["diagnosticWarehouse"]},"PublicApiPostProcessingStageStateDTO":{"type":"string","enum":["ongoing","completed","failed"]},"PublicApiScanDefinitionSlim":{"type":"object","properties":{"id":{"type":"string"},"name":{"type":"string"}},"required":["id","name"]},"ScanStateDTO":{"type":"string","enum":["queuing","executing","started","cancelationRequested","timeOutRequested","canceled","timedOut","failed","completedWithErrors","completedWithFailures","completedWithWarnings","completed"]}}}} ``` ## The PublicApiSecretDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiSecretDTO":{"type":"object","properties":{"created":{"type":"string","format":"date-time"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"}}}}}} ``` ## The PublicApiSetColumnAttributesRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiSetColumnAttributesRequestDTO":{"type":"object","properties":{"columnAttributes":{"type":"object","$ref":"#/components/schemas/MapOfStringToMapOfStringTo_object"}},"required":["columnAttributes"]},"MapOfStringToMapOfStringTo_object":{"type":"object","additionalProperties":true}}}} ``` ## The PublicApiTestConnectionRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiTestConnectionRequestDTO":{"type":"object","properties":{"configurationFileContents":{"type":"string"},"runnerId":{"type":"string"}},"required":["configurationFileContents","runnerId"]}}}} ``` ## The PublicApiTestConnectionStateDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiTestConnectionStateDTO":{"type":"string","enum":["queued","processing","completed","failed","cancelled"]}}}} ``` ## The PublicApiTestConnectionStatusResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiTestConnectionStatusResponse":{"type":"object","properties":{"id":{"type":"string"},"message":{"type":"string"},"started":{"type":"string","format":"date-time"},"state":{"type":"object","$ref":"#/components/schemas/PublicApiTestConnectionStateDTO"}},"required":["id","state"]},"PublicApiTestConnectionStateDTO":{"type":"string","enum":["queued","processing","completed","failed","cancelled"]}}}} ``` ## The PublicApiTestDataStandardRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiTestDataStandardRequestDTO":{"type":"object","properties":{"contents":{"type":"string"},"datasetId":{"type":"string"}},"required":["contents","datasetId"]}}}} ``` ## The PublicApiTestDataStandardResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiTestDataStandardResponse":{"type":"object","properties":{"scanId":{"type":"string"}}}}}} ``` ## The PublicApiTestLoginResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiTestLoginResponse":{"type":"object","properties":{"organisationName":{"type":"string"}}}}}} ``` ## The PublicApiUpdateAttributeRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateAttributeRequestDTO":{"type":"object","properties":{"allowedValues":{"type":"array","items":{"type":"string"}},"description":{"type":"string"},"label":{"type":"string"}}}}}} ``` ## The PublicApiUpdateAttributeResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateAttributeResponse":{"type":"object","properties":{"attribute":{"type":"object","$ref":"#/components/schemas/AttributeContentDTO"}},"required":["attribute"]},"AttributeContentDTO":{"type":"object","properties":{"allowedValues":{"type":"array","items":{"type":"string"}},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"resourceType":{"type":"object","$ref":"#/components/schemas/AttributeResourceTypeDTO"},"type":{"type":"object","$ref":"#/components/schemas/AttributeTypeDTO"}},"required":["id","label","name","resourceType","type"]},"AttributeResourceTypeDTO":{"type":"string","enum":["dataset","check","column"]},"AttributeTypeDTO":{"type":"string","enum":["text","number","checkbox","datetime","singleSelect","multiSelect"]}}}} ``` ## The PublicApiUpdateColumnMetricMonitorRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateColumnMetricMonitorRequestDTO":{"type":"object","properties":{"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"}},"required":["configuration"]},"ColumnMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByColumnConfigurationDTO"}},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}}}}} ``` ## The PublicApiUpdateColumnMetricMonitorResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateColumnMetricMonitorResponse":{"type":"object","properties":{"columnMetricMonitor":{"type":"object","$ref":"#/components/schemas/ColumnMetricMonitorDTO"}},"required":["columnMetricMonitor"]},"ColumnMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/ColumnMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/ColumnMonitorMetricTypeDTO"}},"required":["checkId","columnName","configuration","metricType"]},"ColumnMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumns":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/GroupByColumnConfigurationDTO"}},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"GroupByColumnConfigurationDTO":{"type":"object","properties":{"columnName":{"type":"string"},"excludedValues":{"type":"array","items":{"type":"string"}}},"required":["columnName"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"ColumnMonitorMetricTypeDTO":{"type":"string","enum":["missingPercentage","duplicatePercentage","average","columnFreshness","sum","distinctValuesCount","count","minimumValue","maximumValue","duplicateValuesPercentage","minimumLength","maximumLength","averageLength","standardDeviation","variance","q1","median","q3"]}}}} ``` ## The PublicApiUpdateCustomSqlMonitorRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateCustomSqlMonitorRequestDTO":{"type":"object","properties":{"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/CustomSqlMonitorConfigurationUpdateDTO"},"name":{"type":"string"}}},"CustomSqlMonitorConfigurationUpdateDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}}}}} ``` ## The PublicApiUpdateCustomSqlMonitorResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateCustomSqlMonitorResponse":{"type":"object","properties":{"customSqlMetricMonitor":{"type":"object","$ref":"#/components/schemas/CustomMetricMonitorDTO"}},"required":["customSqlMetricMonitor"]},"CustomMetricMonitorDTO":{"type":"object","properties":{"checkId":{"type":"string"},"columnName":{"type":"string"},"configuration":{"type":"object","$ref":"#/components/schemas/CustomSqlMonitorConfigurationDTO"},"monitorName":{"type":"string"}},"required":["checkId","configuration","monitorName"]},"CustomSqlMonitorConfigurationDTO":{"type":"object","properties":{"groupByColumnNames":{"type":"array","items":{"type":"string"}},"resultMetric":{"type":"string"},"sqlQuery":{"type":"string"},"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}},"required":["resultMetric","sqlQuery"]},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}}}}} ``` ## The PublicApiUpdateDataStandardResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDataStandardResponse":{"type":"object","properties":{"dataStandard":{"type":"object","$ref":"#/components/schemas/DataStandardContentDTO"}}},"DataStandardContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"compliance":{"type":"number"},"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetCount":{"type":"integer","format":"int32"},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["id"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]}}}} ``` ## The PublicApiUpdateDataStandardStatusRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDataStandardStatusRequestDTO":{"type":"object","properties":{"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["status"]},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]}}}} ``` ## The PublicApiUpdateDataStandardStatusResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDataStandardStatusResponse":{"type":"object","properties":{"dataStandard":{"type":"object","$ref":"#/components/schemas/DataStandardContentDTO"}}},"DataStandardContentDTO":{"type":"object","properties":{"checkResults":{"type":"object","$ref":"#/components/schemas/DataStandardCheckResultsDTO"},"cloudUrl":{"type":"string"},"compliance":{"type":"number"},"contents":{"type":"string"},"created":{"type":"string","format":"date-time"},"datasetCount":{"type":"integer","format":"int32"},"description":{"type":"string"},"id":{"type":"string"},"label":{"type":"string"},"lastScanTime":{"type":"string","format":"date-time"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetOwnerDTO"}},"schedule":{"type":"object","$ref":"#/components/schemas/DataStandardScheduleDTO"},"scope":{"type":"object","$ref":"#/components/schemas/DataStandardScopeDTO"},"status":{"type":"object","$ref":"#/components/schemas/DataStandardStatusDTO"}},"required":["id"]},"DataStandardCheckResultsDTO":{"type":"object","properties":{"failingCount":{"type":"integer","format":"int32"},"notEvaluatedCount":{"type":"integer","format":"int32"},"passingCount":{"type":"integer","format":"int32"},"warningCount":{"type":"integer","format":"int32"}}},"DatasetOwnerDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"user":{"type":"object","$ref":"#/components/schemas/UserContentDTO"},"userGroup":{"type":"object","$ref":"#/components/schemas/UserGroupContentBasicDTO"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]},"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]},"DataStandardScheduleDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"DataStandardScopeDTO":{"type":"object","properties":{"rules":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DataStandardScopeRuleDTO"}}}},"DataStandardScopeRuleDTO":{"type":"object","properties":{"datasourceId":{"type":"string"},"name":{"type":"string"},"schema":{"type":"string"}}},"DataStandardStatusDTO":{"type":"string","enum":["draft","active","paused","invalid"]}}}} ``` ## The PublicApiUpdateDatasetDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasetDTO":{"type":"object","properties":{"attributes":{"type":"object","$ref":"#/components/schemas/MapOfStringTo_object"},"computeWarehouseOverride":{"type":"object","$ref":"#/components/schemas/UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO"},"diagnosticsWarehouse":{"type":"object","$ref":"#/components/schemas/UpdateDatasetDiagnosticsWarehouseConfigurationRequestDTO"},"label":{"type":"string"},"metricMonitoring":{"type":"object","$ref":"#/components/schemas/UpdateDatasetMetricMonitoringConfigurationRequestDTO"},"owners":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasetOwnerRequestDTO"}},"partitionColumn":{"type":"string","deprecated":true},"profiling":{"type":"object","$ref":"#/components/schemas/UpdateDatasetProfilingConfigurationRequestDTO"},"tags":{"type":"array","items":{"type":"string"}},"timePartition":{"type":"object","$ref":"#/components/schemas/UpdateDatasetTimePartitionRequestDTO"}}},"MapOfStringTo_object":{"type":"object","additionalProperties":true},"UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO":{"type":"object","properties":{"computeWarehouseConfiguration":{"type":"object","$ref":"#/components/schemas/UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO_WarehouseConfigurationDTO"},"enabled":{"type":"boolean"}}},"UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO_WarehouseConfigurationDTO":{"type":"object","properties":{"name":{"type":"string"}},"required":["name"]},"UpdateDatasetDiagnosticsWarehouseConfigurationRequestDTO":{"type":"object","properties":{"failedRowsConfiguration":{"type":"object","$ref":"#/components/schemas/DatasetDiagnosticsWarehouseFailedRowsConfigurationRequest"},"scanAndResultsConfiguration":{"type":"object","$ref":"#/components/schemas/DatasetDiagnosticsWarehouseScanAndResultsConfigurationRequest"},"schema":{"type":"string"}}},"DatasetDiagnosticsWarehouseFailedRowsConfigurationRequest":{"type":"object","properties":{"enabled":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"maxRowCount":{"type":"integer","format":"int32"},"schema":{"type":"string","deprecated":true},"storageStrategy":{"type":"object","$ref":"#/components/schemas/DiagnosticsWarehouseFailedRowsStorageStrategyDTO"},"uniqueKeyColumnNames":{"type":"array","items":{"type":"string"}}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]},"DiagnosticsWarehouseFailedRowsStorageStrategyDTO":{"type":"string","enum":["fullFailedRows","uniqueKeys","scansAndResultsOnly"]},"DatasetDiagnosticsWarehouseScanAndResultsConfigurationRequest":{"type":"object","properties":{"enabled":{"type":"boolean"}}},"UpdateDatasetMetricMonitoringConfigurationRequestDTO":{"type":"object","properties":{"datasetMetricMonitorsConfiguration":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetMetricMonitorConfigurationDTO"}},"enabled":{"type":"boolean"},"historicalMetricCollectionScanStartDate":{"type":"string","format":"date-time"},"scanSchedule":{"type":"object","$ref":"#/components/schemas/ZonedCronDTO"}}},"DatasetMetricMonitorConfigurationDTO":{"type":"object","properties":{"configuration":{"type":"object","$ref":"#/components/schemas/DatasetMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/DatasetMonitorMetricTypeDTO"}},"required":["configuration","metricType"]},"DatasetMonitorConfigurationDTO":{"type":"object","properties":{"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"DatasetMonitorMetricTypeDTO":{"type":"string","enum":["rowCount","freshness","schema","rowsInserted","totalRowCountChange","timeliness"]},"ZonedCronDTO":{"type":"object","properties":{"cronExpression":{"type":"string"},"timezone":{"type":"string"}},"required":["cronExpression","timezone"]},"PublicApiUpdateDatasetOwnerRequestDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]},"UpdateDatasetProfilingConfigurationRequestDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"profilingSamplingStrategy":{"type":"object","$ref":"#/components/schemas/ProfilingSamplingStrategyConfigurationDTO"},"scanSchedule":{"type":"object","$ref":"#/components/schemas/ZonedCronDTO"}}},"ProfilingSamplingStrategyConfigurationDTO":{"type":"object","properties":{"numberOfRows":{"type":"integer","format":"int32"},"numberOfUnits":{"type":"integer","format":"int32"},"unitOfTime":{"type":"object","$ref":"#/components/schemas/UnitOfTimeDTO"}}},"UnitOfTimeDTO":{"type":"string","enum":["hours","days","weeks"]},"UpdateDatasetTimePartitionRequestDTO":{"type":"object","properties":{"partitionColumn":{"type":"string"},"sqlExpression":{"type":"string"}}}}}} ``` ## The PublicApiUpdateDatasetOwnerRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasetOwnerRequestDTO":{"type":"object","properties":{"type":{"type":"object","$ref":"#/components/schemas/DatasetOwnerTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["type"]},"DatasetOwnerTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The PublicApiUpdateDatasetResponsibilitiesRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasetResponsibilitiesRequestDTO":{"type":"object","properties":{"responsibilities":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetResponsibilityContentRequestDTO"}}},"required":["responsibilities"]},"DatasetResponsibilityContentRequestDTO":{"type":"object","properties":{"roleId":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["roleId","type"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The PublicApiUpdateDatasetRoleResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasetRoleResponse":{"type":"object","properties":{"datasetRole":{"type":"object","$ref":"#/components/schemas/DatasetRoleContentDTO"}},"required":["datasetRole"]},"DatasetRoleContentDTO":{"type":"object","properties":{"configureDataset":{"type":"boolean"},"createCheckProposals":{"type":"boolean"},"deleteDataset":{"type":"boolean"},"executeContracts":{"type":"boolean","deprecated":true},"id":{"type":"string"},"manageChecks":{"type":"boolean"},"manageContracts":{"type":"boolean"},"manageIncidents":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"publishContracts":{"type":"boolean","deprecated":true},"viewFailedRows":{"type":"boolean"},"viewProfilingAndSamples":{"type":"boolean"}},"required":["id","name"]}}}} ``` ## The PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO":{"type":"object","properties":{"configurationFileContents":{"type":"string"},"enabled":{"type":"boolean"},"failedRowsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_FailedRowsConfigurationRequestDTO"},"reuseDatasource":{"type":"boolean"},"scanAndResultsConfiguration":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ScanAndResultsConfigurationRequestDTO"},"tableNameTemplate":{"type":"string"}}},"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_FailedRowsConfigurationRequestDTO":{"type":"object","properties":{"actionButton":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ActionButtonRequestDTO"},"enabled":{"type":"boolean"},"exposeQueries":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"locationMessage":{"type":"string"},"maxRowCount":{"type":"integer","format":"int32"}}},"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ActionButtonRequestDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"title":{"type":"string"},"url":{"type":"string"}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]},"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ScanAndResultsConfigurationRequestDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}}}}} ``` ## The PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO\_ActionButtonRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ActionButtonRequestDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"title":{"type":"string"},"url":{"type":"string"}}}}}} ``` ## The PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO\_FailedRowsConfigurationRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_FailedRowsConfigurationRequestDTO":{"type":"object","properties":{"actionButton":{"type":"object","$ref":"#/components/schemas/PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ActionButtonRequestDTO"},"enabled":{"type":"boolean"},"exposeQueries":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"locationMessage":{"type":"string"},"maxRowCount":{"type":"integer","format":"int32"}}},"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ActionButtonRequestDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"title":{"type":"string"},"url":{"type":"string"}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]}}}} ``` ## The PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO\_ScanAndResultsConfigurationRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasourceDiagnosticsWarehouseRequestDTO_ScanAndResultsConfigurationRequestDTO":{"type":"object","properties":{"enabled":{"type":"boolean"}}}}}} ``` ## The PublicApiUpdateDatasourceRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasourceRequestDTO":{"type":"object","properties":{"configurationFileContents":{"type":"string"},"label":{"type":"string"},"runnerId":{"type":"string"}}}}}} ``` ## The PublicApiUpdateDatasourceResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasourceResponse":{"type":"object","properties":{"datasource":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceDTO"}}},"PublicApiDatasourceDTO":{"type":"object","properties":{"createdAt":{"type":"string","format":"date-time"},"id":{"type":"string"},"label":{"type":"string"},"name":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/PublicApiDatasourceTypeDTO"},"updatedAt":{"type":"string","format":"date-time"}}},"PublicApiDatasourceTypeDTO":{"type":"string","enum":["bigquery","mysql","postgres","snowflake","dask","spark","sparkdf","redshift","duckdb","presto","oracle","athena","trino","dremio","vertica","denodo","db2","db2z","sqlserver","databricks","synapse","fabric"]}}}} ``` ## The PublicApiUpdateDatasourceResponsibilitiesRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasourceResponsibilitiesRequestDTO":{"type":"object","properties":{"responsibilities":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasourceResponsibilityContentRequestDTO"}}},"required":["responsibilities"]},"DatasourceResponsibilityContentRequestDTO":{"type":"object","properties":{"roleId":{"type":"string"},"type":{"type":"object","$ref":"#/components/schemas/ResponsibilityTypeDTO"},"userGroupId":{"type":"string"},"userId":{"type":"string"}},"required":["roleId","type"]},"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The PublicApiUpdateDatasourceRoleResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateDatasourceRoleResponse":{"type":"object","properties":{"datasourceRole":{"type":"object","$ref":"#/components/schemas/DatasourceRoleContentDTO"}},"required":["datasourceRole"]},"DatasourceRoleContentDTO":{"type":"object","properties":{"createDatasets":{"type":"boolean"},"deleteDatasource":{"type":"boolean"},"id":{"type":"string"},"manageDatasourceSettings":{"type":"boolean"},"managePermissions":{"type":"boolean"},"name":{"type":"string"},"viewDatasource":{"type":"boolean"}},"required":["id","name"]}}}} ``` ## The PublicApiUpdateIncidentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateIncidentDTO":{"type":"object","properties":{"description":{"type":"string"},"resolutionNotes":{"type":"string"},"severity":{"type":"object","$ref":"#/components/schemas/IncidentSeverityDTO"},"status":{"type":"object","$ref":"#/components/schemas/IncidentStatusDTO"},"title":{"type":"string"}}},"IncidentSeverityDTO":{"type":"string","enum":["minor","major","critical"]},"IncidentStatusDTO":{"type":"string","enum":["reported","investigating","fixing","resolved"]}}}} ``` ## The PublicApiUpdateSecretRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateSecretRequestDTO":{"type":"object","properties":{"encryptedValue":{"type":"string"},"encryptionKey":{"type":"string"}},"required":["encryptedValue","encryptionKey"]}}}} ``` ## The PublicApiUpdateSecretResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateSecretResponse":{"type":"object","properties":{"secret":{"type":"object","$ref":"#/components/schemas/PublicApiSecretDTO"}}},"PublicApiSecretDTO":{"type":"object","properties":{"created":{"type":"string","format":"date-time"},"id":{"type":"string"},"lastUpdated":{"type":"string","format":"date-time"},"name":{"type":"string"}}}}}} ``` ## The PublicApiUpdateUserGroupRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUpdateUserGroupRequestDTO":{"type":"object","properties":{"userIds":{"type":"array","items":{"type":"string"}}}}}}} ``` ## The PublicApiUserGroupsResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUserGroupsResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/UserGroupContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"UserGroupContentDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"},"users":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/UserContentDTO"}}},"required":["name","userGroupId","users"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]}}}} ``` ## The PublicApiUsersResponse object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiUsersResponse":{"type":"object","properties":{"content":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/UserContentDTO"}},"first":{"type":"boolean"},"last":{"type":"boolean"},"number":{"type":"integer","format":"int32"},"size":{"type":"integer","format":"int32"},"totalElements":{"type":"integer","format":"int32"},"totalPages":{"type":"integer","format":"int32"}},"required":["content","first","last","number","size","totalElements","totalPages"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]}}}} ``` ## The PublicApiVerifyContractRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"PublicApiVerifyContractRequestDTO":{"type":"object","properties":{"checkPaths":{"type":"array","items":{"type":"string"}}}}}}} ``` ## The ReferenceDiffDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ReferenceDiffDiagnosticsV4":{"type":"object","properties":{"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}}}}} ``` ## The ResponsibilityTypeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ResponsibilityTypeDTO":{"type":"string","enum":["user","userGroup"]}}}} ``` ## The RowCountDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"RowCountDiagnosticsV4":{"type":"object","properties":{"checkRowsTested":{"type":"integer","format":"int32"},"datasetRowsTested":{"type":"integer","format":"int32"}}}}}} ``` ## The RowCountDiffDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"RowCountDiffDiagnosticsV4":{"type":"object","properties":{"rowCountDiff":{"type":"integer","format":"int32"},"rowCountDiffPercent":{"type":"number"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}}}}} ``` ## The RowsDiffDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"RowsDiffDiagnosticsV4":{"type":"object","properties":{"changed":{"type":"integer","format":"int32"},"diffCount":{"type":"integer","format":"int32"},"diffPercent":{"type":"number"},"exclusiveInSource":{"type":"integer","format":"int32"},"exclusiveInTarget":{"type":"integer","format":"int32"},"sourceCheckRowsTested":{"type":"integer","format":"int32"},"sourceDatasetRowsTested":{"type":"integer","format":"int32"},"targetCheckRowsTested":{"type":"integer","format":"int32"},"targetDatasetRowsTested":{"type":"integer","format":"int32"}}}}}} ``` ## The ScanStateDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ScanStateDTO":{"type":"string","enum":["queuing","executing","started","cancelationRequested","timeOutRequested","canceled","timedOut","failed","completedWithErrors","completedWithFailures","completedWithWarnings","completed"]}}}} ``` ## The SchemaDiagnosticsV4 object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"SchemaDiagnosticsV4":{"type":"object","properties":{"actual":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}},"expected":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/SchemaDiagnosticsV4_Column"}}}},"SchemaDiagnosticsV4_Column":{"type":"object","properties":{"name":{"type":"string"},"type":{"type":"string"}}}}}} ``` ## The SchemaDiagnosticsV4\_Column object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"SchemaDiagnosticsV4_Column":{"type":"object","properties":{"name":{"type":"string"},"type":{"type":"string"}}}}}} ``` ## The SkeletonContractOperationStateDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"SkeletonContractOperationStateDTO":{"type":"string","enum":["ongoing","completed","failed","canceled"]}}}} ``` ## The ThresholdConditionDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]}}}} ``` ## The ThresholdStrategyDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]}}}} ``` ## The UnitOfTimeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"UnitOfTimeDTO":{"type":"string","enum":["hours","days","weeks"]}}}} ``` ## The UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO":{"type":"object","properties":{"computeWarehouseConfiguration":{"type":"object","$ref":"#/components/schemas/UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO_WarehouseConfigurationDTO"},"enabled":{"type":"boolean"}}},"UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO_WarehouseConfigurationDTO":{"type":"object","properties":{"name":{"type":"string"}},"required":["name"]}}}} ``` ## The UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO\_WarehouseConfigurationDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"UpdateDatasetComputeWarehouseOverrideConfigurationRequestDTO_WarehouseConfigurationDTO":{"type":"object","properties":{"name":{"type":"string"}},"required":["name"]}}}} ``` ## The UpdateDatasetDiagnosticsWarehouseConfigurationRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"UpdateDatasetDiagnosticsWarehouseConfigurationRequestDTO":{"type":"object","properties":{"failedRowsConfiguration":{"type":"object","$ref":"#/components/schemas/DatasetDiagnosticsWarehouseFailedRowsConfigurationRequest"},"scanAndResultsConfiguration":{"type":"object","$ref":"#/components/schemas/DatasetDiagnosticsWarehouseScanAndResultsConfigurationRequest"},"schema":{"type":"string"}}},"DatasetDiagnosticsWarehouseFailedRowsConfigurationRequest":{"type":"object","properties":{"enabled":{"type":"boolean"},"failedRowSamples":{"type":"object","$ref":"#/components/schemas/FailedRowSamplesDTO"},"failedRowsCollectionStrategy":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyDTO"},"maxRowCount":{"type":"integer","format":"int32"},"schema":{"type":"string","deprecated":true},"storageStrategy":{"type":"object","$ref":"#/components/schemas/DiagnosticsWarehouseFailedRowsStorageStrategyDTO"},"uniqueKeyColumnNames":{"type":"array","items":{"type":"string"}}}},"FailedRowSamplesDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"sampleSize":{"type":"integer","format":"int32"}}},"FailedRowsCollectionStrategyDTO":{"type":"object","properties":{"maxRowCountOverride":{"type":"integer","format":"int32"},"threshold":{"type":"number"},"thresholdCondition":{"type":"object","$ref":"#/components/schemas/ThresholdConditionDTO"},"type":{"type":"object","$ref":"#/components/schemas/FailedRowsCollectionStrategyTypeDTO"}},"required":["threshold","thresholdCondition"]},"ThresholdConditionDTO":{"type":"string","enum":["greaterThan","lessThan"]},"FailedRowsCollectionStrategyTypeDTO":{"type":"string","enum":["useDefaultMaxRowCount","absolute","percentage"]},"DiagnosticsWarehouseFailedRowsStorageStrategyDTO":{"type":"string","enum":["fullFailedRows","uniqueKeys","scansAndResultsOnly"]},"DatasetDiagnosticsWarehouseScanAndResultsConfigurationRequest":{"type":"object","properties":{"enabled":{"type":"boolean"}}}}}} ``` ## The UpdateDatasetMetricMonitoringConfigurationRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"UpdateDatasetMetricMonitoringConfigurationRequestDTO":{"type":"object","properties":{"datasetMetricMonitorsConfiguration":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/DatasetMetricMonitorConfigurationDTO"}},"enabled":{"type":"boolean"},"historicalMetricCollectionScanStartDate":{"type":"string","format":"date-time"},"scanSchedule":{"type":"object","$ref":"#/components/schemas/ZonedCronDTO"}}},"DatasetMetricMonitorConfigurationDTO":{"type":"object","properties":{"configuration":{"type":"object","$ref":"#/components/schemas/DatasetMonitorConfigurationDTO"},"metricType":{"type":"object","$ref":"#/components/schemas/DatasetMonitorMetricTypeDTO"}},"required":["configuration","metricType"]},"DatasetMonitorConfigurationDTO":{"type":"object","properties":{"exclusionZones":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/ExclusionZoneDTO"}},"isEnabled":{"type":"boolean"},"isSamplingEnabled":{"type":"boolean"},"sensitivity":{"type":"number"},"thresholdStrategy":{"type":"object","$ref":"#/components/schemas/ThresholdStrategyDTO"},"validRange":{"type":"object","$ref":"#/components/schemas/ValidRangeDTO"}}},"ExclusionZoneDTO":{"type":"object","properties":{"lowerLimit":{"type":"number"},"singleValue":{"type":"boolean"},"upperLimit":{"type":"number"}},"required":["lowerLimit","upperLimit"]},"ThresholdStrategyDTO":{"type":"object","properties":{"isLowerBoundEnabled":{"type":"boolean"},"isUpperBoundEnabled":{"type":"boolean"}},"required":["isLowerBoundEnabled","isUpperBoundEnabled"]},"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}},"DatasetMonitorMetricTypeDTO":{"type":"string","enum":["rowCount","freshness","schema","rowsInserted","totalRowCountChange","timeliness"]},"ZonedCronDTO":{"type":"object","properties":{"cronExpression":{"type":"string"},"timezone":{"type":"string"}},"required":["cronExpression","timezone"]}}}} ``` ## The UpdateDatasetProfilingConfigurationRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"UpdateDatasetProfilingConfigurationRequestDTO":{"type":"object","properties":{"enabled":{"type":"boolean"},"profilingSamplingStrategy":{"type":"object","$ref":"#/components/schemas/ProfilingSamplingStrategyConfigurationDTO"},"scanSchedule":{"type":"object","$ref":"#/components/schemas/ZonedCronDTO"}}},"ProfilingSamplingStrategyConfigurationDTO":{"type":"object","properties":{"numberOfRows":{"type":"integer","format":"int32"},"numberOfUnits":{"type":"integer","format":"int32"},"unitOfTime":{"type":"object","$ref":"#/components/schemas/UnitOfTimeDTO"}}},"UnitOfTimeDTO":{"type":"string","enum":["hours","days","weeks"]},"ZonedCronDTO":{"type":"object","properties":{"cronExpression":{"type":"string"},"timezone":{"type":"string"}},"required":["cronExpression","timezone"]}}}} ``` ## The UpdateDatasetTimePartitionRequestDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"UpdateDatasetTimePartitionRequestDTO":{"type":"object","properties":{"partitionColumn":{"type":"string"},"sqlExpression":{"type":"string"}}}}}} ``` ## The UserContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]}}}} ``` ## The UserGroupContentBasicDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"UserGroupContentBasicDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"}},"required":["name","userGroupId"]}}}} ``` ## The UserGroupContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"UserGroupContentDTO":{"type":"object","properties":{"name":{"type":"string"},"userGroupId":{"type":"string"},"users":{"type":"array","items":{"type":"object","$ref":"#/components/schemas/UserContentDTO"}}},"required":["name","userGroupId","users"]},"UserContentDTO":{"type":"object","properties":{"email":{"type":"string"},"firstName":{"type":"string"},"fullName":{"type":"string"},"lastName":{"type":"string"},"userId":{"type":"string"}},"required":["email","userId"]}}}} ``` ## The ValidRangeDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ValidRangeDTO":{"type":"object","properties":{"max":{"type":"number"},"min":{"type":"number"}}}}}} ``` ## The ValueCountContentDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ValueCountContentDTO":{"type":"object","properties":{"count":{"type":"number"},"value":{"type":"object","$ref":"#/components/schemas/_object"}}},"_object":{"type":"object"}}}} ``` ## The ZonedCronDTO object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"ZonedCronDTO":{"type":"object","properties":{"cronExpression":{"type":"string"},"timezone":{"type":"string"}},"required":["cronExpression","timezone"]}}}} ``` ## The \_object object ```json {"openapi":"3.1.0","info":{"title":"Soda Cloud API v4","version":"v1"},"components":{"schemas":{"_object":{"type":"object"}}}} ``` # Webhook API Webhook integrations allow your system to receive real-time notifications from Soda Cloud when certain events occur, such as check evaluations, incident changes, or contract updates. {% if !(visitor.claims.plan === %} {% hint style="info" %} Webhook integrations are available to organizations with a Soda license. [Contact us](mailto:support@soda.io) to learn more. {% endhint %} Soda Cloud can push real-time events to any endpoint you control, so the moment a check fails, an incident moves, or a contract is published, the systems your team already lives in find out about it. Webhooks turn Soda Cloud into a source of **structured**, **timely** **signal** for the rest of your operational stack. Common patterns include: * **Pipe alerts into your on-call rotation**, so failed checks open pages or tickets in PagerDuty, Opsgenie, or whatever tool your team is on. * **Drive automation:** kick off remediation jobs, pause downstream pipelines, or trigger reruns when specific checks fail. * **Sync data quality events into your own observability stack**, alongside your existing logs and metrics. * **Keep external systems of record in lockstep** with Soda incidents and contract changes, so your catalog, ticketing tool, or governance platform reflects the latest state. {% endif %} *** {% if visitor.claims.plan === %} ## Setup To create a Webhook integration, you must have **Organization Settings** permissions. #### Add Integration {% stepper %} {% step %} Go to **Organization Settings > Integrations** {% endstep %} {% step %} Click **Add Integration** {% endstep %} {% step %} Choose **Webhook** as the integration type
{% endstep %} {% endstepper %} ### Configure Webhook Provide the required fields: * **Name** – The name of your integration * **Webhook URL** – The endpoint to send the events to * **Secret (optional)** – Used to sign the payload
### Event Testing After creation, you can simulate payloads for development and testing. This allows you to preview example payloads for each event type using random values.
{% endif %} ### Scopes Webhooks can be configured to listen for one or more **event scopes**: | Scope | Description | | ----------------------- | --------------------------------------------------------------------------- | | **Alert Notifications** | Triggered when a check result causes an alert (based on notification rules) | | **Incidents** | Triggered when incidents are created or updated in Soda Cloud | | **Contracts** | Triggered when a data contract is published (created or updated) | {% if visitor.claims.plan === %} Alert Notifications require active **notification rules**. Learn more about [Broken mention](broken://pages/GAK0Cdoq2hZmucVmrovV)
{% endif %} *** {% if !(visitor.claims.plan === %} ## Event Types {% endif %} {% if visitor.claims.plan === %} ## Event Types and Payloads {% endif %} Depending on the selected scope, one or more of the following event types will be sent to your endpoint: ### Check Evaluation Sent when a check evaluation triggers an alert. It carries the check's outcome and a link back to the result in Soda Cloud so a human can investigate further. {% if visitor.claims.plan === %} Scope: `alert_notification`\ Triggered when a check result triggers an alert. Alert Notifications require active **notification rules to use the webhook as a recipient**. **Payload structure:** ```bash { "event": "checkEvaluation", "checkResults": [CheckResult, ...] } ``` #### CheckResult | Field | Type | Description | | -------------------------- | ------- | ------------------------------------------------------------------------------------------------------ | | `id` | string | Unique ID of the check result | | `sodaCloudUrl` | string | Link to the check result in Soda Cloud | | `definition` | string | YAML check definition | | `datasets` | array | List of associated datasets (see [#datasets](#datasets "mention") object) | | `column` | string | List of associated datasets (see [#attributes](#attributes "mention") object) | | `attributes` | array | List of associated datasets (see `Dataset` object) | | `value` | numeric | The field shows the actual value calculated for the check (the result used to determine pass or fail). | | `evaluationStatus` | string | `pass`, `fail`, or `notEvaluated` `excluded` | | `outcome` (deprecated) | string | Deprecated, replaced by `evaluationStatus`. | | `dataTimestamp` | string | ISO 8601 timestamp of data | | `diagnostics` (deprecated) | object | Deprecated, replaced by `value`. | | `agreement`(deprecated) | object | Deprecated, V3 concept. | | `name` | string | Name of the check | #### Datasets | Field | Type | Description | | ------------ | ------ | ---------------------------------------------------------------------------- | | id | string | Unique ID of the dataset | | sodaCloudUrl | string | Link to the dataset in Soda Cloud | | name | string | Dataset name (technical name) | | label | string | Display label for the dataset | | tags | array | List of dataset tags | | owners | array | List of dataset owners (see [#owner](#owner "mention")object) | | datasource | object | Datasource information (see [#datasource](#datasource "mention")object) | | attributes | array | List of dataset attributes (see [#attributes](#attributes "mention") object) | #### Owner | Field | Type | Description | | ----- | ------ | ------------------------------- | | type | string | Owner type (for example `user`) | | id | string | Unique ID of the owner | | email | string | Owner email address | #### Datasource | Field | Type | Description | | ----- | ------ | -------------------------------- | | id | string | Unique ID of the datasource | | name | string | Datasource name (technical name) | | label | string | Display label for the datasource | #### Attributes | Field | Type | Description | | ----------- | ------ | ------------------------------- | | id | string | Unique ID of the attribute | | name | string | Attribute name (technical key) | | label | string | Display label for the attribute | | value | string | Attribute value | | {% endif %} | | | *** ### **Incident Created** **S**ent when a new incident is opened, with enough context for a downstream system to create a corresponding ticket, channel, or page. {% if visitor.claims.plan === %} Scope: `incidents`\ Triggered when a new incident is reported ```bash { "event": "incidentCreated", "incident": Incident } ``` #### Incident | Field | Type | Description | | ------------------------- | ------ | ---------------------------------------------------------------------------------------- | | `id` | string | Incident ID | | `sodaCloudUrl` | string | URL to view the incident | | `number` | number | Human-readable incident number | | `title` | string | Incident title | | `description` | string | Incident description | | `severity` | string | `low`, `medium`, `high`, `critical` | | `status` | string | `reported`, `in_progress`, `resolved`, etc. | | `created` | string | ISO timestamp | | `lastUpdated` | string | ISO timestamp | | `reporter` | object | User who reported the incident (see [#reporter](#reporter "mention") object) | | `checkResults` | array | List of related `CheckResult` objects (see [#checkresult](#checkresult "mention")object) | | `incidentLinkCallbackUrl` | string | Link to update or interact with the incident (optional) | #### **Reporter** | Field | Type | Description | | ----- | ------ | ------------------------- | | id | string | Unique ID of the reporter | | email | string | Reporter email address | #### **IncidentLinkCallBackUrl** This property allows you to add links to incidents that can point to external integrations. To add a link, do a POST call with the following body: ```bash { "url": string, "text": string } ``` {% endif %} *** ### Incident Updated Sent when an existing incident changes, so downstream systems can stay in sync as severity, status, or scope evolves. {% if visitor.claims.plan === %} Same structure as `incidentCreated`, but triggered when an incident is updated. ```bash { "event": "incidentUpdated", "incident": Incident } ``` ### Incident | Field | Type | Description | | ----- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | ... | ... | Uses the same structure as the Incident created event. See [#incident](#incident "mention") | | links | array | Links to integration resources set using [#incidentlinkcallbackurl](#incidentlinkcallbackurl "mention") (see [#links](#links "mention") object). | #### Links | Field | Type | Description | | --------------- | ------ | ----------------------------------------- | | integrationType | string | Integration type (for example `slack`) | | url | string | Link to the external integration resource | | {% endif %} | | | *** ### Contract Published Sent when a data contract is published or updated, useful for catalog sync, change auditing, or kicking off downstream validation. {% if visitor.claims.plan === %} Scope: `contracts`\ Triggered when a contract is published or updated. ```bash { "event": "contractPublished", "contract": Contract } ``` #### `Contract` **object** | Field | Type | Description | | -------------- | --------- | ------------------------------------------------------------- | | `dataset` | string | Fully qualified name of the dataset the contract applies to | | `prefixes` | string\[] | *(Deprecated)* – Use `dataset` instead to identify the target | | `datasource` | string | Name of the data source | | `author` | object | Information about the user who published the contract | | `author.id` | string | Soda Cloud user ID | | `author.email` | string | Email of the user who published the contract | | `contents` | string | The full YAML content of the contract as a string | | {% endif %} | | | *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Soda Runner basic concepts Establish a baseline understanding of the concepts involved in deploying a Soda Runner. {% hint style="danger" %} **Nomenclature change: "Agent" is now "Runner"** The Soda Agent has been renamed to [**Soda Runner**](/reference/soda-runner-basic-concepts) across all Soda products and interfaces. This change affects terminology throughout the platform, including: * Soda Cloud UI labels * API permission names: `MANAGE_DATASOURCES_AND_AGENTS` → `MANAGE_DATASOURCES_AND_RUNNERS` * CLI flags: `--use-agent` → `--use-runner` * Python API methods: `verify_contract_on_agent` → `verify_contract_on_runner` The underlying functionality is unchanged, and **both runner/agent nomenclatures are backward compatible** at present. If you are following older documentation or have existing scripts and automation that reference "agent", we recommend to gradually replace every occurrence of "agent" / "Agent" with "runner" / "Runner". {% endhint %} The **Soda Runner** is a tool that empowers Soda Cloud users to securely access data sources to scan for data quality. For a self-hosted runner, create a Kubernetes cluster in a cloud services provider environment, then use Helm to deploy a Soda Runner in the cluster. This setup enables Soda Cloud users to securely connect to data sources (Snowflake, Amazon Athena, etc.) from within the Soda Cloud web application. Any user in your Soda Cloud account can add a new data source via the runner, then write their own no-code checks to check for data quality in the new data source. #### Platform architecture

Soda platform architecture diagram

\*All flows start with instruction retrieval and a Kubernetes Job launch. ### What is Soda Library? **Soda Library** is a **Python library** and command-line tool that serves as the backbone of Soda technology. It is the software that performs the work of converting user-defined input into SQL queries that execute when you run scans for data quality in a data source. Connect Soda Library to a **Soda Cloud** account where you and your team can use the web application to collaborate on data quality monitoring. ### What is Soda Cloud? **Soda Cloud** is Soda's **web application**. It provides a friendly, intuitive and low-barrier interface that can be used by technical and non-technical users alike. Soda Cloud allows for collaboration between teams and for different permission levels depending on your organization's structure. In the web application, for example, non-technical users can propose new checks and contract changes, while engineering teams can review the suggestions and approve or modify them. ### What are data contracts? **Data contracts** are **formal agreements** between your organization's teams (usually between data producers and data consumers), and **they specify the expected shape, quality, and behavior of a dataset**. A data contract sets clear rules for what the data must look like and how it should perform, so that producers and consumers of data share a common understanding. Both **Soda Library** and **Soda Cloud** make use of data contracts written in **YAML language**. Contracts include **checks** for data quality. The checks are tests that Soda Library executes when it runs a scan of your data. > Go to [Contract Language reference](/reference/contract-language-reference) to learn more about contract structure and requirements. ### What is Soda Runner? **Soda Runner** is essentially Soda Library functionality that you deploy in a Kubernetes cluster in your own cloud services provider environment. When you deploy a runner, you also deploy two types of workloads in your Kubernetes cluster from a Docker image: * a **Soda Runner Orchestrator** which creates Kubernetes Jobs to trigger scheduled and on-demand scans of data * a **Soda Runner Scan Launcher** which wraps around Soda Library, the tool which performs the scan itself ### How does Soda integrate with Kubernetes? **Kubernetes** is a system for orchestrating containerized applications; a **Kubernetes cluster** is a set of resources that supports an application deployment. You need a Kubernetes cluster in which to deploy the containerized applications that make up the **Soda Runner**. Kubernetes uses the concept of [**Secrets** ](https://kubernetes.io/docs/concepts/configuration/secret/)that the Soda Runner Helm chart employs to store connection secrets that you specify as values during the Helm release of the Soda Runner. Depending on your cloud provider, you can arrange to store these Secrets in a specialized storage such as [Azure Key Vault](https://learn.microsoft.com/en-us/azure/key-vault/general/basic-concepts) or [AWS Key Management Service (KMS)](https://docs.aws.amazon.com/kms/latest/developerguide/overview.html). > Learn more about [using external secrets](broken://pages/0Cs2lWTZr1mIVYrDs9Y0#using-existing-external-secrets). The Jobs that the runner creates access these Secrets when they execute. > Learn more about [Kubernetes concepts](https://www.youtube.com/watch?v=BOj1sgWVXko). ### Where can a Soda Runner be deployed? Within a cloud services provider environment is *where* you create your Kubernetes cluster. You can deploy a Soda Runner in **any environment in which you can create Kubernetes clusters**, such as: * Amazon Elastic Kubernetes Service (EKS) * Microsoft Azure Kubernetes Service (AKS) * Google Kubernetes Engine (GKE) * Any Kubernetes cluster version 1.21 or greater which uses standard Kubernetes * Locally, for testing purposes, using tools like [Minikube](https://minikube.sigs.k8s.io/docs/), [microk8s](https://microk8s.io/docs), [kind](https://kind.sigs.k8s.io/), [k3s](https://docs.k3s.io/), or [Docker Desktop](https://www.docker.com/products/docker-desktop/) with Kubernetes support. ### What is Helm? **Helm** is a **package manager for Kubernetes** which bundles YAML files together for storage in a public or private repository. This bundle of YAML files is referred to as a **Helm chart**. The Soda Runner is a Helm chart. Anyone with access to the Helm chart’s repo can deploy the chart to make use of YAML files in it. > Learn more about [Helm concepts](https://www.youtube.com/watch?v=-ykwb1d0DXU). The Soda Runner Helm chart is stored on a [public repository](https://helm.soda.io/soda-agent/) and published on [ArtifactHub.io](https://artifacthub.io/packages/helm/soda-agent/soda-agent). Anyone can use Helm to find and deploy the Soda Runner Helm chart in their Kubernetes cluster. ### Why Kubernetes? Kubernetes is the most **powerful** and **future-proof** platform for running the Soda Runner because it delivers the best of both worlds: the **flexibility of raw compute** without the operational burden, and the **scalability of managed services** without their restrictions. * Kubernetes goes far beyond raw compute like EC2 or traditional Virtual Machines (VMs) by **abstracting away the heavy lifting of networking, deployments, and scaling**, while still giving teams precise control when needed. Practically, this makes it easy for Soda’s customers to [deploy, manage, and upgrade Soda Runners](broken://pages/DVzqeL43wc3c9AsZ4TFi) using [Kubernetes](#how-does-soda-integrate-with-kubernetes) and [Helm](#what-is-helm), always staying up to date with the latest releases. * Unlike fully managed options such as AWS Lambda, **Kubernetes has no execution time limits** and is built to handle long-running, stateful, and highly scalable workloads. This means **Soda is not limited to lightweight samples** but can perform complete, row-level operations—powering advanced capabilities like Diagnostics Warehouse, which securely stores the exact failing records inside your own infrastructure, and [Reconciliation Checks](broken://pages/QkO7w20yeaFP3U9Ow5Kk), which compare data at row-level across sources. Whether running in the cloud or on-premises, Kubernetes ensures resilience, portability, and cost-efficient resource use, making it the clear choice for complex, enterprise-grade data quality workloads. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Migrate from v3 to v4 This guide details the steps required to migrate datasets from **v3** to **v4 contracts.** The migration needs to be enabled by the Soda team. Contact us to ensure that the feature flag (`v3ToV4DatasetMigrationEnabled`) is enabled for your organization. {% hint style="warning" %} This action requires the global **Admin** role and can be [enabled for users with **Manage contract** permission](#enable-dataset-owners-to-migrate-datasets-from-v3-v4). Learn more about permissions here: [Broken mention](broken://pages/n914KXqJNBoZ19Ay5W0P) {% endhint %} ## Using Soda Cloud ### Start the migration The migration can be performed in two ways: 1. **Bulk migration:** On the datasets page**,** you can select multiple datasets to migrate them all at once. 2. **Single migration:** On a dataset page, you can migrate the dataset individually.
Once initiated, Soda guides you through each step of the process to ensure a safe and transparent migration. ### Select a v4 data source In the first step, you’ll select which **v4 data source** the dataset(s) should be migrated to. 1. Choose an existing **v4 data source** from the dropdown list. 2. Or click **Add a new v4 Data Source** to create a new one. 3. Click **Next** {% hint style="warning" %} You can only migrate v3 datasets to a **data source of the same type**. For example, from Snowflake to Snowflake. The migration process in Soda Cloud requires a **valid connection** configuration to your data source **through Soda Agent**. {% endhint %}
### Configure migration settings In this step, you can customize optional settings before running the migration. **Available configurations:** * **Contract Schedule:** Set a schedule for automatic contract verification. [Broken mention](broken://pages/adn2iQZu7t1aWGfkGzSJ) * **Migrate History:** include 90 days of historical check results in the migration. * **Migrate Responsibilities:** Migrate the dataset's ownership and assigned responsibilities. * **Migrate Attributes:** Migrate the dataset's attributes. Once your settings are configured, click **Continue** to proceed.
### Review migration Before migrating, you’ll see a detailed summary of what will be migrated and what won’t.\ Each dataset displays: * **Number of checks** that were: * **Successfully translated:** The behavior of the new checks matches that in v3. * **Translated with a warning:** Translated, but not an exact one-to-one match. Review recommended. * **Not translated:** Checks are either not supported in v4 or cannot be translated. See [#current-limitations](#current-limitations "mention") * For each check with a warning or that was not translated, you can view the **definition of the v3 check,** along with an **indicative remark** explaining the reason or impact.
Click the **eye icon** to open a preview of the generated contract before finalizing the migration.It lets you verify the contract structure, filters, variables, and checks. You can test it directly from this view to ensure everything runs as expected before confirming migration.
### Complete migration After reviewing, use the checkboxes to deselect any datasets that are not ready for migration, then click **Migrate** to finalize the process. This action starts a **background process** to migrate the selected datasets. The migration may take a few seconds or minutes, depending on the volume being migrated, to complete.
Once started, you can navigate to the **v3 datasets** to monitor progress.\ Refresh the page after a few moments to view the migration results.
### Post-migration results Once migration is complete: * Your original **v3 dataset** remains accessible but marked as *migrated*. * Your new **v4 dataset** is fully active and ready for contract-based validation. The v3 dataset page shows its migration status as **Completed migration**. In the v4 dataset view, you can now see: * Migrated checks under **Dataset checks** and **Column checks**. * Contract verification results and coverage metrics. * The option to edit or version your new data contract.
## Using Soda CLI ### Install the migration CLI In a virtual environment, install the Soda migration package as well as the Soda Core package for your data source (see [Data source reference for Soda Core](/reference/data-source-reference-for-soda-core)). {% hint style="warning" %} The migration process requires having installed **the `soda-migration` package using the** [**private PyPI**](#private-pypi-installation-flow) with a **Team** or **Enterprise license.** Need access to the PyPI repository ? Please [contact us](https://www.soda.io/contact). {% endhint %} Choose your organization host to install the migration package: {% tabs %} {% tab title="Team EU" %} {% code overflow="wrap" %} ```bash pip install soda-migration -i "https://:@team.pypi.cloud.soda.io" ``` {% endcode %} {% endtab %} {% tab title="Team US" %} {% code overflow="wrap" %} ```bash pip install soda-migration -i "https://:@team.pypi.cloud.us.soda.io" ``` {% endcode %} {% endtab %} {% tab title="Enterprise EU" %} {% code overflow="wrap" %} ```bash pip install soda-migration -i "https://:@enterprise.pypi.cloud.soda.io" ``` {% endcode %} {% endtab %} {% tab title="Enterprise US" %} {% code overflow="wrap" %} ```bash pip install soda-migration -i "https://:@enterprise.pypi.cloud.us.soda.io" ``` {% endcode %} {% endtab %} {% endtabs %} Next to the migration package, **it is also required to install the necessary package to connect to your data source**. See [Data source reference for Soda Core](/reference/data-source-reference-for-soda-core). This is required because Soda connects to your data source to generate the contract skeleton before translating existing checks. ### Configuration Create a `migration.json` file to map v3 dataset IDs to v4 DQNs (Dataset Qualified Names). This file defines which datasets to migrate and their corresponding new DQNs.\ \ Example structure: ```json [ { "dqn": "v4_data_source/postgres/public/dim_employee", "v3DatasetId": "6d706754-4d0d-4aa0-9c7d-25fbe07cb3f6" }, { "dqn": "v4_data_source/postgres/public/dim_employee", "v3DatasetId": "6d706754-4d0d-4aa0-9c7d-25fbe07cb3f6" } ] ``` > Learn more about DQNs in [Contract Language reference](/reference/contract-language-reference#dataset-fully-qualified-name) #### Retrieve the v3 dataset IDs To retrieve the v3 dataset IDs, you want to migrate from Soda Cloud: * Use [Broken mention](broken://pages/35f6a3d29f47259ce552d7bad6bb2b5f7477c480#get-api-v1-datasets) to fetch the datasets' information and their IDs * The IDs can also be found in the dataset URL for a given dataset.
Example Python script to generate the configuration file ```python #!/usr/bin/env python3 """ Soda Cloud Dataset Migration Configuration Generator This script fetches dataset information from Soda Cloud API and generates a migration.json file for dataset migration. This file is used to migrate datasets from V3 to V4. """ import argparse import base64 import json import os import subprocess import sys from dotenv import load_dotenv def generate_basic_auth_token(api_key, api_secret): """Generate Basic Auth token from API key and secret.""" token = f"{api_key}:{api_secret}" return base64.b64encode(token.encode()).decode() def execute_curl_with_auth(url, auth_token): """Execute curl command with Basic Auth and return JSON response.""" try: headers = f"Authorization: Basic {auth_token}" command = ['curl', '-s', url, '--header', headers] result = subprocess.run(command, capture_output=True, text=True, check=True) return json.loads(result.stdout) except subprocess.CalledProcessError as e: print("Curl failed with error:\n", e.stderr) return None except json.JSONDecodeError as e: print(f"Failed to parse JSON response: {e}") return None def fetch_datasets(api_key, api_secret, v3_data_source_name=None): """Fetch datasets from Soda Cloud API.""" url = "https://cloud.soda.io/api/v1/datasets" params = [] if v3_data_source_name: params.append(f"datasourceName={v3_data_source_name}") params.append("size=1000") if params: url += "?" + "&".join(params) auth_token = generate_basic_auth_token(api_key, api_secret) print(f"Fetching datasets from {url}...") response = execute_curl_with_auth(url, auth_token) if response is None: print("Failed to fetch datasets from API") return None if "content" not in response: print("Unexpected API response format") return None return response["content"] def generate_migration_data(datasets, v4_data_source_name): """Generate migration data from datasets.""" migration_data = [] for dataset in datasets: migration_entry = { "v3DatasetId": dataset["id"], "dqn": v4_data_source_name + "/" + dataset["datasource"]["prefix"].replace(".", "/") + "/" + dataset["name"] } migration_data.append(migration_entry) return migration_data def save_migration_file(migration_data, output_path): """Save migration data to JSON file.""" try: with open(output_path, 'w') as f: json.dump(migration_data, f, indent=2) print(f"Migration file saved to: {output_path}") return True except Exception as e: print(f"Failed to save migration file: {e}") return False def main(): """Main application entry point.""" parser = argparse.ArgumentParser( description="Generate Soda Cloud dataset migration file" ) parser.add_argument( "v4_data_source_name", help="V4 data source name for migration" ) parser.add_argument( "v3_data_source_name", help="V3 data source name to filter datasets (used as query parameter)" ) parser.add_argument( "--output", "-o", default="migration.json", help="Output file path (default: migration.json)" ) parser.add_argument( "--env-file", default="../.env", help="Environment file path (default: ../.env)" ) args = parser.parse_args() # Load environment variables load_dotenv(args.env_file) # Get API credentials from environment api_key = os.getenv("SODA_CLOUD_API_KEY") api_secret = os.getenv("SODA_CLOUD_API_SECRET") if not api_key or not api_secret: print("Error: SODA_CLOUD_API_KEY and SODA_CLOUD_API_SECRET must be set in environment") print("You can set them in a .env file or as environment variables") sys.exit(1) print(f"Generating migration for data source: {args.v4_data_source_name}") print(f"Filtering datasets by V3 data source: {args.v3_data_source_name}") # Fetch datasets from API datasets = fetch_datasets(api_key, api_secret, args.v3_data_source_name) if datasets is None: sys.exit(1) print(f"Found {len(datasets)} datasets") # Generate migration data migration_data = generate_migration_data(datasets, args.v4_data_source_name) # Save to file if save_migration_file(migration_data, args.output): print("Migration generation completed successfully!") else: sys.exit(1) if __name__ == "__main__": main() ```
*** ### Generate contracts Run the following command, replacing the paths and file names with your setup. Note that this action requires a valid connection configuration to your data source and Soda Cloud. ```sh soda contract-migrator generate-bulk \ --bulk-config-file migration.json \ --output-directory contracts/ \ --v4-data-source ds.yml \ --soda-cloud soda-cloud.yml \ --schedule "0 0 * * *" \ -v ``` #### Parameters | **Parameter** | **Required** | **Description** | | -------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--bulk-config-file` | Yes | Path to the JSON configuration file (`bulk.json`) that maps v3 dataset IDs to v4 DQNs. | | `--output-directory` | Yes | Directory where the generated v4 contracts will be saved. The structure within this location mirrors the DQN hierarchy. | | `--v4-data-source` | Yes |

Path to the local v4 datasource YAML file (e.g., ds.yml). Used to retrieve schema metadata during contract generation.

See /pages/Ovp9NbJpqBhQz07AImSa

| | `--soda-cloud` | Yes | Configuration required to connect to Soda Cloud. | | `--schedule` | No | Optional cron expression defining the schedule for the generated contracts (e.g., `"0 0 * * *"`). | | `--verbose` | No | Enables verbose output for detailed logs during the migration process. | ### Review migration Ensure that each generated contract includes: * Correct `v4_dqn` in the `dataset` property * Correct `v3_dataset_id` property referencing v3 dataset * Correct **columns and types** * Checks migrated from `sodacl` into `contracts` checks * v3 check IDs present in the `qualifier` field for each check * Accurate **check filters**, expressions, and metadata See [#current-limitation](#current-limitation "mention") to know which checks cannot be automatically migrated yet. Those checks can still be added manually, and the history can be migrated by setting a v3 check ID in the `qualifier` ### Complete migration Once contracts are verified, publish them to Soda Cloud with the following command: ```sh soda contract-migrator publish \ --contract contracts/ \ --soda-cloud soda-cloud.yml \ --migrate-attributes \ --migrate-responsibilities \ --migrate-test-results ``` #### Parameters | Parameter | Required | | | ---------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--contract` | Yes |

Specifies the path to the folder containing the contracts or a specific contract file to be migrated.

This parameter supports recursive directory traversal.


⚠️ Note: If the folder includes contracts for datasets that have already been migrated, those datasets will be migrated again.

| | --`soda-cloud` | Yes | Configuration required to connect to Soda Cloud. | | --`migrate-test-results` | No |

Include test history from v3 to v4. A maximum of 90 days of history is migrated.

Default is false.

| | --`migrate-responsibilities` | No |

Copy responsibilities from v3 datasets to v4 datasets.

Default is false.

| | --`migrate-attributes` | No |

Copy attributes from v3 datasets to v4 datasets

Default is false.

| | --`verbose` | No | Enable verbose mode | ### Post-migration results After publishing, confirm the following: * **Logs:** Check migration status (overall and per dataset). * **UI Overview:** Verify datasets in the Soda Cloud UI. * **Checks:** Ensure checks appear with full history. * **Flags:** Confirm presence of the following indicators: * **v4 flag** present * **v3 link** available * **Migration completed** flag present
## Notes and recommendations * Migration does not delete v3 datasets; it simply marks them as migrated. Once migration is completed, you will be required to update your pipelines or agreements to stop executing v3 checks. Then you can remove the v3 datasets or the v3 data source to remove all its datasets. * Verify that your v4 data source has a valid connection configuration before migrating. Soda connects to your data source to generate the contract structure. * Review the migrated contract in detail before finishing the migration. However, it is possible to re-run the migration if necessary [#re-running-a-migration](#re-running-a-migration "mention") ## Enable dataset owners to migrate datasets from v3→v4 An organization **Admin** can enable users with **Manage Contract** permission to migrate datasets. {% hint style="info" %} **Dataset owners** have the **Manage Contract** permission by default. {% endhint %} {% stepper %} {% step %} Click on your profile and navigate to **Organization Settings**
{% endstep %} {% step %} Under **Dataset migration**, check the option "Allow users with Manage Contract permission to migrate datasets" This option is disabled by default.
{% endstep %} {% endstepper %} Users with permission will now be able to see the migration tool:
*** ## View and filter migrated datasets After migration, you can review and manage your datasets from the **Datasets** page in Soda Cloud. Use the filters to easily identify datasets based on their **migration status** and **version**: * **Migration status filter** — view datasets that are *Pending*, *In progress*, or *Completed migration*. * **Version filter** — filter by dataset version (**v3** or **v4**) to focus on datasets still awaiting migration or already upgraded. This makes it simple to track migration progress, validate completed transitions, and identify datasets that still require attention.
## Re-running a migration Once a dataset has been successfully migrated to version 4 (v4), Soda blocks the migration for the dataset to happen again. To re-run the migration, you will need to delete the v4 dataset and run the migration again. ## How checks are matched The migration process uses the **qualifier** ( [Contract Language reference](/reference/contract-language-reference#check-qualifiers)) field to identify which v3 checks should be migrated. The qualifier values are set to the v3 check IDs.\ \ Because the qualifier is part of the check's identity algorithm, it is important **not to change the qualifier** after migration.\ \ Changing it would result in a **loss of history for the checks** in Soda Cloud. ## Current limitations ### Translation step #### **Check types** The following check types are not yet supported: * [Group By check](/reference/contract-language-reference#group-by-check) * Any [reconciliation](broken://pages/QkO7w20yeaFP3U9Ow5Kk) checks * Reference check * Any checks using [anomaly score](/soda-v3/sodacl-reference/anomaly-score) or [anomaly detection](/soda-v3/sodacl-reference/anomaly-detection) #### **Dataset filters** Dataset filters ([/spaces/oV0A6Eua8LUIyWgHxsjf/pages/yBqe6yM8bkyesdvcy0Pd#in-check-vs.-dataset-filters](https://docs.soda.io/reference/spaces/oV0A6Eua8LUIyWgHxsjf/pages/yBqe6yM8bkyesdvcy0Pd#in-check-vs.-dataset-filters "mention") ) are currently not migrated. #### Column casting Data Contract does not support casting yet.\ When casting is detected in a check, the check will not be translated and will be excluded from the migration. #### **Variables** **Variables in names** If variables are used in the column name, the check will not be translated and will be excluded from the migration. Example**:** ```yaml checks for ECOMMERCE_ORDERS [test]: - missing_count(${ORDER_ID_COL}) = 0: name: Must not have null values ``` **Variables default values** Variables used in SodaCL are automatically added to the data contract. They will not have a default value. The default values can be added manually by the users. *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Documentation access & licensing Learn how Soda features are documented and how to gain full access to the documentation. Soda’s documentation is designed to be accessible and relevant to all users; from those exploring Soda for free, to large-scale enterprise customers. Documentation visibility depends on your **license plan** and your **login status**. {% if visitor.claims.plan === %} > See [Broken mention](broken://pages/nEeVAUUx8V57By8ALc2b) for more information about how licenses work. > {% endif %} **Soda maintains two types of documentation:** ### Public documentation Public documentation is accessible to **anyone**, without requiring a Soda account.\ It covers general topics such as: * getting started guides; * feature overviews and descriptions; * Soda Cloud and Soda Runner setup; * common configurations and syntax references; * open-source integrations and community features; * and more. **Public documentation can be accessed at any time, without login.** *** ### Private documentation Private documentation is **conditional**, meaning access depends on: * Your **Soda license type**, and * Whether you are **logged in to Soda** in your browser. Private documentation includes advanced or licensed-only features such as: * comprehensive description of features, such as architecture and integration details; * curated documentation unique to your organization; * custom data quality policies. If your plan includes a feature, you **automatically gain access to its related documentation.** *** ## How access works Access to private documentation is handled **securely and automatically** using **signed cookies** from your Soda session. 1. Log in to [**Soda Cloud**](https://cloud.soda.io/) **using the same browser** where you plan to read the documentation. 2. [Soda Docs](https://docs.soda.io/) reads your **login token (JWT)** from the browser cookies. 3. The documentation content is unlocked automatically according to your license permissions. **No extra steps are needed.** As long as you’re **logged in on the same browser**, Soda Docs will recognize your plan and grant access accordingly. ### Troubleshooting If you can’t access documentation you expect to see: * Make sure you are **logged in to Soda Cloud** on the same browser. * Refresh the page or clear cookies if access doesn’t update. * Confirm your **license type** under your Soda account settings. * Confirm which version of the documentation you are seeing at the **bottom of any page**. * [**Contact** **Soda**](https://www.soda.io/contact) if your organization has access to private features but can't see the corresponding documentation. {% hint style="info" %} **Access is dynamic:** as your organization’s license changes, your documentation access updates automatically. {% endhint %} *** {% if (visitor.claims.plan === 'datasetStandard') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterprise') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if (visitor.claims.plan === 'enterpriseUserBased') %} {% hint style="success" %} You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). {% endhint %} {% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %} {% hint style="info" %} You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing). If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**. {% endhint %} # Soda v3 documentation Soda is a data quality platform that provides tools to monitor, test, and improve data quality across all stacks. Welcome to the Soda documentation hub, your one-stop resource for everything you need to know about **Soda’s data quality platform**. Dive into our guides, tutorials, reference materials, and integration pages to learn how keep your data quality fresh across your entire stack. Get started! #### Soda v3 vs v4 {% hint style="success" %} This is the documentation for **Soda v3**. If you are using Soda v4 or want to learn more about the next iteration of Soda, head to the [v4 documentation](https://docs.soda.io/). {% endhint %} **Soda v3** is a checks-based, CLI-driven data quality tool. **Soda v4** has incorporated collaborative data contracts and end-to-end observability features to become a unified data-quality platform for all.
CapabilitySoda v3Soda v4

Data Testing

(Checks)

CLI-centric checks written in YAML/SodaCL, run via the Python library or Runner.

Still supports YAML/SodaCL checks.

Adds full Data Testing workflows in both CLI and Web UI.

Data Observability

(Monitoring)

Anomaly dashboards provide threshold-based monitoring configured via Soda Cloud.Metric Monitoring leverages an in-house anomaly detection algorithm to monitor data and metadata metric trends, and provide built-in alerts via Soda Cloud.
Data ContractsFile-based contracts executed via CLI/Git; verification via soda scan.Collaborative data contracts: file-based and UI-based, executed via CLI/Git or the Soda Cloud UI.
*** ### 📚 Guides & Tutorials Learn core concepts and best practices: * Use case guides: Practical Soda usage scenarios * Write checks: Define data quality checks * Run scans: Execute Soda data scans * Organize, alert, investigate: Check results and investigate issues ### 🔌 Integrations Extend Soda into your existing tools and workflows: Integrations ### 📖 Reference Detailed command, API, and configuration docs: Reference *** ### 💬 Community & Support Need help or want to contribute? * **Join our Slack Community**: [Soda Community](https://join.slack.com/t/soda-community/shared_invite/zt-3epazj3kw-00z15nnW4KEt4j_vk8lbdQ) * **Browse GitHub Discussions**: [Soda on GitHub](https://github.com/sodadata/soda-core) *** > **Still have questions?** Use the search bar above or reach out through our community channels for additional help. # Get started Follow this tutorial to set up and run a simple Soda scan for data quality using example data. {% hint style="warning" %} The Soda environment has been updated since this tutorial. > Refer to [v4 documentation](/quickstart) for updated tutorials. > {% endhint %} Is Soda the data quality testing solution you've been looking for? Take a sip and see! 🫧 Use the example data in this quick tutorial to set up and run a simple Soda scan for data quality.
[Set up Soda](#set-up-soda) | 3 minutes\ [Build an example data source](#build-an-example-data-source) | 2 minutes\ [Connect Soda](#connect-soda) | 5 minutes\ [Write some checks and run a scan](#write-some-checks-and-run-a-scan) | 5 minutes\
> 💡 For standard set up instructions, access the [Get started roadmap](/soda-v3/quick-start-sip/get-started-roadmap). > > ✨ Want a total UI experience? Use the out-of-the-box [Soda-hosted agent](/soda-v3/quick-start-sip/managed-agent) to skip the CLI. ## Set up Soda This tutorial references a MacOS environment. 1. Check the following prerequisites: * You have installed Python 3.8, 3.9, or 3.10. * You have installed Pip 21.0 or greater. * (Optional) You have installed [Docker Desktop](https://www.docker.com/products/docker-desktop/) and have access to [GitHub](https://github.com/), to set up an example data source. 2. Contact us at to get an account set up. 3. In your command-line interface, create a Soda project directory in your local environment, then navigate to the directory. ```shell mkdir soda_sip cd soda_sip ``` 3. Best practice dictates that you install the Soda using a virtual environment. In your command-line interface, create a virtual environment in the `.venv` directory, then activate the environment. ```shell python3 -m venv .venv source .venv/bin/activate ``` 3. Execute the following command to install the Soda package for PostgreSQL in your virtual environment. The example data is in a PostgreSQL data source, but there are 15+ data sources with which you can connect your own data beyond this tutorial. ```shell pip install -i https://pypi.cloud.soda.io soda-postgres ``` 4. Validate the installation. ```shell soda --help ``` To exit the virtual environment when you are done with this tutorial, use the command `deactivate`. ## Build an example data source To enable you to take a first sip of Soda, you can use Docker to quickly build an example PostgreSQL data source against which you can run scans for data quality. The example data source contains data for AdventureWorks, an imaginary online e-commerce organization. * (Optional) Access the [`sodadata/sip-of-soda`](https://github.com/sodadata/sip-of-soda) repository in GitHub. * (Optional) Access a quick view of the [AdventureWorks schema](https://github.com/aldibit/soda-docs/blob/main/assets/adventureworks_schema.png). 1. Open a new tab in Terminal. 2. If it is not already running, start Docker Desktop. 3. Run the following command in Terminal to set up the prepared example data source. ```shell docker run \ --name sip-of-soda \ -p 5432:5432 \ -e POSTGRES_PASSWORD=secret \ sodadata/soda-adventureworks ``` When the output reads `data system is ready to accept connections`, your data source is set up and you are ready to proceed.
Troubleshoot {% hint style="danger" %} **Problem:** When you run `docker-compose up` you get an error that reads `[17168] Failed to execute script docker-compose`. {% endhint %} {% hint style="success" %} **Solution:** Start Docker Desktop running. {% endhint %} *** {% hint style="danger" %} **Problem:** When you run `docker-compose up` you get an error that reads `Cannot start service soda-adventureworks: Ports are not available: exposing port TCP 0.0.0.0:5432 -> 0.0.0.0:0: listen tcp 0.0.0.0:5432: bind: address already in use`. {% endhint %} {% hint style="success" %} **Solution:**\ 1\. Execute the command `lsof -i tcp:5432` to print a list of PIDs using the port.\ 2\. Use the PID value to run the following command to free up the port: `kill -9 your_PID_value`. You many need to prepend the commands with `sudo` .\ 3\. Run the `docker run` command again. {% endhint %}
Alternatively, you can use your own data for this tutorial. To do so: 1. Skip the steps above involving Docker. 2. Install the Soda Library package that corresponds with your data source, such as `soda-bigquery`, `soda-athena`, etc. See full list. 3. Collect your data source's login credentials that you must provide to Soda so that it can scan your data for quality. 4. Move on to [Connect Soda](#connect-soda). ## Connect Soda To connect to a data source such as Snowflake, PostgreSQL, Amazon Athena, or GCP BigQuery, you use a `configuration.yml` file which stores access details for your data source. This tutorial also instructs you to connect to a Soda Cloud account using API keys that you create and add to the same `configuration.yml` file. Available for free as a 45-day trial, your Soda Cloud account validates your free trial or license, gives you access to visualized scan results, tracks trends in data quality over time, lets you set alert notifications, and much more. 1. In a code editor such as Sublime or Visual Studio Code, create a new file called `configuration.yml` and save it in your `soda_sip` directory. 2. Copy and paste the following connection details into the file. The `data_source` configuration details connect Soda to the example AdventureWorks data source you set up using Docker. If you are using your own data, provide the `data_source` values that correspond with your own data source. ```yaml data_source adventureworks: type: postgres host: localhost username: postgres password: secret database: postgres schema: public ``` 3. In your Soda account, navigate to **your avatar** > **Profile**, then access the **API keys** tab. Click the plus icon to generate new API keys. Copy+paste the `soda_cloud` configuration syntax, including the API keys, into the `configuration.yml` file, as in the example below. ```yaml data_source adventureworks: type: postgres host: localhost ... schema: public soda_cloud: host: cloud.soda.io api_key_id: 2e0ba0cb-**7b api_key_secret: 5wdx**aGuRg ``` 4. Save the `configuration.yml` file and close the API modal in your Soda account. 5. In Terminal, return to the tab in which the virtual environment is active in the `soda_sip` directory. Run the following command to test Soda's connection to the data source.\ Command: ```shell soda test-connection -d adventureworks -c configuration.yml ``` Output: ```shell Soda Library 1.x.x Soda Core 3.x.x Successfully connected to 'adventureworks'. Connection 'adventureworks' is valid. ``` ## Write some checks and run a scan 1. Create another file in the `soda_sip` directory called `checks.yml`. A check is a test that Soda executes when it scans a dataset in your data source. The `checks.yml` file stores the checks you write using the Soda Checks Language (SodaCL). 2. Open the `checks.yml` file in your code editor, then copy and paste the following checks into the file. ```yaml checks for dim_customer: - invalid_count(email_address) = 0: valid format: email name: Ensure values are formatted as email addresses - missing_count(last_name) = 0: name: Ensure there are no null values in the Last Name column - duplicate_count(phone) = 0: name: No duplicate phone numbers - freshness(date_first_purchase) < 7d: name: Data in this dataset is less than 7 days old - schema: warn: when schema changes: any name: Columns have not been added, removed, or changed ```
What do these checks do? * **Ensure values are formatted as email addresses** checks that all entries in the `email_address` column are formatted as `name@domain.extension`. See [Validity metrics](https://docs.soda.io/soda-cl/validity-metrics.html). * **Ensure there are no null values in the Last Name column** automatically checks for NULL values in the `last_name` column. See [Missing metrics](https://docs.soda.io/soda-cl/missing-metrics.html). * **No duplicate phone numbers** validates that each value in the `phone` column is unique. See [Numeric metrics](https://docs.soda.io/soda-cl/numeric-metrics.html#list-of-numeric-metrics). * **Columns have not been added, removed, or changed** compares the schema of the dataset to the last scan result to determine if any columns were added, deleted, changed data type, or changed index. The first time this check executes, the results show `[NOT EVALUATED]` because there are no previous values to which to compare current results. In other words, this check requires a minimum of two scans to evaluate properly. See [Schema checks](https://docs.soda.io/soda-cl/schema.html). * **Data in this dataset is less than 7 days old** confirms that the data in the dataset is less than seven days old. See [Freshness checks](https://docs.soda.io/soda-cl/freshness.html).
3. Save the changes to the `checks.yml` file, then, in Terminal, use the following command to run a scan. A scan is a CLI command which instructs Soda to prepare SQL queries that execute data quality checks on your data source. As input, the command requires: * `-d` the name of the data source to scan * `-c` the filepath and name of the `configuration.yml` file * the filepath and name of the `checks.yml` file\ \ Command: ```shell soda scan -d adventureworks -c configuration.yml checks.yml ``` Output: ```shell Soda Library 1.0.x Soda Core 3.0.x Sending failed row samples to Soda Cloud Scan summary: 3/5 checks PASSED: dim_customer in adventureworks No changes to schema [PASSED] Emails formatted correctly [PASSED] No null values for last name [PASSED] 2/5 checks FAILED: dim_customer in adventureworks No duplicate phone numbers [FAILED] check_value: 715 Data is fresh [FAILED] max_column_timestamp: 2014-01-28 23:59:59.999999 max_column_timestamp_utc: 2014-01-28 23:59:59.999999+00:00 now_variable_name: NOW now_timestamp: 2023-04-24T21:02:15.900007+00:00 now_timestamp_utc: 2023-04-24 21:02:15.900007+00:00 freshness: 3372 days, 21:02:15.900008 Oops! 2 failures. 0 warnings. 0 errors. 3 pass. Sending results to Soda Cloud Soda Cloud Trace: 4417******32502 ``` 4. As you can see in the Scan Summary in the command-line output, some checks failed and Soda sent the results to your Cloud account. To access visualized check results and further examine the failed checks, return to your Soda account in your browser and click **Checks**.
5. In the table of checks that Soda displays, you can click the line item for one of the checks that failed to examine the visualized results in a line graph, and to access the failed row samples that Soda automatically collected when it ran the scan and executed the checks.\ \ Use the failed row samples, as in the example below, to determine what caused a data quality check to fail.
✨Well done!✨ You've taken the first step towards a future in which you and your colleagues can trust the quality and reliability of your data. Huzzah! If you are done with the example data, you can delete it from your account to start fresh with your own data. 1. Navigate to **your avatar** > **Data Sources**. 2. In the **Data Sources** tab, click the stacked dots to the right of the `adventureworks` data source, then select **Delete Data Source**. 3. Follow the steps to confirm deletion. 4. Connect to your own data by [configuring your data source connections](/soda-v3/quick-start-sip/install#configure-soda) in your existing `configuration.yml` file.\ 5\. Adjust your `checks.yml` to point to your own dataset in your data source, then adjust the checks to apply to your own data. Go ahead and run a scan! ## Go further * Get inspired on how to set up Soda to meet your [use case needs](/soda-v3/use-case-guides). * Use [check suggestions](broken://pages/CpQuK4ZSGNMBZB4LMsfy) to quickly get off the ground with basic checks for data quality. * Learn [how to start](/soda-v3/soda-cl-overview/quick-start-sodacl) writing SodaCL checks. * Read more about [SodaCL metrics and checks](/soda-v3/sodacl-reference/metrics-and-checks) in general. * Learn more about [How Soda works](/soda-v3/learning-resources/how-library-works). ## Need help? * What can Soda do for you? [Request a demo](https://www.soda.io/schedule-a-demo). * Join the [Soda community on Slack](https://community.soda.io/slack). # Get started roadmap Get started with Soda! Use this curated set of instructions to quickly get data quality tests up and running. {% hint style="warning" %} The Soda environment has been updated since this tutorial. > Refer to [v4 documentation](/quickstart) for updated tutorials. > {% endhint %} The roadmap to get started offers a curated experience to help you get from zero to productive with Soda software. However, if a guided experience is not your style, take a different path! * Follow a [15-min tutorial](/soda-v3/quick-start-sip) to set up and run Soda using demo data. * Follow a [Use case guide](/soda-v3/use-case-guides) for implementation instructions that target a specific outcome. * [Request a demo](https://www.soda.io/schedule-a-demo) so we can help you get the most out of your Soda experience. ## Get started roadmap 1. [Choose a flavor of Soda](/soda-v3/quick-start-sip/setup-guide) 🚀 Start here! 2. Set up Soda * [Self-operated](/soda-v3/quick-start-sip/install) * [Soda-hosted agent](/soda-v3/quick-start-sip/managed-agent) * [Self-hosted agent](/soda-v3/quick-start-sip/deploy) * [Programmatic](/soda-v3/quick-start-sip/programmatic) 3. [Write SodaCL checks](/soda-v3/soda-cl-overview) 4. [Run scans and review results](/soda-v3/run-a-scan) 5. [Organize, alert, investigate](/soda-v3/collaborate) Need help? Join the [Soda community on Slack](https://community.soda.io/slack). ## About Soda Soda enables Data Engineers, Data Scientists, and Data Analysts to test data for quality where and when they need to. Is your data fresh? Is it complete or missing values? Are there unexpected duplicate values? Did something go wrong during transformation? Are all the data values valid? These are the questions that Soda answers. * Use Soda with GitHub Actions to test data quality during CI/CD development. * Use Soda to build data quality rules in a collaborative, browser user interface. * Use it with Airflow to test data quality after ingestion and transformation in your pipeline. * Import your dbt tests into Soda to facilitate issue investigation and track dataset health over time. * Integrate Soda with your data catalog to gauge dataset health from within the catalog. ### How it works Soda works by taking the data quality checks that you prepare and using them to run a scan of datasets in a data source. A scan is a command which instructs Soda to prepare optimized SQL queries that execute data quality checks on your data source to find invalid, missing, or unexpected data. When checks fail, they surface bad-quality data and present check results that help you investigate and address quality issues. To test your data quality, you choose a flavor of Soda (choose a deployment model) which enables you to configure connections with your data sources and define data quality checks, then run scans that execute your data quality checks. * **Connect to your data source.**\ Connect Soda to a data source such as Snowflake, Amazon Athena, or BigQuery by providing access details for your data source such as host, port, and data source login credentials. * **Define checks to surface bad-quality data.**\ Define data quality checks using Soda Checks Language (SodaCL), a domain-specific language for data quality testing. A Soda Check is a test that Soda performs when it scans a dataset in your data source. * **Run a scan to execute your data quality checks.**\ During a scan, Soda does not ingest your data, it only scans it for quality metrics, then uses the metadata to prepare scan results1. After a scan, each check results in one of three default states: * pass: the values in the dataset match or fall within the thresholds you specified * fail: the values in the dataset do not match or fall within the thresholds you specified * error: the syntax of the check is invalid, or there are runtime or credential errors * A fourth state, warn, is something you can explicitly configure for individual checks. * **Review scan results and investigate issues.**\ You can review the scan output in the command-line and in your Soda Cloud account. Access visualized scan results, set alert notifications, track trends in data quality over time, and integrate with the messaging, ticketing, and data cataloging tools you already use, like Slack, Jira, and Atlan. 1 An exception to this rule is when Soda collects failed row samples that it presents in scan output to aid with issue investigation, a feature you can [disable](https://docs.soda.io/soda-cl/failed-row-samples.html#disable-all-failed-row-samples). > Access a [Soda product overview](https://docs.soda.io/soda/product-overview.html). > > Learn more about [How Soda works](https://docs.soda.io/soda-library/how-library-works.html). > > Learn more about [SodaCL](https://docs.soda.io/soda-cl/metrics-and-checks.html). > > Access the [Glossary](https://docs.soda.io/soda/glossary.html) for a full list of Soda terminology. # Choose a flavor of Soda Use this guide to help you decide which Soda deployment model best fits your data quality testing needs. {% hint style="warning" %} The Soda environment has been updated since this tutorial. > Refer to [v4 documentation](/quickstart) for updated tutorials. > {% endhint %} A lightweight, versatile tool for testing and monitoring data quality, you have several options for deploying Soda in your environment. As the first step in the **Get started roadmap**, this guide helps you decide how to set up Soda to best meet your data quality testing and monitoring needs. After choosing a flavor of Soda, access the corresponding [Set up Soda](#next) instructions below.
#### Get started roadmap 1. **Choose a flavor of Soda** 📍 You are here! 2. Set up Soda: sign up and install, deploy, or invoke 3. Write SodaCL checks 4. Run scans and review results 5. Organize, alert, investigate ## Choose a flavor of Soda This guide helps you decide how to set up Soda to best meet your data quality testing and monitoring needs. You can set up Soda in one or more of four flavors.
FlavorDescriptionSoda LibrarySoda AgentSoda Cloud
Self-operatedA simple setup in which you install Soda Library locally and connect it to Soda Cloud via API keys.✅✅
Soda-hosted agentRecommended
A SaaS-style setup in which you manage data quality entirely from your Soda Cloud account.		✅✅
Self-hosted agent
A setup in which you deploy a Soda Agent in a Kubernetes cluster in a cloud-services environment and connect it to Soda Cloud via different API keys.✅✅
ProgrammaticA setup in which you invoke Soda Library programmatically.✅✅
Why do I need a Soda Cloud account? To validate your account license or free trial, Soda Library or a Soda Agent must communicate with a Soda Cloud account via API keys. You create a set of API keys in your Soda Cloud account, then use them to configure the connection to Soda Library or a Soda Agent. > [Learn more](/soda-v3/quick-start-sip/get-started-roadmap#about-soda)

### Self-operated This simple setup enables you to `pip install` Soda Library from the command line, then prepare YAML files to: * configure connections to your data sources to run scans * configure the connection to your Soda Cloud account to validate your license and visualize and share data quality check results * write data quality checks Use this setup for: ✅ **A small team**: Manage data quality within a small data engineering team or data analytics team who is comfortable working with the command line and YAML files to design and execute scans for data quality. ✅ **POC**: Conduct a proof-of-concept evaluation of Soda as a data quality testing and monitoring tool. See: [Get started](/soda-v3/quick-start-sip) ✅ **Basic DQ**: Start from scratch to set up basic data quality checks on key datasets. See: [Adopt check suggestions](broken://pages/CpQuK4ZSGNMBZB4LMsfy) ✅ **Data migration**: Migrate good-quality data from one data source to another. See: [Test data before migration](/soda-v3/use-case-guides/quick-start-migration) Requirements: * Python 3.8, 3.9, or 3.10 * Pip 21.0 or greater * Login credentials for your data source

### Soda-hosted agent *Recommended* This setup provides a secure, out-of-the-box Soda Agent to manage access to data sources from within your Soda Cloud account. Quickly configure connections to your data sources in the Soda Cloud user interface, then empower all your colleagues to explore datasets, access check results, customize collections, and create their own no-code checks for data quality. See also: [Soda-hosted vs. self-hosted agent](#soda-hosted-vs-self-hosted-agent) Use this setup for: ✅ **A quick start**: Use the out-of-the-box agent to start testing data quality right away from within the Soda Cloud user interface, without the need to install or deploy any other tools. ✅ **Anomaly detection dashboard**: {% hint style="success" %} Available in 2025 {% endhint %} Use Soda's out-of-the-box **anomaly dashboards** to get automated insights into basic data quality metrics for your datasets. See: [Add anomaly dashboards](/soda-v3/use-case-guides/quick-start-automate) ✅ **Automated data monitoring**: Set up data profiling and automated data quality monitoring. See: [Automate monitoring](/soda-v3/use-case-guides/quick-start-automate) ✅ **Self-serve data quality**: Empower data analysts and scientists to self-serve and create their own no-code checks for data quality. See: [Self-serve Soda](/soda-v3/use-case-guides/quick-start-end-user) ✅ **Data migration**: Migrate good-quality data from one data source to another. See: [Test ](/soda-v3/use-case-guides/quick-start-dev)[data ](/soda-v3/use-case-guides/quick-start-dev)[before migration](/soda-v3/use-case-guides/quick-start-dev) ✅ **Data catalog integration**: Integrate Soda with a data catalog such as Atlan, Alation, or Metaphor. See: [Integrate Soda](/soda-v3/integrate-soda) Soda hosts agents in a secure environment in Amazon AWS. As a SOC 2 Type 2 certified business, Soda responsibly manages Soda-hosted agents to ensure that they remain private, secure, and independent of all other hosted agents. See [Data security and privacy](/soda-v3/learning-resources/data-privacy) for details. Requirements: * Login credentials for your data source; Soda securely stores passwords as [Kubernetes secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
### Self-hosted agent This setup enables a data or infrastructure engineer to deploy Soda Library as an agent in a Kubernetes cluster within a cloud-services environment such as Google Cloud Platform, Azure, or AWS. The engineer can manage access to data sources while giving Soda Cloud end users easy access to Soda check results and enabling them to write their own checks for data quality. Users connect to data sources and create no-code checks for data quality directly in the Soda Cloud user interface. See also: [Soda-hosted vs. self-hosted agent](#soda-hosted-vs-self-hosted-agent) Use this setup for: ✅ **Self-serve data quality**: Empower data analysts and scientists to self-serve and create their own checks for data quality. See: [Self-serve Soda](/soda-v3/use-case-guides/quick-start-end-user) ✅ **Data migration**: Migrate good-quality data from one data source to another. See: [Test ](/soda-v3/use-case-guides/quick-start-dev)[data ](/soda-v3/use-case-guides/quick-start-dev)[before migration](/soda-v3/use-case-guides/quick-start-dev) ✅ **Anomaly detection dashboard**: {% hint style="success" %} Available in 2025 {% endhint %} Use Soda's out-of-the-box **anomaly dashboards** to get automated insights into basic data quality metrics for your datasets. See: [Add anomaly dashboards](/soda-v3/use-case-guides/quick-start-automate) ✅ **Data catalog integration**: Integrate Soda with a data catalog such as Atlan, Alation, or Metaphor. See: [Integrate Soda](/soda-v3/integrate-soda) ✅ **Secrets manager integration**: Integrate your Soda Agent with an external secrets manager to securely access frequently rotated data source login credentials. See: [Integrate with a Secrets Manager](/soda-v3/use-case-guides/quick-start-secrets) Requirements: * Access to your cloud-services environment, plus the authorization to deploy containerized apps in a new or existing Kubernetes cluster * Login credentials for your data source
### Programmatic Use this setup to invoke Soda programmatically in, for example, an Airflow DAG or GitHub Workflow. You provide connection details for data sources and Soda Cloud inline or in external YAML files, and similarly define data quality checks inline or in a separate YAML file. Use this setup for: ✅ **Testing during development**: Test data before and after ingestion and transformation during development. See: [Test data during development](/soda-v3/use-case-guides/quick-start-dev) ✅ **Circuit-breaking in a pipeline**: Test data in an Airflow pipeline so as to enable circuit breaking that prevents bad-quality data from having a downstream impact. See: [Test data in production](/soda-v3/use-case-guides/quick-start-prod) ✅ **Databricks Notebook**: Invoke Soda data quality scans in a Databricks Notebook. See: [Add Soda to a Databricks notebook](/soda-v3/use-case-guides/quick-start-databricks)
Requirements: * Python 3.8, 3.9, or 3.10 * Pip 21.0 or greater * Login credentials for your data source
## Soda-hosted vs self-hosted agent Though similar, the type of Soda agent you choose to use depends upon the following factors. | Factor | Soda-hosted agent | Self-hosted agent | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | Data source compatibility | Compatible with a [limited subset](/soda-v3/quick-start-sip/managed-agent#compatibility) of Soda-supported data sources. | Compatible with [nearly all](/soda-v3/quick-start-sip/deploy#compatibility) Soda-supported data sources. | | Upgrade maintenance | Soda manages all upgrades to the latest available version of the Soda Agent. | You manage all upgrades to your Soda Agent deployed on your Kubernetes cluster. | | External Secrets manager integration | Unable to integrate with an External Secrets manager. | Able to [integrate](/soda-v3/use-case-guides/quick-start-secrets) with an External Secrets manager to better manage frequently rotated login credentials. | | Network connectivity | Access Soda Agent via public networks or passlisting. | Deploy the Soda Agent inside your own private cloud or on-premises network infrastructure. | ## Next 1. ~~Choose a flavor of Soda~~ 2. **Set up Soda.** Select the setup instructions that correspond with your flavor of Soda: * [Self-operated](/soda-v3/quick-start-sip/install) * [Soda-hosted agent](/soda-v3/quick-start-sip/managed-agent) * [Self-hosted agent](/soda-v3/quick-start-sip/deploy) * [Programmatic](/soda-v3/quick-start-sip/programmatic) 3. Write SodaCL checks 4. Run scans and review results 5. Organize, alert, investigate > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Install Soda Library From your command-line interface, execute a pip install command to install Soda Library in your environment. {% hint style="warning" %} The Soda environment has been updated since this tutorial. > Refer to [v4 documentation](/quickstart) for updated tutorials. > {% endhint %} Soda Library is a Python library and command-line interface (CLI) tool that enables Data Engineers to test the data in a data source to surface invalid, missing, or unexpected data. As a step in the **Get started roadmap**, this guide offers instructions to set up, install, and configure Soda in a [self-operated deployment model](/soda-v3/quick-start-sip/setup-guide#self-operated). #### Get started roadmap 1. ~~Choose a flavor of Soda~~ 2. **Set up Soda: self-operated** 📍 You are here!\ a. [Review requirements](#requirements)\ b. [Install Soda Library](#install-soda-library)\ c. [Configure Soda](#configure-soda)
3. Write SodaCL checks 4. Run scans and review results 5. Organize, alert, investigate > 💡 **TL;DR:** Follow a [15-min tutorial](/soda-v3/quick-start-sip) to set up and run Soda with example data. ## Requirements To use Soda Library, you must have installed the following on your system. * Python 3.8, 3.9, or 3.10. To check your existing version, use the CLI command: `python --version` or `python3 --version`\ If you have not already installed Python, consider using [pyenv](https://github.com/pyenv/pyenv/wiki) to manage multiple versions of Python in your environment. * Pip 21.0 or greater. To check your existing version, use the CLI command: `pip --version` * A Soda Cloud account; see next section.
Python versions Soda supports Soda officially supports Python versions 3.8, 3.9, and 3.10.\ Though largely funcntional, efforts to fully support Python 3.11 and 3.12 are ongoing. Using Python 3.11, some users might have some issues with dependencies constraints. At times, extra the combination of Python 3.11 and dependencies constraints requires that a dependency be built from source rather than downloaded pre-built. The same applies to Python 3.12, although there is some anecdotal evidence that indicates that 3.12 might not work in all scenarios due to dependencies constraints.
## Create a Soda Cloud account 1. In a browser, navigate to [cloud.soda.io/signup](https://cloud.soda.io/signup?utm_source=docs) to create a new Soda account, which is free for a 45-day trial. If you already have a Soda account, log in. 2. Navigate to **your avatar** > **Profile**, then access the **API keys** tab. Click the plus icon to generate new API keys. 3. Copy+paste the API key values to a temporary, secure place in your local environment.
Why do I need a Soda Cloud account? To validate your account license or free trial, Soda Library must communicate with a Soda Cloud account via API keys. You create a set of API keys in your Soda Cloud account, then use them to configure the connection to Soda Library. > [Learn more](/soda-v3/quick-start-sip/get-started-roadmap#about-soda)
## Install Soda Library {% tabs %} {% tab title="MacOS, Linux" %} 1. Best practice dictates that you install the Soda Library CLI using a virtual environment. In your command-line interface tool, create a virtual environment in the `.venv` directory using the commands below. Depending on your version of Python, you may need to replace `python` with `python3` in the first command. ```shell python -m venv .venv source .venv/bin/activate ``` 2. Upgrade pip inside your new virtual environment. ```shell pip install --upgrade pip ``` 3. Execute the following command, replacing `soda-postgres` with the install package that matches the type of data source you use to store data. ```shell # For bash interactive shell pip install -i https://pypi.cloud.soda.io soda-postgres # For zsh interactive shell pip install -i https://pypi.cloud.soda.io "soda-postgres" ``` | Data source | Install package | | ------------------------------------------------------------------------------- | ------------------------ | | Amazon Athena | `soda-athena` | | Amazon Redshift | `soda-redshift` | |

Apache Spark DataFrames
(For use with programmatic Soda scans, only.)

| `soda-spark-df` | | Azure Synapse | `soda-sqlserver` | | ClickHouse | `soda-mysql` | | Dask and Pandas | `soda-pandas-dask` | | Databricks | `soda-spark[databricks]` | | Denodo | `soda-denodo` | | Dremio | `soda-dremio` | | DuckDB | `soda-duckdb` | | GCP BigQuery | `soda-bigquery` | | Google CloudSQL | `soda-postgres` | | IBM DB2 | `soda-db2` | | Local file | Use Dask. | | MotherDuck | `soda-duckdb` | | MS SQL Server | `soda-sqlserver` | | MySQL | `soda-mysql` | | OracleDB | `soda-oracle` | | PostgreSQL | `soda-postgres` | | Presto | `soda-presto` | | Snowflake | `soda-snowflake` | | Trino | `soda-trino` | | Vertica | `soda-vertica` | To deactivate the virtual environment, use the following command: ```shell deactivate ``` #### Troubleshoot As of version 1.7.0, Soda Library packages include Pydantic version 2 for data validation. If your systems require the use of Pydantic version 1, you can install an extra package that uses Pydantic version 1. To do so, use the following command, adjusting the type of library to correspond with your data source. ```sh #bash pip install -i https://pypi.cloud.soda.io soda-postgres[pydanticv1] #zsh pip install -i https://pypi.cloud.soda.io "soda-spark-df[pydanticv1]" ``` {% endtab %} {% tab title="Windows" %} 1. Best practice dictates that you install the Soda Library CLI using a virtual environment. In your command-line interface tool, create a virtual environment in the `.venv` directory using the commands below. Depending on your version of Python, you may need to replace `python` with `python3` in the first command. Reference the [virtualenv documentation](https://virtualenv.pypa.io/en/legacy/userguide.html#activate-script) for activating a Windows script. ```shell python -m venv .venv .venv\Scripts\activate ``` 2. Upgrade pip inside your new virtual environment. ```shell pip install --upgrade pip ``` 3. Execute the following command, replacing `soda-postgres` with the install package that matches the type of data source you use to store data. ```shell pip install -i https://pypi.cloud.soda.io soda-postgres ``` | Data source | Install package | | ------------------------------------------------------------------------------ | ------------------------ | | Amazon Athena | `soda-athena` | | Amazon Redshift | `soda-redshift` | |

Apache Spark DataFrame
(For use with programmatic Soda scans, only.)

| `soda-spark-df` | | Azure Synapse | `soda-sqlserver` | | ClickHouse | `soda-mysql` | | Dask and Pandas | `soda-pandas-dask` | | Databricks | `soda-spark[databricks]` | | Denodo | `soda-denodo` | | Dremio | `soda-dremio` | | DuckDB | `soda-duckdb` | | GCP BigQuery | `soda-bigquery` | | Google CloudSQL | `soda-postgres` | | IBM DB2 | `soda-db2` | | MS SQL Server | `soda-sqlserver` | | MySQL | `soda-mysql` | | OracleDB | `soda-oracle` | | PostgreSQL | `soda-postgres` | | Snowflake | `soda-snowflake` | | Trino | `soda-trino` | | Vertica | `soda-vertica` | To deactivate the virtual environment, use the following command: ```shell deactivate ``` Reference the [virtualenv documentation](https://virtualenv.pypa.io/en/legacy/userguide.html#activate-script) for activating a Windows script. #### Troubleshoot As of version 1.7.0, Soda Library packages include Pydantic version 2 for data validation. If your systems require the use of Pydantic version 1, you can install an extra package that uses Pydantic version 1. To do so, use the following command, adjusting the type of library to correspond with your data source. ```sh #bash pip install -i https://pypi.cloud.soda.io soda-postgres[pydanticv1] #zsh pip install -i https://pypi.cloud.soda.io "soda-spark-df[pydanticv1]" ```
{% endtab %} {% tab title="Docker" %} Use Soda’s Docker image in which Soda Scientific is pre-installed. You need Soda Scientific to be able to use SodaCL [distribution checks](/soda-v3/sodacl-reference/distribution) or [anomaly detection checks](/soda-v3/sodacl-reference/anomaly-detection). 1. If you have not already done so, [install Docker](https://docs.docker.com/get-docker/) in your local environment. 2. From Terminal, run the following command to pull Soda Library’s official Docker image; adjust the version to reflect the most [recent release](/soda-v3/release-notes). ```sh docker pull sodadata/soda-library:v1.0.3 ``` 3. Verify the pull by running the following command. ```sh docker run sodadata/soda-library:v1.0.3 --help ``` Output: ```sh Usage: soda [OPTIONS] COMMAND [ARGS]... Soda Library CLI version 1.0.x, Soda Core CLI version 3.0.xx Options: --version Show the version and exit. --help Show this message and exit. Commands: ingest Ingests test results from a different tool scan Runs a scan suggest Generates suggestions for a dataset test-connection Tests a connection update-dro Updates contents of a distribution reference file ``` When you run the Docker image on a non-Linux/amd64 platform, you may see the following warning from Docker, which you can ignore. ```sh WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested ``` 4. When you are ready to run a Soda scan, use the following command to run the scan via the docker image. Replace the placeholder values with your own file paths and names. ```sh docker run -v /path/to/your_soda_directory:/sodacl sodadata/soda-library scan -d your_data_source -c /sodacl/your_configuration.yml /sodacl/your_checks.yml ``` Optionally, you can specify the version of Soda Library to use to execute the scan. This may be useful when you do not wish to use the latest released version of Soda Library to run your scans. The example scan command below specifies Soda Library version 1.0.0. ```sh docker run -v /path/to/your_soda_directory:/sodacl sodadata/soda-library:v1.0.0 scan -d your_data_source -c /sodacl/your_configuration.yml /sodacl/your_checks.yml ```
What does the scan command do? * `docker run` ensures that the docker engine runs a specific image. * `-v` mounts your SodaCL files into the container. In other words, it makes the configuration.yml and checks.yml files in your local environment available to the docker container. The command example maps your local directory to `/sodacl` inside of the docker container. * `sodadata/soda-library` refers to the image that `docker run` must use. * `scan` instructs Soda Library to execute a scan of your data. * `-d` indicates the name of the data source to scan. * `-c` specifies the filepath and name of the configuration YAML file.
**Error: Mounts denied** If you encounter the following error, follow the procedure below. ``` docker: Error response from daemon: Mounts denied: The path /soda-library-test/files is not shared from the host and is not known to Docker. You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing. See https://docs.docker.com/desktop/mac for more info. ``` You need to give Docker permission to acccess your configuration.yml and checks.yml files in your environment. To do so: 1. Access your Docker Dashboard, then select Preferences (gear symbol). 2. Select Resources, then follow the [Docker instructions](https://www.docker.com/blog/file-sharing-with-docker-desktop/) to add your Soda project directory —the one you use to store your configuration.yml and checks.yml files— to the list of directories that can be bind-mounted into Docker containers. 3. Click Apply & Restart, then repeat steps 2 - 4 above. **Error: Configuration path does not exist** If you encounter the following error, double check the syntax of the scan command in step 4 above. * Be sure to prepend `/sodacl/` to both the congifuration.yml filepath and the checks.yml filepath. * Be sure to mount your files into the container by including the `-v` option. For example, `-v /Users/MyName/soda_project:/sodacl`. ``` Soda Library 1.0.x Configuration path 'configuration.yml' does not exist Path "checks.yml" does not exist Scan summary: No checks found, 0 checks evaluated. 2 errors. Oops! 2 errors. 0 failures. 0 warnings. 0 pass. ERRORS: Configuration path 'configuration.yml' does not exist Path "checks.yml" does not exist ``` {% endtab %} {% tab title="Soda Scientific" %} Install Soda Scientific to be able to use SodaCL [distribution checks](/soda-v3/sodacl-reference/distribution) or [anomaly detection checks](/soda-v3/sodacl-reference/anomaly-detection). You have two installation options to choose from: * [Install Soda Scientific in a virtual environment (Recommended)](#install-soda-scientific-in-a-virtual-environment-recommended) * [Use Docker to run Soda Library with Soda Scientific](#use-docker-to-run-soda-scientific) #### Install Soda Scientific in a virtual environment (Recommended) 1. Set up a virtual environment, and install Soda Library in your new virtual environment. 2. Use the following command to install Soda Scientific. ``` pip install -i https://pypi.cloud.soda.io soda-scientific ```
List of Soda Scientific dependencies * pandas<2.0.0 * wheel * pydantic>=1.8.1,<2.0.0 * scipy>=1.8.0 * numpy>=1.23.3, <2.0.0 * inflection==0.5.1 * httpx>=0.18.1,<2.0.0 * PyYAML>=5.4.1,<7.0.0 * cython>=0.22 * prophet>=1.1.0,<2.0.0
**Error: Library not loaded** If you have defined an `anomaly detection` check and you use an M1 MacOS machine, you may get a`Library not loaded: @rpath/libtbb.dylib` error. This is a known issue in the MacOS community and is caused by issues during the installation of the [prophet library](https://github.com/facebook/prophet). There currently are no official workarounds or releases to fix the problem, but the following adjustments may address the issue. 1. Install `soda-scientific` as per the local environment installation instructions and activate the virtual environment. 2. Use the following command to navigate to the directory in which the `stan_model` of the `prophet` package is installed in your virtual environment. ```sh cd path_to_your_python_virtual_env/lib/pythonyour_version/site_packages/prophet/stan_model/ ``` For example, if you have created a python virtual environment in a `/venvs` directory in your home directory and you use Python 3.9, you would use the following command. ```sh cd ~/venvs/soda-library-prophet11/lib/python3.9/site-packages/prophet/stan_model/ ``` 3. Use the `ls` command to determine the version number of `cmndstan` that `prophet` installed. The `cmndstan` directory name includes the version number. ```sh ls cmdstan-2.26.1 prophet_model.bin ``` 4. Add the `rpath` of the `tbb` library to your `prophet` installation using the following command. ```sh install_name_tool -add_rpath @executable_path/cmdstanyour_cmdstan_version/stan/lib/stan_math/lib/tbb prophet_model.bin ``` With `cmdstan` version `2.26.1`, you would use the following command. ```sh install_name_tool -add_rpath @executable_path/cmdstan-2.26.1/stan/lib/stan_math/lib/tbb prophet_model.bin ``` #### Use Docker to run Soda Scientific Use Soda’s Docker image in which Soda Scientific is pre-installed. You need Soda Scientific to be able to use SodaCL [distribution checks](/soda-v3/sodacl-reference/distribution) or [anomaly detection checks](/soda-v3/sodacl-reference/anomaly-detection). 1. If you have not already done so, [install Docker](https://docs.docker.com/get-docker/) in your local environment. 2. From Terminal, run the following command to pull Soda Library’s official Docker image; adjust the version to reflect the most [recent release](/soda-v3/release-notes). ```sh docker pull sodadata/soda-library:v1.0.3 ``` 3. Verify the pull by running the following command. ```sh docker run sodadata/soda-library:v1.0.3 --help ``` Output: ```sh Usage: soda [OPTIONS] COMMAND [ARGS]... Soda Library CLI version 1.0.x, Soda Core CLI version 3.0.xx Options: --version Show the version and exit. --help Show this message and exit. Commands: ingest Ingests test results from a different tool scan Runs a scan suggest Generates suggestions for a dataset test-connection Tests a connection update-dro Updates contents of a distribution reference file ``` When you run the Docker image on a non-Linux/amd64 platform, you may see the following warning from Docker, which you can ignore. ```sh WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested ``` 4. When you are ready to run a Soda scan, use the following command to run the scan via the docker image. Replace the placeholder values with your own file paths and names. ```sh docker run -v /path/to/your_soda_directory:/sodacl sodadata/soda-library scan -d your_data_source -c /sodacl/your_configuration.yml /sodacl/your_checks.yml ``` Optionally, you can specify the version of Soda Library to use to execute the scan. This may be useful when you do not wish to use the latest released version of Soda Library to run your scans. The example scan command below specifies Soda Library version 1.0.0. ```sh docker run -v /path/to/your_soda_directory:/sodacl sodadata/soda-library:v1.0.0 scan -d your_data_source -c /sodacl/your_configuration.yml /sodacl/your_checks.yml ```
What does the scan command do? * `docker run` ensures that the docker engine runs a specific image. * `-v` mounts your SodaCL files into the container. In other words, it makes the configuration.yml and checks.yml files in your local environment available to the docker container. The command example maps your local directory to `/sodacl` inside of the docker container. * `sodadata/soda-library` refers to the image that `docker run` must use. * `scan` instructs Soda Library to execute a scan of your data. * `-d` indicates the name of the data source to scan. * `-c` specifies the filepath and name of the configuration YAML file.
**Error: Mounts denied** If you encounter the following error, follow the procedure below. ```sh docker: Error response from daemon: Mounts denied: The path /soda-library-test/files is not shared from the host and is not known to Docker. You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing. See https://docs.docker.com/desktop/mac for more info. ``` You need to give Docker permission to acccess your configuration.yml and checks.yml files in your environment. To do so: 1. Access your Docker Dashboard, then select Preferences (gear symbol). 2. Select Resources, then follow the [Docker instructions](https://www.docker.com/blog/file-sharing-with-docker-desktop/) to add your Soda project directory—the one you use to store your configuration.yml and checks.yml files—to the list of directories that can be bind-mounted into Docker containers. 3. Click Apply & Restart, then repeat steps 2 - 4 above. **Error: Configuration path does not exist** If you encounter the following error, double check the syntax of the scan command in step 4 above. * Be sure to prepend `/sodacl/` to both the congifuration.yml filepath and the checks.yml filepath. * Be sure to mount your files into the container by including the `-v` option. For example, `-v /Users/MyName/soda_project:/sodacl`. ```sh Soda Library 1.0.x Configuration path 'configuration.yml' does not exist Path "checks.yml" does not exist Scan summary: No checks found, 0 checks evaluated. 2 errors. Oops! 2 errors. 0 failures. 0 warnings. 0 pass. ERRORS: Configuration path 'configuration.yml' does not exist Path "checks.yml" does not exist ``` {% endtab %} {% endtabs %} ## Configure Soda 1. Soda Library connects with Spark DataFrames in a unique way, using programmtic scans. * If you are using Spark DataFrames, follow the configuration details in [Connect to Spark](/soda-v3/data-source-reference/connect-spark). * If you are *not* using Spark DataFrames, continue to step 2. 2. In the same directory and environment in which you installed Soda Library, use a code editor to create a new `configuration.yml` file. This file stores connection details for your data sources and your Soda Cloud account.\ Use the data source-specific connection configurations (see: [Data source reference](/soda-v3/data-source-reference)) to copy+paste the connection syntax into your file, then adjust the values to correspond with your data source’s details, as in the following example for PostgreSQL. * You can use [system variables](/soda-v3/quick-start-sip/install#provide-credentials-as-system-variables) to pass sensitive values, if you wish. * If you want to run scans on multiple schemas in the data source, add one data source config block per schema. ```sh data_source my_datasource: type: postgres host: localhost username: postgres password: secret database: postgres schema: publi ``` 3. Copy+paste the following `soda_cloud` configuration syntax into the `configuration.yml` file, as in the example below. Input the API key values you created in Soda CLoud. * *Do not* nest the `soda_cloud` configuration under the `datasource` configuration. * For `host`, use `cloud.soda.io` for EU region; use `cloud.us.soda.io` for USA region, according to your selection when you created your Soda Cloud account. * Optionally, provide a value for the `scheme` property to indicate which scheme to use to initialize the URI instance. If you do not explicitly include a `scheme` property, Soda uses the default `https`. ``` soda_cloud: # Use cloud.soda.io for EU region # Use cloud.us.soda.io for US region host: https://cloud.soda.io api_key_id: 2e0ba0cb-your-api-key-7b api_key_secret: 5wd-your-api-key-secret-aGuRg scheme: ``` * Save the `configuration.yml` file. Run the following scan to confirm that Soda can successfully connect with your data source. ``` soda test-connection -d my_datasource -c configuration.yml ``` ## Provide credentials as system variables If you wish, you can provide data source login credentials or any of the properties in the configuration YAML file as system variables instead of storing the values directly in the file. System variables persist only for as long as you have the terminal session open in which you created the variable. For a longer-term solution, consider using permanent environment variables stored in your `~/.bash_profile` or `~/.zprofile` files. ### For connection configuration values 1. From your command-line interface, set a system variable to store the value of a property that the configuration YAML file uses. For example, you can use the following command to define a system variable for your password. ```shell export POSTGRES_PASSWORD=1234 ``` 2. Test that the system retrieves the value that you set by running an `echo` command. ```shell echo $POSTGRES_PASSWORD ``` 3. In the configuration YAML file, set the value of the property to reference the environment variable, as in the following example. ```yaml data_source my_database_name: type: postgres host: soda-temp-demo port: '5432' username: sodademo password: ${POSTGRES_PASSWORD} database: postgres schema: public ``` 4. Save the configuration YAML file, then run a scan to confirm that Soda Library connects to your data source without issue. ```shell soda test-connection -d my_datasource -c configuration.yml ``` ### For API key values 1. From your command-line interface, set a system variable to store the value of a property that the configuration YAML file uses. For example, you can use the following command to define a system variable for your password. ```shell export API_KEY=1234 ``` 2. Test that the system retrieves the value that you set by running an `echo` command. ```shell echo $API_KEY ``` 3. In the configuration YAML file, set the value of the property to reference the environment variable, as in the following example. ```yaml data_source my_database_name: type: postgres host: soda-temp-demo port: '5432' username: sodademo password: ${POSTGRES_PASSWORD} database: postgres schema: public soda_cloud: host: cloud.soda.io api_key_id: ${API_KEY} api_key_secret: ${API_SECRET} ``` 4. Save the configuration YAML file, then run a scan to confirm that Soda Library connects to Soda Cloud without issue. ```shell soda test-connection -d my_datasource -c configuration.yml ``` ## Next 1. ~~Choose a flavor of Soda~~ 2. ~~Set up Soda: self-operated~~ 3. [Write SodaCL checks](/soda-v3/soda-cl-overview) 4. Run scans and review results 5. Organize, alert, investigate > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Set up a Soda-hosted agent Use an out-of-the-box Soda-hosted agent to connect to your data sources and begin testing data quality. {% hint style="warning" %} The Soda environment has been updated since this tutorial. > Refer to [v4 documentation](/quickstart) for updated tutorials. > {% endhint %} The **Soda Agent** is a tool that empowers Soda Cloud users to securely access data sources to scan for data quality. Use the secure, out-of-the-box **Soda-hosted agent** made available for every Soda Cloud organization or, alternatively, you can create a Kubernetes cluster in your organization's environment and use Helm to deploy a self-hosted Soda Agent in the cluster; see [Deploy a Soda Agent](/soda-v3/quick-start-sip/deploy). A Soda-hosted agent enables Soda Cloud users to securely connect to [supported data sources](#compatibility) and create checks for data quality in the new data source. As a step in the **Get started roadmap**, this guide offers instructions to set up Soda in a [Soda-hosted agent deployment model](/soda-v3/quick-start-sip/setup-guide). #### Get started roadmap 1. ~~Choose a flavor of Soda~~ 2. **Set up Soda: Soda-hosted agent** 📍 You are here!\ a. [Create a Soda Cloud account](#create-a-soda-cloud-account)\ b. [Add a new data source](#add-a-new-data-source) 3. Write SodaCL checks 4. Run scans and review results 5. Organize, alert, investigate ## Compatibility * BigQuery * Databricks SQL * MS SQL Server * MySQL * PostgreSQL * Redshift * Snowflake ## Create a Soda Cloud account 1. If you have not already done so, create a Soda Cloud account at [cloud.soda.io](https://cloud.soda.io/signup?utm_source=docs). If you already have a Soda account, log in. 2. By default, Soda prepares a Soda-hosted agent for all newly-created accounts. However, if you are an Admin in an existing Soda Cloud account and wish to use a Soda-hosted agent, navigate to **your avatar** > **Organization Settings**. In the **Organization** tab, click the checkbox to **Enable Soda-hosted Agent**. 3. Navigate to **your avatar** > **Data Sources**, then access the **Agents** tab. Notice your out-of-the-box Soda-hosted agent that is up and running.
## Add a new data source In your Soda Cloud account, navigate to **your avatar** > **Data Sources**. Click **New Data Source**, then follow the guided steps to create a new data source. Refer to the sections below for insight into the values to enter in the fields and editing panels in the guided steps. > Already have data source connected to a self-hosted agent? > > You can [migrate](/soda-v3/quick-start-sip/upgrade#migrate-a-data-source-from-a-self-hosted-to-a-soda-hosted-agent) a data source to a Soda-hosted agent. #### 1. Attributes
Field or LabelGuidance
Data Source LabelProvide a unique identifier for the data source. Soda Cloud uses the label you provide to define the immutable name of the data source against which it runs the Default Scan.
Default Scan AgentSelect the Soda-hosted agent, or the name of a Soda Agent that you have previously set up in your secure environment. This identifies the Soda Agent to which Soda Cloud must connect in order to run its scan.
Check ScheduleProvide the scan frequency details Soda Cloud uses to execute scans according to your needs. If you wish, you can define the schedule as a cron expression.
Starting At (UTC)Select the time of day to run the scan. The default value is midnight.
Custom Cron Expression(Optional) Write your own cron expression to define the schedule Soda Cloud uses to run scans.
Anomaly Dashboard Scan Schedule (available in 2025)Provide the scan frequency details Soda Cloud uses to execute a daily scan to automatically detect anomalies for the anomaly dashboard.
#### 2. Connect Enter values in the fields to provide the connection configurations Soda Cloud needs to be able to access the data in the data source. Connection configurations are data source-specific and include values for things such as a database's host and access credentials. Soda hosts agents in a secure environment in Amazon AWS. As a SOC 2 Type 2 certified business, Soda responsibly manages Soda-hosted agents to ensure that they remain private, secure, and independent of all other hosted agents. See [Data security and privacy](/soda-v3/learning-resources/data-privacy) for details. Use the following data source-specific connection configuration pages to populate the connection fields in Soda Cloud. * [Connect to BigQuery](/soda-v3/data-source-reference/connect-bigquery) * [Connect to Databricks SQL](/soda-v3/data-source-reference/connect-databricks) * [Connect to MS SQL Server](/soda-v3/data-source-reference/connect-mssql) * [Connect to MySQL](/soda-v3/data-source-reference/connect-mysql) * [Connect to PostgreSQL](/soda-v3/data-source-reference/connect-postgres) * [Connect to Redshift](/soda-v3/data-source-reference/connect-redshift) * [Connect to Snowflake](/soda-v3/data-source-reference/connect-snowflake) #### 3. Discover During its initial scan of your data source, Soda Cloud discovers all the datasets the data source contains. It captures basic information about each dataset, including dataset names, the columns each contains, and the type of data each column contains such as integer, character varying, timestamp, etc. In the editing panel, specify the datasets that Soda Cloud must include or exclude from this basic discovery activity. The default syntax in the editing panel instructs Soda to collect basic dataset information from all datasets in the data source *except* those with names that begin with `test_`. The `%` is a wildcard character. See [Add dataset discovery](/soda-v3/soda-cl-overview/profile#add-dataset-discovery) for more detail on profiling syntax. {% hint style="warning" %} **Known issue**: SodaCL does not support using variables in column profiling and dataset discovery configurations. {% endhint %} ```yaml discover datasets: datasets: - include % - exclude test_% ``` #### 4. Profile To gather more detailed profile information about datasets in your data source and automatically build an **anomaly dashboard** for data quality observability, you can configure Soda Cloud to profile the columns in datasets. Profiling a dataset produces two tabs' worth of data in a dataset page: * In the **Columns** tab, you can see column profile information including details such as the calculated mean value of data in a column, the maximum and minimum values in a column, and the number of rows with missing data.
* In the **Anomalies** tab, you can access an out-of-the-box anomaly dashboard that uses the column profile information to automatically begin detecting anomalies in your data relative to the patterns the machine learning algorithm learns over the course of approximately five days. (Available in 2025. [Learn more](/soda-v3/collaborate/anomaly-dashboard))
In the editing panel, provide details that Soda Cloud uses to determine which datasets to include or exclude when it profiles the columns in a dataset. The default syntax in the editing panel instructs Soda to profile every column of every dataset in this data source, and, superfluously, all datasets with names that begin with `prod`. The `%` is a wildcard character. See [Add column profiling](/soda-v3/soda-cl-overview/profile#add-column-profiling) for more detail on profiling syntax. Column profiling and automated anomaly detection can be resource-heavy, so carefully consider the datasets for which you truly need column profile information. Refer to [Compute consumption and cost considerations](/soda-v3/soda-cl-overview/profile#compute-consumption-and-cost-considerations) for more detail. ```yaml profile columns: columns: - "%.%" # Includes all your datasets - prod% # Includes all datasets that begin with 'prod' ``` #### 5. Check When Soda Cloud automatically discovers the datasets in a data source, it prepares automated monitoring checks for each dataset. These checks detect anomalies and monitor schema evolution, corresponding to the SodaCL [anomaly detection](/soda-v3/sodacl-reference/anomaly-detection) and [schema evolution](/soda-v3/sodacl-reference/schema#define-schema-evolution-checks) checks, respectively. (Note that if you have signed up for early access to [anomaly dashboards](/soda-v3/collaborate/anomaly-dashboard) for datasets, this **Check** tab is unavailable as Soda performs all automated monitoring automatically in the dashboards.) In the editing panel, specify the datasets that Soda Cloud must include or exclude when preparing automated monitoring checks. The default syntax in the editing panel indicates that Soda will add automated monitoring to all datasets in the data source *except* those with names that begin with `test_`. The `%` is a wildcard character. Refer to [Add automated monitoring checks](/soda-v3/soda-cl-overview/automated-monitoring) for further detail. ```yaml automated monitoring: datasets: - include % - exclude test_% ``` #### (5) 6. Assign This tab is the fifth step in the guided workflow if the **5. Check** tab is absent because you requested access to the anomaly dashboards feature. | Field or Label | Guidance | | --------------------- | ------------------------------------------------------------------------------------------------------------------------- | | Data Source Owner | The Data Source Owner maintains the connection details and settings for this data source and its Default Scan Definition. | | Default Dataset Owner | g6driOiYtQZD | ## Next 1. ~~Choose a flavor of Soda~~ 2. ~~Set up Soda: self-hosted agent~~ 3. [**Write SodaCL checks**](/soda-v3/soda-cl-overview) 4. Run scans and review results 5. Organize, alert, investigate > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Deploy a Soda Agent Learn how to deploy a Soda Agent in a Kubernetes cluster. {% hint style="warning" %} The Soda environment has been updated since this tutorial. > Refer to [v4 documentation](/quickstart) for updated tutorials. > {% endhint %} The **Soda Agent** is a tool that empowers Soda Cloud users to securely access data sources to scan for data quality. Create a Kubernetes cluster, then use Helm to deploy a self-hosted Soda Agent in the cluster. This setup enables Soda Cloud users to securely connect to data sources (BigQuery, Snowflake, etc.) from within the Soda Cloud web application. Any user in your Soda Cloud account can add a new data source via the agent, then write their own no-code checks and agreements to check for data quality in the new data source. Alternatively, if you use a BigQuery, Databricks SQL, MS SQL Server, MySQL, PostgreSQL, Redshift, or Snowflake data source, you can use a secure, out-of-the-box [Soda-hosted agent](/soda-v3/quick-start-sip/managed-agent) made available for every Soda Cloud organization. As a step in the **Get started roadmap**, this guide offers instructions to set up, install, and configure Soda in a [self-hosted agent deployment model](/soda-v3/quick-start-sip/setup-guide#self-hosted-agent). **Get started roadmap** 1. ~~Choose a flavor of Soda~~ 2. **Set up Soda: self-hosted agent** 📍 You are here!\ a. [Create a Soda Cloud account](/soda-v3/quick-start-sip/deploy#create-a-soda-cloud-account)\ b. [Deploy a Soda Agent in a Kubernetes cluster](/soda-v3/quick-start-sip/deploy#deploy-a-soda-agent-in-a-kubernetes-cluster)\ c. [Add a new data source](/soda-v3/quick-start-sip/deploy#add-a-new-data-source) 3. Write SodaCL checks 4. Run scans and review results 5. Organize, alert, investigate ## Create a Soda Cloud account The Soda Agent communicates with your Soda Cloud account using API public and private keys. Note that the keys a Soda Agent uses are *different* from the API keys Soda Library uses to connect to Soda Cloud. 1. If you have not already done so, create a Soda Cloud account at [cloud.soda.io](https://cloud.soda.io/signup?utm_source=docs). If you already have a Soda account, log in. 2. In your Soda Cloud account, navigate to **your avatar** > **Data Sources**, then navigate to the **Agents** tab. Click **New Soda Agent**. 3. The dialog box that appears offers abridged instructions to set up a new Soda Agent from the command-line; more thorough instructions exist in this documentation, below.\ \ For now, copy and paste the values for both the **API Key ID** and **API Key Secret** to a temporary, secure place in your local environment. You will need these values when you deploy the agent in your Kubernetes cluster.
4. You can keep the dialog box open in Soda Cloud, or close it. ## Deploy a Soda Agent in a Kubernetes cluster What follows are detailed deployment instructions according to the type of environment in which you create a cluster to deploy an agent. The high-level steps to complete the deployment remain the same regardless of environment. 1. (Optional) Familiarize yourself with [basic Soda, Kubernetes, and Helm concepts](/soda-v3/learning-resources/basics). 2. Install, or confirm the installation of, a few required command-line tools. 3. Create a new Kubernetes cluster in your environment, or identify an existing cluster you can use to deploy a Soda Agent. 4. Deploy the Soda Agent in the cluster. 5. Verify the existence of your new Soda Agent in your Soda Cloud account. ### Compatibility Soda supports Kubernetes cluster version 1.21 or greater. You can deploy a Soda Agent to connect with the following data sources: | | | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |
  • Amazon Athena
  • Amazon Redshift
  • Azure Synapse
  • ClickHouse
  • Databricks SQL
  • Denodo
  • Dremio
  • DuckDB
  • GCP BigQuery
  • Google CloudSQL
  • IBM DB2
|
  • MotherDuck
  • MS SQL Server1
  • MS Fabric1
  • MySQL
  • OracleDB
  • PostgreSQL
  • Presto
  • Snowflake
  • Trino
  • Vertica
| 1 MS SQL Server/MS Fabric with Windows Authentication does not work with Soda Agent out-of-the-box. {% tabs %} {% tab title="Kubernetes cluster" %} These deployment instructions offer generic guidance for deploying a Soda Agent in a Kubernetes cluster. [Prerequisites](/soda-v3/quick-start-sip/deploy#prerequisites)\ [System requirements](#system-requirements)\ [Deploy an agent](#deploy-an-agent)\ [Deploy using CLI only](#deploy-using-cli-only)\ [Deploy using a values YAML file](#deploy-using-a-values-yaml-file)\ [About the `helm install` command](#about-the-helm-install-command)\ [Decommission the Soda Agent and cluster](#decomission-the-soda-agent-and-cluster)\ [Troubleshoot deployment](#troubleshoot-deployment) *** ### Prerequisites * You have created, or have access to an existing Kubernetes cluster into which you can deploy a Soda Agent. * You have installed v1.22 or v1.23 of [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). This is the command-line tool you use to run commands against Kubernetes clusters. If you have installed Docker Desktop, kubectl is included out-of-the-box. With Docker running, use the command `kubectl version --output=yaml` to check the version of an existing install. * You have installed [Helm](https://helm.sh/docs/intro/install/). This is the package manager for Kubernetes which you will use to deploy the Soda Agent Helm chart. Run `helm version` to check the version of an existing install. ### System requirements Kubernetes cluster size and capacity: 2 CPU and 2GB of RAM. In general, this is sufficient to run up to six scans in parallel. Scan performance may vary according to the workload, or the number of scans running in parallel. To improve performance for larger workloads, consider fine-tuning the cluster size using the `resources` parameter for the `agent-orchestrator` and `soda.scanlauncher.resources` for the `scan-launcher`. Adding more resources to the `scan-launcher` can improve scan times by as much as 30%. Be aware, however, that allocating too many resources may be costly relative to the small benefit of improved scan times. To specify resources, add the following parameters to your `values.yml` file during deployment. Refer to Kubernetes documentation for [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for information on values to supply for `x`. ``` soda: agent: resources: limits: cpu: x memory: x requests: cpu: x memory: x scanlauncher: resources: limits: cpu: x memory: x requests: cpu: x memory: x ``` For reference, a Soda-hosted agent specifies resources as follows: ``` soda: agent: resources: limits: cpu: 250m memory: 375Mi requests: cpu: 250m memory: 375Mi ``` ### Deploy an agent The following table outlines the two ways you can install the Helm chart to deploy a Soda Agent in your cluster. | Method | Description | When to use | | ---------------------------------------------------------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [CLI only](#deploy-using-cli-only) | Install the Helm chart via CLI by providing values directly in the install command. | Use this as a straight-forward way of deploying an agent on a cluster in a secure or local environment. | | [Use a values YAML file](#deploy-using-a-values-yaml-file) | Install the Helm chart via CLI by providing values in a values YAML file. |

Use this as a way of deploying an agent on a cluster while keeping sensitive values secure.
- provide sensitive API key values in this local file
- store data source login credentials as environment variables in this local file or in an external secrets manager; Soda needs access to the credentials to be able to connect to your data source to run scans of your data. See: Soda Agent extras.

| #### **Deploy using CLI only** 1. (Optional) You have familarized yourself with [basic Soda, Kubernetes, and Helm concepts](/soda-v3/learning-resources/basics). 2. Add the Soda Agent Helm chart repository. ``` helm repo add soda-agent https://helm.soda.io/soda-agent/ ``` 3. Use the following comand to install the Helm chart to deploy a Soda Agent in your custer. Learn more about the [`helm install` command](#about-the-helm-install-command). * Replace the values of `soda.apikey.id` and `soda-apikey.secret` with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud account. By default, Soda uses [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) as part of the Soda Agent deployment. The Agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets. * Replace the value of `soda.agent.name` with a custom name for your agent, if you wish. * Specify the value for `soda.cloud.endpoint` according to your local region: `https://cloud.us.soda.io` for the United States, or `https://cloud.soda.io` for all else. * (Optional) Specify the format for log output: `raw` for plain text, or `json` for JSON format. * (Optional) Specify the level of log information you wish to see when deploying the agent: `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. ``` helm install soda-agent soda-agent/soda-agent \ --set soda.agent.name=myuniqueagent \ # Use https://cloud.us.soda.io for US region; use https://cloud.soda.io for EU region --set soda.cloud.endpoint=https://cloud.soda.io \ --set soda.apikey.id=*** \ --set soda.apikey.secret=**** \ --set soda.agent.logFormat=raw \ --set soda.agent.loglevel=ERROR \ --namespace soda-agent ``` The command-line produces output like the following message: ``` NAME: soda-agent LAST DEPLOYED: Thu Jun 16 15:03:10 2022 NAMESPACE: soda-agent STATUS: deployed REVISION: 1 ``` 4. (Optional) Validate the Soda Agent deployment by running the following command: ``` minikube kubectl -- describe pods ``` 5. In your Soda Cloud account, navigate to **your avatar** > **Agents**. Refresh the page to verify that you see the agent you just created in the list of Agents.\ \ Be aware that this may take several minutes to appear in your list of Soda Agents. Use the `describe pods` command in step 3 to check the status of the deployment. When `State: Running` and `Ready: True`, then you can refresh and see the agent in Soda Cloud. ``` ... Containers: soda-agent-orchestrator: Container ID: docker://081*33a7 Image: sodadata/agent-orchestrator:latest Image ID: docker-pullable://sodadata/agent-orchestrator@sha256:394e7c1**b5f Port: Host Port: State: Running Started: Thu, 16 Jun 2022 15:50:28 -0700 Ready: True ... ```
If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs. ``` kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f ``` #### **Deploy using a values YAML file** 1. (Optional) You have familarized yourself with [basic Soda, Kubernetes, and Helm concepts](/soda-v3/learning-resources/basics). 2. Create or navigate to an existing Kubernetes cluster in your environment in which you can deploy the Soda Agent helm chart. 3. Using a code editor, create a new YAML file called `values.yml`. 4. In that file, copy+paste the content below, replacing the following values: * `id` and `secret` with the values you copy+pasted from the **New Soda Agent** dialog box in your Soda Cloud account. By default, Soda uses [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) as part of the Soda Agent deployment. The Agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets. * Replace the value of `name` with a custom name for your agent, if you wish. * Specify the value for `endpoint` according to your local region: `https://cloud.us.soda.io` for the United States, or `https://cloud.soda.io` for all else. * (Optional) Specify the format for log output: `raw` for plain text, or `json` for JSON format. * (Optional) Specify the level of log information you wish to see when deploying the agent: `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. ``` soda: apikey: id: "***" secret: "***" agent: name: "myuniqueagent" logformat: "raw" loglevel: "ERROR" cloud: # Use https://cloud.us.soda.io for US region # Use https://cloud.soda.io for EU region endpoint: "https://cloud.soda.io" ``` 5. Save the file. Then, in the same directory in which the `values.yml` file exists, use the following command to install the Soda Agent helm chart. ``` helm install soda-agent soda-agent/soda-agent \ --values values.yml \ --namespace soda-agent ``` 6. (Optional) Validate the Soda Agent deployment by running the following command: ``` minikube kubectl -- describe pods ``` 7. In your Soda Cloud account, navigate to **your avatar** > **Agents**. Refresh the page to verify that you see the agent you just created in the list of Agents.\ \ Be aware that this may take several minutes to appear in your list of Soda Agents. Use the `describe pods` command in step three to check the status of the deployment. When `State: Running` and `Ready: True`, then you can refresh and see the agent in Soda Cloud. ``` ... Containers: soda-agent-orchestrator: Container ID: docker://081*33a7 Image: sodadata/agent-orchestrator:latest Image ID: docker-pullable://sodadata/agent-orchestrator@sha256:394e7c1**b5f Port: Host Port: State: Running Started: Thu, 16 Jun 2022 15:50:28 -0700 Ready: True ... ```
If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs. ``` kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f ``` If you use private key authentication with a Soda Agent, refer to [Soda Agent extras](/soda-v3/quick-start-sip/extras). ### About the `helm install` command ``` helm install soda-agent soda-agent/soda-agent \ --set soda.agent.name=myuniqueagent \ --set soda.apikey.id=*** \ --set soda.apikey.secret=**** \ --namespace soda-agent ``` | Command part | Description | | ----------------------------- | ------------------------------------------------- | | `helm install` | the action helm is to take | | `soda-agent` (the first one) | a release named soda-agent on your cluster | | `soda-agent` (the second one) | the name of the helm repo you installed | | `soda-agent` (the third one) | the name of the helm chart that is the Soda Agent | The `--set` options either override or set some of the values defined in and used by the Helm chart. You can override these values with the `--set` files as this command does, or you can specify the override values using a values.yml file. | Parameter key | Parameter value, description | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--set soda.agent.name` | A unique name for your Soda Agent. Choose any name you wish, as long as it is unique in your Soda Cloud account. | | `--set soda.apikey.id` | With the apikey.secret, this connects the Soda Agent to your Soda Cloud account. Use the value you copied from the dialog box in Soda Cloud when adding a new agent. You can use a values.yml file to pass this value to the cluster instead of exposing it here. | | `--set soda.apikey.secret` | With the apikey.id, this connects the Soda Agent to your Soda Cloud account. Use the value you copied from the dialog box in Soda Cloud when adding a new agent. You can use a values.yml file to pass this value to the cluster instead of exposing it here. | | `--set soda.agent.logFormat` | (Optional) Specify the format for log output: `raw` for plain text, or `json` for JSON format. | | `--set soda.agent.loglevel` | (Optional) Specify the leve of log information you wish to see when deploying the agent: `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. | | `--namespace soda-agent` | Use the namespace value to identify the namespace in which to deploy the agent. | ### Decomission the Soda Agent and cluster 1. Uninstall the Soda Agent in the cluster. ``` helm uninstall soda-agent -n soda-agent ``` 2. Delete the cluster. ``` minikube delete ``` ``` 💀 Removed all traces of the "minikube" cluster. ``` ### Troubleshoot deployment **Problem:** After setting up a cluster and deploying the agent, you are unable to see the agent running in Soda Cloud. **Solution:** The value you specify for the `soda-cloud-enpoint` must correspond with the region you selected when you signed up for a Soda Cloud account: * Use`https://cloud.us.soda.io` for the United States * Use `https://cloud.soda.io` for all else **Problem:** You need to define the outgoing port and IP address with which a self-hosted Soda Agent can communicate with Soda Cloud. Soda Agent does not require setting any *inbound* rules as it only polls Soda Cloud looking for instruction, which requires only *outbound* communication. When Soda Cloud must deliver instructions, the Soda Agent opens a bidirectional channel. **Solution:** Use port `443` and passlist the fully-qualified domain names for Soda Cloud: * `cloud.us.soda.io` for Soda Cloud account created in the US region\ OR * `cloud.soda.io` for Soda Cloud account created in the EU region\ AND * `collect.soda.io` {% endtab %} {% tab title="Amazon EKS" %} These deployment instructions offer guidance for setting up an Amazon Elastic Kubernetes Service (EKS) cluster and deploying a Soda Agent in it. [Prerequisites](#prerequisites-1)\ [System requirements](#system-requirements-1)\ [Deploy an agent](#deploy-an-agent-1)\ [Deploy using CLI only](#deploy-using-cli-only-1)\ [Deploy using a values YAML file](#deploy-using-a-values-yaml-file-1)\ [(Optional) Connect via AWS PrivateLink](#optional-connect-via-aws-privatelink)\ [About the `helm install` command](#about-the-helm-install-command-1)\ [Decommission the Soda Agent and the EKS cluster](#decommission-the-soda-agent-and-the-eks-cluster)\ [Troubleshoot deployment](#troubleshoot-deployment-1) *** ### Prerequisites * You have an AWS account and the necessary permissions to enable you to create, or gain access to an EKS cluster in your region. * You have installed v1.22 or v1.23 of [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). This is the command-line tool you use to run commands against Kubernetes clusters. If you have installed Docker Desktop, kubectl is included out-of-the-box. Run `kubectl version --output=yaml` to check the version of an existing install. * You have installed [Helm](https://helm.sh/docs/intro/install/). This is the package manager for Kubernetes which you will use to deploy the Soda Agent Helm chart. Run `helm version` to check the version of an existing install. ### System requirements Kubernetes cluster size and capacity: 2 CPU and 2GB of RAM. In general, this is sufficient to run up to six scans in parallel. Scan performance may vary according to the workload, or the number of scans running in parallel. To improve performance for larger workloads, consider: * fine-tuning the cluster size using the `resources` parameter for the `agent-orchestrator` and `soda.scanlauncher.resources` for the `scan-launcher`. Adding more resources to the `scan-launcher` can improve scan times by as much as 30%. * adding more nodes to the node group; see AWS documentation for [Scaling Managed Nodegroups](https://eksctl.io/usage/nodegroup-managed/#scaling-managed-nodegroups). * adding a cluster auto-scaler to your Kubernetes cluster; see AWS documentation for [Autoscaling](https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html) Be aware, however, that allocating too many resources may be costly relative to the small benefit of improved scan times. To specify resources, add the following parameters to your `values.yml` file during deployment. Refer to Kubernetes documentation for [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for information on values to supply for `x`. ``` soda: agent: resources: limits: cpu: x memory: x requests: cpu: x memory: x scanlauncher: resources: limits: cpu: x memory: x requests: cpu: x memory: x ``` For reference, a Soda-hosted agent specifies resources as follows: ``` soda: agent: resources: limits: cpu: 250m memory: 375Mi requests: cpu: 250m memory: 375Mi ``` ### Deploy an agent The following table outlines the two ways you can install the Helm chart to deploy a Soda Agent in your cluster. | Method | Description | When to use | | ------------------------------------------------------------ | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [CLI only](#deploy-using-cli-only-1) | Install the Helm chart via CLI by providing values directly in the install command. | Use this as a straight-forward way of deploying an agent on a cluster. | | [Use a values YAML file](#deploy-using-a-values-yaml-file-1) | Install the Helm chart via CLI by providing values in a values YAML file. |

Use this as a way of deploying an agent on a cluster while keeping sensitive values secure.
- provide sensitive API key values in this local file
- store data source login credentials as environment variables in this local file or in an external secrets manager; Soda needs access to the credentials to be able to connect to your data source to run scans of your data. See: Soda Agent extras.

| #### **Deploy using CLI only** 1. (Optional) You have familarized yourself with [basic Soda, Kubernetes, and Helm concepts](/soda-v3/learning-resources/basics). 2. (Optional) If you wish, you can establish an [AWS PrivateLink](https://aws.amazon.com/privatelink/) to provide private connectivity with Soda Cloud. Refer to [Connect via AWS PrivateLink](#optional-connect-via-aws-privatelink) before deploying an agent. 3. (Optional) If you are deploying to an existing Virtual Private Cloud (VPC), consider supplying public or private subnets with your deployment. Consult the eksctl documentation to [Use existing VPC](https://eksctl.io/usage/vpc-configuration/#use-existing-vpc-other-custom-configuration). 4. Create or navigate to an existing Kubernetes cluster in your environment in which you can deploy the Soda Agent helm chart. Best practices advises [creating a managed node group](https://docs.aws.amazon.com/eks/latest/userguide/create-managed-node-group.html) into which you can deploy the agent. 5. Use Helm to add the Soda Agent Helm chart repository. ``` helm repo add soda-agent https://helm.soda.io/soda-agent/ ``` 6. Use the following command to install the Helm chart which deploys a Soda Agent in your custer. * Replace the values of `soda.apikey.id` and `soda-apikey.secret` with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud. By default, Soda uses [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) as part of the Soda Agent deployment. The agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets. * Replace the value of `soda.agent.name` with a custom name for your agent, if you wish. * Specify the value for `soda.cloud.endpoint` according to your local region: `https://cloud.us.soda.io` for the United States, or `https://cloud.soda.io` for all else. * (Optional) Specify the format for log output: `raw` for plain text, or `json` for JSON format. * (Optional) Specify the level of log information you wish to see when deploying the agent: `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. * Read more [about the `helm install` command](https://docs.soda.io/soda-agent/deploy.html#about-the-helm-install-command-1). ``` helm install soda-agent soda-agent/soda-agent \ --set soda.agent.name=myuniqueagent \ # Use https://cloud.us.soda.io for US region; use https://cloud.soda.io for EU region --set soda.cloud.endpoint=https://cloud.soda.io \ --set soda.apikey.id=*** \ --set soda.apikey.secret=**** \ --set soda.agent.logFormat=raw \ --set soda.agent.loglevel=ERROR \ --namespace soda-agent ``` The command-line produces output like the following message: ``` NAME: soda-agent LAST DEPLOYED: Thu Jun 16 10:12:47 2022 NAMESPACE: soda-agent STATUS: deployed REVISION: 1 ``` 7. (Optional) Validate the Soda Agent deployment by running the following command: ``` kubectl describe pods ``` 8. In your Soda Cloud account, navigate to **your avatar** > **Agents**. Refresh the page to verify that you see the agent you just created in the list of Agents.\ \ Be aware that this may take several minutes to appear in your list of Soda Agents. Use the `describe pods` command in step 3 to check the status of the deployment. When `State: Running` and `Ready: True`, then you can refresh and see the agent in Soda Cloud. ``` ... Containers: soda-agent-orchestrator: Container ID: docker://081*33a7 Image: sodadata/agent-orchestrator:latest Image ID: docker-pullable://sodadata/agent-orchestrator@sha256:394e7c1**b5f Port: Host Port: State: Running Started: Thu, 16 Jun 2022 15:50:28 -0700 Ready: True ... ```
If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs. ``` kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f ``` #### **Deploy using a values YAML file** 1. (Optional) You have familarized yourself with [basic Soda, Kubernetes, and Helm concepts](/soda-v3/learning-resources/basics). 2. (Optional) If you wish, you can establish an [AWS PrivateLink](https://aws.amazon.com/privatelink/) to provide private connectivity with Soda Cloud. Refer to [Connect via AWS PrivateLink](#optional-connect-via-aws-privatelink) before deploying an agent. 3. (Optional) If you are deploying to an existing Virtual Private Cloud (VPC), consider supplying public or private subnets with your deployment. Consult the eksctl documentation to [Use existing VPC](https://eksctl.io/usage/vpc-configuration/#use-existing-vpc-other-custom-configuration). 4. Create or navigate to an existing Kubernetes cluster in your environment in which you can deploy the Soda Agent helm chart. Best practices advises [creating a managed node group](https://docs.aws.amazon.com/eks/latest/userguide/create-managed-node-group.html) into which you can deploy the agent. 5. Using a code editor, create a new YAML file called `values.yml`. 6. To that file, copy+paste the content below, replacing the following values: * `id` and `secret` with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud account. By default, Soda uses [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) as part of the Soda Agent deployment. The agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets. * Replace the value of `name` with a custom name for your agent, if you wish. * Specify the value for `endpoint` according to your local region: `https://cloud.us.soda.io` for the United States, or `https://cloud.soda.io` for all else. * (Optional) Specify the format for log output: `raw` for plain text, or `json` for JSON format. * (Optional) Specify the level of log information you wish to see when deploying the agent: `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. ``` soda: apikey: id: "***" secret: "***" agent: name: "myuniqueagent" logformat: "raw" loglevel: "ERROR" cloud: # Use https://cloud.us.soda.io for US region; use https://cloud.soda.io for EU region endpoint: "https://cloud.soda.io" ``` 7. Save the file. Then, in the same directory in which the `values.yml` file exists, use the following command to install the Soda Agent helm chart. ``` helm install soda-agent soda-agent/soda-agent \ --values values.yml \ --namespace soda-agent ``` 8. (Optional) Validate the Soda Agent deployment by running the following command: ``` kubectl describe pods -n soda-agent ``` 9. In your Soda Cloud account, navigate to **your avatar** > **Agents**. Refresh the page to verify that you see the agent you just created in the list of Agents.\ \ Be aware that this may take several minutes to appear in your list of Soda Agents. Use the `describe pods` command in step four to check the status of the deployment. When `State: Running` and `Ready: True`, then you can refresh and see the agent in Soda Cloud. ``` ... Containers: soda-agent-orchestrator: Container ID: docker://081*33a7 Image: sodadata/agent-orchestrator:latest Image ID: docker-pullable://sodadata/agent-orchestrator@sha256:394e7c1**b5f Port: Host Port: State: Running Started: Thu, 16 Jun 2022 15:50:28 -0700 Ready: True ... ```
If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs. ``` kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f ``` ### (Optional) Connect via AWS PrivateLink If you use AWS services for your infrastructure and you have deployed or will deploy a Soda Agent in an EKS cluster, you can use an [AWS PrivateLink](https://aws.amazon.com/privatelink/) to provide private connectivity with Soda Cloud. 1. Log in to your AWS console and navigate to your **VPC dashboard**. 2. Follow the AWS documentation to [Connect to an endpoint service as the service customer](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html).\ For security reasons, Soda does not publish its **Service name**. Email with your **AWS account ID** to request the PrivateLink service name. Refer to [AWS documentation](https://docs.aws.amazon.com/signin/latest/userguide/console_account-alias.html) for instructions on how to obtain your account ID. 3. After creating the endpoint, return to the **VPC dashboard**. When the status of the endpoint becomes **Available**, the PrivateLink is ready to use. Be aware that this make take more than 10 minutes. 4. Deploy a Soda Agent to your AWS EKS cluster, or, if you have already deployed one, restart your Soda Agent to begin sending data to Soda Cloud via the PrivateLink. ``` kubectl -n soda-agent rollout restart deploy ``` 5. After you have started the agent and validated that it is running, log into your Soda Cloud account, then navigate to **your avatar** > **Agents**. Refresh the page to verify that you see the agent you just created in the list of Agents.
If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs. ``` kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f ``` ### About the `helm install` command ``` helm install soda-agent soda-agent/soda-agent \ --set soda.agent.name=myuniqueagent \ --set soda.apikey.id=*** \ --set soda.apikey.secret=**** \ --namespace soda-agent ``` | Command part | Description | | ----------------------------- | --------------------------------------------- | | `helm install` | the action helm is to take | | `soda-agent` (the first one) | a release named soda-agent on your cluster | | `soda-agent` (the second one) | the name of the helm repo you installed | | `soda-agent` (the third one) | the name of the Helm chart for the Soda Agent | The `--set` options either override or set some of the values defined in and used by the Helm chart. You can override these values with the `--set` files as this command does, or you can specify the override values using a values.yml file. | Parameter key | Parameter value, description | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--set soda.agent.name` | A unique name for your Soda Agent. Choose any name you wish, as long as it is unique in your Soda Cloud account. | | `--set soda.apikey.id` | With the apikey.secret, this connects the Soda Agent to your Soda Cloud account. Use the value you copied from the dialog box in Soda Cloud when adding a new agent. You can use a values.yml file to pass this value to the cluster instead of exposing it here. | | `--set soda.apikey.secret` | With the apikey.id, this connects the Soda Agent to your Soda Cloud account. Use the value you copied from the dialog box in Soda Cloud when adding a new agent. You can use a values.yml file to pass this value to the cluster instead of exposing it here. | | `--set soda.agent.logFormat` | (Optional) Specify the format for log output: `raw` for plain text, or `json` for JSON format. | | `--set soda.agent.loglevel` | (Optional) Specify the leve of log information you wish to see when deploying the agent: `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. | | `--namespace soda-agent` | Use the namespace value to identify the namespace in which to deploy the agent. | ### Decommission the Soda Agent and the EKS cluster 1. Uninstall the Soda Agent in the cluster. ``` helm uninstall soda-agent -n soda-agent ``` 2. Delete the EKS cluster itself. ``` eksctl delete cluster --name soda-agent ``` 3. (Optional) Access your [CloudFormation console](https://eu-central-1.console.aws.amazon.com/cloudformation/home), then click **Stacks** to view the status of your decommissioned cluster.\ If you do not see your Stack, use the region drop-down menu at upper-right to select the region in which you created the cluster. ### Troubleshoot deployment **Problem:** After setting up a cluster and deploying the agent, you are unable to see the agent running in Soda Cloud. **Solution:** The value you specify for the `soda-cloud-enpoint` must correspond with the region you selected when you signed up for a Soda Cloud account: * Use`https://cloud.us.soda.io` for the United States * Use `https://cloud.soda.io` for all else **Problem:** You need to define the outgoing port and IP address with which a self-hosted Soda Agent can communicate with Soda Cloud. Soda Agent does not require setting any *inbound* rules as it only polls Soda Cloud looking for instruction, which requires only *outbound* communication. When Soda Cloud must deliver instructions, the Soda Agent opens a bidirectional channel. **Solution:** Use port `443` and passlist the fully-qualified domain names for Soda Cloud: * `cloud.us.soda.io` for Soda Cloud account created in the US region\ OR * `cloud.soda.io` for Soda Cloud account created in the EU region\ AND * `collect.soda.io`
**Problem:** `UnauthorizedOperation: You are not authorized to perform this operation.` **Solution:** This error indicates that your user profile is not authorized to create the cluster. Contact your AWS Administrator to request the appropriate permissions. {% endtab %} {% tab title="Azure AKS" %} These deployment instructions offer guidance for setting up an Azure Kubernetes Service (AKS) cluster and deploying a Soda Agent in it. [Prerequisites](#prerequisites-2)\ [System requirements](#system-requirements-2)\ [Deploy an agent](#deploy-an-agent-2)\ [Deploy using CLI only](#deploy-using-cli-only-2)\ [Deploy using a values YAML file](#deploy-using-a-values-yaml-file-2)\ [About the `helm install` command](#about-the-helm-install-command-2)\ [Decommission the Soda Agent and the AKS cluster](#decommission-the-soda-agent-and-the-aks-cluster)\ [Troubleshoot deployment](#troubleshoot-deployment-2) *** ### Prerequisites * You have an Azure account and the necessary permissions to enable you to create, or gain access to an existing AKS cluster in your region. Consult the [Azure access control documentation](https://learn.microsoft.com/en-us/azure/role-based-access-control/overview) for details. * You have installed the [Azure CLI tool](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). This is the command-line tool you need to access your Azure account from the command-line. Run `az --version` to check the version of an existing install. Consult the [Azure Command-Line Interface documentation](https://learn.microsoft.com/en-us/cli/azure/) for details. * You have logged in to your Azure account. Run `az login` to open a browser and log in to your account. * You have installed v1.22 or v1.23 of [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). This is the command-line tool you use to run commands against Kubernetes clusters. If you have already installed the Azure CLI tool, you can install kubectl using the following command: `az aks install-cli`.\ Run `kubectl version --output=yaml` to check the version of an existing install. * You have installed [Helm](https://helm.sh/docs/intro/install/). This is the package manager for Kubernetes which you will use to deploy the Soda Agent Helm chart. Run `helm version` to check the version of an existing install. ### System requirements Kubernetes cluster size and capacity: 2 CPU and 2GB of RAM. In general, this is sufficient to run up to six scans in parallel. Scan performance may vary according to the workload, or the number of scans running in parallel. To improve performance for larger workloads, consider fine-tuning the cluster size using the `resources` parameter for the `agent-orchestrator` and `soda.scanlauncher.resources` for the `scan-launcher`. Adding more resources to the `scan-launcher` can improve scan times by as much as 30%. Be aware that allocating too many resources may be costly relative to the small benefit of improved scan times. To specify resources, add the following parameters to your `values.yml` file during deployment. Refer to Kubernetes documentation for [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for information on values to supply for `x`. ``` soda: agent: resources: limits: cpu: x memory: x requests: cpu: x memory: x scanlauncher: resources: limits: cpu: x memory: x requests: cpu: x memory: x ``` For reference, a Soda-hosted agent specifies resources as follows: ``` soda: agent: resources: limits: cpu: 250m memory: 375Mi requests: cpu: 250m memory: 375Mi ``` ### Deploy an agent The following table outlines the ways you can install the Helm chart to deploy a Soda Agent in your cluster. | Method | Description | When to use | | ------------------------------------------------------------ | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [CLI only - regular cluster](#deploy-using-cli-only-2) | Install the Helm chart via CLI by providing values directly in the install command. | Use this as a straight-forward way of deploying an agent on a cluster. | | [Use a values YAML file](#deploy-using-a-values-yaml-file-2) | Install the Helm chart via CLI by providing values in a values YAML file. |

Use this as a way of deploying an agent on a cluster while keeping sensitive values secure.
- provide sensitive API key values in this local file or in an external secrets manager
- store data source login credentials as environment variables in this local file; Soda needs access to the credentials to be able to connect to your data source to run scans of your data. See: Soda Agent extras.

| #### **Deploy using CLI only** 1. (Optional) You have familiarized yourself with [basic Soda, Kubernetes, and Helm concepts](/soda-v3/learning-resources/basics). 2. Create or navigate to an existing Kubernetes cluster in your environment in which you can deploy the Soda Agent helm chart. 3. Use Helm to add the Soda Agent Helm chart repository. ``` helm repo add soda-agent https://helm.soda.io/soda-agent/ ``` 4. Use the following command to install the Helm chart which deploys a Soda Agent in your cluster. (Learn more about the [`helm install` command](/soda-v3/quick-start-sip/deploy#about-the-helm-install-command-2).) * Replace the values of `soda.apikey.id` and `soda-apikey.secret` with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud. By default, Soda uses [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) as part of the Soda Agent deployment. The agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets. * Replace the value of `soda.agent.name` with a custom name for your agent, if you wish. * Specify the value for `soda.cloud.endpoint` according to your local region: `https://cloud.us.soda.io` for the United States, or `https://cloud.soda.io` for all else. * (Optional) Specify the format for log output: `raw` for plain text, or `json` for JSON format. * (Optional) Specify the level of log information you wish to see when deploying the agent: `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. ``` helm install soda-agent soda-agent/soda-agent \ --set soda.agent.name=myuniqueagent \ # Use https://cloud.us.soda.io for US region; use https://cloud.soda.io for EU region --set soda.cloud.endpoint=https://cloud.soda.io \ --set soda.apikey.id=*** \ --set soda.apikey.secret=**** \ --set soda.agent.logFormat=raw \ --set soda.agent.loglevel=ERROR \ --namespace soda-agent ``` The command-line produces output like the following message: ``` NAME: soda-agent LAST DEPLOYED: Mon Nov 21 16:29:38 2022 NAMESPACE: soda-agent STATUS: deployed REVISION: 1 ``` 5. (Optional) Validate the Soda Agent deployment by running the following command: ``` kubectl get pods -n soda-agent ``` ``` NAME READY STATUS RESTARTS AGE soda-agent-orchestrator-ffd74c76-5g7tl 1/1 Running 0 32s ``` 6. In your Soda Cloud account, navigate to **your avatar** > **Agents**. Refresh the page to verify that you see the agent you just created in the list of Agents.\ \ Be aware that this may take several minutes to appear in your list of Soda Agents.
If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs. ``` kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f ``` #### **Deploy using a values YAML file** 1. (Optional) You have familiarized yourself with [basic Soda, Kubernetes, and Helm concepts](/soda-v3/learning-resources/basics). 2. Create or navigate to an existing Kubernetes cluster in your environment in which you can deploy the Soda Agent helm chart. 3. Use Helm to add the Soda Agent Helm chart repository. ``` helm repo add soda-agent https://helm.soda.io/soda-agent/ ``` 4. Using a code editor, create a new YAML file called `values.yml`. 5. To that file, copy+paste the content below, replacing the following values: * `id` and `secret` with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud account. By default, Soda uses [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) as part of the Soda Agent deployment. The agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets. * Replace the value of `name` with a custom name for your agent, if you wish. * Specify the value for `endpoint` according to your local region: `https://cloud.us.soda.io` for the United States, or `https://cloud.soda.io` for all else. * (Optional) Specify the format for log output: `raw` for plain text, or `json` for JSON format. * (Optional) Specify the level of log information you wish to see when deploying the agent: `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. ``` soda: apikey: id: "***" secret: "***" agent: name: "myuniqueagent" logformat: "raw" loglevel: "ERROR" cloud: # Use https://cloud.us.soda.io for US region; use https://cloud.soda.io for EU region endpoint: "https://cloud.soda.io" ``` 6. Save the file. Then, create a namespace for the agent. ``` kubectl create ns soda-agent ``` ``` namespace/soda-agent created ``` 7. In the same directory in which the `values.yml` file exists, use the following command to install the Soda Agent helm chart. ``` helm install soda-agent soda-agent/soda-agent \ --values values.yml \ --namespace soda-agent ``` 8. (Optional) Validate the Soda Agent deployment by running the following command: ``` kubectl describe pods -n soda-agent ``` 9. In your Soda Cloud account, navigate to **your avatar** > **Agents**. Refresh the page to verify that you see the agent you just created in the list of Agents.
If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs. ``` kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f ``` ### About the `helm install` command ``` helm install soda-agent soda-agent/soda-agent \ --set soda.agent.name=myuniqueagent \ --set soda.apikey.id=*** \ --set soda.apikey.secret=**** \ --namespace soda-agent ``` | Command part | Description | | ----------------------------- | --------------------------------------------- | | `helm install` | the action helm is to take | | `soda-agent` (the first one) | a release named soda-agent on your cluster | | `soda-agent` (the second one) | the name of the helm repo you installed | | `soda-agent` (the third one) | the name of the Helm chart for the Soda Agent | The `--set` options either override or set some of the values defined in and used by the Helm chart. You can override these values with the `--set` files as this command does, or you can specify the override values using a values.yml file.
### Decommission the Soda Agent and the AKS cluster 1. Delete everything in the namespace which you created for the Soda Agent. ``` kubectl delete ns soda-agent ``` 2. Delete the cluster. Be patient; this task may take some time to complete. ``` az aks delete --resource-group SodaAgent --name soda-agent-cli-test --yes ``` ### Troubleshoot deployment **Problem:** After setting up a cluster and deploying the agent, you are unable to see the agent running in Soda Cloud. **Solution:** The value you specify for the `soda-cloud-enpoint` must correspond with the region you selected when you signed up for a Soda Cloud account: * Use`https://cloud.us.soda.io` for the United States * Use `https://cloud.soda.io` for all else **Problem:** You need to define the outgoing port and IP address with which a self-hosted Soda Agent can communicate with Soda Cloud. Soda Agent does not require setting any *inbound* rules as it only polls Soda Cloud looking for instruction, which requires only *outbound* communication. When Soda Cloud must deliver instructions, the Soda Agent opens a bidirectional channel. **Solution:** Use port `443` and passlist the fully-qualified domain names for Soda Cloud: * `cloud.us.soda.io` for Soda Cloud account created in the US region\ OR * `cloud.soda.io` for Soda Cloud account created in the EU region\ AND * `collect.soda.io` **Problem:** When you attempt to create a cluster, you get an error that reads, `An RSA key file or key value must be supplied to SSH Key Value. You can use --generate-ssh-keys to let CLI generate one for you`. **Solution:** Run the same command to create a cluster but include an extra line at the end to generate RSA keys. ``` az aks create \ > --resource-group SodaAgent \ > --name SodaAgentCluster \ > --node-count 1 \ > --generate-ssh-keys ``` {% endtab %} {% tab title="Google GKE" %} These deployment instructions offer guidance for setting up a Google Kubernetes Engine (GKE) cluster and deploying a Soda Agent in it. [Prerequisites](#prerequisites-3)\ [System requirements](#system-requirements-3)\ [Deploy an agent](#deploy-an-agent-3)\ [Deploy using CLI only](#deploy-using-cli-only-3)\ [Deploy using a values YAML file](#deploy-using-a-values-yaml-file-3)\ [About the `helm install` command](#about-the-helm-install-command-3)\ [Decommission the Soda Agent and cluster](#decomission-the-soda-agent-and-cluster)\ [Troubleshoot deployment](#troubleshoot-deployment-3) *** ### Prerequisites * You have a Google Cloud Platform (GCP) account and the necessary permissions to enable you to create, or gain access to an existing Google Kubernetes Engine (GKE) cluster in your region. * You have installed the [gcloud CLI tool](https://cloud.google.com/sdk/docs/install). Use the command `glcoud version` to verify the version of an existing install. * If you have already installed the gcloud CLI, use the following commands to login and verify your configuration settings, respectively: `gcloud auth login` `gcloud config list` * If you are installing the gcloud CLI for the first time, be sure to complete [all the steps](https://cloud.google.com/sdk/docs/install) in the installation to properly install and configure the setup. * Consider using the following command to learn a few basic glcoud commands: `gcloud cheat-sheet`. * You have installed v1.22 or v1.23 of [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl). This is the command-line tool you use to run commands against Kubernetes clusters. If you have installed Docker Desktop, kubectl is included out-of-the-box. With Docker running, use the command `kubectl version --output=yaml` to check the version of an existing install. * You have installed [Helm](https://helm.sh/docs/intro/install/). This is the package manager for Kubernetes which you will use to deploy the Soda Agent Helm chart. Run `helm version` to check the version of an existing install. ### System requirements Kubernetes cluster size and capacity: 2 CPU and 2GB of RAM. In general, this is sufficient to run up to six scans in parallel. Scan performance may vary according to the workload, or the number of scans running in parallel. To improve performance for larger workloads, consider fine-tuning the cluster size using the `resources` parameter for the `agent-orchestrator` and `soda.scanlauncher.resources` for the `scan-launcher`. Adding more resources to the `scan-launcher` can improve scan times by as much as 30%. Be aware, however, that allocating too many resources may be costly relative to the small benefit of improved scan times. To specify resources, add the following parameters to your `values.yml` file during deployment. Refer to Kubernetes documentation for [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for information on values to supply for `x`. ``` soda: agent: resources: limits: cpu: x memory: x requests: cpu: x memory: x scanlauncher: resources: limits: cpu: x memory: x requests: cpu: x memory: x ``` For reference, a Soda-hosted agent specifies resources as follows: ``` soda: agent: resources: limits: cpu: 250m memory: 375Mi requests: cpu: 250m memory: 375Mi ``` ### Deploy an agent The following table outlines the two ways you can install the Helm chart to deploy a Soda Agent in your cluster. | Method | Description | When to use | | ------------------------------------------------------------ | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [CLI only](#deploy-using-cli-only-3) | Install the Helm chart via CLI by providing values directly in the install command. | Use this as a straight-forward way of deploying an agent on a cluster in a secure or local environment. | | [Use a values YAML file](#deploy-using-a-values-yaml-file-3) | Install the Helm chart via CLI by providing values in a values YAML file. |

Use this as a way of deploying an agent on a cluster while keeping sensitive values secure.
- provide sensitive API key values in this local file
- store data source login credentials as environment variables in this local file or in an external secrets manager; Soda needs access to the credentials to be able to connect to your data source to run scans of your data. See: Soda Agent extras.

| #### **Deploy using CLI only** 1. (Optional) You have familiarized yourself with [basic Soda, Kubernetes, and Helm concepts](/soda-v3/learning-resources/basics). 2. Create or navigate to an existing Kubernetes cluster in your environment in which you can deploy the Soda Agent helm chart. 3. Add the Soda Agent Helm chart repository. ``` helm repo add soda-agent https://helm.soda.io/soda-agent/ ``` 4. Use the following command to install the Helm chart to deploy a Soda Agent in your custer. (Learn more about the [`helm install` command](#about-the-helm-install-command-3).) * Replace the values of `soda.apikey.id` and `soda-apikey.secret` with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud account. By default, Soda uses [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) as part of the Soda Agent deployment. The agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets. * Replace the value of `soda.agent.name` with a custom name for your agent, if you wish. * Specify the value for `soda.cloud.endpoint` according to your local region: `https://cloud.us.soda.io` for the United States, or `https://cloud.soda.io` for all else. * (Optional) Specify the format for log output: `raw` for plain text, or `json` for JSON format. * (Optional) Specify the level of log information you wish to see when deploying the agent: `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. ``` helm install soda-agent soda-agent/soda-agent \ --set soda.agent.name=myuniqueagent \ # Use https://cloud.us.soda.io for US region; use https://cloud.soda.io for EU region --set soda.cloud.endpoint=https://cloud.soda.io \ --set soda.apikey.id=*** \ --set soda.apikey.secret=*** \ --set soda.agent.logFormat=raw \ --set soda.agent.loglevel=ERROR \ --namespace soda-agent ``` The command-line produces output like the following message: ``` NAME: soda-agent LAST DEPLOYED: Wed Dec 14 11:45:13 2022 NAMESPACE: soda-agent STATUS: deployed REVISION: 1 ``` 5. (Optional) Validate the Soda Agent deployment by running the following command: ``` kubectl describe pods ``` 6. In your Soda Cloud account, navigate to **your avatar** > **Agents**. Refresh the page to verify that you see the agent you just created in the list of Agents.\ \ Be aware that this may take several minutes to appear in your list of Soda Agents. Use the `describe pods` command in step three to check the status of the deployment. When `Status: Running`, then you can refresh and see the agent in Soda Cloud. ``` Name: soda-agent-orchestrator-66-snip Namespace: soda-agent Priority: 0 Service Account: soda-agent Node: Labels: agent.soda.io/component=orchestrator agent.soda.io/service=queue app.kubernetes.io/instance=soda-agent app.kubernetes.io/name=soda-agent pod-template-hash=669snip Annotations: seccomp.security.alpha.kubernetes.io/pod: runtime/default Status: Running ... ```
If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs. ``` kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f ``` #### **Deploy using a values YAML file** 1. (Optional) You have familiarized yourself with [basic Soda, Kubernetes, and Helm concepts](/soda-v3/learning-resources/basics). 2. Create or navigate to an existing Kubernetes cluster in your environment in which you can deploy the Soda Agent helm chart. 3. Using a code editor, create a new YAML file called `values.yml`. 4. In that file, copy+paste the content below, replacing the following values: * `id` and `secret` with the values you copy+pasted from the **New Soda Agent** dialog box in your Soda Cloud account. By default, Soda uses [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) as part of the Soda Agent deployment. The agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets. * Replace the value of `name` with a custom name for your agent, if you wish. * Specify the value for `endpoint` according to your local region: `https://cloud.us.soda.io` for the United States, or `https://cloud.soda.io` for all else. * (Optional) Specify the format for log output: `raw` for plain text, or `json` for JSON format. * (Optional) Specify the level of log information you wish to see when deploying the agent: `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. ``` soda: apikey: id: "***" secret: "***" agent: name: "myuniqueagent" logformat: "raw" loglevel: "ERROR" cloud: # Use https://cloud.us.soda.io for US region; use https://cloud.soda.io for EU region endpoint: "https://cloud.soda.io" ``` 5. Save the file. Then, in the same directory in which the `values.yml` file exists, use the following command to install the Soda Agent helm chart. ``` helm install soda-agent soda-agent/soda-agent \ --values values.yml \ --namespace soda-agent ``` 6. (Optional) Validate the Soda Agent deployment by running the following command: ``` kubectl describe pods ``` 7. In your Soda Cloud account, navigate to **your avatar** > **Agents**. Refresh the page to verify that you see the agent you just created in the list of Agents.\ \ Be aware that this may take several minutes to appear in your list of Soda Agents. Use the `describe pods` command in step four to check the status of the deployment. When `Status: Running`, then you can refresh and see the agent in Soda Cloud. ``` Name: soda-agent-orchestrator-66-snip Namespace: soda-agent Priority: 0 Service Account: soda-agent Node: Labels: agent.soda.io/component=orchestrator agent.soda.io/service=queue app.kubernetes.io/instance=soda-agent app.kubernetes.io/name=soda-agent pod-template-hash=669snip Annotations: seccomp.security.alpha.kubernetes.io/pod: runtime/default Status: Running ... ```
If you do no see the agent listed in Soda Cloud, use the following command to review status and investigate the logs. ``` kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent -f ``` ### About the `helm install` command ``` helm install soda-agent soda-agent/soda-agent \ --set soda.agent.name=myuniqueagent \ --set soda.apikey.id=*** \ --set soda.apikey.secret=**** \ --namespace soda-agent ``` | Command part | Description | | ----------------------------- | ------------------------------------------------- | | `helm install` | the action helm is to take | | `soda-agent` (the first one) | a release named soda-agent on your cluster | | `soda-agent` (the second one) | the name of the helm repo you installed | | `soda-agent` (the third one) | the name of the helm chart that is the Soda Agent | The `--set` options either override or set some of the values defined in and used by the Helm chart. You can override these values with the `--set` files as this command does, or you can specify the override values using a values.yml file. | Parameter key | Parameter value, description | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--set soda.agent.name` | A unique name for your Soda Agent. Choose any name you wish, as long as it is unique in your Soda Cloud account. | | `--set soda.apikey.id` | With the apikey.secret, this connects the Soda Agent to your Soda Cloud account. Use the value you copied from the dialog box in Soda Cloud when adding a new agent. You can use a values.yml file to pass this value to the cluster instead of exposing it here. | | `--set soda.apikey.secret` | With the apikey.id, this connects the Soda Agent to your Soda Cloud account. Use the value you copied from the dialog box in Soda Cloud when adding a new agent. You can use a values.yml file to pass this value to the cluster instead of exposing it here. | | `--set soda.agent.logFormat` | (Optional) Specify the format for log output: `raw` for plain text, or `json` for JSON format. | | `--set soda.agent.loglevel` | (Optional) Specify the leve of log information you wish to see when deploying the agent: `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE`. | | `--namespace soda-agent` | Use the namespace value to identify the namespace in which to deploy the agent. | ### Decommission the Soda Agent and cluster 1. Uninstall the Soda Agent in the cluster. ``` helm uninstall soda-agent -n soda-agent ``` 2. Delete the cluster. ``` gcloud container clusters delete soda-agent-gke ``` Refer to [Google Kubernetes Engine documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/deleting-a-cluster) for details. ### Troubleshoot deployment **Problem:** After setting up a cluster and deploying the agent, you are unable to see the agent running in Soda Cloud. **Solution:** The value you specify for the `soda-cloud-enpoint` must correspond with the region you selected when you signed up for a Soda Cloud account: * Use`https://cloud.us.soda.io` for the United States * Use `https://cloud.soda.io` for all else **Problem:** You need to define the outgoing port and IP address with which a self-hosted Soda Agent can communicate with Soda Cloud. Soda Agent does not require setting any *inbound* rules as it only polls Soda Cloud looking for instruction, which requires only *outbound* communication. When Soda Cloud must deliver instructions, the Soda Agent opens a bidirectional channel. **Solution:** Use port `443` and passlist the fully-qualified domain names for Soda Cloud: * `cloud.us.soda.io` for Soda Cloud account created in the US region\ OR * `cloud.soda.io` for Soda Cloud account created in the EU region\ AND * `collect.soda.io` {% endtab %} {% endtabs %} ## Add a new data source In your Soda Cloud account, navigate to **your avatar** > **Data Sources**. Click **New Data Source**, then follow the guided steps to create a new data source. Refer to the sections below for insight into the values to enter in the fields and editing panels in the guided steps. ### **1. Attributes** | Field or Label | Guidance | | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Data Source Label | Provide a unique identifier for the data source. Soda Cloud uses the label you provide to define the immutable name of the data source against which it runs the Default Scan. | | Default Scan Agent | Select the Soda-hosted agent, or the name of a Soda Agent that you have previously set up in your secure environment. This identifies the Soda Agent to which Soda Cloud must connect in order to run its scan. | | Check Schedule | Provide the scan frequency details Soda Cloud uses to execute scans according to your needs. If you wish, you can define the schedule as a cron expression. | | Starting At | Select the time of day to run the scan. The default value is midnight. | | Cron Expression | (Optional) Write your own [cron expression](https://en.wikipedia.org/wiki/Cron) to define the schedule Soda Cloud uses to run scans. | |

Anomaly Dashboard Scan Schedule
(Available in 2025)

| Provide the scan frequency details Soda Cloud uses to execute a daily scan to automatically detect anomalies for the anomaly dashboard. | ### 2. Connect In the editing panel, provide the connection configurations Soda Cloud needs to be able to access the data in the data source. Connection configurations are data source-specific and include values for things such as a database's host and access credentials. To more securely provide sensitive values such as usernames and passwords, use environment variables in a `values.yml` file when you deploy the Soda Agent. See [Use environment variables for data source connection credentials](/soda-v3/quick-start-sip/extras#use-environment-variables-to-store-data-source-connection-credentials) for details. Access the data source-specific connection configurations listed below to copy+paste the connection syntax into the editing panel, then adjust the values to correspond with your data source's details. Access connection configuration details in \[[Data source reference](/soda-v3/data-source-reference) section of Soda documentation. See also: [Use a file reference for a BigQuery data source connection](#use-a-file-reference-for-a-bigquery-data-source-connection) ### 3. Discover During its initial scan of your datasource, Soda Cloud discovers all the datasets the data source contains. It captures basic information about each dataset, including a dataset names and the columns each contains. In the editing panel, specify the datasets that Soda Cloud must include or exclude from this basic discovery activity. The default syntax in the editing panel instructs Soda to collect basic dataset information from all datasets in the data source *except* those with names that begin with `test_`. The `%` is a wildcard character. See [Add dataset discovery](/soda-v3/soda-cl-overview/profile#add-dataset-discovery) for more detail on profiling syntax. {% hint style="warning" %} *Known issue:* SodaCL does not support using variables in column profiling and dataset discovery configurations. {% endhint %} ```yaml discover datasets: datasets: - include % - exclude test_% ``` ### 4. Profile To gather more detailed profile information about datasets in your data source and automatically build an **anomaly dashboard** for data quality observability (preview, only), you can configure Soda Cloud to profile the columns in datasets. Profiling a dataset produces two tabs' worth of data in a dataset page: * In the **Columns** tab, you can see column profile information including details such as the calculated mean value of data in a column, the maximum and minimum values in a column, and the number of rows with missing data.
* In the **Anomalies** tab, you can access an out-of-the-box anomaly dashboard that uses the column profile information to automatically begin detecting anomalies in your data relative to the patterns the machine learning algorithm learns over the course of approximately five days. [Learn more](/soda-v3/collaborate/anomaly-dashboard) (**available in 2025**)
In the editing panel, provide details that Soda Cloud uses to determine which datasets to include or exclude when it profiles the columns in a dataset. The default syntax in the editing panel instructs Soda to profile every column of every dataset in this data source, and, superfluously, all datasets with names that begin with `prod`. The `%` is a wildcard character. See [Add column profiling](/soda-v3/soda-cl-overview/profile#add-column-profiling) for more detail on profiling syntax. Column profiling and automated anomaly detection can be resource-heavy, so carefully consider the datasets for which you truly need column profile information. Refer to [Compute consumption and cost considerations](/soda-v3/soda-cl-overview/profile#compute-consumption-and-cost-considerations) for more detail. ```yaml profile columns: columns: - "%.%" # Includes all your datasets - prod% # Includes all datasets that begin with 'prod' ``` ### **5. Check** When Soda Cloud automatically discovers the datasets in a data source, it prepares automated monitoring checks for each dataset. These checks detect anomalies and monitor schema evolution, corresponding to the SodaCL [anomaly detection](/soda-v3/sodacl-reference/anomaly-detection) and [schema](/soda-v3/sodacl-reference/schema) checks, respectively. (Note that if you have signed up for early access to [anomaly dashboards](/soda-v3/collaborate/anomaly-dashboard) for datasets, this **Check** tab is unavailable as Soda performs all automated monitoring automatically in the dashboards.) In the editing panel, specify the datasets that Soda Cloud must include or exclude when preparing automated monitoring checks. The default syntax in the editing panel indicates that Soda will add automated monitoring to all datasets in the data source *except* those with names that begin with `test_`. The `%` is a wildcard character. ``` automated monitoring: datasets: - include % - exclude test_% ``` ### **(5) 6. Assign Owner** This tab is the fifth step in the guided workflow if the **5. Check** tab is absent because you requested access to the anomaly dashboards feature. | Field or Label | Guidance | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Data Source Owner | The Data Source Owner maintains the connection details and settings for this data source and its Default Scan Definition. | | Default Dataset Owner | The Datasets Owner is the user who, by default, becomes the owner of each dataset the Default Scan discovers. Refer to [Manage roles and permissions in Soda Cloud](/soda-v3/collaborate/roles-global) to learn how to adjust the Dataset Owner of individual datasets. | ## Use a file reference for a BigQuery data source connection If you already store information about your data source in a JSON file in a secure location, you can configure your BigQuery data source connection details in Soda Cloud to refer to the JSON file for service account information. To do so, you must add two elements: * `volumes` and `volumeMounts` parameters in the `values.yml` file that your Soda Agent helm chart uses * the `account_info_json_path` in your data source connection configuration You, or an IT Admin in your organization, can add the following `scanlauncher` parameters to the existing `values.yml` that your Soda Agent uses for deployment and redeployment in your Kubernetes cluster. Refer to the Google GKE instructions above. ```yaml soda: scanlauncher: volumeMounts: - name: gcloud-credentials mountPath: /opt/soda/etc volumes: - name: gcloud-credentials secret: secretName: gcloud-credentials items: - key: serviceaccount.json path: serviceaccount.json ``` Use the following command to add the service account information to a Kubernetes secret that the Soda Agent consumes according to the configuration above. ```shell kubectl create secret -n gcloud-credentials --from-file=serviceaccount.json= ``` After you make both of these changes, you must redeploy the Soda Agent. Adjust the data source connection configuration to include the `account_info_json_path` configuration, as per the following example. ```yaml my_datasource_name: type: bigquery account_info_json_path: /opt/soda/etc/serviceaccount.json auth_scopes: - https://www.googleapis.com/auth/bigquery - https://www.googleapis.com/auth/cloud-platform - https://www.googleapis.com/auth/drive project_id: *** dataset: sodacore ``` ## Next 1. ~~Choose a flavor of Soda~~ 2. ~~Set up Soda: self-hosted agent~~ 3. [Write SodaCL checks](/soda-v3/soda-cl-overview) 4. Run scans and review results 5. Organize, alert, investigate > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Deploy a self-hosted Soda Agent from Soda's private container registry ## What has changed? As of July 2025, the container images required for the self-hosted Soda agent will be distributed using private registries, hosted by Soda. EU cloud customers will use the EU registry located at `registry.cloud.soda.io`. US cloud customers will use the US registry located at `registry.us.soda.io`. The images currently distributed through Docker Hub will stay available there. New releases will only be available in the Soda-hosted registries. Existing or new Soda cloud API keys can be used to authenticate to the Soda-hosted registries. Starting from version `1.2.0`, the `soda-agent` Helm chart supports working with Soda-hosted image registries. In order to enjoy the latest features Soda has to offer, please upgrade any self-hosted Soda agent you manage using one of the following guides. ## How-to's ### Registry access using your existing API key Follow the self-hosted Soda agent [upgrade](/soda-v3/quick-start-sip/upgrade#upgrade-a-self-hosted-soda-agent) or [redeployment](/soda-v3/quick-start-sip/upgrade#redeploy-a-self-hosted-soda-agent) guides. Don't execute the final `helm install` or `helm upgrade` step yet. Ensure you retrieve the `soda.apikey.id` and `soda.apikey.secret` values first, by using\ `helm get values -n ` . Now pass these values back to the upgrade command via the CLI: ```shell helm upgrade soda-agent/soda-agent --set soda.apikey.id=*** \ --set soda.apikey.secret=**** ``` Or by using a values file: ```shell > cat values-local.yaml soda: apikey: id: *** secret: *** > helm upgrade soda-agent soda-agent/soda-agent \ --values values-local.yml --namespace soda-agent ``` ### Registry access using a separate API key Ensure you have a new API key `id` and `secret` by following the [API key creation guide](/soda-v3/use-case-guides/api-keys) . Follow the self-hosted Soda agent [upgrade](/soda-v3/quick-start-sip/upgrade#upgrade-a-self-hosted-soda-agent) or [redeployment](/soda-v3/quick-start-sip/upgrade#redeploy-a-self-hosted-soda-agent) guides. Don't execute the final `helm install` or `helm upgrade` step yet. Now pass the API keys to use for registry access in the upgrade command via the CLI, using the `imageCredentials.apikey.id` and `imageCredentials.apikey.secret` properties.\ Note that we're also still passing the `soda.apikey.id` and `soda.apikey.secret` values, which are still required for the agent to authenticate to Soda Cloud. ```shell helm upgrade soda-agent/soda-agent --set soda.apikey.id=*** \ --set soda.apikey.secret=**** --set imageCredentials.apikey.id=*** \ --set imageCredentials.apikey.secret=*** ``` Or when using a values file: ```shell > cat values-local.yaml soda: apikey: id: *** secret: *** imageCredentials: apikey: id: *** secret: *** > helm upgrade soda-agent soda-agent/soda-agent \ --values values-local.yml --namespace soda-agent ``` ### Using existing (external) secrets You can also use a self-managed, existing secret to authenticate to the Soda-hosted or your self-hosted private container registry, for example when mirroring container images. You can refer to existing secrets as follows for the CLI: ```shell helm upgrade soda-agent/soda-agent --set soda.apikey.id=*** \ --set soda.apikey.secret=**** --set existingImagePullSecrets[0].name=my-existing-secret # Mind the array and indexing syntax! ``` Or using a values file: ```shell > cat values-local.yaml soda: apikey: id: *** secret: *** existingImagePullSecrets - name: my-existing-secret > helm upgrade soda-agent soda-agent/soda-agent \ --values values-local.yml --namespace soda-agent ``` ### Using the US image registry When you're onboarded on the US region of Soda Cloud, you'll have to use the container registry associated with that region. You can alter the `soda.cloud.region` value to automatically render the correct container registry and Soda Cloud API endpoint. Simply follow any of the above instructions and include the `soda.cloud.region` value. To do so in the CLI: ```shell helm upgrade soda-agent/soda-agent --set soda.apikey.id=*** \ --set soda.apikey.secret=**** --set soda.cloud.region=us ``` Or using a values file: ```shell > cat values-local.yaml soda: apikey: id: *** secret: *** cloud: region: "us" > helm upgrade soda-agent soda-agent/soda-agent \ --values values-local.yml --namespace soda-agent ``` ## FAQ ### Mirroring images If you want to mirror the Soda images into your own registries, you'll need to log in to the appropriate container registry. This will allow you to pull the images into your custom container image registry. ```shell # For Soda Cloud customers in the EU region docker login registry.cloud.soda.io -u -p # For Soda Cloud customers in the US region docker login registry.us.soda.io -u -p ``` The following `values.yaml` file illustrates the changes required for the Helm release to work with mirrored images: ```yaml existingImagePullSecrets - name: my-existing-secret soda: apikey: id: *** secret: *** agent: image: repository: custom.registry.org/sodadata/agent-orchestrator scanLauncher: image: repository: custom.registry.org/sodadata/soda-scan-launcher contractLauncher: image: repository: custom.registry.org/sodadata/soda-contract-launcher hooks: image: repository: custom.registry.org/sodadata/soda-agent-utils ``` ### Do I have to upgrade? What if we can't do that right away? Your existing Soda agent deployments will continue to function. This does mean that your self-hosted agent will not be able to support features like collaborative data contracts and the fully revamped metric monitoring. The images hosted on Docker Hub, required to run the self-hosted agent, will remain there in their current state for a grace period of 6 months. There will be no more maintenance (updates, bug fixes, security patches) for the old self-hosted agent versions. # Soda Agent extras Learn how to adjust the Soda Agent to fit your security standards by leveraging secrets managers, environment variables, and other controls. {% hint style="warning" %} The Soda environment has been updated since this tutorial. > Refer to [v4 documentation](/quickstart) for updated tutorials. > {% endhint %} When you deploy a self-hosted Soda Agent to a Kubernetes cluster in your cloud service provider environment, you need to provide several key parameters and values to ensure optimal operation and to allow the agent to connect to your Soda Cloud account, connect to your data sources, and run data quality scans on the data. ## Handle sensitive values By default, Soda uses [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) as part of the Soda Agent deployment. The agent automatically converts any sensitive values you add to a values YAML file, or directly via the CLI, into Kubernetes Secrets. As these values are sensitive, you may wish to employ the following alternative strategies to keep them secure. *** ### Use a values YAML file to store API key values When you deploy a self-hosted Soda Agent from the command line, you provide values for the API key ID and API key secret which the agent uses to connect to your Soda Cloud account. You can provide these values during agent deployment in one of two ways: * directly in the `helm install` command that deploys the agent and stores the values as [Kubernetes secrets](https://kubernetes.io/docs/concepts/configuration/secret/) in your cluster; see [Deploy using CLI only](/soda-v3/quick-start-sip/deploy#deploy-using-cli-only)\ OR * in a `values.yml` file which you store locally but reference in the `helm install` command, as in the example below. ```yaml soda: apikey: id: "***" secret: "***" agent: name: "myuniqueagent" ``` ```shell helm install soda-agent soda-agent/soda-agent \ --values values.yml \ --namespace soda-agent ``` Refer to the exhaustive [cloud service provider-specific instructions](/soda-v3/quick-start-sip/deploy#deploy-an-agent-in-a-kubernetes-cluster) for more detail on how to deploy an agent using a values YAML file. ### Use a values file to store private key authentication values If you use private key with Snowflake or BigQuery, you can provide the required private key values in a `values.yml` file when you deploy or redeploy the agent. * [Private key authentication with Snowflake](/soda-v3/data-source-reference/connect-snowflake#use-a-values-file-to-store-private-key-authentication-values) * [Private key authentication with BigQuery](/soda-v3/data-source-reference/connect-bigquery#use-a-file-reference-for-a-bigquery-data-source-connection) ### Use environment variables to store data source connection credentials When you, or someone in your organization, follows the guided steps to use a self-hosted Soda Agent to [add a data source](/soda-v3/quick-start-sip/deploy#add-a-new-data-source) in Soda Cloud, one of the steps involves providing the connection details and credentials Soda needs to connect to the data source to run scans. You can add those details directly in Soda Cloud, but because any user can then access these values, you may wish to store them securely in the values YAML file as environment variables. 1. Create or edit your local values YAML file to include the values for the environment variables you input into the connection configuration. ```yaml soda: apikey: id: "***" secret: "***" agent: name: "myuniqueagent" env: POSTGRES_USER: "sodalibrary" POSTGRES_PASS: "sodalibrary" ``` 2. After adding the environment variables to the values YAML file, update the Soda Agent using the following command: ```shell helm upgrade soda-agent soda-agent/soda-agent \ --values values.yml \ --namespace soda-agent ``` 3. In step 2 of the add a data source guided steps, add data source connection configuration which looks something like the following example for a PostgreSQL data source. Note the environment variable values for username and password. ```yaml data_source local_postgres_test: type: postgres host: 172.17.0.7 port: 5432 username: ${POSTGRES_USER} password: ${POSTGRES_PASS} database: postgres schema: new_york ``` 4. Follow the remaining guided steps to add a new data source in Soda Cloud. When you save the data source and test the connection, Soda Cloud uses the values you stored as environment variables in the values YAML file you supplied during redeployment. ### Integrate with a secrets manager Use External Secrets Operator to integrate your self-hosted Soda Agent with your secrets manager, such as Hashicorp Vault, AWS Secrets Manager, or Azure Key Vault, and securely reconcile the login credentials that Soda Agent uses for your data sources. For example, imagine you use Hashicorp Vault to store data source login credentials and your security protocol demands frequent rotation of passwords. In this situation, the challenge is that apps running in your Kubernetes cluster, such as a Soda Agent, need access to the up-to-date passwords. To address the challenge, you can set up and configure ESO in your Kubernetes cluster to regularly reconcile externally stored password values so that your apps always have the credentials they need. Doing so obviates the need to manually redeploy a values YAML file with new passwords for apps running in the cluster each time your system refreshes the passwords. The current integration of Soda Agent and a secrets manager *does not* yet support the configuration of the Soda Cloud credentials. For those credentials, use a tool such as [helm-secrets](https://github.com/jkroepke/helm-secrets) or [vals](https://github.com/helmfile/vals). To integrate Soda Agent with a secret manager, you need the following: * **External Secrets Operator** which is a Kubernetes operator that facilitates a connection between the Soda Agent and your secrets manager * a **ClusterSecretStore** resource which provides a central gateway with instructions on how to access your secret backend * an **ExternalSecret** resource which instructs the cluster on what values to fetch, and references the ClusterSecretStore Read more about the [ESO resource model](https://external-secrets.io/latest/introduction/overview/). The following procedure outlines how to use ESO to integrate with a **Hashicorp Vault** that uses a KV Secrets Engine v2. Extrapolate from this procedure to integrate with another secrets manager such as: * [AWS Secrets Manager](https://external-secrets.io/latest/provider/aws-secrets-manager/) * [Azure Key Vault](https://external-secrets.io/latest/provider/azure-key-vault/) #### Prerequisites * You have set up a Kubernetes cluster in your cloud services environment and deployed a self-hosted Soda Agent in the cluster. * For the purpose of this example procedure, you have set up and are using Hashicorp Vault which contains a key-value pair for `POSTGRES_USERNAME` and `POSTGRES_PASSWORD` at the path `local/soda`. #### Install and set up the External Secrets Operator Consider referencing the [use case guide](/soda-v3/use-case-guides/quick-start-secrets) for integrating an External Secrets Manager with a Soda Agent which offers step-by-step instructions to set everything up locally to see the integration in action. 1. Use Helm to install the External Secrets Operator from the [Helm chart repository](https://charts.external-secrets.io) into the same Kubernetes cluster in which you deployed your Soda Agent. ```shell helm repo add external-secrets https://charts.external-secrets.io helm install external-secrets \ external-secrets/external-secrets \ -n external-secrets \ --create-namespace ``` 2. Verify the installation using the following command: ```shell kubectl -n external-secrets get all ``` 3. Create a `cluster-secret-store.yml` file for the `ClusterSecretStore` configuration. The details in this file instruct the Soda Agent how to access the external secrets manager vault.\ This example uses [Hashicorp Vault AppRole authentication](https://external-secrets.io/latest/provider/hashicorp-vault/#approle-authentication-example). AppRole authenticates with Vault using the [App Role auth mechanism](https://developer.hashicorp.com/vault/docs/auth/approle) to access the contents of the secret store. It uses the SecretID in the Kubernetes secret, referenced by `secretRef` and the `roleID`, to acquire a temporary access token so that it can fetch secrets.\ Access external-secrets.io documentation for configuration examples for: * [AWS Secrets Manager](https://external-secrets.io/latest/provider/aws-secrets-manager/) * [Azure Key Vault](https://external-secrets.io/latest/provider/azure-key-vault/) ```yaml apiVersion: external-secrets.io/v1beta1 kind: ClusterSecretStore metadata: name: vault-app-role spec: provider: vault: auth: appRole: path: approle roleId: 3e****54-****-936e-****-5c5a19a5eeeb secretRef: key: appRoleSecretId name: external-secrets-vault-app-role-secret-id namespace: external-secrets path: kv server: http://vault.vault.svc.cluster.local:8200 version: v2 ``` 4. Deploy the `ClusterSecretStore` to your cluster. ```shell kubectl apply -f cluster-secret-store.yaml ``` 5. Create an `soda-secret.yml` file for the `ExternalSecret` configuration. The details in this file instruct the Soda Agent which values to fetch from the external secrets manager vault. ```yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: soda-agent namespace: soda-agent spec: data: - remoteRef: key: local/soda property: POSTGRES_USERNAME secretKey: POSTGRES_USERNAME - remoteRef: key: local/soda property: POSTGRES_PASSWORD secretKey: POSTGRES_PASSWORD refreshInterval: 1m secretStoreRef: kind: ClusterSecretStore name: vault-app-role target: name: soda-agent-secrets template: data: soda-agent.conf: | POSTGRES_USERNAME={{ .POSTGRES_USERNAME }} POSTGRES_PASSWORD={{ .POSTGRES_PASSWORD }} engineVersion: v2 ``` This example identifies: * the `namespace` of the Soda Agent, * two `remoteRef` configurations, including the file path in the vault, one each for `POSTGRES_USERNAME` and `POSTGRES_PASSWORD`, to detail what the `ExternalSecret` must fetch from Hashicorp Vault, * a `refreshInterval` to indicate how often ESO must reconcile the `remoteRef` values; this ought to correspond to the frequency with which your passwords are reset, * the `secretStoreRef` to indicate the `ClusterSecretStore` through which to access the vault, and * a `target template` that creates a file called `soda-agent.conf` into which it adds the username and password values in the dotenv format that the Soda Agent expects. 6. Deploy the `ExternalSecret` to your cluster. ```shell kubectl apply -n soda-agent -f soda-secret.yaml ``` 7. Use the following command to get the `ExternalSecret` to authenticate to Hashicorp Vault using the `ClusterSecretStore` and fetch secrets. ```shell kubectl get secret -n soda-agent soda-agent-secrets ``` Output: ```shell NAME TYPE DATA AGE soda-agent-secrets Opaque 1 24h ``` 8. Prepare a `values.yml` file to deploy the Soda Agent with the `existingSecrets` parameter that instructs it to access the `ExternalSecret` file to fetch data source login credentials. Refer to complete [deploy instructions](/soda-v3/quick-start-sip/deploy#deploy-using-a-values-yaml-file), or [redeploy instructions](/soda-v3/quick-start-sip/upgrade#redeploy-a-self-hosted-soda-agent) if you already have an agent running in a cluster. ```yaml soda: apikey: id: "154k***889" secret: "9sfjf****ff4" agent: name: "my-soda-agent-external-secrets" scanlauncher: existingSecrets: # from spec.target.name in the ExternalSecret file - soda-agent-secrets cloud: # Use https://cloud.us.soda.io for US region # Use https://cloud.soda.io for EU region endpoint: "https://cloud.soda.io" ``` 9. Deploy the Soda Agent using the following command: ```bash helm install soda-agent soda-agent/soda-agent \ --values values.yml \ --namespace soda-agent ``` Output: ```shell NAME: soda-agent LAST DEPLOYED: Tue Aug 29 13:08:51 2023 NAMESPACE: soda-agent STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Success, the Soda Agent is now running. You can inspect the Orchestrator logs if you like, but if all was configured correctly, the Agent should show up in Soda Cloud. Check the logs using: kubectl logs -l agent.soda.io/component=orchestrator -n soda-agent ``` ### Use Soda Cloud API keys from an existing secret By default, the Soda Agent creates a secret for storing the Soda Cloud API key details securely in your cluster. If you want to use a different secret, you can point the Soda Agent to an existing Kubernetes Secret in your cluster using the `soda.apikey.existingSecret` property. To use an existing Kubernetes secret for Soda Agent cloud API credentials, add `existingSecret` and the `secretKeys` values to your agent's values YAML file, as in the following example. ```yaml soda: apikey: existingSecret: "" secretKeys: idKey: "" secretKey: "" ``` ## Optimize performance The default Soda Agent settings balance performance and cost-efficiency. You can adjust these settings to better suit your needs, optimizing for larger datasets, faster scans, or improved resource management. ### Change sample data and failed rows memory limits The hard query cursor limit setting controls how many rows Soda Library can store in memory during a scan. By default, this value is 10,000 rows, preventing Out-Of-Memory errors by capping the number of rows Soda holds in memory at any given time. If you need to work with larger sets of sample data or failed rows, you can raise the `query_cursor_hard_limit`. Be aware that if you increase or remove the limit, you must ensure that the Soda Agent has enough memory to prevent OOM errors. To turn off the limit completely, set the value of `query_cursor_hard_limit` to `null`. The example below demonstrates how you can clear the limit and increase the memory limit using settings in your `values.yml` file: ```yaml soda: scanlauncher: config: query_cursor_hard_limit: null resources: limits: memory: 2Gi ``` ## Go further * Consider referencing the [use case guide](/soda-v3/use-case-guides/quick-start-secrets) for integrating an External Secrets Manager with a Soda Agent which offers step-by-step instructions to set everything up locally to see the integration in action. * Learn more about [basic Soda, Kubernetes, and Helm concepts](/soda-v3/learning-resources/basics). > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Invoke Soda Library Use Soda Library to programmatically execute scans and automate the checks for bad-quality data. {% hint style="warning" %} The Soda environment has been updated since this tutorial. > Refer to [v4 documentation](/quickstart) for updated tutorials. > {% endhint %} To automate the search for bad-quality data, you can use **Soda library** to programmatically set up and execute scans. As a Python library, you can invoke Soda just about anywhere you need it; the invocation instructions below offers a very simple invocation example to extrapolate from. Consult the [Use case guides](/soda-v3/use-case-guides) for more examples of how to programmatically run Soda scans for data quality. Alternatively, you can install and use the Soda Library CLI to run scans; see [Install Soda Library](/soda-v3/quick-start-sip/install). As a step in the **Get started roadmap**, this guide offers instructions to set up, install, and configure Soda in a [programmatic deployment model](/soda-v3/quick-start-sip/setup-guide#programmatic). #### Get started roadmap 1. ~~Choose a flavor of Soda~~ 2. **Set up Soda: programmatic** 📍 You are here!\ a. [Review requirements](#requirements)\ b. [Create a Soda Cloud account](#create-a-soda-cloud-account)\ c. [Set up basic programmatic invocation in Python](#set-up-basic-programmatic-invocation-in-python) 3. Write SodaCL checks 4. Run scans and review results 5. Organize, alert, investigate ## Requirements To use Soda Library, you must have installed the following on your system. * Python 3.8, 3.9, or 3.10 * Pip 21.0 or greater * A Soda Cloud account; see next section. * A Soda Cloud account; see next section.
Python versions Soda supports Soda officially supports Python versions 3.8, 3.9, and 3.10.\ Though largely functional, efforts to fully support Python 3.11 and 3.12 are ongoing.\ \ Using Python 3.11, some users might have some issues with dependencies constraints. At times, extra the combination of Python 3.11 and dependencies constraints requires that a dependency be built from source rather than downloaded pre-built.\ \ The same applies to Python 3.12, although there is some anecdotal evidence that indicates that 3.12 might not work in all scenarios due to dependencies constraints.
## Create a Soda Cloud account 1. In a browser, navigate to [cloud.soda.io/signup](https://cloud.soda.io/signup?utm_source=docs) to create a new Soda account, which is free for a 45-day trial. If you already have a Soda account, log in. 2. Navigate to **your avatar** > **Profile**, then access the **API keys** tab. Click the plus icon to generate new API keys. 3. Copy+paste the API key values to a temporary, secure place in your local environment.
Why do I need a Soda Cloud account? To validate your account license or free trial, Soda Library must communicate with a Soda Cloud account via API keys. You create a set of API keys in your Soda Cloud account, then use them to configure the connection to Soda Library. [Learn more](/soda-v3/quick-start-sip/get-started-roadmap#about-soda)
## Set up basic programmatic invocation in Python As in the simple example below, invoke the Python library and provide: * your data source connection configuration details, including environment variables, using one of the listed methods; consult [Data source reference](/soda-v3/data-source-reference) for data source-specific connection config * your Soda Cloud account API key values: * use cloud.soda.io for EU region * use cloud.us.soda.io for US region Use the following guidance for optional elements of a programmatic scan. * You can save Soda Library scan results anywhere in your system; the `scan_result` object contains all the scan result information. To import Soda Library in Python so you can utilize the `Scan()` object, [install a Soda Library package](/soda-v3/quick-start-sip/programmatic#set-up-basic-programmatic-invocation-in-python), then use `from soda.scan import Scan`. * If you provide a name for the scan definition to identify inline checks in a programmatic scan as independent of other inline checks in a different programmatic scan or pipeline, be sure to set a unique scan definition name for each programmatic scan. Using the same scan definition name in multiple programmatic scans results in confused check results in Soda Cloud. * If you wish to collect samples of failed rows when a check fails, you can employ a custom sampler; see [Configure a failed row sampler](/soda-v3/run-a-scan/failed-row-samples#configure-a-python-custom-sampler). * Be sure to include any variables in your programmatic scan *before* the check YAML files. Soda requires the variable input for any variables defined in the check YAML files. ```python from soda.scan import Scan scan = Scan() scan.set_data_source_name("events") # Add configuration YAML files ######################### # Choose one of the following to specify data source connection configurations : # 1) From a file scan.add_configuration_yaml_file(file_path="~/.soda/my_local_soda_environment.yml") # 2) Inline in the code # For host, use cloud.soda.io for EU region; use cloud.us.soda.io for US region scan.add_configuration_yaml_str( """ data_source events: type: snowflake host: ${SNOWFLAKE_HOST} username: ${SNOWFLAKE_USERNAME} password: ${SNOWFLAKE_PASSWORD} database: events schema: public soda_cloud: host: cloud.soda.io api_key_id: 2e0ba0cb-your-api-key-7b api_key_secret: 5wd-your-api-key-secret-aGuRg scheme: """ ) # Add variables ############### scan.add_variables({"date": "2022-01-01"}) # Add check YAML files ################## scan.add_sodacl_yaml_file("./my_programmatic_test_scan/sodacl_file_one.yml") scan.add_sodacl_yaml_file("./my_programmatic_test_scan/sodacl_file_two.yml") scan.add_sodacl_yaml_files("./my_scan_dir") scan.add_sodacl_yaml_files("./my_scan_dir/sodacl_file_three.yml") # OR # Define checks using SodaCL ################## checks = """ checks for cities: - row_count > 0 """ # Add template YAML files, if used ################## scan.add_template_files(template_path) # Add the checks to the scan #################### scan.add_sodacl_yaml_str(checks) # OR Add the checks to scan with virtual filename identifier # for advanced use cases such as partial/concurrent scans #################### scan.add_sodacl_yaml_str( checks file_name=f"checks-{scan_name}.yml", ) # Set scan definition name, equivalent to CLI -s option # The scan definition name MUST be unique to this scan, and # not duplicated in any other programmatic scan ################## scan.set_scan_definition_name("YOUR_SCHEDULE_NAME") # Execute the scan ################## scan.execute() # Set logs to verbose mode, equivalent to CLI -V option ################## scan.set_verbose(True) # Do not send results to Soda Cloud, equivalent to CLI -l option; ################## scan.set_is_local(True) # Inspect the scan result ######################### scan.get_scan_results() # Inspect the scan logs ####################### scan.get_logs_text() # Typical log inspection ################## scan.assert_no_error_logs() scan.assert_no_checks_fail() # Advanced methods to inspect scan execution logs ################################################# scan.has_error_logs() scan.get_error_logs_text() # Advanced methods to review check results details ######################################## scan.get_checks_fail() scan.has_check_fails() scan.get_checks_fail_text() scan.assert_no_checks_warn_or_fail() scan.get_checks_warn_or_fail() scan.has_checks_warn_or_fail() scan.get_checks_warn_or_fail_text() scan.get_all_checks_text() ``` See the [Python API Reference page](/soda-v3/run-a-scan/python_api) for detailed documentation of the `Scan` class in Soda Library. ## Tips and best practices * You can save Soda Library scan results anywhere in your system; the `scan_result` object contains all the scan result information. To import Soda Library in Python so you can utilize the `Scan()` object, [install a Soda Library package](/soda-v3/quick-start-sip/install), then use `from soda.scan import Scan`. * Be sure to include any variables in your programmatic scan *before* the check YAML files. Soda requires the variable input for any variables defined in the check YAML files. * Because Soda Library pushes scan results to Soda Cloud, you may not want to change the scan definition name with each scan. Soda Cloud uses the scan definition name to correlate subsequent scan results, thus retaining an historical record of the measurements over time.\ Sometimes, changing the name is useful, like when you wish to configure a single scan to run in multiple environments. Be aware, however, that if you change the scan definition name with each scan for the same environment, Soda Cloud recognizes each set of scan results as independent from previous scan results, thereby making it appear as though it records a new, separate check result with each scan and archives or "disappears" previous results. See also: [Missing check results in Soda Cloud](/soda-v3/sodacl-reference/troubleshoot#missing-check-results-in-soda-cloud). ## Scan exit codes Soda Library’s scan output includes an exit code which indicates the outcome of the scan. | | | | - | ------------------------------------------------------------------------------------ | | 0 | all checks passed, all good from both runtime and Soda perspective | | 1 | Soda issues a warning on a check(s) | | 2 | Soda issues a failure on a check(s) | | 3 | Soda encountered a runtime issue, and was able to submit scan results to Soda Cloud | | 4 | Soda encountered a runtime issue, but was unable to submit any results to Soda Cloud | To obtain the exit code, you can add the following to your programmatic scan. ```python exit_code = scan.execute() print(exit_code) ``` ## Next 1. ~~Choose a flavor of Soda~~ 2. ~~Set up Soda: programmatic~~ 3. [**Write SodaCL checks**](/soda-v3/soda-cl-overview) 4. Run scans and review results 5. Organize, alert, investigate > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Upgrade, redeploy, or uninstall Soda Learn how to upgrade or uninstall Soda Library, or redploy a Soda Runner. {% hint style="warning" %} The Soda environment has been updated since this tutorial. > Refer to [v4 documentation](/quickstart) for updated tutorials. > {% endhint %} *** ## Migrate a data source from a self-hosted to a Soda-hosted runner If you already use a self-hosted Soda Runner deployed in a Kubernetes cluster to connect to your data source(s), you have the option of migrating a connected data source to a Soda-hosted runner. Though you must reconfigure your data source connection to the new Soda runner, your checks, check history, and scan definition remain intact. * Be aware that Soda-hosted runners are only compatible with the following data sources: BigQuery, Databricks SQL, MS SQL Server, MySQL, PostgreSQL, Redshift, Snowflake. * 🔴 When you migrate to a Soda-hosted runner, Soda Cloud *resets* all the connection configuration details for your data source. Be sure to capture all existing data source connection details before migrating so you can re-enter the details for the data source connection. 1. As a user with permission to do so in Soda Cloud, navigate to **your avatar** > **Organization Settings**. In the **Organization tab**, click the checkbox to **Enable Soda-hosted Runner**. 2. Navigate to **your avatar** > **Data Sources**, then access the **Runners** tab. Notice your out-of-the-box Soda-hosted runner that is up and running.
3. Navigate to the **Data Sources** tab, then click to select the data source you wish to migrate to the Soda-hosted runner. 4. In the **2. Connect the Data Source** tab, copy+paste the contents of the editing panel to a temporary, secure, local place in your system. Switching runners resets all connection configuration parameters, so be sure to record existing parameter settings before proceeding. 5. In the **1. Attributes** tab, use the dropdown for **Default Scan Runner** to select `soda-hosted-runner`. 6. Return to the **2. Connect the Data Source** tab, then, using the configuration values you recorded in step 3, use the dropdowns to re-enter the values, then **Test Data Source**. 7. When the test completes successfully, **Save** your changes to the data source. ## Redeploy a self-hosted Soda Runner The **Soda Runner** is a tool that empowers Soda Cloud users to securely access data sources to scan for data quality. Create a Kubernetes cluster in a cloud services provider environment, then use Helm to deploy a sefl-hosted Soda Runner in the cluster. [Read more](/soda-v3/learning-resources/basics). When you delete the Soda Runner Helm chart from your cluster, you also delete all the runner resources on your cluster. However, if you wish to redeploy the previously-registered runner (use the same name), you need to specify the runner ID in your override values in your values YAML file. 1. In Soda Cloud, navigate to **your avatar** > **Runners**. 2. Click to select the runner you wish to redeploy, then copy the runner ID of the previously-registered runner from the URL.\ \ For example, in the following URL, the runner ID is the long UUID at the end. `https://cloud.soda.io/runners/842feab3-snip-87eb-06d2813a72c1`.\ \ Alternatively, if you use the base64 CLI tool, you can run the following command to obtain the runner ID. ```shell kubectl get secret/soda-runner-id -n soda-runner --template={{.data.SODA_RUNNER_ID}} | base64 --decode ``` 3. Open your `values.yml` file, then add the `id` key:value pair under `runner`, using the runner ID you copied from the URL as the value. ```yaml soda: apikey: id: "***" secret: "***" runner: id: "842feab3-snip-87eb-06d2813a72c1" name: "myuniquerunner" ``` 4. To redeploy the runner, you need to provide the values for the API keys the runner uses to connect to Soda Cloud in the values YAML file. Access the values by running the following command, replacing the `soda-runner` values with your own details, then paste the values into your values YAML file. ```shell helm get values -n soda-runner soda-runner ``` Alternatively, if you use the base64 CLI tool, you can run the following commands to obtain the API key and API secret, respectively. ```shell kubectl get secret/soda-runner-apikey -n soda-runner --template={{.data.SODA_API_KEY_ID}} | base64 --decode ``` ```shell kubectl get secret/soda-runner-apikey -n soda-runner --template={{.data.SODA_API_KEY_SECRET}} | base64 --decode ``` 5. In the same directory in which the `values.yml` file exists, use the following command to install the Soda Runner helm chart. ```shell helm install soda-runner soda-runner/soda-runner \ --values values.yml \ --namespace soda-runner ``` 6. Validate the Soda Runner deployment by running the following command: ```shell kubectl describe pods ``` ## Upgrade a self-hosted Soda Runner The **Soda Runner** is a Helm chart that you deploy on a Kubernetes cluster and connect to your Soda Cloud account using API keys. To take advantage of new or improved features and functionality in the Soda Runner, including new features in the Soda Library, you can upgrade your runner when a new version becomes available in [ArtifactHub.io](https://artifacthub.io/packages/helm/soda-runner/soda-runner). Note that there is no downtime associated with the exercise of upgrading a self-hosted Soda Runner. Because Soda does not define the `.spec.strategy` in the deployment manifest of the Soda Runner Helm chart, Kubernetes uses the default `RollingUpdate` to upgrade; refer to [Kubernetes documentation ](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy). 1. If you regularly access multiple clusters, you must ensure that are first accessing the cluster which contains your deployed Soda Runner. Use the following command to determine which cluster you are accessing. ```shell kubectl config get-contexts ``` If you must switch contexts to access a different cluster, copy the name of cluster you wish to use, then run the following command. ``` kubectl config use-context ``` 2. To upgrade the runner, you must know the values for: * namespace - the namespace you created, and into which you deployed the Soda Runner * release - the name of the instance of a helm chart that is running in your Kubernetes cluster * API keys - the values Soda Cloud created which you used to run the runner application in the cluster\ Access the first two values by running the following command. ```shell helm list ``` Output: ```shell NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION soda-runner soda-runner 5 2023-01-20 11:55:49.387634 -0800 PST deployed soda-runner-0.8.26 Soda_Library_1.0.0 ``` 3. Access the API key values by running the following command, replacing the placeholder values with your own details. ```shell helm get values -n ``` From the output above, the command to use is: ```shell helm get values -n soda-runner soda-runner ``` 4. Use the following command to search ArtifactHub for the most recent version of the Soda Runner Helm chart. ```shell helm search hub soda-runner ``` 5. Use the following command to upgrade the Helm repository. ```shell helm repo update ``` 6. Upgrade the Soda Runner Helm chart. The value for the chart argument can be a chart reference such as `example/runner`, a path to a chart directory, a packaged chart, or a URL. To upgrade the runner, Soda uses a chart reference: `soda-runner/soda-runner`. ```shell helm upgrade --set soda.apikey.id=*** \ --set soda.apikey.secret=**** ``` From the output above, the command to use is ```shell helm upgrade soda-runner soda-runner/soda-runner \ --set soda.apikey.id=*** \ --set soda.apikey.secret=**** ``` OR, if you use a values YAML file, ```shell helm upgrade soda-runner soda-runner/soda-runner \ --values values-local.yml --namespace soda-runner ``` ### Upgrade to Soda Runner 1.0.0 or greater Soda Runner 1.0.0 includes several key changes to the way the Soda Runner works. If you already use a Soda Runner, carefully consider the changes that Soda Runner 1.0.0 introduces and make appropriate changes to your configured parameters. Soda Runner 1.0.0 favors manged or self-managed node groups over AWS Fargate, AKS Virtual Nodes, or GKE Autopilot profiles. Though this version of the runner still works with those profiles, the scan performance is slower because the profiles provision new nodes for each scan. To migrate your runner to a managed node group: 1. Add a managed node group to your Kubernetes cluster. 2. Check your cloud-services provider’s recommendations for node size and adapt it for your needs based on volume of scans you anticipate. Best practice dictates that you set your cluster to have at least 2 CPU and 2GB of RAM, which, in general is sufficient to run up to six scans in parallel. 3. [Upgrade to Soda Runner 1.0.0](/soda-v3/quick-start-sip/upgrade#upgrade-a-self-hosted-soda-runner), configuring the helm chart to *not* use Fargate, Virtual Nodes, or GKE Autopilot by: * removing the `provider.eks.fargate.enabled` property, or setting the value to `false` * removing the `provider.aks.virtualNodes.enabled` property, or setting the value to `false` * removing the `provider.gke.autopilot.enabled` property, or setting the value to `false` * removing the `soda.runner.target` property 4. Remove the Fargate profiles, and drain existing workloads from virtual nodes in the namespace in which you deployed the Soda Runner so that the runner uses the node group to execute scans, not the profiles. ### Upgrading from 1.1.x to 1.2.x Starting from version 1.2.0 all images required for the Soda Runner are distributed using a Soda-hosted image registry. For more information, see [Broken mention](broken://spaces/A2PmHkO5cBgeRPdiPPOG/pages/0Cs2lWTZr1mIVYrDs9Y0). #### Set up authentication for the Soda image registry **Using your exising Soda API key and secret** By default we'll use your existing Soda API key and secret values to perform the authentication to the Soda image registry. Ensure these values are still present in your `values.yaml` , no further action is required. ```yaml soda: # These values will also be used to authenticate to the Soda image registry apikey: id: existing-key-id secret: existing-key-secret ``` **Using a separate Soda API key and secret** You might also opt to use a new, separate Soda API key and secret to perform the authentication to the Soda image registry. In this case, ensure the `imageCredentials.apikey.id` and `imageCredentials.apikey.secret` values are set to these new values: ```yaml soda: apikey: id: existing-key-id secret: existing-key-secet imageCredentials: apikey: id: my-new-key-id secret: my-new-key-secret ``` #### Specify existing `imagePullSecrets` If you're providing your own `imagePullSecrets` on the cluster, e.g. when you're pulling images from your own mirroring image registry, you must modify your existing values file. The `imagePullSecrets` property that was present in versions `1.1.x` has been renamed to the more standard `existingImagePullSecrets` . If applicable to you, please perform the following rename in your values file: ```yaml soda: apikey: id: *** secret: *** # This is no longer supported # imagePullSecrets # - name: my-existing-secret ​ # Instead, use this! existingImagePullSecrets - name: my-existing-secret ``` For more information on setting up image mirroring, see [Broken mention](broken://spaces/A2PmHkO5cBgeRPdiPPOG/pages/0Cs2lWTZr1mIVYrDs9Y0#mirroring-images) #### Update the `region` If you are a customer using the US instance of Soda Cloud, you'll have to configure your Runner setup accordingly. Otherwise you can ignore this section. In version `1.2.0` we're introducing a `soda.cloud.region` property, that will be used to determine which registry and Soda Cloud endpoint to use. Possible values are `eu` and `us`. When the `soda.cloud.region` property is not set explicitly, it defaults to the value of `eu`. If applicable to you, please perform the following changes in your values file: ```yaml soda: apikey: id: *** secret: *** cloud: # This also sets the correct endpoint under the covers. region: "us" # This can be removed now, as the region property sets this up correctly. # endpoint: https://cloud.us.soda.io ``` For more information about using the US region, see [Broken mention](broken://spaces/A2PmHkO5cBgeRPdiPPOG/pages/0Cs2lWTZr1mIVYrDs9Y0#using-the-us-image-registry). #### Rename `scanlauncher` to `scanLauncher` The `scanlauncher` section in the values file has been renamed to `scanLauncher`.\ Please ensure the correct name is used in your values file if you have any configuration values there: ```yaml soda: apikey: id: *** secret: *** # Rename this ... # scanLauncher: # to become scanLauncher: existingSecrets: - soda-runner-secrets ``` ## Upgrade Soda Library To upgrade your existing Soda Library tool to the latest version, use the following command, replacing `redshift` with the install package that matches the type of data source you are using. ```shell pip install -i https://pypi.cloud.soda.io soda-redshift -U ``` ## Uninstall Soda Library 1. (Optional) From the command-line, run the following command to determine which Soda packages exist in your environment. ```shell pip freeze | grep soda ``` 2. (Optional) Run the following command to uninstall a specific Soda package from your environment. ```shell pip uninstall soda-postgres ``` 3. Run the following command to uninstall all Soda packages from your environment, completely. ```shell pip freeze | grep soda | xargs pip uninstall -y ``` ## Migrate from Soda Core Soda Core, the free, open-source Python library and CLI tool upon which Soda Library is built, continues to exist as an OSS project in [GitHub](https://github.com/sodadata/soda-core). To migrate from an existing Soda Core installation to Soda Library, simply uninstall the old and install the new from the command-line. 1. Uninstall your existing Soda Core packages using the following command. ```shell pip freeze | grep soda | xargs pip uninstall -y ``` 2. Install a Soda Library package that corresponds to your data source. Your new package automatically comes with a 45-day free trial. Our Soda team will contact you with licensing options after the trial period. ```shell pip install -i https://pypi.cloud.soda.io soda-postgres ``` 3. If you had connected Soda Core to Soda Cloud, you do not need to change anything for Soda Library to work with your Soda Cloud account.\ If you had not connected Soda Core to Soda Cloud, you need to connect Soda Library to Soda Cloud. Soda Library requires API keys to validate licensing or trial status and run scans for data quality. See [Configure Soda](/soda-v3/quick-start-sip/install#configure-soda) for instructions. 4. You *do not need* to adjust your existing `configuration.yml` or `checks.yml` files which will continue to work as before. ## Go further * Learn more about the ways you can use Soda in [Use case guides](/soda-v3/use-case-guides). * Write [custom SQL checks](/soda-v3/sodacl-reference/user-defined) for your own use cases. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Use case guides Access examples of Soda implementations according to use case and data quality testing needs. Use the following guides as example implementations based on how you intend to use Soda for data quality testing. For standard set up instructions, see [Get started](/soda-v3/quick-start-sip/get-started-roadmap). | [Test data in an Airflow pipeline](/soda-v3/use-case-guides/quick-start-prod) | Use this guide as an example for how to set up Soda to test the quality of your data in an Airflow pipeline that uses dbt transformations. |

Soda Library
Soda Cloud

| | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------- | | [Test data quality in an ADF pipeline](/soda-v3/use-case-guides/quick-start-adf) | Learn how to invoke Soda data quality tests in an ETL pipeline in Azure Data Factory. |

Soda Library
Soda Cloud

| | [Test data quality in a Dagster pipeline](/soda-v3/use-case-guides/quick-start-dagster) | Learn how to invoke Soda data quality tests in a Dagster pipeline. |

Soda Library
Soda Cloud

| | [Test data quality in Databricks pipeline](/soda-v3/use-case-guides/quick-start-databricks-pipeline) | Learn how to use Databricks notesbooks with Soda to test data quality before feeding a machine learning model. |

Soda Library
Soda Cloud

| | [Test data before migration](/soda-v3/use-case-guides/quick-start-migration) | Use this guide to set up Soda to test before and after data migration between data sources. |

Soda Library
Soda Cloud

| | [Self-serve Soda](/soda-v3/use-case-guides/quick-start-end-user) | Use this guide to set up Soda Cloud to enable users across your organization to serve themselves when it comes to testing data quality. |

Soda Cloud
Soda Runner

| | [Test data during development](/soda-v3/use-case-guides/quick-start-dev) | Use this guide to set up Soda to test the quality of your data during your development lifecycle in a GitHub Workflow. |

Soda Library
Soda Cloud

| | [Automate monitoring](/soda-v3/use-case-guides/quick-start-automate) | Use this guide to set up Soda to automatically monitor data quality. |

Soda Cloud
Soda Runner

| \ Use the following How tos for practical advice, examples, and instructions for using Soda. | [Build a Sigma dashboard](/soda-v3/use-case-guides/reporting-api-to-overview-dashboards) | Learn how to build a customized data quality reporting dashboard in Sigma using the Soda Cloud API. |

Soda Library
Soda Cloud

| | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | | [Build a Grafana dashboard](/soda-v3/use-case-guides/public-api-to-grafana) | Learn how to build a customized data quality reporting dashboard in Grafana using the Soda Cloud API. | Soda Cloud | | [Invoke Soda in Databricks](/soda-v3/use-case-guides/quick-start-databricks) | Learn how to invoke Soda data quality tests in a Databricks notebook. |

Soda Library
Soda Cloud

| | [Use a Secrets Manager](/soda-v3/use-case-guides/quick-start-secrets) | Learn how to set up a Soda Runner to use an External Secrets Manager to retrieve frequently-rotated data source passwords. |

Soda Cloud
Self-hosted Runner

| | [Generate API keys](/soda-v3/use-case-guides/api-keys) | Learn how to use Soda Cloud API keys to securely communicate with other entities such as Soda Library and self-hosted Soda Runners, and to provide secure access to Soda Cloud via API. | Soda Cloud | | [Manage sensitive data](/soda-v3/use-case-guides/sensitive-data) | Learn how to adjust several configurable settings that help you manage access to sensitive data in Soda Cloud. | Soda Cloud | | [Reroute failed row samples](/soda-v3/run-a-scan/failed-row-samples) | Learn how to programmatically set up Soda Library to display failed row samples in the command-line. |

Soda Library
Soda Cloud

| | [Double-onboard a data source](/soda-v3/use-case-guides/double-onboard-datasource) | Learn how to onboard a data source in Soda Cloud that you have already onboarded via Soda Library. |

Soda Library
Soda Cloud

| > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Test data quality in an Airflow data pipeline Follow this guide to set up and run scheduled Soda scans for data quality in your Airflow data pipeline. Use this guide as an example for how to set up and use Soda to test the quality of your data in an Airflow pipeline. Automatically catch data quality issues after ingestion or transformation to prevent negative downstream impact. > Not quite ready for this big gulp of Soda? 🥤Try [taking a sip](/soda-v3/quick-start-sip), first.
## About this guide The instructions below offer Data Engineers an example of how to execute SodaCL checks for data quality on data in an [Apache Airflow](https://airflow.apache.org/) pipeline. For context, this guide presents an example of a Data Engineer at a small firm who was tasked with building a simple products report of sales by category for [AdventureWorks data](https://github.com/aldibit/soda-docs/blob/main/assets/adventureworks_schema.png). This Engineer uses [dbt](https://www.getdbt.com/) to build a simple model transformation to gather data, then builds more models to transform and push gathered information to a reporting and visualization tool. The Engineer uses Airflow for scheduling and monitoring workflows, including data ingestion and transformation events. The Engineer's goal in this example is to make sure that after such events, and before pushing information into a reporting tool, they run scans to check the quality of the data. Where the scan results indicate an issue with data quality, Soda notifies the Engineer so that they can potentially stop the pipeline and investigate and address any issues before the issue causes problems in the report. Access the [sodadata/sip-of-soda/test-data-in-pipeline](https://github.com/sodadata/sip-of-soda/tree/main/test-in-pipeline/soda) folder to review the dbt models and Soda checks files that the Data Engineer uses. Borrow from this guide to connect to your own data source, set up scan points in your pipeline, and execute your own relevant tests for data quality. ## Install Soda from the command-line With Python 3.8, 3.9, or 3.10 installed, the Engineer creates a virtual environment in Terminal, then installs the Soda package for PostgreSQL using the following command. ```shell pip install -i https://pypi.cloud.soda.io soda-postgres ``` > Refer to [complete install instructions](/soda-v3/quick-start-sip/install) for all supported data sources, if you wish. ## Connect Soda to a data source and Soda Cloud account To connect to a data source such as Snowflake, PostgreSQL, Amazon Athena, or BigQuery, you use a `configuration.yml` file which stores access details for your data source. This guide also includes instructions for how to connect to a Soda Cloud account using API keys that you create and add to the same `configuration.yml` file. Available for free as a 45-day trial, a Soda Cloud account gives you access to visualized scan results, tracks trends in data quality over time, enables you to set alert notifications, and much more. 1. In the directory in which they work with their dbt models, the Data Engineer creates a `soda` directory to contain the Soda configuration and check YAML files. 2. In the new directory, they create a new file called `configuration.yml`. 3. In the `configuration.yml` file, they add the data source connection configuration for the PostgreSQL data source that contains the AdventureWorks data. The example below is the connection configuration for a PostgreSQL data source. Access the [example file](https://github.com/sodadata/sip-of-soda/blob/main/test-in-pipeline/soda/configuration.example.yml). See a complete list of supported [data sources](/soda-v3/data-source-reference). ```yaml data_source soda-demo: type: postgres host: localhost username: postgres password: secret database: postgres schema: public ``` 4. In a browser, they navigate to [cloud.soda.io/signup](https://cloud.soda.io/signup?utm_source=docs) to create a free, 45-day trial Soda account. 5. They navigate to **avatar** > **Profile**, then navigate to the **API Keys** tab and click the plus icon to generate new API keys. * They copy the syntax for the `soda_cloud` configuration, including the values **API Key ID** and **API Key Secret**, and paste it into the `configuration.yml`. * They are careful *not* to nest the `soda_cloud` configuration in the `data_source` configuration. 6. They save the `configuration.yml` file and close the API modal in the Soda account. 7. In Terminal, they run the following command to test Soda's connection to the data source. ```shell soda test-connection -d adventureworks -c configuration.yml ``` ## Write checks for data quality A check is a test that Soda executes when it scans a dataset in your data source. The `checks.yml` file stores the checks you write using the Soda Checks Language (SodaCL). You can create multiple `checks.yml` files to organize your data quality checks and run all, or some of them, at scan time. In this example, the Data Engineer creates multiple checks after ingestion, after initial transformation, and before pushing the information to a visualization or reporting tool.
### Transform checks After building a simple dbt model transformation that creates a new fact table which gathers data about products, product categories, and subcategories ([see dbt model](https://github.com/sodadata/sip-of-soda/blob/main/test-in-pipeline/dbt%20models/transform/fact_product_category.sql)), the Engineer realizes that some of the products in the dataset do not have an assigned category or subcategory, which means those values would erroneously be excluded from the report. To mitigate the issue and get a warning when these values are missing, they create a new checks YAML file and write the following checks to execute after the transformation produces the `fact_product_category` dataset.
[fact\_product\_category.yml](https://github.com/sodadata/sip-of-soda/blob/main/test-in-pipeline/soda/transform-checks/fact_product_category.yml) ```yaml checks for fact_product_category: # Check warns when any NULL values exist in the column - missing_count(category_key): name: All products have a category warn: when > 0 # Check warns when any NULL values exist in the column - missing_count(subcategory_key): name: All products have a subcategory warn: when > 0 # Check warns when any NULL values exist in the column - missing_count(product_key) = 0: name: All products have a key ``` ### Ingest checks Because the Engineer does not have the ability or access to fix upstream data themselves, they create another checks YAML file write checks to apply to each dataset they use in the transformation, *after* the data is ingested, but before it is transformed. For any checks that fail, the Engineer can notify upstream Data Engineers or Data Product Owners to address the issue of missing categories and subcategories.
[dim\_product.yml](https://github.com/sodadata/sip-of-soda/blob/main/test-in-pipeline/soda/ingest-checks/dim_product.yml) ```yaml checks for dim_product: # Check fails when product_key or english_product_name is missing, OR # when the data type of those columns is other than specified - schema: fail: when required column missing: [product_key, english_product_name] when wrong column type: product_key: integer english_product_name: varchar # Check fails when any NULL values exist in the column - missing_count(product_key) = 0: name: All products have a key # Check fails when any NULL values exist in the column - missing_count(english_product_name) = 0: name: All products have a name # Check fails when any NULL values exist in the column - missing_count(product_subcategory_key): name: All products have a subcategory warn: when > 0 # Check fails when the number of products, relative to the # previous scan, changes by 10 or more - change for row_count < 10: name: Products are stable ``` [dim\_product\_category.yml](https://github.com/sodadata/sip-of-soda/blob/main/test-in-pipeline/soda/ingest-checks/dim_product_category.yml) ```yaml checks for dim_product_category: # Check fails when product_category_key or english_product_category name # is missing, OR # when the data type of those columns is other than specified - schema: fail: when required column missing: [product_category_key, english_product_category_name] when wrong column type: product_category_key: integer english_product_category_name: varchar # Check fails when any NULL values exist in the column - missing_count(product_category_key) = 0: name: All categories have a key # Check fails when any NULL values exist in the column - missing_count(english_product_category_name) = 0: name: All categories have a name # Check fails when the number of categories, relative to the # previous scan, changes by 5 or more - change for row_count < 5: name: Categories are stable ``` [dim\_product\_subcategory.yml](https://github.com/sodadata/sip-of-soda/blob/main/test-in-pipeline/soda/ingest-checks/dim_product_subcategory.yml) ```yaml checks for dim_product_subcategory: # Check fails when product_subcategory_key or english_product_subcategory_name # is missing, OR # when the data type of those columns is other than specified - schema: fail: when required column missing: [product_subcategory_key, english_product_subcategory_name] when wrong column type: product_subcategory_key: integer english_product_subcategory_name: varchar # Check fails when any NULL values exist in the column - missing_count(product_subcategory_key) = 0: name: All subcategories have a key # Check fails when any NULL values exist in the column - missing_count(english_product_subcategory_name) = 0: name: All subcategories have a name # Check fails when the number of categories, relative to the # previous scan, changes by 5 or more - change for row_count < 5: name: Subcategories are stable ``` [fact\_internet\_sales.yml](https://github.com/sodadata/sip-of-soda/blob/main/test-in-pipeline/soda/ingest-checks/fact_internet_sales.yml) ```yaml checks for fact_internet_sales: # Check fails when product_key, order_quantity, or sales_amount # is missing, OR # when the data type of those columns is other than specified - schema: fail: when required column missing: [product_key, order_quantity, sales_amount] when wrong column type: product_key: integer order_quantity: smallint sales_amount: money # Check fails when any NULL values exist in the column - missing_count(product_key) = 0: name: All sales have a product associated # Check fails when any order contains no items - min(order_quantity) > 0: name: All sales have a non-zero order quantity # Check fails when the amount of any sales order is zero - failed rows: name: All sales have a non-zero order amount fail query: | SELECT sales_order_number, sales_amount::NUMERIC FROM fact_internet_sales WHERE sales_amount::NUMERIC <= 0 # Check warns when there are fewer than 5 new internet sales # relative to the previous scan resuls # Check fails when there are more than 500 new internet sales # relative to the previous scan resuls - change for row_count: warn: when < 5 fail: when > 500 name: Sales are within expected range # Check fails when the average of the column is abnormal # relative to previous measurements for average sales amount # sales_amount is cast from data type MONEY to enable calculation - anomaly detection for avg(sales_amount::NUMERIC) ``` ## Reports checks Finally, the Engineer builds category and subcategory sales report models [sales report models](https://github.com/sodadata/sip-of-soda/tree/main/test-in-pipeline/dbt%20models/report) using dbt. The checks files they create to run on the new transform models contain similiar user-defined checks. Ultimately, the Engineer wants data quality checks to fail if the sales of uncategorized products rises above normal (0.85%), and if the sum of sales orders in the model that prepares the report differs greatly from the sum of raw sales order number.
[report\_category\_sales.yml](https://github.com/sodadata/sip-of-soda/blob/main/test-in-pipeline/soda/reports-checks/report_category_sales.yml) ```yaml checks for report_category_sales: # Check fails if the percentage of sales of products with no # catgory exceeds 0.90% - uncategorized_sales_percent < 0.9: uncategorized_sales_percent query: > select ROUND(CAST((sales_total * 100) / (select sum(sales_total) from report_category_sales) AS numeric), 2) as uncategorized_sales_percent from report_category_sales where category_key is NULL name: Most sales are categorized # Check fails if the sum of sales produced by the model is different # than the sum of sales in the fact_internet_sales dataset - sales_diff = 0: name: Category sales total matches sales_diff query: > SELECT CAST((SELECT SUM(fact_internet_sales.sales_amount) FROM fact_internet_sales) - (SELECT SUM(report_category_sales.sales_total) FROM report_category_sales) as numeric) AS sales_diff ``` [report\_subcategory\_sales.yml](https://github.com/sodadata/sip-of-soda/blob/main/test-in-pipeline/soda/reports-checks/report_subcategory_sales.yml) ```yaml checks for report_subcategory_sales: # Check fails if the percentage of sales of products with no # subcategory exceeds 0.90% - uncategorized_sales_percent < 0.9: uncategorized_sales_percent query: > select ROUND(CAST((sales_total * 100) / (select sum(sales_total) from report_subcategory_sales) AS numeric), 2) as uncategorized_sales_percent from report_subcategory_sales where category_key is NULL OR subcategory_key is NULL name: Most sales are categorized # Check fails if the sum of sales produced by the model is different # than the sum of sales in the fact_internet_sales dataset - sales_diff = 0: name: Subcategory sales total matches sales_diff query: > SELECT CAST((SELECT SUM(fact_internet_sales.sales_amount) FROM fact_internet_sales) - (SELECT SUM(report_subcategory_sales.sales_total) FROM report_subcategory_sales) as numeric) AS sales_diff ``` ## Create a DAG and run the workflow The Engineer creates an Airflow DAG and [runs the workflow locally](https://airflow.apache.org/docs/apache-airflow/2.2.3/start/local.html). Note that the value for `scan-name` must be unique to every programmatic scan you define. In other words, it cannot be the same as a programmatic scan in another pipeline.\ [Access the DAG](https://github.com/sodadata/sip-of-soda/blob/main/test-in-pipeline/airflow/model_sales_category.py) in the repo. ```python from airflow import DAG from airflow.models.variable import Variable from airflow.operators.bash import BashOperator from airflow.operators.python import PythonVirtualenvOperator from airflow.operators.dummy import DummyOperator from airflow.utils.dates import days_ago from datetime import timedelta default_args = { "owner": "soda", "retries": 1, "retry_delay": timedelta(minutes=5), } PROJECT_ROOT = "" def run_soda_scan(project_root, scan_name, checks_subpath = None): from soda.scan import Scan print("Running Soda Scan ...") config_file = f"{project_root}/soda/configuration.yml" checks_path = f"{project_root}/soda/checks" if checks_subpath: checks_path += f"/{checks_subpath}" data_source = "soda_demo" scan = Scan() scan.set_verbose() scan.add_configuration_yaml_file(config_file) scan.set_data_source_name(data_source) scan.add_sodacl_yaml_files(checks_path) scan.set_scan_definition_name(scan_name) result = scan.execute() print(scan.get_logs_text()) if result != 0: raise ValueError('Soda Scan failed') return result with DAG( "model_adventureworks_sales_category", default_args=default_args, description="A simple Soda Library scan DAG", schedule_interval=timedelta(days=1), start_date=days_ago(1), ): ingest_raw_data = DummyOperator(task_id="ingest_raw_data") checks_ingest = PythonVirtualenvOperator( task_id="checks_ingest", python_callable=run_soda_scan, requirements=[ "-i https://pypi.cloud.soda.io", "soda-postgres", "soda-scientific"], system_site_packages=False, op_kwargs={ "project_root": PROJECT_ROOT, "scan_name": "model_adventureworks_sales_category_ingest", "checks_subpath": "ingest/dim_product_category.yml" }, ) dbt_transform = BashOperator( task_id="dbt_transform", bash_command=f"dbt run --project-dir {PROJECT_ROOT}/dbt --select transform", ) checks_transform = PythonVirtualenvOperator( task_id="checks_transform", python_callable=run_soda_scan, requirements=["-i https://pypi.cloud.soda.io", "soda-postgres", "soda-scientific"], system_site_packages=False, op_kwargs={ "project_root": PROJECT_ROOT, "scan_name": "model_adventureworks_sales_category_transform", "checks_subpath": "transform" }, ) dbt_report = BashOperator( task_id="dbt_report", bash_command=f"dbt run --project-dir {PROJECT_ROOT}/dbt --select report", ) checks_report = PythonVirtualenvOperator( task_id="checks_report", python_callable=run_soda_scan, requirements=["-i https://pypi.cloud.soda.io", "soda-postgres", "soda-scientific"], system_site_packages=False, op_kwargs={ "project_root": PROJECT_ROOT, "scan_name": "model_adventureworks_sales_category_report", "checks_subpath": "report" }, ) publish_data = DummyOperator(task_id="publish_data") ingest_raw_data >> checks_ingest >> dbt_transform >> checks_transform >> dbt_report >> checks_report >> publish_data ``` ### Run Soda scans manually Without using an Airflow DAG, the Engineer can use Soda locally to run scans for data quality using the checks YAML files they created. 1. They use the `soda scan` command to run the ingest checks on the raw data, pointing Soda to the checks YAML files in the `ingest-checks` folder. ```shell soda scan -d soda_demo -c soda/configuration.yml soda/ingest-checks/ ``` 2. If the ingest check results pass, they run dbt to create the new `fact_product_category` dataset. ```shell dbt run --project-dir dbt --select transform ``` 3. Accordingly, they run a scan on the new dataset, pointing Soda to the checks YAML file in the `transform-checks` folder. ```shell soda scan -d soda_demo -c soda/configuration.yml soda/transform-checks/ ``` 4. If the transform check results pass, they run dbt to create the reports. ```shell dbt run --project-dir dbt --select report ``` 5. Lastely, they run a scan on the reports data, pointing Soda to the checks YAML file in the `reports-checks` folder. ```shell soda scan -d soda_demo -c soda/configuration.yml soda/resports-checks/ ``` 6. If the reports check results pass, the data is reliable enough to push to the reporting or visualization tool for consumers. > Learn more about [running Soda scans](/soda-v3/run-a-scan). ## View results and tag datasets 1. In their Soda Cloud account, the Engineer clicks **Checks** to access the **Checks** dashboard. The checks from the scan that Soda performed during the scan appear in the table where they can click each line item to learn more about the results, as in the example below.\
2. To more easily retrieve Soda scan results by dbt model, the Engineer navigates to **Datasets**, then clicks the stacked dots at the right of the `dim_product` dataset and selects **Edit Dataset**. 3. In the **Tags** field, they add a value for `fact_product_category`, the dbt model that uses this dataset, and a tag to indicate the kind of data that Soda is scanning, `raw`, `transformed` or `reporting`, then saves. They repeat these steps to add tags to all the datasets in their Soda Cloud account. 4. Navigating again to the **Datasets** page, they use the filters to display datasets according to **Tags** and **Arrival Time** to narrow the search for the most recent quality checks associated with their models which have failed or warned.\
5. After filtering the datasets according to the tags, the Engineer saves the filter setup as a **Collection** that they can revisit daily. 6. If you were in the Data Engineer's shoes, you may further wish to set up \[Slack notifications]\() for any checks that warn or fail during scans. ✨Hey, hey!✨ Now you know what it's like to add data quality checks to your production data pipeline. Huzzah! ## Go further * [Get organized](/soda-v3/collaborate) in Soda! * [Request a demo](https://www.soda.io/schedule-a-demo). Hey, what can Soda do for you? > Join the [Soda community on Slack](https://community.soda.io/slack). # Test data quality in an Azure Data Factory pipeline Use this guide to invoke Soda data quality tests in an Azure Data Factory pipeline. Use this guide as an example of how to set up Soda to run data quality tests on data in an ETL pipeline in Azure Data Factory.
## About this guide This guide offers an example of how to set up and trigger Soda to run data quality scans from an Azure Data Factory (ADF) pipeline. The Data Engineer in this example has copied data from a PostgreSQL data source to an Azure SQL Server data source and uses Soda reconciliation checks in a Synapse notebook to validate that data copied from the source to the target is the same. Next, they create a second notebook to execute Soda checks to validate the completeness of the ingested data. Finally, the Engineer generates a visualized report of the data quality results. This example uses a programmatic deployment model which invokes the Soda Python library, and uses Soda Cloud to validate a commercial usage license and display visualized data quality test results. Read more: [SodaCL reference](/soda-v3/sodacl-reference)\ Read more: [Soda reconciliation checks](/soda-v3/sodacl-reference/recon)\ Read more: [Choose a flavor of Soda](/soda-v3/quick-start-sip/setup-guide) ## Prerequisites The Data Engineer in this example has the following: * permission to configure Azure Cloud resources through the user interface * access to: * an Azure Data Factory pipeline * a Synapse workspace * a dedicated SQL pool in Synapse * a dedicated Apache Spark pool in Synapse * an external source SQL database such as PostgreSQL * an Azure Data Lake Storage account * an Azure Key Vault * The above-listed resources have permissions to interact with each other; for example the Synapse workspace has permission to fetch secrets from the Key Vault. * Python 3.8, 3.9, or 3.10 * Pip 21.0 or greater
Python versions Soda supports Soda officially supports Python versions 3.8, 3.9, and 3.10.\ Though largely funcntional, efforts to fully support Python 3.11 and 3.12 are ongoing. Using Python 3.11, some users might have some issues with dependencies constraints. At times, extra the combination of Python 3.11 and dependencies constraints requires that a dependency be built from source rather than downloaded pre-built. The same applies to Python 3.12, although there is some anecdotal evidence that indicates that 3.12 might not work in all scenarios due to dependencies constraints.
## Create a Soda Cloud account To validate your account license or free trial, Soda Library must communicate with a Soda Cloud account via API keys. You create a set of API keys in your Soda Cloud account, then use them to configure the connection to Soda Library. 1. In a browser, the engineer navigated to [cloud.soda.io/signup](https://cloud.soda.io/signup) to create a new Soda account, which is free for a 45-day trial. 2. They navigated to **your avatar** > **Profile**, then accessed the **API keys** tab and clicked the plus icon to generate new API keys. 3. They copy+pasted the API key values to their Azure Key Vault. ## Use Soda to reconcile data This example executes checks which, after a data migration, validate that the source and target data are matching. The first ADF Notebook Activity links to a notebook which contains the Soda connection details, the check definitions, and the script to run a Soda scan for data quality which executes the [reconciliation checks](/soda-v3/sodacl-reference/recon).
Download the notebook: [Soda Synapse Recon notebook](https://docs.soda.io/assets/soda-synapse-recon-notebook.ipynb) 1. In the ADF pipeline, the Data Engineer [adds a Notebook activity](https://learn.microsoft.com/en-us/azure/data-factory/transform-data-synapse-notebook#add-a-notebook-activity-for-synapse-to-a-pipeline-with-ui) for Synapse to a pipeline. In the Settings tab, they name the notebook `Reconciliation Checks`. 2. Next, in the Azure Synapse Analytics (Artifacts) tab, they create a [new Azure Synapse Analytics linked service](https://learn.microsoft.com/en-us/azure/data-factory/transform-data-synapse-notebook#azure-synapse-analytics-artifacts-settings) that serves to execute the Notebook activity. 3. In the Settings tab, they [choose the Notebook](https://learn.microsoft.com/en-us/azure/data-factory/transform-data-synapse-notebook#settings-tab) and the base parameters to pass to it. 4. The Spark Pool that runs the notebook must have the Soda Library packages it needs to run scans of the data. Before creating the notebook in the Synapse workspace, they add a `requirement.txt` file to the Spark Pool and include the following contents. Access [Spark Pool instructions](https://sqlofthenorth.blog/2021/02/26/adding-new-python-libraries-to-azure-synapse-spark-pools/#:~:text=txt%20file%20contents%20%E2%80%93,file%20and%20click%20%E2%80%9Capply%E2%80%9D.).\ Because this example runs scans on both the source (PostgreSQL) and target (SQL server) data sources, it requires two Soda Library packages. ``` --extra-index-url https://pypi.cloud.soda.io soda-postgres soda-sqlserver ``` 5. They [create a new notebook](https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-development-using-notebooks) in their Synapse workspace, then add the following contents that enable Soda to connect with the data sources, and with Soda Cloud. For the sensitive data source login credentials and Soda Cloud API key values, the example fetches the values from an Azure Key Vault. Read more: \[Integrate Soda with a secrets manager]\(#integrate-with-a-secrets-manager) ```python from notebookutils import mssparkutils config_str = f""" data_source postgres_data: type: postgres host: soda-demo.xxx.eu-west-1.rds.amazonaws.com port: 5432 username: my_user password: {mssparkutils.credentials.getSecret('soda-vault' , 'postgres-pw')} database: postgres schema: soda_demo_data_testing data_source azure_sql_data: type: sqlserver driver: ODBC Driver 18 for SQL Server host: soda.sql.azuresynapse.net port: xxxx username: my_sql_user password: {mssparkutils.credentials.getSecret('soda-vault' , 'sql-pw')} database: soda_sqlserver schema: soda_demo_data_testing soda_cloud: host: cloud.us.soda.io api_key_id: {mssparkutils.credentials.getSecret('soda-vault' , 'soda-api-key-id')} api_key_secret: {mssparkutils.credentials.getSecret('soda-vault' , 'soda-api-key-secret')} """ ``` 6. They define the SodaCL reconciliation checks inside another YAML string. The checks include [check attributes](/soda-v3/collaborate/check-attributes) which they created in Soda Cloud. When added to checks, the Data Engineer can use the attributes to filter check results in Soda Cloud, build custom views ([Collections](/soda-v3/collaborate#build-check-collections)), and stay organized as they monitor data quality in the Soda Cloud user interface. ```python check_str = """reconciliation retail_customers: label: 'Reconcile Postgres source and Azure SQL target' datasets: source: dataset: retail_customers datasource: postgres_data target: dataset: retail_customers datasource: azure_sql_data checks: - row_count diff = 0 attributes: data_quality_dimension: [Reconciliation, Volume] pipeline: ADF_pipeline_demo pipeline_stage: Migration data_domain: Sales - duplicate_count(customer_id): fail: when diff > 0 attributes: data_quality_dimension: [Reconciliation, Uniqueness] pipeline: ADF_pipeline_demo pipeline_stage: Migration data_domain: Sales - missing_count(customer_id): fail: when diff > 0 attributes: data_quality_dimension: [Reconciliation, Completeness] pipeline: ADF_pipeline_demo pipeline_stage: Migration data_domain: Sales - missing_count(country_code): fail: when diff > 0 attributes: data_quality_dimension: [Reconciliation, Completeness] pipeline: ADF_pipeline_demo pipeline_stage: Migration data_domain: Sales """ ``` 7. Finally, they define the script that runs the Soda scan for data quality, executing the reconcilation checks that validate the matching source and target data. If `scan.assert_no_checks_fail()` returns an `AssertionError` indicating that one or more checks have failed during the scan, then the Azure Data Factory pipeline halts. ```python from soda.scan import Scan scan = Scan() scan.set_data_source_name('azure_sql_data') scan.add_configuration_yaml_str(config_str) scan.set_scan_definition_name('reconciliation') scan.set_verbose(True) scan.add_sodacl_yaml_str(check_str) scan.execute() scan.assert_no_checks_fail() ``` ## Add post-ingestion checks Beyond reconciling the copied data, the Data Engineer uses SodaCL checks to gauge the completeness of data. In a new ADF Notebook Activity, they follow the same pattern as the reconciliation check notebook in which they configured connections to Soda Cloud and the data source, defined SodaCL checks, then prepared a script to run the scan and execute the checks.
> Download the notebook: [Soda Synapse Ingest notebook](https://docs.soda.io/assets/soda-synapse-ingest-notebook.ipynb) ```python ## Configure connections to the data source and Soda Cloud config_str = f""" data_source azure_sql_data: type: sqlserver driver: ODBC Driver 18 for SQL Server host: soda.sql.azuresynapse.net port: xxxx username: my_sql_user password: {mssparkutils.credentials.getSecret('soda-vault' , 'sql-pw')} database: soda_sqlserver schema: soda_demo_data_testing soda_cloud: host: cloud.us.soda.io api_key_id: {mssparkutils.credentials.getSecret('soda-vault' , 'soda-api-key-id')} api_key_secret: {mssparkutils.credentials.getSecret('soda-vault' , 'soda-api-key-secret')} """ ## Define data quality checks using Soda Checks Language (SodaCL) check_str = """checks for retail_customers: - missing_percent(customer_id): name: check completeness of customer_id fail: when > 5% - duplicate_percent(customer_id): name: check uniqueness of customer_id fail: when > 5% - missing_percent(country_code): name: check completeness of country_code fail: when > 5% """ ## Run the Soda scan from soda.scan import Scan scan = Scan() scan.set_verbose(True) scan.set_data_source_name('azure_sql_data') scan.add_configuration_yaml_str(config_str) scan.set_scan_definition_name('retail_customers_scan') scan.add_sodacl_yaml_str(check_str) scan.execute() scan.assert_no_checks_fail() ``` ## Generate a data visualization report The last activity in the pipeline is another Notebook Activity which runs a new Synapse notebook called `Report`. This notebook loads the data into a dataframe, creates a plot of the data, then saves the plot to an Azure Data Lake Storage location. > Download the notebook: [Soda Synapse Report notebook](https://docs.soda.io/assets/soda-synapse-report-notebook.ipynb) ```python # Visualize the number of customers per country # The first step loads the data from the Azure SQL database. import pandas as pd import pyodbc from notebookutils import mssparkutils server = 'soda.sql.azuresynapse.net' database = 'soda' username = 'my_sql_user' password = mssparkutils.credentials.getSecret("soda-vault" , "sql-pw") connection_string = f'DRIVER={{ODBC Driver 18 for SQL Server}};SERVER={server};DATABASE={database};UID={username};PWD={password}' conn = pyodbc.connect(connection_string) query = 'SELECT * FROM soda_demo_data_testing.retail_customer_count_by_country_code' df = pd.read_sql(query, con=conn) df.head() # The second step makes the plot. import pandas as pd import plotly.express as px fig = px.bar( df.sort_values(by=['customer_count', 'country_code'], ascending=True), x='country_code', y='customer_count', color='customer_count', title='Customer Count by Country Code', labels={'country_code': 'Country Code', 'customer_count': 'Number of Customers'} ) fig.show() # Lastly, save the plot. fig.write_html("/tmp/retail_customer_count_by_country_code_hist.html") mssparkutils.fs.cp( "file:/tmp/retail_customer_count_by_country_code_hist.html", "abfss://soda@soda.dfs.core.windows.net/Soda-in-ADF-pipeline/fig/retail_customer_count_by_country_code_hist.html" ) ``` ## Review check results in Soda Cloud After running the ADF pipeline, the Data Engineer can access their Soda Cloud account to review the check results. In the **Checks** page, they apply a filter to narrow the results to display only those associated with the Azure SQL Server data source against which Soda ran the data quality scans. Soda displays the results of the most recent scan.

## Go further * Learn more about [SodaCL checks and metrics](/soda-v3/sodacl-reference/metrics-and-checks). * Learn more about [getting organized](/soda-v3/collaborate) in Soda Cloud. * Set [notification rules](/soda-v3/collaborate/notif-rules) to receive alerts when checks fail. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Test data quality in a Dagster pipeline Use this guide as an example of how to invoke Soda data quality tests in a Dagster pipeline. Use this guide as an example for how to use Soda to test for data quality in an ETL pipeline in Dagster.
## About this guide The instructions below offer an example of how to execute several Soda Checks Language (SodaCL) tests for data quality at multiple points within a Dagster pipeline. For context, the example follows a fictional organization called Bikes 4 All that operates several bicycle retail stores in different regions. The Data Analysts at the company are struggling with their sales forecasts and reporting dashboards. The company has tasked the Data Engineering team to automate the ETL pipeline that uses Dagster and dbt to orchestrate the ingestion and transformation of data before exporting it for use by Data Analysts in their business intelligence tools. The pipeline built in an `assets.py` file in the Dagster project automates a flow which: 1. **Tests data before ingestion:** Uploaded from various stores into S3, the Data Engineers run Soda data quality checks *before* copying the data to a Redshift data source. To do so, they use Soda Library to load the files into DataFrames, then run a Soda scan for data quality to catch any issues with incomplete, missing, or invalid data early in the pipeline. For any Soda checks that fail, the team routes failed row samples, which contain sensitive data, back to their own S3 bucket to use to investigate data quality issues. 2. **Loads data:** After addressing any data quality issues in the retail data in S3, they load the data into Redshift in a staging environment. 3. **Transforms data in staging:** Using dbt, the Data Engineers build the models in a staging environment which transform the data for efficient use by the Data Analysts. 4. **Tests transformed data:** In a Soda Cloud staging environment, Data Analysts can prepare no-code checks for data quality based on their knowledge of the reports and dashboards that the data feeds. The Data Engineers use the Soda Cloud API to execute remote Soda scans for data quality that include the checks the Data Analysts defined. 5. **Transforms data in production:** After addressing any data quality issues after transformation in staging, the Data Engineers build the dbt models in the production environment. 6. **Exports data quality results:** The Data Engineers use the Soda Cloud API to load the data quality results into tables Redshift from which other BI tools can fetch data quality results. As a final step, outside the Dagster pipeline, the Data Engineers also design a dashboard in Tableau to monitor data quality status. ## Prerequisites The Data Engineers in this example uses the following: * Python 3.8, 3.9, or 3.10 * Pip 21.0 or greater * dbt-core and the required database adapter (dbt-redshift) * a Dagster account * access permission and connection credentials and details for Amazon Redshift * access permission and connection credentials and details for an Amazon S3 bucket * access to a Tableau account ## Install dbt, Dagster, and Soda Library Though listed as prerequisites, the following instructions include details for installing and initializing dbt-core and Dagster. 1. From the command-line, a Data Engineer installs dbt-core and the required database adapter for Redshift, and initializes a dbt project directory. Consult [dbt documentation](https://docs.getdbt.com/docs/core/pip-install) for details. ```shell pip install \ dbt-core \ dbt-redshift \ cd project-directory dbt init project-name ``` 2. In the same directory that contains the `dbt_project.yml`, they install and initialize the Dagster project inside the dbt project. Consult the Dagster [install](https://docs.dagster.io/getting-started/install#installing-dagster-into-an-existing-python-environment) and [new project](https://docs.dagster.io/getting-started/create-new-project) documentation for details.cd project-namepip install dagster-dbt dagster-webserver dagster-awsdagster-dbt project scaffold --project-name my-dagster-projectcd my-dagster-project 3. They install the Soda Library packages they need to run data quality scans in both Redshift and on data in DataFrames using Dask and Pandas.pip install -i soda-redshiftpip install -i soda-pandas-dask 4. They create a file in the `~/.dbt/` directory named `profiles.yml`, then add the following configuration to use dbt with Dagster. Consult the [Dagster documentation](https://docs.dagster.io/integrations/dbt/using-dbt-with-dagster-plus). ```yaml project-name: bike_retail target: dev outputs: dev: host: # DB host method: database port: 5439 schema: demo # schema name threads: 2 # threads in the concurrency of dbt tasks type: redshift # DB source connection dbname: # the target db name user: # username password: # password ``` 5. Lastly, they make sure that Dagster can read the dbt project directories in `project.py`. ```python from pathlib import Path from dagster_dbt import DbtProject dbt_project = DbtProject( project_dir=Path(__file__).joinpath("..", "..", "..").resolve(), packaged_project_dir=Path(__file__).joinpath("..", "..", "dbt-project").resolve(), ) dbt_project.prepare_if_dev() ``` ## Create and connect a Soda Cloud account To validate an account license or free trial, Soda Library must communicate with a Soda Cloud account via API keys. You create a set of API keys in your Soda Cloud account, then use them to configure the connection to Soda Library. 1. In a browser, a Data Engineer navigates to [cloud.soda.io/signup](https://cloud.soda.io/signup) to create a new Soda account, which is free for a 45-day trial. 2. They navigate to **your avatar** > **Profile**, access the **API keys** tab, then click the plus icon to generate new API keys. 3. They create a new file called `configuration.yml` in the same directory in which they installed the Soda Library packages, then copy+paste the API key values into the file according to the following configuration. This config enables Soda Library to connect to Soda Cloud via API. ```yaml soda_cloud: # For host, use cloud.us.soda.io for US regions; use cloud.soda.io for European region host: cloud.soda.io api_key_id: 5r-xxxx-t6 api_key_secret: Lvv8-xxx-xxx-sd0m ```
No Redshift connection details in the configuration.yml? Normally, when connecting Soda Library to a data source so it can run data quality scans, you must configure data source connection details in a `configuration.yml` file, as instructed in [Soda documentation](/soda-v3/data-source-reference/connect-redshift). However, in this example, because the Data Engineers need only use Soda Library to programmatically run scans on data loaded as DataFrames from S3, it is not necessary to provide the connection config details. See: [Connect Soda to Dask and Pandas](/soda-v3/data-source-reference/connect-dask). Later in this example, when the Data Engineers run Soda scans remotely, they do so via calls to Soda Cloud API endpoints. Soda Cloud is configured to connect to the Redshift data source and Soda executes the scan via the Soda-hosted Runner included out-of-the-box with a Soda Cloud account. Learn more about the [flavors of Soda](/soda-v3/quick-start-sip/setup-guide).
## Set up Soda To empower their Data Analyst colleagues to write their own no-code checks for data quality, a Data Engineer volunteers to set up Soda to: * connect to the Redshift data source that will contain the ingested data in a staging environment * discover the datasets and make them accessible by others in the Soda Cloud user interface * create check attributes to keep data quality check results organized 1. Logged in to Soda Cloud, the Data Engineer, who, as the initiator of the Soda Cloud account for the organization is automatically the Soda Admin, decides to use the out-of-the-box Soda-hosted runner made available for every Soda Cloud organization to securely connect to their Redshift data source. 2. The Data Engineer follows the guided workflow to [Add a new data source](https://docs.soda.io/soda-agent/managed-agent.html#add-a-new-data-source) to the Soda Cloud account to connect to their Redshift data source, making sure to include all datasets during [discovery](https://docs.soda.io/soda-agent/managed-agent.html#3-discover), and exclude datasets from [profiling](https://docs.soda.io/soda-agent/managed-agent.html#4-profile) to avoid exposing any customer information in the Soda Cloud UI. 3. Lastly, they follow the instructions to create [check attributes](/soda-v3/collaborate/check-attributes), which serve to label and sort check results by pipeline stage, data domain, etc. ## Write pre-ingestion SodaCL checks Before the Data Engineer loads the existing retail data from S3 to Redshift, they prepare several data quality tests using the Soda Checks Language (SodaCL), a YAML-based, domain-specific language for data reliability. Read more: [SodaCL reference](/soda-v3/sodacl-reference/metrics-and-checks) After creating a new `checks.yaml` file in the same directory in which they installed the Soda Library packages, the Data Engineer consults with their colleagues and defines the following checks for four datasets—stores, stocks, customers, and orders—being sure to add attributes to each to keep the check results organized. ```yaml checks for stores: - row_count > 0: name: Invalid row count attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Completeness data_domain: Location weight: 3 - missing_count(store_id) = 0: name: Store must have ID attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Completeness data_domain: Location weight: 2 - invalid_count(email) = 0: valid format: email attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Validity data_domain: Location weight: 1 - invalid_count(phone) = 0: valid format: phone number attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Validity data_domain: Location weight: 1 checks for stocks: - row_count > 0: attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Completeness data_domain: Product weight: 3 - values in (store_id) must exist in stores (store_id): attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Consistency data_domain: Product weight: 2 - values in (product_id) must exist in products (product_id): attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Consistency data_domain: Product weight: 2 - min(quantity) >= 0: name: No negative quantities attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Validity data_domain: Product weight: 2 checks for customers: - missing_count(phone) < 5% : name: Missing phone number attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Completeness data_domain: Product weight: 1 - missing_count(email) < 5% : name: Missing email address attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Completeness data_domain: Product weight: 1 - invalid_count(email) = 0: valid format: email attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Validity data_domain: Location weight: 1 - invalid_count(phone) = 0: valid format: phone number attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Validity data_domain: Location weight: 1 checks for orders: - failed rows: name: Shipment Late fail query: | select order_id as failed_orders from orders where shipped_date < required_date; attributes: pipeline_stage: Pre-ingestion data_quality_dimension: - Timeliness data_domain: Transaction weight: 3 ``` ## Run pre-ingestion checks In the `assets.py` file of their Dagster project, the Data Engineer begins defining the first asset under the `@asset` decorator. Consult the [Dagster documentation](https://docs.dagster.io/concepts/assets/software-defined-assets#defining-assets) for details. The first definition loads the S3 data into a DataFrame, then runs the pre-ingestion checks on the data. Because the data contains sensitive customer information, the Data Engineer also includes a [Soda custom sampler](/soda-v3/run-a-scan/failed-row-samples#configure-a-python-custom-sampler) which sends failed row samples for checks that fail to an S3 bucket instead of automatically pushing them to Soda Cloud. To execute the scan programmatically, the script references two files that Soda uses: * the `configuration.yml` file which contains the Soda Cloud API key values that Soda Library need to validate the user license before executing a scan, and * the `checks.yml` file which contains all the pre-ingestion SodaCL checks that the Data Engineer prepared. ```python import s3fs import boto3 import pandas as pd from soda.scan import Scan from soda.sampler import Sampler from soda.sampler.sample_contex import SampleContext from dagster import asset, Output, get_dagster_logger, MetaDataValue # Create a class for a Soda Custom Sampler class CustomSampler(Sampler): def store_sample(self, sample_context: SampleContext): rows = sample_context.sample.get_rows() json_data = json.dumps(rows) # Convert failed row samples to JSON exceptions_df = pd.read_json(json_data) # Create a DataFrame with failed rows samples # Define exceptions DataFrame exceptions_schema = sample_context.sample.get_schema().get_dict() exception_df_schema = [] for n in exceptions_schema: exception_df_schema.append(n["name"]) exceptions_df.columns = exception_df_schema check_name = sample_context.check_name exceptions_df['failed_check'] = check_name exceptions_df['created_at'] = datetime.now() exceptions_df.to_csv(check_name+".csv", sep=",", index=False, encoding="utf-8") bytestowrite = exceptions_df.to_csv(None).encode() # Write the failed row samples CSV file to S3 fs = s3fs.S3FileSystem(key=AWS_ACCESS_KEY, secret=AWS_SECRET_KEY) with fs.open(f's3://BUCKET-NAME/PATH/{check_name}.csv', 'wb') as f: f.write(bytestowrite) get_dagster_logger().info(f'Successfuly sent failed rows to {check_name}.csv ') @asset(compute_kind='python') def ingestion_checks(context): # Initiate the client s3 = boto3.client('s3') dataframes = {} for i, file_key in enumerate(FILE_KEYS, start=1): try: # Read the file from S3 response = s3.get_object(Bucket=BUCKET_NAME, Key=file_key) file_content = response['Body'] # Load CSV into DataFrame df = pd.read_csv(file_content) dataframes[i] = df get_dagster_logger().info(f"Successfully loaded DataFrame for {file_key} with {len(df)} rows.") except Exception as e: get_dagster_logger().error(f"Error loading {file_key}: {e}") failed_rows_cloud= 'false' # Execute a Soda scan scan = Scan() scan.set_scan_definition_name('Soda Dagster Demo') scan.set_data_source_name('soda-dagster') dataset_names = [ 'customers', 'orders', 'stocks', 'stores' ] # Add DataFrames to Soda scan in a loop try: for i, dataset_name in enumerate(dataset_names, start=1): scan.add_pandas_dataframe( dataset_name=dataset_name, pandas_df=dataframes[i], data_source_name='soda-dagster' ) except KeyError as e: get_dagster_logger().error(f"DataFrame missing for index {e}. Check if all files are loaded correctly.") # Add the configuration YAML file scan.add_configuration_yaml_file('path/config.yml') # Add the SodaCL checks YAML file scan.add_sodacl_yaml_str('path/checks.yml') if failed_rows_cloud == 'false': scan.sampler= CustomSampler() scan.execute() # Runs the scan logs = scan.get_logs_text() scan_results = scan.get_scan_results() context.log.info("Scan executed successfully.") get_dagster_logger().info(scan_results) get_dagster_logger().info(logs) scan.assert_no_checks_fail() # Terminate the pipeline if any checks fail return Output( value=scan_results, metadata={ "scan_results": MetadataValue.json(scan_results), 'logs':MetadataValue.json(logs) # Save the results as JSON }, ) ``` ## Load data into Redshift and define staging transformations After all SodaCL checks pass, indicating that the data quality is good, the next step in the Dagster pipeline loads the data from the S3 bucket into Amazon Redshift. As the Redshift data source is connected to Soda Cloud, both Data Engineers and Data Analysts in the Soda Cloud account can access the data and prepare no-code SodaCL checks to test data for quality. The Data Engineer then defines the dbt models that transform the data and which run under the `@dbt_assets` decorator in the staging environment. ```python from dagster_dbt import DbtCliResource, dbt_assets from dagster import AssetExecutionContext from .project import dbt_project # Select argument selects only models in the models/staging/ directory @dbt_assets(select='staging', manifest=dbt_project.manifest_path) def dbt_staging(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"],context=context, manifest=dbt_project.manifest_path).stream() ```
## Write post-transformation SodaCL checks With the transformed data available in Redshift in a staging environment, the Data Engineer invites their Data Analyst colleagues to define their own [no-code checks](/soda-v3/soda-cl-overview#define-sodacl-checks) for data quality. The Data Analysts in the organization know their data the best, particularly the data feeding their reports and dashboards. However, as they prefer not to write code—SQL, Python, or SodaCL—Soda Cloud offers them a UI-based experience to define the data quality tests they know are required.
When they create a check for a dataset in Soda Cloud, they also make two selections that help gather and analyze check results later: * a scan definition in which to include their check * one or more check attributes The [scan definition](/soda-v3/learning-resources/glossary#scan-definition) is what Soda uses to run regularly-scheduled scans of data. For example, a scan definition may instruct Soda to use the Soda-hosted runner connected to a Redshift data source to execute the checks associated with it every day at 07:00 UTC. Additionally, a Data Engineer can programmatically trigger a scheduled scan in Soda Cloud using the `scanDefinition` identifier; see the next step! The creator of a no-code check can select an existing scan definition, or choose to create a new one to define a schedule that runs at a different time of day, or at a different frequency. In this example, the Data Analysts creating the checks are following the Data Engineer's instruction that they use the same scan definition for their checks, `dagsterredshift_default_scan`, to facilitate running a single remote scan in the pipeline, later. The check attributes that the Data Engineer defined when they [Set up Soda](#set-up-soda) are available in the Soda Cloud user interface for Data Analysts to select when they are creating a check. For example, a missing check on the `store_id` column validates that there are no NULL values in the column. By adding four attributes to the check, the Data Analyst makes it easier for themselves and their colleagues to filter and analyze check results in Soda Cloud, and other BI tools, according to these custom attributes.
## Trigger a Soda scan via API After the Data Analysts have added the data quality checks they need to the datasets in Soda Cloud, the next step in the pipeline triggers a Soda scan of the data remotely, via the [Soda Cloud API](broken://pages/vOqf3xQITkpmDUg8wnKB). To do this, a Data Engineer uses the scan definition that the Data Analysts assigned to checks as they created them. In the Dagster pipeline, the Data Engineer adds a script to first call the Soda Cloud API to trigger a scan via the [Trigger a scan](#trigger-a-soda-scan-via-api) endpoint. Then, using the `scanID` from the response of the first call, they send a request to the [Get scan status](broken://pages/vOqf3xQITkpmDUg8wnKB) endpoint which continues to call the endpoint as the scan executes until the scan status reaches an end state that indicates that the scan completed, issued a warning, or failed to complete. If the scan completes successfully, the pipeline continues to the next step; otherwise, it trips a “circuit breaker”, halting the pipeline. ```python import base64 import requests from dagster import asset, get_dagster_logger, Failure url = 'https://cloud.soda.io/api/v1/scans' # cloud.us.soda.io for US region api_key_id = 'soda_api_key_id' api_key_secret = 'soda_api_key_secret' credentials = f"{api_key_id}:{api_key_secret}" encoded_credentials = base64.b64encode(credentials.encode('utf-8')).decode('utf-8') # Headers, including the authorization token headers = { 'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': f'Basic {encoded_credentials}' } # Data for the POST request payload = { "scanDefinition": "dagsterredshift_default_scan" } def trigger_scan(): response = requests.post(url, headers=headers, data=payload) # Check the response status code if response.status_code == 201: get_dagster_logger().info('Request successful') # Print the response content scan_id = response.headers.get('X-Soda-Scan-Id') if not scan_id: get_dagster_logger().info('X-Soda-Scan-Id header not found') raise Failure('Scan ID not found') else: get_dagster_logger().error(f'Request failed with status code {response.status_code}') raise Failure(f'Request Failed: {response.status_code}') # Check the scan status in a loop while scan_id: get_response = requests.get(f'{url}/{scan_id}', headers=headers) if get_response.status_code == 200: scan_status = get_response.json() state = scan_status.get('state') if state in ['queuing', 'executing']: # Wait for a few seconds before checking again time.sleep(5) get_dagster_logger().info(f'Scan state: {state}') # The pipeline terminates when the scan either warns or fails elif state == 'completed': get_dagster_logger().info(f'Scan: {state} successfully') break else: get_dagster_logger().info(f'Scan failed with status: {state}') raise Failure('Soda Cloud Check Failed') else: get_dagster_logger().info(f'GET request failed with status code {get_response.status_code}') raise Failure(f'Request failed: {get_response.status_code}') @asset(deps=[dbt_staging], compute_kind='python') def soda_UI_check(): trigger_scan() ``` ## Transform data in production When all the Data Analysts' checks have been executed and the results indicate that the data is sound in staging, the Data Engineer adds a step in the pipeline to perform the same transformations on data in the production environment. The production data in Redshift feeds the reports and dashboards that the Data Analysts use, who now with more confidence in the reliability of the data. ```python from dagster_dbt import DbtCliResource, dbt_assets from dagster import AssetExecutionContext from .project import dbt_project @dbt_assets(select='prod', manifest=dagsteretl_project.manifest_path) def dbt_prod(context: AssetExecutionContext, dbt: DbtCliResource): yield from dbt.cli(["build"], context=context, manifest=dagsteretl_project.manifest_path).stream() ``` ## Export data quality test results As a last step in the Dagster pipeline, the Data Engineer goes the extra mile to export data quality check results to tables in Redshift. The script again accesses the Soda Cloud API to gather results, then transforms the API data quality responses into DataFrames and writes them to Redshift. In this example, the check attributes that both the Data Engineers and Data Analysts applied to the checks they created prove useful: during export, the script adds separate columns to the tables in Redshift for the attributes' keys and values so that anyone using the data to create, say, a dashboard in Tableau, can organize the data according to attributes like Data Quality Dimension, Pipeline Stage, or Data Domain. Download this asset definition: [Export results](https://docs.soda.io/assets/soda-dagster-asset-def.ipynb) ## Create a Dagster asset job After defining all the assets for the Dagster pipeline, the Data Engineer must define the asset jobs, schedules, and resources for the Dagster and dbt assets. The `definitions.py` in the Dagster project wires everything together. Consult [Dagster documentation](https://docs.dagster.io/guides/understanding-dagster-project-files) for more information. ```python from dagster import Definitions from dagster_dbt import DbtCliResource from .project import dbt_project from dagster import Definitions, load_assets_from_modules, define_asset_job, AssetSelection, ScheduleDefinition from . import assets from dagster_aws.s3 import S3Resource # Load the assets defined in the module all_assets = load_assets_from_modules([assets]) # Define one asset job with all the assets dagster_pipeline = define_asset_job("dagster_pipeline", selection=AssetSelection.all()) # Create a schedule daily_schedule = ScheduleDefinition( name="Bikes_Pipeline", cron_schedule="0 9 * * *", job=dagster_pipeline, run_config={}, # Provide run configuration if needed execution_timezone="UTC" ) # Wire it all together, along with resources defs = Definitions( assets=[*all_assets], jobs=[dagster_pipeline], schedules=[daily_schedule], resources={ "dbt": DbtCliResource(project_dir=dbt_project), "s3": S3Resource( region_name="your-region", aws_access_key_id="your-aws-key", aws_secret_access_key="your-aws-secret", ) } ) ``` ## Review results To review check results from the latest Soda scan for data quality, along with the historical measurements for each check, both Data Analysts and Data Engineers can use Soda Cloud. They navigate to the **Datasets** page, then select a dataset from those listed to access a dataset overview page which offers info about check coverage, the dataset's health, and a list of its latest check results.
To keep sensitive customer data secure, the Data Engineers in this example chose to reroute any failed row samples that Soda *implicitly* collected for missing, validity, reference checks, and \*explicitly\* collected for failed row checks to an S3 bucket. Those with access to the bucket can review the CSV files which contain the failed row samples which can help Data Engineers investigate the cause of data quality issues.
Further, because the Data Engineer went the extra mile to export data quality check results via the Soda Cloud API to tables in Redshift, they are able to prepare a Tableau dashboard using the check attributes to present data according to Domain, Dimension, etc. To do so in Tableau, they added their data source, selected the Redshift connector, and entered the database connection configuration details. Consult [Tableau documentation](https://help.tableau.com/current/pro/desktop/en-us/examples_amazonredshift.htm) for details.
## Go further * Learn more about [integrating with Atlan](https://docs.soda.io/soda/integrate-atlan.html) to review Soda check results from within the catalog. * Learn more about [integrating with Slack](https://docs.soda.io/soda/integrate-slack.html) to set up [alert notifications](https://docs.soda.io/soda-cloud/notif-rules.html) for data quality checks that fail. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Test data quality in a Databricks pipeline Use this guide as an example of how to invoke Soda data quality tests in a Databricks pipeline. Use this guide as an example for how to set up and use Soda to test the quality of data in a Databricks pipeline. Automatically catch data quality issues after ingestion or transformation, and before using the data to train a machine learning model. > [Jump to Databricks notebooks](#invoke-soda-in-databricks-notebooks)
## About this guide The instructions below offers an example of how to execute Soda Checks Language (SodaCL) checks for data quality within a Databricks pipeline that handles data which trains a machine learning (ML) model. For context, this guide demonstrates a Data Scientist and Data Engineer working with Human Resources data to build a forecast model for employee attrition. The Data Engineer, working with a Data Scientist, uses a Databricks notebook to gather data from SQL-accessible dataset, transforms the data into the correct format for their ML model, then uses the data to train the model. Though they do not have direct access to the data to be able to resolve issues themselves, the Data Engineer can use Soda to detect data quality issues before the data model trains on poor-quality data. The pipeline the Data Engineer creates includes various SodaCL checks embedded at two stages in the pipeline: after data ingestion and after data transformation. At the end of the process, the pipeline stores the checks' metadata in a Databricks table which feeds into a data quality dashboard. The Data Engineer utilizes Databricks workflows to schedule this process on a daily basis. ## Prerequisites The Data Engineer in this example uses the following: * Python 3.8, 3.9, or 3.10 * Pip 21.0 or greater * a Databricks account * access to a Unity catalog ## Create a Soda Cloud account To validate an account license or free trial, Soda Library must communicate with a Soda Cloud account via API keys. You create a set of API keys in your Soda Cloud account, then use them to configure the connection to Soda Library. 1. In a browser, the Data Engineer navigates to [cloud.soda.io/signup](https://cloud.soda.io/signup) to create a new Soda account, which is free for a 45-day trial. 2. They navigate to **your avatar** > **Profile**, access the **API keys** tab, then click the plus icon to generate new API keys. 3. They copy+paste the API key values to a temporary, secure place in their local environment. ## Connect Soda Cloud to Soda Library and data source 1. Within Databricks, the Data Engineer creates two notebooks: * **Data Ingestion Checks**, which runs scans for data quality after data is ingested into a Unity catalog * **Input Data Checks**, which prepares data for training a machine learning model and runs data quality scans before submitting to the model for training 2. In the same directory as the Databricks notebooks, the Data Engineer creates a `soda_settings` directory to contain this configuration file, and, later, the check YAML files that Soda needs to run scans. To connect Soda to the Unity catalog, the Data Engineer prepares a `soda_conf.yml` file which stores the data source connection details. 3. To the file, they add the data source connection configuration to the Unity catalog that contains the Human Resources data the Data Engineer uses, and the Soda Cloud API key connection configuration, then they save the file. ```yaml data_source employees: type: spark method: databricks catalog: unity_catalog schema: employees host: hostname_from_Databricks_SQL_settings http_path: http_path_from_Databricks_SQL_settings token: my_access_token soda_cloud: # Use cloud.soda.io for EU region # Use cloud.us.soda.io for US region host: https://cloud.soda.io api_key_id: soda-api-key-id api_key_secret: soda-api-key-secret ``` > Read more: [Use Soda Library with Spark DataFrames on Databricks](/soda-v3/data-source-reference/connect-spark#use-soda-library-with-spark-dataframes-on-databricks) > > Read more: [How Soda works](/soda-v3/learning-resources/how-library-works) ## Write checks for data quality A check is a test that Soda executes when it scans a dataset in your data source. The `checks.yml` file stores the checks you write using the Soda Checks Language. You can create multiple checks files to organize your data quality checks and run all, or some of them, at scan time. In this example, the Data Engineer creates two checks files in the `soda_settings` directory in Databricks: * `ingestion_checks.yml` to execute quality checks after data ingestion into the Unity catalog in the Data Ingestion Checks notebook * `input_data_checks.yml` to execute quality checks after transformation, and before using it to train their ML model in the Input Data Checks notebook. * `output_data_checks.yml` to execute quality checks after training the model and monitor the performance of your model. The raw data in this example is divided into two main categories. * The first category is Human Resources data, which the Unity catalog contains in three datasets: basic employee information, results of manager surveys, and results of employee surveys. The survey datasets are updated on a frequent basis. * The second category is application login data, which is a file in the Databricks file system; it is updated daily. > Download: [employee\_info\_sample.csv](https://docs.soda.io/assets/employee_info_sample.csv) > > Read more: [SodaCL reference](/soda-v3/sodacl-reference/metrics-and-checks) ### Post-ingestion checks The Data Engineer creates a checks YAML file to write checks that apply to the datasets they use to train their ML model. The Data Ingestion Checks notebook runs these checks after the data is ingested into the Unity catalog. For any checks that fail, the Data Engineer can notify upstream Data Engineers or Data Product Owners to address issues such as missing data or invalid entries. Many of the checks that the Data Engineer prepares include [check attributes](/soda-v3/collaborate/check-attributes) which they created in Soda Cloud; see image below. When added to checks, the Data Engineer can use the attributes to filter check results in Soda Cloud, build custom views ([Collections](/soda-v3/collaborate#build-check-collections)), and stay organized as they monitor data quality in the Soda Cloud user interface. Skip to [Review check results](#review-check-results-in-soda-cloud) to see an example.
The Data Engineer also added a [dataset filter](/soda-v3/sodacl-reference/filters#configure-dataset-filters) to the quality checks that apply to the application login data. The filter serves to partition the data against which Soda executes the checks; instead of checking for quality on the entire dataset, the filter limits the scan to the previous day’s data. ingestion\_checks.yml ```yaml checks for employee_info: - invalid_count(Department) = 0: valid values: ['Sales', 'Research & Development', 'Human Resources'] name: Only correct departments are present in the dataset attributes: dimension: [Validity] pipeline_stage: Ingest team: Data Engineering - missing_count(EmployeeID) = 0: name: No null values in the Employee ID column attributes: dimension: [Completeness] pipeline_stage: Ingest team: Data Engineering - duplicate_count(EmployeeID) = 0: name: No duplicate IDs attributes: dimension: [Uniqueness] pipeline_stage: Ingest team: Data Engineering - invalid_count(Gender) = 0: valid values: ['Female', 'Male', 'Non-binary'] name: Value for gender is valid attributes: dimension: [Validity] pipeline_stage: Ingest team: Data Engineering - invalid_count(Age) = 0: valid min: 18 name: All employees are over 18 attributes: dimension: [Validity] pipeline_stage: Ingest team: Data Engineering - missing_count(MonthlyIncome) = 0: name: No null values in MonthlyIncome attributes: dimension: [Completeness] pipeline_stage: Ingest team: Data Engineering - failed rows: name: Monthly Salary equals or exceeds legally required salary fail condition: MonthlyIncome < 11000 - schema: warn: when schema changes: any name: Columns have not been added, removed, or changed attributes: dimension: [Consistency] pipeline_stage: Ingest team: Data Engineering checks for employee_survey: - invalid_count(EnvironmentSatisfaction) = 0: valid min: 1 valid max: 5 name: Values are formatted in range 1-5 attributes: dimension: [Validity] pipeline_stage: Ingest team: Data Engineering - missing_count(EmployeeID) = 0: name: No null values in Employee ID attributes: dimension: [Completeness] pipeline_stage: Ingest team: Data Engineering - duplicate_count(EmployeeID) = 0: name: No duplicate IDs attributes: dimension: [Uniqueness] pipeline_stage: Ingest team: Data Engineering - invalid_count(WorkLifeBalance) = 0: valid min: 1 valid max: 5 name: Values are formatted in range 1-5 attributes: dimension: [Validity] pipeline_stage: Ingest team: Data Engineering - schema: warn: when schema changes: any name: Columns have not been added, removed, or changed - values in EmployeeID must exist in employee_info EmployeeID: name: EmployeeID Integrity Check for employee survey checks for manager_survey: - invalid_count(PerformanceRating) = 0: valid min: 1 valid max: 5 name: Values are formatted in range 1-5 attributes: dimension: [Validity] pipeline_stage: Ingest team: Data Engineering - schema: warn: when schema changes: any name: Columns have not been added, removed, or changed attributes: dimension: [Consistency] pipeline_stage: Ingest team: Data Engineering - values in EmployeeID must exist in employee_info EmployeeID: name: EmployeeID integrity check for manager survey # This filter partitions data included in the quality scan # because the data in the dataset lags by one day filter login_logout [daily]: where: LogoutTime < CAST(current_date() AS TIMESTAMP) - INTERVAL 1 DAY AND LoginTime > CAST(current_date() AS TIMESTAMP) - INTERVAL 2 DAY checks for login_logout [daily]: - invalid_count(LoginTime): valid regex: "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(\\.\\d+)?(Z|[+-]\\\ d{2}:\\d{2})$" name: Login time format fail: when > 0 attributes: dimension: [Validity] pipeline_stage: Ingest team: Data Engineering - missing_count(LogoutTime) = 0: name: No nulls in LogoutTime attributes: dimension: [Completeness] pipeline_stage: Ingest team: Data Engineering - values in EmployeeID must exist in employee_info EmployeeID: name: EmployeeID Integrity Check for login times - freshness(LogoutTime) < 2d: name: Data is updated attributes: dimension: [Timeliness] pipeline_stage: Ingest team: Data Engineering ``` ## Post-transformation checks The Data Engineer also prepared a second set of SodaCL checks in a separate file to run after transformation in the Input Data Checks notebook. Curious readers can download the [ETL notebook.ipynb](https://docs.soda.io/assets/Data%20Ingestion%20Checks.ipynb) to review transformations and the resulting `input_data_attrition_model` output into a DataFrame. Two of the checks the Data Engineer prepares involve checking groups of data. The [group evolution check](/soda-v3/sodacl-reference/group-evolution) validates the presence or absence of a group in a dataset, or to check for changes to groups in a dataset relative to their previous state; in this case, it confirms the presence of the `Married` group in the data, and when any group changes. Further, the [group by check](/soda-v3/sodacl-reference/group-by) collects and presents check results by category; in this case, it groups the results according to `JobLevel`. input\_data\_checks.yml ```yaml filter input_data_attrition_model [daily]: where: PartitionDate < CAST(current_date() AS TIMESTAMP) - INTERVAL 1 DAY AND PartitionDate > CAST(current_date() AS TIMESTAMP) - INTERVAL 2 DAY checks for input_data_attrition_model [daily]: - missing_count(Attrition)=0:: name: Target value is not missing attributes: pipeline: Transform team: Data Science dimension: [Completeness] - invalid_percent(TotalWorkingYears): valid min: 0 name: Working years can't be negative warn: when > 0% fail: when > 10% attributes: pipeline: Transform team: Data Science attribute: [Validity] - values in EmployeeID must exist in employee_info EmployeeID: name: EmployeeID Integrity Check - failed rows: name: Overtime detected fail query: | SELECT * FROM input_data_attrition_model WHERE WorkingMinutes > 750 attributes: pipeline: Transform team: Data Science - freshness(PartitionDate) < 2d: name: Data is fresh attributes: pipeline: Transform team: Data Science dimension: [Timeliness] - group evolution: name: Marital status query: | SELECT MaritalStatus FROM input_data_attrition_model GROUP BY 1 fail: when required group missing: [Married] warn: when groups change: any attributes: pipeline: Transform team: Data Science dimension: [Consistency] - group by: query: | SELECT JobLevel, min(MonthlyIncome) AS salary FROM input_data_attrition_model GROUP BY 1 fields: - JobLevel checks: - salary: warn: when < 0 fail: when < -1 name: Min Salary Normalised cannot be below -1 attributes: pipeline: Transform team: Data Science dimension: [Accuracy] ``` ## Invoke Soda in Databricks notebooks At the [beginning](#connect-soda-cloud-to-soda-library-and-data-source) of this exercise, the Data Engineer created two notebooks in their Databricks workflow: * **Data Ingestion Checks** to run after data is ingested into the Unity catalog * **Input Data Check** to run after transformation, and before using the data to train the ML model The following outlines the contents of each notebook and the steps included to install Soda and invoke it to run scans for data quality, thereby executing the data quality checks in the checks YAMLfiles. Beyond invoking Soda to scan for data quality, the notebooks also save the checks' metadata for further analysis. #### Data ingestion checks > Download: [Data Ingestion Checks.ipynb](https://docs.soda.io/assets/Data%20Ingestion%20Checks.ipynb) ```python # Install to run checks contained in files pip install -i https://pypi.cloud.soda.io soda-spark-df # Import Scan from Soda Library from soda.scan import Scan import yaml from io import StringIO from pathlib import Path from datetime import datetime, timedelta # Define file directory settings_path = Path('/Workspace/Users/my_user_id/employee_attrition/soda_settings') # Define results file directory result_path = Path('/Workspace/Users/my_user_id/employee_attrition/checks_output') # Define the file partition partition = (datetime.today().date() - timedelta(days=1)).strftime("%Y-%m-%d") # Create a scan object scan = Scan() # Set scan name and data source name scan.set_scan_definition_name("Employee Attrition Scan") scan.set_data_source_name("employees") # Add file to be scanned df = spark.read.option("header", True).csv(f"dbfs:/Workspace/Users/my_user_id/employee_attrition/soda_settings/login_logout/PartitionDate={partition}") # Create temporary View to run the checks df.createOrReplaceTempView("login_logout") # Function to create temporary views of the tables to be included in the same scan def create_temp_views(spark, schema, table_names): for table in table_names: full_table_name = f"{schema}.{table}" df = spark.table(full_table_name) df.createOrReplaceTempView(table) # Create the temp view from the table list schema = "unity_catalog.employees" table_names = ["employee_info", "employee_survey", "manager_survey"] create_temp_views(spark, schema, table_names) # Add Views to the scan object scan.add_spark_session(spark, data_source_name="employees") # Access the checks YAML file with open(settings_path/"ingestion_checks.yml") as ing_checks: ingestion = ing_checks.read() # Create a file-like object from the YAML content ingestion_checks = StringIO(ingestion) # Use the scan.add_sodacl_yaml method to retrieve the checks scan.add_sodacl_yaml_str(ingestion_checks) # Retrieve the configuration file and use the scan.add_sodacl_yaml method with open(settings_path/"soda_conf.yml") as cfg: cfg_content = cfg.read() # Create a file-like object from the YAML content conf = StringIO(cfg_content) # Add the data source connection configuration to the scan scan.add_configuration_yaml_str(conf) # Execute the scan scan.execute() # Check the Scan object for methods to inspect the scan result; print all logs to console print(scan.get_logs_text()) # Save the checks metadata for further analysis metadata = scan.build_scan_results() scan_date = datetime.now().date().strftime("%Y-%m-%d") scan.save_scan_result_to_file(result_path/f"ingestion_result_{scan_date}.json", metadata['checks']) ``` #### Input data checks and model output checks > Download: [Input Data Checks.ipynb](https://docs.soda.io/assets/Input%20Data%20Checks.ipynb) ```python # Install to run checks on data in Unity datasets pip install -i https://pypi.cloud.soda.io soda-spark[databricks] #restart to use updated packages %restart_python # Import Scan from Soda Library from soda.scan import Scan import yaml from io import StringIO from pathlib import Path # Define file directory settings_path = Path('/Workspace/Users/my_user_id/employee_attrition/soda_settings') # Create a scan object scan = Scan() # Set scan name and data source name scan.set_scan_definition_name("Attrition Model - Input Data Checks") scan.set_data_source_name("employee_info") # Attach a Spark session scan.add_spark_session(spark) # Access the checks YAML file with open(settings_path/"input_data_checks.yml") as input_checks: input_data = input_checks.read() # Create a file-like object from the YAML content input_data_checks = StringIO(input_data) # Use the scan.add_sodacl_yaml method to retrieve the checks scan.add_sodacl_yaml_str(input_data_checks) # Retrieve the configuration file and use the scan.add_sodacl_yaml method with open(settings_path/"soda_conf.yml") as cfg: cfg_content = cfg.read() # Create a file-like object from the YAML content conf = StringIO(cfg_content) # Add the connection configuration to the scan scan.add_configuration_yaml_str(conf) scan.execute() # Check the Scan object for methods to inspect the scan result; print all logs to console print(scan.get_logs_text()) ``` Using the same structure the data scientists define some extra checks to validate and monitor the performance of their model after training. They define a ratio between the categories and apply an anomaly detection to make sure that there are no spikes or unexpected swifts in the label distribution. Furthermore, they add a check to ensure that they will notified when the model accuracy is below 60% and/or when the dataset is incomplete. model\_output\_checks.yml ```yaml discover datasets: datasets: - attrition_model_output profile columns: columns: - include attrition_model_output.% filter attrition_model_output [daily]: where: PartitionDate < CAST(current_date() AS TIMESTAMP) - INTERVAL 1 DAY AND PartitionDate > CAST(current_date() AS TIMESTAMP) - INTERVAL 2 DAY checks for attrition_model_output [daily]: - row_count > 0: name: Dataset cannot be empty attributes: pipeline_stage: Training team: Data Science dimension: [Completeness] - missing_count(Attrition) = 0: name: Attrition field is not completed attributes: pipeline_stage: Training team: Data Science dimension: [Completeness] - avg(Accuracy): name: Accuracy is not below 60% fail: when < 0.60 warn: when < 0.70 attributes: pipeline_stage: Training team: Data Science dimension: [Accuracy] - anomaly detection for attrition_ratio: name: Attrition ratio anomaly detection attrition_ratio query: | SELECT (COUNT(CASE WHEN Attrition = true THEN 1 END) * 1.0) / COUNT(*) AS attrition_ratio FROM attrition_model_output attributes: pipeline_stage: Training team: Data Science dimension: [Accuracy] ``` ## Review check results in Soda Cloud After running the notebooks, the Data Engineer accesses Soda Cloud to review the check results. In the **Checks** page, they apply filters to narrow the results to the datasets involved in the Employee Attrition ML model, and distill the results even further by selecting to display only those results with the Pipeline attribute of `Ingest`. They save the results as a Collection labeled **Employee Attrition - Ingestion** to easily access the relevant quality results in the future.
## Review check results in a Unity dashboard After the Data Engineer trains the model to forecast employee attrition, they decide to devise an extra step in the process to use the [Soda Cloud API](broken://pages/vOqf3xQITkpmDUg8wnKB) export all the Soda check results and dataset metadata back into the Unity catalog, then build a dashboard to display the results. {% hint style="info" %} *Coming soon:* a tutorial for building a dashboard using the Soda Cloud API. {% endhint %}
## Go further * Learn more about [SodaCL metrics and checks](/soda-v3/sodacl-reference/metrics-and-checks). * Learn more about [getting organized](/soda-v3/collaborate) in Soda Cloud. * Set [notification rules](/soda-v3/collaborate/notif-rules) to receive alerts when checks fail. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Test data quality before migration Use this guide to set up Soda to check data quality before and after migrating data between data sources. Use this guide to install and set up Soda to test the quality in a data migration project. Test data quality at both source and target, both before and after migration to prevent data quality issues from polluting a new data source. > Not quite ready for this big gulp of Soda? 🥤Try [taking a sip](/soda-v3/quick-start-sip), first.
## About this guide The instructions below offer Data Engineers an example of how to set up Soda and use reconciliation checks to compare data quality between data sources before and after migrating data. For context, this guide presents an example of how you could use Soda to prepare to migrate data from one data source, such as PostgreSQL to another, such as Snowflake. It makes suggestions about how to prepare for a data migration project and use a staging environment to validate data quality before migrating data in production. This example uses a self-operated deployment model which uses Soda Library and Soda Cloud, though you could as easily use a self-hosted runner model (Soda Runner and Soda Cloud) instead. ## Prepare for data migration This example imagines moving data from PostgreSQL to Snowflake. The following outlines the high level steps involved in preparing for and executing such a project. 1. Confirm your access to the source data in a PostgreSQL data source; you have the authorization and access credentials to query the data. 2. Set up or confirm that you have a Snowflake account and the authorization and credentials to set up and query a new data source. 3. Confirm that you have a data orchestration tool such as Airflow to extract data from PostgreSQL, perform any transformations, then load the data into Snowflake. Reference [Migrating data using Airflow](https://medium.com/data-engineering-indonesia/migrating-data-from-sqlite-to-snowflake-using-airflow-90f5e980d88f) for an Airflow setup example. 4. [Install and set up Soda](#install-and-set-up-soda) to perform preliminary tests for data quality in the source data. Use this opportunity to make sure that the quality of the data you are about to migrate is in a good state. Ideally, you perform this step in a production environment, before replicating the source data source in a staging environment to ensure that you begin the project with good-quality data. 5. You have backed up the existing data in the PostgreSQL source data source, and created a staging environment which replicates the production PostgreSQL data source. 6. Use Airflow to execute the data migration from PostgreSQL to Snowflake in a staging environment. 7. In the staging environment, use Soda to run reconciliation checks on both the source and target data sources to validate that the data has been transformed and loaded as expected, and the quality of data in the target is sound. 8. Adjust your data transformations as needed in order to address any issues surfaced by Soda. Repeat the data migration in staging, checking for quality after each run, until you are satisfied with the outcome and the data that loads into the target Snowflake data source. 9. Prepare an Airflow DAG to execute the data migration in production. Execute the data migration in production, then use Soda to scan for data quality on the target data source for final validation. 10. (Optional) For regular migration events, consider invoking Soda scans for data quality after extraction and transformation(s) in the DAG. ## Install and set up Soda What follows is an abridged version of installing and configuring Soda for PostgreSQL. Refer to [full installation instructions](https://docs.soda.io/soda-library/install.html) for details. 1. In a browser, navigate to [cloud.soda.io/signup](https://cloud.soda.io/signup) to create a new Soda account, which is free for a 45-day trial. If you already have a Soda account, log in. 2. Navigate to **your avatar** > **Profile**, then access the **API keys** tab. Click the plus icon to generate new API keys. Copy+paste the API key values to a temporary, secure place in your local environment. 3. With Python 3.8, 3.9, or 3.10 and Pip 21.0 or greater, use the command-line to install Soda locally in a new virtual environment. ```shell python3 -m venv .venv source .venv/bin/activate pip install -i https://pypi.cloud.soda.io soda-postgres ``` 4. In a code editor, create a new file called `configuration.yml`, then copy paste the following config details into the file. Provide your own values for the fields, using your own API key and secret values you created in Soda Cloud. Replace the value of `my_database_name` with the name of your PostgreSQL data source. ```yaml data_source my_database_name: type: postgres host: port: username: password: database: schema: soda_cloud: # For US region, use cloud.us.soda.io # For EU region, use cloud.soda.io host: cloud.soda.io api_key_id: api_key_secret: ``` 5. Save the file. From the command-line, in the same directory in which you created the `configuration.yml` run the following command to test Soda's connection to your data source. Replace the value of `my_datasource` with the name of your own PostgreSQL data source. ```shell soda test-connection -d my_datasource -c configuration.yml ``` 6. To create some basic checks for data quality, run the following command to launch Check Suggestions which auto-generates checks using the Soda Checks Language (SodaCL), a domain-specific language for data quality testing. * Identify one dataset in your data source to use as the value for the `-ds` option in the command below. * Replace the value of `my_datasource` with the name of your own PostgreSQL data source. * Answer the prompts in the command-line and, at the end, select `y` to run a scan using the suggested checks. ``` soda suggest -d my_datasource -c configuration.yml -ds your_dataset_name ``` 7. In a browser, log in to your Soda Cloud account, then navigate to the **Checks** dashboard. Here, you can review the results of the checks that Soda executed in the first scan for data quality. After a scan, each check results in one of three default states: * pass: the values in the dataset match or fall within the thresholds you specified * fail: the values in the dataset do not match or fall within the thresholds you specified * error: the syntax of the check is invalid, or there are runtime or credential errors 8. Based on the check results from the first scan, address any data quality issues that Soda surfaced so that your data migration project begins with good-quality data. Refer to [Run a scan and review results](/soda-v3/run-a-scan) for much more detail. 9. If you wish, open the checks.yml that the check suggestions command saved locally for you and add more checks for data quality, then use the following command to run the scan again. Refer to [SodaCL reference](/soda-v3/sodacl-reference) for exhaustive details on all types of checks. ```shell soda scan -d my_datasource -c configuration.yml checks.yml ``` ## Migrate data in staging 1. Having tested data quality on the PostgreSQL data source, best practice dictates that you back up the existing data in the PostgreSQL data source, then replicate both the PostgreSQL and an empty Snowflake data source in a staging environment. 2. As in the example that follows, add two more configurations to your `configuration.yml` for: * the PostgreSQL staging data source * the Snowflake staging data source ````yaml data_source fulfillment_apac_prod: type: postgres host: 127.0.0.1 port: '5432' username: ${POSTGRES_USER} password: ${POSTGRES_PASSWORD} database: postgres schema: public data_source fulfillment_apac_staging: type: postgres host: localhost port: '5432' username: ${POSTGRES_USER} password: ${POSTGRES_PASSWORD} database: postgres schema: public data_source fulfillment_apac1_staging: type: snowflake username: ${SNOWFLAKE_USER} password: ${SNOWFLAKE_USER} account: my_account database: snowflake_database warehouse: snowflake_warehouse connection_timeout: 240 role: PUBLIC client_session_keep_alive: true authenticator: externalbrowser session_parameters: QUERY_TAG: soda-queries QUOTED_IDENTIFIERS_IGNORE_CASE: false schema: public ``` ```` 3. Run the following commands to test the connection to each new data source in the staging environment. ```shell soda test-connection -d fulfillment_apac_staging -c configuration.yml soda test-connection -d fulfillment_apac1_staging -c configuration.yml ``` 4. Using an orchestrator such as Airflow, migrate your data in the staging environment from PostgreSQL to Snowflake, making any necessary transformations to your data to populate the new data source. Reference [Migrating data using Airflow](https://medium.com/data-engineering-indonesia/migrating-data-from-sqlite-to-snowflake-using-airflow-90f5e980d88f) for an Airflow setup example. ## Reconcile data and migrate in production 1. With both source and target data sources, you can use [SodaCL reconciliation checks](/soda-v3/sodacl-reference/recon) to compare the data in the target to the source to ensure that it is expected and free of data quality issues.\ Begin by using a code editor to prepare a `recon.yml` file in the same directory as you installed Soda, as per the following example which identifies the source and target datasets to compare, and defines basic checks to compare schemas and row counts. ```yaml reconciliation OrdersAPAC: label: "Recon APAC orders" datasets: source: dataset: orders_apac datasource: fulfillment_apac_staging target: dataset: orders_apac datasource: fulfillment_apac1_staging checks: - schema - row_count diff = 0 ``` 2. Referencing the checks that checks suggestions created, add corresponding [metric reconciliation checks](/soda-v3/sodacl-reference/recon#metric-reconciliation-checks) to the file to surface any delta between the metrics Soda measures for the source and the measurements it collects for the target. Refer to the [list of metrics and checks](/soda-v3/sodacl-reference/recon#list-of-compatible-metrics-and-checks-for-metric-reconciliation-checks) that are available as reconciliation checks.\ Examples of `checks.yml` and `recon.yml` files follow. ```yaml # checks.yml prepared by check suggestions filter dim_product [daily]: where: start_date > TIMESTAMP'${NOW}' - interval '1d' checks for dim_product [daily]: - schema: name: Any schema changes fail: when schema changes: - column delete - column add - column index change - column type change - row_count > 0 - anomaly detection for row_count - freshness(start_date) < 398d - missing_count(weight_unit_measure_code) = 0 - missing_count(color) = 0 - duplicate_count(safety_stock_level) = 0 ``` ```yaml # recon.yml reconciliation OrdersAPAC: label: "Recon datasets" ... checks: - schema - row_count diff = 0 - freshness(start_date) diff = 0 - missing_count(weight_unit_measure_code) diff = 0 - missing_count(color) diff = 0 - duplicate_count(safety_stock_level): fail: when diff > 10 warn: when diff between 5 and 9 ``` 3. Run a scan to execute the checks in the `recon.yml` file. When you run a scan against either the source or target data source, the `Scan summary` in the output indicates the check value, which is the calculated delta between measurements, the measurement value of each metric or check for both the source and target datasets, along with the diff value and percentage, and the absolute value and percentage. Review the results Soda Library produces in the command-line and/or in the **Checks** dashboard in Soda Cloud. ```shell soda scan -d fulfillment_apac_staging -c configuration.yml recon.yml ``` 4. Based on the scan results, make adjustments to the transformations in your orchestrated flow and repeat the scans, adding more metric reconciliation checks needed. 5. Compare more source and target datasets by adding more reconciliation blocks to the `recon.yml` file. Tip: You can run check suggestions against new datasets and use those checks as a baseline for writing metric reconciliation checks for other datasets in your data source. ```yaml reconciliation OrdersAPAC: label: "Recon APAC orders" datasets: source: dataset: orders_apac datasource: fulfillment_apac_staging target: dataset: orders_apac datasource: fulfillment_apac1_staging checks: - schema - row_count diff = 0 reconciliation DiscountAPAC: label: "Recon APAC discount" datasets: source: dataset: discount_apac datasource: fulfillment_apac_staging target: dataset: discount_apac datasource: fulfillment_apac1_staging checks: - schema - row_count diff = 0 ``` 6. After reconciling metrics between multiple datasets, consider writing more granular [record reconciliation checks](/soda-v3/sodacl-reference/recon#record-reconciliation-checks) for the most critical data, as in the example below. As these checks execute a row-by-row comparison of data in a dataset, they are resource-heavy relative to metric and schema reconciliation checks. However, for the datasets that matter most, the resource usage is warranted to ensure that the data you migrate remains intact and as expected in the target data source. ```yaml reconciliation CommissionAPAC: label: "Recon APAC commission" datasets: source: dataset: commission_apac datasource: fulfillment_apac_staging target: dataset: commission_apac datasource: fulfillment_apac1_staging checks: - rows diff = 0 ``` 7. After reviewing multiple scan results and correcting any reconciliation issues between source and target datasets, you can execute the migration in production.\ After the migration, use the same `recon.yml` file to run a scan on the migrated data in production to confirm that the data in the target is as expected. Adjust the `soda scan` command to run against your production data source instead of the staging data source. ```shell soda scan -d fulfillment_apac1_prod -c configuration.yml recon.yml ``` 8. (Optional) If you intend to execute the migration of data between data sources frequently, you may wish to invoke Soda scan with the reconciliation checks programmatically within your pipeline orchestration, such as in your Airflow DAG. To access an example of how to include Soda scans in your DAG, see [Test data in production](/soda-v3/use-case-guides/quick-start-prod). ## Go further * Learn more about [reconciliation checks](/soda-v3/sodacl-reference/recon) in general. * Write reconciliation checks that produce [failed row samples](/soda-v3/run-a-scan/failed-row-samples) in Soda Cloud to help you investigate the root cause of data quality issues. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Test data quality during CI/CD development Follow this guide to set up and run automated Soda scans for data quality during CI/CD development using GitHub Actions. Use this guide to install and set up Soda to test the quality of your data during your development lifecycle. Catch data quality issues in a GitHub pull request before merging data management changes, such as transformations, into production. > Not quite ready for this big gulp of Soda? 🥤Try [taking a sip](/soda-v3/quick-start-sip), first.
## About this guide The instructions below offer Data Engineers an example of how to use the [Soda Library Action](https://github.com/marketplace/actions/soda-library-action) to execute SodaCL checks for data quality on data in a Snowflake data source. For context, the example assumes that a team of people use GitHub to collaborate on managing data ingestion and transformation with dbt. In the same repo, team members collaborate to write tests for data quality in SodaCL checks YAML files. With each new pull request, or commit to an existing one, in the repository that adds a transformation or makes changes to a dbt model, the GitHub Action in Workflow executes a Soda scan for data quality and presents the results of the scan in a comment in the pull request, and in Soda Cloud. Where the scan results indicate an issue with data quality, Soda notifies the team via a notification in Slack so that they can investigate and address any issues before merging the PR into production. Borrow from this guide to connect to your own data source, add the GitHub Action for Soda to a Workflow, and execute your own relevant tests for data quality to prevent issues in production. ## Add the GitHub Action for Soda to a Workflow 1. In a browser, navigate to [cloud.soda.io/signup](https://cloud.soda.io/signup) to create a new Soda account, which is free for a 45-day trial. If you already have a Soda account, log in. 2. Navigate to **your avatar** > **Profile**, then access the **API keys** tab. Click the plus icon to generate new API keys. Copy+paste the API key values to a temporary, secure place in your local environment.
Why do I need a Soda Cloud account? To validate your account license or free trial, the Soda Library Docker image that the GitHub Action uses to execute scans must communicate with a Soda Cloud account via API keys.\ Create new API keys in your Soda Cloud account, then use them to configure the connection between the Soda Library Docker image and your account later in this procedure.
3. In the GitHub repository in which you wish to include data quality scans in a Workflow, create a folder named `soda` for the configuration files that Soda requires as input to run a scan. 4. In this folder, create two files: * a `configuration.yml` file to store the connection configuration Soda needs to connect to your data source and your Soda Cloud account. * a `checks.yml` file to store the SodaCL checks you wish to execute to test for data quality; see [next section](#write-checks-for-data-quality). 5. Follow the [instructions](/soda-v3/quick-start-sip/install#configure-soda) to add connection configuration details for both your data source and your Soda Cloud account to the `configuration.yml`, as per the example below. ```yaml data_source my_datasource_name: type: snowflake username: ${ SNOWFLAKE_USER } password: ${ SNOWFLAKE_PASS } account: ${ SNOWFLAKE_ACCOUNT } database: sodadata_test warehouse: compute_wh role: analyst session_parameters: QUERY_TAG: soda-queries QUOTED_IDENTIFIERS_IGNORE_CASE: false schema: public soda_cloud: host: cloud.us.soda.io api_key_id: ${ SODA_CLOUD_API_KEY } api_key_secret: ${ SODA_CLOUD_API_SECRET } ``` 6. In the `.github/workflows` folder in your GitHub repository, open an existing Workflow or [create a new workflow](https://docs.github.com/en/actions/using-workflows/about-workflows#create-an-example-workflow) file. 7. In your browser, navigate to the GitHub Marketplace to access the [Soda Library Action](https://github.com/marketplace/actions/soda-library-action). Click **Use latest version** to copy the code snippet for the Action. 8. Paste the snippet into your new or existing workflow as an independent step, then add the required action inputs and environment variable as in the following example. * Be sure to add the Soda Action *after* the step in the workflow that completes a dbt run that executes your dbt tests. * Best practice dictates that you configure sensitive credentials using GitHub secrets. Read more about [GitHub encrypted secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets).
```yaml # This GitHub Action runs a Soda scan on a Snowflake data source called reporting_api_marts. name: Run Soda Scan on [reporting_api_marts] # GitHub triggers this job when a user creates or updates a pull request. on: pull_request jobs: soda_scan: runs-on: ubuntu-latest name: Run Soda Scan steps: - name: Checkout uses: actions/checkout@v3 - name: Perform Soda Scan uses: sodadata/soda-github-action@main env: SODA_CLOUD_API_KEY: ${{ secrets.SODA_CLOUD_API_KEY }} SODA_CLOUD_API_SECRET: ${{ secrets.SODA_CLOUD_API_SECRET }} SNOWFLAKE_USERNAME: ${{ secrets.SNOWFLAKE_USERNAME }} SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }} with: soda_library_version: v1.0.4 data_source: snowflake_api_marts configuration: ./soda/configuration.yml checks: ./soda/checks.yml ``` 9. Save the changes to your workflow file. ## Write checks for data quality A check is a test that Soda executes when it scans a dataset in your data source. The `checks.yml` file stores the checks you write using the [Soda Checks Language (SodaCL)](/soda-v3/soda-cl-overview). You can create multiple `checks.yml` files to organize your data quality checks and run all, or some of them, at scan time. 1. In your `soda` folder, open the `checks.yml` file, then copy and paste the following rather generic checks into the file. * Replace the value of `dataset_name` with the name of a dataset in your data source. * Replace the value of `column1` with the name of a column in the dataset.\ `yaml checks for dataset_name: # Checks that dataset contains rows - row_count > 0: name: Dataset contains data # Checks that column contains no NULL values - missing_count(column1) = 0: name: No NULL values` 2. Save the `checks.yml` file. ## Trigger a scan and examine the scan results To trigger the GitHub Action and initiate a Soda scan for data quality, create a new pull request in your repository. Be sure to trigger a Soda scan *after* the step in your Workflow that completes the dbt run that executed your dbt tests.
What does the GitHub Action do? To summarize, the action completes the following tasks: 1. Checks to validate that the required Action input values are set. 2. Builds a Docker image with a specific Soda Library version for the base image. 3. Expands the environment variables to pass to the Docker run command as these variables can be configured in the workflow file and contain secrets. 4. Runs the built image to trigger the Soda scan for data quality. 5. Converts the Soda Library scan results to a markdown table using newest hash from 1.0.0 version. 6. Creates a pull request comment. 7. Posts any additional messages to make it clear whether or not the scan failed. See the public [soda-github-action](https://github.com/sodadata/soda-github-action) repository for more detail.
1. For the purposes of this exercise, create a new branch in your GitHub repo, then make a small change to an existing file and commit and push the change to the branch. 2. Execute a [dbt run](https://docs.getdbt.com/reference/commands/run). 3. Create a new pull request, then navigate to your GitHub account and review the pull request you just created. Notice that the Soda scan action is queued and perhaps already running against your data to check for quality. 4. When the job completes, navigate to the pull request's **Conversation** tab to view the comment the Action posted via the github-action bot. The table indicates the states and volumes of the check results.x\
5. To examine the full scan report and troubleshoot any issues, click the link in the comment to **View full scan results**, then click **View Scan Log**. Use \[Troubleshoot SocaCL]\() for help diagnosing issues.
✨Well done!✨ You've taken the first step towards a future in which you and your colleagues prevent data quality issues from getting into production. Huzzah! ## Go further * [Get organized](https://docs.soda.io/soda-cloud/collaborate.html) in Soda! * [Request a demo](https://www.soda.io/schedule-a-demo). Hey, what can Soda do for you? > Not quite ready for this big gulp of Soda? 🥤Try [taking a sip](/soda-v3/quick-start-sip), first. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Self-serve Soda Follow this guide to enable Soda Cloud end users to create no-code checks for data quality for the data that matters to them the most. Use this guide to set up Soda Cloud and enable users across your organization to serve themselves when it comes to testing data quality. Deploy a Soda Runner in a Kubernetes cluster to connect to both a data source and the Soda Cloud, then invite your Data Analyst and Data Scientist colleagues to join the account, start data quality discussions, and begin creating their own SodaCL checks for data quality. ## About this guide The instructions below offer Data Engineers an example of how to set up the Soda Cloud to enable non-coder colleagues to propose, discuss, and create their own data quality tests. After all, data quality testing is a team sport! Once you have completed the set-up, you can direct your non-coding colleagues to log in to Soda Cloud and begin creating Discussions. A Discussion in Soda is a messaging space that facilitates collaboration between data producers and data consumers. Together, colleagues can establish the expected and agreed-upon state of data quality in a dataset by proposing, then approving data quality checks that execute as part of a scheduled scan in Soda. When checks fail during data quality scans, you and your colleagues get alerts via Slack which enable you to address issues before they have a downstream impact on the users or applications that depend upon the data. ## Access or deploy a Soda Runner {% tabs %} {% tab title="Access a Soda-hosted runner" %} 1. If you have not already done so, create a Soda Cloud account at [cloud.soda.io](https://cloud.soda.io/signup?utm_source=docs). If you already have a Soda account, log in. 2. By default, Soda prepares a Soda-hosted runner for all newly-created accounts. However, if you are an Admin in an existing Soda Cloud account and wish to use a Soda-hosted runner, navigate to **your avatar** > **Organization Settings**. In the **Organization** tab, click the checkbox to **Enable Soda-hosted Runner**. 3. Navigate to **your avatar** > **Data Sources**, then access the **Runners** tab. Notice your out-of-the-box Soda-hosted runner that is up and running.
{% endtab %} {% tab title="Deploy a self-hosted runner" %} Soda Runner is a tool that empowers Soda Cloud users to securely access data sources to scan for data quality. Create a Kubernetes cluster in a cloud services provider environment, then use Helm to deploy a self-hosted Soda Runner in the cluster. For context, this example assumes that: * you have the appropriate access to a cloud services provider environment such as Azure, AWS, or Google Cloud that allows you to create and deploy applications to a cluster, * you, or someone on your team, has access to the login credentials that Soda needs to be able to access a data source such as MS SQL, BigQuery, or Athena so that it can run scans of the data. Access the [exhaustive deployment instructions](/soda-v3/quick-start-sip/deploy#deploy-a-soda-runner-in-a-kubernetes-cluster) for the cloud services provider you use. * Cloud services provider-agnostic instructions * Amazon Elastic Kubernetes Service (EKS) * Microsoft Azure Kubernetes Service (AKS) * Google Kubernetes Engine (GKE) See also: [Soda Runner basic concepts](/soda-v3/learning-resources/basics) {% endtab %} {% endtabs %} ## Connect a data source Depending on your deployment model, Soda Runner supports connections with the following data sources. #### **Self-hosted runner** |

Amazon Athena
Amazon Redshift
Azure Synapse
ClickHouse
Databricks SQL
Denodo
Dremio
DuckDB
GCP BigQuery
Google CloudSQL

|

IBM DB2
MotherDuck
MS SQL Server1
MySQL
OracleDB
PostgreSQL
Presto
Snowflake
Trino
Vertica

| | --------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | 1 MS SQL Server with Windows Authentication does not work with Soda Runner out-of-the-box. #### **Soda-hosted runner** |

BigQuery
Databricks SQL
MS SQL Server
MySQL

|

PostgreSQL
Redshift
Snowflake

| | ----------------------------------------------------------- | ---------------------------------------------- | 1. Log in to your Soda Cloud account, then navigate to **your avatar** > **Data Sources**. 2. In the **Runners** tab, confirm that you can see a Soda-hosted runner, or the Soda Runner you deployed, and that its status is “green” in the **Last Seen** column. If not, refer to the Soda Runner documentation to [troubleshoot](/soda-v3/quick-start-sip/deploy#troubleshoot-deployment) its status.
3. Navigate to the **Data Sources** tab, then click **New Data Source** and follow the [guided steps](/soda-v3/quick-start-sip/managed-agent#add-a-new-data-source) to: * identify the new data source and its default scan definition * provide connection configuration details for the data source such as name, schema, and login credentials, and test the connection to the data source * profile the datasets in the data source to gather basic metadata about the contents of each * identify the datasets to which you wish to apply automated monitoring for anomalies and schema changes * assign ownership roles for the data source and its datasets 4. Save the new data source. ## Set up Slack integration and notification rules Use this integration to enable Soda to send alert notifications to a Slack channel to notify your team when check results warn and fail. If your team does not use Slack, you can follow the instructions to integrate with [MS Teams](/soda-v3/integrate-soda/integrate-msteams), instead, or skip this step as Soda sends alert notifications via email by default. 1. Log in to your Soda Cloud account and navigate to **your avatar** > **Organization Settings**, then navigate to the **Integrations** tab and click the **+** icon to add a new integration. 2. Follow the guided steps to authorize Soda to connect to your Slack workspace. If necessary, contact your organization's Slack Administrator to approve the integration with Soda. * **Configuration** tab: select the public channels to which Soda can post messages; Soda cannot post to private channels. * **Scope** tab: select the two Soda features, **Alert Notifications** and **Discussions**, which can access the Slack integration. 3. To dictate where Soda must send alert notifications for checks that fail, create a new notification rule. Navigate to **your avatar** > **Notification Rules**, then click **New Notification Rule**. Follow the guided steps to complete the new rule directly Soda to send check results that fail to a specific channel in your Slack workspace. > Learn more about [Integrating with Slack](/soda-v3/integrate-soda/integrate-slack). > > Learn more about [Setting notification rules](/soda-v3/collaborate/notif-rules). ## Invite your colleagues After testing and saving the new data source, invite your colleagues to your Soda Cloud account so they can begin creating new agreements. Navigate to **your avatar** > **Invite Team Members**, then complete the form to send invitations to your colleagues. ## Begin a discussion and propose checks While waiting for your colleagues to accept your Soda invitation, get a head start on setting up data quality checks on the data that matters the most to your data consumers. 🎥 Watch a [5-minute video](https://www.soda.io/resources/introducing-no-code-checks) of the following procedure, if you like! 1. In Soda Cloud, navigate to **Discussions** from the main navigation bar. 2. Start a **New Discussion**, providing relevant details for a discussion on data quality metrics, and adding people whose perspectives will add value to the data quality of a particular dataset.
3. Kick off the data quality discussion with your colleagues: begin with **Propose Check**, then use the no-code check interface to select from the list available checks for the dataset. Most common baseline data quality checks include: missing, invalid, duplicate, and freshness. Refer to [Define SodaCL checks](/soda-v3/soda-cl-overview) for more detail on how to leverage no-code checks. 4. After filling in the blanks and testing the check, **Propose Check** to add the SodaCL check to the discussion. When your colleagues join and review the Discussions, they can add comments or propose new or different checks to address the data quality issues of this dataset. 5. When you and your team agree on the data quality checks to add to the dataset, you, as the data producer, can **Review & Add** the check to a scan for the dataset – either existing or new – so that Soda begins executing the check as per the data source's default scan schedule. \ ✨Well done!✨ You've taken the first step towards a future in which you and your colleagues can collaborate on defining and maintaining good-quality data. Huzzah! ## Go further? * [Get organized](/soda-v3/collaborate) in Soda! * [Integrate Soda](/soda-v3/integrate-soda) with your data catalog. * Use [failed row samples](/soda-v3/run-a-scan/failed-row-samples) to investigate data quality issues. * [Request a demo](https://www.soda.io/schedule-a-demo). Hey, what can Soda do for you? > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Automate anomaly detection Use this guide to set up Soda and start automatically monitoring your data for quality. {% hint style="success" %} **Available in 2025:** refer to the new [Soda docs](https://docs.soda.io/). {% endhint %} Use this guide to set up Soda and begin automatically monitoring the data quality of datasets in a data source. Use the guided workflow in Soda Cloud to connect to a data source, profile your data, and activate **anomaly dashboards** for your datasets.
## About this guide This guide offers Data Analysts, Data Scientists, and business users instructions to set up Soda to profile and begin monitoring data for quality, right out of the box. This example offers instructions for both self-hosted and Soda-hosted runner deployment models which use Soda Cloud connected to a Soda Runner to securely access data sources and execute scheduled scans for data quality anomaly detections. See: [Choose a flavor of Soda](/soda-v3/quick-start-sip/setup-guide). ## Set up a Soda Runner {% tabs %} {% tab title="Soda-hosted Runner" %} This setup provides a secure, out-of-the-box Soda-hosted Runner to manage access to data sources from within your Soda Cloud account. #### Compatibility |

BigQuery
Databricks SQL
MS SQL Server
MySQL

|

PostgreSQL
Redshift
Snowflake

| | ----------------------------------------------------------- | ---------------------------------------------- | #### Set up 1. If you have not already done so, create a Soda Cloud account at [cloud.soda.io](https://cloud.soda.io/signup?utm_source=docs). If you already have a Soda account, log in. 2. By default, Soda prepares a Soda-hosted runner for all newly-created accounts. However, if you are an Admin in an existing Soda Cloud account and wish to use a Soda-hosted runner, navigate to **your avatar** > **Organization Settings**. In the **Organization** tab, click the checkbox to **Enable Soda-hosted Runner**. 3. Navigate to **your avatar** > **Data Sources**, then access the **Runners** tab. Notice your out-of-the-box Soda-hosted runner that is up and running.
Invite your colleague(s) to your Soda Cloud organization so they can access the newly-deployed Soda Runner to connect to data sources and begin monitoring data quality. In your Soda Cloud account, navigate to **your avatar** > **Invite Team Members** and fill in the blanks. {% endtab %} {% tab title="Self-hosted Runner" %} This setup uses a secure self-hosted Soda Runner to manage access to data sources from within your Soda Cloud account. #### Compatibility |

Amazon Athena
Amazon Redshift
Azure Synapse
ClickHouse
Databricks SQL
Denodo
Dremio
DuckDB
GCP BigQuery
Google CloudSQL

|

IBM DB2
MotherDuck
MS SQL Server1
MySQL
OracleDB
PostgreSQL
Presto
Snowflake
Trino
Vertica

| | --------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | 1 MS SQL Server with Windows Authentication does not work with Soda Runner out-of-the-box. #### Set up A self-hosted Soda Runner is a tool that empowers Soda Cloud users to securely access data sources to scan for data quality. Create a Kubernetes cluster in a cloud services provider environment, then use Helm to deploy a self-hosted Soda Runner in the cluster. For context, the instructions to deploy a self-hosted runner assume: * you have the appropriate access to a cloud services provider environment such as Azure, AWS, or Google Cloud that allows you to create and deploy applications to a cluster, * you, or someone on your team, has access to the login credentials that Soda needs to be able to access a data source such as MS SQL, BigQuery, or Athena so that it can run scans of the data. Access the [exhaustive deployment instructions](/soda-v3/quick-start-sip/deploy#deploy-a-soda-runner-in-a-kubernetes-cluster) for the cloud services provider you use. * Cloud services provider-agnostic instructions * Amazon Elastic Kubernetes Service (EKS) * Microsoft Azure Kubernetes Service (AKS) * Google Kubernetes Engine (GKE) See also: [Soda Runner basic concepts](/soda-v3/learning-resources/basics) {% endtab %} {% endtabs %} ## Automate data quality monitoring *For preview participants, only.* 1. As a user with permission to do so in your Soda Cloud account, navigate to **your avatar** > **Data Sources**. 2. In the **Runners** tab, confirm that you can see your Soda-hosted runner and that its status is "green" in the **Last Seen** column.
3. Navigate to the **Data source** tab, then click **New Data Source** and follow the guided steps to connect to a new data source. Refer to the subsections below for insight into the values to enter in the fields and editing panels in the guided steps. #### 1. Attributes | Field or Label | Guidance | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Data Source Label | Provide a unique identifier for the data source. Soda Cloud uses the label you provide to define the immutable name of the data source against which it runs the Default Scan. | | Default Scan Runner | Select the Soda-hosted runner, or the name of a Soda Runner that you have previously set up in your secure environment. This identifies the Soda Runner to which Soda Cloud must connect in order to run its scan. | | Check Schedule | Provide the scan frequency details Soda Cloud uses to execute scans according to your needs. If you wish, you can define the schedule as a cron expression. | | Starting At | Select the time of day to run the scan. The default value is midnight. | | Cron Expression | (Optional) Write your own [cron expression](https://en.wikipedia.org/wiki/Cron) to define the schedule Soda Cloud uses to run scans. | |

Anomaly Dashboard Scan Schedule
Available in 2025

| Provide the scan frequency details Soda Cloud uses to execute a daily scan to automatically detect anomalies for the anomaly dashboard. | #### 2. Connect In the editing panel, provide the connection configurations Soda Cloud needs to be able to access the data in the data source. Connection configurations are data source-specific and include values for things such as a database's host and access credentials. Access the [data source-specific connection configurations](/soda-v3/data-source-reference) for the connection syntax and descriptions; adjust the values to correspond with your data source’s details. To more securely provide sensitive values such as usernames and passwords in a self-hosted runner deployment model, use environment variables in a `values.yml` file when you deploy the Soda Runner. See [Use environment variables for data source connection credentials](/soda-v3/quick-start-sip/extras#use-environment-variables-to-store-data-source-connection-credentials) for details. #### 3. Discover During its initial scan of your datasource, Soda Cloud discovers all the datasets the data source contains. It captures basic information about each dataset, including a dataset's schema and the columns it contains. In the editing panel, specify the datasets that Soda Cloud must include or exclude from this basic discovery activity. The default syntax in the editing panel instructs Soda to collect basic dataset information from all datasets in the data source *except* those with names that begin with `test_`. The `%` is a wildcard character. See [Add dataset discovery](/soda-v3/soda-cl-overview/profile#add-dataset-discovery) for more detail on profiling syntax. {% hint style="warning" %} *Known issue:* SodaCL does not support using variables in column profiling and dataset discovery configurations. {% endhint %} ```yaml discover datasets: datasets: - include % - exclude test_% ``` #### 4. Profile To gather more detailed profile information about datasets in your data source and automatically build an **anomaly dashboard** for data quality observability, you can configure Soda Cloud to profile the columns in datasets. Profiling a dataset produces two tabs' worth of data in a dataset page: * In the **Columns** tab, you can see column profile information including details such as the calculated mean value of data in a column, the maximum and minimum values in a column, and the number of rows with missing data.
* In the **Anomalies** tab, you can access an out-of-the-box Anomaly Dashboard that uses the column profile information to automatically begin detecting anomalies in your data relative to the patterns the machine learning algorithm learns over the course of approximately five days. **Available in 2025:** [Learn more](/data-observability)
In the editing panel, provide details that Soda Cloud uses to determine which datasets to include or exclude when it profiles the columns in a dataset. The default syntax in the editing panel instructs Soda to profile every column of every dataset in this data source, and, superfluously, all datasets with names that begin with `prod`. The `%` is a wildcard character. See [Add column profiling](/soda-v3/soda-cl-overview/profile#add-column-profiling) for more detail on profiling syntax. Column profiling can be resource-heavy, so carefully consider the datasets for which you truly need column profile information. Refer to [Compute consumption and cost considerations](/soda-v3/soda-cl-overview/profile#compute-consumption-and-cost-considerations) for more detail. ```yaml profile columns: columns: - "%.%" # Includes all your datasets - prod% # Includes all datasets that begin with 'prod' ``` #### 5. Assign Owner | Field or Label | Guidance | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Data Source Owner | The Data Source Owner maintains the connection details and settings for this data source and its Default Scan Definition. | | Default Dataset Owner | The Datasets Owner is the user who, by default, becomes the owner of each dataset the Default Scan discovers. Refer to [Manage global roles, user groups, and settings ](/soda-v3/collaborate/roles-global)to learn how to adjust the Dataset Owner of individual datasets | ## Access an anomaly dashboard After approximately five days, during which Soda’s machine learning studies your data, you can navigate to the **Anomalies** tab on the **Dataset** page on one of the datasets you included in profiling to view the issues Soda automatically detected.
The three **Dataset Metrics** tiles represent the most recent measurement or, in other words, one day’s worth of data anomaly detection. The three **Column Metrics** tiles display the last seven days’ worth of measurements and any anomalies that Soda detected. When you click a **Column Metrics** tile to access more information, the list below details which columns contained anomalies. * A red warning icon for a column indicates that Soda registered an anomaly in the last daily scan of the dataset. * A green check icon for a column indicates that Soda resgisterd no anomalies in the last daily scan of the dataset. * A grayed-out icon for a column indicates that Soda registered an anomaly for a check at least once in the last seven days, but not on the most recent daily scan. Click a Dataset Metric tile or the column name for a Column Metric to open the **Check History** for the anomaly detection check. Optionally, you can add feedback to individual data points in the check history graph to help refine the anomaly detection’s algorithm pattern recognition and its ability to recognize anomalies.
### Set up alert notifications The anomaly dashboard adheres to Soda’s “no noise” policy when it comes to alert notifications for data quality issues. As such, the dashboard does not automatically send any notifications to anyone out of the box. If you wish to received alert notifications for any of the anomalies the dashboard detects, use the bell (🔔) icon. If your Soda Admin has integrated your Soda Cloud account with [Slack](/soda-v3/integrate-soda/integrate-slack) or [MS Teams](/soda-v3/integrate-soda/integrate-msteams) to receive check notifications, you can direct anomaly dashboard alerts to those channels. The dashboard does not support sending alerts via [webhook](/soda-v3/integrate-soda/integrate-webhooks). For a **Dataset Metric**, click the bell to follow the guided instructions to set up a rule that defines where to send an alert notification when Soda detects an anomalous measurement for the metric.
For a **Column Metric**, click the bell next to an individual column name from those listed in the table below the three column metric tiles. Follow the guided instructions to set up a rule that defines where to send an alert notification when Soda detects an anomalous measurement for the metric. For example, if you want to receive notifications any time Soda detects an anomalous volume of duplicate values in an `order_id` column, click the **Duplicate** tile to display all the columns for which Soda automatically detects anomalies, then click the bell for `order_id` and set up a rule. If you also wish to receive notifications for anomalous volumes of missing values in the same column, click the **Missing** tile, then click the bell for `order_id` to set up a second rule.
### Go further * Learn more about the [anomaly dashboard](/soda-v3/collaborate/anomaly-dashboard) for datasets. * Learn more about [organizing check results](/soda-v3/collaborate), setting alerts, and investigating issues. * [Write your own checks](/soda-v3/sodacl-reference/metrics-and-checks) for data quality. * [Integrate Soda with Slack](/soda-v3/integrate-soda/integrate-slack) to send alert notifications directly to channels in your workspace. * Integrate Soda with a data catalog to see data quality results from within the catalog: * [Atlan](/soda-v3/integrate-soda/integrate-atlan) * [Alation](/soda-v3/integrate-soda/integrate-alation) * [Metaphor](/soda-v3/integrate-soda/integrate-metaphor) > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Build a Sigma dashboard This example helps you build a customized data quality reporting dashboard in Sigma using the Soda Cloud API. This guide offers a comprehensive example for building a customized data quality reporting dashboard in Sigma. Use the [Soda Cloud API](broken://pages/vOqf3xQITkpmDUg8wnKB) to capture metadata from your Soda Cloud account, store it in Snowflake, then access the data in Snowflake to create a Sigma dashboard.
## Prerequisites * Python 3.8, 3.9, or 3.10 * Pip 21.0 or greater * access to an account in Sigma * access to a Snowflake data source * a Soda Cloud account; see [Get started](/soda-v3/quick-start-sip/managed-agent#create-a-soda-cloud-account) * permission in Soda Cloud to access dataset metadata; see [Manage dataset roles](/soda-v3/collaborate/roles-dataset) ## Set up a Python script 1. Install an HTTP request library and Snowflake connector. ```shell pip install pandas requests pip install snowflake-connector-python ``` 2. In a new Python script, configure the following details to integrate with Soda Cloud. See [Generate API keys](/soda-v3/use-case-guides/api-keys) for detailed instructions. ```python # Use cloud.us.soda.io in the US region; use cloud.soda.io in the EU region soda_cloud_url = 'https://cloud.us.soda.io' soda_apikey = 'xxx' # API key ID from Soda Cloud soda_apikey_secret = 'xxx' # API key secret from Soda Cloud ``` 3. In the same script, define the tables in which to store the Soda dataset information and check results in Snowflake, ensuring they are in uppercase to avoid issues with Snowflake's case sensitivity requirements. ```python # Tables to store Soda metadata. Use UPPERCASE. datasets_table = 'DATASETS_REPORT' checks_table = 'CHECKS_REPORT' ``` 4. In the same script, configure your Snowflake connection details. This configuration enables your script to securely access your Snowflake data source. ```python # Snowflake connection details snowflake_details = snowflake.connector.connect( user=user, password=password, account=account, warehouse=warehouse, database=database, schema=schema, ) ``` 5. In the script, prepare an HTTP `GET` request to the Soda Cloud API to retrieve dataset information. Direct the request to the [Dataset information](broken://pages/vOqf3xQITkpmDUg8wnKB) endpoint, including the authentication API keys to access the data. This script prints an error if the request is unauthorized. ```python response_datasets = requests.get( soda_cloud_url + '/api/v1/datasets?page=0', auth=(soda_apikey , soda_apikey_secret) ) if response_datasets.status_code == 401 or response_datasets.status_code == 403: print("Unauthorized or Forbidden access. Please check your API keys and/or permissions in Soda.") sys.exit() ``` 6. Run the script to ensure that the `GET` request results in HTTP status code `200`, confirming the successful connection to Soda Cloud. ## Capture and store metadata 1. With a functional connection to Soda Cloud, adjust the API call to extract all dataset information from Soda Cloud, iterating over each page of the datasets. Then, create a Pandas Dataframe to contain the retrieved metadata.\ This adjusted call retrieves information about each dataset's name, its last update, the data source in which it exists, its health status, and the volume of checks and incidents with which it is associated. ```python # Fetch info about all datasets if response_datasets.status_code == 200: dataset_pages = response_datasets.json().get('totalPages') i = 0 while i < dataset_pages: dq_datasets = requests.get( soda_cloud_url + '/api/v1/datasets?page='+str(i), auth=(soda_apikey , soda_apikey_secret)) if dq_datasets.status_code == 200: print("Fetching all datasets on page: "+str(i)) list = dq_datasets.json().get("content") datasets.extend(list) i += 1 elif dq_datasets.status_code == 429: print("API Rate Limit reached when fetching datasets on page: " +str(i)+ ". Pausing for 30 seconds.") time.sleep(30) # Retry fetching the same page else: print("Error fetching datasets on page "+str(i)+". Status code:", dq_datasets.status_code) else: print("Error fetching initial datasets. Status code:", response_datasets.status_code) sys(exit) df_datasets = pd.DataFrame(datasets) ``` 2. Inspect the information you retrieved with the following Pandas command; see example output below. ```python df_datasets.head() ``` ![df-output](/files/jBzt8tJrov1QARJQuN4o){:height="700px" width="700px"}\ 3\. Following the same logic, extract all the check-related information from Soda Cloud using the [Checks information](https://docs.soda.io/api-docs/public-cloud-api-v1.html#/operations/GET/api/v1/checks) endpoint.\ This call retrieves information about the checks in Soda Cloud, including the dataset and column each runs against, the latest check evaluation time and the result—pass, warn, or fail—and any attributes associated with the check. ````python # Fetch info about all checks response_checks = requests.get( soda_cloud_url + '/api/v1/checks?size=100', auth=(soda_apikey , soda_apikey_secret)) if response_checks.status_code == 200: check_pages = response_checks.json().get('totalPages') i = 0 while i < check_pages: dq_checks = requests.get( soda_cloud_url + '/api/v1/checks?size=100&page='+str(i), auth=(soda_apikey , soda_apikey_secret)) if dq_checks.status_code == 200: print("Fetching all checks on page "+str(i)) check_list = dq_checks.json().get("content") checks.extend(check_list) i += 1 elif dq_checks.status_code == 429: print("API Rate Limit reached when fetching checks on page: " +str(i)+ ". Pausing for 30 seconds.") time.sleep(30) # Retry fetching the same page else: print("Error fetching checks on page "+str(i)+". Status code:", dq_checks.status_code) else: print("Error fetching initial checks. Status code:", response_checks.status_code) sys(exit) df_checks = pd.DataFrame(checks) ``` ```` 4\. Again, inspect the output with a Pandas command. ```python df_checks.head() ```
5\. Finally, move the two sets of metadata into your Snowflake data source. Optionally, if you wish to track updates and changes to dataset and check metadata over time, you can store the metadata to incremental tables and set up a flow to update the values on a regular basis using the latest information retrieved from Soda Cloud. ```python write_pandas(snowflake_details, df_checks, checks_table, auto_create_table=True) write_pandas(snowflake_details, df_datasets, datasets_table, auto_create_table=True) ``` 6. Run the script to populate the tables in Snowflake with the metadata pulled from Soda Cloud. ## Build a data quality dashboard in Sigma To build a custom dashboard, this example uses [Sigma](https://www.sigmacomputing.com/product/dashboards), a cloud-based analytics and business intelligence platform designed to facilitate data exploration and analysis. You may wish to use a different tool to build a dashboard such as Metabase, Lightdash, Looker, PowerBI, or Tableau. This example leverages [check attributes](/soda-v3/collaborate/check-attributes), an optional configuration that helps categorize or segment check results so you can better filter and organize not only your views in Soda Cloud, but your customized dashboard. Checks in this example use the following attributes: * Data Quality Dimension: Completeness, Validity, Consistency, Accuracy, Timeliness, Uniqueness * Data Domain: Customer, Location, Product, Transaction * Data Team: Data Engineering, Data Science, Sales Operations * Pipeline stage: Destination, Ingest, Report, Transform * Weight The weight attribute, in particular, is very useful in allocating a numerical level of importance to checks which you can use to create a custom data health quality score. 1. Follow the Sigma documentation to [Connect to Snowflake](https://help.sigmacomputing.com/docs/connect-to-snowflake). 2. Follow Sigma documentation to access the metadata you stored in Snowflake, either by [Modeling data from database tables](https://help.sigmacomputing.com/docs/create-models), or [Creating a dataset by writing custom SQL](https://help.sigmacomputing.com/docs/create-a-dataset-from-sql#create-a-dataset-by-writing-custom-sql). 3. Create a new [workbook](https://help.sigmacomputing.com/docs/workbooks-overview) in Sigma where you can create your [visualizations](https://help.sigmacomputing.com/docs/intro-to-visualizations). The Sigma dashboard below tracks data quality status within an organization. It includes some basic KPI information including the number of datasets monitored by Soda, as well as the number of checks that it regularly executes. It displays a weighted data quality score based on the custom values provided in the `Weight` attribute for each check (here shown according to data quality dimension) which it compares to previous measurements gathered over time.
## Go further * Access full [Soda Cloud API](broken://pages/vOqf3xQITkpmDUg8wnKB) and [Soda Cloud Reporting API](broken://pages/JxhkwBNtkP07gkPsmgpe) documentation. * Learn more about [check attributes](/soda-v3/collaborate/check-attributes) and [dataset attributes](/soda-v3/collaborate/organize-datasets). > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Build a Grafana dashboard This example helps you build a customized data quality reporting dashboard in Grafana using the Soda Cloud API. This guide offers an example of how to create a data quality reporting dashboard using the Soda Cloud API and Grafana. Such a dashboard enables data engineers to monitor the status of Soda scans and capture and display check results. Use the [Soda Cloud API](https://docs.soda.io/api-docs/public-cloud-api-v1.html) to trigger data quality scans and extract metadata from your Soda Cloud account, then store the metadata in PostgreSQL and use it to customize visualized data quality results in Grafana.
## Prerequisites * access to a [Grafana account](https://grafana.com/) * Python 3.8, 3.9, or 3.10 * familiarity with Python, with Python library interactions with APIs * access to a PostgreSQL data source * a Soda Cloud account: [Sign Up](https://cloud.soda.io/select-region) * permission in Soda Cloud to access dataset metadata; see [Manage dataset roles](/soda-v3/collaborate/roles-dataset) * at least one agreement or no-code check associated with a scan definition in Soda Cloud; see [Use no-code checks](/soda-v3/soda-cl-overview#define-sodacl-checks) ## Choose a scan definition Because this guide uses the Soda Cloud API to trigger a scan execution, you must first choose an existing check in Soda Cloud to identify its associated [scan definition](/soda-v3/learning-resources/glossary#scan-definition), which you will use to identify which checks to execute during the triggered scan. > See also: [Trigger a scan via API](/soda-v3/run-a-scan#trigger-a-scan-via-api) 1. Log in to your Soda Cloud account and navigate to the **Checks** page. Choose a check that originated in Soda Cloud, identifiable by the cloud icon, that you can use to complete this exercise. Use the action menu (stacked dots) next to the check to select **Edit Check**.
2. In the dialog that opens, copy the scan definition name from the **Add to Scan Definition** field. Under **Scans**, above the scan definition name, copy the scan definition ID that uses undescores to represent spaces. Paste the scan definition ID in a temporary local file; you will use it in the next steps to trigger a scan via the Soda Cloud API.
3. Be aware that when you use the Soda Cloud API to trigger the execution of this scan definition remotely, Soda executes *all* checks associated with the scan definition. This is a good thing, as you can see the metadata for multiple check results in the Grafana dashboard this guide prepares. ## Prepare to use the Soda Cloud API 1. As per best practice, set up a new Python virtual environment so that you can keep your projects isolated and avoid library clashes. The example below uses the built-in venv module to create, then navigate to and activate, a virtual environment named `soda-grafana`. Run `deactivate` to close the virtual environment when you wish.
```shell python3 -m venv ~/venvs/soda-grafana cd venvs source soda-grafana/bin/activate ``` 2. Run the following command to install the requests libraries in your virtual environment that you need to connect to Soda Cloud API endpoints.\ Because this exercise moves the data it extracts from your Soda Cloud account into a PostgreSQL data source, it requires the `psycopg2` library. Alternatively, you can list and save all the requirements in a `requirements.txt` file, then install them from the command-line using `pip install -r requirements.txt`. If you use a different type of data source, find a corresponding plugin, or check SQLAlchemy's built-in database compatibility. ```shell pip install requests psycopg2 ``` 3. In the same directory, create a new file named `apiscan.py`. Paste the following contents into the file to define an ApiScan class, which you will use to interact with the Soda Cloud API. ```python import os import requests class ApiScan(): def __init__(self): self.url = os.environ.get("SODA_URL") self.auth = (os.environ.get("API_KEY"), os.environ.get("API_SECRET")) def _get(self, endpt: str, headers: dict = None): r = requests.get(url=self.url + endpt, auth=self.auth, headers=headers) print(f"_get result for endpoint {endpt}: {r}") r = r.json() return r def _post(self, endpt: str, headers: dict = None, data: dict = None): r = requests.post(url=self.url + endpt, auth=self.auth, headers=headers, data=data) return r def test(self): # expect response 200 return self._get(endpt="test-login") def checks(self, datasetID: str, size: int = 100): headers = {"Accept": "application/json"} params = f"?size={size}&datasetID={datasetID}" return self._get(endpt=f"checks{params}", headers=headers) def datasets(self, from_ts: str, size: int = 100): headers = {"Accept": "application/json"} params = f"?size={size}&from={from_ts}" return self._get(endpt=f"datasets{params}", headers=headers) def trigger(self, scan: str): headers = {"Accept": "application/json", "Content-Type": "application/x-www-form-urlencoded"} data = {"scanDefinition": scan} return self._post(endpt="scans", headers=headers, data=data) def state(self, scan_id: str): return self._get(endpt=f"scans/{scan_id}") ``` 4. From the command-line, create the following environment variables to facilitate a connection to your Soda Cloud account and your PostgreSQL data source. * `SODA_URL`: use `https://cloud.soda.io/api/v1/` or `https://cloud.us.soda.io/api/v1/` as the value, according to the region in which you created your Soda Cloud account. * `API_KEY` and `API_SECRET`: see [Generate API keys](/soda-v3/use-case-guides/api-keys) ````sh # Soda Cloud API keys, used in apiscan.py SODA_URL = https://cloud.soda.io/api/v1/ API_KEY = xxx API_SECRET = xxx # PostgreSQL access credentials, used in main.py HOST = host_name PG_USER = user_login PG_PASSWORD = user_pass ``` ```` #### Troubleshoot **Problem:** You get an error that reads, "psycopg2 installation fails with error: metadata-generation-failed" and the suggestion "If you prefer to avoid building psycopg2 from source, please install the PyPI 'psycopg2-binary' package instead." **Solution:** As suggested, install the binary package instead, using `pip install psycopg2-binary`. ## Trigger and monitor a Soda scan 1. In the same directory in which you created the `apiscan.py` file, create a new file named `main.py`. 2. To the file, add the following code which: * imports necessary libraries, as well as the ApiScan class from `apiscan.py` * initializes an `ApiScan` object as `ascan`, uses the object to trigger a scan with `scan_definition` as a parameter which, in this case, is `grafanascan0`; replace `grafanascan0` with the scan definition ID you copied to a local file earlier. * stores the scan `id` as a variable * checks the state of the scan every 10 seconds, then only when it is in a completion state (`completedWithErrors`, `completedWithFailures`, `completedWithWarnings`, or `completed`), stores the scan results as variable `r`. ```python from apiscan import ApiScan import os import psycopg2 import time scan_definition = "grafanascan0" ascan = ApiScan() sc = ascan.trigger(scan=scan_definition) id = sc.headers["X-Soda-Scan-Id"] state = "" # do not get logs until scan completed while "completed" not in state: r = ascan.state(scan_id=id) state = r["state"] print(f"Scan state: {state}", end="\r") time.sleep(10) r = ascan.state(scan_id=id) print(f"Scan done!") ``` ## Extract scan results To the `main.py` file, add the following code which: * extracts Soda scan details, from the scan results stored in variable `r` * extracts dataset details, using Soda Cloud API's datasets endpoint * extracts checks details, using Soda Cloud API's checks endpoint * combines scan, dataset and checks details into one dictionary per check, and appends the dictionary to a list of checks ```python # EXTRACT SCAN DETAILS s = {} s["definitionName"] = r["scanDefinition"]["name"] s["scanStartTimestamp"] = r["started"] s["scanEndTimestamp"] = r["ended"] s["hasErrors"] = (False if r["errors"] == 0 else True) # EXTRACT DATASETS DETAILS ### get only the datasets just scanned d_api = ascan.datasets(from_ts=r["started"]) d_all = [{"datasetName": d["name"], "datasetID": d["id"], "dataSource": d["datasource"]["name"]} for d in d_api["content"]] # EXTRACT CHECKS DETAILS c_all = [] ### get only the checks for datasets just scanned for d in d_all: d_checks = ascan.checks(datasetID=d["datasetID"]) c_all = c_all + d_checks["content"] c_cols = ["id", "name", "evaluationStatus", "column"] checks = [] # list of rows, each row has both scan and check details for check in r["checks"]: ### find scanned check in all checks from api c_single = next(c for c in c_all if c["id"] == check["id"]) c = {col:c_single[col] for col in c_cols} # rename keys to standardize and avoid SQL special words c["identity"] = c.pop("id") c["outcome"] = c.pop("evaluationStatus") c["columnName"] = c.pop("column") # add dataset details - datasetName, dataSource c["datasetName"] = c_single["datasets"][0]["name"] d_full = next(d for d in d_all if d["datasetName"] == c["datasetName"]) c["dataSource"] = d_full["dataSource"] checks.append({**s, **c}) # combine scan, dataset and check details in one row ``` ## Process scan results into a PostgreSQL data source The following example code serves as reference for adding data to a PostgreSQL data source. Replace it if you intend to store scan results in another type of data source. 1. To the `main.py` file, add the code below which: * connects to a PostgreSQL data source, using the psycopg2 library * creates a table in the data source in which to store scan results, if one does not already exist * processes the list of dicts, and inserts them into table of scan results 2. From the command-line, run `python3 main.py`. ```python # POSTGRES / SQL target = "my_schema.api_results" # . try: print("postgres.py trying to connect to database...") conn = psycopg2.connect(database="postgres", user=os.environ.get("PG_USER"), password=os.environ.get("PG_PASSWORD"), host=os.environ.get("HOST"), port="5432") print("postgres.py connected to database!") curs = conn.cursor() except Exception as e: print(f"postgres.py failed to connect to database: {e}") # Create a table if one does not exist schema = [] for col in checks[0].keys(): if "Timestamp" in col: schema.append(f"{col} TIMESTAMP") elif col[0:3] == "has": schema.append(f"{col} BOOLEAN") else: schema.append(f"{col} VARCHAR") schema = "(" + ", ".join(schema) + ")" curs.execute(f""" create table if not exists {target} {schema} """ ) curs.connection.commit() # Insert into table # Create list of cols without quotation marks " " cols = "(%s)" % ", ".join(map(str, checks[0].keys())) # create tuples of values to be appended values = [tuple(str(v) for v in check.values()) for check in checks] values = str(values).strip('[]') curs.execute(f""" insert into {target} {cols} values {values} """) curs.connection.commit() print(f"Scan processed to PostgreSQL# !") ``` ## Visualize scan results in a Grafana dashboard 1. Log into your Grafana account, select My Account, then launch Grafana Cloud. 2. Follow Grafana's [PostgreSQL data source](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/data-sources/postgres/) instructions to add your data source which contains the Soda check results. 3. Follow Grafana's [Create a dashboard](https://grafana.com/docs/grafana-cloud/visualizations/dashboards/build-dashboards/create-dashboard/) instructions to create a new dashboard. Use the following details for reference in the Edit panel for Visualizations. * In the Queries tab, configure a Query using Builder or Code, then Run query on the data source. Toggle the Table view at the top to see Query results. * In the Transformations tab, create, edit, or delete Transformations that transform Query results into the data and format that Visualization needs. * Access Grafana's Visualizations documentation for [guidance on Visualizations](https://grafana.com/docs/grafana-cloud/visualizations/panels-visualizations/visualizations/). The example code included in this guide produces the following visualizations.
## Go further * Access full [Soda Cloud API](broken://pages/vOqf3xQITkpmDUg8wnKB) and [Soda Cloud Reporting API](broken://pages/JxhkwBNtkP07gkPsmgpe) documentation. * Learn more about [remotely running a Doda scan](/soda-v3/run-a-scan#trigger-a-scan-via-api). > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Add Soda to a Databricks notebook Use this guide to invoke Soda data quality tests from inside a Databricks notebook. Use this guide to install and set up Soda in a Databricks notebook so you can run data quality tests on data in a Spark data source. > 🎥 Watch a video that demonstrates how to add Soda to your Databricks pipeline: ## About this guide The instructions below offer Data Engineers an example of how to write Python in a Databricks notebook to set up Soda, then write and execute scans for data quality in Spark. This example uses a programmatic deployment model which invokes the Soda Python library, and uses Soda Cloud to validate a commercial usage license and display visualized data quality test results. See: [Choose a flavor of Soda](/soda-v3/quick-start-sip/setup-guide). ## Create a Soda Cloud account To validate your account license or free trial, Soda Library must communicate with a Soda Cloud account via API keys. You create a set of API keys in your Soda Cloud account, then use them to configure the connection to Soda Library. 1. In a browser, navigate to [cloud.soda.io/signup](https://cloud.soda.io/signup) to create a new Soda account, which is free for a 45-day trial. If you already have a Soda account, log in. 2. Navigate to **your avatar** > **Profile**, then access the **API keys** tab. Click the plus icon to generate new API keys. 3. Copy+paste the API key values to a temporary, secure place in your local environment. ## Set up Soda Soda Library has the following requirements: * Python 3.8, 3.9, or 3.10 * Pip 21.0 or greater
Python versions Soda supports Soda officially supports Python versions 3.8, 3.9, and 3.10.\ Though largely funcntional, efforts to fully support Python 3.11 and 3.12 are ongoing. Using Python 3.11, some users might have some issues with dependencies constraints. At times, extra the combination of Python 3.11 and dependencies constraints requires that a dependency be built from source rather than downloaded pre-built. The same applies to Python 3.12, although there is some anecdotal evidence that indicates that 3.12 might not work in all scenarios due to dependencies constraints.
\ Download the notebook: [Soda Databricks notebook](https://github.com/sodadata/docs/blob/main/soda/soda-databricks-notebook.ipynb) ```python # Install a Soda Library package with Apache Spark DataFrame pip install -i https://pypi.cloud.soda.io soda-spark-df # Import Scan from Soda Library # A scan is a command that executes checks to extract information about data in a dataset. from soda.scan import Scan # Create a Spark DataFrame, or use the Spark API to read data and create a DataFrame # A Spark DataFrame is a distributed collection of data organized into named columns which provides a structured and tabular representation of data within the Apache Spark framework. df = spark.table("delta.`/databricks-datasets/adventureworks/tables/adventureworks`") # Create a view that Soda uses as a dataset df.createOrReplaceTempView("adventureworks") # Create a scan object scan = Scan() # Set a scan definition # Use a scan definition to configure which data to scan, and when and how to execute the scan. scan.set_scan_definition_name("Databricks Notebook") scan.set_data_source_name("spark_df") # Attach a Spark session scan.add_spark_session(spark) # Define checks for datasets # A Soda Check is a test that Soda Library performs when it scans a dataset in your data source. You can define your checks in-line in the notebook, or define them in a separate checks.yml fail that is accessible by Spark. checks = """ checks for dim_customer: - invalid_count(email_address) = 0: valid format: email name: Ensure values are formatted as email addresses - missing_count(last_name) = 0: name: Ensure there are no null values in the Last Name column - duplicate_count(phone) = 0: name: No duplicate phone numbers - freshness(date_first_purchase) < 7d: name: Data in this dataset is less than 7 days old - schema: warn: when schema changes: any name: Columns have not been added, removed, or changed sample datasets: datasets: - include dim_% """ # OR, define checks in a file accessible via Spark, then use the scan.add_sodacl_yaml method to retrieve the checks scan.add_sodacl_yaml_str(checks) # Add your Soda Cloud connection configuration using the API Keys you created in Soda Cloud # Use cloud.soda.io for EU region # Use cloud.us.soda.io for US region config =""" soda_cloud: host: cloud.soda.io api_key_id: 39**9 api_key_secret: hN**_W1Q """ # OR, configure the connection details in a file accessible via Spark, then use the scan.add_configuration_yaml method to retrieve the config scan.add_configuration_yaml_str(config) # Execute a scan scan.execute() # Check the Scan object for methods to inspect the scan result # The following prints all logs to the console print(scan.get_logs_text()) ``` ## Go further * Use Soda to [test data in a Databricks pipeline](/soda-v3/use-case-guides/quick-start-databricks-pipeline). * Learn more about [SodaCL checks and metrics](/soda-v3/sodacl-reference/metrics-and-checks). * Access instructions to [Generate API Keys](/soda-v3/use-case-guides/api-keys). > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Integrate an External Secrets Manager with a Soda Runner Follow this guide to set up an integration with an External Secrets Manager for a Soda Runner to use to securely retrieve frequently-rotated passwords. Use this guide to set up a Soda Runner to securely retrieve frequently-rotated and/or encrypted data source login credentials. Rather than managing sensitive login credentials for data sources passed via the Helm chart value `soda.env`, you can set up a Soda Runner to integrate with external secrets managers such as Hashicorp Vault, AWS Secrets Manager, or Azure Key Vault, so that it can securely access up-to-date, externally-stored login credentials for data sources. ## About this guide This exercise points to a GitHub repository from which you can set up a locally-run, example Kubernetes cluster to illustrate what an integration between a Soda Runner and an external secrets manager looks like. When you complete the exercise, you will have examples of the things you need for a Soda Runner to access an external secrets manager: * **External Secrets Operator (ESO)** which is a Kubernetes operator that facilitates a connection between the Soda Runner and your secrets manager; see [external-secrets.io](https://external-secrets.io/latest/). * a **ClusterSecretStore** resource which provides a central gateway with instructions on how to access your secret backend * an **ExternalSecret** resource which instructs the cluster on which values to fetch, and references the ClusterSecretStore Follow the instructions below to use the Terraform files in the repository to: * set up and configure a local Kubernetes cluster * deploy External Secrets Operator * configure both a ClusterSecretStore and ExternalSecrets to access username and password examples in a Hashicorp Vault * set up an example PostgreSQL data source containing NYC bus breakdowns and delays data * deploy a Soda Runner configured to use the external secrets manager to access login credentials for a data source * create a Soda Cloud account and set up a new data source that accesses the data in the PostgreSQL data source via the Soda Runner and the external secrets manager ## Prerequisites For this exercise, you must have installed the following tools: * [Terraform](https://developer.hashicorp.com/terraform/downloads) to build a locally-run example environment * One container runtime that provides containers to use as local Kubernetes cluster nodes, either: * [Docker Desktop](https://docs.docker.com/desktop/), for users who prefer to use a UI * [Docker engine](https://docs.docker.com/engine/install/), for users who prefer to use the command-line\ OR
* [Podman Desktop](https://podman-desktop.io/docs/Installation), for users who prefer to use a UI * [Podman engine](https://podman.io/docs/installation), for users who prefer to use the command-line * (Optional) [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation) to create and run a Kubernetes cluster locally * (Optional) [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) to execute commands against the Kubernetes cluster ## Set up 1. Clone the [`github.com/sodadata/soda-runner-use-cases`](https://github.com/sodadata/soda-runner-use-cases) repository locally. 2. Navigate to the `setup` directory in the repository. ``` cd soda-runner-external-secrets/setup ``` 3. Use the Terraform commands below to: * create a local Kubernetes cluster, * setup a Hashicorp Vault, * deploy External Secrets Operator, * setup a Kubernetes UI dashboard called Headlamp * create a PostgreSQL data source containing a NYC bus breakdowns and delays dataset ```shell terraform init terraform apply -auto-approve ``` Output (last few lines): ```shell ... Apply complete! Resources: 13 added, 0 changed, 0 destroyed. Outputs: base_port = 30200 cluster_admin_token = cluster_name = "soda-runner-external-secrets" dashboard_access = "http://127.0.0.1:30202" soda_runner_namespace = "soda-runner" vault_access = "http://127.0.0.1:30200" vault_init_access = "http://127.0.0.1:30201" vault_root_token = ``` ## Configure 1. Navigate to the `configure` directory in the repository. ``` cd .. cd configure ``` 2. Use the Terraform commands below to: * configure a Hashicorp Vault, * configure the Vault provider of the ESO, * create a Kubernetes secret with a value for `appRoleSecretId` to access the Hashicorp Vault, * create an `ExternalSecret`, * create a `ClusterSecretstore`, * populate some secret values into Vault ```shell terraform init terraform apply -auto-approve ``` Output (last few lines): ```shell ... Apply complete! Resources: 20 added, 0 changed, 0 destroyed. Outputs: dashboard_access = "http://127.0.0.1:30202" dashboard_token = vault_access = "http://127.0.0.1:30200" vault_admin_password = vault_admin_username = "admin" vault_read_only_password = vault_read_only_role_id = "3e94ee54-1799-936e-9cec-5c5a19a5eeeb" vault_read_only_role_secret_id = vault_read_only_token = vault_read_only_username = "soda" ``` ## Access the Hashicorp Vault 1. The configuration output produces a URL value for `vault_access` which, by default, is . Click the link to access the Hashicorp Vault login page in your browser.
2. To log in, change the Method to `Username`, then use the `vault_admin_username` to populate the first field. To extract the value for `vault_admin_password` for the second field, use the following command: ``` terraform output -raw vault_admin_password ``` Hint: To copy the password directly to the clipboard the command, use one of the following commands: ``` # MacOS terraform output -raw vault_admin_password | pbcopy ``` ``` # Linux terraform output -raw vault_admin_password | xclip -selection clipboard ``` 3. Now logged in, from the list of Secret Engines, navigate to `kv/local/soda` to see the example username and password secrets in the vault. If you wish, you can set new secrets that the Soda Runner can use.\
## Deploy a Soda Runner and pass login credentials 1. Access [Deploy a Soda Runner](/soda-v3/quick-start-sip/deploy#create-a-soda-cloud-account) and follow the instructions to create a free, 45-day trial Soda Cloud account and an API key id, and an API key secret for the Soda Runner. 2. Prepare a values YAML file to deploy a Soda Runner in your cluster, as per the following example. ```yaml soda: apikey: id: "value-from-step1" secret: "value-from-step1" runner: name: "my-soda-runner-external-secrets" scanlauncher: existingSecrets: # from spec.target.name in the ExternalSecret file - soda-runner-secrets idle: enabled: true replicas: 1 cloud: # Use https://cloud.us.soda.io for US region # Use https://cloud.soda.io for EU region endpoint: "https://cloud.soda.io" ``` 3. Deploy the Soda Runner using the following command: ```shell helm install soda-runner soda-runner/soda-runner \ --values values.yml \ --namespace soda-runner ``` ## Create the example data source in Soda Cloud To use your newly-deployed Soda Runner, you start by creating a new data source in your Soda Cloud account, then you can create a Soda Agreement to write checks for data quality. 1. In your Soda Cloud account, navigate to your **avatar** > **Scans & Data**. Click **New Data Source**, then follow the guided steps to create a new data source. Refer to [Add a new data source](/soda-v3/quick-start-sip/deploy#add-a-new-data-source) for full instructions for setting up a data source. 2. In step 2 of the flow, use the following data source connection configuration. This connects to the example data source you created during [Set up](#set-up). ```yaml data_source nyc_bus_breakdowns_and_delays: type: postgres connection: host: sodademonyc-postgresql port: "5432" username: ${POSTGRES_USERNAME} password: ${POSTGRES_PASSWORD} database: nyc schema: public ``` 3. Complete the guided workflow to **Save & Run** a scan of the data source to validate that Soda Cloud can access the data in the example data source via the Soda Runner. It uses the external secrets manager configuration you set up to fetch, then pass the username and password to the data source. 4. Follow the instructions to [Define SodaCL checks](/soda-v3/soda-cl-overview#define-sodacl-checks) using no-code checks in Soda Cloud, then run scans for data quality. ## About the ClusterSecretStore The `ClusterSecretStore` is a YAML-configured set of instructions for accessing the external secrets manager which, in this case, is a Hashicorp Vault using a KV Secrets Engine V2. Note that some values in the example are generated; values in your own file vary. ``` apiVersion: external-secrets.io/v1beta1 kind: ClusterSecretStore metadata: name: vault-app-role spec: provider: vault: auth: appRole: path: approle roleId: 3e94ee54-1799-936e-9cec-5c5a19a5eeeb secretRef: key: appRoleSecretId name: external-secrets-vault-app-role-secret-id namespace: external-secrets path: kv server: http://vault.vault.svc.cluster.local:8200 version: v2 ``` ## About the ExternalSecret The `ExternalSecret` is a separate YAML-based set of instructions for which secrets to fetch. The example below references the `ClusterSecretStore` above, which facilitates access to the Hashicorp Vault. The Soda Runner uses the `ExternalSecret` to retrieve data source credential values. ``` apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: soda-runner namespace: soda-runner spec: data: - remoteRef: key: local/soda property: POSTGRES_USERNAME secretKey: POSTGRES_USERNAME - remoteRef: key: local/soda property: POSTGRES_PASSWORD secretKey: POSTGRES_PASSWORD refreshInterval: 1m secretStoreRef: kind: ClusterSecretStore name: vault-app-role target: name: soda-runner-secrets template: data: soda-runner.conf: | POSTGRES_USERNAME={{ .POSTGRES_USERNAME }} POSTGRES_PASSWORD={{ .POSTGRES_PASSWORD }} engineVersion: v2 ``` The `target template` configuration in the `ExernalSecret` creates a file called `soda-runner.conf` into which it adds the username and password values in the [dotenv format](https://saurabh-kumar.com/python-dotenv/#file-format) that the Soda Runner expects. ## Go further * Access standard instructions to [integrate with a secrets manager](/soda-v3/quick-start-sip/extras#integrate-with-a-secrets-manager). * [Get organized](/soda-v3/collaborate) in Soda! * [Request a demo](https://www.soda.io/schedule-a-demo). Hey, what can Soda do for you? > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Generate API keys Generate API keys to securely connect Soda Library or a Soda Runner to Soda Cloud, or to access Soda Cloud via API. Soda Cloud uses API keys to securely communicate with other entities such as Soda Library and self-hosted Soda Runners, and to provide secure access to Soda Cloud via API. There are two sets of API keys that you can generate and use with Soda Cloud: * API keys for communicating with **Soda Library**, the **Soda Cloud API** or **Soda Cloud Reporting API**, and the Soda Library Docker image that the **GitHub Action for Soda** uses * API keys for communicating with a self-hosted **Soda Runner** Note that you can use other authentication methods to access Soda Cloud metadata via the Reporting API such as HTTPBasic authentication with username and password, or authentication using tokens; use API keys to authenticate access if your organization employs Single Sign On (SSO) to access Soda Cloud. ## Generate API keys for use with Soda Library or a Soda Cloud API 1. In your Soda Cloud account, navigate to **your avatar** > **Profile**, then navigate to the **API Keys** tab. Click the plus icon to generate new API keys. 2. Copy the syntax for the `soda_cloud` configuration, including the values **API Key ID** and **API Key Secret**, then apply the keys according to how you intend to use them: * for use in a `configuration.yml` file: follow [Configure Soda](/soda-v3/quick-start-sip/install#configure-soda) * for use with the Reporting API if your organization uses Single Sign On (SSO) to access Soda Cloud: follow [Reporting API authentication](/soda-v3/reporting-api-v1/authentication) ## Generate API keys for use with a Soda Runner 1. In your Soda Cloud account, navigate to **your avatar** > **Data Sources**, then navigate to the **Runners** tab. Click **New Soda Runner**. 2. Copy the values of the **API Key ID** and **API Key Secret** to a secure location, then apply the keys according to the instructions in the [Deploy a Soda Runner](/soda-v3/quick-start-sip/deploy#deploy-a-runner) documentation. ## Go further * Learn more about integrating with third-party tools via a [webhook](/soda-v3/integrate-soda/integrate-webhooks). * Access a list of [all integrations](https://www.soda.io/integrations) that Soda Cloud supports. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Manage sensitive data Learn how to mitigate the exposure of sensitive information in Soda Cloud. Soda provides several capabilities and configurable settings that help you manage access to sensitive data. What follows are several options that you can implement to guard against unauthorized access to sensitive data that Soda may check for data quality. ## Utilize roles and permissions in Soda Cloud Soda Cloud employs roles and permissions that apply to users of an organization's account. These access controls enable you to define who can access, add, change, or delete metadata or access to data in the account. Refer to [Manage roles, user groups, and settings](/soda-v3/collaborate/roles-global) for much more detail and guidance on how to limit access. ## Deploy a self-hosted Soda Runner Soda's self-hosted runner is a containerized Soda Library deployed in a Kubernetes cluster in your cloud services provider environment, such as Azure or AWS. It enables users of Soda Cloud to securely access your data sources so it can perform data quality scanning while meeting your infrastructure team’s security rules and requirements that protect credentials and record-level data from exposure. Consider [deploying a self-hosted runner](/soda-v3/quick-start-sip/deploy) in your own infrastructure to securely manage access to your data sources. See also: [Soda architecture](/soda-v3/learning-resources/soda-cloud-architecture) Further, if you use an external secrets manager such as Hashicorp Vault or AWS Secrets Manager, you may wish to [integrate](/soda-v3/quick-start-sip/extras#integrate-with-a-secrets-manager) your self-hosted Soda Runner with your secrets manager to securely and efficiently grant Soda access to data sources that use frequently-rotated login credentials. ## Limit data sampling During the data source onboarding process, you have the option to [configure Soda](/soda-v3/soda-cl-overview/sample-datasets) to collect and store 100 rows of sample data for the datasets in the data source. This is a feature you must implicitly configure; Soda does not collect sample rows of data by default. These samples, accessible in Soda Cloud as in the example below, enable users to gain insight into the data's characteristics, facilitating the formulation of data quality rules.
### Turn off sample data collection Where your datasets contain sensitive or private information, you may not want to collect, send, store, or visualize *any* samples from your data source to Soda Cloud. In such a circumstance, you can disable the feature completely in Soda Cloud. To prevent Soda Cloud from receiving any sample data or failed row samples for any datasets in any data sources to which you have connected your Soda Cloud account, proceed as follows: 1. As a user with [permission](/soda-v3/collaborate/roles-global#global-roles-and-permissions) to do so, log in to your Soda Cloud account and navigate to **your avatar** > **Organization Settings**. 2. In the **Organization** tab, uncheck the box to **Allow Soda to collect sample data and failed row samples for all datasets**, then **Save**. Alternatively, if you use Soda Library, you can adjust the configuration in your `configuration.yml` to disable all samples for an individual data source, as in the following example. ```yaml data_source my_datasource: type: postgres ... sampler: disable_samples: True ``` ### Limit sample data collection If you wish to provide sample rows for some datasets and only wish to limit the ones for which Soda collects samples, you can add a `sample datasets` configuration to your data source. Navigate to **your avatar** > **Data Sources** > **New Data Source**, or select an existing data source, to begin. You can add this configuration to one of two places: * to either step 3. Discover\ OR * step 4. Profile The example configuration below uses a wildcard character (`%`) to specify that Soda collects sample rows for all datasets with names that begin with `region`, and *not* to send samples for any other datasets in the data source. ```yaml sample datasets: datasets: - include region% ``` The following example excludes a list of datasets from any sampling, and implicitly collects samples for all other datasets in the data source. ```yaml sample datasets: datasets: - exclude [credit_card, birth_date] ``` * If you configure `sample datasets` to include specific datasets, Soda implicitly *excludes* all other datasets from sampling. * If you combine an include config and an exclude config and a dataset fits both patterns, Soda excludes the dataset from sampling. * For excluded datasets, Soda does not generate, store, or offer visualizations of sample rows anywhere. For those datasets without sample rows, users must use another tool, such as a query builder for a data source, to collect any sample data they require. * No other functionality within Soda relies on these sample rows; if you exclude a dataset in a sample configuration, you can still configure individual failed row checks which collect independent failed row samples at scan time. Learn more about [managing failed row samples](/soda-v3/run-a-scan/failed-row-samples). ## Limit data profiling During the data source onboarding process, you have the option to [configure Soda](/soda-v3/soda-cl-overview/profile) to profile the datasets, and/or their columns, when it connects to the data source. When it **discovers datasets**, as in the example below, Soda captures only the names of the datasets in the data source, each dataset's schema, and the data type of each column.
When it **profiles datasets**, as in the example below, Soda automatically evaluates several data quality metrics for each column of a dataset based on the column's data type, such as missing and distinct values, calculated statistical metrics, and frequently occurring values. The majority of these metrics are aggregated which safeguards against the exposure of record-level data. In instances where a column contains categorical data, profiling provides insights into the most extreme and frequent values, which could potentially reveal information about the data. However, as Soda only exposes individual metric values, end-users cannot link these calculated metrics to specific records.
### Limit or turn off dataset discovery If you wish to limit the profiling that Soda performs on datasets in a data source, or limit the datasets which it discovers, you can do so at the data source level as part of the guided workflow to create or edit a data source. Navigate to **your avatar** > **Data Sources** > **New Data Source**, or select an existing data source, to begin. In step 3 of the guided workflow, **Discover**, you have the option of listing the datasets you wish to discover, as in the example below. Refer to [Add dataset discovery](/soda-v3/soda-cl-overview/profile#add-dataset-discovery) for many examples and variations of this configuration. ```yaml discover datasets: datasets: - include % - exclude test% ``` To avoid discovering *any* datasets in your data source, use the following configuration. ```yaml discover datasets: datasets: - exclude % ``` ### Limit or turn off dataset profiling If you wish to limit the profiling that Soda performs on datasets in a data source, or limit the datasets which it profiles, you can do so at the data source level as part of the guided workflow to create or edit a data source. Navigate to **your avatar** > **Data Sources** > **New Data Source**, or select an existing data source, to begin. In step 4 of the guided workflow, **Profile**, you have the option of listing the datasets you wish to profile, as in the example below which excludes columns that begin with `pii` and any columns that contain `email` in their names. Refer to [Add column profiling](/soda-v3/soda-cl-overview/profile#add-column-profiling) for many examples and variations of this configuration. ```yaml profile columns: columns: - exclude %.pii_% - exclude %.%email% ``` To avoid profiling *any* datasets in your data source, use the following configuration. ```yaml profile columns: columns: - exclude %.% ``` Dataset profiling can be resource-heavy, so carefully consider the datasets for which you truly need column profile information. Refer to [Compute consumption and cost considerations](/soda-v3/soda-cl-overview/profile#compute-consumption-and-cost-considerations) for more detail. ## Limit failed row sampling When a scan results in a failed check, the CLI output displays information about the check that failed and why, including the [actual SQL queries](/soda-v3/run-a-scan/failed-row-samples#about-failed-row-samples) that retrieve failed row samples. To offer more insight into the data that failed a check, Soda Cloud displays failed row samples in a check result’s measurement history, as in the example below. There are two ways Soda collects and displays failed row samples in your Soda Cloud account. * **Implicitly:** Soda automatically collects 100 failed row samples for the following checks: * [reference check](/soda-v3/sodacl-reference/reference#failed-row-samples) * checks that use a [missing metric](/soda-v3/sodacl-reference/missing-metrics#failed-row-samples) * checks that use a [validity metric](/soda-v3/sodacl-reference/validity-metrics#failed-row-samples) * checks that use a [duplicate metric](/soda-v3/sodacl-reference/numeric-metrics#failed-row-samples) * [metric reconciliation check](/soda-v3/sodacl-reference/recon#metric-reconciliation-checks) that include missing, validity, or duplicate metrics, or reference checks * [record reconciliation checks](/soda-v3/sodacl-reference/recon#record-reconciliation-checks) * **Explicitly:** Soda automatically collects 100 failed row samples for the following explicitly-configured checks: * [failed rows check](/soda-v3/sodacl-reference/failed-rows-checks) * [user-defined checks](/soda-v3/sodacl-reference/user-defined) that use the `failed rows query` configuration
### Turn off failed row sampling Where your datasets contain sensitive or private information, you may not want to collect, send, store, or visualize *any* samples from your data source to Soda Cloud. In such a circumstance, you can disable the feature completely in Soda Cloud. Users frequently disable failed row sampling in Soda Cloud and, instead, reroute failed row samples to an internal database; see [Reroute failed row samples](/soda-v3/use-case-guides/route-failed-rows) below. ### Customize failed row sampling For checks which implicitly collect failed rows samples, you can add a configuration to prevent Soda from collecting those samples from specific columns or datasets that contain sensitive data. For example, you may wish to exclude a column that contains personal identifiable information (PII) such as credit card numbers from the Soda query that collects samples. Refer to [manage failed row samples](/soda-v3/run-a-scan/failed-row-samples) for extensive options and details. ### Reroute failed row samples If the data you are checking contains sensitive information, you may wish to send any failed rows samples that Soda collects to a secure, internal location rather than Soda Cloud. These configurations apply to checks defined as no-code checks, in an agreement, or in a checks YAML file. To do so, you have two options: 1. **HTTP sampler**: Create a function, such as a lambda function, available at a specific URL within your environment that Soda can invoke for every check result in a data source that fails and includes failed row samples. Use the function to perform any necessary parsing from JSON to your desired format (CSV, Parquet, etc.) and store the failed row samples in a location of your choice. 2. **Python CustomSampler**: If you run programmatic Soda scans of your data, add a custom sampler to your Python script to collect samples of rows with a `fail` check result. Once collected, you can print the failed row samples in the CLI, for example, or save them to an alternate destination. Learn how to define custom samplers in [Manage failed rows samples](/soda-v3/run-a-scan/failed-row-samples). ## Go further * Learn more about how to [manage failed row samples](/soda-v3/run-a-scan/failed-row-samples). * Run scans locally to prevent Soda Library from pushing results to Soda Cloud. Access the **Prevent pushing scan results to Soda Cloud** in the [Run a scan](/soda-v3/run-a-scan#scan-for-data-quality) tab. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Reroute failed row samples Learn how to programmatically use Soda Library with an example script to reroute failed row samples to the CLI output instead of Soda Cloud. Using Soda Library, you can programmatically run scans that reroute failed row samples to display them in the command-line instead of Soda Cloud. By default, Soda Library *implicitly* pushes samples of any failed rows to Soda Cloud for missing, validity, duplicate, and reference checks; see [About failed row samples](/soda-v3/run-a-scan/failed-row-samples#about-failed-row-samples). Instead of sending the results to Soda Cloud, you can use a Python custom sampler to programmatically instruct Soda to display those samples in the command-line. Follow the instructions below to modify an example script and run it locally to invoke Soda to run a scan on example data and display samples in the command-line for the rows that failed missing, validity, duplicate, and reference checks. This example uses Dask and Pandas to convert CSV sample data into a DataFrame on which Soda can run a scan, and also to convert failed row samples into a CSV to route them to, or display them in, a non-Soda Cloud location. Note that although the example does not send failed row samples to Soda Cloud, it does still send dataset profile information and the data quality check results to Soda Cloud. ## Prerequisites * a code or text editor such as PyCharm or Visual Studio Code * Python 3.8, 3.9, or 3.10 * Pip 21.0 or greater ## Set up and run example script Jump to: [script](#example-script) 1. In a browser, navigate to [cloud.soda.io/signup](https://cloud.soda.io/signup) to create a new Soda account, which is free for a 45-day trial. If you already have a Soda account, log in. 2. Navigate to **your avatar** > **Profile**, then access the **API keys** tab. Click the plus icon to generate new API keys. Copy+paste the API key values to a temporary, secure place in your local environment.
Why do I need a Soda Cloud account? To validate your account license or free trial, the Soda Library Docker image that the GitHub Action uses to execute scans must communicate with a Soda Cloud account via API keys.\ Create new API keys in your Soda Cloud account, then use them to configure the connection between the Soda Library Docker image and your account later in this procedure.
3. Best practice dictates that you run Soda in a virtual environment. From the command line, create a new directory in your environment, then use the following command to create, then activate, a virtual environment called `.sodadataframes`. ```shell python -m venv .sodadataframes # MacOS source .sodadataframes/bin/activate # Windows source .sodadataframes\Scripts\activate ``` 4. Run the following commands to upgrade pip, then install Soda Library for Dask and Pandas. ```shell # MacOS pip install --upgrade pip # Windows python.exe -m pip install --upgrade pip pip install -i https://pypi.cloud.soda.io soda-pandas-dask ``` 5. Copy + paste the [script](#example-script) below into a new `Soda-dask-pandas-example.py` file in the same directory in which you created your virtual environment. In the file, replace the above-the-line values with your own Soda Cloud values, then save the file. 6. From the command-line, use the following command to run the example and see both the scan results and the failed row samples as command-line output. ```shell python Soda-dask-pandas-example.py ``` Output: ```shell By downloading and using Soda Library, you agree to Sodas Terms & Conditions (https://go.soda.io/t&c) and Privacy Policy (https://go.soda.io/privacy). Running column profiling for data source: pandas_reference_example Profiling columns for the following tables: - soda_pandas_example Scan summary: 5/7 checks PASSED: soda_pandas_example in pandas_reference_example No blank values in Name [PASSED] No blank values in Age [PASSED] No blank values in City [PASSED] No blank values in Country [PASSED] Emails Addresses are formatted correctly [PASSED] 2/7 checks FAILED: soda_pandas_example in pandas_reference_example Alpha2 Country Codes must be valid [FAILED] value: 1 No duplicate Email Addresses [FAILED] check_value: 1 Oops! 2 failures. 0 warnings. 0 errors. 5 pass. Sending results to Soda Cloud Soda Cloud Trace: 628131**** Failed Rows in a Dataframe Example ----------------------------------- name age city email country failed_check created_at 0 Alice 25 New York alice@example.net US No duplicate Email Addresses 2024-03-12 10:40:55.681690 1 Bob 30 Los Angeles alice@example.net BT No duplicate Email Addresses 2024-03-12 10:40:55.681690 2 Charlie 66 Chicago alice@example.net BO No duplicate Email Addresses 2024-03-12 10:40:55.681690 3 David 87 Chicago1 alice@example.net ABC No duplicate Email Addresses 2024-03-12 10:40:55.681690 4 David 87 Chicago1 alice@example.net ABC Alpha2 Country Codes must be valid 2024-03-12 10:40:55.731225 ``` 7. In your Soda Cloud account, navigate to **Datasets**, then click to open **soda.pandas.example**. Soda displays the check results for the scan you just executed via the command-line.\ If you wish, click the **Columns** tab to view the dataset profile information Soda Library collected and pushed to Soda Cloud.
8. Click the **Alpha2 Country Codes must be valid** row to view the latest check result, which failed. Note that Soda Cloud does not display a tab for **Failed Rows Analysis** which would normally contain samples of failed rows from the scan.
#### Example script ```python import pandas as pd from soda.scan import Scan from soda.sampler.sampler import Sampler from soda.sampler.sample_context import SampleContext from datetime import datetime import json import os # For the US Region, use "cloud.us.soda.io". # For the EU region, use "cloud.soda.io". soda_cloud_host = "cloud.soda.io" # Input the API keys you generated in step 2. cloud_apikeyID = "XXX" cloud_apikeySecret = "XXX" # Set to "true" to view failed row samples in Soda Cloud. # Set to "false" to view samples in the CLI/separate DataFrame. failed_rows_cloud = "false" # ---------------------------------------------------------------------------------------- # Reroute failed row samples (exceptions) class CustomSampler(Sampler): def store_sample(self, sample_context: SampleContext): rows = sample_context.sample.get_rows() json_data = json.dumps(rows) # Convert failed rows to JSON exceptions_df = pd.read_json(json_data) #create dataframe with failed rows # Define exceptions dataframe exceptions_schema = sample_context.sample.get_schema().get_dict() exception_df_schema = [] for n in exceptions_schema: exception_df_schema.append(n["name"]) exceptions_df.columns = exception_df_schema check_name = sample_context.check_name exceptions_df['failed_check'] = check_name exceptions_df['created_at'] = datetime.now() exceptions_df.to_csv(check_name+".csv", sep=",", index=False, encoding="utf-8") # Sample data1 data_list = [ {'name': 'Alice', 'age': 25, 'city': 'New York', 'email': 'alice@example.net', 'country': 'US'}, {'name': 'Bob', 'age': 30, 'city': 'Los Angeles', 'email': 'alice@example.net', 'country': 'BT'}, {'name': 'Charlie', 'age': 66, 'city': 'Chicago', 'email': 'alice@example.net', 'country': 'BO'}, {'name': 'David', 'age': 87, 'city': 'Chicago1', 'email': 'alice@example.net', 'country': 'ABC'} ] # Sample data2 reference_list = [ {'iso2_country': 'US'}, {'iso2_country': 'BT'}, {'iso2_country': 'BO'}, {'iso2_country': 'CN'} ] # Convert Sample data1 to a Pandas DataFrame pandas_frame1 = pd.DataFrame(data_list) # Convert Sample data2 to a Pandas DataFrame pandas_frame2 = pd.DataFrame(reference_list) # Setup Soda data quality scan scan = Scan() scan.add_pandas_dataframe(dataset_name="soda_pandas_example", pandas_df=pandas_frame1, data_source_name="pandas_reference_example") scan.add_pandas_dataframe(dataset_name="reference", pandas_df=pandas_frame2, data_source_name="pandas_reference_example") # reference List scan.set_scan_definition_name("pandas_reference_example") scan.set_data_source_name("pandas_reference_example") if failed_rows_cloud == "false": scan.sampler = CustomSampler() # Define data quality checks using SodaCL checks = """ checks for soda_pandas_example: - missing_count(name) = 0: name: No blank values in Name - missing_count(age) = 0: name: No blank values in Age - missing_count(city) = 0: name: No blank values in City - missing_count(country) = 0: name: No blank values in Country - invalid_count(email) = 0: valid format: email name: Email addresses are formatted correctly - duplicate_count(email) = 0: name: No duplicate email addresses - values in (country) must exist in reference (iso2_country): name: Alpha2 country codes must be valid profile columns: columns: - include soda_pandas_example.% """ config = f""" soda_cloud: host: {soda_cloud_host} api_key_id: {cloud_apikeyID} api_key_secret: {cloud_apikeySecret} """ # Execute a scan scan.add_sodacl_yaml_str(checks) scan.add_configuration_yaml_str(config) # When testing, you can set scan.set_is_local(True) to avoid sending failed row samples to Soda Cloud. scan.set_is_local(False) scan.execute() # Create a DataFrame for any exceptions # Optionally, you can write this DataFrame to an external table. if failed_rows_cloud == "false": current_dir = os.path.dirname(os.path.realpath(__file__)) csv_files = [file for file in os.listdir(current_dir) if file.endswith('.soda')] if len(csv_files) == 0: pass else: dfs = [] for file in csv_files: file_path = os.path.join(current_dir, file) df = pd.read_csv(file_path) dfs.append(df) if len(dfs) == 1: combined_df = dfs[0] else: combined_df = pd.concat(dfs, ignore_index=True) print("Failed Rows in a Dataframe Example") print("-----------------------------------") print(combined_df) # remove the CSV files that were created for file in csv_files: os.remove(os.path.join(current_dir, file)) ``` ## Go further * Learn how to [Manage sensitive data](/soda-v3/use-case-guides/sensitive-data) in Soda Cloud. * Learn how to [Disable failed rows sampling for specific columns](/soda-v3/run-a-scan/failed-row-samples#disable-failed-row-samples). * [Disable samples in Soda Cloud](/soda-v3/soda-cl-overview/sample-datasets#disable-samples-in-soda-cloud) entirely. * Learn how to use a custom sampler to route failed row samples to an [external storage location](/soda-v3/run-a-scan/failed-row-samples#reroute-failed-row-samples). > Not quite ready for this big gulp of Soda? 🥤Try [taking a sip](/soda-v3/quick-start-sip), first. # Double-onboard a data source Learn how to double-onboard a data source to leverage all the features supported by Soda Runners. To scan your data for quality, Soda must connect to a data source using connection configurations (host, port, login credentials, etc.) that you either define in Soda Cloud during onboarding using a Soda Runner, or in a configuration YAML file you reference during programmatic or CLI scans using Soda Library. Soda recognizes each data source you onboard as an independent resource in Soda Cloud, where it displays all scan results and failed row samples for all data sources regardless of onboarding method. However, data sources you connect via a Soda Runner using the guided workflow in Soda Cloud support several features which data sources you connect via Soda Library do not, including: * [no-code checks](/soda-v3/soda-cl-overview#define-sodacl-checks) * [Discussions](/soda-v3/use-case-guides/quick-start-end-user#begin-a-discussion-and-propose-checks) * [scan scheduling](/soda-v3/run-a-scan#scan-for-data-quality) * [anomaly dashboards](/soda-v3/collaborate/anomaly-dashboard) **Available in 2025** If you have onboarded a data source via Soda Library but you wish to take advantage of the features available to Soda Runner-onboarded data sources, you can double-onboard an existing data source. > See also: [Soda overview](/soda-v3/learning-resources/product-overview) > > See also: [Choose a flavor of Soda](/soda-v3/quick-start-sip/setup-guide) > > See also: [Add a new data source](/soda-v3/quick-start-sip/managed-agent) in Soda Cloud ## Prerequisites * You [installed Soda Library](/soda-v3/quick-start-sip/install), you have configured it to connect to your data source, and you have run at least one [scan](/soda-v3/run-a-scan#scan-for-data-quality) programmatically or via the Soda Library CLI. * You have deployed a [self-hosted Soda Runner](/soda-v3/quick-start-sip/deploy) Helm chart in a Kubernetes cluster in your cloud services environment\ OR\ Someone with Soda Admin privileges in your organization’s Soda Cloud account has navigated to **your avatar** > **Organization Settings** check the box to **Enable Soda-hosted Runner**; see [Set up a Soda-hosted runner](/soda-v3/quick-start-sip/managed-agent). * You have access to the connection configurations (host, port, login credentials, etc.) for your data source. * Your data source is compatible with a Soda Runner; refer to tables below. #### Self-hosted runner |

Amazon Athena
Amazon Redshift
Azure Synapse
ClickHouse
Databricks SQL
Denodo
Dremio
DuckDB
GCP BigQuery
Google CloudSQL

|

IBM DB2
MotherDuck
MS SQL Server1
MySQL
OracleDB
PostgreSQL
Presto
Snowflake
Trino
Vertica

| | --------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | 1 MS SQL Server with Windows Authentication does not work with Soda Runner out-of-the-box. #### Soda-hosted runner |

BigQuery
Databricks SQL
MS SQL Server
MySQL

|

PostgreSQL
Redshift
Snowflake

| | ----------------------------------------------------------- | ---------------------------------------------- | ## Onboard an existing data source 1. Log in to Soda Cloud, then navigate to **your avatar** > **Data Sources**. 2. From the list of data sources connected to your Soda Cloud account, click to select and open the one you onboarded via Soda Library and now wish to double-onboard via a Soda Runner. 3. Follow the guided workflow to onboard the existing data source via a Soda Runner, starting by using the dropdown to select the **Default Scan Runner** you wish to use to connect to the data source. 4. Complete the [guided steps](/soda-v3/quick-start-sip/managed-agent#add-a-new-data-source) to: * define a schedule for your default scan definition * provide connection configuration details for the data source such as name, schema, and login credentials, and test the connection to the data source * profile the datasets in the data source to gather basic metadata about the contents of each * identify the datasets to which you wish to apply automated monitoring for anomalies and schema changes * assign ownership roles for the data source and its datasets 5. **Save** your changes, then navigate to the **Datasets** page and select a dataset in the data source you just double-onboarded. 6. (Optional) If you wish, and if you have [requested preview access](https://go.soda.io/join-observability-preview) for the feature, you can follow the instructions to [activate the anomaly dashboard](/soda-v3/collaborate/anomaly-dashboard) for the dataset. 7. (Optional) Click **Add Check** and begin adding [no-code checks](/soda-v3/soda-cl-overview#define-sodacl-checks) to the dataset. ***Known issue***: Double-onboarding a data source renders Soda Library API keys invalid. After double-onboarding a data source, if you run a programmatic or CLI scan of that data source using Soda Library, an error appears to indicate that the API keys are invalid. As a workaround, [generate new API keys](/soda-v3/use-case-guides/api-keys) in Soda Cloud, then, in your configuration YAML, replace the old API key values with the newly-generated ones. ## Go further * Learn more about [automating anomaly detection](/soda-v3/collaborate/anomaly-dashboard) for observability. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Write SodaCL checks Soda Checks Language is a human-readable, domain-specific language for data reliability. You use SodaCL to define Soda Checks in a checks YAML file. **Soda Checks Language (SodaCL)** is a YAML-based, domain-specific language for data reliability. Used in conjunction with Soda tools, you use SodaCL to write checks for data quality, then run a scan of the data in your data source to execute those checks. A **Soda Check** is a test that Soda performs when it scans a dataset in your data source. A **Soda scan** executes the checks you write in an agreement, in a checks YAML file, or inline in a programmatic invocation, and returns a result for each check: pass, fail, or error. Optionally, you can configure a check to warn instead of fail by setting an [alert configuration](/soda-v3/sodacl-reference/optional-config#add-alert-configurations). As a step in the **Get started roadmap**, this guide offers instructions to define your first SodaCL checks in the Soda Cloud UI as no-code checks or in agreements, in a checks YAML file, or within a programmatic invocation of Soda. #### Get started roadmap 1. ~~Choose a flavor of Soda~~ 2. ~~Set up Soda: install, deploy, or invoke~~ 3. **Write SodaCL checks** 📍 You are here! 4. Run scans and review results 5. Organize, alert, investigate *** #### Examples ```yaml # Checks for basic validations checks for dim_customer: - row_count between 10 and 1000 - missing_count(birth_date) = 0 - invalid_percent(phone) < 1 %: valid format: phone number - invalid_count(number_cars_owned) = 0: valid min: 1 valid max: 6 - duplicate_count(phone) = 0 checks for dim_product: - avg(safety_stock_level) > 50 # Checks for schema changes - schema: name: Find forbidden, missing, or wrong type warn: when required column missing: [dealer_price, list_price] when forbidden column present: [credit_card] when wrong column type: standard_cost: money fail: when forbidden column present: [pii*] when wrong column index: model_name: 22 # Check for freshness - freshness (start_date) < 1d # Check for referential integrity checks for dim_department_group: - values in (department_group_name) must exist in dim_employee (department_name) ``` ## Define SodaCL checks {% tabs %} {% tab title="Use no-code checks" %} > 🎥 Watch a [5-minute video](https://www.soda.io/resources/introducing-no-code-checks) for no-code checks and discussions, if you like! ✖️ Requires Soda Core Scientific\ ✖️ Requires Soda Core\ ✖️ Requires Soda Library + Soda Cloud\ ✔️ Requires Soda Runner + Soda Cloud
### Prerequisites * You, or an Admin on your Soda Cloud account, has [deployed a Soda Runner](/soda-v3/quick-start-sip/deploy) version 0.8.52 or greater, and connected it to your Soda Cloud account. * You, or an Admin on your Soda Cloud account, has [added a new datasource](/soda-v3/quick-start-sip/deploy#add-a-new-data-source) via the Soda Runner in your Soda Cloud account *and* configured the data source to [discover the datasets](/soda-v3/soda-cl-overview/profile#add-dataset-discovery) in the data source for which you want to write no-code checks. (Soda must have access to dataset names and column names to present those values in dropdown menus during no-code check creation.) * You must have permission to edit the dataset; see [Manage dataset roles](/soda-v3/collaborate/roles-dataset). ### Create a new check SodaCL includes over 25 built-in metrics that you can use to write checks, a subset of which are accessible via no-codecheck creation. The table below lists the checks available to create via the no-code interface; access [SodaCL reference](/soda-v3/sodacl-reference) for detailed information about each metric or check. |

Missing
Validity
Numeric
Duplicate
Row count

|

Freshness
Schema
SQL Failed rows
SQL Metric

| | --------------------------------------------------------------- | -------------------------------------------------------------------- | 1. As a user with [permission](/soda-v3/collaborate/roles-dataset#dataset-roles-and-permissions) to do so of a dataset to which you wish to add checks, navigate to the dataset, then click **Add Check**. You can only create a check via the no-code interface for datasets in data sources connected via a Soda Runner. 2. Select the type of check you wish to create, then complete the form to create the check. Refer to table below for guidance on the values to enter. 3. Optionally, **Test** your check, then click **Propose check** to initiate a [**Discussion**](/soda-v3/learning-resources/glossary#discussion) with colleagues. Soda executes the check during the next scan according to the schedule you selected, or whenever a Soda Cloud user runs the schedule scan manually.\ Be aware that a schema check requires a minimum of two measurements before it yields a useful check result because it needs at least one historical measurement of the existing schema against which to compare a new measurement to look for changes. Thus, the first time Soda executes this check, the result is `[NOT EVALUATED]`, indicated by a gray, question mark status icon. 4. Click **Add Check** to include the new, no-code check in the next scheduled scan of the dataset. Note that a user with Viewer permissions cannot add a check, they can only propose checks. 5. Optionally, you can manually execute your check immediately. From the dataset’s page, locate the check you just created and click the stacked dots, then select **Execute Check**. Soda executes *only* your check. | Field or Label | Guidance | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Dataset | Select the dataset to which you want the check to apply. | | Check Name | Provide a unique name for your check. | | Add to Scan Definition | Select the scan definition to which you wish to add your check. Optionally, you can click **create a new Scan Definition** if you want Soda to execute the check more or less frequently, or at a different time of day than existing scan definitions dictate. See [Manage scheduled scans](/soda-v3/run-a-scan/scan-mgmt) for details. | | Filter fields | Optionally, add an [in-check filter](/soda-v3/sodacl-reference/optional-config#add-a-filter-to-a-check) to apply conditions that specify a portion of the data against which Soda executes the check. | | Define Metric/Values/Column/SQL |

As each metric or check requires different values, refer to SodaCL reference for detailed information about each metric or check.
Learn more about how Soda uses OpenAI to process the input for SQL and Regex assistants in no-code checks.

| | Alert Level |

Select the check result state(s) for which you wish to be notified: Fail, Warn, or Fail and Warn. See View scan results for details.
By default, alert notifications for your check go to the Dataset Owner. See Define alert notification rules to set up more alert notifications.

| | Fail Condition, Value, and Value Type |

Set the values of these fields to specify the threshold that constitutes a fail or warn check result.
For example, if you are creating a Duplicate Check and you want to make sure that less than 5% of the rows in the column you identified contain duplicates, set:
• Fail Condition to >
• Value to 5
• Value Type to Percent

| | Attribute fields | Select from among the list of existing attributes to apply to your check so as to organize your checks and alert notifications in Soda Cloud. Refer to [Add check attributes](/soda-v3/collaborate/check-attributes) for details. | ### **About Soda AI assistants** Powered by OpenAI's GPT-3.5 & GPT-4, the generative SQL and regular expression assistants available in Soda Cloud's no-code checks helps you write the queries and expressions you can add to validity, missing, SQL failed rows, and SQL metric checks. When creating a Missing or Validity check in the no-code user interface in Soda Cloud, you can click for help from the **Soda AI Regex Assistant** to translate an English request into a regular expression you can use to define missing or valid values. Similarly, access the **Soda AI SQL Assistant** in SQL Failed Rows or SQL Metric checks to generate SQL queries based on requests in plain English.
Soda AI SQL and Regex Assistants are enabled for *new* Soda Cloud accounts by default. If you do not wish to use them, navigate to **your avatar** > **Organization Settings**, then click to remove the check from the box for **Enable SQL and Regex Assistants Powered By Powered by OpenAI**. Existing Soda customers can review and accept the revised [Terms & Conditions](https://www.soda.io/terms-and-conditions), then [request access](https://go.soda.io/join-soda-ai-preview). Soda acknowledges that the output of the assistants may not be fully accurate or reliable. Leverage the assistants’ output, but be sure to carefully review all queries and expressions you add to your checks. Refer to [Soda’s General Terms & Conditions](https://www.soda.io/terms-and-conditions) in the Use of AI for further details. Be aware that Soda shares the content of all SQL and Regex assistant prompts/input and output with OpenAI to perform the processing that yields the output. Following OpenAI’s suggestion, Soda also sends metadata, such as schema information, to OpenAI along with the prompts/input in order to improve the quality of the output. Read more about OpenAI at . The **Ask AI Assistant** is powered by kapa.ai and replaces SodaGPT. While Soda collaborates with third parties to develop certain AI features, it’s important to note that Soda does not disclose any primary data with our partners, such as data samples or data profiling details. We only share prompts and some schema information with OpenAI and kapa.ai to enhance the accuracy of the assistants. Refer to [Soda’s General Terms & Conditions](https://www.soda.io/terms-and-conditions) in the **Use of AI** section for further details. ### Define alert notification rules By default, alert notifications for your no-code check go to the **Dataset Owner** and **Check Owner**. If you wish to send alerts elsewhere, in addition to the owner, create a notification rule. For a new rule, you define conditions for sending notifications including the severity of a check result and whom to notify when bad data triggers an alert. In Soda Cloud, navigate to **your avatar** > **Notification Rules**, then click **New Notification Rule**. Follow the guided steps to complete the new rule. Use the table below for insight into the values to enter in the fields and editing panels. | Field or Label | Guidance | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Name | Provide a unique identifier for your notification. | | For | Select **All Checks**, or select **Selected Checks** to use conditions to identify specific checks to which you want the rule to apply. You can identify checks according to several attributes such as **Data Source Name**, **Dataset Name**, or **Check Name**. | | Notify Recipient | Select the destination to which this rule sends its notifications. For example, you can send the rule’s notifications to a channel in Slack. | | Notify About | Identify the notifications this rule sends based on the severity of the check result: warn, fail, or both. | ### Edit an existing check 1. As a user with permission to do so, navigate to the dataset in which the no-code check exists. 2. To the right of the check you wish to edit, click the stacked dots, then select **Edit Check**. You can only edit a check via the no-code interface if it was first created as a no-code check, as indicated by the cloud icon in the **Origin** column of the table of checks. 3. Adjust the check as needed, test your check, then save. Soda executes the check during the next scan according to the scan definition you selected. 4. Optionally, you can execute your check immediately. Locate the check you just edited and click the stacked dots, then select **Execute Check**. Soda executes *only* your check. {% endtab %} {% tab title="Use an agreement" %} You can write SodaCL checks directly in the Soda Cloud user interface within an **agreement**. An agreement is a contract between stakeholders that stipulates the expected and agreed-upon state of data quality in a data source. In an agreement, use SodaCL checks to define the state of “good quality” for data in this data source, then identify and get approval from stakeholders in your organization. Define whom Soda Cloud will notify when a check in the agreement fails, then set a schedule to regularly execute the Soda Checks to uphold the tenets of the agreement. ✖️ Requires Soda Core Scientific\ ✖️ Requires Soda Core\ ✖️ Requires Soda Library + Soda Cloud\ ✔️ Requires Soda Agent + Soda Cloud
### Prerequisites * You, or an Admin on your Soda Cloud account, has [deployed a Soda Agent](https://docs.soda.io/soda-agent/deploy.html) and connected it to your Soda Cloud account. * You, or an Admin on your Soda Cloud account, has [added a new datasource](https://docs.soda.io/soda-agent/deploy.html#add-a-new-data-source) via the Soda Agent in your Soda Cloud account. ### Create a new agreement For a new agreement, you define several details including which data to check, what checks to execute during a scan, and whom to notify when bad data triggers an alert. In Soda Cloud, navigate to the **Agreements** dashboard, then click **New Agreement**. Follow the guided steps to complete the new agreement. Use the sections below for insight into the values to enter in the fields and editing panels in the guided steps. #### **1. Select a Data Source** You can only create an agreement that uses a data source that has been added to Soda Cloud via a Soda Agent. | Field or Label | Guidance | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Agreement Label | Provide a name for your agreement. | | Data Source |

Select the data source that contains the datasets to which your agreement applies.
If you have no options to select in the dropdown, it is because you have not added a data source via a Soda Agent. You can only create agreements on datasets that are in a data source that has been onboarded into Soda Cloud via a Soda Agent.

| #### **2. Write Checks** Use SodaCL to define the checks that Soda Cloud executes on a regular schedule to uphold the tenets of this agreement. If any of these checks fail during a regularly-scheduled scan, Soda Cloud notifies the stakeholders you specify in the Notifications section. Be sure to click **Test checks** to validate that the SodaCL syntax you have written is valid, and that Soda can execute the checks against your datasets without errors. For help writing your first checks: * browse the library of **SodaCL snippets** that insert correctly-formatted syntax for the most commonly-used checks for basic data quality * use **Ask AI**, a generative AI assistant that turns natural-language requests into production-ready SodaCL checks. [Read more](https://docs.soda.io/soda-cloud/ask-ai.html) * consider following the [Quick start for SodaCL](https://docs.soda.io/soda/quick-start-sodacl.html), including the [Tips and best practices](https://docs.soda.io/soda/quick-start-sodacl.html#tips-and-best-practices-for-sodacl) section * refer to [SodaCL reference](https://docs.soda.io/soda-cl/metrics-and-checks.html) for exhaustive details on every type of metric and check
#### **3. Identify Stakeholders** Add Stakeholders to this agreement who have an interest in maintaining or using the good-quality data in this data source. Consider adding a co-owner to your agreement for redundancy should you, as the agreement author, be absent. Soda Cloud sends emails to request review and approval from all stakeholders, and waits to run scans which execute checks in the agreement until all stakeholders have approved the agreement. #### **4. Set Notifications** By default, Soda Cloud includes an out-of-the-box email notification to all the agreement’s stakeholders when a check in your agreement fails. You can remove or adjust this notification, or use the search bar to add more. Access [View scan results](https://docs.soda.io/soda-library/run-a-scan.html#view-scan-results) to learn more about pass, warn, and fail check results. (Optional) If you have integrated your Soda Cloud account with [Slack](https://docs.soda.io/soda/integrate-slack.html) or another third-party service provider via a [webhook](https://docs.soda.io/soda/integrate-webhooks.html), use the search field to type a channel name to add the channel as a notification recipient. Alternatively, use the field to enter names of individual teammates with whom you collaborate in Soda Cloud. #### **5. Set a Scan Definition** After you have set up a new agreement, Soda Cloud sends approval requests to the stakeholders you identified in step 3. When stakeholders approve or reject your agreement, Soda Cloud sends you an email notification. Regardless of the approval status of the agreement, however, Soda Cloud begins running scans of your data according to the scan definition you set. Soda Cloud sends notifications after each scan according to the settings you defined in step 4. (Optional) You can click the link provided to create a new scan definition if you wish to run a scan to execute the checks in this agreement more or less frequently, or a different time of day, relative to the default scan definition for the data source. To review existing scan definitions, navigate to the **Scans** menu item. ### Agreement tips and best practices Further, take into account the following tips and best practices when writing SodaCL checks in an agreement. * Avoid applying the same [customized check names](https://docs.soda.io/soda-cl/optional-config.html#customize-check-names) in multiple agreements. Soda Cloud associates check results with agreements according to name so if you reuse custom names, Soda Cloud may become confused about which agreement to which to link check results. * If you use an [anomaly detection check](https://docs.soda.io/soda-cl/anomaly-detection.html), be aware that when you **Test Checks**, this type of checks results in `[NOT EVALUATED]`. The ML algorithm that anomaly detection checks use requires a minimum of four, regular-frequency scans before it has collected enough historic measurements against which to gauge an anomaly. Therefore, until it has collected enough historical measurements to use to gauge anomolies, Soda does not evaluate the check. * Note that any checks you test in the context of this step in the agreements workflow *do not* appear as “real” checks in the **Checks** dashboard. * Except with a NOW variable with freshness checks, you cannot use variables in checks you write in an agreement in Soda Cloud as it is impossible to provide the variable values at scan time. > See also: [Tips and best practices for SodaCL](https://docs.soda.io/soda/quick-start-sodacl.html#tips-and-best-practices-for-sodacl) > {% endtab %} {% tab title="Use a YAML file" %} As a Data Engineer, you can write SodaCL checks directly in a `checks.yml` file, or leverage check suggestions in the Soda Library CLI to prepare a basic set of data quality checks for you. Alternatively, you can add SodaCL checks to a programmatic invocation of Soda Library. ### Manually write SodaCL checks ✔️ Some checks require Soda Core Scientific\ ✔️ Supported in Soda Core\ ✔️ Supported in Soda Library + Soda Cloud\ ✔️ Supported in Soda Cloud Agreements + Soda Agent *** The checks YAML file stores the Soda Checks you write using SodaCL. Use this file to manually write your own SodaCL checks. 1. Using a code editor, create a new file called `checks.yml`. 2. Copy+paste the following basic check syntax in your file, then adjust the value for `dataset_name` to correspond with the name of one of the datasets in your data source. ```yaml checks for dataset_name: - row_count > 0 ``` 3. Save the changes to the `checks.yml` file. 4. To test the check and confirm the syntax is valid and error-free, use the following command to run a scan of the data in your data source. Replace the value for `my_datasource` with the name of the data source you added to your `configuration.yml` file. Read more about \[scans]\(). ```shell soda scan -d my_datasource -c configuration.yml checks.yml ``` Command-line Output: ```shell Soda Library 1.0.x Scan summary: 1/1 check PASSED: dim_customer in adventureworks row_count > 0 [PASSED] All is good. No failures. No warnings. No errors. Sending results to Soda Cloud Soda Cloud Trace: 67592***474 ``` 5. Add more checks to the `checks.yml` file to test for multiple data quality metrics. Consult the \[SodaCL tutorial]\() for advice and the \[Use case guides]\() for example checks. Refer to \[SodaCL reference]\() for exhaustive details on every type of metric and check. #### **Add a schema check** This type of check validates the schema, or structure, of your data. It ensures that the columns you expect to exist are present in the dataset, and that they have the correct data type and index location. Refer to [Schema checks](https://docs.soda.io/soda-cl/schema.html) for more information. ```yaml checks for dataset_A: - schema: name: Any schema changes fail: when schema changes: - column delete - column add - column index change - column type change ``` #### **Add row count checks** This step adds two checks: one to confirm that the dataset is not empty, and one to ensure that the current row count is not significantly different from the expected row count. Soda determines the expected row count relative to the previous row count value using a time series-based anomaly detection model. Refer to [Anomaly detection checks](https://docs.soda.io/soda-cl/anomaly-detection.html) for more information. ```yaml checks for dataset_A: - row_count > 0 - anomaly detection for row_count ``` #### **Add time-based partitioning** Also referred to as dataset filtering, this step prompts you to specify a time range on which to apply the data quality checks. By default, check suggestions sets the time-based partition to one day if the column contains DATE type data, and the preceding 24 hours if the column contains DATETIME data. When generating a list of candidate columns to which to apply the time-based partition, the assisstant uses heuristic methods to automatically identify and rank column names.
Read more about heuristic ranking The heuristic ranking strategy identifies the most suitable columns for effectively partitioning your data. The algorithm it uses for ranking applies several criteria and heuristic scoring to assess the columns' incrementality, standard deviation, maximum date, missing values, and column names.
1. Incrementality: This criterion checks whether the values in a time-based column incrementally change over time. It assesses if the date or time values consistently increase as new records are added. Columns with higher incrementality scores are more likely to provide a meaningful partitioning mechanism. 2. Standard Deviation: Check suggestions uses standard deviation between dates to assess the uniformity or distribution of values in a time-based column. Columns with low standard deviation indicate that the dates are closely packed together, suggesting a more consistent and evenly-spaced distribution. 3. Maximum Date: This step examines the maximum date value in a column and compares it to the current date. Columns with a maximum date value that is less than the current date receive a higher score. This criterion helps identify columns with recent data. 4. Missing Value: Check suggestions considers the number of missing values in a column; those with fewer missing values receive a higher score. This criterion helps identify columns with more complete data. 5. Column Name: Check suggestions analyzes the names of the columns to determine their relevance for partitioning. The algorithm assigns higher points to columns with names that contain keywords such as "create", "insert", "generate", etc. This criterion aims to identify columns that are likely to represent meaningful, time-based information. After calculating scores from each of the five criteria, the algorithm combines them to obtain a comprehensive score for each time-based column. The assistant then ranks the columns from highest to lowest score, providing guidance on the partitioning suitability of each column.
Refer to [Configure dataset filters](https://docs.soda.io/soda-cl/filters.html#configure-dataset-filters) for more information. ```yaml filter customer [daily]: where: created_at > TIMESTAMP '${NOW}' - interval '1d' checks for customer [daily]: - missing_count(name) < 5 - duplicate_count(phone) = 0 ``` #### **Add a freshness check** A freshness check ensures that the data in the dataset is up-to-date according to the latest value entered in a column containing date or timestamp values. Check suggestions uses the same heuristic methods with the time based partitioning to rank the columns. After ranking the columns, the CLI estimates the threshold by using the standard error of date differences. It then prompts you to select the column and threshold to use for the freshness check. Refer to [Freshness checks](https://docs.soda.io/soda-cl/freshness.html) for more information. ```yaml checks for dataset_A: - freshness(date_first_purchase) < 24h ``` #### **Add validity checks** A validity check compares the data in text columns to a specific format (see the list that follows) to determine whether the content is valid. For example, such a check can validate that all rows in an `id` column contain UUID-formatted values. Check suggestions prompts you to select the columns that are candidates for validity checks, which must contain text type data such as CHAR, VARCHAR, or TEXT. Valid formats: * UUID * email * phone number * credit card number * IP address (IPv4 and IPv6) * money * timestamp * date * time Refer to [Validity metrics](https://docs.soda.io/soda-cl/validity-metrics.html) for more information. ```yaml checks for dataset_A: - invalid_count(email_address) = 0: valid format: email ``` #### **Add missing checks** A missing check automatically identifies any NULL values within your dataset. Check suggestions prompts you to select the columns to which you want to apply a missing check. By default, it sets each check threshold to `0`, which means that a check fails if there are any NULL values in the column. Refer to [Missing metrics](https://docs.soda.io/soda-cl/missing-metrics.html) for more information. ```yaml checks for dataset_A: - missing_count(customer_key) = 0 - missing_count(geography_key) = 0 - missing_count(customer_alternate_key) = 0 - missing_count(title) = 0 - missing_count(first_name) = 0 - missing_count(middle_name) = 0 - missing_count(last_name) = 0 - missing_count(name_style) = 0 - missing_count(birth_date) = 0 - missing_count(marital_status) = 0 - missing_count(suffix) = 0 - missing_count(gender) = 0 ``` #### **Add duplicate checks** A duplicate check identifies duplicate records or entries within your dataset. By default, it sets each check threshold to `0`, which means that a check fails if there are any duplicate values in the column. Refer to [Numeric metrics](https://docs.soda.io/soda-cl/numeric-metrics.html#list-of-numeric-metrics) for more information. ```yaml checks for dataset_A: - duplicate_count(customer_key) = 0 - duplicate_count(geography_key) = 0 - duplicate_count(customer_alternate_key) = 0 - duplicate_count(title) = 0 - duplicate_count(first_name) = 0 - duplicate_count(middle_name) = 0 - duplicate_count(last_name) = 0 - duplicate_count(name_style) = 0 - duplicate_count(birth_date) = 0 - duplicate_count(marital_status) = 0 - duplicate_count(suffix) = 0 - duplicate_count(gender) = 0 ``` #### Programmatically add checks ✔️ Some checks require Soda Core Scientific\ ✔️ Some checks supported in Soda Core\ ✔️ Supported in Soda Library + Soda Cloud\ ✖️ Supported in Soda Cloud Agreements + Soda Agent *** Follow the [steps above](#manually-write-sodacl-checks) to create a `checks.yml` file to define your checks for data quality. Then, add the file(s) to your Python program as in the example below.\ Be sure to include any variables in your programmatic scan *before* the check YAML files. Soda requires the variable input for any variables defined in the check YAML files. ```python from soda.scan import Scan scan = Scan() scan.set_data_source_name("events") scan.add_configuration_yaml_file(file_path="~/.soda/my_local_soda_environment.yml") # Add variables ############### scan.add_variables({"date": "2022-01-01"}) # Add check YAML files ################## scan.add_sodacl_yaml_file("./my_programmatic_test_scan/sodacl_file_one.yml") scan.add_sodacl_yaml_file("./my_programmatic_test_scan/sodacl_file_two.yml") scan.add_sodacl_yaml_files("./my_scan_dir") scan.add_sodacl_yaml_files("./my_scan_dir/sodacl_file_three.yml") ``` {% endtab %} {% endtabs %} ## Next 1. ~~Choose a flavor of Soda~~ 2. ~~Set up Soda: install, deploy, or invoke~~ 3. ~~Write SodaCL checks~~ 4. [**Run scans and review results**](/soda-v3/run-a-scan) 5. Organize, alert, investigate > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # SodaCL tutorial Follow the quick start tutorial to get started with SodaCL, a human-readable, domain-specific language for data reliability. If you are staring at a blank page wondering what SodaCL checks to write to surface data quality issues, this quick start tutorial is for you. Alternatively, use the [Check suggestions](broken://pages/CpQuK4ZSGNMBZB4LMsfy) assistant in the Soda Library CLI to profile a dataset and auto-generate basic checks for data quality.
✖️ Requires Soda Core Scientific (included in a Soda Runner)\ ✔️ Supported in Soda Core\ ✔️ Supported in Soda Library + Soda Cloud\ ✔️ Supported in Soda Cloud Agreements + Soda Runner\ ✔️ Some available as no-code checks} *** ## SodaCL: In brief **Soda Checks Language (SodaCL)** is a YAML-based, domain-specific language for data reliability. Used in conjunction with Soda software, you use SodaCL to write checks for data quality, then run a scan of the data in your data source to execute those checks. A **Soda Check** is a test that Soda performs when it scans a dataset in your data source. A **Soda scan** executes the checks you defined and returns a result for each check: pass, fail, or error. Optionally, you can configure a check to warn instead of fail by setting an [alert configuration](/soda-v3/sodacl-reference/optional-config#add-alert-configurations). ## About this tutorial With over [25 built-in SodaCL checks and metrics](/soda-v3/sodacl-reference/metrics-and-checks#list-of-sodacl-metrics-and-checks) to choose from, it can be hard to know where to begin. This tutorial offers suggestions for some basic checks you can write to begin surfacing missing, invalid, unexpected data in your datasets. All the example checks in this tutorial use placeholder values for dataset and column name identifiers, but you can copy+paste the examples into your own checks YAML file and adjust the details to correspond to your own data. You do not need to follow the tutorial sequentially. ## Tutorial prerequisites * You have completed the [Get started](/soda-v3/quick-start-sip) tutorial\ OR\ you have followed the instructions on the [Roadmap](/soda-v3/quick-start-sip/get-started-roadmap) on your own. * You have created a new YAML file in your code editor and named it `checks.yml`\ OR\ you are on step 2 in the guided flow to create a new Soda Agreement. * (Optional) You have read the first two sections in [Metrics and checks](/soda-v3/sodacl-reference/metrics-and-checks) as a primer for SodaCL. ## Row count and cross checks One of the most basic checks you can write uses the `row_count` metric. When it executes the following check during a scan, Soda simply counts the rows in the dataset you identify in the `checks for` section header to confirm that the dataset is not empty. If it counts one or more rows, the check result is pass. ```yaml # Check that a dataset contains rows checks for dataset_name: - row_count > 0 ``` The check above is an example that use a numeric metric in a standard check pattern. By contrast, the following unique cross check compares row counts between datasets within the same data source without setting a threshold for volume, like `> 50`. This type of check is useful when, for example, you want to compare row counts to validate that a transformed dataset contains the same volume of data as the source from which it came. ```yaml # Compare row counts between datasets checks for dataset_name: - row_count same as other_dataset_name ``` [Run a scan](/soda-v3/run-a-scan) to execute your checks: ```yaml soda scan -d datasource_name -c configuration.yml checks.yml ``` ### Read more * [Numeric metrics](/soda-v3/sodacl-reference/numeric-metrics) * [Standard check pattern](/soda-v3/sodacl-reference/metrics-and-checks#check-types) * [Cross checks](/soda-v3/sodacl-reference/cross-row-checks) ## Duplicate check For the nearly universal use case that demands uniqueness, you can use the `duplicate_count` or `duplicate_percent` metrics. In the following example, Soda counts the number of duplicate values in the `column_name` column, identified as the argument in parentheses appended to the metric. If there is even one value that is a duplicate of another, the check result is fail. This type of check is useful when, for example, you need to make sure that values in an `id` column are unique, such `customer_id` or `product_id`. ```yaml # Check that a column does not contain any duplicate values checks for dataset_name: - duplicate_count(column_name) = 0 ``` If you wish, you can check for duplicate pairs in multiple columns. In the following example, Soda counts the number of duplicate values in both `column_name1` and `column_name2`. Be sure to add a space between the comma-separated values in the list of column names.
Example of duplicate pairs Rows 1 and 4 are duplicates. | - | column 1 | column 2 | | -- | -------- | -------- | | 1. | apple | banana | | 2. | apple | pear | | 3. | banana | pear | | 4. | apple | banana |
```yaml # Check that duplicate pairs do not exist between columns checks for dataset_name: - duplicate_count(column_name1, column_name2) = 0 ``` [Run a scan](/soda-v3/run-a-scan) to execute your checks: ```yaml soda scan -d datasource_name -c configuration.yml checks.yml ``` ### Read more * [Numeric metrics](/soda-v3/sodacl-reference/numeric-metrics) ## Freshness check If your dataset contains a column that stores timestamp information, you can configure a freshness check. This type of check is useful when, for example, you need to validate that the data feeding a weekly report or dashboard is not stale. Timely data is reliable data! In this example, the check fails if the most-recently added row (in other words, the youngest row) in the `timestamp_column_name` column is more than 24 hours old. ```yaml # Check that data in dataset is less than one day old checks for dataset_name: - freshness(timestamp_column_name) < 1d ``` [Run a scan](/soda-v3/run-a-scan) to execute your checks: ```yaml soda scan -d datasource_name -c configuration.yml checks.yml ``` ### Read more * [Freshness checks](/soda-v3/sodacl-reference/freshness) ## Missing and invalid checks SodaCL's missing metrics make it easy to find null values in a column. You don't even have to specify that `NULL` qualifies as a missing value because SodaCL registers null values as missing by default. The following check passes if there are no null values in `column_name`, identified as the value in parentheses. ```yaml # Check that there are no null values in a column checks for dataset_name: - missing_count(column_name) = 0 ``` If the type of data a dataset contains is TEXT (string, character varying, etc.), you can use an invalid metric to surface any rows that contain ill-formatted data. This type of check is useful when, for example, you need to validate that all values in an email address column are formatted as `name@domain.extension`. The following example fails if, during a scan, Soda discovers that more than 5% of the values in the `email_column_name` do not follow the email address format. ```yaml # Check an email column that all values are in email format checks for dataset_name: - invalid_percent(email_column_name) > 5%: valid format: email ``` If you want to surface more than just null values as missing, you can specify a list of values that, in the context of your business rules, qualify as missing. In the example check below, Soda registers `N/A`, `0000`, or `none` as missing values in addition to `NULL`; if it discovers more than 5% of the rows contain one of these values, the check fails. Note that the missing value `0000` is wrapped in single quotes; all numeric values you include in such a list must be wrapped in single quotes. ```yaml # Check that fewer than 5% of values in column contain missing values checks for dataset_name: - missing_percent(column_name) < 5%: missing values: [N/A, '0000', none] ``` [Run a scan](https://docs.soda.io/soda-library/run-a-scan.html) to execute your checks: ```yaml soda scan -d datasource_name -c configuration.yml checks.yml ``` ### Read more * [Missing metrics](/soda-v3/sodacl-reference/missing-metrics) * [Validity metrics](/soda-v3/sodacl-reference/validity-metrics) ## Reference checks If you need to validate that data in one column of a dataset exists in a column in another dataset, you can use a reference check. The following unique check compares the values of `state_code` to confirm that those values exist in `code` in the `iso_3166-2` dataset in the same data source. The check passes if the values in the `state_code` exist in `code`. ```yaml # Check that values in a column exist in another column in a different dataset checks for dataset_name: - values in (state_code) must exist in iso_3166-2 (code) ``` If you wish, you can compare the values of multiple columns in one check. Soda compares the column names respectively, so that in the following example, `column_name1` compares to `other_column1`, and `column_name2` compares to `other_column2`. ```yaml # Check that values in two columns exist in two other columns in a different dataset checks for dataset_name: - values in (column_name1, column_name2) must exist in different_dataset_name (other_column1, other_column2) ``` [Run a scan](https://docs.soda.io/soda-library/run-a-scan.html) to execute your checks: ```yaml soda scan -d datasource_name -c configuration.yml checks.yml ``` ### Read more * [Reference checks](/soda-v3/sodacl-reference/reference) ## Schema checks To eliminate the frustration of the silently evolving dataset schema, use schema checks with alert configurations to notify you when column changes occur. If you have set up a Soda Cloud account, you can use a catch-all schema check, also known as a schema evolution check, that results in a warning whenever a Soda scan reveals that a column has been added, removed, moved within the context of an index, or changed data type relative to the results of the previous scan. ```yaml # Check for any schema changes to dataset checks for dataset_name: - schema: warn: when schema changes: any ``` If you wish to apply a more granular approach to monitoring schema changes, you can specify columns in a dataset that ought to be present or which should not exist in the dataset. The following example warns you when, during a scan, Soda discovers that `column_name` is missing in the dataset; the check fails if either `column_name1` or `column_name2` exist in the dataset. This type of check is useful when, for example, you need to ensure that datasets do not contain columns of sensitive data such as credit card numbers or personally identifiable information (PII). ```yaml # Check for absent or forbidden columns in dataset checks for dataset_name: - schema: warn: when required column missing: [column_name] fail: when forbidden column present: [column_name1, column_name2] ``` Be aware that a check that contains one or more alert configurations only ever yields a *single* check result; one check yields one check result. If your check triggers both a warn and a fail, the check result only displays the more severe, failed check result. [Read more](/soda-v3/sodacl-reference/schema#expect-one-check-result). [Run a scan](/soda-v3/run-a-scan) to execute your checks: ```yaml soda scan -d datasource_name -c configuration.yml checks.yml ``` ### Read more * [Alert configuration](/soda-v3/sodacl-reference/optional-config#add-alert-configurations) * [Schema checks](/soda-v3/sodacl-reference/schema) * [Expect one check result](/soda-v3/sodacl-reference/optional-config#expect-one-check-result) ## Tips and best practices for SodaCL * Get your logic straight: your check defines a *passing* state, what you expect to see in your dataset. Do not define a failed state. * Take careful note of the data type of the column against which you run a check. For example, if numeric values are stored in a column as data type TEXT, a numeric check such as `min` or `avg` is incalculable. * A check that uses [alert configurations](/soda-v3/sodacl-reference/optional-config#add-alert-configurations) only ever returns *one* check result. See [Expect one check result](/soda-v3/sodacl-reference/optional-config#expect-one-check-result). * The `invalid format` configuration key only works with data type TEXT. See [Specify valid format](/soda-v3/sodacl-reference/validity-metrics#specify-valid-format). * Not all checks support in-check filters. See [List of compatible metrics and checks](/soda-v3/sodacl-reference/filters#list-of-compatible-metrics-and-checks). #### Best practices * To avoid typos or spelling errors, best practice dictates that you copy + paste any dataset or column names into your checks. * It is good practice to add a [custom name](/soda-v3/sodacl-reference/numeric-metrics) to your check. Establish a naming convention – word order, underscores, identifiers – and apply easily-digestible check names for any colleagues with whom you collaborate. #### Syntax tips * Be sure to add a colon to the end of a check whenever you add a second line to a check such as for a missing or invalid configuration key, or if you add a [custom name](/soda-v3/sodacl-reference/numeric-metrics) for your check. * Indentations in the SodaCL syntax are critical. If you encounter an error, check your indentation first. * Spaces in the SodaCL syntax are critical. For example, be sure to add a space before and after your threshold symbol ( `=`, `>`, `>=` ); do *not* add a space between a metric and the column to which it applies, such as `duplicate_count(column1)`. * All comma-separated values in lists in SodaCL use a comma + space syntax, such as `duplicate_count(column1, column2)`; do not forget to add the space. * Note that multi-word checks such as `missing_count` use underscores, but configuration keys, such as `missing regex`, do not. See [List of missing metrics](/soda-v3/soda-cl-overview/quick-start-sodacl#missing-and-invalid-checks) and [List of validity metrics](/soda-v3/soda-cl-overview/quick-start-sodacl). * If you use `missing values` or `invalid values` configuration keys, note that values in a comma-separated list must be enclosed in square brackets. For example, `[US, BE, CN]`. * Column names that contain colons or periods can interfere with SodaCL’s YAML-based syntax. For any column names that contain these punctuation marks, [apply quotes](/soda-v3/sodacl-reference/optional-config#use-quotes-in-a-check) to the column name in the check to prevent issues.\ If you are using a failed row check with a CTE fail condition, however, the syntax checker does not accept an expression that begins with double-quotes. In that case, as a workaround, add a meaningless `true and` to the beginning of the CTE, as in the following example. ```yaml checks for corp_value: - failed rows: fail condition: true and "column.name.PX" IS NOT null ``` ## Go further * Learn more about [SodaCL metrics and checks](/soda-v3/sodacl-reference/metrics-and-checks) in general. * Read about the [Optional configurations](/soda-v3/sodacl-reference/optional-config) you can apply to SodaCL checks. * [Get started](/soda-v3/quick-start-sip) to run a simple data quality scan on example data. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Write checks with Ask AI Use Soda's Ask AI assistant to turn natural language into production-ready data quality checks in SodaCL. **Ask AI** is an in-product generative AI assistant for data quality testing.\ Ask AI replaces SodaGPT, the original implementation of a generative AI assistant. ✖️ Requires Soda Core Scientific (included in a Soda Runner)\ ✖️ Supported in Soda Core\ ✔️ Supported in Soda Library + Soda Cloud\ ✔️ Supported in Soda Cloud + Soda Runner *** [Log in](https://cloud.soda.io/login) to your **Soda Cloud** account, click the **Ask AI** button in the main nav, then provide natural language instructions to the interface to: * receive fully-formed, syntax-correct checks in the [Soda Checks Language (SodaCL)](/soda-v3/sodacl-reference/metrics-and-checks) * get answers to questions about how to configure or use Soda * obtain advice about how to resolve an error while using Soda
## Enable Ask AI If you do not already have an account, [sign up for Soda Cloud](https://cloud.soda.io/signup) for a 45-day free trial. Then, as a user with [permission](/soda-v3/collaborate/roles-global#global-roles-and-permissions) to do so, navigate to **your avatar** > **Organization Settings**, then check the box to **Enable Ask AI powered by Kapa**.
Can't see the Ask AI button? If you are an existing Soda customer, you must accept Soda's revised terms and conditions for service that includes the use of third-party tools that facilitate generative AI capabilites. Reply to Soda's Terms & Conditions email to accept the revisions, or contact [Soda Support](mailto:support@soda.io) to arrange acceptance and enable the feature.\ \ If you have accepted the revised terms and conditions but still cannot see the Ask AI button, as a user with \[permission]\(/soda-cloud/roles-global.html#global-roles-and-permissions) to do so, navigate to **your avatar** > **Organization Settings**, then check the box to **Enable Ask AI powered by Kapa**.
## About Ask AI The **Ask AI Assistant** is powered by kapa.ai and replaces SodaGPT. While Soda collaborates with third parties to develop certain AI features, it’s important to note that Soda does not disclose any primary data with our partners, such as data samples or data profiling details. We only share prompts and some schema information with OpenAI and kapa.ai to enhance the accuracy of the assistants. Refer to [Soda’s General Terms & Conditions](https://www.soda.io/terms-and-conditions) in the **Use of AI** section for further details. ## Go further * Create [no-code checks](/soda-v3/soda-cl-overview#define-sodacl-checks) via the Soda Cloud user interface. * Get started with Soda by following a [tutorial](/soda-v3/quick-start-sip). * Consider using [check suggestions](broken://pages/CpQuK4ZSGNMBZB4LMsfy) to profile your data and suggest basic checks for data quality. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). # Add automated monitoring checks Use a SodaCL automated monitoring check to automatically check for row count anomalies and schema changes. {% hint style="warning" %} This feature is not supported in **Soda Core OSS**. [Migrate](https://docs.soda.io/soda/upgrade.html#migrate-from-soda-core) to **Soda Library** in minutes to start using this feature for free with a 45-day trial. {% endhint %} Use automated monitoring checks to instruct Soda to automatically check for row count anomalies and schema changes in a dataset. ```yaml automated monitoring: datasets: - include % - exclude test% ``` ✔️ Requires Soda Core Scientific (included in a Soda Runner)\ ✖️ Supported in Soda Core\ ✔️ Supported in Soda Library + Soda Cloud\ ✔️ Supported in Soda Cloud Agreements + Soda Runner\ ✖️ Available as no-code checks *** ## About automated monitoring checks When you add automated monitoring checks to a data source connected to your Soda Cloud account via a self-hosted runner, Soda prepares and executes two checks on all the datasets you indicated as `included` in the configuration. **Anomaly score check on row count**: This check counts the number of rows in a dataset during scan and registers anomalous counts relative to previous measurements for the row count metric. Refer to [Anomaly score checks](/soda-v3/sodacl-reference/anomaly-score) for details.\ Anomaly score checks require a minimum of four data points (four scans at stable intervals) to establish a baseline against which to gauge anomalies. If you do not see check results immediately, allow Soda Library to accumulate the necessary data points for relative comparison. **Schema evolution check**: This check monitors schema changes in datasets, including column addition, deletion, data type changes, and index changes. By default, this automated check results in a failure if a column is deleted, its type changes, or its index changes; it results in a warning if a column is added. Refer to [Schema checks](/soda-v3/sodacl-reference/schema) for details.\ Schema checks require a minimum of one data point to use as a baseline against which to gauge schema changes. If you do not see check results immediately, wait until after you have scanned the dataset twice. ## Add automated monitoring checks Add automated monitoring checks as part of the guided workflow to create a new data source only in deployment models that use a self-hosted Soda Runner, not a Soda-hosted Soda Runner. For a Soda-hosted runner, consider using the automated [anomaly dashboard](/soda-v3/collaborate/anomaly-dashboard) for observability into basic data quality in your datasets. If you are using a self-operated deployment model that leverages Soda Library, add the column profiling configuration outlined below to your checks YAML file. In Soda Cloud, navigate to **your avatar** > **Data Sources** > **New Data Source** to begin. In step **5. Check** of the guided workflow, you have the option of listing the datasets to which you wish to automatically add anomaly score and schema evolution checks. (Note that if you have signed up for early access to [anomaly dashboards](/soda-v3/collaborate/anomaly-dashboard) for datasets, this **Check** tab is unavailable as Soda performs all automated monitoring automatically in the dashboards.) The example check below uses a wildcard character (`%`) to specify that Soda Library executes automated monitoring checks against all datasets with names that begin with `prod`, and *not* to execute the checks against any dataset with a name that begins with `test`. ```yaml automated monitoring: datasets: - include prod% - exclude test% ``` You can also specify individual datasets to include or exclude, as in the following example. ```yaml automated monitoring: datasets: - include orders ``` ### Scan results in Soda Cloud To review the check results for automated monitoring checks in Soda Cloud, you can: * navigate to the **Checks** dashboard to see the check results * navigate to the **Datasets** dashboard to see the check results for an individual dataset ## Add quotes to all datasets ```yaml data_source soda_demo: type: sqlserver host: localhost username: ${SQL_USERNAME} password: ${SQL_PASSWORD} quote_tables: true ``` To add those necessary quotes to dataset names that Soda acts upon automatically – discovering, profiling, or sampling datasets, or creating automated monitoring checks – you can add a `quote_tables` configuration to your data source, as in the following example. If your dataset names include white spaces or use special characters, you must wrap those dataset names in quotes whenever you identify them to Soda, such as in a checks YAML file. ## Go further * Learn more about the [anomaly dashboard](/soda-v3/collaborate/anomaly-dashboard) for datasets. * Reference [tips and best practices for SodaCL](/soda-v3/soda-cl-overview/quick-start-sodacl#tips-and-best-practices-for-sodacl). * Use a [freshness check](/soda-v3/sodacl-reference/freshness) to gauge how recently your data was captured. * Use [reference checks](/soda-v3/sodacl-reference/reference) to compare the values of one column to another. > Need help? Join the [Soda community on Slack](https://community.soda.io/slack). --- [Next Page](/llms-full.txt/1)