Link Search Menu Expand Document

Create an agreement

Last modified on 29-Mar-23

An agreement is a contract between stakeholders that stipulates the expected and agreed-upon state of data quality in a data source.

agreement-1

Use SodaCL checks to define the state of “good quality” for data in this data source, then identify and get approval from stakeholders in your organization. Define whom Soda Cloud will notify when a check in this Agreement fails, then set a schedule to regularly execute the Soda Checks to uphold the tenets of the agreement.

agreement-2

Prerequisites
Create a new agreement
Run an ad hoc scan
Edit an agreement
Go further

Prerequisites

Create a new agreement

For a new agreement, you define several details including which data to check, what checks to execute during a scan, and whom to notify when bad data triggers an alert.

In Soda Cloud, navigate to the Agreements dashboard, then click New Agreement. Follow the guided steps to complete the new agreement. Use the sections below for insight into the values to enter in the fields and editing panels in the guided steps.

1. Select a Data Source

Field or Label Guidance
Agreement Label Provide a name for your agreement.
Data Source Select the data source that contains the datasets to which your agreement applies.
If you have no options to select in the dropdown, it is because you have not added a data source via a Soda Agent. You can only create agreements on datasets that are in a data source that has been onboarded into Soda Cloud via a Soda Agent.
See Create a data source.


2. Write Checks

Use SodaCL to define the checks that Soda Cloud executes on a regular schedule to uphold the tenets of this agreement. If any of these checks fail during a regularly-scheduled scan, Soda Cloud notifies the stakeholders you specify in the Notifications section.

  • For help writing your first checks, consider following the Quick start for SodaCL, including the Tips and best practices section.
  • Avoid applying the same customized check names in multiple agreements. Soda Cloud associates check results with agreements according to name so if you reuse custom names, Soda Cloud may become confused about which agreement to which to link check results.
  • Avoid using an anomaly score check to test the agreements workflow. The ML algorithm that anomaly score checks use requires a minimum of four, regular-frequency scans before it has collected enough historic measurements against which to gauge an anomaly. Consider using checks with numeric, missing, or validity metrics, instead.
  • Note that any checks you test in the context of this step in the agreements workflow do not appear as “real” check results in the Check Results dashboard.
  • Do not use variables to define dynamic values in your checks as you cannot provide scan-time values for those variables with scheduled scans. See Configuration details and limitations for details.


3. Identify Stakeholders

Add Stakeholders to this agreement who have an interest in maintaining or using the good-quality data in this data source. Consider adding a co-owner to your agreement for redundancy should you, as the agreement author, be absent.

Soda Cloud sends emails to request review and approval from all stakeholders, but does not prevent Soda Cloud from running scans and executing checks in the agreement if not all stakeholders have approved the agreement.


4. Set Notifications

By default, Soda Cloud includes an out-of-the-box email notification to all the agreement’s stakeholders when a check in your agreement fails. You can remove or adjust this notification, or use the search bar to add more. Access Scan output to learn more about pass, warn, and fail check results.

If you have integrated your Soda Cloud account with Slack or another third-party service provider via a webhook, use the search field to type a channel name to add the channel as a notification recipient. Alternatively, use the field to enter names of individual teammates with whom you collaborate in Soda Cloud.

Note that Soda Cloud does not send a notification when a scan fails because of an error, only when checks pass, warn, or fail. Refer to Scan output for details.


5. Set a Scan Schedule

After you have set up a new agreement, Soda Cloud sends approval requests to the stakeholders you identified in step 3. When stakeholders approve or reject your agreement, Soda Cloud sends you an email notification.

Regardless of the approval status of the agreement, however, Soda Cloud begins running scans of your data according to the scan schedule you set. Soda Cloud sends notifications after each scan according to the settings you defined in step 4.

(Optional) You can create a new scan definition if you wish to run a scan to execute the checks in this agreement more or less frequently, or a different time of day, relative to the default scan definition for the data source.

To review existing scan definitions, navigate to your avatar > Scans & Data > Scan Definitions tab.


Run an ad hoc scan

If you wish to run a scan immediately to see the check results for the checks you included in your agreement, you can run an ad hoc scan within the scan definition.

  1. In Soda Cloud, navigate to your avatar > Scans & Data, then access the Scan Definitions tab.
  2. In the list of scan definitions, click the one that you associated with the agreement you just defined.
  3. In the Scan Defintion page, click Run scan to immediately execute all agreements that use this scan definition.

Edit an agreement

  1. Navigate to the Agreements dashboard, then click the stacked dots at the right of the agreement you wish to edit and select Edit Agreement.
  2. Navigate the tabs to locate the details you wish to change.
  3. When you Save, Soda Cloud sends new approval request emails to all the agreement’s stakeholders. The next scheduled scan applies your changes, regardless of stakeholder approval status.

Go further

Was this documentation helpful?

What could we do to improve this page?


Last modified on 29-Mar-23