Integrate webhooks with Soda Cloud

Last modified on 26-Jan-23

Configure a webhook in Soda Cloud to connect your account to a third-party service provider such as Jira, ServiceNow, PagerDuty, and more.

Use a webhook to:

send alert notifications for failed or warning check results to a third-party, such as ServiceNow
create and track data quality incidents with a third-party, such as Jira

webhook-example

Prerequisites
Configure a webhook
Webhooks for Soda Cloud alert notifications
Webhooks for Soda Cloud incident integrations
Example webhook with Jira for Soda Cloud incidents
Example webhook with ServiceNow for Soda Cloud incidents
Event payloads
Go further

Prerequisites

You have a Soda Cloud account with Admin permissions.
You have connected your Soda Cloud account to Soda Core and run at least one scan so as to produce one or more check results.
OR
You have added a datasource via a Soda Agent and are ready to create an agreement.

Configure a webhook

Confirm that the third-party can provide an incoming webhook URL that meets the following technical specifications:
- can return an HTTP status code between 200 and 400
- can reply to a request within 10 seconds (otherwise the request from Soda Cloud times out)
- provides an SSL-secured endpoint (https://) of TLS 1.2 or greater
In your Soda Cloud account, navigate to your avatar > Organization Settings, then select the Integrations tab.
Click the + at the upper right of the table of integrations to add a new integration.
In the Add Integration dialog box, select Webhook then follow the guided steps to configure the integration. Reference the following tables for guidance on the values to input in the guided steps.

Field or Label	Guidance
Name	Provide a unique name for your webhook in Soda Cloud. Required
URL	Input the incoming webhook URL or API endpoint provided by your service provider. See sections below for details. Required alert notifications incidents
HTTP Headers, Name	For example, `Authorization:`
HTTP Headers, Value	For example, `bearer [token]`
Enable to send notifications to this webhook when a check result triggers an alert.	Check to allow users to select this webhook as a destination for alert notifications when check results warn or fail.
Use this webhook as the default notification channel for all check result alerts.	Check to automatically configure check results alert notifications to this webhook by default. Users can deselect the webhook as the notification destination in an individual check, but it is the prepopulated destination by default.
Enable to use this webhook to track and resolve incidents in Soda Cloud.	Check to allow users to send incident information to a destination. For example, a user creating a new incident can choose to use this webhook to create a new issue in Jira.

Webhooks for Soda Cloud alert notifications

You can use a webhook to enable Soda Cloud to send alert notifications to a third-party provider, such as OpsGenie, to notify your team of warn and fail check results. With such an integration, Soda Cloud enables users to select the webhook as the destination for an individual check or checks that form a part of an agreement, or multiple checks.

To send notifications that apply to multiple checks, see Set notification rules.

Soda Cloud alert notifications make use of the following events:

validate
checkEvaluation

Access a third-party service provider’s documentation for details on how to set up an incoming webhook or API call, and obtain a URL to input into the Soda webhook configuration in step 4, above. The following links may be helpful starting points.

Webhooks for Soda Cloud incident integrations

You can use a webhook to integrate with a third-party service provider, such as Jira, to track incidents. With such an integration, Soda Cloud displays an external link for the integration in the Incident Details. See image below for an example.

Soda Cloud incident integrations make use of the following events:

validate
incidentCreated
incidentUpdated

When Soda Cloud sends an incidentCreated event to a webhook endpoint, the third-party service provider can respond with a link message. In such a case, Soda Cloud adds the link to the incident. The following is an example of the response payload.

// > POST [webhook URL]
{
  "event": "incidentCreated",
  // ...
}
// < 200 OK
{
  "link": {
    "url": "https://sodadata.atlassian.net/browse/SODA-69",
    "text": "[SODA-69] Notification & Incident Webhook"
  }
}

For incident integrations with third-party service providers that do not provide a link message in the response, you can use a callback URL. In such a case, when Soda Cloud sends an incidentCreated event to the third-party, you can configure the third-party response to include an incidentLinkCallbackUrl property. See an example with Jira below.

Configure the third-party response to make a POST request to this callback URL, including the text and url in the body of the JSON payload. Soda Cloud adds the callback URL as an integration link in the incident details.

The following is an example of the response payload with a callback URL.

// > POST [webhook URL]
{
  "event": "incidentCreated",
  "incident": { ... },
  "incidentLinkCallbackUrl": "https://cloud.soda.io/integrations/webhook/8224bbc2-2c80-4c6d-a*****/incident-link/510fad8c-dc43-419a-a122-712a***/uLYosxWNwVGHSdR-_noJjlNAA--WyQwe1ygqGBg*****Q"
}
// < 200 OK
{ }
Followed by a POST request to incidentLinkCallbackUrl:
// > POST https://cloud.soda.io/integrations/webhook/8224bbc2-2c80-4c6d-a002-16***4e/incident-link/510fad8c-dc43-419a-a122-7***97/uLYosxWNwVGHSdR-_noJjlNAA--WyQwe1ygqGBg****IrQ
{
  "url": "https://sodadata.atlassian.net/browse/SODA-69",
  "text": "[SODA-69] Notification & Incident Webhook"
}

Example webhook with Jira for Soda Cloud incidents

In Jira, you can set up an Automation Rule that enables you to define what you want an incoming webhook to do, then provides you with a URL that you use in the URL field in the Soda Cloud integration setup.

This example offers guidance on how to set up a webhook to generate an external link to which Soda Cloud displays in the Incident Details. When you change the status of a Soda Cloud incident, the webhook also updates the status of the Jira issue that corresponds with the incident.

webhook-incident

In Jira, start by navigating to Project settings > Automation, then click Create rule and, for the type of New trigger, select Incoming webhook. Reference the Jira documentation for details on how to create an Incoming webhook.

The images below offer details for an example of an Automation Rule in Jira to set up an incoming webhook that processes incident events from Soda Cloud.

See full Jira Automation example

When configuring the Incoming webhook, note the value for the Webhook URL that Jira provides for the automation; input this URL into the Soda Cloud URL field for the webhook (step 4, above).

webhook-incoming

webhook-config

When configuring Send web request, note the Web request URL that Jira sends back to Soda Cloud for the incident. This example uses incidentLinkCallbackUrl to send a POST request back to Soda Cloud to display a link to the Jira issue in the Incident Details page.

webhook-send-request

When configuring a Branch rule to update the status of an existing Jira issue, note the value in the JQL field that identifies which issue to update when the incident status in Soda Cloud changes.

webhook-branch-rule

Example webhook with ServiceNow for Soda Cloud incidents

In ServiceNow, you can create a Scripted REST API that enables you to prepare a resource to work as an incoming webhook. Use the ServiceNow Resource Path in the URL field in the Soda Cloud integration setup.

This example offers guidance on how to set up a Scripted REST API Resource to generate an external link which Soda Cloud displays in the Incident Details; see image below. When you change the status of a Soda Cloud incident, the webhook also updates the status of the SNOW issue that corresponds with the incident.

webhook-incident-snow

The following steps offer a brief overview of how to set up a ServiceNow Scripted REST API Resource to integrate with a Soda Cloud webhook. Reference the ServiceNow documentation for details:

In ServiceNow, start by navigating to the All menu, then use the filter to search for and select Scripted REST APIs.
Click New to create a new scripted REST API. Provide a name and API ID, then click Submit to save.
In the Scipted Rest APIs list, find and open your newly-created API, then, in the Resources tab, click New to create a new resource.
Provide a Name for your resource, then select POST as the HTTP method.
In the Script field, define a script that creates new tickets when a Soda Cloud incident is opened, and updates existing tickets when a Soda Cloud incident status is updated. Use the example below for reference. You may also need to define Security settings according to your organizations authentication rules.
Click Submit, then copy the value of the Resource path to use in the URL field in the Soda Cloud integration setup.

(function process(/*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) {


	var businessServiceId = '28***';
	var snowInstanceId = 'dev***';
	
	var requestBody = request.body;
	var requestData = requestBody.data;
	gs.info(requestData.event);
	if (requestData.event == 'incidentCreated'){
		gs.log("*** Incident Created ***");
		var grIncident = new GlideRecord('incident');
		grIncident.initialize();
		grIncident.short_description = requestData.incident.description;

		grIncident.description = requestData.incident.sodaCloudUrl;
		grIncident.correlation_id = requestData.incident.id;
		if(requestData.incident.severity == 'critical'){
			grIncident.impact = 1;
		}else if(requestData.incident.severity == 'major'){
			grIncident.impact = 2;
		}else if(requestData.incident.severity == 'minor'){
			grIncident.impact = 3;
		}
		
		grIncident.business_service = businessServiceId;
		grIncident.insert();
		var incidentNumber = grIncident.number;
		var sysid = grIncident.sys_id;
		var callBackURL = requestData.incidentLinkCallbackUrl;
		var req, rsp;
		
		req = new sn_ws.RESTMessageV2();


		req.setEndpoint(callBackURL.toString());
		req.setHttpMethod("post");
		var sodaUpdate = '{"url":"https://'+ snowInstanceId +'.service-now.com/incident.do?sys_id='+sysid + '", "text":"SNOW Incident '+incidentNumber+'"}';
		req.setRequestBody(sodaUpdate.toString());
		resp = req.execute();
		gs.log(resp.getBody());
		

	}else if(requestData.event == 'incidentUpdated'){
		gs.log("*** Incident Updated ***");
		var target = new GlideRecord('incident');
		target.addQuery('correlation_id', requestData.incident.id);
		target.query();
		target.next();

		if(requestData.incident.status == 'resolved'){
			//Change this according to how SNOW is used.
			target.state = 6;
			target.close_notes = requestData.incident.resolutionNotes;
		}else{
			//Change this according to how SNOW is used.
			target.state = 4;
		}
		target.update();
		
	}


})(request, response);

Event payloads

The following list of event payloads outlines the information that Soda Cloud sends when an action triggers a webhook.

validate

Soda Cloud sends this event payload to validate that the integration with the third-party service provider works. Soda Cloud sends this event during the guided workflow to set up an integration.

{
  "event": "validate",
  "sentAt": "2022-10-01T09:12:10.042323Z" 
}

checkEvaluation

Soda Cloud sends this event payload when it receives new check results. If the check is part of an agreement, the payload includes the agreement identifier.

{
  "event": "checkEvaluation",
  "checkResults": [
    {
      "id": "39d706c3-5a48-4f4b***",
      "sodaCloudUrl": "https://cloud.soda.io/checks/39d706c3-5a48-b",
      "definition": "checks for SODATEST_Customers_6f90f4ad:\ncount same as SODATEST_RAWCUSTOMERS_7275c02c in postgres2",
      "datasets": [
        {
          "id": "e8f1fe55-ae3c-44bd-",
          "sodaCloudUrl": "https://cloud.soda.io/datasets/e8f1fe55-ae3c",
          "name": "bnm_orders",
          "label": "bnm_orders",
          "tags": [],
          "owner": {
            "id": "31781df5-93cf-***",
            "email": "person@soda.io"
          },
          "datasource": {
            "id": "5a152025-26f6-",
            "name": "sodaspark",
            "label": "sodaspark"
          },
          "attributes": [
            {
              "id": "f0cd7b0f-4ac6-42a1-",
              "label": "Data Domain",
              "name": "data_domain",
              "value": "Product"
            },
            {
              "id": "32986775-3c7a-4a81-bfdb-5f9853746c39",
              "label": "Origin",
              "name": "origin",
              "value": "Pipeline"
            }
          ]
        }
      ],
      "column": "columnName",
      // pass, warn or fail
      "outcome": "pass",
      "dataTimestamp": "2022-01-04T09:49:48.060897Z",
      "diagnostics": {
        "value": 0.0,
      },
      // included when a check belongs to an agreement
      "agreement": {
        "id": "AGREEMENT-001-0000-0000-0",
        "sodaCloudUrl": "https://cloud.soda.io/agreements/AGREEMEN-T001-0000-0000-0",
        "label": "My new agreement pending",
        "approvalState": "pending",
        "evaluationResult": "warning"
      }
    }
  ]
}

incidentCreated

Soda Cloud sends this event payload when you create a new incident.

{
  "event": "incidentCreated",
  "incident": {
    "id": "e1f399a3-09ea-***",
    "sodaCloudUrl": "https://cloud.soda.io/incidents/e1f399a3-******-1992d2744ef6",
    "number": 196,
    "title": "Invalid customer ids",
    "description": "Invalid customer ids",
    "severity": "major",
    "status": "opened",
    "createdTimestamp": "2022-05-18T06:07:34Z",
    "lastUpdatedTimestamp": "2022-05-18T06:08:23Z",
    "resolutionNotes": "Stan is fixing the issue",
    "resolutionTimestamp": "2022-05-18T06:08:22.620196441Z",
    "links": [
      {
        "integrationType": "slack",
        "name": "soda-inc-196-2022-05-18-invalid-customer-ids",
        "url": "https://example.slack.com/channels/C03FU9GR7P7"
      }
    ],
    "lead": {
      "id": "31781df5-93cf-***",
      "email": "personn@soda.io"
    },
    "reporter": {
      "id": "31781df5-***",
      "email": "person@soda.io"
    },
    "checkResults": [
      // Contains the same payload as 
      // event checkEvaluation
    ]
  },
  "incidentLinkCallbackUrl": "https://cloud.soda.io/integrations/webhook/8224bbc2-******-16907465484e/incident-link/510fad8c-******-712a23f27197/uL******Kr6rvMcIrQ*"
}

incidentUpdated

Soda Cloud sends this event payload when an incident has been updated with, for example, a status change, when a new Lead has been assigned, or when check results have been added to the incident.

{
  "event": "incidentUpdated",
  "incident": {
    // Contains the same payload as 
    // event incidentCreated
  }
}

Go further

Learn more about creating agreements in Soda Cloud.
Set notification rules that apply to multiple checks in your account.
Learn more about creating, tracking, and resolving data quality Incidents in Soda Cloud.
Need help? Join the Soda community on Slack.

Was this documentation helpful?

What could we do to improve this page?

Suggest a docs change in GitHub.
Share feedback in the Soda community on Slack.

Last modified on 26-Jan-23