Link Search Menu Expand Document

Soda Agent basic concepts preview

× This feature is available only for limited preview access.
Interested in getting early access? Let us know!

The Soda Agent is a tool that empowers Soda Cloud users to securely connect to new data sources.

The agent obviates the need to install Soda Core independently and set up configuration YAML files to connect to data sources. Instead, it enables Soda Cloud users to serve themselves when it comes to connecting to new data sources in an organization.

What follows is an extremely abridged introduction to a few basic elements involved in the deployment and setup of a Soda Agent.

Soda Core is an open-source, command-line tool that serves as the backbone of Soda technology. It is the software that performs the work of converting user-defined input into SQL queries that execute when you run scans for data quality in a data source. You can connect Soda Core to a Soda Cloud account where you and your team can use the web application to collaborate on data quality monitoring.

Both Soda Core and Soda Cloud make use of Soda Checks Language (SodaCL) to write checks for data quality. The checks are tests that Soda Core executes when it runs a scan of your data. Read more.

Kubernetes is a system for orchestrating containerized applications; a Kubernetes cluster is a set of resources that support an application environment. You need a Kubernetes cluster in which to deploy the containerized applications that make up the Soda Agent.

Amazon Elastic Kubernetes Service (EKS) is where you create your Kubernetes cluster; Fargate is a type of EKS node that operates as a serverless, pay-as-you-go compute engine, so that you can pay for the compute power your cluster uses. The Kubernetes cluster is also where you store Kubernetes secrets, such as login credentials, which Kubernetes creates independently on the pods that use them. (Pods are a basic workload unit in Kubernetes, usually an instance of one container.) Learn more about Kubernetes concepts.

Helm is a package manager for Kubernetes which bundles YAML files together for storage in a public or private repository. This bundle of YAML files is referred to as a Helm chart. The Soda Agent is a Helm chart. Anyone with access to the Helm chart’s repo can deploy the chart to make use of YAML files in it. Learn more about Helm concepts.

The Soda Agent Helm chart is stored on a public respository on ArtifactHub.io. Anyone can use Helm to find and deploy the Soda Agent Helm chart in their Kubernetes cluster. Deploying the agent also installs two other things in your Kubernetes cluster:

  • a Soda Agent Orchestrator from a Docker image, which creates Kubernetes Jobs and cron jobs to trigger scheduled scans of data
  • a Soda Agent Scan Launcher which wraps around Soda Core, the tool which performs the scan itself and pushes scan results to Soda Cloud

Go further


Was this documentation helpful?

What could we do to improve this page?


Last modified on 10-Aug-22