Link Search Menu Expand Document

Deploy a Soda Agent in Amazon EKS

Last modified on 26-Jan-23

The Soda Agent is a tool that empowers Soda Cloud users to securely access data sources to scan for data quality. Create an Amazon Elastic Kubernetes Service (EKS) cluster, then use Helm to deploy a Soda Agent in the cluster.

This setup enables Soda Cloud users to securely connect to data sources (Snowflake, Amazon Athena, etc.) from within the Soda Cloud web application. Any user in your Soda Cloud account can add a new data source via the agent, then write their own agreements to check for data quality in the new data source.

Deployment overview
Compatibility
Prerequisites
Create a Soda Cloud account and API keys
Create an EKS Fargate cluster
Deploy a Soda Agent
    Deploy using CLI only
    Deploy using a values YAML file
(Optional) Create a practice data source
About the helm install command
Decommission the Soda Agent and the EKS cluster
Troubleshoot deployment
Go further

Deployment overview

  1. (Optional) Familiarize yourself with basic Soda, Kubernetes, and Helm concepts.
  2. Install, or confirm the installation of, a few required command-line tools.
  3. Sign up for a Soda Cloud account and create new API keys.
  4. Use the command-line to create a Kubernetes cluster.
  5. Deploy the Soda Agent in the new cluster.
  6. Verify the existence of your new Soda Agent in your Soda Cloud account.

Compatibility

Soda supports Kubernetes cluster version 1.21 or greater.

You can deploy a Soda Agent to connect with the following data sources:

Amazon Athena
Amazon Redshift
Azure Synapse (Experimental)
ClickHouse (Experimental)
Denodo (Experimental)
Dremio
DuckDB (Experimental)
GCP Big Query
IBM DB2
MS SQL Server †
MySQL
OracleDB
PostgreSQL
Snowflake
Trino
Vertica (Experimental) - Coming soon

† MS SQL Server with Windows Authentication does not work with Soda Agent out-of-the-box.

Prerequisites

  • (Optional) You have familiarized yourself with basic Soda, Kubernetes, and Helm concepts.
  • You have an AWS account and the necessary permissions to enable you to create an EKS cluster in your region.
  • You have installed aws-cli. This is the command-line tool you need to access your AWS account from the command-line. Run aws --version to check the version of an existing install.
  • You have installed eksctl. This is the command-line tool for Amazon EKS that you use to create and manage Kubernetes clusters on EKS. Run eksctl version to check the version of an existing install.
  • You have installed v1.22 or v1.23 of kubectl. This is the command-line tool you use to run commands against Kubernetes clusters. If you have installed Docker Desktop, kubectl is included out-of-the-box. Run kubectl version --output=yaml to check the version of an existing install.
  • You have installed Helm. This is the package manager for Kubernetes which you will use to deploy the Soda Agent Helm chart. Run helm version to check the version of an existing install.

Create a Soda Cloud account and API keys

The Soda Agent communicates with your Soda Cloud account using API public and private keys. Note that the keys a Soda Agent uses are different from the API keys Soda Core uses to connect to Soda Cloud.

  1. If you have not already done so, create a Soda Cloud account at cloud.soda.io.
  2. In your Soda Cloud account, navigate to your avatar > Scans & Data > the Agents tab, then click the New Soda Agent.
  3. The dialog box that appears offers abridged instructions to set up a new Soda Agent from the command-line; more thorough instructions exist in this documentation, below.

    For now, copy and paste the values for both the API Key ID and API Key Secret to a temporary, secure place in your local environment. You will need these values in the next section when you deploy the agent in your Kubernetes cluster.
    deploy-agent
  4. You can keep the dialog box open in Soda Cloud, or close it.

Create an EKS Fargate cluster

The following offers instructions to create a Fargate (serverless) cluster to deploy a Soda Agent, but you can create and use a regular EKS cluster if you wish.

  1. If you are deploying to an existing Virtual Private Cloud (VPC), consider supplying public or private subnets with your deployment. Consult the eksctl documentation to Use existing VPC.
  2. From the command-line, execute the following command to create a new EKS Fargate cluster in your AWS account.
    Replace the value of --region with one that is appropriate for your location.
    eksctl create cluster --name soda-agent --region eu-central-1 --fargate
    
    • If you are not sure which region to use, execute the following command to find your region:
      aws configure get region
      
    • The activity to create a cluster may take awhile, printing messages in the console that read waiting for CloudFormation stack "eksctl-soda-agent-cluster". Be patient! Access AWS CloudFormation troubleshooting documentation for help.
  3. To connect to the newly-created cluster and create a namespace, use the following command.
    kubectl create namespace soda-agent
    
  4. Create a namespace and a Fargate profile for EKS Fargate serverless deployment.
    When you deploy a Soda Agent on EKS Fargate, AWS matches the Fargate Profile using annotation labels in the Soda Agent Helm chart. Without the profile, the Helm chart cannot successfully deploy.
    Refer to Troubleshoot deployment below if you encounter errors.
    eksctl create fargateprofile --cluster soda-agent --name soda-agent-profile --region us-west-1 --namespace soda-agent
    
  5. Run the following command to change the context to associate the current namespace to soda-agent.
    kubectl config set-context --current --namespace=soda-agent
    
  6. Run the following command to verify that the cluster kubectl recognizes soda-agent as the current namespace.
    kubectl config get-contexts
    
    CURRENT   NAME               CLUSTER          AUTHINFO          NAMESPACE
    *         testing@soda...    soda-agent.eu..  testing@soda...   soda-agent
    

Deploy a Soda Agent

The following table outlines the two ways you can install the Helm chart to deploy a Soda Agent in your cluster.

Method Description When to use
CLI only Install the Helm chart via CLI by providing values directly in the install command. Use this as a straight-forward way of deploying an agent on a cluster.
Use a values YAML file Install the Helm chart via CLI by providing values in a values YAML file. Use this as a way of deploying an agent on a cluster while keeping sensitive values secure.
- provide sensitive API key values in this local file
- store data source login credentials as environment variables in this local file; Soda needs access to the credentials to be able to connect to your data source to run scans of your data. See: Manage sensitive values.

Deploy using CLI only

  1. Use Helm to add the Soda Agent Helm chart repository.
    helm repo add soda-agent https://helm.soda.io/soda-agent/
    
  2. Use the following command to install the Helm chart which deploys a Soda Agent in your custer.
    • Replace the values of soda.apikey.id and soda-apikey.secret with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud. The cluster stores these key values as Kubernetes secrets.
    • Replace the value of soda.agent.name with a custom name for your agent, if you wish.
    • Optionally, add the soda.core settings to configure idle workers in the cluster. Launch an idle worker so at scan time, the agent can hand over instructions to an already running idle Scan Launcher to avoid the start-from-scratch setup time for a pod. This helps your test scans from Soda Cloud run faster. You can have multiple idle scan launchers waiting for instructions.
    • Read more about the helm install command.
      helm install soda-agent soda-agent/soda-agent \
       --set soda.agent.target=aws-eks \
       --set soda.agent.name=myuniqueagent \
       --set soda.apikey.id=*** \
       --set soda.apikey.secret=**** \
       --set soda.core.idle=true \
       --set soda.core.replicas=1 \
       --namespace soda-agent
      

      The command-line produces output like the following message:

      NAME: soda-agent
      LAST DEPLOYED: Thu Jun 16 10:12:47 2022
      NAMESPACE: soda-agent
      STATUS: deployed
      REVISION: 1
      
  3. (Optional) Validate the Soda Agent deployment by running the following command:
    kubectl describe pods
    
  4. In your Soda Cloud account, navigate to your avatar > Scans & Data > Agents tab. Refresh the page to verify that you see the agent you just created in the list of Agents.

    Be aware that this may take several minutes to appear in your list of Soda Agents. Use the describe pods command in step 3 to check the status of the deployment. When State: Running and Ready: True, then you can refresh and see the agent in Soda Cloud.
    ...
    Containers:
      soda-agent-orchestrator:
         Container ID:   docker://081*33a7
         Image:          sodadata/agent-orchestrator:latest
         Image ID:       docker-pullable://sodadata/agent-orchestrator@sha256:394e7c1**b5f
         Port:           <none>
         Host Port:      <none>
         State:          Running
           Started:      Thu, 16 Jun 2022 15:50:28 -0700
         Ready:          True
    ...
    

    agent-deployed

  5. Next: Add a data source in Soda Cloud using the Soda Agent you just deployed. If you wish, you can create a practice data source so you can try adding a data source in Soda Cloud using the Soda Agent you just deployed.

Deploy using a values YAML file

  1. Using a code editor, create a new YAML file called values.yml.
  2. To that file, copy+paste the content below, replacing the following values:
    • id and secret with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud account.
    • Replace the value of name with a custom name for your agent, if you wish.
    • Optionally, add the soda.core settings to configure idle workers in the cluster. Launch an idle worker so at scan time, the agent can hand over instructions to an already running idle Scan Launcher to avoid the start-from-scratch setup time for a pod. This helps your test scans from Soda Cloud run faster. You can have multiple idle scan launchers waiting for instructions.
      soda:
         apikey:
            id: "***"
            secret: "***"
         agent:
            name: "myuniqueagent"
         core:
            idle: true
            replicas: 1
      
  3. Save the file. Then, in the same directory in which the values.yml file exists, use the following command to install the Soda Agent helm chart.
    helm install soda-agent soda-agent/soda-agent \
      --values values.yml \
      --namespace soda-agent
    
  4. (Optional) Validate the Soda Agent deployment by running the following command:
    kubectl -- describe pods
    
  5. In your Soda Cloud account, navigate to your avatar > Scans & Data > Agents tab. Refresh the page to verify that you see the agent you just created in the list of Agents.

    Be aware that this may take several minutes to appear in your list of Soda Agents. Use the describe pods command in step four to check the status of the deployment. When State: Running and Ready: True, then you can refresh and see the agent in Soda Cloud.
    ...
    Containers:
      soda-agent-orchestrator:
         Container ID:   docker://081*33a7
         Image:          sodadata/agent-orchestrator:latest
         Image ID:       docker-pullable://sodadata/agent-orchestrator@sha256:394e7c1**b5f
         Port:           <none>
         Host Port:      <none>
         State:          Running
           Started:      Thu, 16 Jun 2022 15:50:28 -0700
         Ready:          True
         ...
    

    agent-deployed

  6. Next: Add a data source in Soda Cloud using the Soda Agent you just deployed. If you wish, you can create a practice data source so you can try adding a data source in Soda Cloud using the Soda Agent you just deployed.

(Optional) Create a practice data source

If you wish to try creating a new data source in Soda Cloud using the agent you deployed, you can use the following command to create a PostgreSQL warehouse containing example data from the NYC Bus Breakdowns and Delay Dataset.

From the command-line, copy+paste and run the following to create the data source as a pod on your new cluster.

cat <<EOF | kubectl apply -n soda-agent -f -
---
apiVersion: v1
kind: Pod
metadata:
  name: nybusbreakdowns
  labels:
    app: nybusbreakdowns
spec:
  containers:
  - image: sodadata/nybusbreakdowns
    imagePullPolicy: IfNotPresent
    name: nybusbreakdowns
    ports:
    - name: tcp-postgresql
      containerPort: 5432
  restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nybusbreakdowns
  name: nybusbreakdowns
spec:
  ports:
  - name: tcp-postgresql
    port: 5432
    protocol: TCP
    targetPort: tcp-postgresql
  selector:
    app: nybusbreakdowns
  type: ClusterIP
EOF

Output:

pod/nybusbreakdowns created
service/nybusbreakdowns created


Once the pod of practice data is running, you can use the following configuration details when you add a data source in Soda Cloud, in step 2, Connect the Data Source.

data_source your_datasource_name:
  type: postgres
  connection:
    host: nybusbreakdowns
    port: 5432
    username: sodacore
    password: sodacore
    database: sodacore
    schema: new_york

About the helm install command

helm install soda-agent soda-agent/soda-agent \
  --set soda.agent.target=azure-aks-virtualnodes \
  --set soda.agent.name=myuniqueagent \
  --set soda.apikey.id=*** \
  --set soda.apikey.secret=**** \
  --namespace soda-agent
Command part Description
helm install the action helm is to take
soda-agent (the first one) a release named soda-agent on your cluster
soda-agent (the second one) the name of the helm repo you installed
soda-agent (the third one) the name of the helm chart that is the Soda Agent

The --set options either override or set some of the values defined in and used by the Helm chart. You can override these values with the --set files as this command does, or you can specify the override values using a values.yml file.

Parameter key Parameter value, description
--set soda.agent.target (Optional) The cluster the command target. Use when deploying to aws-eks or azure-aks-virtualnodes.
--set soda.agent.name A unique name for your Soda Agent. Choose any name you wish, as long as it is unique in your Soda Cloud account.
--set soda.apikey.id With the apikey.secret, this connects the Soda Agent to your Soda Cloud account. Use the value you copied from the dialog box in Soda Cloud when adding a new agent. You can use a values.yml file to pass this value to the cluster instead of exposing it here.
--set soda.apikey.secret With the apikey.id, this connects the Soda Agent to your Soda Cloud account. Use the value you copied from the dialog box in Soda Cloud when adding a new agent. You can use a values.yml file to pass this value to the cluster instead of exposing it here.
--set soda.core.idle=true (Optional) Launch an idle worker so at scan time, the agent can hand over instructions to an already running idle Scan Launcher to avoid the start-from-scratch setup time for a pod. You can have multiple idle scan launchers waiting for instructions.
--set soda.core.replicas=1 (Optional) Replicate an idle worker to have more workers ready to handle instructions without setting up a new pod.
--namespace soda-agent Use the namespace value to identify the namespace in which to deploy the agent.


Decommission the Soda Agent and the EKS cluster

  1. Uninstall the Soda Agent in the cluster.
    helm delete soda-agent -n soda-agent
    
  2. Remove the Fargate profile.
    eksctl delete fargateprofile --cluster soda-agent --name soda-agent-profile
    
  3. Wait for the Fargate profile deletion to complete, then delete the EKS cluster itself.
    eksctl delete cluster --name soda-agent
    
  4. (Optional) Access your CloudFormation console, then click Stacks to view the status of your decommissioned cluster.
    If you do not see your Stack, use the region drop-down menu at upper-right to select the region in which you created the cluster.

Troubleshoot deployment

Refer to Helpful kubectl commands for instructions on accessing logs, etc.


Problem: Scans launched from Soda Cloud take an excessive amount of time to run.

Solution: Consider adjusting the number of replicas for idle workers with kubectl. Launch extra idle workers so at scan time, the agent can hand over instructions to an already running idle Scan Launcher to avoid the start-from-scratch setup time for a pod.

  1. Ensure that the agent was deployed with the soda.core configurations for idle: true and replicas: 1 or more.
  2. Run the following command to increase the number of active replicas to 2.
    kubectl scale deployment/soda-agent-scanlauncher \
      --replicas 2 -n soda-agent
    


Problem: UnauthorizedOperation: You are not authorized to perform this operation.

Solution: This error indicates that your user profile is not authorized to create the cluster. Contact your AWS Administrator to request the appropriate permissions.


Problem: ResourceNotFoundException: No cluster found for name: soda-agent.

Solution: If you get an error like this when you attempt to create a Fargate profile, it may be a question of region.

  1. Access your AWS CloudFormation console, then click Stacks to find the eksctl-soda-agent-cluster that you created. If you do not see the stack, adjust the region of your CloudFormation console (top nav bar, next to your username).
  2. Try running the command: aws eks list-clusters. It likely returns the following.
    {
     "clusters": []
    }
    
  3. Try running the command: aws cloudformation list-stacks --region us-west-1 replacing the value of --region with the value you used for region when you created the EKS Fargate cluster.
  4. If that command returns information about the cluster you just created, add the --region option to the command to create a Fargate profile.
    eksctl create fargateprofile --cluster soda-agent --name soda-agent-profile --region us-west-1 --namespace soda-agent
    

Go further


Was this documentation helpful?

What could we do to improve this page?


Last modified on 26-Jan-23