Soda Agent test deployment

× This feature is available only for limited preview access.
Interested in getting early access? Let us know!

If you are curious about how the Soda Agent works but are not yet ready to deploy to an Amazon Elastic Kubernetes Service (EKS) cluster, you can deploy an agent locally and connect it to your Soda Cloud account.

Access Deploy a Soda Agent for full deployment details.

Prerequisites
Create a Soda Cloud account and API keys
Deploy a Soda Agent locally
Create a practice data source
Deccomission the local cluster and Soda Agent

Prerequisites

You have installed v1.22 or v1.23 of kubectl. This is the command-line tool you use to run commands against Kubernetes clusters. If you have installed Docker Desktop, kubectl is included out-of-the-box. Run kubectl version --output=yaml to check the version of an existing install.
You have installed Helm. This is the package manager for Kubernetes which you will use to deploy the Soda Agent Helm chart. Run helm version to check the version of an existing install.
You have installed Docker in your local environment.

Create a Soda Cloud account and API keys

The Soda Agent communicates with your Soda Cloud account using API public and private keys. Note that the keys a Soda Agent uses are different than the API keys Soda Cloud uses to connect to Soda Core.

If you have not already done so, create a Soda Cloud account at cloud.soda.io.
In your Soda Cloud account, navigate to your avatar > Scans & Data > the Agents tab, then click the New Soda Agent.
The dialog box that appears offers abridged instructions to set up a new Soda Agent from the command-line; more thorough instructions exist in this documentation, below.
For now, copy and paste the values for both the soda.apikey.id and soda.apikey.secret to a temporary, secure place in your local environment. You will need these values in the next section when you deploy the agent in your Kubernetes cluster.
You can keep the dialog box open in Soda Cloud, or close it.

Deploy a Soda Agent locally

Install minikube to use to create a Kubernetes cluster running locally.

Run the following command to create your local Kubernetes cluster. Be aware that this activity can take awhile. Be patient!

minikube start --driver=docker

...
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

To connect to the newly created cluster and create a namespace, use the following command.
```
minikube kubectl -- create namespace soda-agent
```

Run the following command to verify which cluster kubectl regcognizes as the current one.

minikube kubectl -- config get-contexts

The namespace associated with CURRENT must be soda-agent. If it is not, use the following command to change contexts.

minikube kubectl -- config set-context --current --namespace=soda-agent

CURRENT   NAME               CLUSTER          AUTHINFO          NAMESPACE
*         minikube           minikube         minikube          soda-agent

Add the Soda Agent Helm chart repository.

helm repo add soda-agent https://helm.soda.io/soda-agent/

Use the following comand to install the Helm chart to deploy a Soda Agent in your custer.
- Replace the values of soda.apikey.id and soda-apikey.secret with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud account
- Replace the value of soda.agent.name with a custom name for you agent, if you wish
```
helm install soda-agent soda-agent/soda-agent \
--set soda.agent.target=minikube \
--set soda.agent.name=myuniqueagent \
--set soda.apikey.id=*** \
--set soda.apikey.secret=**** \
--namespace soda-agent
```
  The command-line produces output like the following message:
```
NAME: soda-agent
LAST DEPLOYED: Thu Jun 16 15:03:10 2022
NAMESPACE: soda-agent
STATUS: deployed
REVISION: 1
```
(Optional) Validate the Soda Agent deployment by running the following command:
```
minikube kubectl -- describe pods
```
In your Soda Cloud account, navigate to your avatar > Scans & Data > Agents tab. Refresh the page to verify that you see the agent you just created in the list of Agents.
Be aware that this may take several minutes to appear in your list of Soda Agents. Use the describe pods command in step 7 to check the status of the deployment. When State: Running and Ready: True, then you can refresh and see the agent in Soda Cloud.
```
...
Containers:
  soda-agent-orchestrator:
 Container ID:   docker://081*33a7
 Image:          sodadata/agent-orchestrator:latest
 Image ID:       docker-pullable://sodadata/agent-orchestrator@sha256:394e7c1**b5f
 Port:           <none>
 Host Port:      <none>
 State:          Running
   Started:      Thu, 16 Jun 2022 15:50:28 -0700
 Ready:          True
...
```
If you wish, you can create a practice data source so you can try adding a data source in Soda Cloud using the Soda Agent you just deployed.

Create a practice data source

If you wish to try creating a new data source in Soda Cloud using the agent you created locally, you can use the following command to create a PostgreSQL warehouse containing data from the NYC Bus Breakdowns and Delay Dataset.

From the command-line, create the data source as a pod on your local cluster.

cat <<EOF | kubectl apply -n soda-agent -f -
apiVersion: v1
kind: Pod
metadata:
  name: nybusbreakdowns
  labels:
    app: nycbusbreakdowns
    eks.amazonaws.com/fargate-profile: soda-agent-profile
spec:
  containers:
  - image: sodadata/nybusbreakdowns
    imagePullPolicy: IfNotPresent
    name: nybusbreakdowns
  restartPolicy: Always
EOF

Once the pod is running, you can use the following configuration details when you add a data source in Soda Cloud, in step 2, Connect the Data Source.

data_source local_postgres_test:
  type: postgres
  connection:
     host: nybusbreakdowns
     port: 5432
     username: sodacore
     password: sodacore
     database: sodacore
     schema: new_york

Deccomission the local cluster and Soda Agent

When you are satisfied with testing the deployment of a Soda Agent, you can decomission the local cluster you set up for deployment.

Uninstall the Soda Agent in the cluster.
```
helm delete soda-agent -n soda-agent
```

Delete the cluster.

minikube delete

💀  Removed all traces of the "minikube" cluster.

Go further

Deploy a Soda Agent for real!
Need help? Join the Soda community on Slack.

Was this documentation helpful?

What could we do to improve this page?

Suggest a docs change in GitHub.
Share feedback in the Soda community on Slack.

Last modified on 10-Aug-22