Soda Agent test deployment
Interested in getting early access? Let us know!
If you are curious about how the Soda Agent works but are not yet ready to deploy to an Amazon Elastic Kubernetes Service (EKS) cluster, you can deploy an agent locally and connect it to your Soda Cloud account.
Access Deploy a Soda Agent for full deployment details.
Prerequisites
Create a Soda Cloud account and API keys
Deploy a Soda Agent locally
Create a practice data source
Deccomission the local cluster and Soda Agent
Prerequisites
- You have installed v1.22 or v1.23 of kubectl. This is the command-line tool you use to run commands against Kubernetes clusters. If you have installed Docker Desktop, kubectl is included out-of-the-box. Run
kubectl version --output=yaml
to check the version of an existing install. - You have installed Helm. This is the package manager for Kubernetes which you will use to deploy the Soda Agent Helm chart. Run
helm version
to check the version of an existing install. - You have installed Docker in your local environment.
Create a Soda Cloud account and API keys
The Soda Agent communicates with your Soda Cloud account using API public and private keys. Note that the keys a Soda Agent uses are different than the API keys Soda Cloud uses to connect to Soda Core.
- If you have not already done so, create a Soda Cloud account at cloud.soda.io.
- In your Soda Cloud account, navigate to your avatar > Scans & Data > the Agents tab, then click the New Soda Agent.
- The dialog box that appears offers abridged instructions to set up a new Soda Agent from the command-line; more thorough instructions exist in this documentation, below.
For now, copy and paste the values for both the soda.apikey.id and soda.apikey.secret to a temporary, secure place in your local environment. You will need these values in the next section when you deploy the agent in your Kubernetes cluster.
- You can keep the dialog box open in Soda Cloud, or close it.
Deploy a Soda Agent locally
- Install minikube to use to create a Kubernetes cluster running locally.
- Run the following command to create your local Kubernetes cluster. Be aware that this activity can take awhile. Be patient!
minikube start --driver=docker
... 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
- To connect to the newly created cluster and create a namespace, use the following command.
minikube kubectl -- create namespace soda-agent
- Run the following command to verify which cluster kubectl regcognizes as the current one.
minikube kubectl -- config get-contexts
The namespace associated with
CURRENT
must besoda-agent
. If it is not, use the following command to change contexts.minikube kubectl -- config set-context --current --namespace=soda-agent
CURRENT NAME CLUSTER AUTHINFO NAMESPACE * minikube minikube minikube soda-agent
- Add the Soda Agent Helm chart repository.
helm repo add soda-agent https://helm.soda.io/soda-agent/
- Use the following comand to install the Helm chart to deploy a Soda Agent in your custer.
- Replace the values of
soda.apikey.id
andsoda-apikey.secret
with the values you copy+pasted from the New Soda Agent dialog box in your Soda Cloud account - Replace the value of
soda.agent.name
with a custom name for you agent, if you wishhelm install soda-agent soda-agent/soda-agent \ --set soda.agent.target=minikube \ --set soda.agent.name=myuniqueagent \ --set soda.apikey.id=*** \ --set soda.apikey.secret=**** \ --namespace soda-agent
The command-line produces output like the following message:
NAME: soda-agent LAST DEPLOYED: Thu Jun 16 15:03:10 2022 NAMESPACE: soda-agent STATUS: deployed REVISION: 1
- Replace the values of
- (Optional) Validate the Soda Agent deployment by running the following command:
minikube kubectl -- describe pods
- In your Soda Cloud account, navigate to your avatar > Scans & Data > Agents tab. Refresh the page to verify that you see the agent you just created in the list of Agents.
Be aware that this may take several minutes to appear in your list of Soda Agents. Use thedescribe pods
command in step 7 to check the status of the deployment. WhenState: Running
andReady: True
, then you can refresh and see the agent in Soda Cloud.... Containers: soda-agent-orchestrator: Container ID: docker://081*33a7 Image: sodadata/agent-orchestrator:latest Image ID: docker-pullable://sodadata/agent-orchestrator@sha256:394e7c1**b5f Port: <none> Host Port: <none> State: Running Started: Thu, 16 Jun 2022 15:50:28 -0700 Ready: True ...
- If you wish, you can create a practice data source so you can try adding a data source in Soda Cloud using the Soda Agent you just deployed.
Create a practice data source
If you wish to try creating a new data source in Soda Cloud using the agent you created locally, you can use the following command to create a PostgreSQL warehouse containing data from the NYC Bus Breakdowns and Delay Dataset.
- From the command-line, create the data source as a pod on your local cluster.
cat <<EOF | kubectl apply -n soda-agent -f - apiVersion: v1 kind: Pod metadata: name: nybusbreakdowns labels: app: nycbusbreakdowns eks.amazonaws.com/fargate-profile: soda-agent-profile spec: containers: - image: sodadata/nybusbreakdowns imagePullPolicy: IfNotPresent name: nybusbreakdowns restartPolicy: Always EOF
- Once the pod is running, you can use the following configuration details when you add a data source in Soda Cloud, in step 2, Connect the Data Source.
data_source local_postgres_test: type: postgres connection: host: nybusbreakdowns port: 5432 username: sodacore password: sodacore database: sodacore schema: new_york
Deccomission the local cluster and Soda Agent
When you are satisfied with testing the deployment of a Soda Agent, you can decomission the local cluster you set up for deployment.
- Uninstall the Soda Agent in the cluster.
helm delete soda-agent -n soda-agent
- Delete the cluster.
minikube delete
💀 Removed all traces of the "minikube" cluster.
Go further
- Deploy a Soda Agent for real!
- Need help? Join the Soda community on Slack.
Was this documentation helpful?
What could we do to improve this page?
- Suggest a docs change in GitHub.
- Share feedback in the Soda community on Slack.
Last modified on 10-Aug-22