Manage sensitive values for a Soda Agent
Last modified on 31-May-23
When you deploy a Soda Agent to a Kubernetes cluster in your cloud service provider environment, you need to provide a few essential values that the agent needs to connect to your Soda Cloud account (API keys), and connect to your data sources (data source login credentials) so that Soda can run data quality scans on the data.
As these values are sensitive, you may wish to employ the following strategies to keep them secure.
Store Kubernetes secrets
Use a values YAML file to store API key values
Use a values file to store private key authentication values
Use environment variables to store data source connection credentials
Store Kubernetes secrets
Kubernetes uses the concept of Secrets that the Soda Agent Helm chart employs to store connection secrets that you specify as values during the Helm release of the Soda Agent.
Depending on your cloud provider, you can arrange to store these Secrets in a specialized storage such as Azure Key Vault or AWS Key Management Service (KMS).
Use a values YAML file to store API key values
When you deploy a Soda Agent from the command-line, you provide values for the API key id and API key secret which the agent uses to connect to your Soda Cloud account. You can provide these values during agent deployment in one of two ways:
- directly in the
helm install
command that deploys the agent and stores the values as Kubernetes secrets in your cluster; see deploy using CLI only
OR - in a values YAML file which you store locally but reference in the
helm install
command; see below
Values YAML file
soda:
apikey:
id: "***"
secret: "***"
agent:
name: "myuniqueagent"
helm install command
helm install soda-agent soda-agent/soda-agent \
--values values.yml \
--namespace soda-agent
Refer to the exhaustive cloud service provider-specific instructions for more detail on how to deploy an agent using a values YAML file:
- Deploy a Soda Agent in Amazon EKS
- Deploy a Soda Agent in Azure AKS
- Deploy a Soda agent in Google GKE
Use a values file to store private key authentication values
If you use private key with Snowflake or Big Query, you can provide the required private key values in a values.yml
file when you deploy or redeploy the agent.
Use environment variables to store data source connection credentials
When you, or someone in your organization, follows the guided steps to create a data source in Soda Cloud, one of the steps involves providing the connection details and credentials Soda needs to connect to the data source to run scans.
You can add those details directly in Soda Cloud, but because any user can then access these values, you may wish to store them securely in the values YAML file as environment variables.
- Create or edit your local values YAML file to include the values for the environment variables you input into the connection configuration.
soda: apikey: id: "***" secret: "***" agent: name: "myuniqueagent" env: POSTGRES_USER: "sodacore" POSTGRES_PASS: "sodacore"
- After adding the environment variables to the values YAML file, update the Soda Agent using the following command:
helm upgrade soda-agent soda-agent/soda-agent \ --values values.yml \ --namespace soda-agent
- In step 2 of the create a data source guided steps, add data source connection configuration which look something like the following example for a PostgreSQL data source. Note the environment variable values for username and password.
data_source local_postgres_test: type: postgres connection: host: 172.17.0.7 port: 5432 username: ${POSTGRES_USER} password: ${POSTGRES_PASS} database: postgres schema: new_york
- Follow the remaining guided steps to add a new data source in Soda Cloud. When you save the data source and test the connection, Soda Cloud uses the values you stored as environment variables in the values YAML file you supplied during redeployment.
Go further
- Learn more about Soda Agent basic concepts.
- Consider completing the Enable end-user data quality testing guide for more context around setting up a new data source and creating a new agreement.
- Need help? Join the Soda community on Slack.
Was this documentation helpful?
What could we do to improve this page?
- Suggest a docs change in GitHub.
- Share feedback in the Soda community on Slack.
Last modified on 31-May-23