Link Search Menu Expand Document

Manage sensitive values for a Soda Agent

Last modified on 31-May-23

When you deploy a Soda Agent to a Kubernetes cluster in your cloud service provider environment, you need to provide a few essential values that the agent needs to connect to your Soda Cloud account (API keys), and connect to your data sources (data source login credentials) so that Soda can run data quality scans on the data.

As these values are sensitive, you may wish to employ the following strategies to keep them secure.

Store Kubernetes secrets
Use a values YAML file to store API key values
Use a values file to store private key authentication values
Use environment variables to store data source connection credentials

Store Kubernetes secrets

Kubernetes uses the concept of Secrets that the Soda Agent Helm chart employs to store connection secrets that you specify as values during the Helm release of the Soda Agent.

Depending on your cloud provider, you can arrange to store these Secrets in a specialized storage such as Azure Key Vault or AWS Key Management Service (KMS).

Use a values YAML file to store API key values

When you deploy a Soda Agent from the command-line, you provide values for the API key id and API key secret which the agent uses to connect to your Soda Cloud account. You can provide these values during agent deployment in one of two ways:

  • directly in the helm install command that deploys the agent and stores the values as Kubernetes secrets in your cluster; see deploy using CLI only
    OR
  • in a values YAML file which you store locally but reference in the helm install command; see below

Values YAML file 

soda:
  apikey:
    id: "***"
    secret: "***"
  agent:
    name: "myuniqueagent"

helm install command 

helm install soda-agent soda-agent/soda-agent \
  --values values.yml \
  --namespace soda-agent

Refer to the exhaustive cloud service provider-specific instructions for more detail on how to deploy an agent using a values YAML file:

Use a values file to store private key authentication values

If you use private key with Snowflake or Big Query, you can provide the required private key values in a values.yml file when you deploy or redeploy the agent.

Use environment variables to store data source connection credentials

When you, or someone in your organization, follows the guided steps to create a data source in Soda Cloud, one of the steps involves providing the connection details and credentials Soda needs to connect to the data source to run scans.

You can add those details directly in Soda Cloud, but because any user can then access these values, you may wish to store them securely in the values YAML file as environment variables.

  1. Create or edit your local values YAML file to include the values for the environment variables you input into the connection configuration. 
    soda:
 apikey:
   id: "***"
   secret: "***"
 agent:
   name: "myuniqueagent"
 env:
   POSTGRES_USER: "sodacore"
   POSTGRES_PASS: "sodacore"
  2. After adding the environment variables to the values YAML file, update the Soda Agent using the following command: 
    helm upgrade soda-agent soda-agent/soda-agent \
  --values values.yml \
  --namespace soda-agent
  3. In step 2 of the create a data source guided steps, add data source connection configuration which look something like the following example for a PostgreSQL data source. Note the environment variable values for username and password. 
    data_source local_postgres_test:
 type: postgres
 connection:
     host: 172.17.0.7
     port: 5432
     username: ${POSTGRES_USER}
     password: ${POSTGRES_PASS}
     database: postgres
 schema: new_york
  4. Follow the remaining guided steps to add a new data source in Soda Cloud. When you save the data source and test the connection, Soda Cloud uses the values you stored as environment variables in the values YAML file you supplied during redeployment.

Go further

Was this documentation helpful?

What could we do to improve this page?

Last modified on 31-May-23