Dremio

Access configuration details to connect Soda to a Dremio data source.

Connection configuration reference

Make sure you have installed the latest version of soda-core.

Install the following package:

pip install "soda-dremio" -i "https://${SODA_CLOUD_API_KEY_ID}:${SODA_CLOUD_API_KEY_SECRET}@enterprise.pypi.soda.io/simple"

• ${SODA_CLOUD_API_KEY_ID} and ${SODA_CLOUD_API_KEY_SECRET} are environment variables that you should store with your API Key ID and API Key Secret, respectively.

Learn how to generate API keys.

• For US host users, use @enterprise.pypi.us.soda.io For EU host users, use @enterprise.pypi.soda.io

Data source YAML

ds_config.yml

type: dremio
name: my_dremio
connection:
  # Required
  host: ${env.DREMIO_HOST}
  port: ${env.DREMIO_PORT}
  username: ${env.DREMIO_USERNAME}
  password: ${env.DREMIO_PASSWORD}

  # Optional
  # schema: ${env.DREMIO_SCHEMA}                     # schema / space that contains your datasets
  # use_encryption: "true"                           # "false" by default; enable TLS by setting "true"
  # disable_certificate_verification: "true"         # "false" by default; set "true" to skip TLS cert checks

Connection test

Test the data source connection:

soda data-source test -ds ds_config.yml

Limitations & edge cases

• Dremio does not fully support schema evolution in the same way other data sources do it. Tables are still versioned, but Soda cannot create views that point to the latest data.

• Soda does not support Dremio schemas with . in their name.

• Diagnostics Warehouse: When enabling Diagnostics Warehouse advanced configuration > Allow failed row collection during Diagnostics Warehouse setup:

  • A schema configuration must be provided. Dremio requires a top level ID of a configured writable data source (e.g. s3_instance.folder1.soda_diagnostics, gcp_instance.soda_diagnostics). This field defaults to soda_diagnostics and will fail if left with that value.

    The examples above (s3_instance.folder1.soda_diagnostics, gcp_instance.soda_diagnostics) look like object storage paths because, generally, databases are not writable from Dremio, and object storage is.

---

You are not logged in to Soda and are viewing the default public documentation. Learn more about Documentation access & licensing.