> For the complete documentation index, see [llms.txt](https://docs.soda.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soda.io/soda-documentation/data-observability/metadata-data-sources.md).

# Metadata data sources

## Supported data sources

Below, find a snapshot of all the capabilities supported for each data source, including **metadata** and **metadata history**:

<table><thead><tr><th width="185.79998779296875">Data source</th><th width="171.20001220703125">Onboarding</th><th>Metadata</th><th>Metadata history</th><th>Querying data</th></tr></thead><tbody><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/amazon-athena.md">Athena</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/bigquery.md">BigQuery</a></td><td>✅</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/databricks.md">Databricks SQL</a></td><td>✅</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/dremio.md">Dremio</a></td><td>✅</td><td>Upcoming</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/duckdb.md">DuckDB</a> (<a href="/soda-documentation/reference/data-source-reference-for-soda-core/pandas.md">Pandas</a>, <a href="/soda-documentation/reference/data-source-reference-for-soda-core/polars.md">Polars</a>, etc.)</td><td>Partial/Python only</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/microsoft-fabric.md">Fabric</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/mariadb.md">MariaDB</a></td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/mysql.md">MySQL</a></td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/oracle.md">Oracle</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/postgresql.md">PostgreSQL</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/amazon-redshift.md">Redshift</a></td><td>✅</td><td>✅</td><td>✅</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/sap-hana.md">SAP HANA</a></td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/spark-dataframe.md">Spark Dataframe</a></td><td>Partial/Python only</td><td>—</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/snowflake.md">Snowflake</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/sql-server.md">SQL Server</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/synapse.md">Synapse</a></td><td>✅</td><td>✅</td><td>—</td><td>✅</td></tr><tr><td><a data-mention href="/soda-documentation/reference/data-source-reference-for-soda-core/trino.md">Trino</a></td><td>✅</td><td>—</td><td>—</td><td>✅</td></tr></tbody></table>

## Limitations & edge cases

Some data sources present **limitations** when it comes to **metadata** collection or retrieval. Keep in mind these caveats when choosing which data source(s) to connect to Soda.

### Oracle

* **Historical backfilling:** not possible.
* **Row count:** metadata row counts are calculated via `count(*)`. Soda does not use metadata for this metric in Oracle. It requires an additional package and/or is unreliable based on the schedule of that package.
* **Last modification time:** Soda uses metadata

Note that past data is only available for a limited amount of time, which varies depending on the system. The minimum goes back 120 h.

{% hint style="warning" %}
**Non-UTC timestamps are not recommended** when connecting Soda to Oracle data sources. Soda uses timezone data when available, but assumes UTC when the timezone is not provided by the data source.

Some databases convert timestamps to UTC, but **Oracle does not do any implicit conversions** and stores timestamps and timezone information as the user inputs them. Because of Oracle Python client limitations, all timezone information is stripped when Soda retrieves it, which means that Soda will read all timestamps as if they were UTC regardless of the original input.
{% endhint %}

***

### Postgres

Metadata is supported, but it requires some additional setup on Postgres's side.

* **Historical backfilling:** not possible.
* **Row count:** enabled out-of-the-box.
* **Last modification time:** `track_commit_timestamp` must be enabled: <https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-TRACK-COMMIT-TIMESTAMP>
  * If `track_commit_timestamp` is not enabled, Soda will return a warning.

***

### BigQuery

Metadata metrics are available and supported in BigQuery.

* **Historical backfilling:** possible.
* **Partition column:** can be suggested based on metadata available in BigQuery.
  * Soda will prioritize user-suggested columns.
  * If there are no user-suggested columns, Soda will try a metadata approach to find the partition column automatically.
  * If there are no columns found in the metadata of BigQuery, Soda will fall back on its own heuristic.

{% hint style="info" %}
**Partition column availability in BigQuery:**

If the user has configured a partitioning column on BigQuery's side, Soda will use it (given that it is a date/timestamp column).

Otherwise, Soda will fall back on a standard sampling method to detect the partition column.
{% endhint %}

***

### Redshift

* **Historical backfilling** is supported on Redshift and it is limited to 7 days for the metadata.
* **Modification time** does not include schema changes. Only:
  * inserts
  * updates
  * deletes

***

### Synapse

Synapse does not provide metadata history tables.

* **Historical backfilling:** not possible.
* **Last modification time:** not possible.
* **Row count:** current row counts are calculated via `count(*)`.\
  Soda does not use metadata for this metric in Synapse.
* **Quartile metrics** (Q1, median, Q3): not possible.\
  Synapse does not support quartile metrics.

***

### MariaDB

MariaDB does not expose usable warehouse metadata. `information_schema.TABLES.TABLE_ROWS` is an estimate rather than a count, so Soda does not read it.

* **Historical backfilling:** not possible for the metadata.\
  MariaDB exposes no change log for Soda to reconstruct metadata history from. A backfill still runs, on data-based metrics only.
* **Last modification time:** not possible.
* **Row count:** current row counts are calculated via `count(*)`.\
  Soda does not use metadata for this metric in MariaDB.
* **Quartile metrics** (Q1, median, Q3): not possible.\
  MariaDB has neither a percentile function nor `WITHIN GROUP`.

### MySQL

MySQL does not expose usable warehouse metadata. `information_schema.TABLES.TABLE_ROWS` is an estimate for InnoDB tables rather than a count, so Soda does not read it.

* **Historical backfilling:** not possible for the metadata.\
  MySQL exposes no change log for Soda to reconstruct metadata history from. A backfill still runs, on data-based metrics only.
* **Last modification time:** not possible.
* **Row count:** current row counts are calculated via `count(*)`.\
  Soda does not use metadata for this metric in MySQL.
* **Quartile metrics** (Q1, median, Q3): not possible.\
  MySQL has neither a percentile function nor `WITHIN GROUP`.

***

### SAP HANA

SAP HANA exposes no metadata provider to Soda, so Metric Monitoring runs on data-based metrics only. Monitors that read the warehouse's own catalog are unavailable.

* **Historical backfilling:** not possible for the metadata.\
  There is no change log for Soda to reconstruct metadata history from. A backfill still runs, on data-based metrics only.
* **Last modification time:** not possible.
* **Row count:** current row counts are calculated via `count(*)`.\
  Soda does not use metadata for this metric in SAP HANA.
* **Quartile metrics** (Q1, median, Q3): supported.\
  Soda splits the query when a percentile and a `DISTINCT` aggregate over the same column would otherwise land in one `SELECT`, which SAP HANA rejects. The same approach is used for Redshift.

<br>

***

{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Licensing & documentation access](/soda-documentation/reference/documentation-access-and-licensing.md).
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.soda.io/soda-documentation/data-observability/metadata-data-sources.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.