DuckDB advanced usage

Soda supports DuckDB as a flexible, lightweight SQL engine that can be used with files and in-memory data frames such as Pandas and Polars.

Install the following package:

pip install -i https://pypi.cloud.soda.io/simple --pre -U "soda-duckdb>4"

From Pandas DataFrame

import pandas as pd
import duckdb
from soda_core.contracts import verify_contract_locally
from soda_duckdb import DuckDBDataSource

df = pd.read_parquet("adventureworks.parquet")
conn = duckdb.connect(database=":memory:")
cursor = conn.cursor()
cursor.register(view_name="adventureworks", python_object=df)

result = verify_contract_locally(
    data_sources=[DuckDBDataSource.from_existing_cursor(cursor, name="duckdb")],
    contract_file_path="adventureworks.yml"
)

---

From Polars DataFrame

import polars as pl
import duckdb
from soda_core.contracts import verify_contract_locally
from soda_duckdb import DuckDBDataSource

df = pl.read_parquet("adventureworks.parquet")
conn = duckdb.connect(database=":memory:")
cursor = conn.cursor()
cursor.register(view_name="adventureworks", python_object=df)

result = verify_contract_locally(
    data_sources=[DuckDBDataSource.from_existing_cursor(cursor, name="duckdb")],
    contract_file_path="adventureworks.yml"
)

---

In-Memory with DuckDB SQL

import duckdb
from soda_core.contracts import verify_contract_locally
from soda_duckdb import DuckDBDataSource

db_connection = duckdb.connect(database=":memory:")
cursor = db_connection.cursor()

cursor.execute("CREATE SCHEMA analytics")
cursor.execute("CREATE TABLE analytics.adventureworks AS SELECT * FROM read_parquet('adventureworks.parquet')")

result = verify_contract_locally(
    data_sources=[DuckDBDataSource.from_existing_cursor(cursor, name="duckdb")],
    contract_file_path="adventureworks.yml"
)

---

Data from Parquet File

You can point directly to a .parquet file as a DuckDB "database":

ds_config.yml

type: duckdb
name: duckdb
connection:
  database: "adventureworks.parquet"

Then you can verify a contract on this database using the CLI:

soda contract verify -ds ds_config.yml -c adventureworks.yml

Or Python API:

from soda_core.contracts import verify_contract_locally

result = verify_contract_locally(
    data_source_file_path="ds_config.yml",
    contract_file_path="adventureworks.yml"
)

---

You are not logged in to Soda and are viewing the default public documentation. Learn more about Documentation access & licensing.