Migrate from Soda SQL to Soda Core

Last modified on 31-May-23

Soda SQL, the original command-line tool that Soda created to test for data quality, has been deprecated and replaced by Soda Core. As the primary tool for data reliability, you can install Soda Core, connect to your data sources, and convert your Soda SQL tests into SodaCL checks.

Use the following guide to understand the migration path from Soda SQL to Soda Core.
Contact support@soda.io for help migrating from Soda SQL to Soda Core.

Migrate to Soda Core
(Optional) Adjust in Soda Cloud
Migration implications

Migrate to Soda Core

1. Install Soda Core.
2. Configure Soda Core to connect to your data sources.
3. Convert your Soda SQL tests in your scan YAML file to SodaCL checks in a checks YAML file. Refer to SodaCL documentation and some example conversions, below.
   
4. Use Soda Core to run fresh scans of data in your data sources.
5. (Optional) Revise your programmatic scans to use Soda Core.

Example conversions from tests to checks

# SODA SQL - change over time
# Requires a Soda Cloud account
table_name: orders.yaml
metrics
  - row_count
  - missing_count
  - missing_percentage
  - ...
columns:
  ID:
    metrics:
      - distinct
      - duplicate_count
      - valid_count
      - avg
    historic metrics:
      - name: avg_dup_7
        type: avg
        metric: duplicate_count
        count: 7
    tests:
      - duplicate_count < avg_dup_7

# SODA CORE - change over time
# Requires a Soda Cloud account
checks for orders:
  - change avg last 7 for duplicate_count < 50

# SODA SQL - invalid
table_name: fulfillment
metrics:
  - row_count
  - missing_count
  - missing_percentage
  - values_count
  - ...
columns:
  country:
    valid_values:
      - US
      - UK
      - CN
    tests:
      - invalid_percentage == 0

# SODA CORE - invalid
checks for fulfillment:
  - invalid_percent(country) = 0%:
      valid values: [US, UK, CN]

(Optional) Adjust in Soda Cloud

If you connected Soda SQL to a Soda Cloud account, you can continue to use the same Soda Cloud account with Soda Core, with some implications.

1. Connect Soda Core to your existing Soda Cloud account.
2. Prepare your team to review Check Results in lieu of Monitor Results. Checks replace the concept of Monitors in Soda Cloud.
3. Write new checks in Soda Cloud in the context of Soda Agreements.
4. (Optional) Add a new data source in Soda Cloud using a Soda Agent.

Migration implications

There are a couple of implications to take into consideration when migrating from Soda SQL to Soda Core.

• The historical test result data accumulated in Soda Cloud for a Soda SQL dataset does not carry over to the same dataset that you add using Soda Core. In other words, all previously recorded historical test results do not influence new scan result data in Soda Cloud. Soda SQL and Soda Core results remain forever separate from each other.
• Any Monitors that you or your team created in Soda Cloud do not apply to the same datasets that you add using Soda Core. You must convert the monitors into SodaCL checks within the context of an Agreement.

Go further

• Contact support@soda.io for help migrating from Soda SQL to Soda Core.
• Need help? Join the Soda community on Slack.
• Learn more about writing checks with SodaCL.
  

Was this documentation helpful?

What could we do to improve this page?

• Suggest a docs change in GitHub.
• Share feedback in the Soda community on Slack.

Last modified on 31-May-23

Sign up for Soda