Last modified on 26-Jan-23
Soda SQL, the original command-line tool that Soda created to test for data quality, has been deprecated and replaced by Soda Core. As the primary tool for data reliability, you can install Soda Core, connect to your data sources, and convert your Soda SQL tests into SodaCL checks.
Use the following guide to understand the migration path from Soda SQL to Soda Core.
Contact email@example.com for help migrating from Soda SQL to Soda Core.
- Install Soda Core.
- Configure Soda Core to connect to your data sources.
- Convert your Soda SQL tests in your scan YAML file to SodaCL checks in a checks YAML file. Refer to SodaCL documentation and some example conversions, below.
- Use Soda Core to run fresh scans of data in your data sources.
- (Optional) Revise your programmatic scans to use Soda Core.
# SODA SQL - change over time # Requires a Soda Cloud account table_name: orders.yaml metrics - row_count - missing_count - missing_percentage - ... columns: ID: metrics: - distinct - duplicate_count - valid_count - avg historic metrics: - name: avg_dup_7 type: avg metric: duplicate_count count: 7 tests: - duplicate_count < avg_dup_7 # SODA CORE - change over time # Requires a Soda Cloud account checks for orders: - change avg last 7 for duplicate_count < 50
# SODA SQL - invalid table_name: fulfillment metrics: - row_count - missing_count - missing_percentage - values_count - ... columns: country: valid_values: - US - UK - CN tests: - invalid_percentage == 0 # SODA CORE - invalid checks for fulfillment: - invalid_percent(country) = 0%: valid values: [US, UK, CN]
If you connected Soda SQL to a Soda Cloud account, you can continue to use the same Soda Cloud account with Soda Core, with some implications.
- Connect Soda Core to your existing Soda Cloud account.
- Prepare your team to review Check Results in lieu of Monitor Results. Checks replace the concept of Monitors in Soda Cloud.
- Write new checks in Soda Cloud in the context of Soda Agreements.
- (Optional) Add a new data source in Soda Cloud using a Soda Agent.
There are a couple of implications to take into consideration when migrating from Soda SQL to Soda Core.
- The historical test result data accumulated in Soda Cloud for a Soda SQL dataset does not carry over to the same dataset that you add using Soda Core. In other words, all previously recorded historical test results do not influence new scan result data in Soda Cloud. Soda SQL and Soda Core results remain forever separate from each other.
- Any Monitors that you or your team created in Soda Cloud do not apply to the same datasets that you add using Soda Core. You must convert the monitors into SodaCL checks within the context of an Agreement.
- Contact firstname.lastname@example.org for help migrating from Soda SQL to Soda Core.
- Need help? Join the Soda community on Slack.
- Learn more about writing checks with SodaCL.
- Review the features and functionalities that Soda Cloud offers.
Was this documentation helpful?
What could we do to improve this page?
Last modified on 26-Jan-23