Link Search Menu Expand Document

Monitor schema evolution

In Soda Cloud, you can create a monitor that automatically detects changes in your dataset’s schema.

Schema Evolution is a type of monitor that notifies you when columns in a dataset have been added, removed, or changed since the previous Soda scan of the dataset. You can set the type of alert – Warning or Critical – associated with each evolution event. For example, you can configure the monitor so that additions and changes to your dataset’s schema issue a Warning, while a removal of a column in the dataset issues a Critical alert.

Use a schema evolution monitor

Soda Cloud automatically adds a schema evolution monitor to every new dataset you onboard using Soda SQL. In other words, when you use Soda SQL to connect to a new data source, then run soda analyze to discover all of its datasets, Soda SQL sends the dataset information to your Soda Cloud account. Soda Cloud then automatically creates a new schema evolution monitor for each of the datasets in your data source.

To manually create a schema evolution monitor, you must be an Admin of the organization, or have a Manager or Editor role for the monitor’s dataset. See Roles and rights for details.

  1. Note that you can create only one Schema Evolution monitor per dataset. Follow the workflow in Soda Cloud to create a new monitor and, when prompted, select Schema Evolution as the Monitor Type.
  2. Complete the monitor creation flow, including the type of alert, Warning or Critical, that you wish to associate with each schema evolution event: column(s) added, column(s) removed, and column(s) changed.

When you access the Monitors dashboard to review the monitor’s test results, the bar chart displays the volume of each type of event that has occurred in your schema. The Diagnostics and Schema tabs below the chart offer more granular insight into the events that occurred. Note that the monitor displays results only after Soda Cloud has collected at least two measurements, or in other words, after it has completed two scans of the same dataset.

Go further

Last modified on 26-Nov-21

Was this documentation helpful?
Give us your feedback in the #soda-docs channel in the Soda community on Slack or open an issue in GitHub.