In Soda Cloud, you can create a monitor that automatically detects anomalies in your time-series data.
Anomaly detection is a monitor Evaluation Type powered by a machine learning algorithm that works with measurements that occur over time. The algorithm learns the patterns of your data – its trends and seasonality – to identify and flag anomalous measurements in time-series data. Seasonality is a term that describes a common pattern of time-series data and means “something cyclical, irrespecitve of the general direction of the data”. For example, the number of orders that occur on your platform might show a clear drop during the weekend, or peak during the holiday season, irrespecitve of whether your platform is growing or not.
The algorithm automatically adapts to your data. It learns how your data generally behaves over time to build a model of data patterns, then compares actual measurements to that model. If the two disagree, the algorithm identifies the actual measurement as an anomaly and calculates:
a. the certainty of the algorithm’s mental model, or how well it knows your data, and
b. the size of the anomaly, or how far the measurement deviates from the expected value.
Soda Cloud uses the certainty and size calculations to derive the thresholds for triggering alerts:
- a large, certain anomaly triggers a critical alert
- a small, less certain anomaly triggers a warning
For example, if the test you define in your monitor measures the row count of one of your datasets every hour as part of your transformation pipeline, you can use anomaly detection to discover unexpected volumes of entries in the dataset. If the test you define measures the price of an asset at the end of each day, you can use anomaly detection to get an alert if the price jumps wildly high or unexpectedly low.
To use anomaly detection, you must be an Admin of the organization, or have a Manager or Editor role for the monitor’s dataset. See Roles and rights for details.
Note that when Soda SQL first pushes scan results to your connected Soda Cloud account, Soda Cloud automtically creates an anomaly detection monitor for each dataset that contains time-series data.
To manually add an anomaly detection monitor, follow the steps to create a new monitor and select Anomaly Detection as the Evaluation Type in step two of the creation flow. Beyond that, you do not need to specify any other details about your data; Soda Cloud automatically begins learning about your data and triggering alerts when Soda scans reveal anomalies.
When you access the Monitors dashboard to review the monitor’s test results, the chart that displays the time-series data and its anomalies gives you the opportunity to manually provide feedback on an anomalous measurement. Your feedback on the accuracy of the calculated anomaly helps the algorithm to adapt its recognition of exceptions and deviations, and refine its thresholds for triggering alerts.
- Sign up for a free Soda Cloud account.
- Learn how to Create monitors and alerts.
- Learn more about examining failed rows in Soda Cloud.
- Need help? Join the Soda community on Slack.
Last modified on 26-Nov-21