Anomaly score checks
Last modified on 31-May-23
Use an anomaly score check to automatically discover anomalies in your time-series data.
Requires Soda Cloud and Soda Core Scientific.
checks for dim_customer:
- anomaly score for row_count < default
About anomaly score checks
Prerequisites
Install Soda Core Scientific
Define an anomaly score check
Anomaly score check results
Optional check configurations
List of comparison symbols and phrases
Troubleshoot Soda Core Scientific installation
Go further
About anomaly score checks
The anomaly score check is powered by a machine learning algorithm that works with measured values for a metric that occur over time. The algorithm learns the patterns of your data – its trends and seasonality – to identify and flag anomalies in time-series data.
If you have connected Soda Core to a Soda Cloud account, Soda Core pushes check results to your cloud account where Soda Cloud stores all the previously-measured, historic values for your checks in the Cloud Metric Store. SodaCL can then use these stored values to establish a baseline of normal metric values against which to evaluate future metric values to identify anomalies. Therefore, you must have a created and connected a Soda Cloud account to use anomaly score checks.
Prerequisites
- You have a Soda Cloud account and have connected Soda Core to Soda Cloud.
- You have installed Soda Core Scientific in the same directory or virtual environment in which you installed Soda Core.
Install Soda Core Scientific
To use an anomaly score check, you must install Soda Core Scientific in the same directory or virtual environment in which you installed Soda Core. Best practice recommends installing Soda Core and Soda Core Scientific in a virtual environment to avoid library conflicts, but you can Install Soda Core Scientific locally if you prefer.
- Set up a virtual environment, as described in the Soda Core install documentation.
- Install Soda Core in your new virtual environment.
- Use the following command to install Soda Core Scientific.
pip install soda-core-scientific
Note that installing the Soda Core Scientific package also installs several scientific dependencies. Reference the soda-core-scientific setup file in the public GitHub repository for details.
Refer to Troubleshoot Soda Core Scientific installation for help with issues during installation.
Define an anomaly score check
The following example demonstrates how to use the anomaly score for the row_count
metric in a check. You can use any numeric, missing, or validity metric in lieu of row_count
.
checks for dim_customer:
- anomaly score for row_count < default
- Currently, you can only use
< default
to define the threshold in an anomaly score check. - By default, anomaly score checks yield warn check results, not fails.
You can use any numeric, missing, or validity metric in anomaly score checks. The following example detects anomalies for the average of order_price
in an orders
dataset.
checks for orders:
- anomaly score for avg(order_price) < default
The following example detects anomalies for the count of missing values in the id
column.
checks for orders:
- anomaly score for missing_count(id) < default:
missing_values: [None, No Value]
Anomaly score check results
Because the anomaly score check requires at least four data points before it can start detecting what counts as an anomalous measurement, your first few scans will yield a check result that indicates that Soda does not have enough data.
Soda Core 3.0.0xx
Anomaly Detection Frequency Warning: Coerced into daily dataset with last daily time point kept
Data frame must have at least 4 measurements
Skipping anomaly metric check eval because there is not enough historic data yet
Scan summary:
1/1 check NOT EVALUATED:
dim_customer in adventureworks
anomaly score for missing_count(last_name) < default [NOT EVALUATED]
check_value: None
1 checks not evaluated.
Apart from the checks that have not been evaluated, no failures, no warnings and no errors.
Sending results to Soda Cloud
Though your first instinct may be to run several scans in a row to product the four measurments that the anomaly score needs, the measurements don’t “count” if the frequency of occurrence is too random, or rather, the measurements don’t represent enough of a stable frequency.
If, for example, you attempt to run eight back-to-back scans in five minutes, the anomaly score does not register the measurements resulting from those scans as a reliable pattern against which to evaluate an anomaly.
Consider using the Soda Core Python library to set up a programmatic scan that produces a check result for an anomaly score check on a regular schedule.
Optional check configurations
Supported | Configuration | Documentation |
---|---|---|
✓ | Define a name for an anomaly score check. | - |
✓ | Add an identity to a check. | Add a check identity |
Define alert configurations to specify warn and fail thresholds. | - | |
✓ | Apply an in-check filter to return results for a specific portion of the data in your dataset; see example. | Add an in-check filter to a check |
✓ | Use quotes when identifying dataset names; see example. Note that the type of quotes you use must match that which your data source uses. For example, BigQuery uses a backtick (`) as a quotation mark. | Use quotes in a check |
Use wildcard characters ( % or * ) in values in the check. | - | |
✓ | Use for each to apply anomaly score checks to multiple datasets in one scan; see example. | Apply checks to multiple datasets |
Apply a dataset filter to partition data during a scan; see example. | Scan a portion of your dataset |
Example with quotes
checks for "dim_customer":
- anomaly score for row_count < default
Example with for each
for each dataset T:
datasets:
- dim_customer
checks:
- anomaly score for row_count < default
List of comparison symbols and phrases
<
Troubleshoot Soda Core Scientific installation
While installing Soda Core Scientific works on Linux, you may encounter issues if you install Soda Core Scientific on Mac OS (particularly, machines with the M1 ARM-based processor) or any other operating system. If that is the case, consider using one of the following alternative installation procedures.
- Install Soda Core locally
- Troubleshoot Soda Core Scientific installation in a virtual env
- Use Docker to run Soda Core
Need help? Ask the team in the Soda community on Slack.
Install Soda Core Scientific Locally
- Set up a virtual environment, as described in the Soda Core install documentation.
- Install Soda Core in your new virtual environment.
- Use the following command to install Soda Core Scientific.
pip install soda-core-scientific
Note that installing the Soda Core Scientific package also installs several scientific dependencies. Reference the soda-core-scientific setup file in the public GitHub repository for details.
Use Docker to run Soda Core
Use Soda’s Docker image in which Soda Core Scientific is pre-installed.
- If you have not already done so, install Docker in your local environment.
- From Terminal, run the following command to pull the latest Soda Core’s official Docker image.
docker pull sodadata/soda-core
- Verify the pull by running the following command.
docker run sodadata/soda-core --help
Output:
Usage: soda [OPTIONS] COMMAND [ARGS]... Soda Core CLI version 3.0.xxx Options: --help Show this message and exit. Commands: scan runs a scan update-dro updates a distribution reference file
When you run the Docker image on a non-Linux/amd64 platform, you may see the following warning from Docker, which you can ignore.
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
- When you are ready to run a Soda scan, use the following command to run the scan via the docker image. Replace the placeholder values with your own file paths and names.
docker run -v /path/to/your_soda_directory:/sodacl sodadata/soda-core scan -d your_data_source -c /sodacl/your_configuration.yml /sodacl/your_checks.yml
Optionally, you can specify the version of Soda Core to use to execute the scan. This may be useful when you do not wish to use the latest released version of Soda Core to run your scans. The example scan command below specifies Soda Core version 3.0.0.
docker run -v /path/to/your_soda_directory:/sodacl sodadata/soda-core:v3.0.0 scan -d your_data_source -c /sodacl/your_configuration.yml /sodacl/your_checks.yml
What does the scan command do?
docker run
ensures that the docker engine runs a specific image.-v
mounts your SodaCL files into the container. In other words, it makes the configuration.yml and checks.yml files in your local environment available to the docker container. The command example maps your local directory to/sodacl
inside of the docker container.sodadata/soda-core
refers to the image thatdocker run
must use.scan
instructs Soda Core to execute a scan of your data.-d
indicates the name of the data source to scan.-c
specifies the filepath and name of the configuration YAML file.
Error: Mounts denied
If you encounter the following error, follow the procedure below.
docker: Error response from daemon: Mounts denied:
The path /soda-core-test/files is not shared from the host and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.
See https://docs.docker.com/desktop/mac for more info.
You need to give Docker permission to acccess your configuration.yml and checks.yml files in your environment. To do so:
- Access your Docker Dashboard, then select Preferences (gear symbol).
- Select Resources, then follow the Docker instructions to add your Soda project directory – the one you use to store your configuration.yml and checks.yml files – to the list of directories that can be bind-mounted into Docker containers.
- Click Apply & Restart, then repeat steps 2 - 4 above.
Error: Configuration path does not exist
If you encounter the following error, double check the syntax of the scan command in step 4 above.
- Be sure to prepend
/sodacl/
to both the congifuration.yml filepath and the checks.yml filepath. - Be sure to mount your files into the container by including the
-v
option. For example,-v /Users/MyName/soda_core_project:/sodacl
.
Soda Core 3.0.xxx
Configuration path 'configuration.yml' does not exist
Path "checks.yml" does not exist
Scan summary:
No checks found, 0 checks evaluated.
2 errors.
Oops! 2 errors. 0 failures. 0 warnings. 0 pass.
ERRORS:
Configuration path 'configuration.yml' does not exist
Path "checks.yml" does not exist
Troubleshoot Soda Core Scientific installation in a virtual env
If you have defined an anomaly score
check and you use an M1 MacOS machine, you may get aLibrary not loaded: @rpath/libtbb.dylib
error. This is a known issue in the MacOS community and is caused by issues during the installation of the prophet library. There currently are no official workarounds or releases to fix the problem, but the following adjustments may address the issue.
- Install soda-core-scientific as per the virtual environment installation instructions and activate the virtual environment.
- Use the following command to navigate to the directory in which the
stan_model
of theprophet
package is installed in your virtual environment.cd path_to_your_python_virtual_env/lib/pythonyour_version/site_packages/prophet/stan_model/
For example, if you have created a python virtual environment in a
/venvs
directory in your home directory and you use Python 3.9, you would use the following command.cd ~/venvs/soda-core-prophet11/lib/python3.9/site-packages/prophet/stan_model/
- Use the
ls
command to determine the version number ofcmndstan
thatprophet
installed. Thecmndstan
directory name includes the version number.ls cmdstan-2.26.1 prophet_model.bin
- Add the
rpath
of thetbb
library to yourprophet
installation using the following command.install_name_tool -add_rpath @executable_path/cmdstanyour_cmdstan_version/stan/lib/stan_math/lib/tbb prophet_model.bin
With
cmdstan
version2.26.1
, you would use the following command.install_name_tool -add_rpath @executable_path/cmdstan-2.26.1/stan/lib/stan_math/lib/tbb prophet_model.bin
Go further
- Need help? Join the Soda community on Slack.
- Reference tips and best practices for SodaCL.
Was this documentation helpful?
What could we do to improve this page?
- Suggest a docs change in GitHub.
- Share feedback in the Soda community on Slack.
Last modified on 31-May-23