Install Soda Library

1. Best practice dictates that you install the Soda Library CLI using a virtual environment. In your command-line interface tool, create a virtual environment in the .venv directory using the commands below. Depending on your version of Python, you may need to replace python with python3 in the first command.

   python -m venv .venv
   source .venv/bin/activate
   

2. Upgrade pip inside your new virtual environment.

   pip install --upgrade pip
   

3. Execute the following command, replacing soda-postgres with the install package that matches the type of data source you use to store data.

   # For bash interactive shell
   pip install -i https://pypi.cloud.soda.io soda-postgres
   # For zsh interactive shell
   pip install -i https://pypi.cloud.soda.io "soda-postgres"
   

To deactivate the virtual environment, use the following command:

deactivate

Troubleshoot

As of version 1.7.0, Soda Library packages include Pydantic version 2 for data validation. If your systems require the use of Pydantic version 1, you can install an extra package that uses Pydantic version 1. To do so, use the following command, adjusting the type of library to correspond with your data source.

#bash
pip install -i https://pypi.cloud.soda.io soda-postgres[pydanticv1]

#zsh
pip install -i https://pypi.cloud.soda.io  "soda-spark-df[pydanticv1]"

1. Best practice dictates that you install the Soda Library CLI using a virtual environment. In your command-line interface tool, create a virtual environment in the .venv directory using the commands below. Depending on your version of Python, you may need to replace python with python3 in the first command. Reference the virtualenv documentation for activating a Windows script.

   python -m venv .venv
   .venv\Scripts\activate
   

2. Upgrade pip inside your new virtual environment.

   pip install --upgrade pip
   

3. Execute the following command, replacing soda-postgres with the install package that matches the type of data source you use to store data.

   pip install -i https://pypi.cloud.soda.io soda-postgres
   

To deactivate the virtual environment, use the following command:

deactivate

Reference the virtualenv documentation for activating a Windows script.

Troubleshoot

As of version 1.7.0, Soda Library packages include Pydantic version 2 for data validation. If your systems require the use of Pydantic version 1, you can install an extra package that uses Pydantic version 1. To do so, use the following command, adjusting the type of library to correspond with your data source.

#bash
pip install -i https://pypi.cloud.soda.io soda-postgres[pydanticv1]

#zsh
pip install -i https://pypi.cloud.soda.io  "soda-spark-df[pydanticv1]"

Use Soda’s Docker image in which Soda Scientific is pre-installed. You need Soda Scientific to be able to use SodaCL distribution checks or anomaly detection checks.

1. If you have not already done so, install Docker in your local environment.
2. From Terminal, run the following command to pull Soda Library’s official Docker image; adjust the version to reflect the most recent release.

   docker pull sodadata/soda-library:v1.0.3
   

3. Verify the pull by running the following command.

   docker run sodadata/soda-library:v1.0.3 --help
   

   Output:

    Usage: soda [OPTIONS] COMMAND [ARGS]...
   
      Soda Library CLI version 1.0.x, Soda Core CLI version 3.0.xx
   
    Options:
      --version  Show the version and exit.
      --help     Show this message and exit.
   
    Commands:
      ingest           Ingests test results from a different tool
      scan             Runs a scan
      suggest          Generates suggestions for a dataset
      test-connection  Tests a connection
      update-dro       Updates contents of a distribution reference file
   

   When you run the Docker image on a non-Linux/amd64 platform, you may see the following warning from Docker, which you can ignore.

   WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
   

4. When you are ready to run a Soda scan, use the following command to run the scan via the docker image. Replace the placeholder values with your own file paths and names.

   docker run -v /path/to/your_soda_directory:/sodacl sodadata/soda-library scan -d your_data_source -c /sodacl/your_configuration.yml /sodacl/your_checks.yml
   

   Optionally, you can specify the version of Soda Library to use to execute the scan. This may be useful when you do not wish to use the latest released version of Soda Library to run your scans. The example scan command below specifies Soda Library version 1.0.0.

   docker run -v /path/to/your_soda_directory:/sodacl sodadata/soda-library:v1.0.0 scan -d your_data_source -c /sodacl/your_configuration.yml /sodacl/your_checks.yml
   

Error: Mounts denied

If you encounter the following error, follow the procedure below.

docker: Error response from daemon: Mounts denied: 
The path /soda-library-test/files is not shared from the host and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.
See https://docs.docker.com/desktop/mac for more info.

You need to give Docker permission to acccess your configuration.yml and checks.yml files in your environment. To do so:

1. Access your Docker Dashboard, then select Preferences (gear symbol).
2. Select Resources, then follow the Docker instructions to add your Soda project directory – the one you use to store your configuration.yml and checks.yml files – to the list of directories that can be bind-mounted into Docker containers.
3. Click Apply & Restart, then repeat steps 2 - 4 above.

Error: Configuration path does not exist

If you encounter the following error, double check the syntax of the scan command in step 4 above.

• Be sure to prepend /sodacl/ to both the congifuration.yml filepath and the checks.yml filepath.
• Be sure to mount your files into the container by including the -v option. For example, -v /Users/MyName/soda_project:/sodacl.

Soda Library 1.0.x
Configuration path 'configuration.yml' does not exist
Path "checks.yml" does not exist
Scan summary:
No checks found, 0 checks evaluated.
2 errors.
Oops! 2 errors. 0 failures. 0 warnings. 0 pass.
ERRORS:
Configuration path 'configuration.yml' does not exist
Path "checks.yml" does not exist

Install Soda Scientific to be able to use SodaCL distribution checks or anomaly detection checks.

You have two installation options to choose from:

• Install Soda Scientific in a virtual environment (Recommended)
• Use Docker to run Soda Library with Soda Scientific

Install Soda Scientific in a virtual environment (Recommended)

1. Set up a virtual environment, and install Soda Library in your new virtual environment.
2. Use the following command to install Soda Scientific.

pip install -i https://pypi.cloud.soda.io soda-scientific

Error: Library not loaded

If you have defined an anomaly detection check and you use an M1 MacOS machine, you may get aLibrary not loaded: @rpath/libtbb.dylib error. This is a known issue in the MacOS community and is caused by issues during the installation of the prophet library. There currently are no official workarounds or releases to fix the problem, but the following adjustments may address the issue.

1. Install soda-scientific as per the local environment installation instructions and activate the virtual environment.
2. Use the following command to navigate to the directory in which the stan_model of the prophet package is installed in your virtual environment.

   cd path_to_your_python_virtual_env/lib/pythonyour_version/site_packages/prophet/stan_model/
   

   For example, if you have created a python virtual environment in a /venvs directory in your home directory and you use Python 3.9, you would use the following command.

   cd ~/venvs/soda-library-prophet11/lib/python3.9/site-packages/prophet/stan_model/
   

3. Use the ls command to determine the version number of cmndstan that prophet installed. The cmndstan directory name includes the version number.

   ls
   cmdstan-2.26.1		prophet_model.bin
   

4. Add the rpath of the tbb library to your prophet installation using the following command.

   install_name_tool -add_rpath @executable_path/cmdstanyour_cmdstan_version/stan/lib/stan_math/lib/tbb prophet_model.bin
   

   With cmdstan version 2.26.1, you would use the following command.

   install_name_tool -add_rpath @executable_path/cmdstan-2.26.1/stan/lib/stan_math/lib/tbb prophet_model.bin
   

Use Docker to run Soda Scientific

Use Soda’s Docker image in which Soda Scientific is pre-installed. You need Soda Scientific to be able to use SodaCL distribution checks or anomaly detection checks.

1. If you have not already done so, install Docker in your local environment.
2. From Terminal, run the following command to pull Soda Library’s official Docker image; adjust the version to reflect the most recent release.

   docker pull sodadata/soda-library:v1.0.3
   

3. Verify the pull by running the following command.

   docker run sodadata/soda-library:v1.0.3 --help
   

   Output:

    Usage: soda [OPTIONS] COMMAND [ARGS]...
   
      Soda Library CLI version 1.0.x, Soda Core CLI version 3.0.xx
   
    Options:
      --version  Show the version and exit.
      --help     Show this message and exit.
   
    Commands:
      ingest           Ingests test results from a different tool
      scan             Runs a scan
      suggest          Generates suggestions for a dataset
      test-connection  Tests a connection
      update-dro       Updates contents of a distribution reference file
   

   When you run the Docker image on a non-Linux/amd64 platform, you may see the following warning from Docker, which you can ignore.

   WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
   

4. When you are ready to run a Soda scan, use the following command to run the scan via the docker image. Replace the placeholder values with your own file paths and names.

   docker run -v /path/to/your_soda_directory:/sodacl sodadata/soda-library scan -d your_data_source -c /sodacl/your_configuration.yml /sodacl/your_checks.yml
   

   Optionally, you can specify the version of Soda Library to use to execute the scan. This may be useful when you do not wish to use the latest released version of Soda Library to run your scans. The example scan command below specifies Soda Library version 1.0.0.

   docker run -v /path/to/your_soda_directory:/sodacl sodadata/soda-library:v1.0.0 scan -d your_data_source -c /sodacl/your_configuration.yml /sodacl/your_checks.yml
   

Error: Mounts denied

If you encounter the following error, follow the procedure below.

docker: Error response from daemon: Mounts denied: 
The path /soda-library-test/files is not shared from the host and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.
See https://docs.docker.com/desktop/mac for more info.

You need to give Docker permission to acccess your configuration.yml and checks.yml files in your environment. To do so:

1. Access your Docker Dashboard, then select Preferences (gear symbol).
2. Select Resources, then follow the Docker instructions to add your Soda project directory – the one you use to store your configuration.yml and checks.yml files – to the list of directories that can be bind-mounted into Docker containers.
3. Click Apply & Restart, then repeat steps 2 - 4 above.

Error: Configuration path does not exist

If you encounter the following error, double check the syntax of the scan command in step 4 above.

• Be sure to prepend /sodacl/ to both the congifuration.yml filepath and the checks.yml filepath.
• Be sure to mount your files into the container by including the -v option. For example, -v /Users/MyName/soda_project:/sodacl.

Soda Library 1.0.x
Configuration path 'configuration.yml' does not exist
Path "checks.yml" does not exist
Scan summary:
No checks found, 0 checks evaluated.
2 errors.
Oops! 2 errors. 0 failures. 0 warnings. 0 pass.
ERRORS:
Configuration path 'configuration.yml' does not exist
Path "checks.yml" does not exist