### **Enhanced Configuration Options**
For detailed information on configuring custom attribute syncing, see the [Custom Attribute Syncing](https://github.com/sodadata/soda-collibra-integration/blob/main/documentation.md#-custom-attribute-syncing) section above.
### **Custom Logging**
```python
# In your code
import logging
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
```
### **Environment Variables**
```python
# Set custom config path
export SODA_COLLIBRA_CONFIG=/path/to/custom/config.yaml
# Enable debug mode
export SODA_COLLIBRA_DEBUG=true
```
***
## Troubleshooting
### **Common Issues**
#### **Performance Issues**
* **Slow Processing**: Increase `BATCH_SIZE` and `DEFAULT_PAGE_SIZE`
* **Rate Limiting**: Increase `RATE_LIMIT_DELAY`
* **Memory Usage**: Decrease `CACHE_MAX_SIZE`
#### **Connection Issues**
* **API Timeouts**: Check network connectivity and API endpoints
* **Authentication**: Verify credentials and permissions
* **Rate Limits**: Monitor API usage and adjust delays
#### **Data Issues**
* **Missing Assets**: Ensure required asset types exist in Collibra
* **Relation Failures**: Verify relation type configurations
* **Domain Mapping**: Check domain IDs and JSON formatting
#### **Diagnostic Metrics Issues**
* **Missing Diagnostic Attributes**: Check if Soda checks have `lastCheckResultValue.diagnostics` data
* **Incomplete Metrics**: Some diagnostic types may only have partial metrics (e.g., `aggregate` checks lack `failedRowsCount`)
* **Attribute Type Configuration**: Verify diagnostic attribute type IDs are configured correctly in `config.yaml`
* **Zero Division Errors**: System automatically prevents division by zero when calculating fractions
### **Debug Commands**
```bash
# Full debug output
python main.py --debug 2>&1 | tee debug.log
# Verbose logging with timestamps
python main.py --verbose
# Test specific components
python main.py --test-soda --debug
python main.py --test-collibra --debug
```
### **Log Analysis**
Look for these patterns in debug logs:
#### **General Operation Patterns:**
* `Rate limit prevention`: Normal throttling behavior
* `Successfully updated/created`: Successful operations
* `Skipping dataset`: Expected filtering behavior
* `ERROR`: Issues requiring attention
#### **Diagnostic Processing Patterns:**
* `Processing diagnostics`: Diagnostic data found in check result
* `Found failedRowsCount in 'X'`: Successfully extracted failure count from diagnostic type X
* `Found checkRowsTested in 'X'`: Successfully extracted row count from diagnostic type X
* `Using datasetRowsTested from 'X' as fallback`: Fallback mechanism activated
* `No diagnostics found in check result`: Check has no diagnostic data (normal for some check types)
* `Calculated check_rows_passed`: Successfully computed passing rows
* `Added check_X_attribute`: Diagnostic attribute successfully added to Collibra
***
## Reference
### **Common Commands**
```bash
# Basic run with default config
python main.py
# Debug mode with detailed logging
python main.py --debug
# Use custom configuration file
python main.py --config custom.yaml
# Test individual components
python main.py --test-soda --debug
python main.py --test-collibra --debug
```
### **Key Configuration Sections**
* **Collibra Base**: `collibra.base_url`, `collibra.username`, `collibra.password`
* **Soda API**: `soda.api_key_id`, `soda.api_key_secret`
* **Custom Attributes**: `soda.attributes.custom_attributes_mapping_soda_attribute_name_to_collibra_attribute_type_id`
* **Domain Mapping**: `collibra.domains.soda_collibra_domain_mapping`
* **Ownership Sync**: `collibra.responsibilities.owner_role_id`
### **Essential UUIDs to Configure**
* Asset types (table, check, dimension, column)
* Attribute types (evaluation status, sync date, diagnostic metrics)
* Relation types (table-to-check, check-to-dimension)
* Domain IDs for asset creation
***
### Support
For issues and questions:
1. Check the [#troubleshooting](#troubleshooting "mention") section
2. Enable [#debug-logging](#debug-logging "mention") for detailed information
3. Review the performance metrics for bottlenecks
4. Consult the [#unit-tests](#unit-tests "mention") for usage examples
5. Contact for additional help
***
{% if visitor.claims.plan === 'datasetStandard' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}
{% if visitor.claims.plan === 'enterprise' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}
{% if visitor.claims.plan === 'enterpriseUserBased' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}
{% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %}
{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**.
{% endhint %}
{% endif %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.soda.io/integrations/collibra/operations-and-advanced-usage.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.