> For the complete documentation index, see [llms.txt](https://docs.soda.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.soda.io/data-observability/scan-time-and-scan-time-strategy.md).

# Scan time strategy

Selecting the right scan time is essential for accurate data monitoring and reliable metric collection. Scans that occur too early may run before the data has been fully loaded into the database, leading to false positives or misleading results. This guide outlines how to determine the best scan time based on your data load patterns and operational needs.

## Scan frequency

Scans can be scheduled to occur from **hourly** to **weekly**. The time jumps are meant to fit into a 24-hour cycle that matches **hourly/daily seasonalities** related to how humans organize their day. Metric Monitoring can happen every:

|                       |                          |
| --------------------- | ------------------------ |
| <ul><li>1 h</li></ul> | <ul><li>8 h</li></ul>    |
| <ul><li>2 h</li></ul> | <ul><li>12 h</li></ul>   |
| <ul><li>3 h</li></ul> | <ul><li>1 day</li></ul>  |
| <ul><li>4 h</li></ul> | <ul><li>1 week</li></ul> |
| <ul><li>6 h</li></ul> |                          |

## Key considerations

### Data load completion time

When is the database load expected to be complete?

* Determine when the relevant tables or datasets are expected to be fully loaded.
* Factor in common variances: if a load is expected to complete by `00:00 UTC` but occasionally finishes at `00:10 UTC`, account for the expected, albeit sporadic, delay.

Knowing this helps avoid scanning too early and capturing incomplete data.

### Acceptable load delay tolerance

When is a delayed load considered late or "problematic"?

* If data arriving by `02:30 UTC` is still valid for monitoring purposes, it may be better to delay the scan to reduce false alerts.
* Scanning immediately after the earliest expected load time is not always necessary.

Understanding what qualifies as "late data" helps define the tolerance window for scan timing.

### Response window & team availability

How fast after the load can someone respond to issues flagged by monitors?

* If nobody can take action until `09:00 UTC`, scanning earlier may not be useful unless scans feed downstream processes or dashboards.

Choose a scan time that aligns with both data readiness and team readiness.

{% hint style="success" %}
**Consistency is key**

Running scans at the **same time every day** allows to build up a reliable baseline of expected behavior. This helps surface anomalies clearly when something deviates from the norm.
{% endhint %}

***

## Example scenario

* **Scan frequency**: daily
* **Expected load completion:** `00:00 UTC`
* **Occasional load delay:** up to `00:10 UTC`
* **Team available from:** `08:00 UTC`

#### Scan options

<table><thead><tr><th width="183.81668090820312">Strategy</th><th width="126.4000244140625">Scan time</th><th>Rationale</th></tr></thead><tbody><tr><td>Minimal buffer</td><td><code>00:15 UTC</code></td><td>Captures data soon after load with minor delay tolerance.</td></tr><tr><td>Conservative buffer</td><td><code>01:30 UTC</code></td><td>Allows extra time for delayed loads, reduces risk of false positives.</td></tr><tr><td>Operationally aligned</td><td><code>07:30 UTC</code></td><td>Ensures scan results are fresh and complete when the team starts reviewing.</td></tr></tbody></table>

## Scan scheduling at scale

When scanning large volumes of tables:

* It is acceptable to configure scans for the **same scheduled time** (e.g. `00:00 UTC`).
* Scans that are scheduled in large volumes (thousands of tables) may be configured to run at the same logical time, but **the system naturally distributes execution** based on queuing and available resources, so the actual execution will be **staggered**.

***

## Historical scans

* **Historical metric collection scans** (for metric baseline backfilling) run only **once** at configuration time.
* These scans are **not governed** by the scan schedule. They occur once and they are typically the most resource-intensive.

***

## Best practices

* **Consistency is key:** Using the same scan daily establishes a stable baseline for anomaly detection.
* **Early scans should be avoided:** Scheduling scans before the last acceptable load time is not recommended unless business needs require it.
* **Time zones should be centralized:** Aligning scan time with the database time zone is ideal, especially when your time partitioning column is based on the insert/load time in that time zone.
* **Monitoring and adjusting:** If load patterns or SLAs change, scan times should be revisited and adjusted accordingly.

***

{% if visitor.claims.plan === 'datasetStandard' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Dataset Standard license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}

{% if visitor.claims.plan === 'enterprise' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Team license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}

{% if visitor.claims.plan === 'enterpriseUserBased' %}
{% hint style="success" %}
You are **logged in to Soda** and seeing the **Enterprise license** documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).
{% endhint %}
{% endif %}

{% if !(visitor.claims.plan === 'enterprise' || visitor.claims.plan === 'enterpriseUserBased' || visitor.claims.plan === 'datasetStandard') %}
{% hint style="info" %}
You are **not logged in to Soda** and are viewing the default public documentation. Learn more about [Documentation access & licensing](/reference/documentation-access-and-licensing.md).

If you do have a Soda license, make sure to **log in to Soda Cloud in this same browser**.
{% endhint %}
{% endif %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.soda.io/data-observability/scan-time-and-scan-time-strategy.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.