Imagine re-running a backtest script you haven't touched in six months. Not a single line of your trading code has changed, yet your strategy's annual return suddenly drops from 18% to 9%.
How is this possible? You didn't touch the code—so what changed? The answer is almost always the underlying data.
Financial data feeds are living organisms, not static historical tablets. Providers silently restate earnings, adjust historical splits, patch bad tick quotes, and revise index membership retroactive to past dates. Without Data Provenance—documenting where your data originated, when it was captured, and what transformations occurred—your backtests are built on shifting sands.
Data Provenance is the auditable lineage record that documents a dataset's origin, capture timestamp, preprocessing rules, and modification history. In financial research, prices and fundamentals are constantly rewritten after the fact (e.g., quarterly accounting restatements, revised corporate actions, retroactive survivor filtering). Tracking provenance ensures that your backtest reflects what a trader *actually knew at the historical moment in time* (Point-in-Time accuracy), rather than future knowledge silently patched into a downloaded CSV file.
| Data Mutation Type | What Actually Changed Behind the Scenes? | Distortion on Backtest Results |
|---|---|---|
| Retroactive Earnings Restatements | A company restated its 2021 earnings in late 2023 due to an accounting audit | Backtest uses the 2023 corrected number in 2021, creating lookahead bias |
| Silent Corporate Action Adjustments | Vendor changes how it computes reverse split or spin-off price ratios | Historical moving averages shift, generating artificial buy signals |
| Survivorship Cleansing | Vendor removes liquidated or delisted penny stocks from historical archives | Portfolio looks dramatically safer and more profitable than it was in reality |
| Timezone & Bar Timestamp Shifts | Vendor switches historical daily bars from 4:00 PM close to 5:00 PM settlement | Strategy executes orders at price levels that were unavailable during active trading |
- Never Overwrite Raw Data: Treat primary downloads as immutable archives. Store them in read-only directories labeled with download dates (e.g., `raw_data_2026_09_01/`).
- Record Point-in-Time Timestamps: Ensure your fundamental dataset records two distinct dates for every financial figure: the fiscal period end date (e.g., Dec 31) and the actual public filing date (e.g., Feb 28).
- Generate Cryptographic Checksums (Hashes): Compute an MD5 or SHA-256 hash for your baseline data files and store it in your research notebook to confirm data files have not been modified.
- Document Cleaning Transformations in Code: Never manually delete bad ticks or format cells inside Excel. Perform all cleaning transformations in script pipelines that leave an executable audit trail.
Frequently Asked Questions
Why do free web scrapers pose high data provenance risks?
Free public financial portals constantly modify their back-end scraping formats, adjust pricing histories without notice, and rarely preserve point-in-time filing dates, making it impossible to audit past backtest results.
What is 'Point-in-Time' (PIT) data, and why is it expensive?
Point-in-time data preserves a snapshot of what was publicly known on every single calendar day. Storing daily historical revisions requires massive database architecture, which is why institutional vendors charge premium subscription rates for PIT feeds.
Can data provenance be tracked simply by naming folders well?
For small-scale independent research, structured folder naming conventions combined with a lightweight `manifest.json` or text log detailing the data source, download parameters, and download date is sufficient to maintain basic provenance.
