What was expected, stated first
Written before the test, so it can be checked against what the test actually found.
Building an Auditable Research Workflow
A result without its hypothesis, data source, parameters, and limitations attached is a claim someone has to take on faith. Recording all four is what turns it into something a reviewer can actually check.
- Identify the four fields an auditable research log needs: hypothesis, data source, parameters, and limitations.
- Explain why omitting parameter count hides overfitting risk from a reviewer.
- Explain why the data source and range need to be recorded for reproducibility.
- Recognize known limitations, like survivorship bias, as something to disclose rather than omit.
- Apply a documentation habit that lets research actually be reviewed, not just reported.
“Data came from an API” is not enough to reproduce the research
Reproducibility means another researcher can rerun the same process with the same inputs, code, parameters, and version context. Financial data revisions make timestamps and snapshots part of the evidence.
Research Pipeline
Which missing step makes a research result impossible to reproduce?
Research is not an equity curve. A minimal evidence chain preserves Source → Raw Data → Transformation → Rule → Result → Conclusion. Find the broken link, then build a Research Record.
SourceRaw dataRuleResultConclusion
Which step is missing?Build a Research Record
A field is still empty, so another researcher cannot fully replay the path.
A Result Without Its Context Is a Claim, Not Evidence
Everything covered in this module — sample size, curve-fitting, survivorship bias, lookahead bias — is only checkable if the research process is actually written down. A bare result, with no record of the hypothesis, data, parameters, or known limitations behind it, asks a reader to simply trust the conclusion rather than evaluate it.
Exactly which dataset and range
What lets a reviewer reproduce or independently check the result.
Every adjustable value used
What lets a reviewer assess overfitting risk.
Toggle Fields Off and See What a Reviewer Loses
Turn fields on and off and read what a reviewer can no longer check without each one.
A
Hypothesis
Missing this field: a later reviewer can't tell whether the test was designed to check a real idea or discovered by trying many things until something worked.
B
Data Source
Missing this field: the result can't be reproduced or checked against a different, independent dataset.
C
Parameters
Missing this field: a reviewer can't assess how many parameters were tuned, which is central to judging overfitting risk.
D
Limitations
Missing this field: the result can be mistaken for more robust than it actually is.
Disclosing a Weakness Is Different From Having One
Every piece of research has some limitation — a short test period, a dataset with known survivorship bias, a small number of independent sub-periods. Disclosing these doesn't make the research worse; it gives a reviewer what they need to weigh the result appropriately. Omitting them doesn't remove the limitation — it only hides it from view.
What Does Each Research Writeup Actually Support?
Pick a case and judge whether the writeup gives a reviewer enough to actually evaluate the result.
A
No parameter count recorded
A research writeup states the final result but does not mention how many parameters were tested or tuned along the way. Without a record of how many parameters were tried, a reviewer can't assess the overfitting risk behind the result.
B
A complete writeup
A research writeup states the hypothesis before testing, the exact data source and range, every parameter used, and known limitations of the test. This writeup gives a reviewer enough information to reproduce, check, and judge the reliability of the result.
C
No period stated
A research writeup presents a result without stating which historical period was tested. Without the tested period, a reviewer can't check for known risks like survivorship bias or an unusually favorable market stretch.
What a Reviewable Research Writeup Needs to Show
1
Hypothesis
What was expected, stated before the test ran.
2
Data source
The exact dataset, provider, and time range used.
3
Parameters
Every adjustable value tuned, and how many were tried.
4
Limitations
Known weaknesses like survivorship bias, sample size, or test-period sensitivity.
A reproducible result is automatically correct
A repeatable error, biased sample or unsuitable permission can also be reproduced.
A final result is the whole research record
The discarded trials, data version, parameters and revision reasons may be necessary to audit selection effects.
Leave a Research Trail Someone Else Can Rebuild
Hypothesis, data source, parameters, and limitations together let a reviewer actually check the work.
Omitting it hides exactly what a reviewer needs to judge the result.
They let a reviewer weigh the result appropriately instead of overtrusting it.
Put Your Understanding to the Test
Submit your answers to see detailed explanations.
Question 1 of 3
A research writeup states a final result but omits how many parameters were tuned along the way. What problem does this create?
Question 2 of 3
Why should known limitations, like a dataset's survivorship bias, be disclosed in a research writeup rather than left out?
Question 3 of 3
A colleague's writeup names the hypothesis, the exact dataset and date range, all six parameters tuned, and the survivorship bias in the data. The finding itself is modest. What does the completeness of that log establish?
Stuck? Ask Mira to Break It Down
Describe your research process, and Mira can help you check it against the four fields covered in this lesson — it won't run or validate the research for you.
Checking sign-in status...
Put this concept into practice
Validating a strategy: from historical evidence to new evidence
Build on backtest limitations from earlier in the course: data used to design a rule can't be the only evidence for it; a new time window that had no part in shaping the rule (out-of-sample, in beginner terms) is more valuable for validation; simulation, paper trading, and small-scale live observation each have their own limits. The core takeaway: evidence accumulates, certainty does not. Walk-forward, cross-validation, and Monte Carlo are out of scope.