A
3 parameters or fewer
A small number of parameters relative to a typical dataset — easier to reason about and less prone to fitting noise.
Overfitting and the Curve-Fitting Trap
Add enough adjustable parameters to any strategy and it will eventually fit its own test data perfectly. That perfect fit is often a description of noise, not a discovery of a real, repeatable pattern.
- Explain overfitting as fitting a strategy to noise rather than a repeatable pattern.
- Recognize how parameter count relative to data size affects overfitting risk.
- Identify a strategy that only performs well within one narrow, specific test window.
- Value consistent performance across independent sub-periods over a single aggregate result.
- Apply skepticism toward parameters added only to improve a historical fit.
Pick the best of 20 random parameter sets. Did you discover an edge?
Overfitting occurs when a model or rule adapts too closely to the training sample and mistakes noise for repeatable structure. More search freedom creates more chances to find a lucky historical winner. Selecting a historical champion from many candidates creates data snooping.
Overfitting Visual Lab
Give 20 no-edge parameter sets enough history and one will look like a champion
This is explicitly a simulation: all 20 parameter sets are generated from noise and have no true edge. First see only in-sample performance and choose the one you would deploy; only then reveal out-of-sample.
Choose the parameter to deployEnough Adjustable Parameters Can Fit Almost Any Historical Data
Every historical dataset contains some amount of pure noise — coincidental patterns that happened once and have no reason to repeat. A strategy with enough adjustable parameters can be tuned to capture that noise as if it were a real, repeatable edge. The result looks impressive on the exact data it was tuned against and often falls apart anywhere else.
See How Parameter Count Relates to Overfitting Risk
Move the slider and read how a strategy with that many free parameters is typically regarded, relative to a standard amount of test data.
B
4 to 8 parameters
A moderate number of parameters — worth checking performance across several independent sub-periods before trusting the result.
C
More than 8 parameters
A large number of adjustable parameters — the more free parameters relative to the data available, the easier it becomes to fit coincidental noise rather than a real pattern.
Two Signals That a Result May Be Overfit
A strategy whose performance depends heavily on the exact start and end dates chosen for the test — where shifting the window by a few weeks meaningfully changes the conclusion — is showing a classic overfitting warning sign. A strategy that performs consistently across several distinct, independent sub-periods is showing the opposite, more reassuring pattern.
What Does Each Case Suggest About Overfitting Risk?
Pick a case and judge whether it points toward a robust result or a warning sign of overfitting.
A
Added purely for fit
A researcher adds a ninth adjustable parameter to a strategy specifically because it improves the historical result by a small amount, with no other justification for including it. Adding parameters purely because they improve the historical fit, without independent justification, raises real overfitting risk.
B
Consistent across sub-periods
A researcher tests a strategy with two parameters across two decades of data spanning multiple distinct market conditions, and it performs consistently across each sub-period. Consistent performance across multiple independent sub-periods is a stronger signal than a single aggregate result.
C
Sensitive to the test dates
A researcher notices a strategy's historical performance depends heavily on the exact starting and ending dates chosen for the test. Extreme sensitivity to the exact test period is a warning sign that the result may reflect noise rather than a robust pattern.
Four Tests for a Strategy That Has Been Tuned
1
Parameter count
How many free parameters were tuned, relative to the amount of data tested?
2
Justification
Was each parameter added for an independent reason, or only because it improved the fit?
3
Sub-period consistency
Does the result hold up across several distinct, independent sub-periods?
4
Date sensitivity
Does shifting the test window slightly change the conclusion significantly?
A Good Fit Is Not Yet Good Evidence
Weigh parameter count against the amount of independent data available.
A pattern holding up separately in several periods is stronger than one aggregate result.
A robust pattern shouldn't vanish because the test window shifted slightly.
Put Your Understanding to the Test
Submit your answers to see detailed explanations.
Question 1 of 3
A researcher adds a ninth adjustable parameter to a strategy specifically because it improves the historical result, with no other justification. What does this suggest?
Question 2 of 3
A strategy's historical performance changes dramatically when the test's start date shifts by just a few weeks. What does this suggest?
Question 3 of 3
Two candidates are on the table. A uses nine parameters tuned on one three-year window and posts the higher aggregate result. B uses two parameters, posts a weaker aggregate result, and holds up separately across four decades of different conditions. Which read is defensible?
Stuck? Ask Mira to Break It Down
Describe how many parameters your strategy uses and how much data you've tested it on, and Mira can help you think through the overfitting risk — it won't tell you whether your specific strategy will work going forward.
Checking sign-in status...
Put this concept into practice
A Backtest Isn't a Crystal Ball
Understand that a backtest applies rules to historical data to examine past performance and risk characteristics, but it can't prove future profitability, and ignoring costs overstates performance.
Validating a strategy: from historical evidence to new evidence
Build on backtest limitations from earlier in the course: data used to design a rule can't be the only evidence for it; a new time window that had no part in shaping the rule (out-of-sample, in beginner terms) is more valuable for validation; simulation, paper trading, and small-scale live observation each have their own limits. The core takeaway: evidence accumulates, certainty does not. Walk-forward, cross-validation, and Monte Carlo are out of scope.