A good score on a test you kept changing is weak proof. Set aside some history. Lock the rules before you look. Then keep the result, even when it is poor or too small to judge.
What does out of sample mean?
In-sample data is the history you use to build or tune a strategy. Out-of-sample data is the history you save for a later check. The key is whether that data helped choose the rules, not the label on the report.
TradingView’s strategy guide describes this split: tune on the first set, then test on the reserved set without more tuning. It also warns that no test can guarantee future results. This is a research method, not a claim about a TradeLab control.
Write the plan before the first look
Save the market, data source, dates, time frame, entry, exit and trade size. Add fees, price-slip assumptions and when an order could be filled. Write what you will check and why it matters. Use our trading-rule template to settle vague rules first.
Keep time in order. Later data must not shape an earlier decision. Use the look-ahead bias check to review that risk. A date split alone cannot fix a rule that uses facts from the future.
A made-up test record
This example shows how to keep a record. We did not run these tests. It names no stock, gives no return and sets no rule for how much data you need.
| Step | What the record says | What it means |
|---|---|---|
| Lock version A | Use 2023–2024 to build the rules. Reserve January–June 2025. Save all settings before opening the test. | These dates are made up. The saved plan is the reference. |
| First test | Run version A on the reserved dates. Keep the full output, costs, trades and errors. | Do not change the rules to improve this score. |
| Change to version B | After seeing A’s test, change an entry rule. Record why and keep A’s result. | The same test period has now helped shape B. A rerun is more research, not a fresh independent check. |
For a revised plan, say which new evidence is still untouched. If you have already used all the history to choose rules, say so. Do not rename an old test to make it seem new.
Keep the failed tries too
The original research on backtest overfitting by Bailey and colleagues explains a key limit: a holdout by itself does not account for all the trials used to find a strategy. Trying many choices and showing only the best can hide how that choice was made.
Keep a trial log: version, date, change, reason and result. Include poor results, empty trade lists and errors. If many versions were compared on the reserved period, report that too. One strong row does not erase the other tests.
What can the result show?
- A poor result: the fixed plan did not hold up on this test. Keep it. Check the rules and errors before you decide what to change.
- A small sample: there may be too little evidence to judge. Do not treat “not enough data” as a pass.
- A strong result: the plan passed the checks you set for this period. It still does not prove future profit or real fills.
Do not pick one pass mark for every strategy. Review trade count, market conditions, large losses and whether a few trades drove the score. Read how to judge the trade sample and test the cost assumptions before drawing a conclusion.
What if you need to tune again?
You can return to research. Keep the old result and name the new version. A walk-forward test repeats a set process: build with earlier data, then check the next period. Set that process before judging its score. Changing the windows to find a better score creates the same problem again.
Paper trading can add observations as new data arrives. It is not a promise of live results. Our backtesting and paper trading guide explains what each step checks.
Save this with the test
- The question and rule version.
- The build dates and reserved test dates.
- The data, settings, trade size, costs and fill rules.
- The first result and every later change.
- What remains unknown and what you will check next.
Keep this in your research notes, even if your tool has no built-in form for it. Use the full backtest checklist before treating a report as useful evidence.
Backtests use past data and assumptions. They are hypothetical and do not predict future results. Out-of-sample tests do not remove market, model, liquidity or execution risk. Paper trading also differs from live trading. This guide is for education. It is not financial advice or a recommendation to trade.