How to Backtest a Crypto Strategy Without Fooling Yourself

A backtest is not evidence that a strategy will make money. It is evidence that a precisely defined set of rules produced a result on a particular dataset under a particular execution model. The gap between those two statements is where most false confidence enters.

The objective of a serious backtest is not to maximize historical return. It is to expose a strategy to conditions that should make a weak idea fail.

Start with a falsifiable hypothesis

“Buy when momentum is strong” is not a testable strategy. A backtest needs exact inputs, timing and decisions.

A complete hypothesis states:

• the market and venue;
• the data frequency;
• the signal formula;
• when the signal becomes observable;
• the order submitted after the signal;
• position sizing and leverage;
• exit, stop and time-out rules;
• costs and funding assumptions;
• conditions under which the idea should not trade.

If the rules change every time the equity curve looks bad, the process is fitting a story to history rather than testing an idea.

The five biases that break most backtests

1. Look-ahead bias

Look-ahead bias uses information before it was available. A common example is calculating a daily indicator with the day's closing price and assuming an order filled at that same close. The strategy could not know the final close until the period ended.

Use explicit event timing. If a candle closes at 12:00, the earliest realistic market decision starts after 12:00. For slower data releases, use the publication timestamp rather than the period the statistic describes.

2. Survivorship bias

Testing only assets that exist today removes delisted, failed or illiquid markets from history. This can make a strategy appear safer than it was. Universe membership must be reconstructed for each date.

In crypto, listing and delisting rules are especially important. A strategy that trades “the top 20 assets” needs the top 20 as they were known then, not today's winners projected backward.

3. Data leakage

Leakage is broader than look-ahead bias. It includes normalization with future data, choosing parameters after seeing the test period, or using a feature whose construction quietly incorporates later observations.

Fit scalers, thresholds and models on training data only. Then freeze them before validation.

4. Multiple-testing bias

If hundreds of parameter combinations are tried, one can look excellent by chance. Selecting the best curve hides the number of failed attempts.

Track every run. Compare the chosen result with the distribution of all tested variants. A strategy that only works at one exact parameter value is fragile even if that point has a high Sharpe ratio.

5. Unrealistic execution

A signal can be real while the simulated fills are impossible. Small-cap crypto markets make this obvious, but the problem exists everywhere.

The execution model should include:

• maker or taker fees;
• bid-ask spread;
• slippage that grows with order size;
• funding payments for perpetual positions;
• latency between signal and order;
• partial fills and rejected orders;
• minimum size and tick constraints;
• market impact where capacity matters.

CME's backtesting research explicitly notes that before-cost Sharpe ratios are overstated when transaction costs are omitted. That warning becomes more important as turnover rises.

A robust data split

A single train/test split is better than testing on the whole history, but it can still make one period carry too much weight.

The test set stops being a test set once its result changes the strategy. If you modify the rule after seeing it, create a new untouched period or move to forward testing.

For strategies that adapt over time, walk-forward testing is often more realistic. Train on a rolling window, freeze the rule, test on the next period, then advance. Join only the out-of-sample segments into the reported equity curve.

Build an execution model before reading the result

Costs should not be added only after a strategy looks profitable. Define them first.

A simple net return model is:

Net return = gross price return − fees − spread − slippage − funding − borrow costs

Each component can vary by venue, instrument, time and order size. A conservative model is preferable to a precise-looking constant that is too optimistic.

Run at least three cost scenarios:

1. observed or baseline costs;
2. stressed costs, such as wider spread and higher slippage;
3. break-even costs, where the edge disappears.

The break-even test answers a practical question: how much execution error can the strategy survive?

Evaluate more than return

Total return is easy to improve with leverage. It does not reveal the path or capacity of the strategy.

Review:

• maximum drawdown and drawdown duration;
• volatility and downside deviation;
• hit rate and average win-to-loss ratio;
• turnover and average holding period;
• gross and net exposure;
• largest single-trade contribution;
• performance by year, regime and asset;
• sensitivity to delayed entry and worse fills;
• capacity relative to market volume;
• number of independent trades.

A strategy with 12 trades and a beautiful curve has little evidence. A high-frequency result with thousands of highly correlated observations may also have less independent evidence than the count suggests.

Test parameter stability

Do not report only the best parameter. Plot or tabulate the neighborhood around it.

If a 21-period setting works but 20 and 22 collapse, the edge may be noise. A broad plateau of acceptable outcomes is more credible than a sharp isolated peak. The same principle applies to stop distance, holding period and rebalance frequency.

Stress tests should also shift timestamps, remove the best trades, worsen fills and delay execution. The purpose is not to prove the strategy indestructible. It is to identify the assumption that carries the result.

Compare across regimes without cherry-picking

Segment results using rules defined independently of the strategy. Examples include high versus low volatility, trending versus ranging periods, and liquid versus illiquid assets.

Avoid creating a regime definition after seeing where the strategy failed. That simply moves overfitting from parameters into labels.

A robust idea may still have bad regimes. The important questions are whether those regimes can be identified with information available at the time and whether turning the strategy off creates a second layer of overfitting.

A Formion validation workflow

Keep the research record and the performance output together. In Formion, document the hypothesis, selected universe, data timestamps, fee model and rejected parameter variants before comparing the final scorecard.

Use the same scorecard for every candidate. Separate model approval from live execution: a passing historical result should enter forward paper testing before it can control real capital. If a required input is unknown, record it as unavailable rather than silently substituting zero cost.

A strong final report includes:

• strategy specification and version;
• data sources and known gaps;
• in-sample and out-of-sample boundaries;
• all cost assumptions;
• parameter search size;
• baseline and stress results;
• failure regimes;
• forward-testing status.

Primary reference

• CME Group: Backtesting

The best backtest is not the one with the highest historical return. It is the one whose assumptions are visible, whose test period remained untouched and whose edge survives realistic friction. That standard produces fewer approved strategies, which is exactly the point.