amibroker

From time to time we receive questions about why you can get different results back-testing the same code.

There are many reasons for differences in backtest results:

1. Different data

   For example if past history is updated/changed due to splits for example or backfill or external data update.

2. Different settings / parameters

   May happen if your formula uses Param() functions that output values that may be changed from the Parameter window.

   Note that Parameters in Analysis window use shared ChartID=0, which means that if two formulas use same parameter names they would share parameter values as well.

3. Different formula

   Sometimes even slight change to the formula causes big change in the results, for example if your formula uses #include and included code has changed

4. The formula that self-references its previously generated results.

   Such code produces some data that is later used to produce next run output (for example your code produces composites that are later used – if those composites change – the input data change so the results change, or if your formula uses previous backtest equity or if your formula uses STATIC variables without deleting/clearing them at the beginning)

5. The formula uses random number generator

   This occurs when your formula directly or indirectly calls any function that produces random numbers such as Random or mtRandom

6. The formula reads external files/data that might have changed

   This occurs when your formula directly or indirectly calls any functions that read external files or data that have changed between backtests. It can be as trivial as using current time in your formula.

7. The formula sets global (per-backtest) settings to non-constant value – for example sets global option differently for each symbol.

   This one is subtle and can be easily overlooked by non-experienced users. One has to understand that global (per-backtest) settings are applied to entire portfolio. If one changes those settings on per-symbol basis, then “last write” counts, and it means that result of your backtest will depend on which execution thread ended last. In Windows OS, it is generally unpredicatable which thread ends first because of so many factors that affect execution time. So your backtest may use different settings in different runs as a result of this. Bottom line: never change global settings in your formula to non-constant value.

Filed by Tomasz Janeczko at 6:01 am under Problems
Comments Off on Why do backtest results change?