One common issue that comes up with backtestingis that it often does not match what happened in the real market. This isn't really a deficiency, at least from the perspective of your charting provider. It relates more to the data provided. Inaccurate historical backtests come up in any charting package, including NinjaTrader and MetaTrader.
Consider a typical bar. It has a high, low, open and close. That's only 4 data points. If you're looking at something like an H1 bar, there could be several thousand ticks - maybe tens of thousands of ticks - that comprise that entire bar. And you're trying to encapsulate it by only referencing 4 points.
MetaTrader brokers in particular do not want to provide tick data. The sheer volume of information would put a lot of strain on the servers. The brokers are the ones paying MetaQuotes for MT4. Their concerns go ahead of you as the trader. Thus, no tick data is available unless you go to a ton of extra effort.
That leaves the backtester guessing what should have happened at all of the points in between. Strategies that trade intrabar or EAs that place the stop and take profit so closely together that they might get hit on the same bar are particularly vulnerable to bad assumptions.
Example:
You placed an order to buy a forex pair at 100, which is also the open. You set the take profit at 105 and the stop loss at 97. The low of the bar is 96, the high of the bar is 110 and the close is 102. Is there any way to know whether the stop loss or take profit hit first?
No, you cannot tell. The price might have fell from 100 down to 96, climbed up to 110 and then fallen to 102. Alternatively, it may have climbed from 100 to 110, then fallen to 96, then climbed back to 102. More than likely, the price followed a complex path that vaguely resembles one of those paths.
MetaTrader and NinjaTrader have no way of knowing which answer is correct. They do their best to work around the gap in knowledge. NinjaTrader simply assumes the worst case scenario, which is probably the best policy to avoid false positives. If a stop or take profit both hit within a bar, then NinjaTrader assumes that the stop loss was hit first. There are no exceptions. The slightly good news is that if your NinjaTrader backtest looks abysmal and faces this issue, it may not be as bad as you first thought.
MetaTrader approaches the backtest assumptions differently. It tries to build the path of the bar by inferring from the relationship between the high and low and the open and close. If the bar closes up, I believe that MetaTrader assumes that the low comes first, then retraces up to the high of the bar before settling back down to the close. I don't have a theory on the algorithm that powers what happens in between; it's usually not relevant. If your stops and take profits are so close that the up/down bar assumption cannot sort it out, then no amount of backtesting will provide an accurate answer until you finally import tick data.
The spread at the time of entry, and thus the cost of trading, are also unknown. This can heavily distort the results. MetaTrader makes assumptions about the spread. Although it generally reflects the average spread, it's a safe bet that it does not reflect the actual spread at the time you traded. Whether or not the assumed spread works out in your favor over hundreds of trades varies from Expert Advisor to Expert Advisor.
NinjaTrader can work around the tick assumptions, but you must anticipate it within the strategy's programming. I plan to write a future blog post that will explain how to force NinjaTrader to use tick data for both the bid and ask to generate accurate backtests in NinjaTrader.
