One exciting opportunity popped up while I'm in Dublin, Ireland, next week. Best of all, it's free and open to the public. If you're in the western Europe and would like to discuss trading in person, I'd love to meet you.
Trinity College Dublin invited me to present a graduate level seminar to MSc students in Finance and Alternative Investments on Wednesday, February 8, at 6 pm. The seminar will be hosted in the MBA room, which is on the second floor of the business school. The topic will be high frequency market making in forex.
Topics for the high frequency forex trading seminar (about 10 minutes per subject):
Market making versus price taking
Comparing frequency to expectation. The more you trade, the more you make
Liquidity risks and self-feedback loops
Technical approaches and limitations
Friday, February 3, 2012
Backtesting for Efficiency
Your forex backtests are absolutely worthless if you do not test the statistical entry efficiency and exit efficiency of the strategy. Everyone that runs a backtest inevitably reports the dollars earned as the outcome. Other factors exist like the average win to loss, the profit factor and the Sharpe ratio, but they do not tell you anything useful until the final step of designing an automated trading system.
The correct approach to testing a strategy should focus on the question, "is my strategy a piece of garbage?" Most people try to prove themselves right. The real test is to not be able to prove yourself wrong. The only way to do that is through a statistical approach.
Entry and Exit Efficiency
Efficiency puts a hard number to what percentage of an available trading range that a strategy captures. The trading window starts on the bar where a trade entered the market. The window closes when the trade exits.
The total available window is the highest high minus the lowest low in the window. Calculating the entry and exit efficiency simply measures what percentage of that window that your strategy tends to capture. Take the average of all of the trades and you get the overall efficiency.
Entry efficiency formula
Formula for a long trade: (Highest high - entry price) ÷ (Highest high - lowest low)
Formula for a short trade: (entry price - lowest low) ÷ (Highest high - lowest low)
Exit efficiency formula
Formula for a long trade: (Exit price - lowest low) ÷ (Highest high - lowest low)
Formula for a short trade: (Highest high - exit price) ÷ (Highest high - lowest low)
Take an example where you buy a hypothetical currency at 150 and sell it at 170. The lowest low between the time of entry and exit was 140. The price then ran all the way up to 200 before settling back down to 170, which is where the exit took place.
The entry efficiency is (200-150) ÷ (200-140) = 50 ÷ 60 = 83%. Nearly anyone would agree that this makes for a great entry.
The exit efficiency is (170-140) ÷ (200-140) = 30 ÷ 60 = 50%. Most would agree that the exit would have ideally occurred sooner than it did.
Efficiencies do not change by instrument or time frame
One major problem that we encounter with forex backtests is the limited data set. This is especially true for those interested in testing long term strategies like those on the H4 or D1 charts. The wonderful thing about entry and exit efficiencies is that they do not vary from chart to chart or even period to period.
I like to jump down to M1 charts for efficiency testing. The data is nearly endless. I never have to worry about running out. The great thing is that I know when I shift back to the H4 chart, the efficiencies should not change more than ±5%.
If you see the efficiency vary too much, then you may not have enough trades to form a statistically significant group. My experience tells me that 75 trades usually gets very close to the actual efficiency. 100 trades or more is better. When I run tests on M1 charts, I often get several thousand trades over the course of a few months. Numbers that large can tell you with a great deal of confidence just how robust a strategy's parameters truly are.
Usually, you can assume that any results that fall within 45-55% are the result of a random, stochastic process. When I see backtests that creep right up to those barriers like 54.9% or even 55.1%, the results inevitably tank back to around the 50% mark.
Random trade outcomes and dollar profits
I wish this section was about how to make money with a random efficiency. Alas, we must cover how randomness can result in unjustified eurphoria.
I've been interested in the concept of randomness for several years now. Mathematicians refer to it with the more opaque name of a "stochastic process". Despite the non-sensical name, it's just a fancy way of saying the study of randomness - how it changes, its distribution, how far it "walks", etc.
Yesterday, I used the analogy of coin flips to describe how Martingale strategies are probabilistically doomed to failure. One interesting concept that I did not mention relates to Brownian motion. Even with a set of random outcomes, trades will go on a random walk away from the starting point.
Einstein gets the real credit for solving the math behind the concept, even though his name is not on the term. He demonstrated that the distance a random process will follow is the square root of the number of trials. If we decide to flip a coin 60 times, we know that 50% of the time should fall on heads and the other 30 on tails.
It actually turns out that we should expect a very slight bias in the number of either winners or losers, although we do not know which one. It's random. The precise bias, whichever way it prefers to go, should equal √60, which works out ~7. The heads outcomes should typically range from 23-37, with the tails outcomes making up the difference.
Seven trades out of sixty strongly alters the percentages, even if we know that it's really supposed to be 50%. If heads only came up 23 times out of 60, that's 38%. The problem is not with the coin. It's with the number of trials. As you do an increasing large number of trails, the random bias decreases in significance in terms of the percent accuracy. 50,000 trades, for example, should show a surplus of roughly 223 trades in favor of winning or losing. The accuracy range falls within 1% of 50% on either side, a dramatic improvement.
Risks of curve fitting
Curve fitting a random efficiency relates to the idea of Brownian motion. Let's say that we use a strategy that I know will never show an entry or exit efficiency: the moving average crossover. I've gone through this strategy six ways from Sunday, almost exclusively at the behest of clients. It does not work as a fully automated strategy. There is no secret set of fast and slow periods that will unlock the hidden keys to profit.
Most traders, experienced or not, abuse the backtester by searching for a set of parameters that yield the most dollar profit. They curve fit their test to optimize for maximum profit. What really happens is that the traders optimize for amount of random drift that already occurred.
When I used the example of 50,000 trades creating a natural drift of 223, I cited it with the purpose of showing how little it reduces the error in the real percent accurracy. The other consequence for trading systems is that as the error percentage decreases, the natural bias in your outcomes increases. Blindly running the optimizer only selects the set of combination that yields a combination of two criteria:
- The drift that happened to work out in favor of that set of parameters
- The profit and loss that varies with those parameters. The dollar profit naturally changes because the two moving averages cross at different points
You need a tool like efficiency to guard against these types of random outcomes. It's the only method that I know of that definitively states whether or not a strategy behaves in a random manner. I especially like the fact that it breaks those elements down into two of the three basic components of a trading strategy: the entry, the exit, and the position sizing.
Efficient strategies do not work all of the time
Position sizing marks the final obstacle to building your fully automated trading strategy. A set of rules that yields a statistically efficient entry that is paired with an efficient exit does not necessarily make money. The value of each trading setup can vary, too.
Each strategy contains different sets of winners and losers. Each winner and loser varies in its dollar value. Whatever money management approach that you take requires balancing the ratio of the winners and losers in a way that normalizes the outcome of each trade. You ideally want to eliminate the variation in dollar value. 20 pip trades should earn or lose you exactly as much as the 100 pip trades.
That seems counter-intuitive. Most traders want to win in proportion with the size of the opportunity. It's better from a system perspective to entirely ignore the size of the opportunity and to make each trade worth the same amount. Betting more or less with each trade effectively normalizes the value of each trade.
Using a stop loss stands out as an obvious candidate to fix how much a trade is worth. The severe disadvantage is that it almost always negatively affects the exit efficiency. Whenever I can get away with it, I always recommend using a market based exit instead of an arbitrary stop loss. Traders usually scream at the top of their lungs when they hear me say this. I'm just speaking as a systems developer. The numbers are what they are.
Monday, January 30, 2012
Martingale Strategy
Fear constitutes the strongest emotion facing any forex trader. Fear of losing, in particular, keeps many in bed with one eye on the monitor while the other rests. Martingale strategies theoretically remove that problem by ensuring that a strategy always comes out on top.
The idea is to bet more after a loss, usually twice the amount of a previous bet. Let's do an example where the value of winning and losing is identical. We either win $10 or we lose $10.
The first bet puts $10 at risk. We start out on a run of bad luck, resulting in a $10 loss. Now we double the risk. We either get paid $20 or will lose another $20. We lost again. So far we are down $30. We double the risk again, so now we're either going to make $40 or lose another $40.
So far the strategy doesn't seem that bad. 3 losses in a row only occurs 12.5% of the time (0.5 * 0.5 * 0.5). This is as bad as it gets the majority of the time. But, for the sake of the example, let's continue.
We lose again. Now we're down $70. Double the risk to either make or lose $80.
Another loss. The floating loss grows to $150. The risk grows to $160.
You're no doubt noticing that the risk numbers are getting a little more uncomfortable. For the sake of making it clear how quickly they can get out of hand, I created a small table below.
However, if tenth trade rescues the account, you earn back everything that you lost, plus the $10 that you originally attempted to earn. Although you sweat bullets during a cold streak, the strategy comes out on top the vast majority of the time.
Here is where the problems start to roll in. I did not say all of the time. In the real world, you do not have an infinite supply of money. If you have a $5,000 trading account and risk $10 on the first trade, then you can only withstand a maximum of 10 consecutive losses; you don't have enough money to make it to the 11th loss.
Consider a game of coin flips where heads means that you win and tails means that you lose. The odds are 1:1, 50%, that you will win or lose on any given flip. Although the probability of winning remains constant at 50%, the actual distribution of those wins and losses will come in the form of hot and cold streaks. Tossing the coin ten times does not always result in 5 winners and 5 losers.
Throwing a coin 10 times will result in 10 consecutive wins 0.09% (0.510)of the time. Conversely, 0.09% of the time also results in 10 consecutive losses. So, the coin flips results in behavior that looks anything like a random outcome 0.18% of the time. The scenario that keeps us up at night - fear in forex trading never goes away - is that that unlucky streak of 10 winners will actually come to pass. 0.09% of the time occurs once in every 1,111 trials.
The next question to consider is the number of wins required to make this worthwhile. Everyone's goal post is different, but it's safe to assume that doubling the account balance makes anyone a happy camper. Earning $10 for a total of $5,000 means that you must have a total of 500 winning Martingale sets. Each set is the group of trades required to earn that $10. Sometimes, the first trade wins and that finishes the set. Sometimes it takes 7 trades. Sometimes (gulp), it takes 20.
500 winning sets requires 1,000 total trades. We get this number by dividing the winning sets, 500, by the probability of winning, 0.5. If the closeness of that number to the scary threshold of 1,111 jumps out at you, it should.
Remember that the evil 10th consecutive losing trade could possibly occur at any point at or after 10 total trades. It could occur after 100 trades or, very rarely, after 10,000 trades. On the whole, it will occur 0.09% of the time, although it will randomly cluster. No method exists to predict where.
Think of the issue graphically. Our hurdle of 1,000 trades to double the account constitutes the danger zone, or the bad space. The safety zone is only the total trades at risk, 1,111, minus the benchmark of 1,000, which is a safety zone of 111.
As the sea of red danger zone relative to the green safety zone makes clear, the odds of coming out on top are not exactly in your favor. Specifically, you face a 90% chance of blowing up against a meager 10% chance of doubling your money.
Look for more fertile strategy strategy ideas and money management techniques. I like the simplicity behind Martingale, but that's about it. A simple mathematical analysis tells you that it's not going to end well the vast majority of the time. If you wish to continue a Martingale trading strategy, it only gets worse from this point forward.
The odds are not really 50%. We assumed that the price either goes up or down. We forgot to consider the spread. Pairs like the EURUSD show a typical spread of 1.0-1.5 pips at most reasonably priced brokerages. That, unfortunately, affects the probabilities. The price must move an additional 1.5 pips to reach the exit point.
A Martingale system trading a 10 pip interval now must earn 11.5 pips to cover the 10 pips it risks. The wins will only occur 46.51% of the time instead of the theoretical 50%.
Moving the size of the pip interval does help bring the probability closer to 50%. A 25 pip interval achieves 48.54% accuracy, while a 50 pip interval yields 49.46% accuracy. Going further and further up the ladder gets closer and close to 50%. It never quite gets there, though.
The other assumption that we made is that markets are akin to tossing a coin. Well, that's definitely not the case. Although markets exhibit a high degree of randomness, they do show periods of time where they are distinctly non random. They are, in fact, fractal. Mandelbrot enjoyed stumping on the fractal nature of markets at every opportunity. The assumptions that we made with the coin tossing analogy do not always apply. I strongly recommend reading The (Mis)behavior of Markets if you're interested in a layman friendly explanation of the topic.
The idea is to bet more after a loss, usually twice the amount of a previous bet. Let's do an example where the value of winning and losing is identical. We either win $10 or we lose $10.
The first bet puts $10 at risk. We start out on a run of bad luck, resulting in a $10 loss. Now we double the risk. We either get paid $20 or will lose another $20. We lost again. So far we are down $30. We double the risk again, so now we're either going to make $40 or lose another $40.
So far the strategy doesn't seem that bad. 3 losses in a row only occurs 12.5% of the time (0.5 * 0.5 * 0.5). This is as bad as it gets the majority of the time. But, for the sake of the example, let's continue.
We lose again. Now we're down $70. Double the risk to either make or lose $80.
Another loss. The floating loss grows to $150. The risk grows to $160.
You're no doubt noticing that the risk numbers are getting a little more uncomfortable. For the sake of making it clear how quickly they can get out of hand, I created a small table below.
| Consecutive losses | Floating loss |
| 6 | -$160 |
| 7 | -$320 |
| 8 | -$640 |
| 9 | -$1,280 |
| 10 | -$2,560 |
However, if tenth trade rescues the account, you earn back everything that you lost, plus the $10 that you originally attempted to earn. Although you sweat bullets during a cold streak, the strategy comes out on top the vast majority of the time.
Here is where the problems start to roll in. I did not say all of the time. In the real world, you do not have an infinite supply of money. If you have a $5,000 trading account and risk $10 on the first trade, then you can only withstand a maximum of 10 consecutive losses; you don't have enough money to make it to the 11th loss.
Measuring the payout of a Martingale strategy
Consider a game of coin flips where heads means that you win and tails means that you lose. The odds are 1:1, 50%, that you will win or lose on any given flip. Although the probability of winning remains constant at 50%, the actual distribution of those wins and losses will come in the form of hot and cold streaks. Tossing the coin ten times does not always result in 5 winners and 5 losers.
Throwing a coin 10 times will result in 10 consecutive wins 0.09% (0.510)of the time. Conversely, 0.09% of the time also results in 10 consecutive losses. So, the coin flips results in behavior that looks anything like a random outcome 0.18% of the time. The scenario that keeps us up at night - fear in forex trading never goes away - is that that unlucky streak of 10 winners will actually come to pass. 0.09% of the time occurs once in every 1,111 trials.
The next question to consider is the number of wins required to make this worthwhile. Everyone's goal post is different, but it's safe to assume that doubling the account balance makes anyone a happy camper. Earning $10 for a total of $5,000 means that you must have a total of 500 winning Martingale sets. Each set is the group of trades required to earn that $10. Sometimes, the first trade wins and that finishes the set. Sometimes it takes 7 trades. Sometimes (gulp), it takes 20.
500 winning sets requires 1,000 total trades. We get this number by dividing the winning sets, 500, by the probability of winning, 0.5. If the closeness of that number to the scary threshold of 1,111 jumps out at you, it should.
Remember that the evil 10th consecutive losing trade could possibly occur at any point at or after 10 total trades. It could occur after 100 trades or, very rarely, after 10,000 trades. On the whole, it will occur 0.09% of the time, although it will randomly cluster. No method exists to predict where.
Think of the issue graphically. Our hurdle of 1,000 trades to double the account constitutes the danger zone, or the bad space. The safety zone is only the total trades at risk, 1,111, minus the benchmark of 1,000, which is a safety zone of 111.
As the sea of red danger zone relative to the green safety zone makes clear, the odds of coming out on top are not exactly in your favor. Specifically, you face a 90% chance of blowing up against a meager 10% chance of doubling your money.
Look for more fertile strategy strategy ideas and money management techniques. I like the simplicity behind Martingale, but that's about it. A simple mathematical analysis tells you that it's not going to end well the vast majority of the time. If you wish to continue a Martingale trading strategy, it only gets worse from this point forward.
Other Martingale problems
The odds are not really 50%. We assumed that the price either goes up or down. We forgot to consider the spread. Pairs like the EURUSD show a typical spread of 1.0-1.5 pips at most reasonably priced brokerages. That, unfortunately, affects the probabilities. The price must move an additional 1.5 pips to reach the exit point.
A Martingale system trading a 10 pip interval now must earn 11.5 pips to cover the 10 pips it risks. The wins will only occur 46.51% of the time instead of the theoretical 50%.
Moving the size of the pip interval does help bring the probability closer to 50%. A 25 pip interval achieves 48.54% accuracy, while a 50 pip interval yields 49.46% accuracy. Going further and further up the ladder gets closer and close to 50%. It never quite gets there, though.
The other assumption that we made is that markets are akin to tossing a coin. Well, that's definitely not the case. Although markets exhibit a high degree of randomness, they do show periods of time where they are distinctly non random. They are, in fact, fractal. Mandelbrot enjoyed stumping on the fractal nature of markets at every opportunity. The assumptions that we made with the coin tossing analogy do not always apply. I strongly recommend reading The (Mis)behavior of Markets if you're interested in a layman friendly explanation of the topic.
Thursday, January 26, 2012
Robust Expert Advisors
When you want to program an Expert Advisor, it is important to consider how you are likely to use the file. The most common mistake that I see new MQL programmers make is that they rely too heavily on memory. Events like a power failure, a hard drive crash or weak internet connection can wreak havoc when trading with an Expert Advisor. The EA should ideally be able to pick up and run whenever you are able to restore conditions to normal.
Problems sometimes result from something mundane like changing the settings. I learned that the hard way when a client came to our Dallas office for two weeks to order a very complex EA. Everything worked perfectly in the backtester. The overnight tests, however, never lasted for more than a few hours before something went haywire. I instructed the client not to change the settings; just turn it on and leave it alone. For whatever reason, the importance of those instruction never sank in. It took me ages to realize that his definition of "leaving it alone" didn't quite mesh with mine. The little tweaks that he made during trading messed with the settings that I stored in memory.
I can instruct a client to use software in a certain manner. More often than not, they deviate from the instructions to some degree. It's human nature. I learned from those challenges, like any professional should. Our programming templates changed in structure to minimize the need for special instructions.
Global variables are a common tool to avoid the issue of relying on memory. When an EA is removed from the chart, the global variables that it created are accessible for you, the EA or any other EA to read from anywhere inside of MetaTrader.
Although the use global variables sounds sophisticated, it's actually a very simplistic approach. MetaTrader maintains a hidden text file in its installation directory with the variable name, the value that it is storing and the time it was last modified. Whenever more than 6 weeks elapsed since the last time MQL used the global variable, MetaTrader automatically deletes it.
A Martingale system might opt to use a global variable to track a simple statistic. Martingales are wild with their approach to risk, so a trader could realistically want to know at a glance the maximum level traded. Global variables make a trivial task out of that need. If the next level to trade is greater than the global variable, then the MQL code should update the global variable.
The only item to remember is that the global variables expire after six weeks. The EA might need to refresh the information by resetting the global value to its current value. Doing so would keep the variable fresh from MetaTrader's perspective.
I prefer to rely on MQL's OrderSelect() command to recreate whatever information I need. Although it is more complex, the advantage is that the EA can function any time, any place, exactly as designed. If the hard drive crashes, it's not a problem from a recovery perspective. You are still in trouble during the down time, though. You can load the same EA on your MT4 account and have it immediately reconstruct the information.
Taking the above Martingale scenario, we could reconstruct the number of levels traded by comparing the open and close times of trades. If trade #1 has a close time of 2011 Dec. 5 00:00 and trade #2 also has a close time of 2011 Dec. 5 00:00, it tells us that they are part of the same trade group. So far, we have 2 levels traded. If trade #5 closed at 2011 Dec. 5 00:00 but no trades did thereafter, then we know that we traded 5 levels total. The EA can then loop through all possible combinations and pick out the highest number from that.
The approach of reading from the order history is a bit of overkill for this simple example, but it comes in handy when you need to calculate more complicated information dynamically and with a minimal risk of error.
Memory issues
Problems sometimes result from something mundane like changing the settings. I learned that the hard way when a client came to our Dallas office for two weeks to order a very complex EA. Everything worked perfectly in the backtester. The overnight tests, however, never lasted for more than a few hours before something went haywire. I instructed the client not to change the settings; just turn it on and leave it alone. For whatever reason, the importance of those instruction never sank in. It took me ages to realize that his definition of "leaving it alone" didn't quite mesh with mine. The little tweaks that he made during trading messed with the settings that I stored in memory.
I can instruct a client to use software in a certain manner. More often than not, they deviate from the instructions to some degree. It's human nature. I learned from those challenges, like any professional should. Our programming templates changed in structure to minimize the need for special instructions.
Global variables
Global variables are a common tool to avoid the issue of relying on memory. When an EA is removed from the chart, the global variables that it created are accessible for you, the EA or any other EA to read from anywhere inside of MetaTrader.
Although the use global variables sounds sophisticated, it's actually a very simplistic approach. MetaTrader maintains a hidden text file in its installation directory with the variable name, the value that it is storing and the time it was last modified. Whenever more than 6 weeks elapsed since the last time MQL used the global variable, MetaTrader automatically deletes it.
A Martingale system might opt to use a global variable to track a simple statistic. Martingales are wild with their approach to risk, so a trader could realistically want to know at a glance the maximum level traded. Global variables make a trivial task out of that need. If the next level to trade is greater than the global variable, then the MQL code should update the global variable.
The only item to remember is that the global variables expire after six weeks. The EA might need to refresh the information by resetting the global value to its current value. Doing so would keep the variable fresh from MetaTrader's perspective.
Read information instead of storing it
I prefer to rely on MQL's OrderSelect() command to recreate whatever information I need. Although it is more complex, the advantage is that the EA can function any time, any place, exactly as designed. If the hard drive crashes, it's not a problem from a recovery perspective. You are still in trouble during the down time, though. You can load the same EA on your MT4 account and have it immediately reconstruct the information.
Taking the above Martingale scenario, we could reconstruct the number of levels traded by comparing the open and close times of trades. If trade #1 has a close time of 2011 Dec. 5 00:00 and trade #2 also has a close time of 2011 Dec. 5 00:00, it tells us that they are part of the same trade group. So far, we have 2 levels traded. If trade #5 closed at 2011 Dec. 5 00:00 but no trades did thereafter, then we know that we traded 5 levels total. The EA can then loop through all possible combinations and pick out the highest number from that.
The approach of reading from the order history is a bit of overkill for this simple example, but it comes in handy when you need to calculate more complicated information dynamically and with a minimal risk of error.
How Backtests Simulate Ticks
One common issue that comes up with backtestingis that it often does not match what happened in the real market. This isn't really a deficiency, at least from the perspective of your charting provider. It relates more to the data provided. Inaccurate historical backtests come up in any charting package, including NinjaTrader and MetaTrader.
Consider a typical bar. It has a high, low, open and close. That's only 4 data points. If you're looking at something like an H1 bar, there could be several thousand ticks - maybe tens of thousands of ticks - that comprise that entire bar. And you're trying to encapsulate it by only referencing 4 points.
MetaTrader brokers in particular do not want to provide tick data. The sheer volume of information would put a lot of strain on the servers. The brokers are the ones paying MetaQuotes for MT4. Their concerns go ahead of you as the trader. Thus, no tick data is available unless you go to a ton of extra effort.
That leaves the backtester guessing what should have happened at all of the points in between. Strategies that trade intrabar or EAs that place the stop and take profit so closely together that they might get hit on the same bar are particularly vulnerable to bad assumptions.
Example:
You placed an order to buy a forex pair at 100, which is also the open. You set the take profit at 105 and the stop loss at 97. The low of the bar is 96, the high of the bar is 110 and the close is 102. Is there any way to know whether the stop loss or take profit hit first?
No, you cannot tell. The price might have fell from 100 down to 96, climbed up to 110 and then fallen to 102. Alternatively, it may have climbed from 100 to 110, then fallen to 96, then climbed back to 102. More than likely, the price followed a complex path that vaguely resembles one of those paths.
MetaTrader and NinjaTrader have no way of knowing which answer is correct. They do their best to work around the gap in knowledge. NinjaTrader simply assumes the worst case scenario, which is probably the best policy to avoid false positives. If a stop or take profit both hit within a bar, then NinjaTrader assumes that the stop loss was hit first. There are no exceptions. The slightly good news is that if your NinjaTrader backtest looks abysmal and faces this issue, it may not be as bad as you first thought.
MetaTrader approaches the backtest assumptions differently. It tries to build the path of the bar by inferring from the relationship between the high and low and the open and close. If the bar closes up, I believe that MetaTrader assumes that the low comes first, then retraces up to the high of the bar before settling back down to the close. I don't have a theory on the algorithm that powers what happens in between; it's usually not relevant. If your stops and take profits are so close that the up/down bar assumption cannot sort it out, then no amount of backtesting will provide an accurate answer until you finally import tick data.
The spread at the time of entry, and thus the cost of trading, are also unknown. This can heavily distort the results. MetaTrader makes assumptions about the spread. Although it generally reflects the average spread, it's a safe bet that it does not reflect the actual spread at the time you traded. Whether or not the assumed spread works out in your favor over hundreds of trades varies from Expert Advisor to Expert Advisor.
NinjaTrader can work around the tick assumptions, but you must anticipate it within the strategy's programming. I plan to write a future blog post that will explain how to force NinjaTrader to use tick data for both the bid and ask to generate accurate backtests in NinjaTrader.
Consider a typical bar. It has a high, low, open and close. That's only 4 data points. If you're looking at something like an H1 bar, there could be several thousand ticks - maybe tens of thousands of ticks - that comprise that entire bar. And you're trying to encapsulate it by only referencing 4 points.
MetaTrader brokers in particular do not want to provide tick data. The sheer volume of information would put a lot of strain on the servers. The brokers are the ones paying MetaQuotes for MT4. Their concerns go ahead of you as the trader. Thus, no tick data is available unless you go to a ton of extra effort.
That leaves the backtester guessing what should have happened at all of the points in between. Strategies that trade intrabar or EAs that place the stop and take profit so closely together that they might get hit on the same bar are particularly vulnerable to bad assumptions.
Example:
You placed an order to buy a forex pair at 100, which is also the open. You set the take profit at 105 and the stop loss at 97. The low of the bar is 96, the high of the bar is 110 and the close is 102. Is there any way to know whether the stop loss or take profit hit first?
No, you cannot tell. The price might have fell from 100 down to 96, climbed up to 110 and then fallen to 102. Alternatively, it may have climbed from 100 to 110, then fallen to 96, then climbed back to 102. More than likely, the price followed a complex path that vaguely resembles one of those paths.
MetaTrader and NinjaTrader have no way of knowing which answer is correct. They do their best to work around the gap in knowledge. NinjaTrader simply assumes the worst case scenario, which is probably the best policy to avoid false positives. If a stop or take profit both hit within a bar, then NinjaTrader assumes that the stop loss was hit first. There are no exceptions. The slightly good news is that if your NinjaTrader backtest looks abysmal and faces this issue, it may not be as bad as you first thought.
MetaTrader approaches the backtest assumptions differently. It tries to build the path of the bar by inferring from the relationship between the high and low and the open and close. If the bar closes up, I believe that MetaTrader assumes that the low comes first, then retraces up to the high of the bar before settling back down to the close. I don't have a theory on the algorithm that powers what happens in between; it's usually not relevant. If your stops and take profits are so close that the up/down bar assumption cannot sort it out, then no amount of backtesting will provide an accurate answer until you finally import tick data.
The spread at the time of entry, and thus the cost of trading, are also unknown. This can heavily distort the results. MetaTrader makes assumptions about the spread. Although it generally reflects the average spread, it's a safe bet that it does not reflect the actual spread at the time you traded. Whether or not the assumed spread works out in your favor over hundreds of trades varies from Expert Advisor to Expert Advisor.
NinjaTrader can work around the tick assumptions, but you must anticipate it within the strategy's programming. I plan to write a future blog post that will explain how to force NinjaTrader to use tick data for both the bid and ask to generate accurate backtests in NinjaTrader.
Wednesday, January 25, 2012
Iran's Ahmadinejad ups rates to stem money crisis
http://www.reuters.com/article/2012/01/25/us-iran-economy-rates-idUSTRE80O0V520120125 (Reuters) - Iran increased bank interest rates on Wednesday and indicated it would further restrict sales of foreign currency, hoping to halt a spiraling currency crisis after new Western sanctions accelerated a dash for dollars by Iranians worried about their economic future.
"The economy minister has announced that (Iranian President Mahmoud) Ahmadinejad has agreed with the approval of the Money and Credit Council to increase interest rates on bank deposits to up to 21 percent," the official IRNA news agency reported.
Thursday, January 19, 2012
EES VPS for Forex Traders
Elite E Services offers Virtual Private Server for Forex traders, close to our EES FX Trading Environment (in the same data center) as well as a CNS bundling package for premium accounts.
To learn more, visit: http://eesfx.com/portal/trading/vps
See bundling info at CNS: https://cp.commercialnetworkservices.net/announcements.php?_m=news&_a=viewnews&newsid=31
http://helpdesk.commercialnetworkservices.net/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=195
EES also offers dedicated servers for professional traders at http://fxsystemhosting.com
To learn more, visit: http://eesfx.com/portal/trading/vps
See bundling info at CNS: https://cp.commercialnetworkservices.net/announcements.php?_m=news&_a=viewnews&newsid=31
http://helpdesk.commercialnetworkservices.net/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=195
EES also offers dedicated servers for professional traders at http://fxsystemhosting.com
Wednesday, January 18, 2012
Local currencies: 'In the U.S. we don't trust'
NEW YORK (CNNMoney) -- It may seem like Monopoly money to outsiders, but a growing number of communities across the U.S. are using homegrown local currencies to stimulate their economies and protect themselves from the nation's broader economic woes.
While there were only about 20 active community currencies in the United States in 2009, there has been a recent resurgence, with at least a dozen communities developing their own currencies in the past couple of years, estimates Loren Gatch, a professor of political science at the University of Central Oklahoma who researches these alternative currencies. In addition, currencies that have been around for years have seen a spike in interest, with membership doubling in some cases.
http://money.cnn.com/2012/01/17/pf/local_currency/index.htm
While there were only about 20 active community currencies in the United States in 2009, there has been a recent resurgence, with at least a dozen communities developing their own currencies in the past couple of years, estimates Loren Gatch, a professor of political science at the University of Central Oklahoma who researches these alternative currencies. In addition, currencies that have been around for years have seen a spike in interest, with membership doubling in some cases.
http://money.cnn.com/2012/01/17/pf/local_currency/index.htm
Friday, January 13, 2012
EES Article FX Trader Mag
See Elite E Services recent article in FX Trader Mag - registration is free and you can download online.
"Expert Advisors and Retail Forex"
Also EES recently was published in Stocks & Commodities magazine (January, 2012) "A Practical Guide to Forex Arbitrage" see an excerpt here: http://www.traders.com/index.php/sac-magazine/current-contents/forex-focus
"Expert Advisors and Retail Forex"
Also EES recently was published in Stocks & Commodities magazine (January, 2012) "A Practical Guide to Forex Arbitrage" see an excerpt here: http://www.traders.com/index.php/sac-magazine/current-contents/forex-focus
Tuesday, January 10, 2012
MetaTrader is more popular than NinjaTrader
For all my gripes about MetaTrader, I must admit that they do the most important stuff very well. It's very easy to download and install. Clicking on the desktop icon results in MT4 loading near instantaneously. When you log into the platform, charts immediately appear. The buttons at the top of the screen are obvious to anyone with trading experience. In short, it's about as idiot proof as software gets.
All of the things that MetaTrader does well, NinjaTrader does very poorly. The first time that I downloaded the 42 MB installation file, NinjaTrader said that I couldn't use it because I didn't have the .NET 3.5 framework installed. That makes sense to me as a programmer. To people that do not program for a living (i.e., the people reading this blog), that error message is next to worthless. NinjaTrader ought to handle this type of issue automatically.
Once I installed the program, I tried to open a chart. Nothing happened. I messed with it for about 5 minutes until I finally gave up and decided that the program wasn't worth the effort.
I decided to try again a few months later, at which point I realized that NinjaTrader did not include a datafeed. So then I had to go through their list of 50+ vendors to figure out where I might find a decent forex feed to try using their platform. The feed that I happened to have, Interactive Brokers, required that I download a special version of their software. Naturally, I did not figure this out until I got endless error messages and had to contact NinjaTrader suppport. An hour and several handfuls of hair later, I finally was able to do what MetaTrader allows me to do in seconds - look at a forex chart.
My Windows XP machine with 2 GB RAM, which is pretty typical of your average software user, takes about 20 seconds to load NinjaTrader. I have the nasty habit of closing and opening applications repeatedly. It gets on my nerves when I close NinjaTrader because it takes so long to get it back up again. I would consider leaving the application open, but I usually have 5-6 windows open within their platform. The taskbar always looks so cluttered. Call me crazy, but I find it very difficult to work on the computer with so many window tabs showing at the same time. It feels like there is too much going on visually.
Whenever you have a question about MetaTrader, you simply pick up the phone and call any MetaTrader broker. They answer your question immediately. NinjaTrader, in spite of its excellent online support, largely takes the help yourself approach. You can figure out most problems eventually, but it requires digging through forums and reading a few articles before you stumble upon the one with the answer. That type of self-help approach requires patience. Most people, especially us Americans that expect answers on the spot, are not blessed with that particular virtue.
MetaTrader is like Google Android and NinjaTrader is like Apple. MetaQuotes takes the Android style, hands-off approach. Anything that does not interfere with the MetaTrader itself is allowed. "Allow" isn't even the right word. MetaTrader is entirely removed from the process. The result is there are thousands of products available for sale that work exclusiely in MetaTrader. It's an accidental ecosystem for trading products that relates directly to the approach so prevalent in social media and mobile phones.
NinjaTrader takes the Apple, big brother approach; all products are screened and vetted. This does come with some advantages. You're a lot less likely to buy a total piece of garbage for NinjaTrader than you are for MetaTrader. That said, the centralized also stifles the products on offer. Maybe it's just because I'm Texan and resent anything that feels like authority making choices for me, but I think you see a lot more life and innovation when the people selling a related a product don't have to ask permission to do so.
You need to take a week's worth of online training to really get a sense for how to use the product. NinjaTrader, for some odd reason, does not offer YouTube videos or any friendly beginners guides to get up and running. They require that you attend online webinars, which are always scheduled in the middle of the work day, in order to thoroughly learn each of the product's main features.
When you open the software, it's not at all apparent what exactly it's supposed to do. Is it an automated trading platform? Is it a charting platform? Is it a tool for active traders? NinjaTrader does all of the above, but my opinion is that the software itself does a terrible job making this clear. Little things like a first-time walk after downloading through would help traders get over the steep learning curve and give the product a chance.
I actually love NinjaTrader, but it nearly takes a graduate degree to figure out how to use it. That's the only reason that I'm so hesistant to recommend it to my MetaTrader programming clients. Once you figure out how to use the tools, they are incredibly powerful. The built in trading analytics eliminate the need for MetaTrader related sites like MyFXBook and MT4Stats for viewing and studying my daily performance. Comparing MetaTrader and NinjaTrader in the backtester is not even worth doing: NinjaTrader is infinitely superior. It's too bad that NinjaTrader gets in its own way.
The features that NinjaTrader offers should make it way more popular than MetaTrader. I just don't see that happening any time soon with how long it takes to learn the software.
Fumbling through NinjaTrader
All of the things that MetaTrader does well, NinjaTrader does very poorly. The first time that I downloaded the 42 MB installation file, NinjaTrader said that I couldn't use it because I didn't have the .NET 3.5 framework installed. That makes sense to me as a programmer. To people that do not program for a living (i.e., the people reading this blog), that error message is next to worthless. NinjaTrader ought to handle this type of issue automatically.
Once I installed the program, I tried to open a chart. Nothing happened. I messed with it for about 5 minutes until I finally gave up and decided that the program wasn't worth the effort.
I decided to try again a few months later, at which point I realized that NinjaTrader did not include a datafeed. So then I had to go through their list of 50+ vendors to figure out where I might find a decent forex feed to try using their platform. The feed that I happened to have, Interactive Brokers, required that I download a special version of their software. Naturally, I did not figure this out until I got endless error messages and had to contact NinjaTrader suppport. An hour and several handfuls of hair later, I finally was able to do what MetaTrader allows me to do in seconds - look at a forex chart.
My Windows XP machine with 2 GB RAM, which is pretty typical of your average software user, takes about 20 seconds to load NinjaTrader. I have the nasty habit of closing and opening applications repeatedly. It gets on my nerves when I close NinjaTrader because it takes so long to get it back up again. I would consider leaving the application open, but I usually have 5-6 windows open within their platform. The taskbar always looks so cluttered. Call me crazy, but I find it very difficult to work on the computer with so many window tabs showing at the same time. It feels like there is too much going on visually.
Whenever you have a question about MetaTrader, you simply pick up the phone and call any MetaTrader broker. They answer your question immediately. NinjaTrader, in spite of its excellent online support, largely takes the help yourself approach. You can figure out most problems eventually, but it requires digging through forums and reading a few articles before you stumble upon the one with the answer. That type of self-help approach requires patience. Most people, especially us Americans that expect answers on the spot, are not blessed with that particular virtue.
Products and Add Ons
MetaTrader is like Google Android and NinjaTrader is like Apple. MetaQuotes takes the Android style, hands-off approach. Anything that does not interfere with the MetaTrader itself is allowed. "Allow" isn't even the right word. MetaTrader is entirely removed from the process. The result is there are thousands of products available for sale that work exclusiely in MetaTrader. It's an accidental ecosystem for trading products that relates directly to the approach so prevalent in social media and mobile phones.
NinjaTrader takes the Apple, big brother approach; all products are screened and vetted. This does come with some advantages. You're a lot less likely to buy a total piece of garbage for NinjaTrader than you are for MetaTrader. That said, the centralized also stifles the products on offer. Maybe it's just because I'm Texan and resent anything that feels like authority making choices for me, but I think you see a lot more life and innovation when the people selling a related a product don't have to ask permission to do so.
What does Ninjatrader do, anyway?
You need to take a week's worth of online training to really get a sense for how to use the product. NinjaTrader, for some odd reason, does not offer YouTube videos or any friendly beginners guides to get up and running. They require that you attend online webinars, which are always scheduled in the middle of the work day, in order to thoroughly learn each of the product's main features.
When you open the software, it's not at all apparent what exactly it's supposed to do. Is it an automated trading platform? Is it a charting platform? Is it a tool for active traders? NinjaTrader does all of the above, but my opinion is that the software itself does a terrible job making this clear. Little things like a first-time walk after downloading through would help traders get over the steep learning curve and give the product a chance.
I actually love NinjaTrader, but it nearly takes a graduate degree to figure out how to use it. That's the only reason that I'm so hesistant to recommend it to my MetaTrader programming clients. Once you figure out how to use the tools, they are incredibly powerful. The built in trading analytics eliminate the need for MetaTrader related sites like MyFXBook and MT4Stats for viewing and studying my daily performance. Comparing MetaTrader and NinjaTrader in the backtester is not even worth doing: NinjaTrader is infinitely superior. It's too bad that NinjaTrader gets in its own way.
The features that NinjaTrader offers should make it way more popular than MetaTrader. I just don't see that happening any time soon with how long it takes to learn the software.
Forex Correlation
Correlation strategies appeal to forex traders because removes the stress associated with picking market direction. When two correlated pairs diverge from one another, the idea is to simply buy one pair and sell the other.
Correlation offers a mathematical probability of two "time series" moving in the same direction. Applying the idea to forex, it means that we need to pick two currency pairs. EURUSD and USDCHF are two popular choices due to their extremely high correlation, so we'll use those.
Now we ask a simple question: "If the EURUSD rises, what is the probability of that the USDCHF will also rise"? Our calculations will pump out a simple a number between -1 and +1. +1 means that that if Currency A rose in value, then it is 100% certain that Currency B rose in value. -1 means that if Currency A increased in value, then it is 100% certain that Currency B decreased in value. A value of 0 means that the movement of Currency A exercises no effect at all on Currency B.
Traders generally consider a correlation significant whenever the number is greater than 70%. EURUSD and USDCHF are so popular because they hold the strongest correlation among the major currency pairs. When market volatility was very low a few years ago, it was around -93%. Today, the correlation tends to hang around -80%. The European debt problems and Swiss National Bank's intervention have a lot to do with the decrease in this number. Their trading relationships are far less stable.
Let's move back into familiar territory with my favorite example, the moving average. If you take the average over the past 20 bars, you know from experience that the average will differ if you study a 50 period versus a 200 period average. If you look at the average on a 5 minute chart versus an hourly chart, the number will vary yet again.
The take-away here is that the correlations work the same way. The correlation between EURUSD and USDCHF might even be positive if you look at a short enough time scale. As you back away in time, you will notice that the further out you go, the more steady the correlation numbers look. If the weekly correlation of the EURUSD and USDCHF is -80%, you would expect the numbers to get more wild and erratic as you scale all the way down to a tick chart.
The same problem with the moving average also appears. Studying the correlation over 50 periods provides a responsive number, but it is also far less consistent than the 200 period correlation. What a short period gains in responsiveness, it loses in stability.
You should also consider whether the correlation that you're studying makes fundamental sense. Just because the temperature change in Mongolia predicted the direction of USDJPY for the past week does not make it a good idea to use in the future. The same goes with pair trading.
EURUSD and USDCHF should be highly correlated for two reasons. They both contain the same currency in the pair (USD), which half weights them with the same instrument. Additionally, the EUR and CHF both have strong trading relationships with the US. You would expect both the Euro Zone and Switzerland to share a need for buying and selling US dollars. They need them for buying oil, importing and exporting to the US, etc. Anyone with a cursory understanding of macroeconomics could explain why this relationship makes sense.
Correlation traders typically settle on pairs that share a common currency. The EURUSD and USDCHF trade both share the US dollar. When you buy EURUSD and buy USDCHF, you are really:
Buying EUR andselling USD
Buying USD and selling CHF
Notice that the USD cancels itself out. What you are really doing is buying EUR and selling CHF. This is commonly known as the EURCHF pair. Assuming that the spread is not outrageous, it makes more sense to simply buy or sell EURCHF directly rather than going through the convoluted process of managing two open trades.
If you decide to pursue the two pair approach in a MetaTrader expert advisor, you must consider the need to balance the trade sizes against each other. Using standard lots as the example, 100,000 EUR is 137,500 USD. 100,000 USD is 90,900 CHF. If you buy one standard lot of EURUSD, you are buying $137,500 of it. When you buy a standard lot of USDCHF, you are only buying $100,000.
$137,500 obviously does not equal $100,000. Unless you intentionally decided to trade different sizes, you may want to consider equalizing them.
Solve for x:
€100,000 / $137,500 = x * (₣90,900/$100,000)
x = €100,000 / ₣90,00 * $100,000 / $137,500 = 0.803
You would need your EURUSD trade to be 80% of the size of the USDCHF trade.
Correlation only provides insight into the probability of direction. It says absolutely nothing about the strength of a particular move. A few months ago the USDCHF climbed 1,000 points in value within a single day. The EURUSD only moved a few hundred pips. The USDCHF moved dramatically further than the EURUSD both in terms of pips, but more importantly, as a percentage of price.
Consider if you were short EURUSD that day and short USDCHF. You lost a ton of money. On the flip side, if you were long EURUSD and long USDCHF, then you got lucky and earned the move. Regardless of what happened, correlation told you nothing about the outcome when they move in the same direction. For that reason, I prefer looking at a less intuitive method called cointegration.
Conintegration turns the problem on its head. Rather than asking whether or not two pairs move in the same direction, it asks how likely are they to remain a certain distance apart. Naturally, that distance tends to vary with time. What you want the cointegration formula to tell you is how likely two pairs are to come back to a standard distance. If you see two pairs spread unusually far apart and the numbers tell you that they usually come back together, then it makes sense to consider a pair trade.
Ernest Chan has a friendly introduction to cointegration that I highly recommend. A much uglier, math intensive introduction to the subject, albeit one that is also far more thorough, is in the book Pairs Trading by Ganapathy Vidyamurthy.
What are correlated currency pairs?
Correlation offers a mathematical probability of two "time series" moving in the same direction. Applying the idea to forex, it means that we need to pick two currency pairs. EURUSD and USDCHF are two popular choices due to their extremely high correlation, so we'll use those.
Now we ask a simple question: "If the EURUSD rises, what is the probability of that the USDCHF will also rise"? Our calculations will pump out a simple a number between -1 and +1. +1 means that that if Currency A rose in value, then it is 100% certain that Currency B rose in value. -1 means that if Currency A increased in value, then it is 100% certain that Currency B decreased in value. A value of 0 means that the movement of Currency A exercises no effect at all on Currency B.
Traders generally consider a correlation significant whenever the number is greater than 70%. EURUSD and USDCHF are so popular because they hold the strongest correlation among the major currency pairs. When market volatility was very low a few years ago, it was around -93%. Today, the correlation tends to hang around -80%. The European debt problems and Swiss National Bank's intervention have a lot to do with the decrease in this number. Their trading relationships are far less stable.
Risks of correlation strategies
Let's move back into familiar territory with my favorite example, the moving average. If you take the average over the past 20 bars, you know from experience that the average will differ if you study a 50 period versus a 200 period average. If you look at the average on a 5 minute chart versus an hourly chart, the number will vary yet again.
The take-away here is that the correlations work the same way. The correlation between EURUSD and USDCHF might even be positive if you look at a short enough time scale. As you back away in time, you will notice that the further out you go, the more steady the correlation numbers look. If the weekly correlation of the EURUSD and USDCHF is -80%, you would expect the numbers to get more wild and erratic as you scale all the way down to a tick chart.
The same problem with the moving average also appears. Studying the correlation over 50 periods provides a responsive number, but it is also far less consistent than the 200 period correlation. What a short period gains in responsiveness, it loses in stability.
You should also consider whether the correlation that you're studying makes fundamental sense. Just because the temperature change in Mongolia predicted the direction of USDJPY for the past week does not make it a good idea to use in the future. The same goes with pair trading.
EURUSD and USDCHF should be highly correlated for two reasons. They both contain the same currency in the pair (USD), which half weights them with the same instrument. Additionally, the EUR and CHF both have strong trading relationships with the US. You would expect both the Euro Zone and Switzerland to share a need for buying and selling US dollars. They need them for buying oil, importing and exporting to the US, etc. Anyone with a cursory understanding of macroeconomics could explain why this relationship makes sense.
Correlation traders typically settle on pairs that share a common currency. The EURUSD and USDCHF trade both share the US dollar. When you buy EURUSD and buy USDCHF, you are really:
Buying EUR and
Notice that the USD cancels itself out. What you are really doing is buying EUR and selling CHF. This is commonly known as the EURCHF pair. Assuming that the spread is not outrageous, it makes more sense to simply buy or sell EURCHF directly rather than going through the convoluted process of managing two open trades.
If you decide to pursue the two pair approach in a MetaTrader expert advisor, you must consider the need to balance the trade sizes against each other. Using standard lots as the example, 100,000 EUR is 137,500 USD. 100,000 USD is 90,900 CHF. If you buy one standard lot of EURUSD, you are buying $137,500 of it. When you buy a standard lot of USDCHF, you are only buying $100,000.
$137,500 obviously does not equal $100,000. Unless you intentionally decided to trade different sizes, you may want to consider equalizing them.
Solve for x:
€100,000 / $137,500 = x * (₣90,900/$100,000)
x = €100,000 / ₣90,00 * $100,000 / $137,500 = 0.803
You would need your EURUSD trade to be 80% of the size of the USDCHF trade.
What correlation is not
Correlation only provides insight into the probability of direction. It says absolutely nothing about the strength of a particular move. A few months ago the USDCHF climbed 1,000 points in value within a single day. The EURUSD only moved a few hundred pips. The USDCHF moved dramatically further than the EURUSD both in terms of pips, but more importantly, as a percentage of price.
Consider if you were short EURUSD that day and short USDCHF. You lost a ton of money. On the flip side, if you were long EURUSD and long USDCHF, then you got lucky and earned the move. Regardless of what happened, correlation told you nothing about the outcome when they move in the same direction. For that reason, I prefer looking at a less intuitive method called cointegration.
Cointegration
Conintegration turns the problem on its head. Rather than asking whether or not two pairs move in the same direction, it asks how likely are they to remain a certain distance apart. Naturally, that distance tends to vary with time. What you want the cointegration formula to tell you is how likely two pairs are to come back to a standard distance. If you see two pairs spread unusually far apart and the numbers tell you that they usually come back together, then it makes sense to consider a pair trade.
Ernest Chan has a friendly introduction to cointegration that I highly recommend. A much uglier, math intensive introduction to the subject, albeit one that is also far more thorough, is in the book Pairs Trading by Ganapathy Vidyamurthy.
Wednesday, December 28, 2011
Euro drops to near 10 year low
http://www.businessweek.com/news/2011-12-28/euro-drops-to-almost-10-year-low-versus-yen-on-ecb-loans-concern.html
Dec. 28 (Bloomberg) -- The euro dropped against the yen to almost the lowest level since 2001 as the European Central Bank’s balance sheet soared to a record after it lent regional banks more money last week to keep credit flowing.
The 17-nation currency fell against the dollar to the lowest level since January as concern increased that the region’s sovereign-debt crisis will curb growth, even as rates fell at an Italian bill sale. The dollar gained as stocks dropped, boosting demand for haven assets. The yen gained on a safety bid after a U.S. Treasury report criticized Japan for intervening in the currency market and as economic reports signaled slowing economic growth.
S&P 500 Snaps 5-Day Rally Amid Europe Concern
http://www.bloomberg.com/news/2011-12-28/u-s-stock-index-futures-decline-before-italian-debt-auction.html
U.S. stocks declined, halting a five- day advance in the Standard & Poor’s 500 Index, as the European Central Bank’s balance sheet increased to a record after a surge of bank lending to stem the region’s debt crisis.
All 10 groups in the S&P 500 fell as measures of commodity, financial and industrial shares slid at least 1.4 percent. Alcoa (AA) Inc. andCaterpillar Inc. (CAT) dropped more than 1.9 percent. Bank of America Corp. (BAC) sank 2.9 percent, extending yesterday’s slump.
The S&P 500 lost 1.1 percent to 1,252.03 at 11:33 a.m. New York time, after rallying 5 percent over the previous five days. The Dow Jones Industrial Average fell 120.41 points, or 1 percent, to 12,170.94. About 1.5 billion shares changed hands on U.S. exchanges, or 35 percent below the same time a week ago.
Wednesday, December 21, 2011
Monday, December 19, 2011
Repainting Indicator
Repainting indicators are one of the most common problems facing would-be EA designers. Sometimes it seems like an indicator really has its thumb on the market. It perfectly predicts when the market will go up or down. Making an EA run off of the indicator is a no-brainer.
If this sounds familiar, then you need to consider whether or not the indicator repaints.
Now you move on to October 26. You look back at October 25, 24, and so on, until you get the final RSI value for October 26. All of the examples move forward into the future, then look back into the past to get the current value.
We gather data from October 26, 27, and so on. Now, having learned the future, we jump back to October 25 and pretend that we knew the future all along. The RSI looks amazing.
I receive copies of the 3_Level_ZZ_Semafor every once in a while and it tends to cause problems. It's based on the same idea as the zig zag indicator, which helps it to pick perfect tops and bottoms. Trend traders love looking at it. The information seems like it would be so valuable.
Making an EA off of it, however, presents numerous problems. The trades that the Expert Advisor makes do not match up with what the indicator looks like a few hours later. What actually happens is that the indicator changes its mind. It doesn't leave behind any evidence of its incorrect prediction. The novice EA designer naturally concludes that the programming is all wrong and that we need to fix it.
The first image below shows the value of the semafor on a EURSUD M5 chart at 00:00 on October 10, 2011. You'll notice that a purple dot with the number 2 appears on that bar. The market is moving strongly upwards. The indicator at this point predicts that the move is likely overextended.
If we jump 2 bars into the future at 00:10, now magically a number 3 dot appears. What's worse is that the number 2 dot disappears entirely from the 00:00 bar! The indicator sees that the price jumped up, so it goes back in time and deletes the evidence of its incorrect prediction. If you weren't watching the indicator or taking screenshots, it is as though the number 2 dot was never there.
I like to do this by selecting an Expert Advisor without caring too much about which one I pick. If you don't have any installed, you can grab one from the free Expert Advisor page for the sake of testing.
You'll need to select settings to get the backtester to run. The options that you choose are not important, so long as you have selected something. I highlighted the required items in red below. I personally prefer to use M5 EURUSD charts. M5 is good because the backtester moves over the data quickly on full speed without going so fast that I cannot control the process. It strikes a nice balance between speed and following what's actually happening. I like EURUSD simply because I always have EURUSD data loaded. I don't have to worry about the tester not working, whereas that would likely be an issue if I selected an exotic currency like TRY/ZAR.
When the visual backtest chart pops up, the first thing to do is to load your indicator onto the chart. As the test progresses, watch how the indicator behaves. Some repainting indicators like the Semafor are completely obvious. They repaint on most bars, so it's easy to tell that it is not consistent. You can safely assume that it repaints.
Other indicators are more tricky. They only plot once every 50 bars and sometimes even less frequently. You have to watch closely to make sure that where plots appear, that they don't subsequently dance onto another bar. I like to use the same technique in the first screenshot that I posted with a vertical red line. Whenever the indicator plots something on the chart, mark it with a vertical line. Continue doing so for about a day's worth of data on the M5 chart. Stop the test, then scroll back through the chart. If 100% of the plotted values match the vertical lines, then your indicator does not repaint.
If this sounds familiar, then you need to consider whether or not the indicator repaints.
What is a repainting indicator?
First consider a baseball analogy. Last week, the St. Louis Cardinals beat the Texas Rangers to win the World Series. A repainting indicator is like asking you today, "Would you bet on the Cardinals or Rangers?," then pretending like you placed your bet 3 weeks ago and not today. You had no way of knowing for certain who would actually win.How normal indicators work
If that sounds stupid, that's because it is. Repainting indicators use knowledge of the "historical future" to "predict" which way the market was going to go. Indicators typically start with old bars on the left side of the chart, then walk forward to the right side until they obtain enough information to make a decision. Say, for example, that you want to create the RSI indicator on a daily chart and you want to start on October 25. You start at October 25, and make decisions based on information from Oct. 24, 23, etc., until you've collected 14 bars worth of data or however long you set the period. You then say definitively that the RSI on October 25 is X.Now you move on to October 26. You look back at October 25, 24, and so on, until you get the final RSI value for October 26. All of the examples move forward into the future, then look back into the past to get the current value.
A hypothetical repainting indicator
Now let's make a hypothetical RSI indicator that repaints. Say our first date under consideration is October 25. We might start by looking into the past, which is what we did last time. But for the sake of making the example obvious, let's look the other way - into the future.We gather data from October 26, 27, and so on. Now, having learned the future, we jump back to October 25 and pretend that we knew the future all along. The RSI looks amazing.
A Real Example
I receive copies of the 3_Level_ZZ_Semafor every once in a while and it tends to cause problems. It's based on the same idea as the zig zag indicator, which helps it to pick perfect tops and bottoms. Trend traders love looking at it. The information seems like it would be so valuable.
Making an EA off of it, however, presents numerous problems. The trades that the Expert Advisor makes do not match up with what the indicator looks like a few hours later. What actually happens is that the indicator changes its mind. It doesn't leave behind any evidence of its incorrect prediction. The novice EA designer naturally concludes that the programming is all wrong and that we need to fix it.
The first image below shows the value of the semafor on a EURSUD M5 chart at 00:00 on October 10, 2011. You'll notice that a purple dot with the number 2 appears on that bar. The market is moving strongly upwards. The indicator at this point predicts that the move is likely overextended.
If we jump 2 bars into the future at 00:10, now magically a number 3 dot appears. What's worse is that the number 2 dot disappears entirely from the 00:00 bar! The indicator sees that the price jumped up, so it goes back in time and deletes the evidence of its incorrect prediction. If you weren't watching the indicator or taking screenshots, it is as though the number 2 dot was never there.
Test your indicator and look for repainting
No automated process exists to determine whether or not your indicator repaints. It's something that you have to do by staring at the chart during a backtest to see if the values change or not.I like to do this by selecting an Expert Advisor without caring too much about which one I pick. If you don't have any installed, you can grab one from the free Expert Advisor page for the sake of testing.
You'll need to select settings to get the backtester to run. The options that you choose are not important, so long as you have selected something. I highlighted the required items in red below. I personally prefer to use M5 EURUSD charts. M5 is good because the backtester moves over the data quickly on full speed without going so fast that I cannot control the process. It strikes a nice balance between speed and following what's actually happening. I like EURUSD simply because I always have EURUSD data loaded. I don't have to worry about the tester not working, whereas that would likely be an issue if I selected an exotic currency like TRY/ZAR.
When the visual backtest chart pops up, the first thing to do is to load your indicator onto the chart. As the test progresses, watch how the indicator behaves. Some repainting indicators like the Semafor are completely obvious. They repaint on most bars, so it's easy to tell that it is not consistent. You can safely assume that it repaints.
Other indicators are more tricky. They only plot once every 50 bars and sometimes even less frequently. You have to watch closely to make sure that where plots appear, that they don't subsequently dance onto another bar. I like to use the same technique in the first screenshot that I posted with a vertical red line. Whenever the indicator plots something on the chart, mark it with a vertical line. Continue doing so for about a day's worth of data on the M5 chart. Stop the test, then scroll back through the chart. If 100% of the plotted values match the vertical lines, then your indicator does not repaint.
Subscribe to:
Posts (Atom)