How good is a trading system? How can you compare the performance of two different systems and make a judgement as to which you would rather trade? Originally published in “Traders Magazine”, in this article I explore the various different ways of assessing the merit of a trading strategy.
Strategy Development Process
There are many different software platforms available which offer the ability to develop and test trading strategies on historic data and many traders are out there working away trying to develop a winning system that will make them rich. Whilst the software functionality makes the process of testing a strategy relatively straight-forward many have found to their cost it is not as easy as it looks to translate these virtual test results into actual hard cash. Many a trader has come unstuck when his system, which performs wonderfully on historic data, falls flat on its face when trading with real-money. This leads to an important question: how do you measure how good your system actually is?
Typically when developing a system there is an iterative process which involves coding up an idea and seeing how it performs and then assessing that performance according to some figure of merit. The system is then adjusted or often parameters are optimised to achieve the best figure of merit for the test data. When system developers work on systems they often assume that the only figure of merit worth looking at is the net profits that are generated by a system and when optimising over a range of parameters they will often use this figure alone to select the optimal parameter set. However this can be a far from ideal approach to system development and can often be misleading. For example the system could have one huge winning trade and the rest are losers but still come out as the best profit figure. Whilst this system might have generated the highest total return over the test period it is relying on the one huge trade to make up for everything else and the circumstances which lead to that trade may have been unusual. This means that the test results won’t be very reliable. It is therefore important to look at a number of the strategy performance figures when assessing how good a system is.
Percentage Profit
There is an equation for the average profit per trade of any trading strategy which (before costs) is:
Avg Profit Per Trade = Pwin x Wavg – Plose x Lavg
where
Pwin = percentage winners
Wavg = average win size
Plose = percentage loses
Lavg = average lose size
At break-even this average profit per trade is zero, so re-arranging this and using the fact that Plose=1-Pwin gives us:
Wavg/Lavg = (1-Pwin)/Pwin (Equation 1)
This represents the relationship between the respective average trade sizes and the percentage winners for a break-even system before costs. It is always amazing that system vendors (usually the less reputable ones) advertise systems as 90% winners in S&P’s etc. From the above equation it can be seen that just because a system has 90% winners there is no guarantee that it is profitable. For example a naked out of the money option writing strategy might have a 90% win rate but the losses when they occur will be many times the average premium collected and the strategy could still be a net loser. Without knowing the relative sizes of the wins and losses as well it is not possible to assess whether the system is net profitable or not on the basis of percentage profitability.
Whilst a high percentage win isn’t necessarily a pre-requisite to a profitable system, it does give an indication of how frequently you might expect profits to come along. It’s no good if you develop a long-term trend-following system if you are a short-term scalper at heart so this figure is important in giving you an idea of what your system is like. It helps to think of what sort of system you are developing: a long-term trend following system might for example only be profitable 20% of the time but the average winning trade is much larger than the average losing trade. Using our equation above we can see that for the system to break-even you need Wavg/Lavg to be 4, that is the average winner needs to be 4 times the average loser for the system to break even with this win percentage.
Case Study 1
Recently a client asked me to look at a set of parameters for a system that he was developing. He had chosen the parameter values based on optimisation of the total profit. When I looked at the performance figures the percentage profitability was only 7%! The optimisation process had resulted in the stops being incredibly tight which meant that the position sizing algorithm selected very large sizes to trade. This meant that most of the time he was being stopped out and in fact there were only three winning trades in the whole test but they had made very large profits compared to the numerous losses. Whilst this might still be a viable system it is important to think what it’s going to be like to trade this method. Day after day of losses waiting for that occasional big win – and what happens if you happen not to take that trade? That one trade makes up for all the losses and without it you have a losing strategy. This means that the system will psychologically be difficult to trade and also the test results are not very reliable because of the small number of winning trades.
The Number of Trades
This figure is not important in it’s own right except that too few trades means that the performance metrics of the system are less reliable. It also gives you an idea of what it will feel like to trade the system: will you be waiting for long periods for a trade to be generated or will you be constantly in and out of the market?
Case Study 2
A client had done an optimisation and wanted to know why the optimal parameters that he was getting were different from the ones that I suggested. I looked at his optimisation results from which he had simply picked the best Net Profit and noticed that his parameter set had only traded once during the entire test period and had held the position open for many months on what was supposed to be a short-term swing trading model. It is always a good idea to check that the trade results are sensible and what you expect from your model.
Average Profit Per Trade
This is always an important figure to look at because of trading costs. Equation 1 above did not include trading costs but these are an important part of the reality of being a trader. When testing you should always include realistic estimates of total trading costs including commissions and slippage. The average profit per trade after costs tells you how much profit there is left after you have paid all the costs. If this is rather small then it is important that you make sure that your trading cost estimate are correct. For example if you estimate that the total costs are 1.5 ticks for your system and your average profit per trade before costs is only 2.0 ticks per trade then you have better hope that your 1.5 tick estimate is correct because if the real costs are 2.0 ticks per trade then you won’t be making any money. This also tells you that your costs are 75% of your gross profits which is very high. For a high frequency trading strategy this could still be viable but you might want to work on reducing your costs in order to increase profitability.
Maximum Drawdown
Drawdowns are always an important figure to look at when assessing a system. They are a somewhat tricky figure to assess as they depend on a run of consecutive losing trades in order to create the drawdown which means that the order in which the trades occur makes a difference to the size of the drawdown. Now a run of losing trades can occur by pure chance just as when tossing a coin it is possible for it to come down as head eight times in a row, say. Alternatively a drawdown can occur because the prevailing market conditions at the time of the drawdown were not suited to the method. It is always useful to have a look at your system to try to work out whether a drawdown was generated by chance or because of the prevailing market conditions.
It is often useful to look at the ratio between the drawdown and the annualised profit return. For example the annualised return might be 20% and the maximum drawdown might be 15%. This is a ratio of 1.33 which is a reasonably healthy one. Anything over 2 is good, over 1 is ok, below 1 and your annual profits are starting to look a little small compared to the drawdown figure.
Drawdowns also have an impact on the initial account size: if you experience the maximum drawdown when you start trading a system you are obviously going to need at least the maximum drawdown plus margins to trade the system. However due to the uncertain nature of drawdowns there is always the possibility that the next drawdown will be even larger than the maximum one that occurred during your back-testing. You want to consider how large a drawdown you or your investing clients could stand before pulling the plug on the system. It is an unfortunate fact that investors tend to bail out of a system during a drawdown just before it turns round and tend to invest after a good run up just before it starts to draw down. It is therefore a good idea to keep the drawdowns to a manageable size even though this will mean trading smaller size and therefore reducing the profit potential. A professional fund manager for example might scale his trading size for a system down so that the maximum drawdown is only 5% if that is all he feels that his clients could stand.
It is also worth looking at the duration of a drawdown: a drawdown that lasts a year is pretty tough psychologically as well as meaning no income (from performance fees or from trading profits) for a whole year. How well could you handle such as situation? It’s one thing to look at an equity curve over 15 years with a good general up trend and to see a six month drawdown half way through this period in this context. It’s another thing actually to be in the middle of it wondering when or even if it will ever end. It’s at times like these that you can start to doubt your system. Therefore it is always worth studying your system’s drawdowns carefully.
Annualised Return
This is an often-overlooked performance figure but one which is very important as it tells you how much money you are going to make each year using it. If you are looking to make a living from a system you need to know whether it is going to make sufficient profits for you to live off. Professional money managers for example would are happy with a return of 20% with a maximum drawdown figure of 5% but if you have only 10,000 Euros or Pounds to trade then, unless your living standards are pretty modest, you are probably not going to be able to live off the profits that such a system would generate. Individual traders tend to tolerate larger drawdowns in order to gain increased profits but you still need to do this calculation and to check that you cal tolerate the larger drawdowns that go with larger profit potential.
Trade Distribution
When looking at the trade distribution you are looking for this to be fairly even. You don’t in general want all the profits to come from one or two huge trades though of course for some long-term trend following methods this may in fact be the way that that the system works. The more widely distributed the trades are the more volatile the equity curve is going to be. In fact when we come to look at the Sharpe ratio we’ll see that in general the lower the standard deviation of the trade results the better the system is.
The trade distribution will tend to vary according to the nature of the system anyway: a short term scalper will tend to have lots of trades all the same size whereas a long term trend follower is looking for a few huge winners to offset the numerous relatively small losses.
Sharpe Ratio
This is a measure that is particularly used by fund managers. It is a measure of the amount of profit that is being generated relative to the risk that is being taken.
Sharpe Ratio = Rann / SDann
where
Rann = Adjusted Annualised Return
SDann = Annualised Standard Deviation of returns
The Adjusted Annualised Return is meant to be the annual return over and above the risk-free rate of return, the idea being that you would otherwise have your capital in the bank earning risk-free interest. Whether you need to subtract off the risk-free interest rate from your rate of return depends on what instrument you are trading: for highly leveraged ones such as futures you are only tying up a small amount of your capital to the trade so you can still be earning interest on the bulk of your account whereas for an unleveraged stock position you are losing the interest that you would otherwise earn and so have to subtract it off from the return. In general to keep it simple and since I tend to work mostly with leveraged instruments I leave off the risk-free interest rate subtraction.
The annualised SD of returns is the standard deviation of the return figures multiplied by a scaling factor to annualise it. The scaling factor will be explained below.
The calculation of the Sharpe Ratio is best explained by way of a simple example. Consider the monthly returns of a hypothetical trading system over two years below:
Month Year 1 Year 2
Jan 2.45% -0.91%
Feb 0.84% 1.23%
Mar 1.20% -0.24%
Apr -1.34% 0.89%
May 0.67% -0.28%
Jun 0.80% 0.44%
Jul -0.20% -0.61%
Aug 3.20% 1.16%
Sep -0.59% 1.74%
Oct 0.65% -0.41%
Nov 1.18% -0.68%
Dec 1.35% 3.10%
Using standard spreadsheet functionality one can determine that the average monthly return is 0.65% and the standard deviation of the returns is 1.21%. To annualised the return you simply multiply the average monthly return by 12. To annualise the standard deviation of the monthly returns you multiply it by the square root of 12. This gives annualised figures of 7.81% and 4.18% respectively. Thus the Sharpe Ratio comes out to be 7.81 / 4.18 = 1.87 in this case which is a pretty reasonable figure.
You can probably see that in calculating the Sharpe Ratio the numerator has the scaling factor (12 in this case) and the denominator has the square root of the scaling factor which means that the formula can be written as:
SR = squareroot( Scaling Factor ) * Ravg / SDret
Where
Ravg = average of returns
SDret = standard deviation of returns.
For monthly returns the scaling factor is 12, for weekly it is 52 and for daily it is around 260. You will get slightly different answers for your Sharpe Ratio calculations according to your sampling interval. The standard that is usually used amongst fund managers is the monthly returns.
The point of the Sharpe Ratio is that it measures performance relative to risk. If you take the system results in the table above and leverage them all up by 10:1 then the annualised return goes up to 78.1% but the annualised standard deviation will go up by the same factor of 10 so the Sharpe Ratio will come out the same. This makes it an extremely useful figure for assessing relative performance figures when different leverage is being used.
A very good Sharpe Ratio is anything over 2 but anything over 1 is worth looking at and 1.5 is a reasonable figure. This is a very useful figure of merit and one which is well worth calculation. Performing optimisations using the Sharpe Ratio can often give better parameter selection than other methods.
K Ratio
This ratio was developed by Lars Kestner. It is based on the simple idea that the best equity curve is one which goes up as a straight-line with as steep a slope as possible. Therefore the best measure of a trading system is how closely it follows the ideal. A straight line fit is first made to the equity curve and the slope of this line is measured. Next the deviation of the equity curve from the straight-line fit is evaluated: usually the standard error of the straight-line fit is used. The ratio is then given by:
K-Ratio = Seqc / ( Eslp * Square Root (Np) )
Seqc = Slope of Equity Curve
Eslp = standard error in slope from straight-line fit
Np = number of point in equity curve.
Again this is best illustrated by way of an example. We will take the same returns as in the previous table and use them to construct an equity curve starting with 100,000 in whatever currency you choose:
Month Year1 Year2
Jan 102,450.00 109,595.18
Feb 103,306.46 110,942.01
Mar 104,548.96 110,671.60
Apr 103,149.93 111,656.57
May 103,842.94 111,344.48
Jun 104,669.91 111,834.40
Jul 104,456.87 111,148.68
Aug 107,799.49 112,438.00
Sep 107,159.13 114,394.42
Oct 107,859.80 113,929.28
Nov 109,130.45 113,154.18
Dec 110,599.76 116,661.96
Next because the equity curve is compounded (each 1% gain is applied to the previous equity value not just the original 100,000) we need to take the log of the equity value. Otherwise the ideal equity curve is going to resemble an exponentially increasing curve rather than a straight line. Note however that if the equity curve is uncompounded then there is no need to take logs first. Next we fit a straight-line to this log equity curve. In Excel this is easily done using the LINEST function. This will give you the slope of the curve which is 0.002221 and the error in this slope value which is 0.000118. The K-Ratio is then given by:
K-Ratio = 0.002221 / ( 0.000118 x sqrt(24) ) = 3.84
which is a very good value. From the plot below you can see that the curve looks good and the straight-line fit closely matches it which is why the K-Ratio value is so good.
This is another indicator which is a good general performance figure and again which can be used for optimisations.
Visual Inspection
We’ve looked a variety of different ways of assessing a system’s performance including some more sophisticated methods such as the Sharpe and K- Ratios. However, one of the best ways of assessing a system quickly is simply to look at the shape of the equity curve. A quick glance at this tells you a lot about the system and what it is like. Does the curve slope smoothly upwards or are there prolonged periods of drawdown? Are there a few large spikes upwards indicating that a few huge winners are making up the bulk of the profits? Are there periods of prolonged sideways movement and if so to what period in the market do they correspond? This can often tell you something about the nature of the system and what its weaknesses are. It is for this reason that the equity curve plot is the first thing that I look at for any system.
Spreadsheet and Equity Curve Model
To help you get an understanding of some of these metrics there is a spreadsheet available for download here. Here you can see the details of the calculations of the Sharpe Ratio and the K-Ratio.
In addition to the above calculations, in the spreadsheet there is a formula for generating hypothetical equity curves using the model:
Current Return = Previous Return * ( 1 + Trend + Noise * (0.5-RAND))
where Trend and Noise and user-input parameters for the equity curve model and RAND is a random number between 0 and 1 so 0.5-RAND is a random number between +0.5 and –0.5. You can vary the Trend and Noise parameters and see what sort of equity curves are generated (they are plotted in a chart) and look at the Sharpe and K Ratio values and other performance metrics. From this you can see that the larger the noise factor compared to the trend factor the more variable the performance is and the lower the performance metrics. Nevertheless it is surprising that even with a low trend factor for some of these essentially random equity curves they look good and have good figures. This is always worth bearing in mind least we look back at a good period of trading and conclude that we must indeed be expert traders – it could all be random!
Conclusions
Unfortunately there is not single measure of system performance that can be used to give an indication of how good a system is. Instead the developer needs to go through a variety of different measures in order to get a feel for the strengths and weaknesses of a system. We have seen how the Sharpe Ratio gives a good measure of the returns for a given amount of risk and how the K-Ratio gives a good assessment of the overall equity curve shape. It is important to take time when assessing a system, to understand why it is behaving in the way that it is so that one can incorporate this information into improving it still further. With this insight you can develop better trading systems.