We judge probabilities on a warped but consistent scale. We overestimate the chance of low probability events and underestimate the probability of high probability events. This explains why we are willing to pay for lottery tickets while happily setting off on road trips. We can exploit this distorted world view in the world of investing because it consistently mis-prices some assets and boosts their long-term returns. Read on to understand which asset prices are most distorted by this phenomenon and why.
We Overestimate Low Probability Events
The odds of winning the UK Lotto are one in 45 million. Assuming you win and you have to travel to pick up your winnings be careful! That's because you are equally likely to die on the way as win in the first place if you:
- Cycle 0.6 miles
- Ride 0.2 miles on a motorbike
- Drive 8.7 miles in a car
Given these odds it is remarkable that 70% of the UK population take part in the lottery, and that is over 45 million people. The EuroMillions lottery has even longer odds of one in 117 million. Gambling companies generate a great deal of profit thanks to our natural tendency to overestimate low probabilities.
If we plot the "true" probability versus our "perceived" probability this distortion is clear. This was described by Kahneman and Tversky and led them to develop Prospect Theory. This theory attempts to explain how we make decisions and rests on two pillars, one of which is this probability weighting and the other is our loss aversion which leads to underinvestment.
Test your ability to judge probabilities by estimating the odds of these events.
Being born with an extra digit
Living to 100
Being killed in a car accident
Dying of cancer
How many people have to be in a room to have a 50% chance of a shared birthday?
Chance of being killed by terrorism
Probability Weighting & Asset Prices
Mentally boosting the probability of rare events means that investors misprice assets, but they misprice some more than others. Let's imagine two different assets. The one on the left gives a small, positive payoff almost all the time but once in while gives a bumper payoff. The one on the right pays off more most of the time but once in a while takes a huge loss. A rational investor would pay the same for both because the expected payoff is £2 for both assets.
Which would you prefer?
Positive Skew
- £1.01 with 99% probability
- £100 with 1% probability
Negative Skew
- £3 with 99% probability
- -£97 with 1% probability
Because of our warped sense of probability we tend to worry too much about the 1% probability outcome so we pay too much for the asset with a large positive payoff and too little for the one with a large loss. The asset with a large, low-probability upside payoff is called positively skewed, and the one with a large low-probability loss is called negatively skewed.
Nicholas Barberis and Ming Huang, authors of Stocks as Lotteries: The Implications of Probability Weighting for Security Prices, American Economic Review, Volume 98, No. 5, December 2008 (pp. 2066-2100)
Using this intuition two behavioural economists Nicholas Barberis and Ming Huang applied some reasonable assumptions, and some mathematical models, to derive the price distortion implied as they varied the skew of an asset's returns. They found that the more positively skewed an asset (more low-probability high returns) the lower the expected return of that asset. This is because the models overpaid for positive skew.
Applications of Warped Probability
The reason why this seemingly simple cognitive flaw is so powerful is that it explains so many phenomena, in particular why some assets generate higher long-term returns based on the skewness of their returns. Barberis and Huang point out that assets that are like lottery tickets, with a positive skew, are overpriced and tend to generate lower long-term returns as a result, for example:
- Initial Public Offerings (IPO): When a company decides to raise capital by issuing shares for the first time it is difficult to choose the initial price at which the share is offered. Will it be the next unicorn, going from success to success or, more likely, will it sink into obscurity. Because expectations are usually positive and have a low-probability large payoff we overestimate the new share's value and this means it will have a low return.
- Private Equity Returns: Some companies are held privately rather than having publicly traded shares. The owner could be a venture capital company funding a startup, or it could be a private equity company that buys ailing companies, turns them around by shaking up their business model and execution and then sells the company at a higher valuation. Again this is like a lottery ticket because such companies have the potential, albeit unlikely, to significantly increase the wealth of investors. However they tend to have returns that are not commensurate with their high risk.
- Undiversified Portfolios: Stocks held by undiversified investors have a greater skew than those held by diversified investors. It seems that investors willingly take an undiversified position in a skewed stock in order to add skewness to their portfolio.
So if it is the case that negatively skewed assets are consistently underpriced, how does the skewness compare across assets in practice?
Here is the skewness of a selection of global equity indices and commodities with positively skewed indices at the top in blue and negatively skewed indices at the bottom in red. What is perhaps surprising is that at the index level some share indices, such as the FTSE 250, have quite strong negative skew. Based on the work of Barberis and Huang the FTSE 250 should be underpriced relative to the FTSE 100 which has larger companies. This is consistent with the finding that over the long-term stocks of smaller companies outperform those of larger companies. High yield corporate bonds also tend to have negative skew because most of the time they produce steady, small positive returns. However when a credit crisis strikes the market it sells off very sharply and this bumps up the negative skew of returns. Credit indices should also be underpriced as a result.
Fixing Your Probability Sense
While it is true that history does not repeat it is the only guide we have for the future. So it pays to look at the past as a crude sketch for the future. A good way to start is to always consider the base probability. In other words don't start with "A is happening so B will happen", instead start with "how often has B happened?". For example many people say the following:
"Market valuations are high so markets will crash"
Instead the way to start gauging the probability of a market crash is to look at the number of times that the market has crashed historically, perhaps defined by some percentage fall from the latest peak. In other words how likely is a crash disregarding whether stocks are expensive? The reason why we do this is that it avoids a flaw called base rate neglect.
Let's apply this to the S&P 500. If we look at the number of times the index has fallen by 30% or more over a 12 month period using data going back to 1881, this has happened just 5% of the time based on monthly returns. The base rate of a crash is low given we don't know anything else.
Now let's consider valuation.
In this table we simply count how often things happen. Crashes happened in 82 months out of our sample of 1627 months, which is where our 5% probability comes from (82/1627 is 5.5%).
Now lets count how many times there was a crash in the year following months where valuation was high. We defined this as Robert Shiller's price to earnings stock valuation measure being above 27.3 which only occurs 10% of the time since 1881. The S&P 500 crashed in 8 of the 82 months when there was a high valuation so the odds of this is 10% (because 8/82 is 9.8%). This is twice the chance of a crash following months where valuations were not high which is 5% (81/1545 is 5.2%).
High valuations do increase the chance of a 30%+ fall in the S&P 500 but high valuation is not a reliable indicator of a crash.
Making such tables is easy and lets you estimate the chance of events co-occurring (it's called a Naive Bayes Classifier if you're into Machine Learning). But, crucially, this approach avoids base rate neglect. Let's imagine a world in which high valuation was a really good indicator of market crashes.
You can see that when a crash occurs it is preceded by high valuation 99% of the time (99 out of 100 crash months were preceded by high valuation). And only 1% of non-crash months were preceded by high valuation. So what's the chance of a crash given high valuation? It's just 99 out of 10098 which is 1%! That's because we've made crashes very rare - the base rate of crashes is just 1% and it dominates the calculation.
Don't neglect the base rate, always begin by finding out how often an event has happened in the past.
Get our Latest Updates
Subscribe to get updates on our latest educational videos and blogs, how all the global shares, bonds and commodity prices are reacting, and upcoming economic data releases and how we think they will drive markets
