Are shares priced based on rational expectations of the future? And do shares immediately adjust their price to incorporate all available information i.e. are markets "efficient"? Most people would have answered "yes" to both questions until a revolutionary publication by the economist Robert Shiller in 1981 completely changed our view of how markets are priced and set the stage for the new field of behavioural economics.

Shiller asked a simple question: why do stock markets move up and down so much? In finance the size of these up and down movements is called volatility and we measure it as a percentage price change per year.

Shiller laid out the four factors that investors could use to drive fluctuations in their estimation of the "fair" price of shares:

1. ​Changing rational forecast of future cash flows. We buy shares to get our hands on future earnings either directly as a cash payment (a fraction of earnings are paid to shareholders as a dividend) or indirectly via an increase in the share price.
2. Changing rational forecast of future interest rates. If we can earn a return risk-free then the risky return on shares less valuable. As risk-free interest rates rise this pushes down the price people are willing to pay for shares.
3. Changing rational forecast of future risk. If shares are going to become more risky then their value will decrease.
4. Price of risk. As investor appetite for risk decreases the price paid for shares falls.

Shiller set the problem up by calculating what the rational price would have been if markets had correctly guessed future changes in dividends. In this figure from his paper he calls this "rational" price p*. In the graph you can see the actual price of the S&P share index as a solid line and the rational price as a dotted line. The actual stock price wobbles about much more than the rational price.

Shiller went on to cross the mathematical t's and dot the statistical i's. But what is clear from this picture is that we can cross off item 1 from our list of rational drivers of share prices. The sedate variation in dividends simply cannot explain wild share price gyrations. Shiller gives the most catastrophic market crash ever as an example of market overreaction:

In later work Shiller and others would go on to look at each of the remaining rational drivers of market prices; future interest rates and future risk. It turned out that the huge share price fluctuations that we observe cannot be explained by rational forecasts of future interest rates or risk.

By a process of elimination the primary driver of market volatility is risk appetite.

This assertion by Shiller caused a storm in the academic world because it showed that markets are not all-knowing or even forward-looking. Instead they are driven to a very large extent by emotion. This was a problem for economic models which never took this into account. The result was that Shiller revitalised a whole new field of research into behavioural economics and behavioural finance which has flourished ever since.

What does this mean for investors?

Firstly let's remember this doesn't mean that markets are not rational at all. Eventually prices come back into line with earnings. However, fluctuations in risk appetite can keep this from happening for long periods of time. In turn this means that markets can remain too expensive or too cheap for years.

Unfortunately this research provides almost no help with market timing over the space of a few years, but it can help over the space of a decade. In his book Irrational Exuberance, Shiller includes this graph of market valuation today (along the bottom axis) vs. the return on stock market in the following decade (left-hand axis).

When valuation is very high, when people are willing to pay above $25 to $30 for each dollar of earnings, the return over the next decade has tended to be poor. And when people pay $15 or less for each dollar of earnings return over the next decade has usually been much higher.

Markets are predictable over long periods of time.

If markets are driven by this mysterious force called risk appetite, what does that actually mean? A rough interpretation would be excessive pessimism or optimism. Shiller offers this definition:

"Irrational exuberance is the psychological basis of a speculative bubble. I define a speculative bubble as a situation in which news of price increases spurs investor enthusiasm, which spreads by psychological contagion from person to person, in the process amplifying stories that might justify the price increases and bringing in a larger and larger  class of investors, who, despite doubts about the real value of an investment, are drawn to it partly through envy of others' successes and partly through a gambler's excitement."

More recent research has come up with two explanations for the fluctuations in risk appetite. One explanation is rational and the other is not.

Habit Formation (Rational)

John Campbell and John Cochrane came up with the Habit Formation Hypothesis. The maths behind this is really elegant and John Cochrane's book "Asset Pricing" goes through it in detail but Cochrane summarised it as follows:

"As consumption rises, you slowly get used to the higher level of consumption. Then, as consumption declines relative to the level you’ve gotten used to, it hurts more than the same level did back when you were rising. As I once overheard a hedge-fund manager’s wife say at a cocktail party, 'I’d sooner die than fly commercial again.'"

A key factor in the model is that we adjust slowly to higher levels of consumption, as can be seen in this stylised sketch of the model by Cochrane:

The model shows that at the top of a market boom prices, (P) as measured relative to dividends (D) seem too high. Consumption (C) is high relative to a subsistence level (X), and risk aversion is low. Despite the representative investor knowing that expected return (ER) will be low from these price levels their thought process is along these lines:

"I know prices are so high that expected returns will be low. But my job's secure and I'm earning good money, so I can afford to take some risk, and what else could I do with the money? I'm going to reach for yield."

Everything does an about face during a financial crisis. The internal narrative of our representative investor now runs as follows:

"I know prices are now low so expected returns will be high. But I'm about to lose my job, my car is about to be repossessed, and what if the market carries on falling instead of rebounding? I can't take any risks right now."

By design this model can explain the excess volatility in markets via this slowly varying consumption creep amplifying rallies and crashes. And it can also explain how over the long-term the ratio of prices to dividends can be used to forecast returns.

Overextrapolation (Irrational)

Nicholas Barberis defines overextrapolation as follows: "when forming beliefs about the future, people put too much weight on the recent past". In fact for many problems where numbers vary smoothly like forecasting the average daily temperature, this approach works quite well, but not in financial markets where there are often rapid reversals.

"Recently good corporate earnings have been very strong, share prices have been rising as a result. ​Why shouldn't this continue in future?"

A poll of expected stock returns by individuals and institutional investors (pension funds, hedge funds, insurance companies) shows that this does indeed seem to be the case, which lends weight to this theory and cannot be explained so easily by the Habit Formation hypothesis.

Animal Spirits

Robert Shiller co-authored another great pop economics book with George Akerlof which was published in 2009 just as the world was emerging from the Global Financial Crisis. They say that the most important driver of the economy and financial markets is the old idea of Animal Spirits. It really is an old idea. Here it is described by the economist William Wood in 1722 as the driving force behind Britain's hugely successful global trade.

Shiller and Akerlof develop the idea of animal spirits to explain economic booms and busts. They lay out five ideas that, they claim, explain how the economy works:

1. Confidence. Feedback mechanisms between consumers and the economy amplify booms and busts.
2. Fairness. The level of wages, which falls during busts and rises during booms, is concerned with fairness.
3. Corrupt and anti-social behaviour. When there are profits people are tempted to break rules, or even do away with regulation to lay hands on them.
4. Money Illusion. Inflation is extremely important to investors over the long-term but is usually ignored.
5. Stories. Normally we think about who we are and what we do in terms of anecdotes. Our individual stories can be aggregated into a national and international story which itself drives the economy. Stories spread via contagion like epidemics e.g. Bitcoin is soaring in value, houses will always be a safe investment, tech stocks will soar in value, markets are about to crash...

They describe how many market and economic phenomena can be explained with these five ideas, such as why we get economic depressions, the power of central banks, reasons for unemployment and its relationship with inflation, the volatility of markets (which we discussed above), why house prices go through cycles, and inter-generational poverty in minorities.

It is ironic that these two authors, both of whom have devoted their careers to building mathematical economic models, are describing things which don't fit easily into a spreadsheet. This may be why economic forecasts are so poor: to truly forecast you would need to model human irrationality which is beyond our current technology.

Using Animal Spirits to Invest

Animal Spirits can lead to speculative frenzy or complete capitulation and despair in markets. Given where we are in the current environment we will use the example of a bubble, but most of these indicators can simply be reversed for markets during a trough. Detecting that you are in a bubble is a little bit like telling that you are drunk. You can do it but it takes practice and brutal honesty. But there are some warning signs:

• Fear Of Missing Out. People are buying (houses/shares/...) as if they could not buy them tomorrow because prices are going up so quickly. You hear daily stories about someone that became a millionaire from buying X.
• Remember inflation. If you hear stories about spectacular long-term returns consider what the return would be if you subtract the rate of inflation during the lifetime of the investment.
• News Story Epidemics. Avoid anecdotes, look at the big picture. If mainstream and social media are being swamped with stories about property, or a certain type of share (tech, biotechnology...), or a digital currency then it might be time to step back and consider the true value of those investments.
• "Sure Winner". Story epidemics during the final stages of a bubble often have this key phrase, which to any investor is self-contradictory as there can be no return without risk.
• Value models are irrelevant. When dot com valuations went through the roof for companies that hadn't generated a penny of profit this was said to be irrelevant. The future was so bright that the small matter of no profit was irrelevant. Similarly when shares were at their lows in February 2009 even companies whose business model was strong were discounted with the rest because valuation was irrelevant.