ICR302: Professor David Hand, The Improbability Principle
Why is it that incredibly unlikely things actually happen quite regularly and why should we, in fact, expect such things to happen?
My guest on the podcast today is Professor David Hand and we’re talking about his book, The Improbability Principle.
If you’re interested in coincidences, probability or gambling, this conversation with eminent statistician David Hand is especially for you.
David has a long and impressive bio. He’s Senior Research Investigator and Emeritus Professor of Mathematics at Imperial College, London.
He’s also Chief Scientific Advisor to Winton Group, which is where we connected.
David is a Fellow of the British Academy, and an Honorary Fellow of the Institute of Actuaries, and has served (twice) as President of the Royal Statistical Society.
David is a prolific academic, having published 300 scientific papers and 29 books, including Principles of Data Mining, Information Generation, Measurement Theory and Practice, The Wellbeing of Nations, and The Improbability Principle, the subject of our conversation today.
He’s recipient of the Guy Medal of the Royal Statistical Society, the George Box Medal and, in 2013, was made OBE for services to research and innovation.
Here’s my conversation with Professor David Hand, author of The Improbability Principle, in episode 302 of Informed Choice Radio.
Subscribe in Apple Podcasts | Click to listen now | Right click to download episode
Useful links
–Professor David Hand at Imperial College London
–The Improbability Principle: Why coincidences, miracles and rare events happen all the time
Interview transcript
Martin: Well, I’m delighted to welcome Professor David Hand to the podcast today. David, you are Chief Scientific Advisor to Winton Capital Management. Could you start by telling us a bit about that role and what it involves?
David: Sure. I think, to answer that question, I have to say a little bit about Winton. The point is that although Winton works in a financial sector, it’s very much a scientific and data science driven company. It’s all about analysing real world data to inform its investment decisions.
I sometimes caricature this as saying it doesn’t have a bunch of people hanging around, putting their finger in the air and saying, “Well, in my opinion, the market’s going to move this way or that way.” Instead, what we do is actually look at what’s really going on in the world and base our predictions on solid scientific and statistical analysis of data from the world.
That really tells you what my job is. What I do, my role is to talk to the researchers about what they’re doing, discuss any methodological problems they’ve encountered and make sure they’re up to date with cutting edge statistical machine learning and artificial intelligence sorts, so that’s my role to talk to the researchers about the kinds of analysis they’re doing and how they’re doing it.
Martin: I’ve got a copy of your biography in front of me and it’s fair to say I think you are a prolific academic. You’ve published 300 scientific papers, 28 books, and I’m particularly interested in your book: The Improbability Principle. Tell us, why is it that unlikely things happen quite regularly?
David: The improbability principle is all about, as you say, highly unlikely events coincidences and so on. Basically, the principle is a combination of five mathematical laws and they describe the way chance and probability behave. I think of those five laws as analogists and Newton’s three laws of motion or the four laws of thermodynamics and so on.
Each of those five laws show, in different ways, why unlikely things happen far more often than we’d expect. Let me give you just one example of one of them and that will answer your question. The law of truly large numbers, and I think this is one, which people will be most familiar with in a sense. The law of truly large numbers says that if you give a highly improbable event enough opportunities to happen, then it almost certain to occur. For example, the national lottery gives us … Well, I like working with lotteries, talking about lotteries when I’m talking about the improbability principle because it strips away all ambiguities and uncertainties in human language and so on. We’re actually talking about something concrete, so let’s take that as an example.
The UK National Lottery is a 6/59 lottery, which means each ticket consists of six numbers chosen from 59. If you do the maths on that that shows you that there’s about a one in 45,000,000 chance that any one ticket will win. That’s right. If you have 24 pennies and you threw more up in the air that it will come down heads, so it’s a very small chance.
Here’s the point and here’s where the law of truly large numbers comes in. Every week, millions and millions of people buy lottery tickets, so although each one of them has got an absolutely tiny chance of occurring; if there are enough of them, if enough millions of them is quite likely that one of them will win and so you do get people winning the jackpot quite frequently. In fact, you can take this further. You can apply to it at yourself as an individual. If you buy one ticket a week and you do that for long enough; you give yourself enough opportunities to win, then you’re almost certain to win eventually just because the tiny probability has been multiplied by the number of opportunities.
However, a cautionary note here. I wouldn’t recommend that strategy because you’d expect to have to play for … Buy one ticket a week for nearly a million years before you’d win, so I wouldn’t recommend that as an investment strategy.
Martin: I’ve also heard it said that you shouldn’t buy your national lottery ticket until after about 4 o’clock on a Saturday afternoon because if you buy it beforehand there’s more chance of you being knocked down and killed by a bus than there is you would end up at the jackpot.
David: Yes, and that shows you just how tiny the chance of winning the jackpot is.
Martin: How did you come across this improbability principle? Is it widely known in your field or is this something that you’ve developed yourself?
David: I think some of the laws of the improbability principle are well-known, others less subtle, at least, in the way I phrase them. I think I came across it … I’m a statistician; I’ve been working with chance and probability all my life. I just, like everybody else I think, I found these intriguing coincidences totally intriguing. The apparent contradictions that can sometimes occur with probability, I found fascinating so I decided to focus some research attention on this. This question of gradually realising over many years that you could capture the occurrence of these coincidences and rare events by its five laws.
Martin: You mentioned the law of truly large numbers as it relates to the national lottery. Are there other examples of how this improbability principle occurs in everyday life maybe when we look at things and think, “Wow, that was a coincidence.” Is that the improbability principle occurring?
David: That’s exactly right. It occurs all the time, and once your awareness of these things has been raised, you see even more coincidences like this. Let me just give you a couple of examples: lotteries, yeah fine, let me talk about something else.
When my book came out, I was contacted by a man named John Ironmonger. He’s a novelist and he had just had a book published as well. His book was called The Coincidence Authority. It was about a London based academic expert in coincidences whose birthday was the 30th of June, which is my birthday. A total coincidence. He said he wasn’t stalking me; it was just a total coincidence.
Here’s another little example not a personal one for me but one, which I think is very nice, and you can describe it why it happens in terms, and then you can describe why all of these happen in terms of the improbability principle with this one. The King James Bible was published in the year that Shakespeare turned 46. Now, if you go to Psalms 46 of that bible, you’ll find that the 46th word from the start is Shake and the 46th word from the end is Spear. Now, what I love about that is you look at that and you see, “Wow, that’s incredible,” and that’s not a coincidence. Surely, it must have been done deliberately but I’m certain it was a coincidence.
The point is that if you look for that kind of thing long enough in enough books, in enough published material, you’re bound to find that sort of thing. What’s particularly amusing about that, I think, is that somebody actually went through all that effort.
Martin: Absolutely. Going back to your example of John Ironmonger’s book, I’m assuming, again, this is back to relatively large numbers, probably millions of fiction books being published, some of them will about statisticians, improbability and coincidences, and there’s only a limited number of birth dates to choose from.
David: That’s exactly right. One of my other laws is the law of inevitability and this says if you’ve got a finite possible number of outcomes, one of them must occur. For instance and I can relate these lotteries: one other lottery, the set of six lottery numbers is bound to come up. The UK National Lottery, there are only 45 million of them. If I bought all 45 million lottery tickets, I would be guaranteed to win the jackpot. It is certain it will occur. It’s just like there are only 365 birthdays in a year.
Martin: I understand in Australia actually some years back, there was a weekend where the jackpot was such that a team of people realised if they bought sufficient number of tickets, they were more or less guaranteed a profit because of the numbers involved.
David: That’s exactly right. Some lotteries … In the UK lottery, the jackpot rolls over. Actually, it’s changed a bit now, but some lotteries, now, if the jackpot rollover enough what they do is stop it rolling over anymore and distribute the money amongst the lower number of matches: amongst five matches and so to six matches and so on. If you do that, sometimes, exactly what these Australians did, you can guarantee winning more than you invest.
Martin: Does this all take away some of the magic, house of miracles, so called miracles and coincidences? We’re assuming that the law of inevitability, the law of truly large numbers, this isn’t the illuminati at work; this is magic. This is just naturally going to occur.
David: That’s a really good question and is one I’m often asked. When I encounter a coincidence, people say, “Where do you go?” Well, you’d expect it happen, very boring. I say, “No, the opposite is the case.” The example I usually give is not to do with the improbability but to do with physics. The example I usually give is the rainbow. Now, when you don’t know how the rainbow works, you walk out and you see this amazing band of colours arching across the sky and you think, “Wow, that’s incredible.” But then when you do understand how it works, you understand how light reflects and refracts around inside the rain drops. You walk outside and you still think, “Wow, this is incredible.” You look at it, “This is beautiful. This is wonderful.” And probably, even more wonderful because you understand the complexity of the physics behind it.
I think exactly the same is true of the improbability principle. In fact, one little coincidence, which happened to me happened five, six years ago now: 2012 at the Royal Statistical Society conference in Telford. I went up to the reception desk to register and said my name is David Hand, and the receptionist looked at her screen and said, “But you’ve already registered.” Then I said, “No, I haven’t.” She looked at her screen again and said, “I see. There are two David Hands with us this week.”
It’s not a very common name, so I was really tickled by this. It wasn’t like the magic go away; it’s just as excited even though I then went away, did the calculations and worked out what’s the chance of me meeting somebody else with my name given the number of conferences I go to, the number of hotels I’ve stayed in and all this sort of thing. It’s not surprising in retrospect, but that in no way takes away the excitement of it.
Martin: What makes us as humans so intrigued by coincidences? Is it the knowledge that there is law behind or a science behind but also it still feels a little bit magical when it happens?
David: I think you’ve put your finger on it there. I mean, this sort of coincidence is seen to contravene causality. I mean, a strange town and suddenly I bump into my brother and I have no idea he was going to be there, and you think something must have caused this, so this is hint that there’s something magical about it even though you can’t actually explain that sort of thing.
They come as a surprise and they seem to imply causal mechanism that we’re not aware of even though they don’t really, and I think that’s why they’re moleticulous so much.
Martin: Bringing this background to your role at Winton, how does the improbability principle relates to investment management?
David: It’s all part of understanding the nature of probability, chance, random fluctuations and so on. I think as you dig down into the five laws, the improbability principle gives an explanation and understanding what’s happened in the past and based on that what might happen in the future.
It also shows how careful we have to be to get your models right. in my book, for instance, I talk about the history of financial crushes and you get people asserting things like, you’ve probably read this sort of thing, people saying things are incredibly unlikely, saying things like something like this would be expected to happen only once in a billion years. But then it turns out that that probability calculation, once in a billion years, was based on a slightly incorrect model. If you use the correct model you’ll find that’s not so surprising at all, which is perhaps my over alarming.
That’s the point, really. The improbability principle, the understanding of change and probability that it gives sheds light on the way the financial markets behave.
Martin: You mentioned at the start of our conversation, I don’t think this is unfair characterization that most fund management groups, most investment managers take this put-your-finger-in-the-air approach. They have their crystal ball, they have their opinions. How does the approach taken a scientific method of investing differ from that more traditional approach of investment management?
David: I think that’s absolutely right although I guess I should say that they probably wouldn’t characterise it as a finger in the air I guess, but their expertise is what they’re applying only. I think that’s how they put it.
I think the traditional approach, their approach is based on what they would call their expertise, but we all know how unreliable humans are at making predictions. You’re probably familiar with Daniel Kahneman’s book: Thinking, Fast and Slow, a Nobel Prize winning economist. He describes some of the subconscious biases that humans are susceptible; the mistakes that they can make. Things like overconfidence, a hindsight bias, confirmation bias, these sorts of things.
We’re just not very good at doing intuitive probability calculations. We do get things wrong. We make assumptions, make mistakes all subconsciously, of course. I suppose the point is that Winton doesn’t do that. Winton actually looks at the numbers, crunches the numbers to come up with the objectively right conclusions.
Martin: There’s huge amounts of debate in the profession at the moment about active investment management versus passive management. Do you think that active fund management can be evidence based by applying this scientific approach in the same way that passive investing clearly do?
David: I think … I’ve had enough so. In fact, I think it would be right to characterise Winton as entirely evidence based.
I sometimes say that data is a synonym of evidence, so what we do and what other companies like us do is look at the data, look at the evidence and see what’s actually going on.
Like I said at the start, what we do is we use the most advanced data analytics statistical machine learning artificial intelligence methods and ideas that guide us in extracting understanding and predictions from those data, often, those evidence.
Martin: David, thank you for joining us on the podcast. That’s been a really interesting conversation. Before you go, where can we find out more about your work and about the book?
David: It’s been a great pleasure. I’ve got a webpage at Imperial College where you can see a list of, as you say, the 300 papers and, indeed now, 29 books and the other things I do. As for the book itself: The Improbability Principle itself, I think the best way is to buy a copy of the book.
Martin: Absolutely. We’ll make sure we put links in our show notes as well so our listeners can find your webpage and find the book nice and easily. David, thanks for your time.
David: Great. Thank you very much indeed. It’s been a great pleasure. Thanks.
