Do you read business blogs where the author has failed three times without success?
No, because you want to learn from success, not hear about “lessons learned” from a guy who hasn’t yet learned those lessons himself.
However, the fact that you are learning only from success is a deeper problem than you imagine.
Some stories will expose the enormity of this fallacy.
Bullet holes: A brain teaser
During World War II the English sent daily bombing raids into Germany. Many planes never returned; those that did were often riddled with bullet holes from anti-air machine guns and German fighters.
Wanting to improve the odds of getting a crew home alive, English engineers studied the locations of the bullet holes. Where the planes were hit most, they reasoned, is where they should attach heavy armor plating. Sure enough, a pattern emerged: Bullets clustered on the wings, tail, and rear gunner’s station. Few bullets were found in the main cockpit or fuel tanks.
The logical conclusion is that they should add armor plating to the spots that get hit most often by bullets. But that’s wrong.
Planes with bullets in the cockpit or fuel tanks didn’t make it home; the bullet holes in returning planes were “found” in places that were by definition relatively benign. The real data is in the planes that were shot down, not the ones that survived.
This is a literal example of “survivor bias” — drawing conclusions only from data that is available or convenient and thus systematically biasing your results.
Doesn’t most business advice suffer from this fallacy? You read about successes but what about the businesses that “never made it home?” Like the downed planes, could failure contain more lessons than success?
Burying the other evidence
Scientific journals like to publish extraordinary results, so studies that don’t show anything of statistical significance aren’t published but rather are abandoned or silently stowed away in academic filing cabinets.
This practice is called the “file-drawer effect,” and it’s a particularly insidious form of survivor bias because it’s invisible. Peter Norvig sums it up nicely:
When a published paper proclaims “statistically, this could only happen by chance one in twenty times,” it is quite possible that similar experiments have been performed twenty times, but have not been published.
Pharmaceutical companies have exploited this effect to intentionally skew results. It’s gotten so bad that journals are calling for a public database to prevent fraud:
More than two-thirds of studies of anti-depressants given to depressed children, for instance, found the medications were no better than sugar pills, but companies published only the positive trials.
If all the studies had been registered from the start, doctors would have learned that the positive data were only a fraction of the total.
–Washington Post
Doesn’t most business advice suffer from this fallacy? Harvard Business School’s famous case studies include only success stories. To paraphrase Peter, what if twenty other coffee shops had the same ideas, same product, and same dedication as Starbucks, but failed? How does that affect what we can learn from Starbucks’s success?
Experimental proof of ESP
Dr. Joseph Rhine brought the rigor of experimental psychology to the study of the paranormal, and ESP (Extra Sensory Perception) in particular. He made waves in the 1930s with controlled experiments testing whether a person was able to predict the order of the cards in a shuffled Zener deck (with symbols like circle, square, star, and wavy lines).
In a typical experiment, 500 people are screened for “strong telepathic ability,” measured by significantly above-average performance in a 25-card deck. Those selected are tested again, and more drop away. Tested a third time, perhaps one person passes again and we conclude that such a repeat performance is statistical evidence of genuine ESP.
To see why this is just a different face of survivor bias, consider the following experiment. I believe some people are “heady” when it comes to coin-flipping — getting heads more often than chance alone would suggest. So I put 1000 people in a room and tell them to flip a coin ten times. Sure enough, a woman named Margaret makes “heads” ten times in a row! The chance of her getting heads ten times in a row is only 1-in-1024, so I conclude Margret has special abilities.
Actually that last statement is true but misleading. The chance that Margaret would flip ten heads in a row is 1-in-1024, but that’s not the experiment I ran was it? I let 1000 people flip and “found” Margaret in the crowd.
The chance that somebody in a crowd of a thousand would flip heads ten times is a whopping 62%! Because so many people are attempting the feat, some normally-unlikely events will happen. This isn’t a test of Margaret’s abilities at all!
Doesn’t most business advice suffer from this fallacy? Take me for instance. I’ve done three consecutive successful startup companies, so that’s proof that I know what I’m doing and that you should do everything I say, right? Except maybe I’m just the one in the crowd who guessed right on the Zener cards three times, and there’s no reason to believe I would be successful a fourth time.
Specific examples of survivor bias in business advice
So far I’ve been asking rhetorically whether survivor bias might be severely skewing business advice. Steven Levitt (of Freakonomics fame) investigated this question directly.
He was reading Good to Great by Jim Collins, a book that analyzed eleven companies that were mediocre — just pooping along — but then transformed themselves into stock market sensations. A conclusion was that the common trait was a “culture of discipline.” This book has sold many millions of copies, so it’s a good example of popular writing on business advice.
One of the eleven “great” companies was Fannie Mae, and Steven Levitt was reading this book just as Fannie was collapsing in financial disaster. Hmm, he thought, I wonder how those other “great” companies are doing.
Turns out, had you invested in those eleven companies in 2001 (when the book came out), your portfolio would have underperformed the S&P 500! (Fannie Mae wasn’t even the only case of total disaster — also extolled was the now-bankrupt Circuit City.)
Why didn’t these companies continue to succeed? It turns out Jim started by combing through 1435 companies looking for good candidates for the book, and picked eleven. It’s the ESP experiment all over again!
On top of that, Jim doesn’t bother asking whether any of the 1424 other companies also displayed a “culture of discipline.” Maybe that’s something that many public companies have regardless of performance.
Is this book an aberration? Nope, Steven investigated another business book from the 1980s — In Search of Excellence — and found the same effect.
Steven then comes to the same conclusion that I’m coming to:
These business books are mostly backward-looking: what have companies done that has made them successful? The future is always hard to predict, and understanding the past is valuable; on the other hand, the implicit message of these business books is that the principles that these companies use not only have made them good in the past, but position them for continued success.
To the extent that this doesn’t actually turn out to be true, it calls into question the basic premise of these books, doesn’t it?
Oops, did I just invalidate my blog?
Lately I’ve been wondering if a lot of business advice — both mine and others — is really a case of survivor bias. I mean, I didn’t start out at Smart Bear with a load of philosophy and a fixed idea of who the customer was or even what the products would be.
How do I know this post-hoc philosophy and advice isn’t just a case of survivor bias? Am I not like the ESP-savant, successful not by force of nature but by simple chance of surviving?
Or perhaps I’m like Dr. Rhine the ESP experimenter — convinced I’ve discovered something important with “objective measures of success” — and yet I’m actually living in a dream world.
More to the point, how can you, dear reader, ascertain whether my articles or any advice from anywhere suffers from this fallacy?
In the end of course you don’t know. But here’s something: Just the fact that you’re aware of survivor bias means you’re less likely to be fooled by it. So, reading this article has helped a little.
Beyond that, prefer advice that makes you think and forces you to answer tough questions of yourself, not advice that simply tells you to march in a certain direction. Use advice as a sounding board rather than gospel.
What do you think? How do you decide which advice to take? Leave a comment and join the conversation.
65 responses to “Business Advice Plagued by Survivor Bias”
"The chance that someone in the crowd would flip 10 heads in a row is a whopping 62%."
Here’s the derivation. It’s easier to compute the chance that nobody flips 10 heads in a row. There are 2^10=1024 possible outcomes to flipping a coin 10 times, only one of which is to have a run of 10 heads.
So, the chance the first person doesn’t run 10 heads is 1023/1024, a probability I’ll name P. That the first and second person both don’t run 10 heads is P*P, or P^2. That 1000 people don’t run 10 heads is P^1000, the value of which is 38%.
Remember this is the chance a run of 10 heads does not happen, so the chance it does happen is 62%.
Fun fact: Whenever the probability of an event is (S-1)/S and you run S trials, as S grows the probability of not seeing the event approaches 1/e. In this little problem this was approximately the case.
Jason, does this also explain the "hero complex"? I.E. why we pay CEO’s trillions of dollars, because we think they have some magic or know something that makes them awesome? Or hedge fund managers?
To expound on your conclusion, I try to listen to the "how" of what makes people successful instead of the "what." How lucky was the person/company? You hafta figure out signal to noise, and what fundamentally the person did right/wrong, if any.
@Dale — Good point, certainly makes sense. I know there’s hard data about hedge fund managers and mutual fund managers unable to beat the index in the long haul, and most analysts believe this is a case of survivor bias.
I was going to put that in the article but it was too long and complex. For those who are interested, Wikipedia has a good summary.
Agreed on the "how." Also, since luck always plays a part, the question is "How did the person recognize and capitalize on luck?"
Jason, this is a great post, and inline with my own writing/experiment. I have had some success, but I am not a massive success. That said, I have tried to take lessons learned from the experiences I have had as an operator and an investor.
One of the more startling conclusions I came to before starting on the project was how many of the mistakes I made, even though I should have known better based on my own experience. There’s nothing like actually doing, versus the theoretical, as a classroom.
I enjoyed the post, but I really wanted to look at the source material for the Steven Levitt quote. Could you link that?
"Why didn’t these companies continue to succeed? It turns out Jim started by combing through 1435 companies looking for good candidates for the book, and picked eleven. It’s the ESP experiment all over again!"
Actually he did have ‘control groups’ throughout the book (who didn’t follow the patterns he suggests). He also has another book (How the Mighty Fall) where he basically says those companies that failed e.g. Fannie Mae, stopped doing what got them there. Whether he is right on all of this is up in the air as far as I’m concerned though.
@Brandon — No doubt, school is useless in the real world of doing business. Mistakes == learning.
@Brian — I can’t believe I forgot to link it! Nice catch. Here’s the source to the Steven Levitt quote.
@ww — It’s not a "control group" if you select who goes in the "control." I know he said that, but that concept only counts if you take a population and randomly put some in control and some get some treatment. You’re looking to see if the treatment does anything. If you hand-pick which goes in which groups on the basis of a theory (that you invented only after already observing the data), you’re just selecting what you want to find. There’s nothing scientific about it.
Your point about the company no longer doing what got them there — that’s valid of course, and I agree with you that it’s hard to tell whether that’s true or not.
Of course you often see just the opposite argued: That companies NEED to change how they operate as their markets change, their competitors change, technology changes, economy changes, etc.. So just "sticking with what works" is surely not an acceptable rule anyway.
There’s an old scam based on the same principle.The scammer sends a sports tip sheet to 100 people predicting the winner of a game. He sends a tip sheet to another 100 people predicting the opposing team will win. The following week, the 100 who received the correct answer are divided into two groups and fed another pick. This is repeated until a few victims have received a perfect series of correct picks, then the final pick is offered for sale.
Citing the Harvard cases as an example of survivor bias is true, but misleading. Most (all?) of the cases come from survivors, but they rarely (never?) include the actual decisions made by the company in the student versions of the cases. In other words, there is no "advice". The cases set up the real-life business scenario and ask open ended questions about strategies for dealing with the scenario. They do exactly what you suggest we should prefer: make you think and answer tough questions for yourself.
Speaking of fallacies, how many fallacies are in the statement "School is useless in the real world of doing business"? ;-)
All that said, I appreciate this post. The point is well-taken that business advice is often non-scientific and suffers from various fallacies. Buyer beware.
@Mark — I almost included that exact example in the article, but I didn’t because in that case the "bias" is a deliberate scam, and I wanted to focus on accidental misjudgements. But yes!
@Brandon — Some of the case studies are open-ended questions — the made-up companies — but they also write up actual businesses and then it’s a post-hoc history/analysis. Yes they ask you to evaluate the current choices and the idea is to decide what you would do next. But are there cases about how such-and-such a company failed and what went wrong?
I would go further and say that essentially no business advice is scientific at all…
Just to be ‘that guy’, I quote:
"Is this book an aberration? Nope, Steven investigated another business book from the 1980s — In Search of Excellence — and found the same effect."
You now have Steven Levitt choosing 2 popular business books which have failed ‘due to some expected effect’ out of a population of thousands. …Precisely the conditions under which this sort of bias prevails.
I largely admire the point of your post; I just find that funny :P
Conversely though, I have actually seen a number of blogs and articles from people who failed, and have never felt that they would be less useful. Do most people feel they are useless? That seems surprising.
On top of that, the ‘success-story’ blogs out there rarely have the story "We did A and succeeded" – the story usually goes "We did A and failed, because of _____, so we tried B and things got better but didn’t work, because ______, and it was only when we’d realized C to fix _____ that things got moving along". I find people tend to mention their failures, and yes, while they talk about how they coped with those (a sort of success tale vulnerable to this bias), knowledge just of the failure is a valuable thing to gain, and hearing a route that provided success doesn’t give zero data. Not to say that every one is so useful – but a great deal are.
And a final point in agreement with you: if you recognize the survivor bias, it may be possible to almost completely counteract it. The engineers with the warplanes should have recognized "okay, bullets here don’t seem to cause problems" and thought "so where did these planes _not_ get hit, that would be likely to cause problems?" – as long as they know the system (the plane), and the success data (hit in wings, tail), just as with probability, they can solve for the complement. Perhaps we can as well.
if it hasn’t been mentioned, Founders at Work by Jessican Livingston is a treat filled with false starts and total guesses. The kinds of things we all celebrate as luck, a conspiracy of the universe in our favor, just at the moment we might expect it, like a groove or something.
and failures everywhere. 1000s from a guy who makes things like the first light bulb they say.
but whatever. it’s all of it manufactured in the most ticklish sense anyhooo.
was that satisfying? who the fuck knows?
do you remember? were you even aware of the moment?
no, but seriously, what a good read this was, Jason. thank you.
Nice post, it reminds me of the book "Black swan" from Nassim Taleb which is about all these bias we all experience. I really recommand the book if you want to go deeper!
Very good article, I have the same feeling about business advice and books for years.
I’ve written a year ago about survivor bias and technology choices – it’s the same as with business advice.
http://codemonkeyism.com/rails-scrum-cmmi-and-survivor-bias/
"We hear lots of success stories from companies and developers about tools, languages, frameworks, plattforms and processes. Those people praise Rails, Scrum or CMMI. But looking at funds and transfering our knowledge to those praise stories, we could assume that those involved were just lucky."
Cheer
Stephan
@wcarss — I love it! Yes Steven picked 2 books (and not at random), so it’s not a test. Ha!
Of course there do exist books and blogs and articles about failure; I admit I was being extreme when I claimed "you" never read such things. I think my main assertion is true — that the vast majority of popular writings on business, marketing, sales, etc, is a backward look at something that was successful, not for example comparing failures with successes to see what’s different.
Yes I know the usual story is "We did A and failed, B, C, and eventually D worked." However, that is also the story of companies who failed. Well, not always the story, but often is too.
This is my point — that it’s not enough to say "We failed but continued," but rather to compare completely failures with successes.
It could turn out that the only difference is some people stopped sooner, maybe because of money or life circumstance, or maybe because they didn’t have the stamina, but if that’s the case than this "advice" isn’t really leading you to success or failure is it?
@Sherif — It’s true that recovering from mistakes is essential. However see the previous few paragraphs for why you can’t conclude that this is key to success.
@Clement — Agreed, the Taleb book is a classic accounting of biases like this. He has been criticized for not bringing anything "new" to the table, but who cares as long as it’s a nice exposition of the point! None of my stories above are "new" either…
@Stephan — I totally agree. What’s the solution though? I admit I didn’t offer much…
Hi Jason,
This is not an erudite post at all. I simply want to say that I relish – yes, relish – your essays as such and I look forward to receiving them very much. I’m actually a therapist but I work with some clients who are involved in company startups and I will be recommending your website and blog to them. Many thanks indeed.
Sharon
As well as the Black Swan – I would recommend his ‘Fooled By Randomness"
It goes directly into some of the mathematics in some of these biases (ie the coin toss)
Regards
Jason,
I enjoyed your post very much. You did an excellent job of summarizing several aspects of survivor bias in corporations.
Recently, I’ve been doing a little personal research into an area that extends "survivor bias" into personal success and wealth. In other words, trying to answer the question "Is Bill Gates the wealthiest man on earth because he’s smarter/better in some way than everyone else or was it just luck?"
This has lead me to doubt the existence (or even the possibility) of meritocracy. The idea that the cream with float to the top and "losers" will drop to the bottom. Man is responsible for his own destiny. As much as Google and others claim to be meritocracies, I believe that luck has more to do with advancement and career "success." Don’t get me wrong, there are some very smart people at Google. Advancement has more to do with opportunity (ie. openings, expansions, etc.), timing, and personal relationships.
It used to be that the poor were called "the unfortunate" and now my thinking is that it was a more accurate term than anything we’ve used in the last 40 years.
Regards,
Mark
Hey Mark,
Have you read “Outliers?” I think Gladwell has done all the research that might need to be done on the Bill Gates/luck question.
That said, if BG wasn’t brilliant he would never have taken advantage of his luck.
Also, understanding how to access opportunities is part of an overall “package” needed for success. Calling out “luck” as the biggest factor in success is a bit of a cop-out.
I know personally I always succeed most when I follow the mantra “do the right things and good things will happen” (although of course you have to be strategic in knowing what the right things are).
Brilliant post Jason. Absolutely LOVE the WWI word picture.
There are many lessons to be learned from survivors who lived to tell the tale. The problem in business is many survivors aren’t willing to share their battle stories. Not only are they not willing to share their battle scars (instead, they gird up their loins and rejoin the fray) – we not LOOKING for the lessons from the "losers" – as a culture, we’re not interested.
One exception to this is Roger Ehrenberg who shares some brilliant insights in his "hindsight is always 20/20" analysis of the failure of his business. I wrote about his powerful revelations Requiem for Monitor 110.
One of the most "startling" conclusions Ehrenberg comes to in his post mortem analysis – that VC funding made his business weaker – not stronger!.
That’s the kind of "truth" that can only come from the crash survivor himself. While outsiders were gazing longingly at the flood of VC funding – inside the business – the flood of capital was slowly killing the business.
I personally was amazed that the story didn’t garner much press – probably because as a culture, even business leaders are enamored by the gold and the glitz. While there’s a lot to be learned – this story never made it to the cover of a major business magazine (as far as I knew… I read about it "online" via "social media".)
Statistically the BEST performing stocks are the ones you NEVER hear about in the media. Maybe the same is true of business lessons as well – that the real "gems" like this lesson are shared discreetly for those smart enough to be reading your blog. :)
@Sharon — Thanks so much! You’re the highlight of the week. :-)
@Elliot — Good book mention, yes.
@Mark — I have to agree with you that "cream rises to the top" is insufficient. I agree luck has a strong component. I would argue that effort can make some impact, and without effort it takes far more luck to rise, so in that sense success favors the bold and hard-working. But of course it’s not sufficient. I would argue against you that just having "connections" helps. I had no connections at all in all the market spaces that I built companies in, and I don’t think e.g. Bill Gates did either.
@Kathy — You’re so right, and the media is to blame for covering what’s convenient for them rather than what’s useful and instructive for us. And the business books/blogs where we hope there’s more journalistic integrity are often more rah-rah and less truth.
I think a lot of people ARE interested in the failures, the mistakes, the truth. And you’re right some people write about it. Maybe we just need more of that. I’m certainly willing to commit to writing about more mistakes right here!
I think it’s even worse than only Survivor Bias. The Outlier effect probably kicks in too if you focus on the outperformers.
Jason, great post! Very thought-provoking.
Stirs up a few observations for me:
* Beware black and white thinking. Only looking at success without understanding the dynamics that could have changed the end result to a failure is a recipe for . . . failure
* Statistics can be very misleading, especially when the mental models (assumptions, frame of reference, theories, bias) that underlie them are not explicitly stated
* Success itself is not objective; it’s in the eyes of the beholder
As CEO of a Marketing Operations company (a field that is still evolving), I regularly see a bias toward the "sure-fire" best practice(s) modeled by a respected company (or, hopefully, several companies) as being most valued and worthy of being emulated. Often lost in the shuffle because of this bias: the value of the learning process itself.
Gary Katz
Marketing Operations Partners
The end result is often an organizational learning disability.
Here’s relevant comic strip from Dilbert
——
We’ll start ten mutual funds, each with randomly chosen stocks. Later we’ll build our advertisements around whichever one does the best …
——
@Gijs — I completely agree, more biases and fallacies are present here.
@Gary — Great points on what to take away from this. Your advice is more specific and useful than mine.
@Dennis — Ha, awesome. Unfortunately too close to the truth…
Nice thought-provoking post, Jason.
Just to add my $0.02 to the discussion:
I see the study of business/management much like the study of chess. in chess, you can read all what you want but unless you go out and play, you won’t really be able to progress. And what’s more, most experienced chess players say you should play against stronger opponents, and that you should constantly analyze all the games you lost. The more games you lose, the better you will become provided you learn from your analysis. In business, is the same. You can read about companies that succeeded and about those who failed but until you try it yourself, you will never really learn.
Aren’t those who then fail the most potentially the most successful?
One of the more startling conclusions I came to before starting on the project was how many of the mistakes I made, even though I should have known better based on my own experience. There’s nothing like actually doing, versus the theoretical, as a classroom.
I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.
Margaret
http://businesseshome.net
One of the more startling conclusions I came to before starting on the project was how many of the mistakes I made, even though I should have known better based on my own experience. There’s nothing like actually doing, versus the theoretical, as a classroom.
It is pretty clear from learning theory that:
– Teaching about mistakes and worst case is far more effective for changing behavior and practical learning
– Someone with 8-10 years of experience is the most resistant to learning and ineffective in their behavior
It’s a pity you didn’t read Good to Great, because Jim Collins was smarter than that. He picked up 11 companies under certain criteria, yes. And the first thing he did was search for another 11, that were in the same markets and dimensions (and matched the original 11 as much as possible) but didn’t have the same results. His conclusions came from the differences.
As for the results, Jim Collins never claimed they couldn’t be undone, he provided several examples to the contrary. His work is not perfect, but if someone did the claims you wrote, I’d doubt he read more than the FAQ on statistical significance in the end of the book.
Best regards,
João Miguel Neves
Thanks for the rebuttal! I didn’t read the book, no, I was simply reporting quoting and paraphrasing Steven Levitt, who did read the book and I’m sure you’ll agree has a keen mind for these sorts of things.
“…how can you, dear reader, ascertain whether my articles or any advice from anywhere suffers from this fallacy?”
Easy.
When I read contradictory advice from approximately equally credible sources, I choose the advice that I ***think*** is correct.
One good example would be your advice to stop lying about being a big company and admit I’m a one-man shop.
I followed that advice.
Was it a good idea? I don’t know. But it certainly feels better this way.
…and I don’t expect a 1000-seat order from Lokheed Martin any time soon anyway… ;-)
For me, the ideas in your blog posts seem to be right and reasonable. And your rationale is always convincing.
Thank you!
I have heard “survivor bias” also called the “narrative fallacy” by Nicolas Nassim Taleb in his book “The Black Swan.” Humans want to see agency in the world — we want to understand everything in terms of intent, but sometimes (many times) the “cause” is pure noise. This tendency is probably at the root of religion.
–Len.