How to measure the accuracy of forecasts

How do you hold forecasters accountable, when the forecast is only a probability? The answer appears tricky, then simple, then tricky again, then ends up being simple enough to answer it with Google Spreadsheets.

It’s a journey worth taking, because building better forecasts is invaluable for businesses:

Take lead scoring — putting a value on a new sales lead, predicting the ultimate value of that lead after 9 months have passed and it’s either converted or not. The forecast is what chance this lead has of converting, or what dollar value it has. Like the weather, the lead will convert or it won’t, and if it does, it has a definite dollar value.

If you could predict the chance that a given customer might churn in the next thirty days, you could be proactive and perhaps avert the loss.

If you could predict the chance that a given customer would be amenable to an upgrade, you could focus your internal messaging efforts accordingly.

But how do you measure the accuracy of a prediction which itself is expressed only as a probability?

Read this article

How to measure the accuracy of forecasts

How do you hold forecasters accountable, when the forecast is only a probability? The answer appears tricky, then simple, then tricky again, then ends up being simple enough to answer it with Google Spreadsheets.

It’s a journey worth taking, because building better forecasts is invaluable for businesses:

Take lead scoring — putting a value on a new sales lead, predicting the ultimate value of that lead after 9 months have passed and it’s either converted or not. The forecast is what chance this lead has of converting, or what dollar value it has. Like the weather, the lead will convert or it won’t, and if it does, it has a definite dollar value.

If you could predict the chance that a given customer might churn in the next thirty days, you could be proactive and perhaps avert the loss.

If you could predict the chance that a given customer would be amenable to an upgrade, you could focus your internal messaging efforts accordingly.

But how do you measure the accuracy of a prediction which itself is expressed only as a probability?

Read this article

Invention is Drudgery

Having skipped to the last page of other peoples’ book, we forget that everyone has to take the journey described by the whole book. So we feel bad about ourselves when we’re only on Chapter Four, having already toiled quite a lot thank you very much for asking, and when exactly are we going to get to the good part?

Read this article

Darwinian company growth doesn’t always select the best companies

“Survival of the fittest” is not the same thing as “survival of the best,” though not apparent at first glance.

There’s nothing inherently wrong in obsessing over growth; in fact, using the same argument you can point out that in winner-take-all markets, growth is objectively the main requirement.

The problem is “growth at all costs.” When “growth rate” becomes essentially the only important metric for company “fitness,” other metrics are left unsolved.

Read this article

Your non-linear problem of 90% utilization

Does it feel like everyone is working very hard, all the time, and yet accomplishing 1/10th of what it seems they should? Maybe this is why.

Read this article

How do I figure out who my next important hire should be?

Founders typically revert to whatever they’re already expert in, and decide they need more of that. So, a technical founder decides she needs another developer, or a sales-oriented founder decides she needs another sales person.

Let’s highlight the underlying justification you’ll use to rationalize this probably-erroneous decision, so we can see how to avoid the fallacy.

Read this article

Refutation: An acquisition is always a failure

Not all acquisitions go well. Sometimes the product is ruined, sometimes the culture is annihilated, sometimes jobs are destroyed, sometimes it’s an unmitigated disaster.

Of course, that same thing happens to independent companies. People love quoting (presumably made-up) stats about how most acquisitions are failures, but they forget to mention that most independent startups also fail.

Read this article

Balance is weak. Tension is strong.

When you fire a gun you pull with one hand while pushing with another because in tension there is stability. A pull-up bar in a doorframe can hold 200 pounds because it’s in tension, not in balance. Ethernet contains a pair of wires twisted together, with electrons moving in opposing directions, because their opposing magnetic fields cancel either other but also act as a shield to foreign fields that would otherwise disrupt the electrons’ movements.

It’s often said that “extremes are bad” and thus you should “seek balance.” But balance is fragile; it’s the wrong thing to seek.

Read this article

Why it’s nice to compete against a large, profitable company

A big, profitable company seems like the hardest thing for a small company to compete against. They have everything: money, brand, momentum, existing customers, press, product teams, distribution channels, expertise, market insight, analysts, sales offices, product features, and, by definition, a working business model.

All the little startup has is a decent idea and extremely greasy elbows.

But David has a clear path to slaying Goliath.

Read this article

Zero-sum marketing channels: Good or bad for a startup to pursue?

You might conclude that it’s wise to try to win zero-sum games, because there’s “double” value. That is, not only do you get the sale, also all your competitors don’t. But, especially with auction-style zero-sum games, this means paying top-dollar. Even then, the available inventory is limited, e.g. the top few slots in AdWords or SEO.

Whereas if you invest in non-zero-sum channels like social media, or long-tail SEO, or brand-building, or content, or resellers, the limit of the channel is closer to the limit of the market itself, rather than the sliver of the market which might reach you via a single advertising channel.

Read this article

On the value of Judgement

We’re told not to judge, lest we be judged.

But we are judged, by our customers choosing between us and a competitor, by our employees choosing whether to entwine their careers with our fate, by our peers, allies, and passive-aggressive antagonists on social media.

Rather than avoid judgement as a sin, we should invest in it as a skill.

Read this article

Scaling by “delegation” isn’t good enough

Young founders may fancy themselves wizards of coding, design, and salesmanship, because they’re individually excellent; I did! But it should be obvious that those skills don’t mean they can build a team of 75 engineers that balance quality with speed, or build an international sales team guided by principles other than overwhelming exuberance, or develop a consistent brand with a voice and adherents, or manage cash flows once the P&L becomes abbreviated “in millions,” or navigate HR and insurance and office leases and those various “industry-standard boilerplate” contracts that nevertheless all differ in ways that aren’t rounding off in your favor.

The usual solution to this is “delegation,” but that doesn’t scale, nor does it lead to greatness. Here’s how to think about it properly.

Read this article

Fermi estimation for startup business models

Early in a company’s life, you don’t know anything. Often your best estimate of any metric or market behavior or business model assumption is accurate only within a power of ten, for example “expected conversion rate between 0.5% and 5%” or “cost to acquire a customer between $50 and $500” or “average monthly revenue per customer between $20 and $200.”

Fortunately, being accurate only within a power-of-ten can be surprisingly useful. The method is called “Fermi Estimation.” It’s useful because it’s easy to get to a rough answer to an otherwise-unknowable question, but also ensures you don’t erroneously ascribe too much precision to that answer.

Read this article