When do I *stop* doing customer interviews and start writing code?

Robert Graham of WhiteTail Software (and this awesome guest post on cold calling) asks:

@whitetailsoft When do you stop #custdev efforts and build the product? I’ve been wrestling with the details of #leanstartup.

(from his blog post) …some civil engineers think the world of project management in their field is ripe for revolution, but do I know enough of them? How many should I talk to? 5? 30? 100?

I talked to 30 people before I realized that a certain idea of mine was a crappy idea, and about 40 people before starting WP Engine. Here are the details of both of those customer development experiences.

But there’s no one “number.” Food on the Table — a now-famous lean juggernaut in Austin run by IMVU alum Manuel Rosso — talked to 120. Capital Factory 2011 alum GroupCharger talked to 50 before building and another 50 after that. At AppSumo, another Austin startup with startling growth, Noah Kagan (this guy and that guy) talked to 0 people initially, but maintains ruthless pressure on a tight and measurable product.

There’s two ways to decide when you should stop talking and start building.

Way #1: Go until boredom.

Recently at WP Engine I did some brand new customer development for a new project that we think will revolutionize WordPress blog management. I spent 30 hours talking to WordPress consultants, but I didn’t have “30” preset in my head. I knew to stop when the process got boring.

The first dozen calls were a blizzard of activity — my note-taking fingers furiously trying to keep up with new information being revealed, theories getting alternately validated and blown away, unexpected customer segments arising, and new ideas recombining from the primordial soup generated by introspective, honest, provocative conversation with thoughtful people who were “living the pain” we’re trying to solve.

This is what it feels like to learn, remember? If it feels like learning, you’re probably, you know, learning, and that’s the point. If you’re not learning, what are you accomplishing?

But then as the conversations wore on, I felt a change. As I emphasized what I (now) thought was interesting, sure enough the person on the other end of the phone gets excited. As I attempt to push old ideas which just two weeks ago seemed so fresh and smart, sure enough they fall flat again.

The change is exactly like those charts from biology showing the rate that a chemical imbalance diffuses into a cell — fast at first when the imbalance is high, then lessening asymptotically.

 

Facdiff

With my recent interviews, it got to that flat place in the graph. I could segment the customer from the first two questions, and then I could predict almost perfectly what the answers would be to the other questions.

No more learning is happening. Now you’re bored. Time to stop.

 

Does that mean there’s nothing else to learn about the market? Of course not — it means that this particular exercise is no longer useful right now.

So now it’s time to do something else to progress the company and peel back the next layer, and that means coding up and releasing your MVP.

Way #2: Get ten paying customers.

Remember this is before you’ve built the product, so if ten people give you cash anyway, that’s the best possible signal that the world wants this product to exist.

If it takes talking to 200 people to get those 10, that’s not a good sign. You might think “5% conversion is pretty good,” but not when you’re getting face-time. If the founder sits down with what ought to be the perfect customer and chats for an hour and cannot convince more than one in twenty of the value of the project, it’s too difficult to sell.

But with WP Engine I had lunch with 40 people and 30 said they’d give me $50/mo as soon as I built it, and 20 actually did before launch. I should have asked for the $50 on the spot, or at least had them sign something committing to it; then I would have focussed my remaining efforts on making those 20 people happy instead of diluting it with all 40.

“But it’s impossible to get non-technical people to pay for ‘software’ that’s nothing more than sketches on a napkin,” I hear you say.

Bullshit. Both Food on the Table and GroupCharger did it, one with busy moms (i.e. consumers), the other from small associations (i.e. business). Two of the other Capital Factory companies did it successfully too.

In fact, almost all the people I tell this to are initially incredulous, but you should see the look on their faces when it works. It separates the polite interviewees from the truly interested, and the truly interested are frequently up for a reasonable pre-pay.

And once you get 10 of those, that’s enough opinions. Every customer wants something different, all see a different direction for your company, each have their pet features they want, each will find some part of the product confusing or unnecessary or critical. More cooks in the kitchen — at this very early stage — is not better, because design is not a field for the Wisdom of the Crowds.

Fewer than 5 and you’re designing an idiosyncratic product, not much better than “building it for myself.” More than 15 and you’ll have too many voices not converging on simple truths.

What are your tips for knowing when customer interviews are still valuable and when they’re a waste of time? Let’s accumulate more wisdom in the comments.

, ,

  • http://smartsoftwaremarketing.co.uk/ Giles Farrow

    I think the key point here is keep going until you are not learning much

  • Brian

    or learning too many distracting things. 

  • shparekh

    It’s quite reassuring to see my experiences are in similar vein. I haven’t reached a point where it has gotten boring for me yet.

    I decided to build a prototype after my first interview. But that lead did not materialize. Since then I discovered other niches and paying customers.

    I try to figure out common features desired by customers. Often times I find that when they want a feature they are quite amenable to changes to it as long as their purpose is served. So it helps me to formulate a specific feature that can satisfy many related end goals.

    • http://blog.asmartbear.com Jason Cohen

      Most interviews won’t result in a customer; that’s why you need 20 of them before prototyping.

      Customer are bad about figuring out features, but good about explaining their pain or what they’d like solved.  It’s up to you to provide them something in which its obvious they can be successful, regardless of what the actual features are.

      • shparekh

        You explain better the difference between need and a feature.

        Thx.

  • Dricupero

    What a great post. I currently am about to start an accelerator and we were struggling in what goal to establish with biz dev. Tell people what we’re building and build a list of people that are interested or Sell the idea and collect money now. We plan to collect the money now. Thanks Jason.

    • http://blog.asmartbear.com Jason Cohen

      You bet.  Asking for money is always best.  People paying means everything; anything else is just talk.

  • Alfred Smith

    Hi,
    I am looking for some good quality websites or blogs as I would like to contribute my good quality original articles in those sites. I found your site: (http://blog.asmartbear.com) informative and interesting.
    I was wondering if you can allow me to write for your site. I use to add a text link in my articles. And in return I will provide you backlink from one of the relevant sites. Waiting for your positive reply.

     

  • http://www.whitetailsoftware.com/ Robert Graham

    I asked quite a few people that question and Jason’s answer was the best one I got. Thanks!

  • Pingback: When do I *stop* doing customer interviews and start writing code? | Texas Entrepreneur Networks()

  • http://sehabitat.com/ Ian Mason

    Wow. Lincoln Murphy (Free Trial Dominator) told me to check out this post since I had asked him some burning questions about my startup woes with SEHabitat.

    Thank you for this – I talked to 20 people. I didn’t ask for any money and didn’t have pricing set, not even close…

    • http://blog.asmartbear.com Jason Cohen

      That’s OK, you still can!  Call them back.

      Here’s more motivation: http://blog.asmartbear.com/customer-validation.html

      On a positive note, when it’s done well, pricing itself is a catalyst: http://blog.asmartbear.com/perfect-pricing.html

  • http://twitter.com/sylvainww Sylvain Gauchet

    Thanks again for a great post! 

    I have a kind a stupid question (I usually ask them anyway): with relatively few people that could be potential users of your product in your connections, what do you think would be the best way to find these 20/30/40/etc. people to talk to (and 10 to pay you)?

    • http://blog.asmartbear.com Jason Cohen

      It’s not at all a stupid question.  Of course the answer depends on the market and the tool.  Selling something to restaurant owners will be different than selling to Twitter experts.

      Here’s some general advice: http://blog.asmartbear.com/get-first-customers.html

      And a specific technique which I’ve used myself: http://blog.asmartbear.com/cold-calling.html

      • http://twitter.com/sylvainww Sylvain Gauchet

        Thanks Jason. I got a bad memory, I had already read them..I guess I was secretly hoping that the answer wasn’t “cold calls” ;)

  • Pingback: Startup Links « The Blog of Gobi Capital()

  • Pingback: Twitter Link Roundup #135 – Small Business, Social Media, Design, Copywriting, Marketing And More « crowdSPRING Blog()

  • Pingback: When do I *stop* doing customer interviews and start writing code? | Evan M Rose()

  • Pingback: 26 Resources to Help You Master Customer Development Interviews()

  • Pingback: | ConnectingtotheCloud.com()

  • Pingback: 26 Resources to Help You Master Customer Development Interviews - SkyOffice Consulting | SkyOffice Consulting()

  • Pingback: 26 Resources to Help You Master Customer Development Interviews - Oscar Pena()

  • Pingback: The Art Of Neighboring… In Product Management | Product Management Meets Pop Culture()

  • Pingback: Sounds great, let me know when you launch | David Tuite()

  • Adam

    What if your potential customers aren’t paying in cash but eyeballs? How do you sit down with 20 people who are representative of the millions you would need to sustain a business through advertising, and determine whether your content-site/etc is viable to capture their attention over a significant period of time?

    • http://blog.asmartbear.com Jason Cohen

      The short answer is that this post is written for a product company, so it doesn’t necessarily apply. :-)

      I do think it makes sense for any organization to understand the mindset of the people they cater to. Just because there are millions of viewers doesn’t mean it’s unimportant to know what those viewers are like. So yes, in fact talking to 20 people who would be power-sharers or otherwise power-users, would be smart. Maybe 20 people who actively share on Twitter with large followings and significant engagement — whatever THEY think is probably important.