Product development — how NOT to conduct a survey

Edited excerpt from Surveys and focus groups by Seth Godin:

The story is told of a focus group for a new $100 electronic gadget. The response in the focus group was fabulous. All the people talked about the features of the new device with excitement.

At the end of the session, the moderator said “Thanks for coming. As our gift to you, you can have your choice of the device or $25.”

Everyone took the cash.

Surveys that ask your customers about their preferences, their net promoter intent, their media habits — they’re essentially useless compared to watching what people actually do when they have a chance.

(1) Seth Godin is right that surveys which ask people what they think or would do are ineffective. But that doesn’t mean your only option is to watch directly what people do. Sometimes you don’t have the opportunity to observe actual customer behavior. For example, if you’re trying to assess potential demand for a product you haven’t yet developed, you might want to know what similar products your customers buy. In cases like that, surveys can be very useful.
(2) How then should you run a survey to avoid the pitfall Seth describes? Ask questions that get your customers to share facts and experiences rather than opinions. In the words of Benson Garner: Don’t ask “Would you..?”. Ask “When is the last time you..?” or “Tell me about a time when you..?”. See How to interview customers to get great product insights.
(3) Cf. The survey question you should never ask.

2 thoughts on “Product development — how NOT to conduct a survey

    • Hi Eli, your right that actual user testing is always better where you can do it cheaply and fast. Your suggestion is similar to other tactics used in the lean startup methodology, such as “false doors”, where you offer a product (including pricing) which doesn’t yet exist, and measure attempted purchases.

      Having said that, surveys have two advantages:

      First, observable data often don’t answer the question of “why”. This can be very important for product development. If I can understand why you did something, perhaps I can provide a product which addresses your need better than what’s currently available. This is the heart of the “job to be done” framework. Sometimes I can’t understand your “job to be done” by observing what you do. In Clayton Christenson’s famous milkshake example, he saw that people were buying milkshakes, but didn’t really understand why until his team started to interview customers. When they did that, they learned that there were two jobs to be done, one requiring slow-to-drink milkshakes for commuting adults, and one requiring fast-to-drink milkshakes for parents buying for children.

      Second, it can be much faster to do surveys than to do tests which require changes to your production environment. The key is to get a framework for surveys set up so that the product managers can run surveys whenever they want to without involving tech. There are off-the-shelf products which do this for websites.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s