Is It Statistically Significant?

Having done a great deal of interview-driven analysis for clients, I always smile to myself when this scenario comes up:

  • Consultant:  “Based on 20 interviews, your customers give you a 1.5 out of 5 on your software installation and customer service, and they say that it’s the primary reason they don’t buy more products from you.  You need to fix that stuff to grow revenue.”
  • Client:  “Hmm, that’s very interesting.  But is that result statistically significant?”
  • Consultant:  “Well…”

Of course, the client is really asking whether they can rely on this information, and that’s an important question.  But to answer that question, I usually have to force myself to ignore what they’re actually saying because statistical significance is essentially irrelevant to business research based on in-depth interviews, as opposed to surveys with large sample sizes.

Why is that?  Primarily because common measures of statistical significance assume that your results come from a discrete uniform distribution (like rolling a die) and thus a normal distribution (a bell curve) for the mean.  Real-life probability distributions in business are not normal, nor do they necessarily obey the central limit theorem.  For example, they could be bimodal – some people like you, others hate you.  So right from the get-go, you’re throwing out the main assumption behind standard models of “statistical significance.”

Statistical significance also assumes a uniform population.  Unfortunately, real markets have segments, and those segments will respond to research questions in distinct ways.  So that’s another central assumption of the model that does not hold true in reality.

Finally, you have measurement error.  Strictly conducted social science experiments conduct each interview exactly the same way to avoid biasing the answers.  In a business setting, the value of pulling out more information from each interview outweighs the risk of introducing bias, so often a more conversational approach is taken.  You get more insight into the topic, but at the expense of yet another tenet of statistical significance.

So, while statistical significance is a good concept in theory, like efficient markets, it doesn’t really exist in the real world outside of perhaps a survey setting.

I don’t typically bring all this up with the client since it would just bog the conversation down in procedural talk that doesn’t really settle anything.  Instead, I just say something like, “The results of the research reliably show that this is an important issue, and you should act on it.”


Related Posts:

  • No Related Posts
blog comments powered by Disqus