For the last few years I’ve been trying to convince businesses to run experiments in order to learn how to do things better. Why is it that experimentation is the gold standard in science, but rarely exploited in corporations? My own hunch is that the main reason is what economists call “path dependence” — in other words, businesses don’t run experiments because they rarely have in the past. If, by chance, Henry Ford‘s innovation had been to run experiments rather than to develop the automated assembly line, experiments would be commonplace today in business.
A former student of mine named Ken Kovash, who now works at Mozilla, provides a nice example of how the simplest of experiments can provide answers that are otherwise elusive. As Ken describes in greater detail on the Mozilla blog, one of the ways Mozilla acquires new customers is through pay-per-click ads on search engines. The question Mozilla had is the following: if someone types “firefox” into a search engine, usually the first result they will see is the Mozilla site, so does it really do Mozilla any good to pay search engines to do featured links? Do ads actually generate more traffic, or do they just shift customers around — e.g., instead of getting the customers free, Mozilla ends up paying the search engine because of the pay-per-click ads? Without performing an experiment of some kind, this is a hard question to answer.
So over a two-week period, Mozilla experimented with turning their pay-per-click ads on and off more or less at random. The findings, as Ken reports, are somewhat mixed. Looking at the data one way, it appears that two-thirds of the customers who normally come to Mozilla through pay-per-click ads would get there anyway. On the other hand, the absolute number of downloads was substantially higher when the paid ads were running. This suggests either that (1) their treatment and control periods were different for an unknown reason; or (2) that the pay-per-click ads lead people to download more often through other channels. My guess is that (1) is more likely to be the explanation. But how can one really be certain?
The answer: by running more experiments.
and 
. You can also watch it on 
I have worked in R&D for most of my professional career as an engineer, often working hand-in-glove with marketing and product development.
Quantitative science has an aura of credibility that is greatly desired by those who work in the “softer” fields, so most companies do run experiments of a sort, although neither a scientist nor an engineer would recognize these as anything approaching robust experiments.
I think that while companies are interested in appearing to apply science to their work, there is little real interest in (or examples of) actually doing so.
The best I think I can offer in support of this idea is the following:
– Experiments are designed to test a clearly stated theory.
– Criteria for success and for failure must be (explicitly) understood prior to conducting an experiment to avoid hindsight bias.
– Experiments that refute theories are generally more valuable that experiments that confirm theories. By learning where a theory or thinking is flawed, one can learn to ask better questions and design better experiments.
As a result, much of an engineer’s or scientist’s time is spent thinking about and discussing failures and errors in their work.
Can you imagine a business or marketing manager doing this?
Case-in-point: A cosmetics manufacturer I worked for for several years conducted extensive market research (focus groups, consumer testing, etc) on a major new product line, which showed that the products were well received, easy to use, and priced appropriately.
Once in the market, however, the entire line flopped and was ultimately withdrawn from the market at a cost of many millions of dollars. Complaints were mostly that the products were difficult to use, nearly impossible to remove from the package, and far too expensive.
That marketing department made a mistake that had been repeated in the company for at least the last decade: trying to sell in middle America while doing all their market research just outside their offices in Manhattan.
Of course, rather than ask a different question, the company did what it always did when it had a flop (and they had and still have a lot of flops): they fired the marketing department.
I have worked in R&D for most of my professional career as an engineer, often working hand-in-glove with marketing and product development.
Quantitative science has an aura of credibility that is greatly desired by those who work in the “softer” fields, so most companies do run experiments of a sort, although neither a scientist nor an engineer would recognize these as anything approaching robust experiments.
I think that while companies are interested in appearing to apply science to their work, there is little real interest in (or examples of) actually doing so.
The best I think I can offer in support of this idea is the following:
– Experiments are designed to test a clearly stated theory.
– Criteria for success and for failure must be (explicitly) understood prior to conducting an experiment to avoid hindsight bias.
– Experiments that refute theories are generally more valuable that experiments that confirm theories. By learning where a theory or thinking is flawed, one can learn to ask better questions and design better experiments.
As a result, much of an engineer’s or scientist’s time is spent thinking about and discussing failures and errors in their work.
Can you imagine a business or marketing manager doing this?
Case-in-point: A cosmetics manufacturer I worked for for several years conducted extensive market research (focus groups, consumer testing, etc) on a major new product line, which showed that the products were well received, easy to use, and priced appropriately.
Once in the market, however, the entire line flopped and was ultimately withdrawn from the market at a cost of many millions of dollars. Complaints were mostly that the products were difficult to use, nearly impossible to remove from the package, and far too expensive.
That marketing department made a mistake that had been repeated in the company for at least the last decade: trying to sell in middle America while doing all their market research just outside their offices in Manhattan.
Of course, rather than ask a different question, the company did what it always did when it had a flop (and they had and still have a lot of flops): they fired the marketing department.
I really don’t like the way he does his analysis. Pie charts aren’t that strong of evidence to support his claims or to accept/deny a theory. He should have produced hard numbers for real comparison to see what happens plus analysis of changes from week to week with the same strategy. Two weeks just isn’t enough to draw conclusions.
I really don’t like the way he does his analysis. Pie charts aren’t that strong of evidence to support his claims or to accept/deny a theory. He should have produced hard numbers for real comparison to see what happens plus analysis of changes from week to week with the same strategy. Two weeks just isn’t enough to draw conclusions.
Any reader of Dilbert knows that companies run experiments. The question is whether they act on experimental results or on the voice from upstairs.
Any reader of Dilbert knows that companies run experiments. The question is whether they act on experimental results or on the voice from upstairs.
Aren’t focus groups basically experiments? I’ve never heard anything about a shortage of focus groups.
Aren’t focus groups basically experiments? I’ve never heard anything about a shortage of focus groups.