For the last few years I’ve been trying to convince businesses to run experiments in order to learn how to do things better. Why is it that experimentation is the gold standard in science, but rarely exploited in corporations? My own hunch is that the main reason is what economists call “path dependence” — in other words, businesses don’t run experiments because they rarely have in the past. If, by chance, Henry Ford‘s innovation had been to run experiments rather than to develop the automated assembly line, experiments would be commonplace today in business.
A former student of mine named Ken Kovash, who now works at Mozilla, provides a nice example of how the simplest of experiments can provide answers that are otherwise elusive. As Ken describes in greater detail on the Mozilla blog, one of the ways Mozilla acquires new customers is through pay-per-click ads on search engines. The question Mozilla had is the following: if someone types “firefox” into a search engine, usually the first result they will see is the Mozilla site, so does it really do Mozilla any good to pay search engines to do featured links? Do ads actually generate more traffic, or do they just shift customers around — e.g., instead of getting the customers free, Mozilla ends up paying the search engine because of the pay-per-click ads? Without performing an experiment of some kind, this is a hard question to answer.
So over a two-week period, Mozilla experimented with turning their pay-per-click ads on and off more or less at random. The findings, as Ken reports, are somewhat mixed. Looking at the data one way, it appears that two-thirds of the customers who normally come to Mozilla through pay-per-click ads would get there anyway. On the other hand, the absolute number of downloads was substantially higher when the paid ads were running. This suggests either that (1) their treatment and control periods were different for an unknown reason; or (2) that the pay-per-click ads lead people to download more often through other channels. My guess is that (1) is more likely to be the explanation. But how can one really be certain?
The answer: by running more experiments.
and 
. You can also watch it on 
I think your premise that companies don’t run experiments is false. What are focus groups? What is R&D spending? What is price differentiation for a product (based on time, color, style, etc.) I’m also very sure large corporations analyze their advertising spending, which is exactly what was tested in your example.
I think your premise that companies don’t run experiments is false. What are focus groups? What is R&D spending? What is price differentiation for a product (based on time, color, style, etc.) I’m also very sure large corporations analyze their advertising spending, which is exactly what was tested in your example.
Businesses do run experiments, but not always the way you think of in science – concurrent studies with controls.
Businesses don’t want to do expensive experiments and they don’t want to do experiments that will disrupt a revenue source.
But when a business is failing they are more likely to change things around and experiment with different things.
Businesses do run experiments, but not always the way you think of in science – concurrent studies with controls.
Businesses don’t want to do expensive experiments and they don’t want to do experiments that will disrupt a revenue source.
But when a business is failing they are more likely to change things around and experiment with different things.
Corporations do do experiments, but rarely publish the results (because it is rarely in their interests to do so, the results form competitive advantage).
There are exceptions to this, in my own field of software development one of the newer programming methodologies – Agile/Extreme Programming – started at Chrysler and, coincidentally, also spawned wikis.
Sometimes companys themselves *are* experiments, such as Cygnus and RedHat experimenting in commercialising Open Source software.
Not to mention the direct, hard science experiments done by the likes of Biotech companys.
I think a more interesting question is how to get companys to publish the results of these experiments?
Corporations do do experiments, but rarely publish the results (because it is rarely in their interests to do so, the results form competitive advantage).
There are exceptions to this, in my own field of software development one of the newer programming methodologies – Agile/Extreme Programming – started at Chrysler and, coincidentally, also spawned wikis.
Sometimes companys themselves *are* experiments, such as Cygnus and RedHat experimenting in commercialising Open Source software.
Not to mention the direct, hard science experiments done by the likes of Biotech companys.
I think a more interesting question is how to get companys to publish the results of these experiments?
“[H]ow can one really be certain? The answer: by running more experiments.”
Not so. Certainty, in the philosophical sense of the word, is inaccessible to empirical analysis. In reasoning from effects to cause, one cannot in general guarantee that all the relevant data have been accounted for and that the model is correct.
Of course, one may also argue that certainty is unavailable in all realms. But how can we be certain of this?
“[H]ow can one really be certain? The answer: by running more experiments.”
Not so. Certainty, in the philosophical sense of the word, is inaccessible to empirical analysis. In reasoning from effects to cause, one cannot in general guarantee that all the relevant data have been accounted for and that the model is correct.
Of course, one may also argue that certainty is unavailable in all realms. But how can we be certain of this?