There are some quotes about design of experiments on the inside of Box, Hunter, and Hunter (BHH). Here are the ones that I enjoy the most. I think I extracted about 25% of the quotes that are in the book.
Only in exceptional circumstances do you need or should you attempt to answer all questions with one experiment.
I like this. BHH are talking about sequential experimentation. They recognize the difficulty of getting things right the first time. And the benefit of optimizing the experiment to the goals of the study. When there are multiple goals, multiple experiments can be good.
The business of life is to find out what you don’t know from what you do. – Duke of Wellington
The best time to plan an experiment is after you’ve done it (RA Fisher)
Yes, often we see the flaws only after we see the data (or the scope of data collection)
It is the data that are real.
This is a Bayesian take to me, but I don’t think that is the intent of the quote.
Don’t fall in love with a model.
Yes, taking a model too seriously is just one way of not taking it seriously at all. It’s dangerous to put too much emphasis on the model because we have less incentive to falsify that model, and because we might do something stupid with it (like extrapolation).
It is not unusual for a well designed experiment to analyze itself.
Yes, a factorial experiment can be analyzed without any software or statistical cleverness.
To find out what happens when you change something, it is necessary to change it.
Reminds me of the phrase “no causation without manipulation”.
An engineer who does not know experimental design is not an engineer (Toyota Motor Company executive)
Where there are three or four machines, one will be substantially better or worse than the others (Ellis Ott)
Discovering the unexpected is more important than confirming the known.
One must try by doing the thing. For though you think you know it, you have no certainty until you try. (Sophocles)
Among the factors to be considered there will be the vital few and the trivial many. (Tukey)
I disagree with this simplification, but it was probably useful in Tukey’s day.
Not everything that can be counted counts and not everything that counts can be counted. (Einstein)
Seek computer programs that allow you to do the thinking.
I agree, but I don’t think there is one. All computer software for statistics is so “good” now, an analyst can go on autopilot and easily do something stupid. I include R in that bucket.
Many useful time series models can be regarded as recipes for transforming serial data into white noise.
A factorial experiment makes every observation do double duty. (Jack Youden)
I don’t know what it means… but factorial experiments are optimal when the randomization and data are cheap.