Rethink experiment design

The traditional approach to experimentation, often referred to as the “scientific method,” requires changing only one factor at a time (OFAT). This should not be confused with the full factorial method (Table 1).

Table 1. An example of a full-factorial test matrix with 2 factors.
1. Average of minimum of two replicates at certain conditions.
2. Factors are variables chosen for degree of influence on response.

The OFAT approach not only suffers from being extremely inefficient, but more importantly, it cannot detect interactions of factors – much less map a complex response surface. For example, consider the effects of two factors “A” and “B”, such as temperature and pressure, on a response — for example, chemical yield. Let’s say these factors affect the response as shown by the surface in Figure 1.

Figure 1. The real contour of the response to factors A and B: interactions are important.

Unfortunately, an experimenter who adheres to the old-fashioned scientific method of OFAT can see things only one dimension at a time. The points on Figure 1 outline the path taken by OFAT done first on factor A from left to right. The results can be seen in Figure 2. They look good.

Figure 2. This is how OFAT sees the relationship between the response variable and factor A.

Next the OFAT experimenter sets A at the value where response is maximized and then varies factor B. Figure 3 shows the result. The experiment cries “Eureka — I have found it!” However, it is clear from our perspective in Figure 1 that the outcome falls far short of the potential increase in response.

Figure 3. This is how OFAT sees the relationship between the response variable and factor B.

Minimum-run designs

By simply varying factors only at two levels each, but simultaneously rather than one at a time, experimenters can uncover important interactions such as the one depicted in Figure 4.

Figure 4. Response surface shows a pure interaction of two factors.

Furthermore, this parallel testing scheme is much more efficient (cost-effective) than the serial approach of OFAT. For up to four factors, the number of runs required to do all combinations, 16 (= 24), may not be prohibitive. These full factorials provide resolution of all effects. However, as a practical matter all you really need to know are the main effects (MEs) and two-factor interactions (2FIs) — higher-order effects are so unlikely that they can be assumed to be negligible. Making this assumption (ignoring effects of 3FI or more) opens the door to fractional two-level combinations that suffice for estimating MEs and 2FIs.

Statisticians have spelled these out more than half a century ago. Our previous article on DOE in Chemical Processing [1] details the classical high-resolution fractional design options for up to 11 factors. They can be constructed with the aid of a textbook on DOE [2].

If you dig into traditional options for experiment designs, you will find designs available with as little as k+1 runs, where k equals the number of factors you want to test.  One popular class of designs, called Plackett-Burman after the two statisticians who invented them during World War II, offers experiments that come in multiples of four, for example, 11 factors in 12 runs or 19 factors in 20 runs. However, these “saturated” designs provide very poor resolution — main effects will be confused with 2FIs. We advise that you avoid running such low resolution designs. They seemingly offer something for nothing, but the end result may be similar to banging a pipe wrench against a noisy pump — the problem goes away only temporarily.

Pulling the pump apart and determining the real cause is a much more effective approach. Similarly, you will be better off running a bigger design that at least resolves main effects clear of any 2FIs. Better yet would be an experimental layout that provides solid estimates of the 2FIs as well. Taking advantage of 21st century computer technology, statisticians developed a very attractive class of designs that require minimum runs, or nearly so (some have an extra one to maintain an equal replication of the lows versus the highs of each factor). [3]. The inventors deemed these designs “MR5,” where MR stands for minimum run and the number 5 refers to the relatively high resolution — sufficient to clearly estimate not only main effects, but also the 2FIs. Table 2 shows the number of runs these MR5s require for 6 to 11 factors — far fewer than full factorials or even the standard fractions.