FrF2 Catalogue Findings

1Summary

We checked all 3,099 designs in the FrF2 catalogue catlg. For each entry, we built the design with FrF2::FrF2(design = name) and computed its word length pattern (WLP) from the design matrix. We then compared the result with the WLP that the catalogue stores, and with DoE.base::GWLP(). We also compared the number of generators of each entry with the number of added factors.

• In findings 2 and 3, the WLP that we computed equals DoE.base::GWLP() for all 89 entries. So the error is in the stored values, not in the designs.
• 143 entries with a stored-WLP problem or a generator-count problem (findings 1 to 5) are FrF2's default design for their number of runs and factors.
• Section 9 lists six behaviours of FrF2 design objects that a tool built on FrF2 must handle. They are not errors, but each one caused a defect in our first plan.

2Environment and method

• R 4.6.1 on Windows 11, FrF2 2.3-5 and DoE.base 1.2-5 from CRAN.
• For a two-level design with $N$ runs and $n$ factors, coded $-1$ and $+1$, let $d_{ab}$ be the Hamming distance between runs $a$ and $b$. The word length pattern is (Xu and Wu, 2001):

• A regular fraction is distance-invariant, so the script uses the distances from run 1 only:

• Three sources are compared for each entry:
  1. the stored WLP, catlg[[name]]$WLP (element $j$ is $A_j$);
  2. the WLP computed from DoE.base::desnum() of the built design;
  3. DoE.base::GWLP() of the built design, for each entry where sources 1 and 2 differ.
• Stored values that are NA are not compared.
• A design with $n$ factors in $N = 2^k$ runs has $n - k$ added factors, and needs one generator for each of them. The script compares the number of stored generators, length(catlg[[name]]$gen), with $n - k$. Entries with fewer generators (Finding 4) or more generators (Finding 5) are not in the WLP comparison of findings 2 and 3.
• The check of all entries takes about 5 minutes.

3Finding 1: stored word length patterns with NA values

• 129 entries store NA for some lengths: 28 in 256 runs and 101 in 512 runs.
• Each of them is FrF2's default design for its size.
• The values that are not NA agree with the computed WLP.

Example:

FrF2::catlg[["46-38"]]$WLP
#> 0 0 0 NA NA

Effect. The designs are correct. A program that reads the stored WLP, for example to report $A_4$ or to compare designs, gets NA. A check that compares the stored WLP with a computed WLP must skip the NA values.

4Finding 2: garbled stored word length patterns in 32 runs

• All 84 entries for 21 factors (21-16.1 to 21-16.50) and 22 factors (22-17.1 to 22-17.34) in 32 runs have a garbled stored WLP.
• From length 6 on, one number is split into two numbers, and the later values move one place.

Reproduction:

d <- FrF2::FrF2(design = "21-16.1", randomize = FALSE)
FrF2::catlg[["21-16.1"]]$WLP       # stored
unname(DoE.base::GWLP(d))[2:9]     # A_1 to A_8 from the design

Effect. The designs are correct. $A_3$, $A_4$ and $A_5$ are correct in the stored values, so the resolution and the first terms of the aberration ranking are not affected. The stored values from length 6 on are wrong.

5Finding 3: wrong last value of the stored word length pattern

Five entries in 4096 runs store a last value that differs from the computed WLP and from GWLP():

21-9.1, 22-10.1, 23-11.1 and 24-12.1 (4096 runs) show the same pattern as 20-8.1: only the last stored length is wrong.

Reproduction:

d <- FrF2::FrF2(design = "20-8.1", randomize = FALSE)
FrF2::catlg[["20-8.1"]]$WLP[13]    # 130
unname(DoE.base::GWLP(d))[14]      # 0 (A_13; GWLP() gives A_0 first)

Effect. The designs are correct, and all five are FrF2 defaults. Only the last stored value is wrong, so the resolution and the first terms of the aberration ranking are not affected.

Version 1 of this report also listed 26-17.1 here. Its stored WLP differs from the WLP of the design that FrF2 builds, but the cause is the number of generators of the entry (Finding 5).

6Finding 4: entries with fewer generators than added factors (4096 runs)

• Five entries in 4096 runs have 15 generators, but their designs need more:

• Each of them is FrF2's default for its size. FrF2::FrF2(nruns = 4096, nfactors = 28) returns a design in which every value of factor 28 is NA. FrF2 gives no error and no warning.

Reproduction:

d <- FrF2::FrF2(nruns = 4096, nfactors = 28, randomize = FALSE)
names(DoE.base::design.info(d)$catlg.entry)   # "28-16"
length(FrF2::catlg[["28-16"]]$gen)             # 15, but 28 - 12 = 16 are needed
anyNA(DoE.base::desnum(d))                     # TRUE

Effect. A user who requests 28 to 32 factors in 4096 runs gets an incomplete design. This is the only finding in which the design itself is incomplete.

7Finding 5: entries with more generators than added factors (512 runs)

• Two entries in 512 runs ($k = 9$) store 19 generators, but their designs need fewer. Both entries store the same 19 generators.

• Each of them is FrF2's default for its size. FrF2::FrF2(nruns = 512, nfactors = 26) selects 26-17.1, and nfactors = 27 selects 27-18.1. Neither call gives a warning.
• FrF2 builds the design from the first 17 (or 18) generators: added column $i$ of the design equals the product of the base columns that generator $i$ gives. So the design is a regular fraction.
• But design.info(d)$aliased names factors that the design does not have. In 26-17.1, the last factor is a, and alias chains such as Bc=RV name a factor c.
• The stored list of clear two-factor interactions (2FIs) is incomplete. 26-17.1 lists 267 clear 2FIs, but the design has 283; 27-18.1 lists 287, but the design has 297. Each listed 2FI is clear in the design.
• The stored WLP of 26-17.1 gives $A_1$ to $A_6$ = 0 0 0 6 158 548. The design that FrF2 builds has 0 0 0 7 161 539 (equal to GWLP()), and the design from all 19 generators has 0 0 0 13 236 860. Of the 171 sets of 17 of the 19 generators, only one gives the stored values: the set without generators 2 and 19.
• The stored WLP of 27-18.1 ($A_1$ to $A_6$ = 0 0 0 9 195 689) equals that of the design that FrF2 builds. Of the 19 sets of 18 of the 19 generators, only the set without generator 19 gives these values.

Reproduction:

d <- FrF2::FrF2(nruns = 512, nfactors = 26, randomize = FALSE)
names(DoE.base::design.info(d)$catlg.entry)          # "26-17.1"
length(FrF2::catlg[["26-17.1"]]$gen)                  # 19, but 26 - 9 = 17 are needed
ncol(DoE.base::desnum(d))                             # 26
tail(DoE.base::design.info(d)$aliased$legend, 1)      # "a=a": the last factor is a
grep("[bc]", DoE.base::design.info(d)$aliased$fi2, value = TRUE)[1:3]
#> "Bc=RV" "BR=CQ=Vc" "BV=Rc"
FrF2::catlg[["26-17.1"]]$WLP                          # 0 0 0 6 158 548
unname(DoE.base::GWLP(d))[2:7]                        # 0 0 0 7 161 539

Effect. A user who requests 26 or 27 factors in 512 runs gets a regular design, but its alias information names factors that do not exist, and the stored clear 2FIs leave out 2FIs that are clear. For 26-17.1, the stored WLP is not that of the design that FrF2 builds. If the minimum aberration ranking of the 512-run, 26-factor designs used the stored WLP, the rank of the built design needs a check.

8Finding 6: catalogue names

FrF2's help page (topic CatalogueAccessors) says that the number after the dot is the rank by minimum aberration (MA), and that an entry without a number is "good" but not ranked. Two groups of names do not fit this description.

1. Rank-1 ties. 19 entries in 128 runs (41 to 50 factors) carry a letter after the rank, for example 41-34.1a to 41-34.1h. Their stored WLPs are equal (41-34.1a and 41-34.1b both store 0 0 0 1648 0 70146). FrF2 selects the entry with the letter "a". The help page does not describe the letter.
2. Rank 1 outside the documented MA region. The help page lists MA designs in 256 runs for up to 36 factors. It also mentions "good" resolution IV designs with up to 80 factors in 256 runs. But 12 entries in 256 runs with 69 to 80 factors are named as rank 1: 69-61.1 to 80-72.1.

Effect. For these 31 entries, the documentation does not tell whether the design is MA. A tool should not claim minimum aberration for them without a separate check.

9Behaviour of design objects (for tool builders)

These are not errors. Each one caused a defect in our first plan, so we record them for others who build tools on FrF2.

Behaviour 5 is easy to miss: without center points the level order equals the run numbers, so the error appears only when center points are added.

10How the Fractional Factorial Design Translator handles each finding

11Reproduction

From the folder that holds the script, with R and the packages FrF2 and DoE.base installed:

Rscript frf2-findings.R

• The script prints the versions, a summary of each finding with examples, the entries whose number of generators differs from the number of added factors, and the six behaviours.
• It writes frf2-affected-entries.csv with one row for each affected entry and these columns: name, nruns, nfactors, problem, stored_wlp, computed_wlp, gwlp_agrees, default_for_size.
• The script uses only FrF2 and DoE.base. It defines the WLP formula of Section 2 itself.

12Appendix: draft note to the FrF2 maintainer (not sent)

Status: draft for the authors. Not sent. The maintainer address is in the DESCRIPTION file of FrF2. The package has no bug-report URL.

Dear Professor Grömping,

We use FrF2 2.3-5 to build designs in a research tool, and we checked every entry of catlg against the word length pattern computed from the built design and against DoE.base::GWLP(). We found the following, and we thought you would want to know:

1. 129 entries in 256 and 512 runs store NA in the word length pattern (for example 46-38).
2. The stored word length patterns of 21-16.1 to 21-16.50 and 22-17.1 to 22-17.34 (32 runs) are garbled from length 6 on. For example, 21-16.1 stores 0 0 40 220 641 160 8 3640, while the design gives 0 0 40 220 641 1608 3640 6470.
3. 20-8.1, 21-9.1, 22-10.1, 23-11.1 and 24-12.1 (4096 runs) store a nonzero last value where the design gives 0.
4. 28-16 to 32-20 (4096 runs) have 15 generators where 16 to 20 are needed. FrF2(nruns = 4096, nfactors = 28) returns a design with a column of NA, without a warning.
5. 26-17.1 and 27-18.1 (512 runs; the defaults for 26 and 27 factors) have 19 generators where 17 and 18 are needed. FrF2 builds the design from the first generators, but design.info(d)$aliased names factors that the design does not have (for example the chain Bc=RV for 26-17.1, whose last factor is a), and the stored clear 2FIs leave out 2FIs that are clear in the design. The stored word length pattern of 26-17.1 (0 0 0 6 158 548) is not that of the design that FrF2 builds (0 0 0 7 161 539); it equals $A_1$ to $A_6$ of the design from generators 1 and 3 to 18.
6. The help page does not describe the letter in names such as 41-34.1a, nor the rank-1 names 69-61.1 to 80-72.1 in 256 runs, which lie beyond the 36 factors that the page lists for MA designs in 256 runs.

A script that reproduces each point and a CSV of the affected entries are attached. Thank you for FrF2; the catalogue made our tool possible.

With kind regards, Alan R. Vazquez and Fadel M. Megahed

13References

• Groemping, U. (2014). R package FrF2 for creating and analyzing fractional factorial 2-level designs. Journal of Statistical Software, 56(1), 1-56.
• Xu, H., and Wu, C. F. J. (2001). Generalized minimum aberration for asymmetrical fractional factorial designs. Annals of Statistics, 29(4), 1066-1077.