What Is Orthogonal Array Testing?
Orthogonal Array Testing (OAT) uses mathematical structures called orthogonal arrays to generate test suites. These arrays guarantee that every pair of parameter values appears an equal number of times across all test cases, providing balanced, uniform coverage.
Origins: Taguchi Method
OAT originated from the Taguchi method in manufacturing quality engineering. Dr. Genichi Taguchi developed orthogonal arrays to efficiently test the impact of multiple factors on product quality. Software testing adopted this technique for combinatorial test design.
Understanding Orthogonal Arrays
An orthogonal array is a matrix where:
- Rows = test cases
- Columns = parameters (factors)
- Values = levels of each parameter
The key property: for any two columns, every possible pair of values appears the same number of times.
Notation: L_n(k^f)
- L = Latin square (the array type)
- n = number of rows (test cases)
- k = number of values per parameter (levels)
- f = number of parameters (factors)
Example: L4(2^3) — 4 test cases, 3 parameters, 2 values each
| Test | P1 | P2 | P3 |
|---|---|---|---|
| 1 | 0 | 0 | 0 |
| 2 | 0 | 1 | 1 |
| 3 | 1 | 0 | 1 |
| 4 | 1 | 1 | 0 |
Check any two columns: every pair (0,0), (0,1), (1,0), (1,1) appears exactly once.
Common Orthogonal Arrays
| Array | Tests | Parameters | Values | Exhaustive |
|---|---|---|---|---|
| L4(2^3) | 4 | 3 | 2 | 8 |
| L8(2^7) | 8 | 7 | 2 | 128 |
| L9(3^4) | 9 | 4 | 3 | 81 |
| L16(2^15) | 16 | 15 | 2 | 32,768 |
| L18(3^7) | 18 | 7 | 3 | 2,187 |
| L25(5^6) | 25 | 6 | 5 | 15,625 |
OAT vs. Pairwise Testing
| Aspect | OAT | Pairwise |
|---|---|---|
| Coverage | Every pair appears equally | Every pair appears at least once |
| Balance | Perfectly balanced | May be unbalanced |
| Flexibility | Fixed array sizes | Adaptive to any parameter count |
| Mixed values | Harder with mixed levels | Handles easily |
| Optimality | Mathematically optimal | Near-optimal (heuristic) |
Selecting the Right Array
Step 1: Count your parameters and values per parameter. Step 2: Find the smallest standard array that fits. Step 3: If parameters have different numbers of values, use mixed-level arrays or the “collapse” technique.
Example: 4 parameters with 3 values each → L9(3^4), which gives 9 test cases instead of 81.
Advanced OAT Techniques
Real-World Example: Browser Compatibility
Parameters:
- Browser: Chrome, Firefox, Edge (3 levels)
- OS: Windows, macOS, Linux (3 levels)
- Resolution: 1080p, 1440p, 4K (3 levels)
- Language: EN, ES, RU (3 levels)
Using L9(3^4):
| Test | Browser | OS | Resolution | Language |
|---|---|---|---|---|
| 1 | Chrome | Windows | 1080p | EN |
| 2 | Chrome | macOS | 1440p | ES |
| 3 | Chrome | Linux | 4K | RU |
| 4 | Firefox | Windows | 1440p | RU |
| 5 | Firefox | macOS | 4K | EN |
| 6 | Firefox | Linux | 1080p | ES |
| 7 | Edge | Windows | 4K | ES |
| 8 | Edge | macOS | 1080p | RU |
| 9 | Edge | Linux | 1440p | EN |
9 tests instead of 81. Every pair of values from any two parameters appears exactly once.
Mixed-Level Arrays
When parameters have different numbers of values, use mixed-level orthogonal arrays:
L18(2^1 x 3^7): 18 tests for 1 binary parameter + 7 three-level parameters.
For parameters with fewer levels than the array supports, map multiple array values to the same parameter value (collapsing).
Strength Beyond 2
Standard OAT provides strength 2 (all pairs). Higher-strength arrays cover triples, quadruples, etc.:
- Strength 2: Every pair covered → catches 2-way interaction defects
- Strength 3: Every triple covered → catches 3-way interaction defects
- Strength t: Every t-tuple covered
Higher strength = more tests but catches rarer interaction defects.
When OAT Outperforms Pairwise
- Statistical analysis. OAT’s balanced coverage allows statistical analysis of which parameters most affect the outcome.
- Regulated industries. When you need mathematically provable coverage, OAT’s properties are auditable.
- Hardware testing. Where test execution is expensive, OAT’s optimal test count matters.
Tools for OAT
- NIST covering arrays repository — pre-computed arrays
- Jenny — open-source covering array generator
- ACTS (NIST) — Automated Combinatorial Testing for Software
- Manual lookup — standard arrays are published in textbooks and online catalogs
Exercise: Apply OAT to a Configuration Test
Scenario: You need to test a database configuration with:
- DB Engine: MySQL, PostgreSQL, SQLite (3)
- Connection Pool: 5, 20, 100 (3)
- Cache: Enabled, Disabled, Partial (3)
- Logging: Debug, Info, Error (3)
Tasks:
- What orthogonal array fits these parameters?
- How many test cases does it generate vs. exhaustive?
- Fill in the test matrix using L9(3^4)
- Verify that any two columns contain all 9 pairs exactly once
Hint
4 parameters x 3 values each = L9(3^4). Exhaustive = 3^4 = 81. Map the standard L9 array to your specific parameter values.
Solution
Array: L9(3^4) — 9 tests instead of 81 (89% reduction).
Standard L9 with mapped values:
| Test | DB Engine | Pool | Cache | Logging |
|---|---|---|---|---|
| 1 | MySQL | 5 | Enabled | Debug |
| 2 | MySQL | 20 | Disabled | Info |
| 3 | MySQL | 100 | Partial | Error |
| 4 | PostgreSQL | 5 | Disabled | Error |
| 5 | PostgreSQL | 20 | Partial | Debug |
| 6 | PostgreSQL | 100 | Enabled | Info |
| 7 | SQLite | 5 | Partial | Info |
| 8 | SQLite | 20 | Enabled | Error |
| 9 | SQLite | 100 | Disabled | Debug |
Verification (DB Engine x Pool):
- MySQL-5, MySQL-20, MySQL-100 (each once)
- PostgreSQL-5, PostgreSQL-20, PostgreSQL-100 (each once)
- SQLite-5, SQLite-20, SQLite-100 (each once) All 9 pairs appear exactly once. Same holds for every other column pair.
Pro Tips
- Use OAT when balance matters. If you need to statistically analyze which factor causes failures, OAT’s balanced design is essential.
- Combine with BVA for value selection. OAT tells you which parameter combinations to test; BVA tells you which specific values to use for each level.
- Fall back to pairwise for irregular parameter sets. If you have 12 parameters each with different numbers of values, pairwise tools like PICT are more practical than finding a matching orthogonal array.
- Know the standard arrays. L4, L8, L9, L16, L18, L25, L27 cover most practical scenarios. Keep a reference table handy.
- OAT reduces cost in hardware testing. When each test run costs real money (physical devices, cloud resources), the mathematical optimality of OAT saves budget.