Research Methods Every Architect Should Know

Architects make claims constantly: this design will scale, that approach will be easier to maintain. Many of those claims go untested. Borrowing a few methods from empirical research turns opinions into evidence and makes decisions defensible. You do not need statistical machinery for most of it, only discipline about questions, comparisons and what counts as a result.

Match the method to the question

| Question type | Method | Example | |---|---|---| | How does it behave under load? | Benchmark or load test | Throughput of two queue brokers | | Does it work in our context? | Spike or prototype | Can the ORM handle our access patterns? | | What happened and why? | Case study, postmortem | Why did the cache stampede occur? | | Which option is better overall? | Trade-off analysis | ATAM-style scenario evaluation | | Does this practice help? | Before-and-after study | Lead time before and after trunk-based development | | What do users or teams experience? | Interviews, surveys | Developer friction in the deploy pipeline |

Choosing the wrong method is a common failure. A benchmark cannot tell you whether the team can operate the system.

Benchmarking done properly

A trustworthy benchmark has a hypothesis, a realistic workload, and a controlled setup.

• Use production-shaped data and access patterns. Uniform random keys flatter caches.
• Warm up, then run long enough to observe garbage collection, compaction and thermal effects.
• Report percentiles, such as p50, p95 and p99, plus throughput at saturation.
• Repeat runs and report variance. A single run is an anecdote.
• Change one variable at a time, and tune the baseline fairly.
• Beware coordinated omission: load generators that wait for slow responses before sending the next request understate latency.

Spikes and prototypes

A spike is a time-boxed experiment to answer a specific unknown. Write the question before starting, for example "Can we sustain 5,000 writes per second on this schema with p99 under 50 ms?" End with a written answer and the code thrown away or clearly labelled. Spikes fail when they quietly become production.

Case studies and postmortems

A case study examines a real system in depth. To avoid storytelling, structure it:

Context: what the system and constraints were
Event: what happened, with timestamps
Contributing factors: technical and organisational
Evidence: logs, metrics, traces
Counterfactuals: what would have prevented or limited it
Actions: changes, owners, how we will verify

Case studies cannot prove causation, but they generate hypotheses and reveal mechanisms that metrics miss. Separating contributing factors from a single root cause reflects how real failures behave.

Structured trade-off analysis

For decisions with several competing qualities, use scenario-based analysis. The SEI's Architecture Tradeoff Analysis Method is a formal version. A lightweight form:

1. List the quality attribute scenarios that matter, for example "regional outage: users in other regions see no degradation".
2. Describe how each option responds.
3. Identify sensitivity points (small changes with big effects) and risks.
4. Weight scenarios by business importance, then compare.

The value is less in the final score than in surfacing disagreements about priorities early.

Before-and-after and observational studies

When evaluating a practice, record baseline metrics first. Useful engineering measures include lead time for changes, deployment frequency, change failure rate and time to restore, as popularised by DORA research. Be aware of confounders: if you introduced a new pipeline and hired two senior engineers the same quarter, you cannot attribute improvement to either.

Interviews and qualitative methods

Numbers will not reveal why teams bypass the platform. Short structured interviews with five to eight engineers often expose the dominant friction. Ask about specific recent events rather than opinions, and look for patterns across interviews instead of quoting the most vivid one.

Threats to validity, in plain terms

• Selection bias: you examined only the successes.
• Confounding: something else changed at the same time.
• Measurement error: the metric does not capture what you care about.
• Generalisation: it worked for one team or domain, and may not elsewhere.

State these threats in your write-up; naming them makes conclusions stronger.

Key takeaways

• Choose the method that fits the question: benchmark, spike, case study, trade-off analysis, before-and-after, or interviews.
• Good benchmarks use realistic workloads, warm-up, repetition, percentiles and fair baselines.
• Time-box spikes and write down the answer; do not let them become production.
• Structure postmortems around contributing factors and evidence.
• Record baselines before changing practices and watch for confounders.
• Always state threats to validity alongside your conclusions.