Performance Testing Framework

Performance testing is a family of experiments, each answering a different question. A framework fixes the targets, the workload, the environment, and the comparison method so that results are repeatable and meaningful.

Performance work is most effective when it is continuous. A small smoke load test on each merge, a nightly baseline, and a quarterly stress test provide layered feedback without requiring a dedicated performance team for every change.

How it works

Test types differ by shape. Load tests apply expected peak traffic to validate objectives. Stress tests increase load until failure to find the breaking point and failure mode. Soak tests run for hours at moderate load to reveal leaks and slow degradation. Spike tests apply sudden surges to test autoscaling and queues. A workload model describes request mix, arrival rate, think time, and data distribution, derived from production logs. Open-model arrival, where requests arrive regardless of response time, avoids the coordinated omission that hides latency.

Implementation steps

1. Set targets as percentiles, for example p95 under 300 ms at 500 requests per second with error rate below 0.1 percent.
2. Build the workload model from production traces and logs.
3. Prepare an environment sized and configured like production, with realistic data volume.
4. Script scenarios as code and store them with the service.
5. Run a baseline, then add the test to CI with thresholds that fail the build.
6. Observe resource saturation, queue depth, and traces during runs.
7. Fix the dominant bottleneck, re-run, and record the change.
8. Schedule larger stress and soak tests before major events.

Common pitfalls

• Testing against empty databases, which hides slow queries.
• Load generators saturating before the system does.
• Comparing runs with different data or configuration.
• Reporting only averages.

Metrics to watch

Latency percentiles (p50, p95, p99), throughput, error rate, CPU and memory saturation, connection pool utilization, garbage collection pauses, queue depth, and the delta against the previous baseline.

Store results in a time-series system so trends are visible. A slow upward creep in p99 across ten releases is easy to miss in a single report but obvious on a chart, and it identifies exactly where to investigate.

Example load test script

import http from "k6/http";
import { check } from "k6";

export const options = {
  scenarios: {
    peak: {
      executor: "constant-arrival-rate",
      rate: 500, timeUnit: "1s", duration: "15m",
      preAllocatedVUs: 200, maxVUs: 800,
    },
  },
  thresholds: {
    http_req_duration: ["p(95)<300"],
    http_req_failed: ["rate<0.001"],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/api/products`);
  check(res, { ok: (r) => r.status === 200 });
}

Key takeaways

• Define targets and workload before writing scripts.
• Use arrival-rate load and percentile thresholds.
• Compare to a baseline in CI to catch regressions.
• Match production data and configuration, or results mislead.