Why CI took three minutes or fifteen

CI runs for this blog ranged from three minutes on a warm runner to nearly sixteen minutes on a cold one. The wall time wasn't the real problem; the variance was. A pipeline that takes three minutes or fifteen minutes, depending on runner state, feels unreliable even when every run is passing.

I wanted consistency first, speed second. The investigation started with the wall of apt-get output that preceded every accessibility test run.

Caching the browser binaries#

Playwright stores browser binaries in ~/.cache/ms-playwright. Puppeteer stores its binaries in ~/.cache/puppeteer. Neither path was cached, so every CI run downloaded browsers regardless of whether anything had changed.

The fix is the same for both: add an actions/cache@v5 step immediately before the browser install, keyed on the relevant lockfile.

- name: Cache Playwright Browsers
  uses: actions/cache@v5
  with:
    path: ~/.cache/ms-playwright
    key: ${{ runner.os }}-playwright-${{ hashFiles('pnpm-lock.yaml') }}
    restore-keys: |
      ${{ runner.os }}-playwright-
 
- name: Install Playwright Browsers
  run: pnpm exec playwright install chromium

I got the ordering wrong on the first attempt. The Playwright cache step was placed after the build step, so the restore ran too late to help the install. The cache still saved correctly at the end of the job, which made it look like caching was working. It took a second look at the step timings to spot the problem. The install step was still downloading browsers on every run, even though the next run showed a cache hit on restore.

The cache key matters too. The initial key used hashFiles('**/pnpm-lock.yaml'), a glob that matched both the root workspace lockfile and the Lambda's separate lockfile. A Lambda dependency bump would invalidate the browser cache even though browser versions are governed only by the root lockfile. Scoping the hash to just hashFiles('pnpm-lock.yaml') fixed the false invalidations.

The Puppeteer browser install was also using pnpm dlx instead of pnpm exec. Since Puppeteer is an explicit workspace dependency, dlx would fetch the latest version at runtime rather than using the pinned version from the lockfile. Switching to pnpm exec puppeteer browsers install chrome-headless-shell fixed that.

Dropping WebKit#

The Playwright config was running tests across two browser engines: Chromium and WebKit. Two engines roughly doubled both install time and test execution time.

For this site's accessibility checks, the axe-core violations were effectively identical across Chromium and WebKit. The tests that did differ (keyboard navigation, focus management) had accumulated significant WebKit-specific workarounds: Alt+Tab branches, browserName conditionals, and a post-navigation focus retry that existed entirely to handle Safari quirks.

I dropped WebKit from CI and test it locally when making changes that touch keyboard behaviour. That does reduce cross-engine coverage, but for this site I decided the stability and simplicity were worth more than always testing Safari quirks on every push. After removing it from the Playwright config, the dead code came out too: three browserName parameters, multiple if (browserName === 'webkit') branches, and a multi-step focus detection workaround.

What --with-deps is actually doing#

Even with a browser cache hit, the Playwright install step was still taking around twenty seconds. The cache hit was visible in the log (249 MB restored successfully) but the step duration stayed high.

Reading further into the log explained it. The --with-deps flag tells Playwright to run apt-get to verify and install system-level dependencies alongside the browser binary. It runs on every invocation regardless of whether the binary is already present.

Most packages in the output reported "already the newest version". The only newly installed packages were fonts: fonts-ipafont-gothic, fonts-wqy-zenhei, and several others used for rendering. The GitHub Actions ubuntu-latest runner already ships with the core Chromium system dependencies.

Removing --with-deps skipped the apt step entirely. An install step that takes twenty seconds after a confirmed cache hit is worth investigating. In this case, the CI log made it clear that the time was spent in apt-get, not in Playwright itself.

The result#

After these changes, warm-cache runs settle around two minutes. Cold-cache runs come in around seven minutes rather than fifteen. The warm-cache improvement is modest; the more meaningful number is the cold-cache floor. That improvement comes from the structural changes (dropping WebKit, removing --with-deps), not from caching. Those fixes help every run regardless of cache state.

The remaining time is mostly in the Next.js static build itself. There may be room to parallelise the build and test steps, but the more useful result was not the best-case run time but the narrower range. The pipeline now behaves predictably enough to trust.