The XCUITest Speed-Ups We Measured and Dropped

Parts 8 to 10 covered the optimisations that paid. This part covers three that did not, and it is arguably the most useful of the five, because each of the three looks like an obvious win from a reading of the code. Each was priced from the result bundle against a condition written down in advance, and each was dropped when the measurement came in well under the condition. The method for pricing a candidate is the subject; the three candidates are the worked examples. The part closes with what actually makes result bundles large, a change that made the suite slower on purpose, and ten rules that summarise the arc.

Pricing a candidate before building it

A performance candidate usually arrives with a number attached: a timeout, a call-site count, a semaphore budget. None of those is a cost. A timeout is a bound on the worst case; a call-site count is a bound on exposure. The cost is what the path actually takes on a typical run, and it can be read from the result bundle's activity tree before any code is written.

1. Write the drop condition first: the share of setUp or of the run below which the candidate is not worth its risk. In this suite the threshold for a setUp-path candidate was 20 percent of setUp.
2. Segment the setUp from the activity tree's timestamps: from the start of setUp to the launch, from the launch to the first landing probe, from the probe to the first test-body activity. Validate the parse on one method against xcresulttool's own activity listing before trusting it on a run.
3. Where a cost should scale with something countable, check the ordering. If launches with more fixtures do not take longer, the fixtures are not on the path being timed.
4. Price the ceiling: what the whole run would save if the candidate removed the cost entirely. A candidate whose ceiling is under 1 percent of the run is not worth a phase.

Candidate one: caching provisioning per class

Every test provisions its tenant over HTTP before it launches: mint a token if needed, create or reset the test user and company, read back the membership. The code guarding that path waits on a semaphore with a 30 second timeout, and the first cost estimate priced it as a 30 second blocking request per test. Measured, the blocking provisioning span was a median 0.79 seconds of a 39.0 second setUp on the pilot classes, 2.1 percent; 2.8 percent pooled across 264 launched methods; 17.3 percent on the single worst method; and no method reached the 20 percent condition. Under the per-class lifecycle of part 8, a reusing method already pays 0.00 seconds for it. The ceiling for a cache was 64.7 seconds of 22,486, 0.29 percent, on a change that would have skipped nine places where classes write per-instance state in the provisioning hook. Dropped.

Candidate two: keeping the session across relaunches

A typed login is 33.1 of a 39.1 second setUp, so a relaunch that preserved the signed-in session looked like the natural complement to the per-class lifecycle. The condition had two halves: enough fallback relaunches from the per-class reset, and enough classes that opt out and relaunch anyway. Measured, the fallback rate was one in 58 reuse attempts, about 1.2 per run. The opt-out surface came to about 212 seconds net per run, 0.94 percent, and seven of those relaunches belonged to one pseudolocalisation class.

The measurement also corrected the design. The launch flag believed to sign each launch out was not the mechanism: the app cleared its session on every UI-testing launch, flag or no flag, and the flag believed to preserve the session preserved tokens only. None of the baseline's 264 launched setUps had ever taken the already-signed-in short-circuit. What looked like extending an existing seam was a new app-side change on the product's own sign-out path, for a gain smaller than the run's own variance. Dropped.

Candidate three: seeding fixtures once per class

Sixty classes seed fixtures through sixteen launch arguments, and re-establishing them on every launch looked expensive. No artefact timed a seed directly, because the app's own log lines did not persist in the unified log across the run windows, so the cost was priced as a group difference on the setUp span the seeds run in, from the company-picker tap to the first test activity: 12.62 seconds on seed-bearing launches against 12.04 on seed-free ones. That is a difference of 0.58 seconds, with a standard deviation of 1.28 seconds on the seed-free population alone.

The ordering check settled it before the timing argument did. One-fixture seed sets ranged from 11.97 to 12.84 seconds, a seven-fixture set took 13.16, and a class with five seed arms took 11.90. Unordered by fixture count means not on the critical path, and a read of the code confirmed why: every seed runs in a detached task after login, behind a per-process latch, and nothing on the login path awaits it. The app already seeded once per process, and the per-class lifecycle had already made the process the class. Dropped.

Do

• Keep the dropped candidates in the record with their numbers and the condition they failed. Without that, the next person re-derives them.
• Price from the activity tree; validate the parser against xcresulttool on one method to within a hundredth of a second before using it on a run.
• Run the ordering check before the timing argument whenever the cost should scale with a count.

Don't

• Don't rank a candidate by a timeout, a semaphore budget or a call-site count.
• Don't assume a launch flag does what its name says. Read the app-side gate it reaches.
• Don't trust the app's own logging as a timing source on macOS unless it has been shown to persist for the run window.

What makes a result bundle large

The semantic screenshot captures kept for an AI vision judge looked like a large cost class: 229 call sites. Measured, each capture is about 0.42 megabytes and at most about half a second; the 223 that fired on a full run cost about 96 megabytes and about 90 seconds of 22,486. What made the bundle large was the four tests that hung or failed: 44.4 percent of the exported bytes. A test that ran to the 600 second allowance kept 313 automatic UI snapshots (76.8 MB) from a scroll hunt, and one failed test kept a 70.5 MB screen recording. A second, unexpected family was 41 whole-app screenshots with no rubric attached, 125.8 MB, more than every judged capture combined; that was passed to the suite owners as a finding.

• Do: read attachment sizes per test from the bundle before deciding captures are the cost.
• Don't: remove judged captures to shrink a bundle. Remove the hangs, and audit unjudged whole-app screenshots.

A change that made the suite slower

Not every change in a performance phase should be faster. The iPad reveal loops scrolled with the two-argument press-and-drag, which is a fling: 284 points of content per 103 points of finger travel, at a ratio that differs per scroll view, so no reveal loop could size its step. They were replaced with slow, bounded drags whose content step is limited by the finger step, and every exit now records how many steps it took and why it stopped. Coarse flings went from two to zero, recorded reveals from zero to eight, and five planted blocked destinations each ended at their named assertion. The runs got 35.7 seconds slower. The change was accepted as a correctness change, its cost was reported, and it was never offset against another change's saving.

The whole-run number

A performance arc is usually summarised by one whole-run wall clock, and this one does not have a single-invocation figure for the macOS target on the final tree. The verification run was stopped from outside at 253 of 315 tests, leaving a bundle the instrument could not read, and the remaining 31 classes ran as a separate tail on byte-identical products. The two invocations sum to 22,716 seconds against the baseline's 22,486, which is not a comparison: the first ran beside heavy foreign load and four 600 second timeouts, three of which passed in the tail on the same code. What the record does establish: the per-test median fell from 58.6 to 55.4 seconds over the 245 tests both runs completed, all 23 per-class classes held their savings, and no class ran fewer tests or skipped more than the baseline. The single wall clock is owed to one uncontended run, and it is better left owed than filled with a number that cannot be re-derived.

Ten rules for UI-test performance

1. Build the measuring instrument before the first fix, and make it refuse: zero parsed tests, missing durations and a contended host are refusals, not greens.
2. Accept every change on pass-set identity by test id. An exit code and a ran-and-asserted count cannot see a failure.
3. Treat host load as a share of capacity, and never claim a wall clock measured beside foreign work.
4. Price a candidate from the result bundle's activity tree. A timeout budget or a site count is a ceiling, and the four candidates ranked that way here measured at 5 percent of it or less.
5. Launch once per class where the class allows it, opt in one class at a time, and log every fallback relaunch.
6. Skip wrong-lane tests before super.setUpWithError(), so a skip never pays for a launch.
7. Put a positive landing above every negative wait, keep the negative at one or two seconds, and prove it red at its own line.
8. Disable animations from inside the app, process-scoped and never persisted, and measure the arm by idle-wait duration.
9. Parallelise by shared state, not by speed; the serial remainder sets the ceiling.
10. Keep the dropped candidates in the record with their numbers, so nobody measures them twice.

That closes the series. Parts 1 to 6 are about making a UI-test suite tell the truth; parts 7 to 11 are about making it faster without letting it stop. The principle is the same in both halves: a result that cannot be re-derived today is a claim, whether it says green, red, or 30 percent faster.

Previously (Part 10): Animations Off and Parallel Simulators: Measured XCUITest Speed-Ups

Start of the performance arc (Part 7): Measuring an XCUITest Suite Before You Speed It Up