Diagnosing a Red XCUITest: Environment, Order, and Process

This closes the series. Parts 1 to 5 covered strategy, harness, identifiers, mechanics, oracles and isolation. What remains is the part that decides whether any of it survives contact with a real red: the environment the tests run in, the order in which to suspect things, and the process that keeps the corpus honest over a year.

Environment and host hygiene

The iOS UI-test project drives the installed app. Its xctestrun has no target app path; build-for-testing builds only the runner, and the app is resolved by bundle id at launch. So an iOS run launches whatever build is on that simulator, however old. Five days stale, once. Every iOS and iPad leg needs its own app build and simctl install on every simulator used, and the installed binary's modification time belongs in the run write-up. Prove freshness with a positive control by grepping the debug dylib for a literal your change added; never the app executable, which is a stub that answers zero either way. And never pass CODE_SIGNING_ALLOWED=NO: it strips entitlements, the Keychain save fails, and login never completes.

Rule out sibling simulator contention before any iOS red. Two sessions targeting one simulator produced four runs killed mid-flight at wandering points and two reproducible assertion failures at an untouched line; every one was contention, because the runner bundle id collides and the second run displaces the first. A signal kill is never an assertion failure, whatever the terminal marker says.

On macOS, Timed out while enabling automation mode is zero tests, not a failure, and it has at least two causes: a lapsed Accessibility grant for the runner, and a locked or asleep screen. It can appear mid-lane: two tests that passed minutes earlier failed in three seconds on the same machine. Re-run exactly once; a second consecutive zero-test run is the grant. And after about seven hours of continuous UI testing, Failed to activate application stopped being a one-retry flake and failed three times consecutively with zero stray processes. Re-running a different, untouched, previously-green test costs 60 seconds and converts my test broke into the host is dead with no ambiguity.

That control test is the cheapest thing in the whole system. We carry the engine's smoke tests inside every domain run. A domain red next to a green control is a domain defect; a domain red next to a red control is a broken lane; a run with no control cannot tell them apart.

Diagnosis order when a UI test goes red

Every step below exists because a red was once attributed to the wrong layer. The order is the point.

1. Did anything execute? Read Executed N tests. Zero tests, or every method skipped and failed in a second, is the host or the runner environment, never the product.
2. Is the lane alive? Run the in-bundle control. Red control means broken lane; stop.
3. Is someone else on this simulator? Check for another test-without-building process on your UDID.
4. Is the binary fresh? Compare the installed app's modification time against the source you think you are testing.
5. Is it the right session, tenant and device state? Screenshot the simulator. Read the container's stored session. Check orientation. Check for a stranded modal.
6. Export the AX snapshot before theorising. Look for an ancestor's id sitting on the leaf, a zero-frame row, a keyboard frame intersecting the target, an interrupting-element line.
7. Did the harness actually attempt the action? Find the synthesised tap coordinate; check whether a best-effort helper declined silently; check whether near-identical durations point at one shared blocker.
8. Is the oracle vacuous or pre-state-true? Could this assertion have passed or failed for a reason that belongs to a different state or screen?
9. Run a control leg on the same binary. An identical failure on an adjacent untouched leg attributes it away from your change in one run.
10. Only now suspect the product, and disambiguate client from server with an HTTP probe before touching app code.

# Step 1: did anything execute?
grep -E "Executed [0-9]+ tests" run.log
grep -cE 't = +[0-9.]+s' run.log     # 0 on a failed init, 1000+ on a real run

# Step 3: is anyone else on this simulator?
ps ax -o pid,command | grep test-without-building

# Step 5: what is actually on screen?
xcrun simctl io <udid> screenshot now.png

# Step 6: the AX hierarchy at every failure
xcrun xcresulttool export attachments --path Run.xcresult --output-path attachments/

A blocked run, one that could not authenticate or could not launch, is unmeasured, never red. Record it as such with the blocker named. Never mark it green, and never red, which would falsely accuse the change under test.

Process: how the suite stays honest

Each suite has one living document with fixed sections: scope, test classes, a test-id index with per-id state, how to run, the last run (replaced, never appended), and an append-only learnings section where every entry is dated and states what was measured. Durable material is promoted into learnings before the last-run section is overwritten. Cross-cutting findings go in one shared file, not in whichever suite found them.

Any change that adds, renames, removes or materially alters a user-visible surface flips the affected suites to retest, re-runs the touched slice, and authors new coverage at the cheapest sufficient layer. Test sessions do not patch the product; a discovered defect escalates to a separate fix with its own record. And every red gets a root cause classified as product, test or environment, with the lesson going where the next author will read it. Several lessons in this series were written down and still reproduced one phase later, which is why written down is not enough: promote to the shared base class, add a structural guard, or make the helper record the value the plan told the operator to record.

The meta-rule

Nearly every lesson in this series was reached the same way: a claim recorded in prose was re-measured and found wrong. Blockers dissolved on contact. A cannot-expand-this-menu canon was overturned by a passing sibling. A header comment described an override that did not exist in the file it headed. A component's doc comment described where it stamped an id, and it stamped it nowhere.

A green you cannot re-derive today is a claim, not a result. So is a red.

Ten lines, if you only keep ten

1. UI tests are the deliberately thin top of the pyramid; anything a tree query or a unit test can prove does not belong in a screenshot.
2. The accessibility identifier is both the VoiceOver handle and the test handle; an unnamed control is two bugs.
3. Stamp the leaf. A container id overwrites every child's id; .contain keeps them, .combine destroys them.
4. On iOS the authoring form decides whether your stamp surfaces: Button with a title and systemImage yes, Button with a bare Image label no.
5. isHittable is not visibility, exists is not unoccluded, firstMatch is not the visible one, and the tail of a collection is not the new one.
6. Never coordinate-tap a non-hittable element; resolve a hittable match or fail loudly with the frame in the message.
7. Budget waits in queries, not seconds; never call activate inside a read-only poll.
8. A green built from absences proves nothing; a landing oracle waits on destination content that did not exist before.
9. A restored session, a rotated simulator and a stale binary each produce a red that names the wrong subsystem; screenshot the device and read Executed N tests before theorising.
10. Every red gets a root cause in product, test or environment, and the lesson is promoted to the shared base, not left in a suite's bullet list.

Previously (Part 5): Oracles and Isolation: When a Green XCUITest Proves Nothing