Oracles and Isolation: When a Green XCUITest Proves Nothing

Parts 1 to 4 covered strategy, harness, identifiers and mechanics. This part is about the assertions themselves, and the state that leaks between tests. Both produce the most dangerous outcome a suite can have: a green in exactly the broken state.

Oracles: the rules a green-but-void run teaches

One suite went twelve for twelve green only after ten intervening reds, all ten test defects and none a product defect. The common shape: the cheap predicate answers a different question than the one you asked, and the suite is green in exactly the broken state.

A green built only from absences carries no information about the state it is named for. Our harness's own login smoke test asserted two things, both negative: login button absent, email field absent. So a blank screen, a stuck loader, the onboarding carousel and the company picker all satisfied it. It was vacuous for its entire life. Every absence assertion needs a positive control in the same flow, at a scroll position where its surface is rendered. This is the single most repeated rule in our corpus.

• Prove an absence oracle's identifier can ever appear. A save-button id stamped only on the create form made an assertion that it does not exist hold on the editable edit shell too; it could never fail, whatever the role.
• A landing oracle waits on the destination's content, never on the control that navigates to it. A tab id stamped on a NavigationLink row resolves one push short of the tab's content on compact width; the flow reported a landing and measured a surface that was never on screen.
• A navigation helper returning Bool must return that the selection took, not that a tap was dispatched. Five call sites read true from a tap fired as CRM is selected. That weak oracle hid a live product defect, every iPad hub unselectable on cold start, across the whole lane, and made the two suites asserting a real anchor look like the broken ones. A weak oracle does not merely miss a bug; it misattributes it.
• Silent-return helpers (setToggle, closeWindow, openForm, save) should XCTFail by default, attributed to the caller's file and line, with the non-asserting core split out for deliberately unreachable cases. For every XCTAssertTrue wrapped around a helper, ask whether the true means what the message claims.
• A presence oracle on a collection cannot distinguish the control exists from the control has nothing in it. Three legs asserted the picker is present and were green across three sessions while every candidate list was empty; the defect was real product code. Prefer a count oracle.
• An either-a-list-or-an-empty-state disjunction let a fixture that had never once succeeded certify green for a suite's whole life. If the fixture cannot produce the row, that is a red, not a variation.
• A prefix or CONTAINS selector over a mixed list degrades to confidently wrong, not not-found. A firstMatch slapped on an ambiguous query is worse than the raise it silences; Multiple matching elements found is loud and honest.
• A conditional leg that no-ops silently is an invisible skip. An if around waitForExistence that wraps the whole leg vanishes with no skip, no attachment, no log. Instrument the else, and an unentered leg owes an XCTSkip or an attachment stating what was observed.
• Count executions, not failures. Executed 5 tests, with 4 tests skipped and 1 failure reads as one pre-existing red while four of five asserted nothing. Neither the TEST SUCCEEDED marker nor the exit code is the oracle: xcodebuild exited 0 on a run that executed zero tests.
• A green UI-test lane cannot see a crash. A fully green four-test run contained a SIGSEGV that landed after the last assertion. Sweep crash reports before and after the run.

The habit that ties these together: red-proof every new assertion. Invert the anchor or delete the product line and see the predicted red on the predicted line. A new test whose red has never been seen is a claim, not a measurement, and repeatedly the instrument was the finding.

Isolation: state that outlives a test

On an installed-app iOS target the session survives between test classes, and a login prologue that short-circuits on a sidebar row resolves cannot tell whose session was restored. A class that logs in as a restricted identity and never signs out donates that identity to everything scheduled after it. Eight export legs died at the hub because the previous class had logged in as a zero-grant member. Four suites rediscovered this independently.

What makes it worse than a fixture bug: a tenant pin does not fix it, because the pin takes effect only through a real login, which a restored session skips. The green is the dangerous outcome, because a render assertion passes on a wrong tenant configured the same way. And a navigation helper proving its push by the home row being gone was unfailable for exactly the restricted identity, because that row is permission-gated and absent at the root. It returned true having issued zero taps.

The remedy is a force-sign-out launch argument applied by any class that pins an identity or a tenant, composed with the superclass's launch arguments and never a bare literal array. We measured 95 overrides: 42 composed super, and around 44 were bare literals each of which silently drops the base's company pin the day its class needs one. A per-class identity must also be re-threaded at every re-login site, not just setUp; a mid-test relaunch calling a bare login prologue used the default identity, and nothing flagged it because the wrong credentials are perfectly valid credentials.

Register a restore as a defer before the first mutation, and never default an unread original to the destructive value. An unresolvable toggle recorded its original as off via a nil-coalesce to false, so the restore actively drove state off; and the restore was the last statement rather than a defer, so any early exit skipped it. A bad restore does not fail its own test. It fails every other test on the same state, with a symptom that looks nothing like its cause.

Each UI suite owns its own deterministically seeded test tenant, reset at setUp, and a dedicated identity holding exactly one membership. A shared account that a human also uses turns every red into seed drift. Group test classes by whether a mutation can invalidate a sibling's oracle, never by suite membership: four classes that each commit a real create shared a pool safely because the fixture surface was sentinel-keyed, while one class writing a single config row had to be isolated because that row is a hard save-blocker for exactly the commits the others depend on.

Previously (Part 4): XCUITest Mechanics: Waits, Hittability, and Platform Traps

Next up (Part 6): Diagnosing a Red XCUITest: Environment, Order, and Process