Designing an XCUITest Harness the App Cooperates With

Part 1 covered where UI tests sit in the pyramid. This part is about the machinery: how roughly 180 XCUITest classes across a macOS lane and an iOS/iPad lane share one engine, how we assert on state a tree query cannot see, and what the app itself has to do to make any of it deterministic.

One base class, two lanes, two tiers

Every UI-test flow inherits one open XCUITestCase base. It owns the launch (XCUIApplication started with a --uitesting argument), a deterministic login prologue that drives the login form by identifier and selects the automation company by a company-card identifier, and identifier-driven navigation that replaced an earlier label-based helper. Navigation resolves elements by identifier, never by a static text label.

It also owns the form-factor scoping (a window on macOS, a column on iPad, the screen on iPhone), so one flow scopes its queries and screenshots correctly per device. And it owns the resilience wrappers we learned the hard way: bring the app frontmost before acting, retry on an empty accessibility tree, resolve rows hittability-aware, and poll every candidate in an any-of wait rather than just the first. Shared helpers live on an extension of the base or a lane-wide support file, never as private members of one test class; and because setup hooks are inherited, an override on a shared base silently changes every descendant with no compile error and no red, so any audit of which classes are configured this way has to walk the superclass chain.

macOS and iOS UI tests live in two separate Xcode projects. Dual target membership is impossible, so the engine files are copied byte-for-byte into the iOS project and a checksum test fails if the copies drift. That twin test was itself vacuous for six weeks: every test constructed a type compiled into the macOS target, so deleting the iOS folder would have stayed green. We fixed it by pinning the register size, treating a missing file as a failure rather than a skip, and adding a positive control that asserts the file being read really contains the login prologue. A guard's name, and a doc's summary of it, are not evidence that it does what it says.

Tier A and Tier B

Assertions come in two tiers. Tier A is structural: ordinary XCUITest assertions on identifiers, existence, hittability and values. Deterministic, pass or fail inside the run. Tier B is for state a tree query cannot see: a section rendered as a grouped box with a header, a landscape layout, a map pane actually painting, a redacted field genuinely absent from the region where it would appear. For those the test attaches a screenshot and a JSON rubric as paired XCTAttachments with keepAlways lifetime, and an out-of-band vision judge pairs them by step id, with no source access, and returns a per-assertion verdict.

The contract is an authored, positive-and-negative rubric, never a does-this-look-right prompt:

{
  "stepID": "CRM-01.step3",
  "screen": "Customer create window, Identity section",
  "platform": "macOS",
  "assertions": [
    { "id": "a1", "must":    "A 'Business Name' field is visible and contains 'Acme Industries'" },
    { "id": "a2", "must":    "The commit control shows a checkmark, NOT a warning triangle" },
    { "id": "a3", "mustNot": "No field labelled 'Credit Limit' is present (redacted for this role)" }
  ],
  "intentionalVariation": "Spacing, theme, accent colour and Dynamic Type may differ; judge meaning, not pixels."
}

The rules that make this tier trustworthy took a year to accumulate:

• Positive and negative, always. Positives make a blank or half-loaded screen fail; negatives catch redaction leaks. A vision model will happily rubber-stamp an empty screen, and captures taken with every DisclosureGroup collapsed made every mustNot pass by there being nothing there at all. Assert the expanded precondition.
• Vision is never the sole proof of something deterministically checkable. If a tree query can see it, Tier A asserts it. That shrinks the judge's surface to the visual-only slice.
• Trust or escalate. Per assertion the judge returns pass, fail or uncertain plus a confidence. High-confidence passes pass. Any high-confidence fail is a real defect. Anything uncertain goes to a human, never auto-pass. When XCUITest finds an element the judge says is missing, escalate too.
• The intentionalVariation field is the false-fail guard. It authorises layout and theme drift so the judge validates meaning, not pixels. This is the whole reason the layer is not a pixel-diff baseline.
• A Tier B step carries zero pass/fail information inside the XCUITest run. The capture call contains no assertion. A green suite is never evidence its screenshots were adjudicated; report that the structural legs passed and say whether a judge run happened. When Xcode changed the attachment-name mangling for one file type, a partial pairing failure dropped contracts out of adjudication and exited 0; pairing now fails hard unless paired equals contracts.
• The judge cannot see step order. A capture whose rubric is satisfied by the state the previous step already established proves nothing. We compare the md5 of every paired capture, pairwise across the whole run, before adjudicating. A hidden/shown pair was once byte-identical because the scroll helper never ran.
• Ground every clause in the frame you have, not the source you expect. Rows that specified 0.65 where the locale renders 0,65, a GroupBox the form never had, a caption a macOS toolbar never paints: each a latent false red on a correct screen. When a clause quotes a literal, look at a real capture first.
• A clause whose verdict depends on the judge's charity carries no information. Fourteen clauses across nine suites asserted a standard light toolbar while the lane ran in dark appearance. Write appearance-independent claims: the toolbar band is distinct from the content, not lighter than it.
• Prefer element-scoped captures over whole-screen ones. On iPad in landscape a whole-screen capture is stored rotated 90 degrees. On macOS a whole-desktop default once captured the operator's mail inbox, real sender names and subject lines included, into a result bundle attached to a pull request. Verify every capture's scope by its PNG dimensions, because a scoped capture that silently falls back to whole-screen looks identical in a green run.
• A reveal sufficient to prove a section exists is not sufficient to prove what it contains. A scroll helper that stops the moment the target is hittable leaves it flush against the edge with the fields whose absence was the whole point still clipped. Scroll to the rubric's first and last subject and bracket them.

The app has to cooperate

A UI-test harness cannot make a non-deterministic app deterministic from the outside. Ours honours a determinism contract only under a --uitesting launch argument, doubly gated so it cannot reach a shipped build: the reader body sits inside #if DEBUG, so in TestFlight and the App Store the code literally does not exist, and at runtime the argument must match exactly.

enum UITestingMode {
    /// True only in DEBUG builds launched with the exact `--uitesting` flag.
    static var isActive: Bool {
        #if DEBUG
        return CommandLine.arguments.contains("--uitesting")
        #else
        return false
        #endif
    }

    /// Per-feature arms need the master flag *and* their own argument.
    static func arm(_ name: String) -> Bool {
        isActive && CommandLine.arguments.contains(name)
    }
}

When active, the flag points the app at the automation environment, disables and latches inactivity auto-lock so a login or company switch cannot silently re-arm it mid-run, marks the onboarding carousel as seen so every run lands on a stable login screen, and makes company auto-select deterministic. Per-feature arms follow the same triple gate: force sign-out before the session manager initialises, bypass only the environmental microphone check, make one load site throw before the network call so the run lands on the view's real error branch, present the paywall at the root.

The rules that fell out of these arms matter more than the arms themselves. An injected failure must land on the product's real branch, never a test-only placeholder: a hook that rendered a bare view covering the window exercised a presentation that exists nowhere in the shipped app, and every identifier oracle stayed green over the wrong screen for weeks. A non-deterministic failure cannot substitute for an injected one, because an intermittently green test cannot distinguish the fix works from the request happened to succeed. A hook declined without the master gate must decline loudly. And a timing quirk the harness works around is a measurement of product behaviour: a 0.3 second wait in a helper's doc comment turned out, five weeks later, to be roughly 570 milliseconds of a wrong screen on every cold launch of the shipped app.

One product-side trick paid for itself many times over. A macOS SwiftUI Table with a hundred lazily rendered rows never renders the off-screen row for a seeded fixture, and scrolling to an arbitrary row is fragile. So the --uitesting seed records each fixture id, and a DEBUG-only overlay renders one hidden Button per fixture carrying a seed-prefixed identifier whose action fires the app's own openWindow call. The test clicks the marker and the exact record opens directly. The marker must be a real hittable control. A clear colour alone is not an accessibility element; a Button whose label is a clear colour with a rectangular content shape at around 30 to 44 points is both queryable and clickable. The id goes in the identifier suffix, never in the accessibility value, which truncates. And markers need geometry as well as identifiers: a top-leading overlay put the first marker in the corner macOS reserves for traffic lights, resolvable by id and permanently unclickable. Offsetting it took three tests from about 170 seconds to about 45.

Previously (Part 1): Where UI Tests Belong in a SwiftUI App

Next up (Part 3): SwiftUI Accessibility Identifiers: Why Your Stamp Never Surfaces