mobile architecture · auditThe pull request description is short: "Added unit tests to the booking module." Forty new tests in the diff, all green, and the coverage report has moved from the thirties to the seventies. An agent wrote it, two people approved it, and the total review time was one coffee break. Three weeks later support files a ticket: a passenger who cancelled exactly 48 hours before departure got half their money back. There are five tests on the refund calculation. One of them asserts precisely that behaviour.
Our position: AI is very good at the mechanical part of writing tests and cannot do the part where someone decides what correct means, because the only source of information it has is the code under test. Until a human or a ticket states the expected behaviour, a generated test is a signature approving whatever the code does today. This article is about recognising that signature, what to ask for in its place, and how to check that your tests catch anything at all. The running example is the cancellation and refund flow of a hypothetical travel booking app.
Coverage tells you a line ran, and nothing about whether it was checked
A coverage tool measures one thing: which lines executed during the test run. It has no idea whether any assertion looked at the result. A test that calls a function and throws the return value away produces the same green line as a test that probes every boundary.
When people write tests by hand, this gap stays small on its own, because writing a pointless test still costs effort. For an agent the cost is zero. Tell it to "raise coverage to eighty percent" and it finds the shortest path to the number: call every public function once and expect whatever comes back. You set the wrong target and it was hit accurately.
What generated tests tend to look like
First the code under test and the ticket it came from:
BOOK-1423 Cancellation refund rules
- Flexible fare, cancelled 48 hours or more before departure: full refund
- Flexible fare, between 24 hours (inclusive) and 48 hours: half refund
- Flexible fare, less than 24 hours before or after departure: no refund
- Non-refundable fare: no refund in any case
enum Fare: Sendable { case flexible, nonRefundable }
struct Booking: Sendable {
let id: String
let price: Decimal
let departure: Date
let fare: Fare
}
struct RefundPolicy: Sendable {
func refund(for booking: Booking, cancelledAt now: Date) -> Decimal {
guard booking.fare == .flexible else { return 0 }
let hoursLeft = booking.departure.timeIntervalSince(now) / 3600
if hoursLeft > 48 { return booking.price } // the ticket says "48 or more"
if hoursLeft >= 24 { return booking.price / 2 }
return 0
}
}
There is a boundary bug: > where the ticket calls for >=. Ask an agent to "write tests for this file" without giving it the ticket and you get something like this:
import Testing
import Foundation
struct RefundPolicyGeneratedTests {
let policy = RefundPolicy()
let departure = Date(timeIntervalSince1970: 1_800_000_000)
// 1) A mirror of the implementation: the formula written out a second time
@Test func refundMatchesFormula() {
let booking = Booking(id: "B1", price: 200, departure: departure, fare: .flexible)
let now = departure.addingTimeInterval(-30 * 3600)
let hoursLeft = booking.departure.timeIntervalSince(now) / 3600
let expected: Decimal = hoursLeft > 48 ? booking.price
: hoursLeft >= 24 ? booking.price / 2 : 0
#expect(policy.refund(for: booking, cancelledAt: now) == expected)
}
// 2) A photograph of the current bug: expected is whatever the code returns
@Test func refundAtExactly48HoursIsHalf() {
let booking = Booking(id: "B1", price: 200, departure: departure, fare: .flexible)
let now = departure.addingTimeInterval(-48 * 3600)
#expect(policy.refund(for: booking, cancelledAt: now) == 100)
}
// 3) A test that cannot go red
@Test func refundIsNeverNegative() {
let booking = Booking(id: "B1", price: 200, departure: departure, fare: .flexible)
#expect(policy.refund(for: booking, cancelledAt: .now) >= 0)
}
}
The first test copies the calculation out of the production code and compares the result with itself. The mistake in the formula is on both sides, so the test passes along with the bug. The second is the dangerous one: its name and body agree with each other, and a reader comes away believing there is a half-refund rule at 48 hours. The bug is now documented behaviour, and whoever fixes it is greeted by a red test. The third takes a different branch depending on the day you run it and cannot fail in any of them.
The fourth shape is over-mocking, which we will show on the Kotlin side in a moment. All four share one property: the expected value comes from the code.
iOS: a useful test for the same code
The expected values come from the ticket, the test uses the public interface, and the boundaries are spelled out in a table:
import Testing
import Foundation
struct RefundCase: Sendable, CustomTestStringConvertible {
let hoursLeft: Double
let fare: Fare
let expected: Decimal
var testDescription: String { "\(fare), \(hoursLeft)h before departure -> \(expected)" }
}
@Suite("Cancellation refund (BOOK-1423)")
struct RefundPolicyTests {
static let departure = Date(timeIntervalSince1970: 1_800_000_000)
static let cases: [RefundCase] = [
RefundCase(hoursLeft: 72, fare: .flexible, expected: 200),
RefundCase(hoursLeft: 48, fare: .flexible, expected: 200), // boundary, inclusive
RefundCase(hoursLeft: 47.99, fare: .flexible, expected: 100),
RefundCase(hoursLeft: 24, fare: .flexible, expected: 100), // boundary, inclusive
RefundCase(hoursLeft: 23.99, fare: .flexible, expected: 0),
RefundCase(hoursLeft: -1, fare: .flexible, expected: 0), // after departure
RefundCase(hoursLeft: 72, fare: .nonRefundable, expected: 0),
]
@Test("Refund follows the fare rules", arguments: RefundPolicyTests.cases)
func refund(_ testCase: RefundCase) {
let booking = Booking(id: "B1", price: 200,
departure: Self.departure, fare: testCase.fare)
let cancelledAt = Self.departure.addingTimeInterval(-testCase.hoursLeft * 3600)
#expect(RefundPolicy().refund(for: booking, cancelledAt: cancelledAt) == testCase.expected)
}
}
Against today's code this goes red on the second row, which is exactly what we want. Swift Testing reports each argument of a parameterized test as its own test case, so the output tells you which row broke. The table is also a deliverable in its own right: you can show it to the product manager and ask whether a row is missing.
Android: the same trap, dressed as mocks
On the Kotlin side the price is a Long in minor units and the policy is called from a use case:
class RefundPolicy {
fun refundMinor(booking: Booking, cancelledAt: Instant): Long {
if (booking.fare != Fare.FLEXIBLE) return 0
val left = Duration.between(cancelledAt, booking.departure)
return when {
left > Duration.ofHours(48) -> booking.priceMinor
left >= Duration.ofHours(24) -> booking.priceMinor / 2
else -> 0
}
}
}
class CancelBookingUseCase(
private val policy: RefundPolicy,
private val api: BookingApi,
private val clock: Clock,
) {
suspend fun cancel(booking: Booking): CancellationResult {
val refund = policy.refundMinor(booking, clock.instant())
api.cancel(booking.id, refund)
return CancellationResult(refundMinor = refund)
}
}
The generated test (with MockK):
@Test
fun `cancel calls policy and api`() = runTest {
val policy = mockk<RefundPolicy>()
val api = mockk<BookingApi>(relaxed = true)
val clock = mockk<Clock>()
val booking = mockk<Booking>(relaxed = true)
every { clock.instant() } returns Instant.EPOCH
every { policy.refundMinor(any(), any()) } returns 10_000L
val result = CancelBookingUseCase(policy, api, clock).cancel(booking)
verify(exactly = 1) { policy.refundMinor(any(), any()) }
coVerify(exactly = 1) { api.cancel(any(), any()) }
assertEquals(10_000L, result.refundMinor)
}
The test mocks four objects, one of them a data class. What it verifies is "the number I told the mock to return came back" and "two functions were called". Delete the entire body of RefundPolicy and it passes. Send the wrong booking id to the API and it still passes, because any() accepts anything. On top of that, it breaks on every refactoring that reorders the lines in the use case. A test that raises false alarms and misses the real fault costs more than having no test.
The useful version for the same use case runs the real policy, a fixed clock, and a hand-written fake for the one dependency that sits on the boundary:
class FakeBookingApi : BookingApi {
val cancellations = mutableListOf<Pair<String, Long>>()
override suspend fun cancel(bookingId: String, refundMinor: Long) {
cancellations += bookingId to refundMinor
}
}
fun aBooking(
id: String = "B1",
priceMinor: Long = 20_000,
departure: Instant = Instant.parse("2027-01-15T09:00:00Z"),
fare: Fare = Fare.FLEXIBLE,
) = Booking(id, priceMinor, departure, fare)
@Test
fun `cancelling 30 hours before departure reports a half refund to the API`() = runTest {
val departure = Instant.parse("2027-01-15T09:00:00Z")
val clock = Clock.fixed(departure.minus(Duration.ofHours(30)), ZoneOffset.UTC)
val api = FakeBookingApi()
val useCase = CancelBookingUseCase(RefundPolicy(), api, clock)
val result = useCase.cancel(aBooking(priceMinor = 20_000, departure = departure))
assertEquals(10_000L, result.refundMinor)
assertEquals(listOf("B1" to 10_000L), api.cancellations)
}
Our rule is simple: mocks belong at the boundary where the process ends (network, disk, clock, system services). When a test mocks your own domain classes, send it back in review.
What to ask for instead
When we ask an agent for tests, we pin down four things.
Behaviour comes first, and it does not come from the code. Put the ticket's acceptance criteria, the API contract, or your own three-line list of rules into the prompt. If you have none of those, you need to write the behaviour down yourself before any test is generated; that is the part of the job that cannot be skipped.
Edge cases are enumerated before code. Ask for the list of scenarios only, and the code afterwards. The list takes five minutes to read and that is where the real review happens: is the 48-hour boundary inclusive, what happens after departure, which way does half of an odd number of minor units round. That last question is not in the ticket. If the agent asks it, it did its job well; if it quietly picks an answer, catching that is on you.
Tests are written against the public interface. Tests that reach private helpers through @testable import or read internal fields lock themselves to the implementation. If the externally visible inputs and outputs are not enough to exercise a behaviour, look for the problem in the design of the class.
For a bug, a red test comes first. Give the agent two separate steps: write the test that reproduces the bug and fails, show the output, and only then make the fix. A test that arrives together with the fix and has never been seen red is no evidence that it catches the bug.
We keep this in a template so nobody retypes it:
Task: write tests for RefundPolicy.
Source of behaviour: the BOOK-1423 acceptance criteria below. If the code and a criterion
disagree, do not assume the code is right; report the conflict and expect what the criterion says.
Step 1 (no code): produce the scenario table: input, expected output, which criterion it
comes from. List boundary values and questions the criteria do not answer. Wait for approval.
Step 2: write tests against the public interface only. Do not mock domain types.
Step 3: for each test, name the one-line change to production code that turns it red,
apply it, run the test, show the output, revert the change.
Can the test go red
Step three is mutation testing done by hand, and it is the cheapest check we know for trusting a test suite. You make a small change to the production code (> for >=, || for &&, a hard-coded return value, a deleted side-effect call) and run the tests. If none of them goes red, that line is executed and never verified.
Doing this by hand during review takes five minutes, and you can have the agent do it too. Just do not settle for its statement that the test "went red"; read the test output it pastes, and confirm from the diff that the change was reverted.
You do not know what a test protects until you have seen it fail.
iOS: Muter
The open-source tool that automates this for Swift is Muter. At the time of writing its repository is not archived and has received commits during 2026, though the latest tagged release on GitHub dates from 2023. It installs from a Homebrew formula or from source. Try it on a small package with your own Xcode and Swift version before you wire it into CI.
brew install muter-mutation-testing/formulae/muter
cd Packages/BookingDomain
muter init # writes muter.conf.yml: the test command and files to exclude
muter --files-to-mutate Sources/BookingDomain/RefundPolicy.swift
Know its two limits. Muter reruns the configured test command for every mutant; with xcodebuild on a whole app target that takes hours. We use it on domain packages that run quickly under swift test, and only on changed files. The second limit is the small operator set: it negates conditional operators (>= becomes <=, == becomes !=), swaps logical connectors, removes side-effect calls and swaps ternary branches. It does not produce the boundary mutation between >= and >. The bug in our example is exactly of that kind, so boundary checking is still done by hand and by table.
Android: PIT
On the JVM the standard tool is PIT (pitest). It is actively released, and its default operators include a boundary mutator (CONDITIONALS_BOUNDARY). For Gradle you use the info.solidsoft.pitest plugin, whose own documentation says it does not directly support Android modules. A separate fork for Android exists (pl.droidsonroids.pitest) and describes itself as experimental in its README.
One more caveat for Kotlin: PIT works on bytecode, and the extra bytecode the Kotlin compiler emits leads to meaningless mutants. PIT's site points to the Arcmutate plugin for Kotlin support, and that plugin requires a licence. The older open-source pitest-kotlin repository is archived. Our preference is to run PIT on pure Kotlin domain modules with no Android dependency and to read the report as an audit:
// booking/domain/build.gradle.kts (pure Kotlin/JVM module)
plugins {
kotlin("jvm")
id("info.solidsoft.pitest") version "1.19.0"
}
pitest {
pitestVersion.set("1.30.0")
junit5PluginVersion.set("1.2.3") // check compatibility with your JUnit Platform version in the plugin README
targetClasses.set(setOf("com.example.booking.domain.*"))
threads.set(4)
outputFormats.set(setOf("HTML"))
}
On neither platform do we make the mutation score a CI gate. The day the score becomes a target, what happened to coverage happens to it. Running it once a month on the three most frequently changed files and reading the surviving mutants teaches you plenty.
Where AI is strong
Once the expected behaviour is known, what remains is largely clerical work, and an agent does it faster than a person and often more carefully.
Characterization tests before a refactoring come first. Before you split an eight-hundred-line pricing class nobody dares to touch, you want its current behaviour locked down. Here, recording the current output as the expected value is a deliberate choice. What separates it from the "photograph of the bug" above is the label: these tests say they are characterization tests in the suite name and title, nobody reads them as a specification, and they are replaced with behaviour tests when the refactoring is done. An agent is good at sweeping the input space and patiently filling in a hundred-row table.
Table-driven tests are the second strong area. Once you have approved the scenario list, turning it into a table is mechanical. In a pure Kotlin module with JUnit Jupiter:
class RefundPolicyTest {
private val departure = Instant.parse("2027-01-15T09:00:00Z")
@ParameterizedTest(name = "{0} minutes before departure, {1} -> {2}")
@CsvSource(
"4320, FLEXIBLE, 20000",
"2880, FLEXIBLE, 20000", // 48 hours, inclusive
"2879, FLEXIBLE, 10000",
"1440, FLEXIBLE, 10000", // 24 hours, inclusive
"1439, FLEXIBLE, 0",
"-60, FLEXIBLE, 0", // after departure
"4320, NON_REFUNDABLE, 0",
)
fun `refund follows the fare rules`(minutesLeft: Long, fare: Fare, expectedMinor: Long) {
val booking = aBooking(priceMinor = 20_000, departure = departure, fare = fare)
val cancelledAt = departure.minus(Duration.ofMinutes(minutesLeft))
assertEquals(expectedMinor, RefundPolicy().refundMinor(booking, cancelledAt))
}
}
The same goes for fixtures and builders. Factory functions with default values like aBooking() above, hand-written fakes, models built from JSON samples: tedious, error-prone, and work an agent does well. Our one caveat is that you decide how the fake behaves.
Mechanical migrations are the fourth area. Moving from XCTest to Swift Testing is the typical case:
// Before: XCTest
final class FareParserTests: XCTestCase {
var parser: FareParser!
override func setUp() { parser = FareParser(locale: Locale(identifier: "tr_TR")) }
func testParsesFlexibleFare() throws {
let fare = try XCTUnwrap(parser.parse("ESNEK"))
XCTAssertEqual(fare, .flexible)
}
func testRejectsUnknownCode() {
XCTAssertThrowsError(try parser.parseStrict("???"))
}
}
// After: Swift Testing
struct FareParserTests {
let parser = FareParser(locale: Locale(identifier: "tr_TR"))
@Test func parsesFlexibleFare() throws {
let fare = try #require(parser.parse("ESNEK"))
#expect(fare == .flexible)
}
@Test func rejectsUnknownCode() {
#expect(throws: FareParser.UnknownCode.self) { try parser.parseStrict("???") }
}
}
The conversion is mechanical; the review should not be. Two differences in Apple's migration guide change behaviour silently. XCTest runs synchronous test methods on the main actor, while Swift Testing runs test functions on an arbitrary task, so tests that need the main thread must gain @MainActor. Swift Testing runs tests in parallel by default; a suite that leans on shared state needs .serialized or, better, needs to lose that state. The first number to check in a migration PR is the test count: how many ran before, how many run now. UI automation (XCUIApplication) and performance tests stay in XCTest. That has been Apple's guidance since Swift Testing was introduced; check the release notes of the Xcode version you use all the same.
UI test scaffolding is the fifth, on one condition: elements are found by a stable identifier and never by visible text. Adding those identifiers to production code is also a good job to hand to an agent.
// Production code (SwiftUI)
Text(viewModel.refundText)
.accessibilityIdentifier("cancellation.refundAmount")
Button("Cancel booking") { viewModel.confirm() }
.accessibilityIdentifier("cancellation.confirmButton")
// UI test (XCTest)
final class CancellationUITests: XCTestCase {
@MainActor
func testFlexibleBookingShowsFullRefund() {
let app = XCUIApplication()
app.launchArguments = ["-uiTestScenario", "flexibleBooking72h"]
app.launch()
let refund = app.staticTexts["cancellation.refundAmount"]
XCTAssertTrue(refund.waitForExistence(timeout: 5))
XCTAssertEqual(refund.label, "₺200.00")
XCTAssertTrue(app.buttons["cancellation.confirmButton"].isEnabled)
}
}
The Compose equivalent is Modifier.testTag and onNodeWithTag:
import androidx.compose.ui.test.junit4.v2.createComposeRule
class CancellationScreenTest {
@get:Rule val composeRule = createComposeRule()
@Test
fun confirmButtonReportsCancellation() {
var confirmed = false
composeRule.setContent {
CancellationScreen(
state = CancellationUiState(refundText = "₺200.00"),
onConfirm = { confirmed = true },
)
}
composeRule.onNodeWithTag("cancellation.refundAmount").assertTextEquals("₺200.00")
composeRule.onNodeWithTag("cancellation.confirmButton").performClick()
composeRule.runOnIdle { assertTrue(confirmed) }
}
}
Look at the import line. The v2 Compose test APIs (the junit4.v2 package that arrived with ui-test-junit4 1.11) use StandardTestDispatcher by default, and v1 is deprecated. Agents still write the old import because of its weight in their training data, and once you move to v2, tests that assume coroutines run immediately break. Put this in your instruction file.
Where it is weak
Concurrency and timing
An agent's first reflex with time is to wait:
// Generated: real time, real dispatcher
@Test
fun `hold expires`() = runBlocking {
val hold = SeatHold(CoroutineScope(Dispatchers.Default), holdFor = 100.milliseconds)
hold.start()
delay(150)
assertEquals(HoldState.Expired, hold.state.value)
}
This passes on the developer's machine, fails now and then on a loaded CI runner, and never tests the ten-minute duration used in production: it changed the parameter to avoid the wait. With kotlinx-coroutines-test the same thing is written in virtual time and deterministically, provided SeatHold receives its scope from outside:
class SeatHold(
private val scope: CoroutineScope,
private val holdFor: Duration = 10.minutes,
) {
private val _state = MutableStateFlow(HoldState.Idle)
val state: StateFlow<HoldState> = _state.asStateFlow()
fun start() {
scope.launch {
_state.value = HoldState.Active
delay(holdFor)
_state.value = HoldState.Expired
}
}
}
@OptIn(ExperimentalCoroutinesApi::class)
@Test
fun `seat is held for ten minutes and not released earlier`() = runTest {
val hold = SeatHold(scope = backgroundScope)
hold.start()
runCurrent()
assertEquals(HoldState.Active, hold.state.value)
advanceTimeBy(9.minutes)
assertEquals(HoldState.Active, hold.state.value)
advanceTimeBy(1.minutes)
runCurrent() // advanceTimeBy does not run the task scheduled at exactly that instant
assertEquals(HoldState.Expired, hold.state.value)
}
That last comment is the kind of detail agents skip regularly. Without runCurrent() the test is red; to "fix" it, the agent usually reaches for advanceUntilIdle() and deletes the "not released earlier" check without noticing.
Swift is harder. The generated test waits with try await Task.sleep(for: .milliseconds(150)) and is fragile in the same way. The remedy is to inject any Clock<Duration> into the type, but the standard library has no test clock you can advance by hand: you write a small one or take one from a library. Swift Testing's confirmation API does not solve waiting either. The migration guide is explicit that the event has to happen before the confirmation closure returns. An agent that does not know this translates the XCTestExpectation pattern one to one.
Lifecycle
Process death, backgrounding, configuration changes, scene reconnection: none of these appear in a function signature, so the agent does not see them. The test it produces calls viewDidLoad() or onCreate() by hand and assumes that reproduces the order the system really uses. Which tool covers these scenarios (on-device tests, a unit test with SavedStateHandle, a manual procedure) is your decision.
Flaky UI tests
Ask an agent to "fix" a flaky UI test and the usual answers are an added sleep, a larger timeout, or a retry wrapper. All three hide the symptom. The cause is typically one of three: a test that reaches the real network, animations left on, or state leaking between tests. Finding them means running the app and watching.
Deciding what matters
An agent gives every public function equal attention. The refund calculation and a description property weigh the same to it. The risk ranking that asks for deep tests on money, permissions, data loss and irreversible operations, and a thin layer on the rest, is yours to make.
Testable design still decides everything
Every good test above rests on one thing: RefundPolicy takes the time as a parameter, CancelBookingUseCase takes its dependencies through the constructor, SeatHold takes its scope from outside. Had those classes called Date(), reached for APIClient.shared, or created their own CoroutineScope, the agent would have been left with two options: stomp on global state in the test, or lean on a framework that mocks everything. Both produce the tests we called useless earlier.
So the quality of generated tests depends on the shape of your code more than on the model. Constructor injection and small, single-purpose modules are what make it possible for an agent to write good tests. We covered how to lay that ground in the singletons article, and how a container can ruin it in the DI anti-pattern articles for iOS and Android. If an agent can only produce a forest of mocks for a class, read that as a finding about the class's design.
Workflow: instruction file, review checklist, CI
A test section in the instruction file
Agents read the instruction file at the repository root in every session. AGENTS.md is an open format supported by many tools; if yours expects a different file (CLAUDE.md, .github/copilot-instructions.md), the same content goes there. Our test section looks roughly like this:
## Tests
- The source of expected behaviour is the ticket or task description. If the code disagrees, stop and ask.
- Produce the scenario table before writing code and wait for approval.
- Test against the public interface only. Do not mock domain types; for boundary
dependencies (network, disk, clock) use the fakes under Tests/Fakes, or add one.
- For a bug fix, write the failing test first and show its output.
- No real waiting in tests: sleep, Thread.sleep and delay on a real dispatcher are banned.
- iOS: new unit tests use Swift Testing; UI tests stay in XCTest.
- Android: coroutine tests use runTest; Compose tests use the junit4.v2 APIs.
- UI elements are found by accessibilityIdentifier / testTag, never by visible text.
- Do not delete or disable an existing test or loosen its assertions. If it is necessary,
state why in the commit message on a "Test-Change-Reason:" line.
- Commands: swift test --package-path Packages/BookingDomain | ./gradlew :booking:domain:test
Review checklist for generated tests
Read generated test code with the attention you give production code; there is no "it's only a test" discount. What we look at:
- Where does the expected value come from: the ticket, or the code's current output?
- Does the test name state a behaviour, or repeat a function name?
- Which one-line change to production code turns this test red? If nobody can find one, the test is deleted.
- Is the mocked thing on the process boundary, or one of our own domain types?
- Is there a copy of production logic (the same formula, the same condition) inside the test?
- Any real clock, real network,
sleep, random value, or dependence on test order? - Does the diff delete an older test, mark one
@Disabled/.disabled, or change an expected value?
The last item deserves separate emphasis. The shortest way for an agent to turn a red test green is often to change the test, and it will report that as "updated the test to match the new behaviour".
CI rules
We set two rules. First: no test is deleted or weakened without a stated reason. The script below is crude but does the job. If more assertions were removed from test files than added, or a test file was deleted, it looks for a reason line in the commit messages:
#!/usr/bin/env bash
set -euo pipefail
base="${1:-origin/main}"
files=('*Tests.swift' '*Test.kt')
pattern='@Test|func test|#expect|#require|XCTAssert|assert(Equals|True|False|That)|verify'
count() { # $1: diff line prefix (+ or -)
git diff "$base"...HEAD --unified=0 -- "${files[@]}" \
| grep -E "^\\$1[^$1]" | grep -cE "$pattern" || true
}
removed=$(count '-')
added=$(count '+')
deleted=$(git diff "$base"...HEAD --diff-filter=D --name-only -- "${files[@]}" | wc -l | tr -d ' ')
if { [ "$removed" -gt "$added" ] || [ "$deleted" -gt 0 ]; } \
&& ! git log "$base"..HEAD --format=%B | grep -q '^Test-Change-Reason:'; then
echo "Tests were deleted or weakened (removed: $removed, added: $added, files: $deleted)."
echo "Add 'Test-Change-Reason: ...' to the commit message."
exit 1
fi
Second: coverage is a floor. We set a lower bound slightly below today's value so that a large deletion of tests gets noticed, and we never announce a number as a goal. On Android with Kover:
kover {
reports {
verify {
rule { minBound(60) } // a few points below your current value: an alarm threshold
}
}
}
On iOS the same check is a few lines of script reading the coverage data in the .xcresult bundle through xcrun xccov view --report --json. Do not hand the job of raising the number to an agent; you would be back at the first section of this article.
How to tell whether confidence improved
A coverage percentage does not answer this. These are the indicators we look at, and you can derive all of them from your own data.
Escaped defects: for every bug found in production or QA, ask two questions and write the answers on the ticket. Was there a test on the code where the bug lived? If so, why did it not catch it? After three months those notes tell you which kind of bug your tests miss. If the share of "there was a test and it missed" is growing, your coverage went up and your confidence did not.
Flaky rate: count the CI test runs that fail on the first attempt and pass on rerun. If that count rises after generated tests were added, timing and shared-state problems came in with them. Once the team waves a red build through as "that test again", the suite has lost its value.
Time to reproduce a bug: when a bug report arrives, how long does it take to turn it into a failing test? Minutes if fixtures, fakes and an injectable clock are in place, days if they are not. This duration says far more about the state of your test infrastructure than coverage does.
Share of fixes seen red first: in how many bug-fix PRs is there a test that failed before the fix? A checkbox in the PR template is enough to measure it.
Mutation sampling: once a month, mutate a few of the most frequently changed files by hand or with a tool and count the survivors. The trend is all you need.
Where to stop
For a three-screen app whose business rules live on the server, scenario-table approvals and a mutation tool are overkill; a handful of UI flow tests and the review checklist will do. For throwaway prototype code, generate no tests at all. The discipline in this article is for code where a wrong calculation costs money, data or trust.
Checklist
- In test tasks given to an agent, expected behaviour comes from the ticket or a written list of rules.
- The scenario table and edge cases are approved before any code is written.
- For every generated test, the one-line change that turns it red is known and was tried at least once.
- Domain types are not mocked; shared fakes exist for boundary dependencies.
- Bug-fix PRs include the output of a test that failed before the fix.
- No real waiting in tests:
runTeston Android, an injected clock on iOS. - UI tests find elements by
accessibilityIdentifier/testTag. - The instruction file has a test section that includes the test commands.
- CI fails when a test is deleted or weakened without a stated reason.
- Coverage is defined as a floor and given to nobody as a goal.
- Escaped defects, flaky rate and time to reproduce a bug are reviewed every quarter.
Need support on this?
Let's review together whether these controls are set up correctly in your app. We're here with security audits and advisory.
Sources
- Apple · Swift Testing
- Apple · Migrating a test from XCTest
- Android Developers · Testing Kotlin coroutines on Android
- Android Developers · Migrate to Compose testing v2 APIs
- JUnit · Parameterized classes and tests
- Muter · Mutation testing for Swift
- PIT · Mutation operators
- gradle-pitest-plugin · Documentation
- Kotlin · Kover Gradle plugin
- AGENTS.md · Open format for guiding coding agents
Comments and questions
Disagree with something, or stuck on this in your own project? Write to us and let's talk.