TesterArmy e2e
TesterArmy e2e pilot: verify a trace and a deliberate app change
Use two runs and one controlled UI mutation before trusting cached actions
What you will learn
- Create a safe target
- Observe first and second run
- Make a release decision
Before you start
- A disposable app or test account
- A web or mobile engine
- A deliberate model-provider choice for agent steps
Use two runs and one controlled UI mutation before trusting cached actions
Key takeaways
- A cache hit still needs a final assertion.
- A changed app tests replay limits.
- CI adoption needs a trust review.
Create a safe target
Use a disposable app or account with a reversible workflow. Start with one no-model locator test, then add a bounded `agent.act` and a concrete postcondition.
Pin the e2e revision, engine and provider. Turn off or document telemetry according to your policy, and decide where recordings and screenshots will live.
Observe first and second run
On the first agent run, record actions, result, model calls and trace status. On a second identical run, inspect whether replay completed and whether the final assertion still passed.
Then make a controlled UI change, such as moving a button without changing the intended outcome. Check whether replay relocates safely or stops and yields to a live step; do not call any fallback a guaranteed success.
Make a release decision
Review the trace diff and test logs for sensitive data, false passes and unexpected navigation. Compare with a scripted baseline before enabling privileged CI.
This is an experiment design, not a result we observed. The pinned README describes pre-1.0 development, so keep an update and rollback plan.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Pass a deterministic baseline.
- 2
Run twice and inspect trace acceptance.
- 3
Change the UI, then review fallback and safety.
Copy-ready example
pilot:
target: disposable_app
checks: [no_model_baseline, first_agent_run, replay_run, changed_ui]
trace_review: required
result: not-runFrequently asked questions
When is the pilot complete?
When the no-model baseline, replay attempt, changed-app response and trace safety are documented.
Were tests executed for this article?
No. The series uses fixed primary sources and proposes a reproducible test.
Sources
- TesterArmy e2e / README.mdSource checked 2026-10-14
- TesterArmy e2e / docs/quickstart.mdxSource checked 2026-10-14
- TesterArmy e2e / packages/e2e/src/agent/replay.tsSource checked 2026-10-14
- TesterArmy e2e / packages/e2e/src/cache/recorder.tsSource checked 2026-10-14
- TesterArmy e2e / docs/security.mdxSource checked 2026-10-14
- TesterArmy e2e / docs/telemetry.mdxSource checked 2026-10-14