TesterArmy e2e
TesterArmy e2e performance and cost: measure replay honestly
Separate a model-free first test, model-backed actions and conditional cached runs
What you will learn
- Count where calls occur
- Budget the whole job
- Treat decision models as optional
Before you start
- A disposable app or test account
- A web or mobile engine
- A deliberate model-provider choice for agent steps
Use two runs and one controlled UI mutation before trusting cached actions
Key takeaways
- No-model tests and agent tests have different costs.
- Replay savings depend on verified reuse.
- Provider confidence needs context.
Count where calls occur
The README says tests with no agent steps need no model. Adaptive `agent.act` or `agent.assert` steps use a configured model route; a verified recorded trace may replay without model calls on a later run.
Those are different workloads. Compare them separately and capture whether the trace was accepted, stopped or fell back to a live step. A quoted saving without the replay hit rate would hide the interesting part.
Budget the whole job
Web and mobile engines have startup and observation costs even when model calls are zero. Hosted browsers, simulators or paid model providers can add separate charges.
Use the same app revision, engine, target device and provider for a fair before-and-after measurement. Include retries, failed assertions, artifact retention and human review time.
Treat decision models as optional
The decision-model executor can select bounded semantic actions and assertions through a provider, but its confidence statistic and gates are provider-specific. A missing confidence value is represented as zero in the pinned guide.
We did not run a benchmark or inspect billing terms. Evaluate correctness and stability before optimizing tokens, because a cheap but wrong test is expensive to maintain.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Record the no-model baseline.
- 2
Measure model calls and replay hit rate on the same fixture.
- 3
Include engine startup, retries and review time.
Copy-ready example
run,engine,model_calls,replay_status,elapsed_seconds,provider_cost,review_minutes
baseline,,,,,,not-measuredFrequently asked questions
Does every second run avoid a model call?
Only a suitable verified trace on an unchanged-enough app can use the zero-turn replay path.
Are hosted engines mandatory for speed?
The README lists optional hosted packages; test local engines first for your workload.
Sources
- TesterArmy e2e / README.mdSource checked 2026-10-14
- TesterArmy e2e / docs/decision-models.mdxSource checked 2026-10-14
- TesterArmy e2e / docs/reference/runner.mdxSource checked 2026-10-14
- TesterArmy e2e / packages/e2e/src/agent/replay.tsSource checked 2026-10-14
- TesterArmy e2e / packages/e2e/src/cache/recorder.tsSource checked 2026-10-14