TesterArmy e2e
Deploying TesterArmy e2e in CI: engines, models and secrets
Keep the app target, model route, recordings and pull-request trust boundary explicit
What you will learn
- Assemble the job
- Control credentials
- Handle recorded traces
Before you start
- A disposable app or test account
- A web or mobile engine
- A deliberate model-provider choice for agent steps
Use two runs and one controlled UI mutation before trusting cached actions
Key takeaways
- CI includes app, engine and optional model route.
- Test code executes with job permissions.
- Unsigned recordings are a trust boundary.
Assemble the job
A CI job needs the application under test, an e2e runner, its chosen web or mobile engine, and any model provider required by agent steps. The runner reference describes target selection and artifacts.
Pin package versions and browser or simulator images. A hosted device or browser package is an option, not a prerequisite for every installation. Test the same engine locally before adding CI concurrency.
Control credentials
The security guide says test code, config, engines, tools, reporters, stores and secret providers run with your permissions. Untrusted pull-request code should therefore run in an external sandbox without secrets or write access.
Keep provider keys, test accounts and deployment credentials separate. The model may see observations and earlier test steps; app output itself can reveal information even when a secret-fill mechanism masks a credential.
Handle recorded traces
Recordings may speed later runs, but they are not signed and can replay with the job’s permissions. Treat committed traces as executable test input: review changes and avoid accepting them from untrusted contributors automatically.
Set a retention policy for screenshots, traces and failure logs. We did not run a CI job, test isolation or measure parallel scaling.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Pin packages and target images.
- 2
Run untrusted changes without secrets or write access.
- 3
Review trace updates and retain only needed artifacts.
Copy-ready example
test_job:
target: disposable_app
engine: pinned_web_or_mobile
model: required_only_for_agent_steps
secrets: none_for_untrusted_pr
recordings: reviewedFrequently asked questions
Can pull-request jobs use production secrets?
The pinned security guide advises an external sandbox without secrets or write permissions for untrusted code.
Is a cloud browser required?
No. The README lists local web and mobile engines alongside optional hosted packages.
Sources
- TesterArmy e2e / README.mdSource checked 2026-10-14
- TesterArmy e2e / docs/reference/runner.mdxSource checked 2026-10-14
- TesterArmy e2e / docs/reference/engine.mdxSource checked 2026-10-14
- TesterArmy e2e / docs/security.mdxSource checked 2026-10-14
- TesterArmy e2e / packages/e2e/src/cache/recorder.tsSource checked 2026-10-14