TesterArmy e2e
TesterArmy e2e security and operations: actions run with your rights
Examine model-visible data, trace provenance, telemetry and failure artifacts
What you will learn
- Protect secrets and app data
- Treat traces as input
- Review network and retention
Before you start
- A disposable app or test account
- A web or mobile engine
- A deliberate model-provider choice for agent steps
Use two runs and one controlled UI mutation before trusting cached actions
Key takeaways
- The app can leak a masked secret through its own UI.
- Unsigned traces need review.
- Telemetry is opt-out in the documented CLI.
Protect secrets and app data
The security guide describes a secret-fill path intended to keep credentials out of model text, but it also warns that an app can echo a secret into visible output. The model can see observations, a redacted URL and earlier steps.
A test agent may navigate to a different site or click destructive controls. Use disposable accounts, origin restrictions and limited permissions. Test the chosen secret route with synthetic data before using private credentials.
Treat traces as input
Recorded entries store typed values except secret fills. The guide says traces are unsigned and replay with your permissions; the recorder also marks poisoned values non-replayable.
Review trace changes as carefully as test code. Do not load untrusted recordings into a privileged CI job merely because they look like cached performance data.
Review network and retention
The security guide names the app, model provider and optional telemetry endpoint as connections. The README gives `npx e2e telemetry disable` or `E2E_TELEMETRY_DISABLED=1` for opt-out.
Check the telemetry field list and your own artifact store before rollout. We did not audit the implementation, test exfiltration resistance or verify deletion of stored results.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Use disposable targets and minimal credentials.
- 2
Review traces and pull-request code before privileged CI.
- 3
Decide telemetry and artifact retention explicitly.
Copy-ready example
test account -> app under test -> observed screen -> model
secret fill -> masked path, but app output still needs review
recording -> code review -> privileged replayFrequently asked questions
Does secret fill guarantee the model never sees a secret?
No. The security guide warns that app output can still expose it.
Can I disable CLI telemetry?
The pinned README documents a command and environment variable for opt-out.
Sources
- TesterArmy e2e / README.mdSource checked 2026-10-14
- TesterArmy e2e / docs/security.mdxSource checked 2026-10-14
- TesterArmy e2e / docs/telemetry.mdxSource checked 2026-10-14
- TesterArmy e2e / packages/e2e/src/cache/recorder.tsSource checked 2026-10-14
- TesterArmy e2e / packages/e2e/src/agent/replay.tsSource checked 2026-10-14