Hindsight explained: what an agent can keep between sessions
Build a Hindsight memory evaluation notebook
Create a small, auditable fixture that records corrections, retrieval misses and costs.
What you will learn
- Start with a bank ledger
- Capture evidence from each operation
- Set a useful acceptance rule
Before you start
- Python client basics
- A synthetic memory test case and a deployment boundary
Create a small, auditable fixture that records corrections, retrieval misses and costs.
Key takeaways
- A reproducible fixture can start without private data.
- A changed answer needs its own record.
- Unknown metrics stay unknown.
Start with a bank ledger
Use fictional people and decisions. Each retained line needs a source ID, timestamp and expected bank. Add a later correction to one fact, then write questions that distinguish the old and new state. The ledger is an exercise built around the documented API, not a feature shipped by Hindsight.
Keep the fixture in a separate test bank and pin the service revision. Avoid importing private conversation history just to make the demo look realistic. Synthetic records are enough to test the reporting workflow.
Capture evidence from each operation
Store the retain response, recall candidates and reflected answer with the query used. For every expected answer, link back to the original record and mark stale or unsupported outputs. Include a separate field for the provider and model configuration.
Run the same questions after a restart and, if applicable, after background consolidation. If the answer changes, keep both results and the elapsed time. This makes the difference inspectable instead of silently overwriting an earlier run.
Set a useful acceptance rule
A notebook page should show which questions passed, which failed and which were not run. Add measured latency and token cost only when the service returns or your platform records them. Never fill a missing number from the upstream benchmark chart.
The final artifact is a local report with traceable inputs and unresolved failures. We have not built or executed that notebook here; readers can use the schema below as a starting point for their own pilot.
Decision guide
| Criterion | Option A | Option B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
Implementation steps
- 1
Create dated synthetic records with one correction.
- 2
Save operation responses and expected evidence.
- 3
Keep failed and unrun cases visible in the report.
Copy-ready example
{"case":"correction-1","bank":"fictional-project","expected_source":"record-2","actual_source":null,"verdict":"not-run","latency_ms":null}Frequently asked questions
Does the notebook ship with Hindsight?
No. It is a proposed exercise using the documented API.
What counts as a pass?
The answer must cite the correct current evidence within the chosen budget.
Sources
- Hindsight / README.mdSource checked 2026-09-26
- Hindsight / hindsight-clients/python/hindsight_client/hindsight_client.pySource checked 2026-09-26