Scientific Agent Skills
Scientific Agent Skills 性能与成本：预算完整证据链
从数据库延迟、重试、上下文、API 支出和人工审核，建立可复现的技能基准。
你将学会
- Define a reproducible skill benchmark
- Build a latency and cost ledger
- Optimize while preserving provenance
开始前需要
- Basic Git and command-line usage
- Comfort reading a project README
You can explain the project, run its documented first step, and decide what to verify before adopting it.
先看结论
- Measure the complete evidence path, including external calls and review.
- Use p95, retries, cache state, and identifiers to explain latency.
- Keep provider-specific prices variable when the source does not publish a universal number.
Define the unit of work
技能成本不只是 prompt token，还包括数据库调用、分页、重试、本地解析和人工审核。固定问题、技能版本、模型与缓存状态，分别记录 p50/p95、标识符、引用和成本变量。
Record model, context window, database endpoint, result count, network region, and cache state. A second run can be faster because of HTTP or local caches; keep cold and warm numbers separate.
Measure provider and orchestration latency
Break a run into queue time, model latency, external API latency, parsing, and citation assembly. Track p50/p95, timeout rate, retry count, and bytes returned. Bounded concurrency can improve throughput but may trigger rate limits or increase spend, so treat it as a controlled variable.
For large literature or chemistry lookups, pagination and result filtering often dominate. Prefer the smallest query that answers the question, then expand only when the reviewer requests more evidence.
Build a cost ledger
Log input/output tokens, paid API calls, database quotas, storage, and operator minutes. If a provider has no published price in the captured evidence, leave the amount as an environment-specific variable rather than inventing a dollar figure. Separate one-time installation from per-run cost.
A useful ledger includes query hash, skill revision, model, cache hit, identifiers returned, and reviewer outcome. This lets a team compare a cheaper but less complete path with a slower path that returns stronger provenance.
Optimize without weakening evidence
Cache immutable identifiers and metadata, deduplicate repeated lookups, and summarize only after raw evidence is stored. Never optimize by dropping citations or allowing the model to answer when an adapter timed out. Add a budget ceiling that fails visibly and preserves the partial record.
Re-run the benchmark after skill, host, model, or database upgrades. The right target is predictable evidence-per-minute at an accepted cost, not the lowest token count in isolation.
如何选择
| 比较维度 | 方案 A | 方案 B |
|---|---|---|
| Best when | You need predictable behavior and easy auditing | You need adaptive optimization and have reliable telemetry |
| Main risk | May leave performance on the table | Can become difficult to explain or debug |
实施步骤
- 1
Fix one question, skill revision, model, and database fixture.
- 2
Capture cold/warm latency, retries, tokens, bytes, and identifiers.
- 3
Write a per-run ledger with cache and reviewer fields.
- 4
Canary upgrades and enforce a visible budget ceiling.
可复制示例
run_id,skill_rev,model,cache_hit,api_calls,retries,input_tokens,output_tokens
p50_ms,p95_ms,identifiers,citations,reviewer_status,cost_variable常见问题
Can I publish one benchmark number for all skills?
No. Skills call different databases and models; publish the fixture, revisions, cache state, and measurement method with every result.
Should caching remove source records?
No. Cache stable metadata, but retain the source URL, retrieval date, and raw evidence needed for review.
资料来源
- Scientific Agent Skills README (captured 2026-08-31)来源核查 2026-08-31
- Scientific Agent Skills repository来源核查 2026-08-31