By @noctemSourceGitHub

Tamper-evident AI agent audit harness. Cryptographic witness chain, pre-call policy interception, and byte-stable replay. Prove what your agent did, tried to do, and was blocked from doing. Written in Rust.

Fork
Clone
git clone https://github.com/noctem-o/cogitator.git
Cite
Version link
https://openversions.com/noctem/cogitator/commit/85113731660e66320fb30d0e3470f33dbe418065
Version identifier (commit hash)
85113731660e66320fb30d0e3470f33dbe418065

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

Cogitator is a harness for deterministic, auditable evaluation of autonomous agents that commits to the witnessed semantics of a run using canonical JSON (JCS) and a domain-separated BLAKE3 hash chain, yielding a witness root that recomputes identically in replay mode from recorded transcripts and in live mode when prompts/seeds and tool behaviour are held constant.

Witness roots remain invariant across thread counts (1, 2, 4, 8, 16) and explicit CPU pinning (cores {1,3,7,13}) on NixOS, as well as across thread counts (1 and 16) on Windows.

Under deterministic fault injection (timeouts and drops at a two-per-million rate), Cogitator's drift reports localise regressions to specific tool calls and fields without hiding or amplifying agent regressions, even when aggregate metrics (pass rates and scores) are identical.

Cogitator's domain-separated hash chains and manifest hashes detect subtle tampering, including altered metadata fields and reordered events in trace.jsonl.

A 200,000-case synthetic core evaluation with seed=42 under Cogitator's core evaluation produces stable run-level statistics (55,660 passed, 144,340 failed, 27.83% pass rate, 0.362 mean score).

Empirical results confirm that canonicalisation and hashing in Cogitator incur modest overhead.

Methods

Domain-separated BLAKE3 hash chain over JCS (RFC 8785) canonicalised events: initialization hashes witnessed metadata under COGITATOR/WITNESS/V1/INIT, and each subsequent step updates di=H(COGITATOR/WITNESS/V1/STEP∥|PREV|∥di−1∥|LENGTH|∥u64le(∣ei∣)∥|CONTENT|∥ei)d_i = H(\text{COGITATOR/WITNESS/V1/STEP} \parallel \text{|PREV|} \parallel d_{i-1} \parallel \text{|LENGTH|} \parallel \text{u64le}(|e_i|) \parallel \text{|CONTENT|} \parallel e_i).

Source excerpts

“Concretely, initialization hashes the witnessed metadata bytes under the domain string COGITATOR/WITNESS/V1/INIT. Each subsequent step updates di = H(COGITATOR/WITNESS/V1/STEP ∥ |PREV| ∥ di−1 ∥ |LENGTH| ∥ u64le(|ei|) ∥ |CONTENT| ∥ ei), where ei is the canonical byte string for the ith witnessed event and di is encoded as lowercase hex for chaining.” · Show in paper

Witness/provenance split at the tool-call boundary: tool transcripts record both a committed witness view (tool name, request, response/outcome, success, optional fault) and uncommitted provenance (timestamps, hostnames, thread counts, latencies).

Source excerpts

“Witness view A projection of a tool transcript entry containing only the fields committed by the witness chain—tool name, request, response and success. It omits provenance-only fields such as latency or fault injection state.” · Show in paper

“In agent mode, Cogitator commits only the witness view of each tool call (semantics), excluding provenance such as timestamps and latencies.” · Show in paper

Deterministic fault injection using a BLAKE3-based PRNG keyed by seed and message (run_id, step, call_idx, fault_domain), drawing a 32-bit integer modulo 10610^6 to trigger Timeout, Drop, Corrupt, or LatencySim faults.

Source excerpts

“Cogitator designs fault injection around a pseudorandom number generator derived from BLAKE3, seeded by the run seed, step index and a user-configured fault profile. At each tool call the generator draws an integer from the range [0, 10^6); if the result is below the configured per-million rate, a fault is applied. Four fault types are supported: Timeout, producing a stable error outcome and a simulated delay; Drop, omitting the response entirely; Corrupt, applying deterministic bit flips or value modifications to the response; and LatencySim, introducing deterministic latency recorded only in provenance.” · Show in paper

“state = blake3 ( key = seed , message = ( run_id , step , call_idx , fault_domain ))” · Show in paper

Deterministic event ordering by run_id in core evaluation and by (step, tool_call_idx) in agent mode prior to hashing.

Source excerpts

“In core evaluation, runs are indexed by run_id and outputs are emitted in that order even when work executes in parallel (e.g., via Rayon). In agent mode, each agent step is numbered, and tool calls within a step carry a monotonically increasing tool_call_idx. Verification reconstructs the per-run witness stream by sorting witnessed tool-call entries by tool_call_idx within each step before hashing; this removes dependence on wall-clock timing and scheduler interleavings while preserving causality at the agent–tool boundary.” · Show in paper

Two-level verification workflow comprising meta+trace verification (replaying trace.jsonl and checking witness root and per-artefact hashes against meta.json) and witness bundle verification (recomputing the hash chain over per-case witness views).

Source excerpts

“Meta+trace verification. A verifier replays the canonical trace (trace.jsonl) under the same deterministic scheduling and recomputes the witness root from scratch. The recomputed root is compared against the expected root recorded in meta.json; any discrepancy indicates that either the trace has been tampered with or the implementation is nondeterministic.” · Show in paper

“Witness bundle verification. Each agent run (a “case”) produces a subdirectory containing the tool transcript, agent decisions, drift report (if any) and witness root. A witness bundle includes both the transcript and a witness view of each tool call—only the fields committed to the witness root. When verifying a bundle, a third party recomputes the hash chain over witness views and compares it against the stored root.” · Show in paper

Synthetic core evaluator scoring function that hashes seed||run_id with SHA-256 and computes score_ppm=base_ppm⋅(1−difficulty_ppm/106)\text{score\_ppm} = \text{base\_ppm} \cdot (1 - \text{difficulty\_ppm}/10^6) clamped to [0,106][0, 10^6].

Source excerpts

“The synthetic core evaluator deterministically maps {seed, run_id} to a fixed-point score in parts-per-million (ppm). It hashes seed||run_id with SHA-256, derives a difficulty value from the first byte, seeds a PRNG from the first eight bytes, and computes: score_ppm = base_ppm · (1 − difficulty_ppm/10^6), clamped to [0, 10^6], with base_ppm ∈ [450,000, 1,000,000].” · Show in paper

Research files

Commit by @noctem
Open the manuscript PDF
Publishes automatically from main. Every published version is archived.