OpenVersions
SearchPublishHow it works

George Gallagher

@noctem

Publications

  • cogitatorPublic

    By @noctem

    0 starsAug 19, 2026

Claims 6

cogitator

  • Cogitator is a harness for deterministic, auditable evaluation of autonomous agents that commits to the witnessed semantics of a run using canonical JSON (JCS) and a domain-separated BLAKE3 hash chain, yielding a witness root that recomputes identically in replay mode from recorded transcripts and in live mode when prompts/seeds and tool behaviour are held constant.Verified
Show 5 major claimsHide 5 major claims
  • Witness roots remain invariant across thread counts (1, 2, 4, 8, 16) and explicit CPU pinning (cores {1,3,7,13}) on NixOS, as well as across thread counts (1 and 16) on Windows.Verified
  • Under deterministic fault injection (timeouts and drops at a two-per-million rate), Cogitator's drift reports localise regressions to specific tool calls and fields without hiding or amplifying agent regressions, even when aggregate metrics (pass rates and scores) are identical.Verified
  • Cogitator's domain-separated hash chains and manifest hashes detect subtle tampering, including altered metadata fields and reordered events in trace.jsonl.Verified
  • A 200,000-case synthetic core evaluation with seed=42 under Cogitator's core evaluation produces stable run-level statistics (55,660 passed, 144,340 failed, 27.83% pass rate, 0.362 mean score).Verified
  • Empirical results confirm that canonicalisation and hashing in Cogitator incur modest overhead.Unsupported

© 2026 OpenVersions

How it worksSearch APIPrivacyTerms