SpotAgentIQ

Tamper-evident audit logs in 40 lines: hash chains explained

Jordan McKoy6 minute read

Why an append-only file with a SHA-256 chain is better evidence than a database row, how verification finds the exact line someone edited, and what it can't protect you from.

SpotAgentIQ logs every tool call an AI agent makes. The obvious place for that is a database table. We used a flat file with a hash chain instead, and this post explains why, with the core of the implementation.

The problem with a table

An audit log exists for the bad day. Something went wrong and you need to show what happened. On that day, the most important property of the log is that nobody could have changed it to look better, and a normal table can't show that. UPDATE audit SET approver = 'priya' WHERE id = 42 leaves no trace unless you've built something extra to catch it.

We don't need to make edits impossible. That's a much harder problem involving separate machines and write-once storage. We need edits to be evident: if someone changes the record, anyone checking it can tell, and can tell where.

The idea

Give every entry a fingerprint (its SHA-256 hash) and make each entry include the fingerprint of the one before it. Now each entry vouches for the whole history behind it.

  • Edit entry 42 and its content no longer produces its stored hash.
  • Edit it and recompute its hash, and entry 43's prev_hash no longer matches.
  • Delete entry 42, and 43 points at a hash that isn't there, and there's a gap in the sequence numbers.

This is the same structure under Git commits and Certificate Transparency logs. It's been around for a long time because it works.

The code

Here's a minimal version in Go. The real one adds file handling, fsync and locking, but the logic is the same.

package chain

import (
	"crypto/sha256"
	"encoding/hex"
	"encoding/json"
	"fmt"
)

type Entry struct {
	Seq      int    `json:"seq"`
	Event    string `json:"event"`
	PrevHash string `json:"prev_hash"`
	Hash     string `json:"hash"`
}

// The hash covers everything in the entry, including PrevHash, except itself.
func hashOf(e Entry) string {
	e.Hash = ""
	b, _ := json.Marshal(e)
	sum := sha256.Sum256(b)
	return hex.EncodeToString(sum[:])
}

func Append(log []Entry, event string) []Entry {
	e := Entry{Seq: len(log) + 1, Event: event}
	if len(log) > 0 {
		e.PrevHash = log[len(log)-1].Hash
	}
	e.Hash = hashOf(e)
	return append(log, e)
}

func Verify(log []Entry) error {
	prev := ""
	for i, e := range log {
		switch {
		case e.Seq != i+1:
			return fmt.Errorf("entry %d: expected seq %d, found %d", i+1, i+1, e.Seq)
		case e.PrevHash != prev:
			return fmt.Errorf("entry %d: does not follow the entry before it", e.Seq)
		case hashOf(e) != e.Hash:
			return fmt.Errorf("entry %d: content does not match its hash", e.Seq)
		}
		prev = e.Hash
	}
	return nil
}

Three checks, in order: is anything missing, is this entry attached to the right predecessor, and is it unchanged. The first failure tells you where the tampering starts. Everything before it is still trustworthy.

You can try this on the SpotAgentIQ home page: change who approved an entry and watch every hash after it stop matching.

Why a file and not a table

The chain would work in a table too. We chose a JSON Lines file for reasons that matter to the people who'll read it:

  • Anyone can verify it. An auditor can check it with our CLI, or with twenty lines of Python, without access to your database.
  • Append-only is easy to enforce. Open with O_APPEND, fsync each write, ship it to versioned object storage.
  • It's portable. grep, jq and a spreadsheet all work on it.
  • No dependencies. SpotAgentIQ is a single binary. Requiring a database to record what the agent did would be a strange first install step.

What it can't do

Someone with write access to the file can rewrite it from the edit onward, recomputing every hash after it. The chain is consistent again, and Verify passes.

The fix is to keep a recent hash somewhere the attacker can't reach. If you posted entry 1,204's hash to a Slack channel yesterday, today's file must still contain entry 1,204 with that exact hash, or someone rewrote history. Good places to put it:

  • Object storage with versioning or object lock.
  • A scheduled message to a channel or ticket.
  • An RFC 3161 timestamping authority, which signs "this hash existed at this time".

A hash chain also says nothing about whether the right things were logged in the first place. That's the job of the proxy writing it: every call passes through one place, so there's no path for a tool call that doesn't produce an entry.

In SpotAgentIQ

spotagentiq verify --log audit.jsonl
# OK: 1,204 entries, chain intact.

spotagentiq verify --log audit.jsonl --json | jq -r .last_hash
# the value to anchor somewhere else

Exit code 3 means tampered, so it drops into cron or CI. The audit log docs list every field and event, and our audit trail guide covers what to record and for how long.