Observability You Can See But Not Keep: A Live Stream Is Not a Record

I watched a corpus-analyzer run start at 05:51:26, stream a stage transition every few seconds, land four insights, and finish clean at 05:53:24. The dashboard did exactly what a dashboard is supposed to do: I saw the agent think. An hour later I wanted to know one thing about that same run. How many tool calls it made, whether it retried anything, what the final stop reason was. I had nothing. Not a slow query. Not a partial record. Nothing, because I had built the kind of observability that only exists while you are looking at it.

That gap is the subject of this post, and it is not a bug in the ordinary sense. Every piece involved works correctly. The event bus emits. The ring buffer holds a thousand entries. The SSE connection streams them to a connected browser in real time, and if you have the dashboard open during a run, you get a genuinely good experience: stage names, progress counters, a live feed that reads like a status bar for a mind. The problem only shows up when you close the tab, or when the run finished before you opened it, or when someone asks you three days from now whether a particular agent type has been getting slower. The system was never wired to answer that question. It was wired to look alive right now.

I want to walk through why that shape is so easy to build by accident, what the empirical test for it looks like, and what the minimum durable record actually needs to contain — because "add a database" is not the fix. I had a database. It was empty.

Watching a pipeline live, then having nothing an hour later

The corpus-analyzer run is a good example because it is unremarkable. It is a scheduled background job, part of the automation layer that mines session data for reusable insights, and on 2026-08-04 it ran for 118 seconds and produced four insights, the kind of thing you'd want a weekly summary of six months from now. While it ran, the dashboard's activity feed showed pipeline stage events arriving in order, a progress counter climbing, and a completion event with a green status.

Then I asked the database the question the dashboard had implicitly promised it could answer: what happened during that run. select count(*) from agent_events came back 0. Not filtered to zero by a workspace scope I forgot to pass, not zero because the run predates the table. Zero, full stop, for a table whose entire purpose is to record agent activity, immediately after a real successful run that had visibly emitted a dozen or more events into the system.

The agent_events table itself is not the problem. It is a properly designed table, with three indexes chosen for exactly the queries you'd want to run against it:

export const agentEvents = pgTable(
  "agent_events",
  {
    id: text("id").primaryKey().$defaultFn(() => crypto.randomUUID()),
    traceId: text("trace_id").notNull(),
    parentEventId: text("parent_event_id"),
    workspaceId: text("workspace_id")
      .notNull()
      .references(() => workspaces.id, { onDelete: "cascade" }),
    agentType: text("agent_type").notNull(),
    agentRunId: text("agent_run_id"),
    eventType: text("event_type").notNull(),
    level: text("level").notNull().default("info"),
    payload: jsonb("payload").notNull().default({}),
    createdAt: timestamp("created_at").notNull().defaultNow(),
  },
  (table) => [
    index("agentEvents_traceId_idx").on(table.traceId),
    index("agentEvents_workspace_time_idx").on(table.workspaceId, table.createdAt),
    index("agentEvents_eventType_idx").on(table.eventType),
  ]
);

An index by trace ID for pulling every event from one run. A composite index on workspace and time for the "what happened this week" query. An index on event type for filtering pipeline stages from errors. Whoever designed this schema was thinking about exactly the durability question I ran into. The design is not the failure. Nothing ever writes to it.

Three layers, and the one that's missing

The event path that actually carries data has three real hops, and I can name all three from the code, because I read them to write this post.

First, an in-process event bus. Every emit call goes through a small EventEmitter wrapper that also maintains a fixed-size ring buffer:

class ObservabilityEventBus {
  private emitter = new EventEmitter();
  private ringBuffer: AgentEvent[] = [];

  emit(event: AgentEvent): void {
    if (this.disabled) return;
    this.ringBuffer.push(event);
    if (this.ringBuffer.length > RING_BUFFER_SIZE) {
      this.ringBuffer.shift();
    }
    try {
      this.emitter.emit(EVENT_NAME, event);
    } catch {
      // Never let subscriber errors propagate to the emitter
    }
  }

  subscribe(handler: EventHandler): () => void {
    this.emitter.on(EVENT_NAME, handler);
    return () => this.emitter.off(EVENT_NAME, handler);
  }
}

RING_BUFFER_SIZE is 1000. That number is the whole retention policy for the system: the last thousand events, in process memory, gone on restart, gone once event 1001 pushes event 1 out. Every real caller across the codebase, including the agent runner, the automation publisher, the integration health checker, and the retry publisher, funnels through this one emit() method. I counted more than a dozen call sites feeding it, all correctly instrumented, all writing into a buffer that was never meant to be the system of record.

Second, an SSE broadcaster subscribes to the bus and fans events out to connected browsers:

class SSEBroadcaster {
  private connections = new Map<string, Set<SSEConnection>>();
  constructor() {
    eventBus.subscribe((event) => {
      this.broadcast(event);
    });
  }
}

That subscription is the only consumer registered against the bus in the entire codebase. One subscriber, and its job is to push events to whichever browser tabs happen to be open. That is the second and last hop, and nothing else listens.

The third layer is where a persistence subscriber should live and doesn't. The natural shape is obvious once you say it out loud: a second eventBus.subscribe() call, sitting next to the SSE broadcaster's, whose handler does exactly one thing: insert the event into agentEvents and return. It would cost a few lines and one more subscriber slot; setMaxListeners(100) on the emitter already budgets for more than one. I searched the codebase for any call that constructs an insert against agentEvents. There is none. Every read path, from the activity feed to the metrics endpoint to the per-run detail panel, queries a table that no write path ever populates. Those read endpoints are not broken either. They are querying the empty half of a system that was only ever finished on the streaming side.

Why this shape happens

I don't think this is carelessness so much as a predictable outcome of what gets rewarded during development. Streaming is the demo. You wire up an EventEmitter, add an SSE route, and within an hour you have a dashboard that moves — stages ticking by, a progress bar filling in, something visibly alive on screen. It is immediately, viscerally satisfying, and it is also the thing a stakeholder, a teammate, or your own sense of progress will notice and reward first. Persistence is the opposite of that. It is a subscriber that writes to a table and produces no visible feedback of its own. Nobody watches an insert happen. The only way to notice persistence is missing is to go looking for a record after the fact, which is exactly the moment I finally did.

There's a second, quieter reason: the streaming path and the persistence path look like the same feature from the outside. "We have an events table, we have live events on screen" reads as done. The dashboard's own existence becomes evidence that the underlying system works, because the dashboard is the thing anyone actually looks at. I built the SSE hop, watched it work, and moved on to the next thing convinced that observability, as a category, had been handled. It had been handled for exactly the duration a browser tab stays open.

The same pattern shows up twice more in this codebase, in ways that make the point sharper. agent_runs.result_metadata is a jsonb column, defined and ready, meant to hold token counts, cost, turn count, and stop reason for every completed run. It is NULL on every row I checked, because nothing populates it after a run finishes — the schema anticipated the need and the write path was never built to satisfy it. And there is a withRetry() helper, fully implemented with exponential backoff and rate-limit detection, that returns an attempt count alongside its result:

export async function withRetry<T>(
  fn: () => Promise<T>,
  options: RetryOptions,
): Promise<RetryResult<T>> {
  // ...loop, backoff, return { result, attempts: attempt }
}

Grep the application code for withRetry( and there are zero call sites. agent_runs.attempt_count defaults to 0 and there is no code path anywhere that increments it. The dashboard's run-detail panel has a whole UI affordance for "attempt #N" and a filter for runs where attemptCount > 1, built against a column that can structurally never hold anything but its default. Three different features, three different shapes of the same problem: the visible half exists, the recording half doesn't, and nothing in the running system tells you that.

The empirical test

The test I'd recommend to anyone building agent observability is not "does the dashboard show activity." That question is always going to answer yes if the streaming layer works, and streaming layers are the easy half to get right. The test that actually distinguishes observability from a UI feature is this: run something real, let it finish, walk away for an hour, then answer a question about it using only stored data.

I did exactly that. corpus-analyzer ran from 05:51:26 to 05:53:24 on 2026-08-04, completed successfully, produced four insights. An hour after it finished I ran select count(*) from agent_events scoped to that run's window, with no filters that could have hidden real rows. The measured result was 0. Not an estimate, not a sample — the literal count returned by the query, against the table whose sole purpose is to hold exactly this kind of record. If you can watch a run and then cannot query it, you have built a live stream, not an observability system, no matter how good the stream looks while it's running.

What a minimum durable record needs

The fix is not complicated, and it doesn't require a different architecture. It requires one more subscriber on a bus that already broadcasts to one. The shape of a durable record, at minimum, needs:

• A foreign key to the run: every persisted event should carry the agentRunId that already flows through every emit() call today, so a query by run ID returns a complete, ordered history without joining across log formats.
• The tool call and its result: not just "a tool was called," but which tool, what arguments, and what came back, because that is the difference between "the run failed" and "the run failed because this specific call returned this specific error."
• Prompt and response content, redacted: enough of the actual exchange to reconstruct what the agent was reasoning about, with secrets and PII stripped before the write, not after.
• Usage: input tokens, output tokens, and cost, captured once at the end of the run and written into the result_metadata column that has been sitting empty this whole time.
• A stop reason: why the run ended, whether completed, errored, or killed for stalling, stored as a value you can group by rather than reconstructed by squinting at the last few events in a feed.

None of that requires touching the streaming layer, which works and should be left alone. It requires accepting that a system with a live feed and an empty table is not halfway to observability. It's a UI with a database-shaped decoration next to it. The distinguishing question, going forward, is the one I should have asked before I ever opened the dashboard: not "can I watch it," but "can I still answer a question about it tomorrow." On this system, as of today, the honest answer to that second question is the number I measured: zero.