Skip to content

feat(ingest): split ingest into orchestrated steps with deterministic hand-offs

Paolo Réant--Hamadi requested to merge feat/ingest-step-orchestration into main

agents/ingest was one 471-line conversation in which computing a hash, judging whether a sentence is grounded, and asking the user which folder to use were three instructions at the same level of trust. Split it so that everything computable is removed from the model's hands, and every hand-off between two reasoning steps is a schema-validated artifact.

Pipeline (11 steps, 7 with no LLM at all):

config → plan → glossary-pre → write-page (one per page, fanned out) → seal-citations → lint → verify-citations ⇄ fix-citations → verify-adr → glossary-post → nav → log-index → report

  • agents/ingest/steps/{plan,write-page,fix-citations,nav}/ — the reasoning steps, each self-contained because it starts cold, each with a JSON output schema the CLI enforces via --json-schema.
  • agents/ingest/references/scripts/ — the orchestrator, driving one claude -p per reasoning step and running the mechanical ones itself.
  • ingest.md keeps the overview and the orchestration contract, and inlines the steps via {{step: …}} so the interactive path stays a single source of truth.

Three properties motivate the split:

  • Resumable. State lives in .wiki-runs//, so a failed step is retried without redoing the ones before it. wiki.config.json is snapshotted, so a mid-run edit can't silently change the rules under a resumed step.
  • Interruptible without guessing. A headless process cannot ask a question, so any step may return status: "needs-input" with precise questions instead. This is what preserves ingest's rule 1 ("never assume") under automation; a step that guesses to stay "ok" has broken its contract.
  • Auditable. Each step's inputs and outputs are files, and the final report is derived from them rather than from a model's recollection.

Deterministic citation handling:

  • citationSeal.ts (new) stamps every missing sources[].sha from the real file and never overwrites an existing one. Agents no longer write a sha at all — transcribing ten hex characters is a manual step whose only contribution is an error, so fabrication becomes structurally impossible rather than merely detectable. Deleting a sha is how an agent signals it re-verified a source; leaving one keeps drift visible.
  • citationDump.ts (new) resolves a page's citations to plain text. It was referenced by agents/verify-citations' mandatory Step 2, docs/wiki-lint.md and scripts/wiki-lint/README.md, but had never existed in git history — that agent could not run. It also resolves directory-layout pages from their real file path, which a plan's relPath is: without that, a whole batch reported as "page not found".

Also:

  • metadata.json for agents/adr and agents/verify-adr-relevance, which had none and were invisible to any delivery generation.
  • Commit ce047c9b missed agents/: 40+ references to scripts/*.mjs that are now .ts. Fixed, and every script path cited in agents/ and docs/ now verified to exist — that gap is what let citationDump stay missing.
  • .claude/{agents,commands}/ ignored: generated from agents/ by external tooling.

85 tests (57 unit, 28 integration); typecheck clean. No full end-to-end ingestion has run yet — the mechanical steps, state machine and both citation scripts are exercised, the four reasoning prompts have not met a real batch.

Merge request reports

Loading