Wiki Generation Pipeline
Purpose and Scope
OpenWiki’s indexing pipeline is the path that turns a GitHub repository input into a published, navigable, source-grounded wiki. The README defines the product contract in reader-facing terms: OpenWiki accepts a repository, plans a docs-style outline, writes source-cited pages, publishes the result, and keeps published repositories fresh as source changes. In code, that promise is split across an authorized eve channel, an eve tool that starts the server-side job runner, output schemas that validate generated artifacts, and a publishing step that persists a wiki revision and revalidates the web app.
Sources: README.md, agent/channels/index-repository.ts, agent/tools/run_index_repository.ts, agent/lib/indexing/output.ts, agent/lib/indexing/publish.ts
The pipeline is intentionally job-centered. A repository may already have public pages, but regeneration should not replace the last good wiki until a new revision is validated and published. That design shows up in the phase changes and storage calls used by the indexing route and publisher: the job moves through startup, validation, publication, revalidation, and completion, while failures are recorded against the job instead of silently disappearing. For users, this means a wiki route can keep serving the previous revision while a new index job works in the background.
Sources: README.md, agent/channels/index-repository.ts, agent/lib/indexing/publish.ts
Relevant Source Files
- README.md — States the product-level generation flow: plan an outline, write cited pages, publish a navigable wiki, support repository chat from the same indexed context, and refresh stale repositories.
- agent/channels/index-repository.ts — Defines the authorized eve channel route that accepts repository indexing requests, validates the JSON payload, starts an eve task session, attaches that session to an index job, and records startup failures.
- agent/tools/run_index_repository.ts — Exposes the guarded eve tool that runs the repository indexing job only inside an authorized indexing session and returns the final job status summary.
- agent/lib/indexing/run-index-job.ts — Provides the server-side job runner invoked by the eve tool to perform the repository indexing workflow for the current job and repository identifiers.
- agent/lib/indexing/output.ts — Defines schemas for outline and page-generation output, including page slugs, titles, citations, coverage notes, related pages, markdown lines, navigation, priorities, and source paths.
- agent/lib/indexing/publish.ts — Validates the final typed response, normalizes publishable pages against the file inventory, publishes a wiki revision, finishes the job, and requests repository page revalidation.
System-to-Code Mapping
The public product flow begins outside the agent, when a user enters or visits a repository. The README frames that as giving OpenWiki a GitHub repository and receiving a living wiki. The indexing channel is the server-authorized boundary that accepts the concrete job payload. Its request schema requires an index job id, repository id, and repository URL, with an optional web URL. That schema is important because every later phase logs, publishes, or revalidates against the same identifiers rather than trusting free-form model output.
Sources: README.md, agent/channels/index-repository.ts
Once the request is authenticated and parsed, the channel sets the job phase to starting the eve run and sends a task-mode message to eve. The channel also creates the continuation token and task state used by the indexing session, then attaches the returned eve session id to the index job. This makes the model run observable from storage: application code can ask for the job, inspect its current phase, and correlate it with the eve session that is doing the generation work.
Sources: agent/channels/index-repository.ts
Inside the eve session, the only supported entry point for executing the pipeline is the run_index_repository tool. The tool checks indexing session state and verifies that the current session id matches the authorized context before it calls the runner. That guard matters because the tool description explicitly says it should only run when the index-repository channel asks the agent to start indexing. After the runner returns, the tool reloads the job, treats a failed job as an exception, and returns a compact status payload with the job id, repository id, phase, and status.
Sources: agent/tools/run_index_repository.ts, agent/lib/indexing/run-index-job.ts
Execution Flow
A successful run can be read as a sequence of handoffs. First, application code creates or chooses an index job and calls the eve channel route. Second, the channel authenticates the request, validates the request body, marks the job as starting, and launches an eve task session. Third, the eve tool enters the server-side runner with the job and repository identifiers. Fourth, the runner performs repository indexing work and eventually leaves a final typed response in the adapter state for publication.
Sources: agent/channels/index-repository.ts, agent/tools/run_index_repository.ts, agent/lib/indexing/run-index-job.ts
The final handoff is deliberately strict. The publisher refuses to proceed if the adapter state has no final typed response. It logs validating output, sets the job phase to validating-output, parses the final message, builds a set of valid source file paths from the file inventory, and normalizes pages against that set. If no publishable pages remain, publishing fails rather than creating an empty wiki. This gives the generation process a hard quality gate before any public artifact is written.
Sources: agent/lib/indexing/publish.ts, agent/lib/indexing/output.ts
After validation, the publisher changes the phase to publishing and calls publishWikiRevision with the branch, commit SHA, file inventory, job id, repository id, normalized pages, and source index metadata. The source index includes official docs, the generated outline, the repository summary, skipped files, and the workspace manifest path. That bundle explains how a wiki page can be source-grounded after generation: the published revision does not only contain markdown, it also carries the inventory and planning context needed to understand what evidence was used.
Sources: agent/lib/indexing/publish.ts
Output Contracts and Quality Gates
OpenWiki accepts two page-output shapes at different moments. Draft page parsing supports either a markdown string or markdownLines joined with newlines, along with citations, coverage notes, related pages, slug, and title. Page-generation responses use a stricter object with pages, where each page has markdownLines, citations with nullable positive line numbers, coverage notes, related pages, slug, and title. The schemas also normalize weak or invalid line numbers so citation metadata is predictable before storage or rendering uses it.
Sources: agent/lib/indexing/output.ts
The outline contract is broader than page markdown. It includes a repository title, summary, concepts, navigation, and pages. Each planned page has a priority, purpose, slug, title, and sourcePaths, while navigation supports nested nodes with titles and optional slugs. This is the bridge between planning and writing: the pipeline can first decide what documentation should exist, then ask page generation to produce exactly the requested pages with the expected source paths and citation discipline.
Sources: agent/lib/indexing/output.ts
The output module also enforces bounds and defaults that protect the publisher from malformed generation. Page priority is coerced to required, recommended, or optional, and missing arrays such as concepts, navigation, citations, coverage notes, related pages, and source paths default to empty arrays where appropriate. These choices make the agent contract tolerant of small formatting variations while still requiring the important fields: a wiki must have pages, pages must have titles and slugs, and publishable pages must contain markdown content.
Sources: agent/lib/indexing/output.ts
Publishing, Revalidation, and Failure Behavior
Publishing is not the final visible step. After publishWikiRevision succeeds, the job is finished, the adapter state is marked published, and the pipeline logs revalidation with the new revision id and web URL. It then requests repository revalidation so the Next.js app can update cached wiki routes. Revalidation errors are logged with a readable error description, but the code continues to set the phase to completed after the request attempt. This preserves the distinction between a generated revision and a cache-refresh notification.
Sources: agent/lib/indexing/publish.ts
Startup failures are handled earlier at the channel boundary. If sending the eve task or attaching the session fails, the channel logs a failed indexing event with the job id, repository id, and repository URL, marks the index job failed with the error message, and returns an error response. This keeps failures attached to durable job state. A caller that receives an error response can still rely on storage to explain that the job did not start successfully.
Sources: agent/channels/index-repository.ts
Compact Reference
| Stage | Code entry point | Main contract | Observable phase or result |
|---|---|---|---|
| Request validation | agent/channels/index-repository.ts | JSON body with indexJobId, repositoryId, repoUrl, optional webUrl | HTTP 202 response with job and repository on successful task start |
| Eve task launch | agent/channels/index-repository.ts | Authenticated task-mode send with continuation token and indexing state | starting-eve-run, eve session attached to job |
| Tool execution | agent/tools/run_index_repository.ts | Empty input schema, authorized indexing session state | Returns indexJobId, repositoryId, phase, and status |
| Job orchestration | agent/lib/indexing/run-index-job.ts | Repository and job identifiers passed from the tool | Produces the final indexing state consumed by publishing |
| Output validation | agent/lib/indexing/output.ts and agent/lib/indexing/publish.ts | Outline and pages parsed, normalized, and checked against valid file paths | validating-output, error if no publishable pages |
| Revision publishing | agent/lib/indexing/publish.ts | branch, commitSha, fileInventory, pages, sourceIndex, repositoryId | publishing, finished job, published revision id |
| Web refresh | agent/lib/indexing/publish.ts | repository owner, repo, repositoryId, revisionId, webUrl | revalidating then completed |
For developers extending the pipeline, the safest next step is to identify which boundary they are changing. Request-shape changes belong in the channel schema and any caller that creates the indexing request. Generation-format changes belong in the output schemas and should be reflected in the subagent instructions that produce outlines or pages. Publication changes belong after normalization, where the revision payload is assembled. Keeping those boundaries separate preserves OpenWiki’s key guarantee: generated documentation is accepted only after it is tied back to source inventory and stored as a revision.
Sources: agent/channels/index-repository.ts, agent/lib/indexing/output.ts, agent/lib/indexing/publish.ts