OpenWikivercel-labs/openwiki

Indexing Engine

Purpose and Scope

The indexing engine is the preparation and coordination layer that turns a public GitHub repository URL into the evidence OpenWiki uses to generate a source-grounded wiki. It does not simply clone a repository and hand it to a model. It validates the repository address, resolves GitHub metadata, creates a bounded file inventory, hydrates an eve sandbox workspace, selects high-signal snippets, assembles the parent indexing prompt, and connects eve lifecycle events back to durable indexing job state. This lets every generated page refer back to a known repository URL, default branch, commit SHA, workspace path, and source inventory instead of relying on ad hoc model context. Sources: agent/lib/github-repo.ts, agent/lib/indexing/context.ts, agent/lib/indexing/prompt.ts, agent/lib/indexing/adapter.ts

OpenWiki intentionally separates deterministic repository preparation from agent execution. Repository helpers own URL parsing, GitHub metadata lookup, file inventory construction, raw file reads, language detection, skipped-file tracking, and sandbox manifest writing. Context and prompt helpers compress the prepared snapshot into planning evidence and quality instructions for the parent indexing run. Adapter, logging, and session-state helpers then translate eve channel activity into observable job phases, failures, and active session context. That boundary matters because indexing is both a reproducibility problem and an agent orchestration problem: the source snapshot must be stable while outline and page workers remain flexible enough to plan useful documentation. Sources: agent/lib/github-repo.ts, agent/lib/indexing/context.ts, agent/lib/indexing/prompt.ts, agent/lib/indexing/adapter.ts, agent/lib/indexing/session-state.ts

Relevant Source Files

  • agent/lib/github-repo.ts — parses and normalizes GitHub repository URLs, fetches repository metadata and tree data, builds the useful-file inventory, records skipped files, reads raw file contents, detects languages, and writes the prepared workspace manifest into the eve sandbox.
  • agent/lib/indexing/adapter.ts — handles eve channel lifecycle events and maps them to OpenWiki indexing state, storage phase updates, failure handling, logging, and active session state.
  • agent/lib/indexing/context.ts — chooses priority context files from the repository inventory, filters internal planning documentation, scores paths by likely documentation value, and truncates snippets before prompt assembly.
  • agent/lib/indexing/log.ts — emits structured OpenWiki indexing logs with consistent phase, repository, job, session, and detail fields.
  • agent/lib/indexing/prompt.ts — builds the parent indexing prompt from repository identity, source inventory summaries, context snippets, official docs hints, public-surface candidates, and quality targets.
  • agent/lib/indexing/run-index-job.ts — serves as the job-runner boundary around repository preparation, prompt construction, eve session execution, and output publication.
  • agent/lib/indexing/session-state.ts — defines the shared eve state key that exposes the active repository indexing session to tools and runtime code.

Repository Access and Snapshot Preparation

Repository access starts with parseGitHubRepoUrl. The accepted shape is deliberately narrow: a public GitHub URL with an owner and repository name, an optional git suffix, and safe owner and repository characters. The parser trims input, rejects unsupported hosts or malformed paths, removes the optional suffix from the repository name, and returns a normalized URL plus a stable workspace path of the form repos owner repo. By enforcing this contract before any network or sandbox work starts, the engine keeps later stages from repeatedly interpreting user input and gives callers a clear failure mode when a repository address is not acceptable. Sources: agent/lib/github-repo.ts

After parsing, getGitHubRepoSnapshot resolves the repository into a PreparedRepositoryWorkspace. The snapshot carries identity fields, commit SHA, default branch, optional description, optional homepage URL, topics, useful file inventory, and skipped-file records. The file inventory is intentionally bounded: the engine selects at most 25,000 files and excludes any seeded file larger than 512,000 bytes. Files above the byte limit are recorded with a file_too_large reason, while files beyond the cap are recorded with file_limit_exceeded. Those skipped records are useful operationally because OpenWiki can explain why a repository was only partially seeded without letting unusually large trees dominate cost or prompt construction. Sources: agent/lib/github-repo.ts

prepareRepositoryWorkspace turns the snapshot into sandbox state. It creates the repository-specific OpenWiki metadata directory, writes each useful inventory file under the normalized workspace path, and writes a manifest file that records commit SHA, default branch, files, generation time, repository URL, skipped files, seeding limits, and workspace path. The manifest is the compact handoff between deterministic GitHub fetching and agentic indexing. When downstream workers cite a source path, the indexing engine has already attached that path to a specific commit and inventory decision, so publication can remain source-grounded even though the wiki text is generated later. Sources: agent/lib/github-repo.ts

Context Construction and Prompt Assembly

The context reader narrows the prepared inventory before the parent agent sees file contents. readIndexingContext filters to priority paths, excludes internal planning documentation, skips priority files larger than 160,000 bytes, sorts by a path-scoring function, reads at most 100 files, and truncates each snippet to 8,000 characters. The path rules favor root README files, package manifests, workspace metadata, build and test configuration, entrypoints, tests, documentation directories, examples, templates, GitHub workflow files, and source-tree files. This creates a representative initial view without treating every repository file as equally important for documentation planning. Sources: agent/lib/indexing/context.ts

The scoring rules encode a practical documentation hierarchy. Root README files and root package metadata receive the strongest weights because they usually define the project’s user-facing purpose, installation path, and public entrypoints. Package-level README and manifest files, workspace configuration, Rust crate manifests, and source or test entrypoints then add architecture and implementation clues. The reader also filters paths identified as internal planning documentation so status notes, gap analyses, feedback files, and implementation plans do not become public wiki structure. The result is context that favors reader-facing truth first and source implementation detail second. Sources: agent/lib/indexing/context.ts

createIndexingPrompt is the prompt assembly boundary for the parent run. It combines the prepared snapshot, selected context snippets, and optional official documentation index into a structured indexing task. The prompt names the repository, repository URL, default branch, commit SHA, useful-file count, skipped-file count, prepared workspace, and manifest path. It also instructs the agent to generate a real first-party documentation tree, prefer reader journeys over repository topology, treat first-party docs as primary information-architecture evidence when present, ignore internal planning docs, and use source paths as grounding rather than as the wiki outline itself. Sources: agent/lib/indexing/prompt.ts

Prompt construction uses capped summaries instead of dumping the full repository into the first turn. The prompt module defines limits for inventory samples, documentation source candidates, documentation information-architecture groups and paths, public-surface candidates, repository-map groups, outline context snippets, and page context snippets. These caps are a quality and cost control mechanism. The parent agent receives enough repository breadth to plan mature pages, but it is nudged toward systems, workflows, APIs, and reader tasks rather than a raw directory listing. For large repositories, this design helps the outline cover architecture, build behavior, runtime surfaces, examples, and operations when evidence supports those areas. Sources: agent/lib/indexing/prompt.ts

Session State, Events, and Job Phases

Once eve starts running the indexing session, indexChannelEvents becomes the bridge between agent activity and OpenWiki job state. On turn.started, the adapter requires both a repository URL and an index job identifier before it updates shared session state. That state is stored under openwiki.indexingSession and includes indexJobId, repositoryId, repoUrl, sessionId, and optional webUrl. This gives tools and runtime code a scoped way to discover the active repository indexing context without threading those identifiers through every function call. The default value is null, so code can distinguish an active indexing session from ordinary execution. Sources: agent/lib/indexing/adapter.ts, agent/lib/indexing/session-state.ts

The adapter deliberately filters noisy events before updating user-visible progress. On message.completed, it ignores tool execution and worker roles, then records a non-empty parent message as lastMessage, logs an agent-response-received phase, and updates the indexing job phase in storage. On turn.completed, it again ignores tool execution, worker roles, and already published channels before logging turn-completed. This keeps job status aligned with meaningful parent-run milestones instead of every nested worker or tool event. In practice, that distinction makes progress reporting easier to understand while still allowing the agent to delegate work internally. Sources: agent/lib/indexing/adapter.ts

Failure handling is centralized in the same adapter so incomplete runs do not silently look successful. When a tool-mode session completes before the repository job reaches a terminal state, the adapter loads the job and skips work if it is already completed or failed. Otherwise, it marks the job failed with a message explaining that the eve indexing session completed before the repository job finished. On session.failed, worker failures are ignored when they should not terminate the entire job, already published channels are protected, and the storage failure helper receives either the provided error message or a default indexing failure message. Sources: agent/lib/indexing/adapter.ts

Logging and Operational Signals

logIndexing provides the shared operational log shape for the engine. It always includes the phase and conditionally includes indexJobId, repositoryId, eveSessionId, repoUrl, and a human-readable repository label derived from owner and repo when available. Additional details are merged into the same JSON payload and emitted with the stable openwiki index prefix. Because the adapter uses this helper for response receipt, turn completion, and failure paths, operators can correlate storage job phases, eve session behavior, repository identity, and error details without reconstructing the run from unrelated logs. Sources: agent/lib/indexing/log.ts, agent/lib/indexing/adapter.ts

createEmptyIndexAdapterState shows the fields the runtime expects to accumulate across an indexing run. The initial state includes branch, commit SHA, context snippets, file inventory, job identifier, owner, repository name, repository identifier, repository URL, skipped files, and workspace manifest path. As the run proceeds, channel state can also hold values such as lastMessage and published status. Together, these fields describe the indexing state machine: collect repository facts, prepare prompt evidence, run the parent agent, observe progress, publish generated output, and avoid turning a successfully published channel into a later failure. Sources: agent/lib/indexing/adapter.ts

Compact Reference

  • parseGitHubRepoUrl(value) returns owner, repo, normalized URL, and workspace path after validating a public GitHub repository URL. Sources: agent/lib/github-repo.ts
  • getGitHubRepoSnapshot(repoUrl) returns repository metadata, commit, branch, useful file inventory, skipped-file records, topics, and optional descriptive fields. Sources: agent/lib/github-repo.ts
  • prepareRepositoryWorkspace(repoUrl, sandbox) writes inventory files and the OpenWiki manifest into the sandbox workspace. Sources: agent/lib/github-repo.ts
  • readIndexingContext(snapshot) returns bounded path-plus-text snippets selected from priority repository files. Sources: agent/lib/indexing/context.ts
  • createIndexingPrompt(input) emits the parent indexing prompt from snapshot data, context snippets, official docs evidence, repository summaries, and quality instructions. Sources: agent/lib/indexing/prompt.ts
  • indexChannelEvents handles turn, message, session completion, and failure events for indexing channels. Sources: agent/lib/indexing/adapter.ts
  • indexingSessionState stores the active repository indexing session under openwiki.indexingSession. Sources: agent/lib/indexing/session-state.ts

Execution Flow and Extension Guidance

A normal indexing run proceeds in a predictable order. The job runner boundary starts around a user-provided repository URL, then repository helpers validate and normalize it, fetch metadata, build the file inventory, and hydrate the sandbox workspace. Context selection reads a prioritized subset of files, prompt assembly creates the parent task with repository summaries and quality targets, and the eve session performs planning, drafting, validation, and publication work. During that run, adapter events update session state, storage phases, logs, and failure status. The runner boundary ties those stages together, while the adjacent modules keep their individual responsibilities small and testable. Sources: agent/lib/github-repo.ts, agent/lib/indexing/context.ts, agent/lib/indexing/prompt.ts, agent/lib/indexing/adapter.ts, agent/lib/indexing/run-index-job.ts

When extending the indexing engine, keep changes in the layer that owns the decision. GitHub limits, safe URL handling, raw reads, inventory metadata, skipped-file reasons, and sandbox manifests belong in the repository helper. Priority path rules, context byte limits, and snippet truncation belong in the context reader. Wiki quality instructions, repository-map summaries, official-docs treatment, and public-surface discovery belong in the prompt builder. Lifecycle filtering, published-state guards, job phase updates, and failure handling belong in the adapter. If a change affects user-visible progress or reliability, update logging and phase transitions together so one repository run remains traceable from preparation through publication. Sources: agent/lib/github-repo.ts, agent/lib/indexing/context.ts, agent/lib/indexing/prompt.ts, agent/lib/indexing/adapter.ts, agent/lib/indexing/log.ts

Related next steps: read Wiki Generation Pipeline for the full outline-to-page workflow, Source Grounding and Citations for evidence rules, Subagents and Instructions for worker responsibilities, and Indexing Jobs API for how the web application starts and observes repository indexing.

Ask a question...