Indexing Jobs API

Purpose and Scope

The indexing jobs API is the status boundary between the OpenWiki web application and the asynchronous wiki generation run. When a user creates or refreshes a repository wiki, the application reserves an index job, dispatches the eve-backed repository indexing process, and then polls a small status endpoint until the job completes, fails, or needs to be restarted. This page explains that lifecycle from the route handler down to the helper functions that decide whether an existing job should be reused, failed as stale, or replaced by a new run.

The important concept is an index job: a persisted record representing one attempt to generate and publish a repository wiki. Jobs move through user-visible statuses such as pending, running, completed, and failed, while the agent records a more detailed phase for progress text. OpenWiki deliberately separates job reservation from agent dispatch so the app can prevent duplicate generation, recover from stale startup attempts, and surface an actionable restart state in the UI rather than leaving users with an indefinitely spinning progress indicator.

Sources: app/api/index-jobs/[jobId]/route.ts, lib/repository-indexing.ts, lib/index-job-staleness.ts, app/components/index-job-progress.tsx, agent/lib/indexing/types.ts

Relevant Source Files

• app/api/index-jobs/[jobId]/route.ts — Defines the public GET status endpoint for a single indexing job, including not-found handling, stale active job failure, and replacement with another active job for the same repository.
• lib/repository-indexing.ts — Provides startRepositoryIndexing and the RepositoryIndexingStart return contract used by repository creation or refresh flows to reuse, reserve, dispatch, or fail jobs.
• lib/index-job-staleness.ts — Centralizes timeout rules and error messages for jobs that never receive an eve session and jobs that stop updating while active.
• app/components/index-job-progress.tsx — Implements the client polling component, progress labels, restart callback behavior, and completed-job page refresh flow.
• agent/lib/indexing/types.ts — Defines IndexAdapterState, IndexingLogContext, official docs metadata, context snippets, and the MAX_WIKI_PAGES limit used by the indexing agent state model.

Route Reference

The status endpoint is GET /api/index-jobs/{jobId}. It receives the dynamic jobId route parameter, loads the job with getIndexJob, and returns a JSON response. If no job exists, the handler returns status 404 with the error message Index job not found. Otherwise, the normal successful shape is an object with a job property. The job payload is the storage-level job object expected by the progress UI, including id, status, phase, errorMessage, finishedAt, and eveSessionId where available.

The route also performs a small amount of repair work instead of only reporting state. If the stored job is pending or running and isActiveJobStale returns true, the handler marks that job failed with ACTIVE_JOB_STALE_ERROR_MESSAGE, reloads it, and returns restart: true alongside the failed job. This tells the caller that the current polling target should no longer be treated as recoverable progress and that the surrounding repository page may need to initiate a new indexing attempt.

Failed jobs have one more convenience path. If the requested job is failed, the endpoint checks getActiveIndexJobForRepository for the same repositoryId. When a different active job exists, the response returns that active job instead of leaving the client pinned to the stale failed job. This allows a user interface that still holds an older job id in the URL to follow the current repository generation attempt without needing to know the repository-level job lookup rules itself.

Sources: app/api/index-jobs/[jobId]/route.ts

Start and Dispatch Contract

The application entry point for starting a repository index is startRepositoryIndexing. Its input includes a repository, and may include a request, an explicit webUrl, and a beforeCreateJob callback. Its return type, RepositoryIndexingStart, contains created, job, and repository. The created flag is significant: false means OpenWiki found or reserved an already active job and callers should report that job rather than start another expensive agent run; true means this call created a new job reservation and attempted dispatch.

Before creating anything, startRepositoryIndexing looks for an active job for the repository. A pre-session job is a job whose eveSessionId is still null. If that startup phase has exceeded the pre-session timeout, OpenWiki fails stale pre-session jobs for the repository using PRE_SESSION_STALE_ERROR_MESSAGE and a staleBefore cutoff. If the active job already has session activity but has not been updated within the active timeout, OpenWiki fails that one job using ACTIVE_JOB_STALE_ERROR_MESSAGE. Otherwise, the existing job is returned unchanged with created set to false.

After stale cleanup, the optional beforeCreateJob callback runs immediately before reserving a new job. That placement lets repository flows perform last-moment validation, rate-limit accounting, or related setup only when OpenWiki is actually going to attempt a new reservation. The reservation step is still concurrency-aware: reserveIndexJobForRepository may return an existing job if another request won the race. In that case, the helper again returns created false and avoids duplicate agent dispatch.

When a new reservation is created, dispatchRepositoryIndexing posts to the OpenWiki eve URL for the index-repository action. The JSON body includes indexJobId, repoUrl, repositoryId, and webUrl. The request uses getEveServerHeaders and content-type application/json. A non-OK response is parsed for an error field when possible, otherwise it raises an error containing the HTTP status. Dispatch failures are written back to the job with failIndexJob, so callers see a failed job rather than a silent exception-only state.

Sources: lib/repository-indexing.ts

Staleness and Restart Semantics

OpenWiki uses two timeout classes because indexing can fail before and after the eve session begins. The pre-session timeout covers jobs that were reserved but never received an eveSessionId. By default, that window is twenty minutes. The active job timeout covers jobs whose updatedAt timestamp stops moving, defaulting to fifteen minutes. Both values can be overridden by positive integer environment variables: OPENWIKI_PRE_SESSION_STALE_MS and OPENWIKI_ACTIVE_JOB_STALE_MS.

The staleness helpers are intentionally defensive. isPreSessionJobStale returns false as soon as eveSessionId is present, because that means the job has moved beyond startup. Both staleness checks parse stored timestamps and return false when a timestamp is not finite, avoiding accidental failure from an unparsable value. getPreSessionStaleBefore converts the configured pre-session window into an ISO cutoff string so storage helpers can fail all old pre-session jobs for a repository in one operation.

Restart behavior is exposed at the status route rather than hidden entirely in the starter. A job can become stale while the user is watching progress, so the polling endpoint may fail it and return restart: true. The client component treats that flag specially: if an onRestartNeeded callback was supplied, it invokes the callback and stops processing that response as ordinary progress. This gives repository pages a clear hook for restarting generation or presenting a restart affordance.

Sources: lib/index-job-staleness.ts, app/api/index-jobs/[jobId]/route.ts, app/components/index-job-progress.tsx

Client Polling and Progress Labels

IndexJobProgress is a client component that polls /api/index-jobs/{jobId} every two seconds with cache set to no-store. It keeps the latest job, the latest polling error, and a count of consecutive errors. A missing job in an otherwise successful response is treated as an error. After three consecutive polling failures before any job has loaded, the component changes the display from a transient checking state to a persistent progress-unavailable state and can show the error message to the user.

The component maps internal phases to user-facing labels through PHASE_LABELS. The labels describe the pipeline in product terms: Index job queued, Reading GitHub repository, Preparing source context, Planning wiki structure, Generating wiki pages, Publishing wiki, Refreshing static pages, and Wiki published. This mapping keeps agent and storage phase values useful for operations while presenting readers with understandable milestones. Unknown phases are still displayed directly, which makes new phases visible during development without requiring an immediate UI update.

When a job completes, IndexJobProgress removes the job query from the current route with router.replace(pathname) and calls router.refresh. That is the handoff from polling mode to normal wiki rendering: the completed job should have published artifacts and revalidated pages, so the route can refresh into the new wiki view. Failed jobs stop the loading state and show either the job errorMessage or, for persistent polling problems, the polling error. The UI therefore distinguishes generation failure from inability to read progress.

Sources: app/components/index-job-progress.tsx

Agent State Model

The status API only exposes a compact job object, but the indexing agent maintains richer state while generating the wiki. IndexAdapterState records repository identity, branch, commitSha, source file inventory, selected context snippets, skipped files, workspace manifest path, indexJobId, repositoryId, repoUrl, owner, repo, and optional webUrl. It can also track the hydrated sandbox id, last message, execution role, execution mode, official docs metadata, and whether output has been published.

That state model explains why the job phase is only one visible projection of a larger process. The agent needs repository context, official docs discovery, source inventory, and publication flags to produce a source-grounded wiki. MAX_WIKI_PAGES caps wiki generation at 128 pages, and IndexingLogContext allows logging with partial adapter state plus eveSessionId. Together, these types show that a job id is the stable coordination key linking storage, the web polling API, and the agent run.

Sources: agent/lib/indexing/types.ts

Compact Reference

Surface	Contract	Behavior
GET /api/index-jobs/{jobId}	Returns { job } or { error }	404 when the job does not exist; may fail stale active jobs and return restart: true.
startRepositoryIndexing(input)	Promise	Reuses active jobs, fails stale jobs, reserves a new job when needed, and dispatches eve indexing.
RepositoryIndexingStart	{ created, job, repository }	created indicates whether this call created and dispatched a new job.
OPENWIKI_PRE_SESSION_STALE_MS	Positive integer milliseconds	Overrides the default twenty minute startup timeout.
OPENWIKI_ACTIVE_JOB_STALE_MS	Positive integer milliseconds	Overrides the default fifteen minute active progress timeout.
IndexJobProgress	React client component	Polls every two seconds, displays phase labels, handles restart signals, and refreshes the route on completion.

Next Steps

Use this API when building repository creation, refresh, or progress experiences that need to coordinate with asynchronous wiki generation. Call startRepositoryIndexing from server-side repository flows, pass the resulting job id into a route or component, and let IndexJobProgress poll the status endpoint until publication. For broader context, read the repository API page for the creation endpoints that initiate indexing, the wiki generation pipeline page for the agent stages behind each phase, and the rate limits and auth page before exposing generation on a public deployment.