Keeping Wikis Fresh
Purpose and Scope
OpenWiki treats a published wiki as a living artifact rather than a one-time export. The README defines the product as a Next.js app backed by an eve agent that plans an outline, writes source-cited pages, publishes a navigable wiki, and keeps published repositories fresh as their source changes. For readers operating a deployment, the important promise is that stale repositories are regenerated on a daily schedule without replacing the last good wiki. That framing matters because freshness is balanced against availability: a refresh should improve documentation when source changes, but a failed regeneration should not remove readable pages that users already depend on.
Sources: README.md
This page focuses on the refresh model visible from the requested source paths. The public documentation establishes the daily refresh behavior, while the GitHub repository helper shows how OpenWiki obtains a deterministic snapshot of a repository at a particular commit. Together, those pieces explain the core freshness loop: discover or schedule a repository for refresh, resolve the current GitHub state, compare or persist the commit and file inventory, regenerate when needed, and continue serving the most recent successful wiki until a replacement is successfully published.
Sources: README.md, agent/lib/github-repo.ts
Relevant Source Files
- README.md — Describes OpenWiki as a living, source-grounded wiki system and explicitly lists daily refresh scheduling through eve, stale repository regeneration, and preservation of the last good wiki among the product capabilities.
- agent/lib/github-repo.ts — Implements GitHub repository parsing, snapshot creation, file inventory generation, file size and count limits, workspace seeding, and manifest metadata that make a refresh run reproducible and comparable across commits.
Freshness Model
The top-level freshness contract starts with user value: OpenWiki should keep generated documentation aligned with the source repository. A scheduled refresh is not described as a blind rewrite of every page; it is a regeneration path for repositories that have become stale. Staleness, in this context, means the indexed wiki no longer represents the current GitHub repository state. The README’s phrase “keeps published repositories fresh as the source changes” gives the operator-facing goal, and the repository snapshot helper supplies the source-level data needed to make that goal concrete: owner, repo, normalized URL, default branch, commit SHA, file inventory, and skipped-file metadata.
Sources: README.md, agent/lib/github-repo.ts
The phrase “without replacing the last good wiki” is an important operational constraint. A wiki generation pass can involve remote GitHub requests, model calls, storage writes, and validation. Any of those can fail or produce output that should not be published. OpenWiki’s documented behavior is to keep the currently published wiki available while stale repositories are regenerated. That makes refreshes safe for public deployments: scheduled work can run in the background, but readers should continue seeing the previous successful artifact until a newer wiki is fully generated and published.
Sources: README.md
Repository Snapshot and Stale Detection
The refresh process depends on a stable repository identity before it can decide whether new work is needed. The helper accepts a GitHub repository URL, trims and validates it, rejects malformed input, and only allows owner and repository names made from letters, numbers, dots, dashes, and underscores. It then normalizes the repository URL and workspace path. This protects refresh jobs from ambiguous repository names, unexpected URL shapes, and unsafe workspace paths before any files are fetched or written into the sandbox.
Sources: agent/lib/github-repo.ts
After parsing, OpenWiki asks GitHub for repository metadata and the tree for the current commit on the default branch. The returned snapshot includes the commit SHA and default branch, which are the primary pieces of information a refresh system needs to recognize source drift. If a stored wiki revision was generated from an older commit, the current snapshot can justify a new indexing job. If the commit has not changed, a deployment can avoid unnecessary model and storage work while continuing to serve the existing wiki.
Sources: agent/lib/github-repo.ts
The file inventory adds a second level of grounding beyond the commit identifier. Each selected file contributes its path, detected language, size, and GitHub SHA as a hash. That inventory is exactly the kind of compact source map the page-generation pipeline can use to choose useful evidence and track what was available during a run. It also gives operators a way to reason about edge cases: a repository may change in files that are too large or beyond the seeded-file limit, and those files will be recorded as skipped rather than silently mixed into the generated context.
Sources: agent/lib/github-repo.ts
Reindexing Flow
A reindex run begins by creating a fresh repository snapshot. The helper limits seeded files to a maximum count and excludes individual files above the configured byte limit. Those limits are not just cost controls; they make scheduled refreshes predictable. A public deployment may refresh many repositories every day, so the agent needs bounded input size before it starts reading raw files, writing into the sandbox, and asking the model to plan or revise pages. Oversized files are recorded with a reason, preserving transparency about what was not included.
Sources: agent/lib/github-repo.ts
Once the snapshot is assembled, the workspace preparation function writes repository contents into the eve sandbox. It creates the repository workspace, fetches each selected file from GitHub, writes that file into the sandbox at its repository-relative path, and then writes an OpenWiki manifest under the workspace. The manifest includes the commit SHA, default branch, selected files, generation timestamp, limit settings, normalized repository URL, skipped files, and workspace path. That manifest is the handoff between source discovery and the downstream indexing agent.
Sources: agent/lib/github-repo.ts
Because OpenWiki publishes source-cited pages, a refresh is more than updating a timestamp. The regenerated wiki must be tied to a concrete source snapshot so citations and page content correspond to the files the agent actually saw. If a repository changes rapidly, each run should still be explainable by its own commit SHA and file inventory. This is also why skipped-file records matter during refreshes: they keep the published artifact honest about the bounded source context used to produce the updated documentation.
Sources: agent/lib/github-repo.ts
Cache and Page Revalidation Expectations
The README frames OpenWiki as a hosted app with generated wiki artifacts stored outside the running process. It also states that deployment provisions Postgres for repository metadata, jobs, revisions, and chat state, plus Blob storage for generated wiki artifacts. In the freshness flow, that means the scheduler and reindexer are not expected to mutate an in-memory page cache as the source of truth. Instead, successful generation should produce a new durable revision and artifact set, after which application pages can resolve the latest readable wiki from storage.
Sources: README.md
For operators, the practical rule is to treat revalidation as the final step after a successful publish. Public pages, featured repository prerendering, chat context, and markdown export should all converge on the same latest successful wiki revision. If generation fails, the old artifact remains the safe fallback. If generation succeeds, any route or metadata cache can be refreshed so readers see updated navigation, page titles, source citations, and repository descriptions. The supplied README does not expose the route-level implementation here, so operational details should be read with the internal maintenance and indexing job API references.
Sources: README.md
Edge Cases and Operating Guidance
Several edge cases are visible in the snapshot helper. Invalid GitHub URLs fail before metadata lookup, which keeps scheduled jobs from wasting time on non-repository input. Large files are excluded from the seeded inventory, and repositories with more than the maximum seeded-file count have additional files marked as skipped. GitHub API and raw-file requests use explicit timeouts, so refresh work is bounded against slow upstream responses. These guardrails are especially important for daily scheduling because stale detection and regeneration should remain reliable across small libraries, large monorepos, and temporarily slow network conditions.
Sources: agent/lib/github-repo.ts
When you operate OpenWiki, think of freshness as a sequence of checkpoints. First, ensure the deployment has durable storage and a GitHub token if you need higher public API limits. Second, let the scheduler identify repositories that should be checked. Third, rely on the snapshot to capture the current commit and file inventory. Fourth, allow the indexing pipeline to generate and validate the replacement wiki. Finally, only expose the refreshed pages after the new artifact is published. This sequencing preserves both accuracy and availability for readers.
Sources: README.md, agent/lib/github-repo.ts
Related Pages
Read the indexing engine documentation next if you need the details of how the sandboxed source snapshot becomes outline and page-generation context. Use the indexing jobs API page to understand status checks and job-start decisions, and the internal maintenance API page for scheduled refresh and repository revalidation endpoints. The storage setup page explains the database and artifact stores that keep old and new revisions durable, while the source grounding and citations page explains how file inventories and citations become reader-facing evidence.