Storage Setup

Purpose and Scope

OpenWiki needs storage before it can behave like a durable wiki product. The application generates documentation from GitHub repositories, but the generated result is not just rendered in memory for one request. Repository identity, indexing job progress, refresh metadata, chat-related state, and the pointer to the currently published wiki must survive across page loads and deployments. The storage layer therefore separates structured records from generated artifacts: Postgres holds the records that describe repositories and jobs, while Blob-style artifact storage holds the larger generated wiki outputs. Sources: lib/storage.ts

This separation matters operationally. A repository route can know which revision is currently published by reading database state, while page content and navigation can be loaded from an artifact referenced by that state. That lets OpenWiki preserve the last good wiki while a new indexing job is still pending, running, validating, or publishing. The Repository type includes a currentIndexedRevisionId field, and the indexing model includes explicit job status and phase fields, which together describe a system designed for observable, durable generation rather than one-off request processing. Sources: lib/storage.ts

The recommended deployment shape is Neon Postgres plus Vercel Blob. The storage implementation imports the Neon serverless client, Vercel Blob get and put, and Vercel OIDC token support, so the code is aligned with managed Vercel storage on hosted deployments. The same module also imports filesystem helpers and defines a local://openwiki/ artifact prefix, which supports a local artifact mode for isolated smoke tests. Sources: lib/storage.ts

Relevant Source Files

• lib/storage.ts — Defines the storage contracts used by OpenWiki, including repository records, indexing job phases, wiki page inputs, citation inputs, artifact availability errors, Blob authentication options, and the local artifact URL prefix.
• app/lib/storage-error.ts — Normalizes missing database configuration into a recognizable storage setup failure and returns a consistent HTTP 503 JSON response for route handlers.

Required Services

A complete OpenWiki environment needs a Postgres database and an artifact store. Postgres is the authoritative store for repository metadata, job state, revisions, refresh bookkeeping, and related application records. Generated wiki documents are larger and are better treated as artifacts, so the project uses Blob storage for the published page bundle rather than storing every rendered markdown body directly in the database. The storage module’s imports and types reflect that split by combining database access, Blob access, and publication-oriented data contracts in one shared library. Sources: lib/storage.ts

For the standard Vercel path, provision Neon Postgres and a private Vercel Blob store, then pull the resulting environment variables into local development. If the project was created from the one-click deploy flow, those integrations may already exist and vercel env pull is the main setup step. If not, the first-party setup flow uses these commands:

vercel integration add neon
vercel blob create-store openwiki-artifacts --access private --yes
vercel env pull .env.local --yes

The database boundary is intentionally strict. app/lib/storage-error.ts defines the canonical missing database message as DATABASE_URL is required for OpenWiki storage. and exposes isStorageConfigurationError to recognize that exact failure. When the app sees this condition, storageConfigurationErrorResponse returns a JSON response with status 503 and a reader-facing message telling the developer to set DATABASE_URL before generating or reading repository wikis. Sources: app/lib/storage-error.ts

Blob access has two credential shapes in the storage code. The BlobAuthOptions type includes oidcToken, storeId, and token, which matches hosted Vercel identity as well as token-based local access. In practice, BLOB_STORE_ID is the preferred Vercel-backed setting after environment pull, while BLOB_READ_WRITE_TOKEN is useful for local development when explicit Blob credentials are needed. The important point is that generated wiki artifacts must be readable by the runtime that serves wiki pages, markdown exports, metadata, and chat context. Sources: lib/storage.ts

Local Artifact Mode

OpenWiki also supports a local-only artifact path for smoke testing. The storage module defines LOCAL_ARTIFACT_URL_PREFIX as local://openwiki/ and imports filesystem functions such as mkdir, readFile, and writeFile. That combination indicates that the artifact layer can use disk-backed storage when local artifacts are enabled, while still preserving the same higher-level publication flow. Sources: lib/storage.ts

Use local artifact mode when you want to validate the indexing and rendering path without writing generated outputs to a remote Blob store. The first-party command is:

OPENWIKI_LOCAL_ARTIFACTS=1 pnpm dev

Local artifacts should be paired with an isolated local database. Artifact references become part of persisted repository and revision state; if a shared deployment later reads those records, it will not be able to access files that only exist on a developer laptop. For that reason, local artifact mode is appropriate for disposable development data, smoke tests, and experiments. It is not appropriate for production, shared preview deployments, or any database also used by a hosted OpenWiki instance.

Data Flow Through Storage Helpers

Repository storage starts with identity. The Repository type records id, owner, name, fullName, githubUrl, optional ownerAvatarUrl, updatedAt, and currentIndexedRevisionId. That shape gives route handlers and UI surfaces a stable way to identify a GitHub repository and determine whether a wiki has already been published. The current revision pointer is especially important because it lets readers keep seeing a complete wiki even while another job is preparing a replacement. Sources: lib/storage.ts

Indexing progress is represented by IndexJob and IndexJobPhase. The phase union includes created, fetching-repository, reading-context, starting-eve-run, waiting-for-agent, outlining-wiki, generating-pages, agent-response-received, validating-output, publishing, revalidating, completed, and failed. Those phase names mirror the end-to-end wiki-generation pipeline, so storage can support progress UIs, status APIs, retry decisions, and troubleshooting without requiring callers to inspect agent logs directly. Sources: lib/storage.ts

Generated output is passed into storage using source-grounded page contracts. SourceFileInput records a file path, language, size, and hash. CitationInput records a cited path plus optional start and end line numbers. WikiPageInput combines a slug, title, markdown body, and citations. These types encode OpenWiki’s core promise: wiki pages are not just generated prose; they are generated documents connected to source files and citation ranges that can be rendered, exported, and used for repository-aware chat. Sources: lib/storage.ts

Refresh state is modeled separately through RepositoryRefreshTarget. It extends repository identity with active job information, current commit SHA, generator version, current indexed time, featured status, and timestamps for refresh checks and enqueue attempts. The storage module also imports wikiGeneratorVersion, which means refresh decisions can compare not only repository source changes but also changes in the generator itself. That design lets scheduled maintenance decide whether a repository is stale, already being refreshed, or already current for the same commit and generator version. Sources: lib/storage.ts

Error Handling and Availability

OpenWiki distinguishes configuration failure from artifact unavailability. A missing DATABASE_URL is a setup problem, so storageConfigurationErrorResponse converts it into a clear 503 response with { error: ... }. That keeps local and hosted failures understandable at the route boundary: a user or developer sees that storage is not configured instead of an unrelated stack trace. Sources: app/lib/storage-error.ts

Artifact reads have their own typed failure. ArtifactUnavailableError uses the message OpenWiki artifact is unavailable., and isArtifactUnavailableError lets callers detect that condition. This matters because the database can be healthy while a generated artifact is unavailable, not yet published, or unreadable. Treating artifact failures separately gives callers room to fall back to repository metadata, show a loading or unavailable state, retry later, or preserve the last known route behavior without pretending the database itself is misconfigured. Sources: lib/storage.ts

Compact Reference

Practical Setup Checklist

Start by configuring Postgres, because OpenWiki explicitly treats a missing database URL as a service-level storage failure. Confirm that DATABASE_URL is present in the environment used by the Next.js runtime. Next, configure artifact storage through Vercel Blob using BLOB_STORE_ID on Vercel or BLOB_READ_WRITE_TOKEN for local token-based access. Only use OPENWIKI_LOCAL_ARTIFACTS=1 with an isolated local database, because generated artifact references are persisted and must be readable by whichever runtime later serves the wiki.

After storage is configured, create or visit a repository route and watch indexing progress through the persisted job phases. If the app returns a 503 storage configuration response, fix DATABASE_URL before debugging generation. If repository metadata loads but page artifacts are unavailable, treat that as an artifact-store issue rather than a database issue. Next, read Environment Variables for the full configuration surface, Run Locally for the development loop, Wiki Generation Pipeline for publication flow, and Indexing Jobs API for status handling.