Repository Search and Creation

OpenWiki turns a GitHub repository into a navigable, source-grounded wiki. The repository search and creation flow is the part of the application that accepts a user’s repository intent, normalizes it into a safe GitHub reference, checks whether the repository is already known, and decides whether to create or start an indexing job. This page documents that flow from both sides of the boundary: the public API routes that expose repository data and the client component that automatically starts generation when a user lands on an unindexed repository route.

Sources: app/api/repositories/route.ts, app/api/repositories/search/route.ts, app/api/repositories/[owner]/[repo]/route.ts, lib/github-repo-url.ts, lib/repository-creation.ts, app/components/repository-auto-index.tsx

Purpose and Scope

Repository discovery in OpenWiki supports two related reader tasks. First, a user can search GitHub by name and select a public repository suggestion. Second, a user can paste or navigate directly to a GitHub repository URL and ask OpenWiki to create or refresh the wiki for that repository. The code separates those tasks deliberately: search is a cached read-through query against GitHub’s repository search API, while creation is a state-changing operation that validates input, checks storage, enforces deployment policy, verifies the GitHub repository, and may start indexing.

The creation path is intentionally conservative because public deployments can incur storage, model, and GitHub API costs. The route accepts only public GitHub repository URLs that match OpenWiki’s URL parser, refuses unknown repositories when creation is disabled, and applies repository-generation rate limits before inserting a new repository or creating an indexing job. Featured repositories receive special handling: they are treated as configured public examples and can bypass generation rate-limit reservation in the creation route. This keeps the public demo experience smooth while preserving guardrails for arbitrary user input.

Search is also intentionally narrow. The search route returns an empty result set until the query has at least two trimmed characters, requests public non-fork repositories from GitHub, filters out private or malformed results, normalizes avatar URLs, ranks exact and prefix matches ahead of looser matches, and returns at most nine suggestions. The route is not a general GitHub proxy; it produces a small UI-oriented response shape that is safe to render in the repository picker.

Relevant Source Files

• app/api/repositories/route.ts - Implements the collection endpoint for listing repositories with GET and creating or indexing repositories with POST. It owns input validation, repository lookup, GitHub existence checks, creation-disable policy, generation rate limits, and indexing startup coordination.
• app/api/repositories/search/route.ts - Implements the repository search endpoint. It reads the q query parameter, calls GitHub search with OpenWiki-specific headers, filters and ranks results, and returns compact repository cards for the UI.
• app/api/repositories/[owner]/[repo]/route.ts - Implements lookup for a single repository wiki by route params and returns either { wiki } or a 404 error.
• lib/github-repo-url.ts - Defines the repository URL contract used across creation, routing, fallback avatars, and search avatar normalization.
• lib/repository-creation.ts - Encapsulates the deployment-level switch that disables public repository creation and provides the stable error code and message used by API and UI.
• app/components/repository-auto-index.tsx - Client-side component that posts to /api/repositories, handles indexing job responses, and renders retry, rate-limit, or disabled-creation states.

Public API Components

The main collection route is /api/repositories. Its GET handler calls storage to list known repositories and returns them as { repositories }. It wraps storage configuration failures through the application’s storage-error response helper, which means callers can distinguish a deployment setup problem from an unexpected server error. The POST handler is the creation and generation entry point. It expects a JSON body with repoUrl and optional force, validates the shape with Zod, parses the URL into a normalized GitHub reference, and then branches based on existing storage state and deployment policy.

The search route is /api/repositories/search?q=.... It is optimized for typeahead-style discovery rather than repository creation. A short query returns { repositories: [] } immediately. For longer queries, the route calls https://api.github.com/search/repositories with per_page=30 and a query constrained to in:name,full_name fork:false. It sends accept: application/vnd.github+json and user-agent: openwiki, and attaches authorization: Bearer ... only when GITHUB_TOKEN is set. The GitHub response is cached with next: { revalidate: 60 * 10 }, so repeated searches can reuse results for ten minutes.

The individual repository route is /api/repositories/[owner]/[repo]. This endpoint is a read-only lookup for the generated wiki associated with an owner/name pair. It awaits the dynamic route params, calls getRepositoryWiki({ owner, name: repo }), and returns { wiki } when found. If storage has no wiki for that pair, the route returns 404 with { error: "Repository not found." }. This route is useful for clients that already know the repository identity and need the wiki payload rather than a search card or indexing job.

URL and Search Result Contracts

lib/github-repo-url.ts is the shared normalization layer. parseGitHubRepoUrl(value) accepts HTTPS GitHub repository URLs of the form https://github.com/{owner}/{repo}, optionally ending in .git or a trailing slash. It rejects paths with unsafe owner or repository characters by requiring each part to match letters, numbers, underscores, dots, and hyphens. A successful parse returns { owner, name, fullName, url }, where url is normalized back to the canonical GitHub HTTPS URL. This parser is what keeps repository creation tied to public GitHub repository URLs instead of arbitrary user-provided locations.

The same utility module also defines getRepoHref(repo), which maps a repository reference to OpenWiki’s public route shape /{owner}/{name}. Avatar helpers are nearby because search and featured-card presentation need safe GitHub image URLs. normalizeGitHubAvatarUrl(value, size = 40) accepts only HTTPS URLs from avatars.githubusercontent.com, adds the requested size as the s query parameter, and returns null for anything else. Search results therefore expose iconSrc only when GitHub provided an avatar URL that matches OpenWiki’s allowlist.

Search result ranking is deterministic and intentionally simple. After filtering malformed items, the search route computes a rank using the normalized query, full repository name, repository name, and the original GitHub result index. Exact full-name matches come first, exact repository-name matches come next, then full-name prefix matches, repository-name prefix matches, and finally all remaining results. The endpoint slices the ranked list to nine results and returns only description, fullName, iconSrc, repoUrl, and starCount. This compact shape keeps the UI independent from GitHub’s larger response schema.

Creation and Indexing Flow

The POST /api/repositories flow starts by reading JSON. Invalid JSON returns 400 with Expected a JSON request body. Invalid schema or an unparsable GitHub URL returns 400 with Expected a public GitHub repository URL. Once parsing succeeds, the route checks whether the full name is configured as a featured repository and whether repository creation is disabled for the deployment. It then tries to load an existing repository record by owner and name. Storage configuration errors are converted to stable responses; other errors are allowed to surface.

If no repository record exists, policy is applied before any write. When OPENWIKI_DISABLE_REPOSITORY_CREATION is enabled, the route returns a disabled-creation response instead of creating storage rows. Otherwise it calls githubRepositoryExists(ref.fullName) to verify the public GitHub repository. A GitHub verification failure becomes 502, while a missing repository becomes 404. Only after the repository is known to exist does the route reserve the generation rate limit and call upsertRepository(ref). Rate-limit failures are translated into the repository-generation rate-limited response code expected by the client.

If a repository already has currentIndexedRevisionId and the request did not set force, the route returns 200 with { job: null, repository }. That response tells the caller that the wiki is already indexed and no new generation job is needed. If the repository is unindexed or the caller requested a forced refresh, the route re-checks the creation-disabled policy and then calls startRepositoryIndexing, passing a beforeCreateJob callback that enforces the rate limit exactly once for non-featured repositories. This design avoids charging the same request twice while still protecting both new repository creation and job creation.

Client Auto-Index Behavior

RepositoryAutoIndex is the client component that connects repository pages to the creation endpoint. It receives repoLabel for display and repoUrl for the API call. The component waits until the page is visible before starting work, stores the last started URL in a ref, and avoids duplicate POST /api/repositories calls for the same URL. This matters because React rendering, navigation, and browser tab visibility changes can otherwise produce repeated indexing attempts for a route that is still loading or temporarily hidden.

When the auto-index request succeeds, the response controls the next UI state. If job is missing or null, the component calls router.refresh() because the server can already render the indexed repository state. If a job object is returned, the component stores it and renders indexing progress through IndexJobProgress and WikiGenerationState. If the API responds with repository_generation_rate_limited, the component shows a muted rate-limit state. If it responds with repository_creation_disabled, it shows the deployment policy message. Other failures become a generic start-generation error with a retry path.

The retry behavior is local and explicit. restartIndexing clears the remembered URL, resets creation-disabled, rate-limit, error, and job state, and increments a startAttempt counter so the effect can run again. This retry model is useful for temporary GitHub, storage, or network failures, but it does not bypass API-side policy. If creation is disabled or the generation rate limit is still active, the next attempt will receive the same stable code from the server and render the corresponding state again.

Configuration and Error Reference

The creation-disable switch is centralized in lib/repository-creation.ts. isRepositoryCreationDisabled() reads OPENWIKI_DISABLE_REPOSITORY_CREATION; unset or blank means creation is allowed, and the values 0, false, no, and off also mean allowed. Any other non-empty value disables repository creation. The module exports repositoryCreationDisabledCode as repository_creation_disabled and repositoryCreationDisabledMessage as Repository creation is disabled for this OpenWiki deployment. The client component duplicates the code string so it can branch on API responses without importing server-only logic.

Surface	Method	Input	Success response	Important errors
/api/repositories	GET	none	{ repositories }	storage configuration response
/api/repositories	POST	{ repoUrl: string, force?: boolean }	{ job, repository } or { job: null, repository }	400 invalid JSON or URL, 404 repository not found, 502 GitHub verification failure, repository_generation_rate_limited, repository_creation_disabled
/api/repositories/search	GET	q query string	{ repositories: SearchResult[] }	502 when GitHub search fails
/api/repositories/[owner]/[repo]	GET	dynamic route params	{ wiki }	404 repository not found

Next Steps

When extending repository discovery, keep the distinction between search, lookup, and creation intact. Search should remain a small cached adapter over public GitHub metadata. Lookup should return stored wiki data without side effects. Creation should be the only flow that verifies a GitHub repository, writes storage records, applies rate limits, and starts indexing jobs. For adjacent behavior, read the repository API reference for exact route contracts, the indexing jobs API for job polling, and the storage setup page for the database and artifact services behind these calls.