Migrate registry-docgen to registry-dev - #791
Conversation
Sorry, something went wrong.
Move the docgen implementation into the monorepo, define and verify the canonical documentation format, and add separate replaceable documentation storage with filesystem and S3 coverage. Rename package tarball storage for clarity and add historical compatibility verification.
| @@ -0,0 +1,145 @@ | |||
| module Registry.Docgen.Decoder where | |||
There was a problem hiding this comment.
The Docgen tree is brought in basically as-is, which includes modules like this one — not sure if we need this separate JSON decoding module, but I didn't want to futz around with the implementation.
Sorry, something went wrong.
There was a problem hiding this comment.
This was because:
- Legacy decoder doesn't need to round-trip. That would bloat the decoders significantly since there's a lot of sum types.
- We aren't depending on argonaut, AFAIK
There's not a great ecosystem for purescript-json, other than codec.
Sorry, something went wrong.
Greptile SummaryThis PR migrates purescript-registry-docgen into the monorepo as the top-level docgen package, introduces a separate DocsStorage effect for canonical documentation artifacts, and renames the existing Storage effect to PackageStorage to distinguish the two concerns.
Confidence Score: 5/5Safe to merge. The new DocsStorage effect is well-isolated, tested at both the unit and E2E levels, and the Storage→PackageStorage rename is mechanical and consistent throughout. The DocsStorage FS backend achieves atomic create-only semantics with link and atomic replacement with rename, both verified by concurrent-upload unit tests. The S3 TOCTOU limitation is explicitly documented in code comments, AGENTS.md, the PR description, and tracked in a separate issue. The PackageStorage rename touches many files but is a pure identifier change with no behavioural difference. No incorrect logic, missing effects, or broken contracts were found. Files Needing Attention: No files require special attention. The WireMock state-machine mappings in nix/test/config.nix are the most complex addition but are well-structured and consistent with the E2E test fixture values. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[PackageInput\nmanifest + compiler docs + source CST] --> B[generatePackage]
B --> C{Validate modules\nnames · uniqueness\nsource paths · packages}
C -->|error| D[GenerationError]
C -->|ok| E[Convert from Legacy]
E --> F[Reexports.modulesWithReexports\nselective · hidden · aliased · transitive]
F -->|ReexportError| D
F -->|ok| G[DocPackage\nschemaVersion=1]
G --> H[Codec.docPackage\nJSON serialisation]
H --> I{DocsStorage}
I -->|upload create-only| J[FS: link tempfile to path\natomic EEXIST detection]
I -->|upload create-only| K[S3: preflight exists check\nthen PutObject - TOCTOU noted]
I -->|replace| L[FS: rename tempfile to path\natomic overwrite]
I -->|replace| M[S3: PutObject unconditional]
I -->|download| N[FS / S3: read + validateIdentity]
I -->|exists| O[FS: FS.Sync.exists\nS3: listObjects prefix check]
I -->|delete| P[FS: conditional remove\nS3: deleteObject idempotent]
subgraph PackageStorage
Q[Upload / Download / Delete / Query\npackage tarballs in purescript-registry bucket]
end
subgraph DocsStorageBucket
J
K
L
M
N
O
P
end
Reviews (4): Last reviewed commit: "Vendor documentation styles and harden t..." | Re-trigger Greptile |
Sorry, something went wrong.
Decode filesystem error codes safely and cover transitively blocked reexports.
| const absoluteUriRegex = /^(https|mailto):/i; | ||
|
|
||
| function isAllowedUri(uri) { | ||
| // return uri.startsWith('#') || uri.startsWith('/') || absoluteUriRegex.test(uri); |
There was a problem hiding this comment.
I don't remember what this is about exactly, but it probably needs some consideration. Right now only absolute uri's are allowed but I don't know if we want to do any other kind of linking, or even munging of links.
Sorry, something went wrong.
Bundle the existing Pursuit styles with rendered documentation while letting consumers choose their asset URLs. Make converted type precedence exhaustive and cover function and kind arguments.
| , foldMap stylesheet assets.fontStylesheet | ||
| , stylesheet assets.normalizeStylesheet | ||
| , stylesheet assets.pursuitStylesheet | ||
| , stylesheet assets.extraStylesheet |
There was a problem hiding this comment.
It's not obvious to be that we need to turn each stylesheet into an individual asset in the record. Rather, just configuring stylesheets is probably ok.
type DocumentAssets =
{ stylesheets :: Array String
}
Sorry, something went wrong.
Reviewers
At least 1 approving review is required to merge this pull request.
Closes #704. This moves @natefaubion's purescript-registry-docgen into this repository as the top-level docgen package and makes it the foundation for registry-owned package documentation, as discussed on Discord.
The goal is to build doc artifacts that we can store in a bucket, and those become the source of truth for Pursuit. HTML, search indexes, and other data can be rebuilt from the docs without needing to reach back out to GitHub or any dependency on purs publish directly.
What changed
The new docgen package includes lots of good things built by Nathan:
We use the registry manifest for almost everything, but since we have all the Pursuit backups we can use that for the documentation content itself. 107 package versions out of the ~6000 total have different dependency ranges in the registry and the docs used technically invalid versions, so we'd want to regenerate those.
Documentation storage
The main registry change in here, beyond the docgen target, is adding a separate DocsStorage effect with filesystem and S3 interpreters. Documentation is derived and replaceable, unlike immutable package tarballs, so it supports checking for existing artifacts, downloading, create-only uploading, explicitly replacing, and deleting docs.
The existing package storage effect has been renamed from Storage to PackageStorage to make the distinction clearer.
The production Nix configuration now supplies DOCS_BUCKET, defaulting to the purescript-registry-docs space that I created last week. DigitalOcean spaces cannot make the existence check and upload atomic, so production generation will still need to ensure there is only one writer for a given package version. That constraint is tracked in #525.
Historical documentation
This adds scripts for rendering documentation locally and validating historical Pursuit artifacts against registry manifests.
The migration audit found:
All 107 regeneration candidates have stored tarballs and at least one recorded compatible compiler. This PR adds the conversion and validation machinery but does not perform the production migration.
Scope
This PR is for defining, validating, rendering, and storing documentation artifacts, but it doesn't do any of the following:
Those are covered by #525, #789, #790, and #750.
The imported and adapted code is included under the repository's BSD-3-Clause license with permission from Nathan.