Markdown in, documentation site out — at any scale, anywhere you need it.

astro-huge-doc is the MicroWebStacks rendering engine that turns one or many Markdown repositories into a polished documentation website: code highlighting, Mermaid / PlantUML diagrams, image galleries, sortable tables, math, file-tree navigation and per-page TOC — with virtually no limit on content size. Content is parsed once into a database and rendered from there, so tens of thousands of pages behave like ten.

One engine, exactly three use cases:

flowchart LR
    A[Markdown repos<br/>local folders or GitHub] --> B[astro-huge-doc<br/>engine]
    B --> C[🧩 VS Code extension<br/>lite + json + SSR preview]
    B --> D[📦 Static export<br/>full + json + static]
    D --> D2[⚙️ GitHub Action<br/>deploys it to Pages]
    B --> E[🖥️ Node SSR server<br/>full + sqlite, render + cache on demand]

The GitHub Action is not a fourth engine home — it is the static build packaged for CI (see specification/reusable-render/spec.md). The full SQLite SSR server is the reference implementation of the engine's complete feature set and is always maintained as such.

astro-huge-doc is free, MIT-licensed open source — and it runs entirely on your machines. Zero telemetry, zero data collection, zero phone-home: not in the engine, not in the VS Code extension, not in the GitHub Action. Your content, your builds, and even your performance logs stay with you; what you document is nobody's business but yours. Performance is measured with a local benchmark (pnpm bench:lite) and local logs — never by watching users.

• SSR renders each page the moment it is requested, then caches it — true content-based ISR (Incremental Static Regeneration) with cache warmup. This is what the VS Code extension uses: pages are rendered lightweight and on demand as you browse.
• Static pre-renders the entire site into plain files. Push them to GitHub Pages (or any static host) and you have a complete documentation website with no dedicated server running anywhere.

Same code, no fork — a runtime switch selects the profile. Everything about rendering is shared; the full profile adds generation and storage on top. Profile, backend, and output are three independent axes; the three use cases above are the supported combinations (see specification/engine-profiles/spec.md).

Dropped from both profiles: Dataset SQL (duckdb) and Plotly charts, plus the MUI and Mantine UI kits that rode on them. May return later as full-only features.

Anything that is generation (collect) is full-only; rendering is shared. The lite build loads no native dependencies. In the extension it is fully lazy: startup walks the file tree only (labels and URLs derive from filenames — see specification/engine-profiles/spec.md), each page is parsed the first time it is viewed and cached by content hash, and an edit re-parses just that page with the server kept alive. The static export instead reads a pre-collected content.json plus a blobs/ folder. In both, images use Astro's passthrough image service, and model-viewer is aliased to an empty stub (dropping its ~980 kB chunk — the lite dist/client is ~634 kB).

• list of github repo and folders to fetch
• content is parsed and stored on sqlite db and blobs folder
• db and blobs can are mirrored on cloud (Iceberg + Bucket)
• content is versioned with viewable history
• content is rendered on demand with an Astro SSR handler
• rendered pages are streamed and cached with content sensitive ETag

• db manages versionning for documents and assets
• astro and server code are separately git managed
• astro handler has framework managed assets (css and js) and custom assets endpoint

• Multi levels cache from cloud to server disk to memory
• on demand html can be streamed and cached
• warming up can ensure cache filling

cache libraries considerations :

• lru-cache: simple, in-memory LRU with TTL.
• cacache: content-addressable disk cache; storing bodies keyed by hash.
• apicache (with a storage adapter) or express-cache-middleware if you want a plug-and-play layer, but hand-rolling with lru-cache gives more control.

• all pages content is stored as db items with pages ids
• astro renders pages on the server with React/Astro and streams html
• interactive islands will be hydrated on the client with the shipped page js assets

Point the renderer at a folder of Markdown and read it as the full website in your normal browser. No clone, no manifest.yaml, no content/ directory, no SQLite:

cd path/to/markdown
npx @microwebstacks/md-render preview

Or install it once and keep it:

npm install -g @microwebstacks/md-render
cd path/to/markdown
md-render preview

Both fetch the same package. npx installs nothing permanently but pays the download on a cold cache; the global install pays it once. Either way the first run also has to build the site, so give it a moment - the command prints building full site... and then the URL.

What it does and does not do:

• Node 22+ is the only prerequisite.
• The folder you are standing in is the content root. Markdown may sit directly in it; subfolders become the navigation tree.
• Rendering is the full profile with the json backend and static output - the same website a static deployment produces, including the file tree, TOC, code highlighting, client Mermaid/PlantUML, and PhotoSwipe galleries. No native/SQLite dependency is involved.
• Everything generated (dataset, blobs, built site) goes to a temporary directory outside your folder. Nothing is written next to your notes.
• Ctrl+C stops the server and removes the temporary files.
• Dependency and build directories beside your notes (node_modules/, dist/, build/, out/, target/, vendor/, venv/, __pycache__/, coverage/, and any hidden directory) are never collected. Override with collect.ignore in a manifest.yaml if your documents really do live in one of them.

md-render preview [workspace]

  --docs-root <path>   content root inside the workspace (default: the workspace itself)
  --manifest <path>    manifest to use
  --host <host>        default 127.0.0.1
  --port <port>        default 4321 (0 picks any free port)
  --no-open            do not open a browser - for CI, remote shells, and containers
  --keep               keep the temporary artifact and print its location

The content root is resolved in this order: --docs-root, then a manifest.yaml in the workspace if the folder it names exists, then the workspace itself. A workspace that is already a configured astro-huge-doc project therefore still renders what its manifest points at.

To produce a static artifact instead of serving one, use md-render build (see specification/reusable-render/spec.md):

npx @microwebstacks/md-render build --workspace . --out-dir ./dist

The engine can be configured from four places:

Start by copying the example file:

copy .env.example .env

config.js imports src/libs/load-env.js first, so environment resolution happens before the manifest/default config is finalized.

Notes:

• In standalone use, .env wins because dotenv.config({override: true}) is the default behavior in src/libs/load-env.js.
• In VS Code preview, the extension sets MICROWEBSTACKS_DOTENV_OVERRIDE=false, so the workspace .env fills gaps but cannot replace runtime-critical values that the extension already injected.
• DOCS_BACKEND defaults from DOCS_PROFILE when unset: lite -> json, otherwise sqlite.
• Relative env paths resolve from the workspace root, except MICROWEBSTACKS_OUTDIR, which resolves from the engine root.

These settings live under the microwebstacks.preview.* namespace in VS Code.

engineSource and enginePath are extension-only settings; they do not map to engine env vars. docsRoot and krokiServer affect the engine by being translated into runtime env vars by the extension.

User-facing env vars:

Advanced or runtime-injected env vars:

Mermaid and PlantUML render client-side in the browser and VS Code preview. They require no Java, Docker, or external diagram server. PlantUML uses the official @plantuml/core browser engine and loads it only on pages containing PlantUML diagrams.

BlockDiag and other languages routed to kroki use MICROWEBSTACKS_KROKI_SERVER. Set it through .env, the shell, or the VS Code setting microwebstacks.preview.krokiServer when those diagrams are present. PlantUML can be routed back to Kroki as a compatibility fallback:

diagram:
  languages:
    plantuml: kroki

Client PlantUML cannot automatically read arbitrary local !include files, and the npm engine omits large optional sprite bundles. Use the explicit Kroki route when affected content is not compatible with client mode.

Use this for BlockDiag or explicitly Kroki-routed diagrams without sending diagram source to an external service:

MICROWEBSTACKS_KROKI_SERVER=http://localhost:18000

Start the local renderer:

pnpm kroki:up
# equivalent to: docker compose up -d

Then run one of the normal preview flows:

pnpm dev

or regenerate the data explicitly:

pnpm collect
pnpm diagrams
pnpm dev

To force a fresh diagram render before testing, clear generated diagram rows and static SVG blobs first:

pnpm clean:diagrams
pnpm diagrams
pnpm dev

For the lite/JSON profile, a fresh collect already recreates dataset/json/content.json and dataset/json/blobs. To explicitly test diagram rendering from a clean JSON dataset:

$env:DOCS_BACKEND="json"
pnpm collect
pnpm clean:diagrams
pnpm diagrams
pnpm dev

After pnpm clean:diagrams, the next pnpm diagrams run calls the configured Kroki URL again only for Kroki-routed diagrams. Client Mermaid and PlantUML are skipped by the collection renderer.

Rendered diagram SVG files are served from /blobs/<12-hex>.svg; the full content hash remains in the dataset metadata.

When you are done with the local renderer:

pnpm kroki:down
# equivalent to: docker compose down

Use the public service only when it is acceptable to send diagram source to kroki.io:

MICROWEBSTACKS_KROKI_SERVER=https://kroki.io

Then run:

pnpm collect
pnpm diagrams
pnpm dev

For a company-hosted or otherwise custom Kroki-compatible endpoint, set the same variable to the internal base URL:

MICROWEBSTACKS_KROKI_SERVER=https://kroki.example.internal

Then run the same commands:

pnpm collect
pnpm diagrams
pnpm dev

• Configure fetch.github in manifest.yaml (single object or list). Use fetch.select to run one repo from a list of examples, or omit it/use all to fetch every entry. Example:

  fetch:
    select: MicroWebStacks/astro-big-doc
    github:
      - repo: MicroWebStacks/astro-big-doc
        branch: main
        folders: [content]
        dest: content
      - repo: VectorMind/alm-ontology
        branch: main
        dest: content
  output:
    content: content
  render:
    folder: demo

• folders pulls those subfolders and flattens their contents into dest; omit folders to copy the whole repo. dest defaults to the repo name and is cleared before copying.
• output.content remains the fetch destination / legacy default docs root. Set render.folder when you want to render a different folder such as a bundled local demo/ tree while still fetching remote content into content/.
• Switch examples by changing fetch.select to another repo value.
• Set GITHUB_TOKEN to avoid GitHub rate limits.
• Run pnpm fetch (or node scripts/fetch.js) after installing dependencies.

• All configs are optional and have defaults
• Configure collect in manifest.yaml. Example:

  collect:
  folder_single_doc: false
  file_link_ext: ["svg","webp","png","jpeg","jpg","xlsx","glb"]
  file_compress_ext: ['txt','md','json','csv','tsv','yaml','yml']
  external_storage_kb: 512
  inline_compression_kb: 32

  • folder_single_doc default is false for one document per file, when true, generates one document per folder merging its markdown files.
  • file_link_ext : only these extensions will be considered as assets to manage
  • file_compress_ext : files subject to compressions in blobs storage
  • external_storage_kb : threshold to manage blobs in folders and not in db
  • inline_compression_kb : threshold above which db blobs get compressed
• Run pnpm collect to parse the content directory Markdown and referenced assets and store them in dataset/content.db

The repository root doubles as a composite GitHub Action that renders a consumer's Markdown workspace into a static site artifact, ready for actions/upload-pages-artifact + actions/deploy-pages:

- name: Render static site
  uses: MicroWebStacks/astro-huge-doc@<pinned-tag-or-sha>
  with:
    engine-version: '<exact @microwebstacks/md-render version>'
    workspace: '.'
    out-dir: 'dist'
    base: '/my-repo/'

See action.yml for all inputs and .github/workflows/render-example.yml for a complete reference workflow including the Pages deploy jobs.

The repository includes a first-pass desktop VS Code extension in packages/vscode-extension. It previews Markdown documentation from the opened workspace through the existing astro-huge-doc SSR renderer in an embedded VS Code panel. It follows active .md files and can be locked from the panel toolbar; .mdx is not treated as a supported document type.

Configuration for extension mode is documented in Usage -> configuration. This section covers install and run only.

The extension follows VS Code storage conventions:

• Markdown, supporting assets, and optional manifest.yaml are read from the opened workspace.
• Generated preview databases and cache files are stored in VS Code workspace-scoped extension storage, not in the workspace and not in the installed extension directory.
• The standalone repo flow still works with manifest.yaml, pnpm collect, pnpm build, and pnpm server.

From this repository:

pnpm install
pnpm build

The extension expects the Astro SSR build at dist/server/entry.mjs.

You do not need to publish the extension to test it against another workspace. Launch an Extension Development Host and point it at the Markdown workspace:

code --extensionDevelopmentPath C:\dev\MicroWebStacks\astro-huge-doc\packages\vscode-extension C:\path\to\docs-workspace

In the Extension Development Host window, run:

MicroWebStacks: Preview Docs in VS Code

Additional commands:

MicroWebStacks: Open Docs in Browser
MicroWebStacks: Restart Docs Preview Server
MicroWebStacks: Stop Docs Preview Server

For local installation without Marketplace publishing, build/package the extension and install the generated .vsix:

pnpm ext:release
code --install-extension .\packages\vscode-extension\markdown-site-preview.vsix

The installed extension normally does not require system Node/npm: in auto mode it uses the bundled lite/json engine shipped inside the VSIX and uses VS Code's bundled runtime to run it. The pinned published @microwebstacks/md-render package remains available as a fallback or when you explicitly force microwebstacks.preview.engineSource=registry. microwebstacks.preview.enginePath is now only an advanced development override when you want the extension to run against a local checkout instead of the bundled or published engine.

• XLSX files support dropped but could potentially generate two assets, original file for download and asset table for direct asset vieweing