Migrate to spago v1 - #316
Conversation
Sorry, something went wrong.
|
Oh no, I tried running locally with same memory settings as production build (set -o noglob && stack exec trypurescript -- +RTS -N2 -A128m -M3G -s -RTS 8081 $(spago sources)) and ran out of memory. ❯ (set -o noglob && stack exec trypurescript -- +RTS -N2 -A128m -M3G -s -RTS 8081 $(spago sources))
Reading Spago workspace configuration...
✓ Selecting package to build: try-purescript-server
Warning: nix (Nix) 2.34.7+1 is on the PATH (at /run/current-system/sw/bin/nix)
but Stack's Nix integration is disabled. To mute this message in
future, set notify-if-nix-on-path: false in Stack's configuration.
trypurescript: Heap exhausted;
trypurescript: Current maximum heap size is 3221225472 bytes (3072 MB).
trypurescript: Use `+RTS -M<size>' to increase it.
36,907,900,144 bytes allocated in the heap
7,635,067,328 bytes copied during GC
3,479,980,704 bytes maximum residency (28 sample(s))
21,874,656 bytes maximum slop
3739 MiB total memory in use (0 MB lost due to fragmentation)
Tot time (elapsed) Avg pause Max pause
Gen 0 108 colls, 42 par 2.344s 3.292s 0.0305s 0.1313s
Gen 1 28 colls, 5 par 39.524s 111.686s 3.9888s 7.0771s
Parallel GC work balance: 86.50% (serial 0%, perfect 100%)
TASKS: 9 (1 bound, 8 peak workers (8 total), using -N2)
SPARKS: 7427 (7381 converted, 0 overflowed, 0 dud, 0 GC'd, 46 fizzled)
INIT time 0.001s ( 0.080s elapsed)
MUT time 42.664s ( 9.678s elapsed)
GC time 41.868s (114.978s elapsed)
EXIT time 0.002s ( 0.005s elapsed)
Total time 84.534s (124.742s elapsed)
Alloc rate 865,091,047 bytes per MUT second
Productivity 50.5% of total user, 7.8% of total elapsed
Running with -M4G succeeds. |
Sorry, something went wrong.
|
Dumping the result of investigating this with Opus: https://gist.github.com/pete-murphy/bd776708d72a00a80ba1b7e7845f2982. Option 3 seems the most attractive if it could work. |
Sorry, something went wrong.
|
Option 3 is indeed attractive and I’m perfectly happy experimenting with reworking try PureScript, it’s definitely not received a lot of love recently. I wouldn’t dismiss option 2 offhand — Try PureScript is browser based so certainly node packages, etc. can safely be excluded |
Sorry, something went wrong.
I did end up looking into Option 2 after all since it seems much easier and I am less likely to get to the harder Option 3 in near term. If I trim out Node packages & some large packages that have no dependents, I am able to run the server with (set -o noglob && stack exec trypurescript -- +RTS -N2 -A128m -M3G -s -RTS 8081 $(spago sources)) without exceeding that max heap size. The result was this commit: 0fd79d9. The non-Node packages (largest packages with no dependents) that were omitted:
#4269d0 are packages that were kept 
Here's a notebook for exploring what packages to omit if it's useful: https://new.observablehq.com/@pete-murphy/try-pure-script-package-set-trim-explorer (mostly made by Opus 4.8) The "no dependents" criterion is arbitrary, open to ideas if you can think of something that would be more fair? (Age of package, some other popularity metric...) |
Sorry, something went wrong.
|
Quick followup on your questions in the gist:
I think the option 2 is the way to go, but before we merge, could you measure the trimmed set's peak (maxLive from +RTS -s) on a cold boot (no output)? If it's much above ~2.5 GB, it's probably worth trimming a little further. Longer term, option 3 looks even better, but we don't need to do it now. |
Sorry, something went wrong.
|
Got a little access to Fable, so ran it over this change — here are results, lightly proofread: <---- fable ----> Results, combined with the dependency graph from spago.lock:
So my suggestion: trim by "cannot execute in the playground" instead of size/dependents. It cuts ~24% more modules than the current trim (more memory headroom), it's fully principled — nothing is removed for being big or unpopular, only for being unable to run — and it lets spec, test-unit, and react-basic-dom-beta come back. Two caveats:
{
"unshimmed": {
"affjax-node": [
"url",
"xhr2"
],
"yoga-pino": [
"@opentelemetry/api-logs",
"pino"
],
"motion": [
"motion/react",
"motion/react-m"
],
"yoga-om-strom": [
"node:fs"
],
"hylograph-d3-kernel": [
"d3-drag",
"d3-force",
"d3-selection"
],
"react-testing-library": [
"@testing-library/react/pure.js",
"@testing-library/user-event"
],
"yoga-opentelemetry": [
"@opentelemetry/api",
"@opentelemetry/api-logs",
"@opentelemetry/exporter-logs-otlp-http",
"@opentelemetry/exporter-trace-otlp-http",
"@opentelemetry/instrumentation-http",
"@opentelemetry/instrumentation-pino",
"@opentelemetry/resources",
"@opentelemetry/sdk-logs",
"@opentelemetry/sdk-node",
"@opentelemetry/sdk-trace-base",
"@opentelemetry/sdk-trace-node",
"@opentelemetry/semantic-conventions"
],
"node-fs": [
"node:fs",
"util"
],
"org-doc": [
"instaparse",
"node:fs"
],
"nanoid": [
"nanoid"
],
"react-virtuoso": [
"react-virtuoso"
],
"yoga-redis": [
"ioredis"
],
"chartjs": [
"chart.js/auto"
],
"yoga-fastify": [
"@fastify/cors",
"@fastify/helmet",
"@fastify/rate-limit",
"@fastify/websocket",
"argon2",
"fastify",
"jose"
],
"yoga-test-docker": [
"child_process"
],
"node-net": [
"net",
"node:net"
],
"react-markdown": [
"react-markdown",
"remark-breaks",
"remark-gfm"
],
"graphql-client": [
"@apollo/client/core/index.js",
"@apollo/client/link/context/index.js",
"@apollo/client/link/subscriptions/index.js",
"@apollo/client/utilities/index.js",
"@urql/core",
"graphql-ws",
"wonka"
],
"yoga-fetch": [
"node-fetch"
],
"visx": [
"@visx/annotation",
"@visx/axis",
"@visx/brush",
"@visx/clip-path",
"@visx/curve",
"@visx/event",
"@visx/geo",
"@visx/gradient",
"@visx/grid",
"@visx/group",
"@visx/heatmap",
"@visx/hierarchy",
"@visx/legend",
"@visx/mock-data",
"@visx/network",
"@visx/pattern",
"@visx/responsive",
"@visx/scale",
"@visx/shape",
"@visx/stats",
"@visx/threshold",
"@visx/tooltip",
"@visx/zoom",
"d3-format",
"d3-time-format",
"topojson-client"
],
"postgresql": [
"buffer",
"pg",
"pg-copy-streams",
"postgres-interval",
"postgres-range",
"stream"
],
"node-child-process": [
"node:child_process"
],
"react-aria": [
"@react-aria/button",
"@react-aria/combobox",
"@react-aria/focus",
"@react-aria/interactions",
"@react-aria/listbox",
"@react-aria/overlays",
"@react-aria/utils"
],
"gojs": [
"gojs"
],
"echarts-simple": [
"echarts"
],
"fakerjs": [
"@faker-js/faker"
],
"yoga-heroui": [
"@heroui/react"
],
"xterm": [
"xterm",
"xterm-addon-fit",
"xterm-addon-web-links",
"xterm-addon-webgl"
],
"recharts": [
"recharts"
],
"yaml-next": [
"js-yaml"
],
"react-basic-storybook": [
"@storybook/addon-actions",
"@storybook/addon-docs",
"@storybook/testing-library"
],
"nextui": [
"@nextui-org/react",
"next-themes"
],
"url-regex-safe": [
"url-regex-safe"
],
"node-os": [
"node:os",
"node:util"
],
"spec-discovery": [
"fs",
"path",
"url"
],
"which": [
"which"
],
"yoga-dynamodb": [
"@aws-sdk/client-dynamodb",
"@aws-sdk/lib-dynamodb"
],
"droplet": [
"pg"
],
"oak": [
"virtual-dom/create-element",
"virtual-dom/diff",
"virtual-dom/h",
"virtual-dom/patch"
],
"yoga-jaeger": [
"jaeger-client"
],
"simple-ulid": [
"crypto"
],
"yoga-postgres": [
"pg"
],
"node-zlib": [
"node:zlib"
],
"elmish-hooks": [
"stacktrace-parser"
],
"debounce": [
"debouncing"
],
"yoga-react-native": [
"react-native",
"react-native-fs",
"react-native-macos",
"twrnc"
],
"rough-notation": [
"rough-notation"
],
"reactix": [
"@testing-library/react",
"react-dom/test-utils"
],
"framer-motion": [
"motion/react"
],
"choku": [
"chalk"
],
"hylograph-simulation": [
"d3-drag",
"d3-force",
"d3-selection"
],
"yoga-next-fastify": [
"next"
],
"yoga-shadcn": [
"cmdk",
"embla-carousel-react",
"input-otp",
"radix-ui",
"react-day-picker",
"react-resizable-panels",
"sonner",
"vaul"
],
"node-readline": [
"node:readline"
],
"node-human-signals": [
"os"
],
"next-purs-rsc": [
"next/cache",
"next/dynamic",
"next/font/google",
"next/font/local",
"next/headers",
"next/image",
"next/link",
"next/navigation",
"next/og",
"next/script",
"next/server",
"next/web-vitals"
],
"systemd-journald": [
"systemd-journald"
],
"node-sqlite3": [
"sqlite3"
],
"ink": [
"ink"
],
"address-rfc2821": [
"address-rfc2821"
],
"node-http": [
"node:http",
"node:https"
],
"deno": [
"https://deno.land/std@0.144.0/crypto/mod.ts",
"https://deno.land/std@0.144.0/dotenv/mod.ts",
"https://deno.land/std@0.144.0/fs/mod.ts",
"https://deno.land/std@0.144.0/http/server.ts",
"https://deno.land/std@0.145.0/datetime/mod.ts",
"https://deno.land/std@0.145.0/log/mod.ts",
"https://deno.land/std@0.145.0/uuid/mod.ts"
],
"elmish-enzyme": [
"enzyme"
],
"yoga-config": [
"smol-toml",
"yaml"
],
"cbor-stream": [
"cbor-x"
],
"blessed": [
"@shamansir/everblessed",
"blessed",
"fs",
"path",
"reblessed",
"terminal-size",
"url",
"util"
],
"node-tls": [
"node:tls"
],
"elmish-testing-library": [
"@happy-dom/global-registrator",
"react-dom/test-utils.js"
],
"node-process": [
"process"
],
"node-buffer": [
"node:buffer",
"util"
],
"react-dnd-kit": [
"@dnd-kit/abstract/modifiers",
"@dnd-kit/collision",
"@dnd-kit/dom",
"@dnd-kit/dom/modifiers",
"@dnd-kit/dom/utilities",
"@dnd-kit/geometry",
"@dnd-kit/helpers",
"@dnd-kit/react",
"@dnd-kit/react/sortable"
],
"rito": [
"three/examples/jsm/loaders/GLTFLoader.js",
"three/examples/jsm/postprocessing/BloomPass.js",
"three/examples/jsm/postprocessing/EffectComposer.js",
"three/examples/jsm/postprocessing/GlitchPass.js",
"three/examples/jsm/postprocessing/RenderPass.js",
"three/examples/jsm/postprocessing/ShaderPass.js",
"three/examples/jsm/postprocessing/UnrealBloomPass.js",
"three/examples/jsm/renderers/CSS2DRenderer.js",
"three/examples/jsm/renderers/CSS3DRenderer.js",
"three/src/cameras/PerspectiveCamera.js",
"three/src/core/BufferAttribute.js",
"three/src/core/BufferGeometry.js",
"three/src/core/InstancedBufferAttribute.js",
"three/src/core/Raycaster.js",
"three/src/geometries/BoxGeometry.js",
"three/src/geometries/CapsuleGeometry.js",
"three/src/geometries/CylinderGeometry.js",
"three/src/geometries/PlaneGeometry.js",
"three/src/geometries/SphereGeometry.js",
"three/src/lights/AmbientLight.js",
"three/src/lights/DirectionalLight.js",
"three/src/lights/PointLight.js",
"three/src/loaders/CubeTextureLoader.js",
"three/src/loaders/TextureLoader.js",
"three/src/materials/MeshBasicMaterial.js",
"three/src/materials/MeshLambertMaterial.js",
"three/src/materials/MeshPhongMaterial.js",
"three/src/materials/MeshStandardMaterial.js",
"three/src/materials/RawShaderMaterial.js",
"three/src/materials/ShaderMaterial.js",
"three/src/math/Box3.js",
"three/src/math/Color.js",
"three/src/math/Euler.js",
"three/src/math/Matrix4.js",
"three/src/math/Quaternion.js",
"three/src/math/Sphere.js",
"three/src/math/Vector2.js",
"three/src/math/Vector3.js",
"three/src/objects/Group.js",
"three/src/objects/InstancedMesh.js",
"three/src/objects/Mesh.js",
"three/src/objects/Points.js",
"three/src/renderers/WebGLRenderer.js",
"three/src/scenes/FogExp2.js",
"three/src/scenes/Scene.js"
],
"milkis": [
"node-fetch"
],
"node-http2": [
"node:http2"
],
"webextension-polyfill": [
"webextension-polyfill"
],
"i18next": [
"i18next",
"i18next-browser-languagedetector"
],
"supabase": [
"@supabase/ssr",
"@supabase/supabase-js"
],
"lumi-components": [
"jss",
"jss-preset-default",
"qrcode.react",
"react-media-hook",
"react-password-strength-bar"
],
"marked": [
"marked"
],
"elmish": [
"react-dom/server.js"
],
"prospero": [
"@fastify/multipart",
"graphql",
"node:crypto"
],
"axon": [
"bun",
"node:http",
"node:net",
"node:stream",
"stream"
],
"markdown-it-js": [
"markdown-it",
"markdown-it-collapsible"
],
"react-basic-dnd": [
"react-dnd",
"react-dnd-html5-backend",
"react-dnd-test-backend",
"react-dnd-touch-backend"
],
"optparse": [
"child_process"
],
"node-path": [
"node:path"
],
"node-url": [
"node:url"
],
"z3": [
"z3-solver"
],
"ulid": [
"ulid"
],
"node-streams": [
"node:stream"
],
"leveldb": [
"level"
],
"whine-core": [
"glob",
"micromatch",
"vscode",
"vscode-languageclient/node.js",
"vscode-languageserver-textdocument",
"vscode-languageserver/node",
"yaml"
],
"ace": [
"ace-builds",
"ace/anchor",
"ace/background_tokenizer",
"ace/document",
"ace/edit_session",
"ace/editor",
"ace/ext/language_tools",
"ace/range",
"ace/scrollbar",
"ace/search",
"ace/selection",
"ace/token_iterator",
"ace/tokenizer",
"ace/undomanager",
"ace/virtual_renderer"
],
"node-stream-pipes": [
"stream"
],
"node-event-emitter": [
"node:events"
],
"golem-fetch": [
"node:url"
],
"express": [
"cookie-parser",
"express",
"http",
"https"
],
"idb": [
"idb"
],
"yoga-elasticsearch": [
"@elastic/elasticsearch"
],
"mysql": [
"mysql"
],
"yoga-sqlite": [
"@libsql/client"
],
"meowclient": [
"keys-converter",
"meowclient"
],
"n3": [
"n3"
],
"milkdown": [
"@milkdown/crepe",
"@milkdown/kit/core",
"@milkdown/kit/utils"
],
"yoga-acp-om": [
"@anthropic-ai/claude-agent-sdk"
],
"vitest": [
"vitest"
],
"ezfetch": [
"buffer",
"stream"
],
"tanstack-query": [
"@tanstack/react-query"
],
"yoga-better-auth": [
"better-auth",
"better-auth/client",
"better-auth/db",
"better-auth/plugins",
"pg"
],
"node-execa": [
"process"
],
"webb-commandline": [
"child_process"
],
"csv-stream": [
"csv-parse",
"csv-stringify"
],
"react-icons": [
"react-icons/ai",
"react-icons/bi",
"react-icons/bs",
"react-icons/cg",
"react-icons/ci",
"react-icons/di",
"react-icons/fa",
"react-icons/fa6",
"react-icons/fc",
"react-icons/fi",
"react-icons/gi",
"react-icons/go",
"react-icons/gr",
"react-icons/hi",
"react-icons/hi2",
"react-icons/im",
"react-icons/io",
"react-icons/io5",
"react-icons/lia",
"react-icons/lu",
"react-icons/md",
"react-icons/pi",
"react-icons/ri",
"react-icons/rx",
"react-icons/si",
"react-icons/sl",
"react-icons/tb",
"react-icons/tfi",
"react-icons/ti",
"react-icons/vsc",
"react-icons/wi"
],
"react-basic-emotion": [
"@emotion/react"
],
"faker-ffi": [
"@faker-js/faker"
],
"nextjs": [
"next/document",
"next/head",
"next/image.js",
"next/link",
"next/navigation",
"next/script",
"swr"
],
"node-workerbees": [
"worker_threads"
],
"jsdom": [
"jsdom"
]
},
"broken_closure": [
"ace",
"address-rfc2821",
"affjax-node",
"axon",
"benchlib",
"blessed",
"cbor-stream",
"chartjs",
"chartjs-halogen",
"choku",
"compile-fail",
"css-class-name-extractor",
"csv-stream",
"debounce",
"deno",
"dotenv",
"droplet",
"echarts-simple",
"elmish",
"elmish-enzyme",
"elmish-hooks",
"elmish-html",
"elmish-testing-library",
"elmish-time-machine",
"environment",
"express",
"ezfetch",
"fahrtwind",
"faker-ffi",
"fakerjs",
"framer-motion",
"gojs",
"golden-test",
"golem-fetch",
"graphql-client",
"halogen-echarts-simple",
"halogen-xterm",
"httpurple",
"hylograph-d3-kernel",
"hylograph-simulation",
"hylograph-simulation-halogen",
"i18next",
"idb",
"ink",
"jsdom",
"leveldb",
"logging-journald",
"lumi-components",
"markdown-it-js",
"marked",
"meowclient",
"milkdown",
"milkis",
"motion",
"mysql",
"n3",
"nanoid",
"next-purs-rsc",
"nextjs",
"nextui",
"node-buffer",
"node-child-process",
"node-event-emitter",
"node-execa",
"node-fs",
"node-glob-basic",
"node-http",
"node-http2",
"node-human-signals",
"node-net",
"node-os",
"node-path",
"node-process",
"node-readline",
"node-sqlite3",
"node-stream-pipes",
"node-streams",
"node-tls",
"node-url",
"node-workerbees",
"node-zlib",
"oak",
"oak-debug",
"open-mkdirp-aff",
"optparse",
"org-doc",
"postgresql",
"prospero",
"psa-utils",
"react-aria",
"react-basic-dnd",
"react-basic-emotion",
"react-basic-storybook",
"react-dnd-kit",
"react-icons",
"react-markdown",
"react-testing-library",
"react-virtuoso",
"reactix",
"recharts",
"rito",
"rough-notation",
"simple-ulid",
"spec-discovery",
"spec-node",
"spec-reporter-xunit",
"supabase",
"systemd-journald",
"tanstack-query",
"toestand",
"ts-bridge",
"ulid",
"url-regex-safe",
"visx",
"vitest",
"webb-commandline",
"webb-directory",
"webb-file",
"webb-test",
"webextension-polyfill",
"which",
"whine-core",
"xterm",
"yaml-next",
"yoga-acp-om",
"yoga-better-auth",
"yoga-config",
"yoga-docker-compose",
"yoga-dynamodb",
"yoga-elasticsearch",
"yoga-fastify",
"yoga-fastify-om",
"yoga-fetch",
"yoga-fetch-om",
"yoga-heroui",
"yoga-jaeger",
"yoga-next-fastify",
"yoga-om-strom",
"yoga-om-workerbees",
"yoga-opentelemetry",
"yoga-pino",
"yoga-postgres",
"yoga-react-native",
"yoga-redis",
"yoga-shadcn",
"yoga-sqlite",
"yoga-test-docker",
"z3"
],
"cut": [
"affjax-node",
"axon",
"benchlib",
"blessed",
"cbor-stream",
"compile-fail",
"css-class-name-extractor",
"css-frameworks",
"csv-stream",
"dotenv",
"environment",
"express",
"ezfetch",
"framer-motion",
"golden-test",
"golem-fetch",
"graphql-client",
"httpurple",
"ink",
"lumi-components",
"marked",
"meowclient",
"motion",
"next-purs-rsc",
"node-buffer",
"node-child-process",
"node-event-emitter",
"node-execa",
"node-fs",
"node-glob-basic",
"node-http",
"node-http2",
"node-human-signals",
"node-net",
"node-os",
"node-path",
"node-process",
"node-readline",
"node-stream-pipes",
"node-streams",
"node-tls",
"node-url",
"node-workerbees",
"node-zlib",
"open-mkdirp-aff",
"optparse",
"org-doc",
"pmock",
"postgresql",
"prospero",
"psa-utils",
"react-basic-dom-beta",
"react-icons",
"react-testing-library",
"rito",
"simple-ulid",
"spec",
"spec-discovery",
"spec-mocha",
"spec-node",
"spec-reporter-xunit",
"supabase",
"test-unit",
"ts-bridge",
"webb-commandline",
"webb-directory",
"webb-file",
"webb-test",
"whine-core",
"yoga-acp-om",
"yoga-config",
"yoga-docker-compose",
"yoga-fastify-om",
"yoga-fetch-om",
"yoga-om-strom",
"yoga-om-workerbees",
"yoga-opentelemetry",
"yoga-sqlite",
"yoga-test-docker"
]
}import os, re, json, collections
SHIMMED = {"big-integer", "decimal.js", "react", "react-dom",
"react-dom/client", "react-dom/server", "uuid"}
PAT = re.compile(
r'''(?:\bfrom\s*|(?<!\w)import\s*\(\s*|(?<!\w)require\s*\(\s*)['"]([^'"\n]+)['"]''')
SKIP = {'node_modules', 'test', 'tests', '.git', 'examples', 'example', 'bench'}
bare = collections.defaultdict(set)
for pkgdir in sorted(os.listdir('.spago/p')):
root = os.path.join('.spago/p', pkgdir)
if not os.path.isdir(root): continue
name = re.sub(r'-\d+\.\d+\.\d+.*$', '', pkgdir)
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames if d not in SKIP]
for f in filenames:
if not f.endswith(('.js', '.mjs', '.cjs')): continue
try: text = open(os.path.join(dirpath, f), encoding='utf-8', errors='replace').read()
except OSError: continue
for m in PAT.finditer(text):
spec = m.group(1)
if spec.startswith(('.', '/')) or ' ' in spec or spec.startswith('['): continue
bare[name].add(spec)
unshimmed = {p: sorted(s - SHIMMED) for p, s in bare.items() if s - SHIMMED}
lock = json.load(open('spago.lock'))
deps = {p: set(v.get('dependencies', [])) for p, v in lock['packages'].items()}
rev = collections.defaultdict(set)
for p, ds in deps.items():
for d in ds: rev[d].add(p)
broken = set(unshimmed) & set(deps)
frontier = list(broken)
while frontier:
p = frontier.pop()
for r in rev[p]:
if r not in broken:
broken.add(r); frontier.append(r)
print(json.dumps({"unshimmed": unshimmed, "cannot_run_closure": sorted(broken)}, indent=2)) |
Sorry, something went wrong.
maxLive for the current (on this branch) trimmed package set is 2,064 MiB (2,164 MB). If I apply the suggestion from Fable (trim by "cannot execute in the playground" instead of size/dependents), maxLive is down to 1,899 MiB (1,991 MB). I had Fable run some experiments on a branch in my fork. The HTML report of findings: report.html, and a tl;dr chart: 
The recent findings / this chart show the warm case (a populated .psci_modules cache) is actually worse memory-wise than cold boot, for reasons that I am not yet fully understanding (Fable says that cold start is able to do sharing that warm start is not able to do.) If that's true, I think that refutes the reasoning behind Option 3, which was based on a bad assumption that wasn't tested. I ran the server manually a couple times, with and without a .psci_modules cache, and was able to reproduce the results that Fable reported, but haven't had a chance to look into it. I also didn't realize, til looking at these findings, that the full package set is just barely over the limit 🧐 .
Hmm, that's not great: even the trimmed package set here is bigger than what it currently is. Do you think it would be best to pare it down even further (trim the "cannot run" packages Fable found above)? That would get it to a lower module-count than the currently-deployed package set: 4,482 modules on this branch (trim-current in chart above), 3,769 if trim "cannot run" (trim-cannot-run), and 3,946 in what's currently deployed according to what I measured here. It seems like good news, that the warm cache is the worst case for max-live heap usage and could seemingly be optimized to behave more like the cold cache. But then, it's a new mystery to me why the server would OOM with the current package set size, I'd expect its memory usage to be somewhere between trim-current and trim-cannot-run (just based on module count as a rough approximation of size). |
Sorry, something went wrong.
|
OK, I had some time today to run this branch locally and things look good. Walking through my findings so I can remember this as part of merging: #! /usr/bin/env bash
set -ex
set -o noglob
export XDG_CACHE_HOME="$PWD/.spago-cache"
spago install
# Note: the heap cap below must fit within the droplet's RAM; the package set
# as of psc-0.15.15-20260701 (641 packages) needs more than the 3G that
# sufficed for psc-0.15.13-20231219 (459 packages).
exec trypurescript +RTS -N2 -A128m -M6G -RTS 8081 $(spago sources)
So for this pull request, could you update the deploy script to use spago fetch instead? export XDG_CACHE_HOME="$PWD/.spago-cache"
spago fetch --pure
exec trypurescript +RTS -N2 -A128m -M3G -RTS 8081 $(spago sources)...plus I will fix the droplet to upgrade node to ≥22.5, install new spago/purescript, and remove the legacy spago. Will also delete the stale legacy staging/.spago contents on the server. Other stuffThis branch trimmed thingss down, but we don't have record of the exclusions — the next time someone runs spago install $(spago ls packages --json ...) they'd reinstall the full set right? Should we introduce an exclusion list as an artifact, like staging/excluded-packages.txt with a one-line rationale per package or per-category, and have the RELEASE.md install step filter through it? $ spago install $(spago ls packages --json --quiet | jq -r 'to_entries[] | select(.value.type == "registry") | .key' | grep -vxF -f excluded-packages.txt)It would also be useful to add your maxLive measurement method (+RTS -s on a cold boot, the way you measured 2,064 MiB in this thread) to the existing "check the memory" paragraph in RELEASE.md; right now it says that a reviewer should check memory but not how, and this would be useful to retain. |
Sorry, something went wrong.
There was a problem hiding this comment.
This note isn't true anymore since the new spago doesn't rely on the metadata package. We can delete this line.
Sorry, something went wrong.
| - zipperarray | ||
| workspace: | ||
| packageSet: | ||
| registry: 77.6.0 |
There was a problem hiding this comment.
We've advanced to 77.10.1 if you want to update this
Sorry, something went wrong.
| "esbuild": "^0.14.43", | ||
| "http-server": "^14.1.0", | ||
| "purescript": "^0.15.2", | ||
| "purescript": "^0.15.16", |
There was a problem hiding this comment.
Everything else in here is pinned to 0.15.15; this works, but should use the same version for consistency?
Sorry, something went wrong.
|
Regarding your research -- I see; while I don't know why the numbers diverge necessarily, at least we know that production is basically always in the warm case so that's the number for us to look at. So yeah:
...and otherwise we can merge this, and I'll make the droplet changes when it's deploy time. |
Sorry, something went wrong.
and bring compiler up to date with latest Hackage release (0.15.15)
Addresses PR feedback that the trim in 0fd79d9 left no record of what was removed or why, so the next `spago install $(spago ls packages ...)` per RELEASE.md would have silently reinstalled the full set. - Add staging/excluded-packages.txt: 148 packages, one line each with the reason. The only criterion is that a package cannot execute in the playground: its foreign modules (or a dependency's) import bare JS specifiers that the import map in client/public/frame.html does not shim. Packages previously cut for size despite being runnable (spec, spec-mocha, test-unit, pmock, react-basic-dom-beta, css-frameworks) are back in. - Filter the RELEASE.md install step through the exclusion list, and document the criterion and the un-exclude path (add a shim, delete the line). - Upgrade the package set 77.6.0 -> 77.10.2; spago.{yaml,lock} regenerated by running the documented steps for real: 492 of 640 packages installed. The 77.10.2 cannot-run closure is identical to 77.6.0's; both packages new to the set (harmonia, yoga-format) are clean. - Verified at production RTS settings (-M3G): cold boot compiles all 3,798 modules (peak residency 1,165 MB); warm restart peaks at 1,907 MB, ~160 MB below the previous trim, leaving ~1.1 GB headroom. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addressed here: ff61412
Addressed all of these in this commit: bd91097 Note that I just added a staging/excluded-packages.txt containing the list of packages, but no script for reproducing that list. I'm open to including a script, but I'm also hoping that this is a stopgap solution, and the memory management can be improved such that this exclusion list is no longer necessary. But, maybe we should plan on it being more permanent?
|
Sorry, something went wrong.
because spago needs node:sqlite (added in v22)
Looked into this a little bit, and I think it's essentially this issue: purescript/purescript#3842. Calling loadModules → readFileDeserialise creates a bunch of copies of source span names that are duplicated at each node of the externs.cbor. There are other things that seem to be duplicated when deserializing, haven't looked into how much each amounts to. This proposal seems related too: purescript/purescript#4352. Fable came up with a workaround downstream in this repo, that reduced the max live bytes substantially, but it's kinda gnarly 😓 I'll see if there's a maybe a more minimal solution that gets substantial savings, but for now just leaving this comment as a reminder to self. Also maybe relevant issues: |
Sorry, something went wrong.
Description of the change
Resolves #315
This migrates the client and staging Spago projects to use latest (1.0.4), and updates compiler version and package set to latest while I was at it. Note that 0.15.15 is latest version available on Hackage at time of writing.
For client updates, I just followed these instructions in Spago repo: https://github.com/purescript/spago#migrate-from-spagodhall-to-spagoyaml.
For staging/Haskell updates I followed the instructions in RELEASE.md that I also updated in these changes. I did not change stack.yaml resolver because 0.15.15 is still on lts-20.9.
This is a large diff, but most of that is in generated files. Ignoring staging/* (effectively generated from following the RELEASE.md instructions) and client/spago.lock, the diff is much more reasonable:
That said, I am open splitting this up into separate MRs.
Checklist: