crawlee-cloud/crawlee-cloud: Self-hosted, open-source platform for running Apify Actors. Drop-in compatible with the Apify SDK. · GitHub

Self-hosted, open-source platform for running Apify Actors on your own infrastructure.

Dashboard · Quick Start · Documentation · Community · Contributing

A purpose-built operator console — monitor runs in real time, debug webhook deliveries, browse datasets, and triage failures without leaving the page.

Operator dashboard — actor count, runs, success rate, hourly throughput, recent activity feed

Webhooks — fire test events per subscription, inline delivery log with HTTP code + body, last-seen status	Run detail — cost analysis vs Apify, live logs, container runtime, dataset / KV / queue IDs
Settings — live version, scaler state, storage health probes (PG / Redis / S3) with latency, API keys	KV stores — click any record to pretty-print its JSON inline, no new tab needed
Runs — full execution history with status filters, durations, dataset links, per-run cost	Actors — your deployed scrapers with version + last-modified at a glance

Love the Crawlee/Apify ecosystem but want the freedom to run things your way? Crawlee Cloud brings the same great developer experience to your own infrastructure. Keep using the tools you love — just host them wherever you want.

• 🏠 Your infrastructure — Deploy on your own servers, cloud, or anywhere you like
• 🔒 Complete privacy — Your data stays exactly where you want it
• ⚡ SDK compatible — Works seamlessly with the Apify SDK you already know
• 🐳 Container-based — Each Actor runs in an isolated Docker container
• 📊 Beautiful dashboard — Monitor runs, explore datasets, manage everything visually
• 💰 Cost transparency — See what each run actually cost you vs what Apify would have charged, per run and at a glance across the runs list

# Instead of pointing to Apify's servers...
export APIFY_API_BASE_URL=https://api.apify.com/v2

# Point to your own Crawlee Cloud instance
export APIFY_API_BASE_URL=https://your-server.com/v2
export APIFY_TOKEN=your-token

Your existing Actor code works without any modifications:

import { Actor } from 'apify';

await Actor.init();
await Actor.pushData({ title: 'Scraped data' });
await Actor.exit();

• Node.js 20+
• Docker & Docker Compose
• PostgreSQL, Redis, and S3-compatible storage (or use our Docker setup)

git clone https://github.com/crawlee-cloud/crawlee-cloud.git
cd crawlee-cloud
npm install

# Starts PostgreSQL, Redis, and MinIO
npm run docker:dev

cp .env.example .env
# Edit .env with your settings

npm run build
npm run db:migrate
npm run dev

The API server starts at http://localhost:3000.

Deploy your own instance in minutes:

Method	Status	Description
	✅ Supported	Automated full stack — App Platform, managed PG/Redis, Runner Droplet
VPS Deploy Script	✅ Supported	Full stack on any Ubuntu VPS with auto-HTTPS via Caddy
Railway	🚧 Coming soon	One-click PaaS deploy (template scaffolding present, not yet verified)
Render	🚧 Coming soon	One-click PaaS deploy (blueprint present, not yet verified)

See deploy/ for detailed instructions.

┌─────────────────────────────────────────────────────────────────┐
│                        Your Actors                              │
│            (using official Apify SDK, no changes)               │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                     Crawlee Cloud API                           │
│              (Apify-compatible REST endpoints)                  │
└─────────────────────────────────────────────────────────────────┘
        │                    │                    │
        ▼                    ▼                    ▼
   ┌──────────┐         ┌─────────┐         ┌─────────┐
   │PostgreSQL│         │  Redis  │         │ S3/MinIO│
   │ metadata │         │ queues  │         │  blobs  │
   └──────────┘         └─────────┘         └─────────┘

v1.6.0 — disk-pressure protection for the runner fleet plus an Apify-parity fix on dataset reads. A near-full runner disk fails every image pull in seconds, and because work is runner-pull, a disk-full runner would out-claim healthy capacity and fast-fail the whole READY queue while hiding the demand from the scaler; the runner now stops claiming at RUNNER_DISK_CLAIM_MAX_PCT (default 90%) and, when a registry is configured, evicts unused registry-tagged actor images at RUNNER_DISK_EVICT_PCT (default 80%). Infra failures (pull errors, missing images) also get a small retry floor that never counts against the actor's own retry budget. On the API side, GET /v2/datasets/:id/items without a limit now streams the entire dataset like real Apify does, instead of silently truncating to 100 items. See the full changelog.

The 1.1–1.5 line that led here covered: the zombie-run reliability overhaul from a live production incident — Redis-blip-proof dead-runner detection, a zombie-run reaper, OOM kills made visible, failed-run logs archived to KV, prebuilt runner images cutting ~4.5 min off scale-up (v1.1.x); memory-aware placement, fast dead-runner reap, and the claim-time cost-attribution stamps (v1.2.0); ingest hot-path performance and runner-key self-healing (v1.2.1/1.2.2); the run-details cost analysis card (v1.3.0); per-run cost across the runs list, backed by the batch cost endpoint GET /v2/actor-runs/costs (v1.4.0); and safe actor force-deletion, webhook SSRF loopback fixes, and the first enforced test-coverage floors in CI (v1.5.0).

Upgrading from v1.5.0 to v1.6.0 is a drop-in: no schema migration, no required env-var changes — redeploy the API (dataset fix) and runners (disk gate). The new RUNNER_DISK_* thresholds only need tuning if the 90/80 defaults don't fit your disk sizes. From v1.0.x or earlier, walk the CHANGELOG forward — deploy notes are flagged inline at each release.

Crawlee Cloud is young, and feedback from real deployments shapes what gets built next.

• 💬 GitHub Discussions — ask questions, share what you're running, weigh in on the roadmap
• 🚀 Deployed it? Tell us how it went — 5 minutes of honest notes about your deploy experience is the most useful thing an early user can send
• 🤝 Free 15-minute setup help — stuck deploying? Email me and I'll personally help you get running, free, in exchange for your candid feedback
• ☕ Support the project — Open Collective · GitHub Sponsors · Ko-fi

We welcome contributions! Please see our Contributing Guide for details.

# Run tests
npm test

# Type checking
npm run typecheck

# Linting
npm run lint

Crawlee Cloud's infrastructure is generously supported by DigitalOcean through their Open Source Credits Program.

This project is licensed under the MIT License.