A high-performance web crawler and scanner — recursively crawls websites, respects robots.txt, handles rate-limiting, and produces comprehensive JSON or text reports. Available as a single, self-contained binary via npm.

DeepScanBot is a feature-rich, concurrent web crawler that makes website crawling fast, reliable, and developer-friendly. It handles the complexities of web crawling automatically:

• Discovers all pages on a website by following internal links up to a configurable depth
• Respects crawl rules defined in robots.txt with an option to bypass when needed
• Handles rate limits with automatic retry and exponential backoff
• Filters content by MIME type for targeted crawling
• Exports results in structured JSON or readable text formats with detailed analytics

• Security Researchers - Audit websites for exposed endpoints
• SEO Specialists - Discover all pages, find broken links, analyze site structure
• QA Engineers - Verify page availability, test website coverage
• Data Analysts - Collect structured data from websites for analysis
• DevOps Engineers - Monitor website health, validate deployments

Here's what DeepScanBot can do for you:

• Concurrent Crawling: Multi-threaded architecture with configurable concurrency, per-host rate limiting, and CPU-aware auto-scaling
• Robots.txt Compliance: Automatically respects robots.txt rules with optional bypass
• Retry & Rate-Limit Handling: Automatic retry with exponential backoff and Retry-After header parsing
• Rich Output: JSON or text reports with detailed summaries, status code distribution, and skip reason breakdowns
• Sitemap Discovery: Auto-discover and crawl URLs from sitemap.xml, including nested sitemap indexes
• Content-Type Filtering: Filter downloads by MIME type, enforce page size limits
• Proxy Support: Route traffic through HTTP/HTTPS proxy servers
• Resume Mode: Resume interrupted crawls without recrawling already visited URLs
• Cross-Domain Crawling: Optionally follow links to external domains
• TLS Options: Disable TLS verification for self-signed certificates

DeepScanBot ships as a single, self-contained binary with zero runtime dependencies. Choose the method that fits your workflow.

# One-time tap setup (uses dedicated branch in the same repository)
brew tap mindfiredigital/DeepScanBot homebrew-tap

# Install
brew install deepscanbot

# Upgrade later
brew upgrade deepscanbot

Note: The Homebrew tap is automatically maintained via a dedicated homebrew-tap branch in this repository. No separate repository is needed. After a new release, run brew update before installing to ensure Homebrew recognizes the updated cask.

npm install -g @mindfiredigital/deepscanbot

The npm package automatically detects your OS and architecture. No additional runtime dependencies are required.

deepscanbot --help

curl -fsSL https://raw.githubusercontent.com/mindfiredigital/DeepScanBot/main/scripts/install.sh | bash

Install a specific version:

curl -fsSL https://raw.githubusercontent.com/mindfiredigital/DeepScanBot/main/scripts/install.sh | bash -s -- -b /usr/local/bin -v v1.0.0

What the script does:

1. Detects your OS (linux / darwin) and architecture (amd64 / arm64)
2. Fetches the latest release version from the GitHub API
3. Downloads the binary and checksums.txt from the release
4. Verifies the binary against its SHA256 checksum
5. Installs to /usr/local/bin (or custom path via -b)
6. Runs deepscanbot --version to confirm the installation

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/mindfiredigital/DeepScanBot/main/scripts/install.ps1'))

Install a specific version:

.\install.ps1 -InstallDir "C:\tools" -Version "v1.0.0"

The script installs to $env:ProgramFiles\DeepScanBot by default and offers to add it to your PATH.

Requires Go 1.22+:

go install github.com/mindfiredigital/DeepScanBot/apps/cli@latest

The binary is placed in $GOPATH/bin (typically ~/go/bin).

Download the binary for your platform from the latest release:

Linux / macOS:

chmod +x deepscanbot_*
sudo mv deepscanbot_linux_amd64 /usr/local/bin/deepscanbot

Windows (PowerShell as Administrator):

Move-Item .\deepscanbot_windows_amd64.exe C:\Windows\System32\deepscanbot.exe

Run these three commands to confirm DeepScanBot is installed correctly:

# 1. Check the version
deepscanbot --version

# 2. Run the diagnostic checks
deepscanbot doctor

# 3. Verify the help output
deepscanbot --help

Expected output for deepscanbot --version:

DeepScanBot CLI v1.0.0

Checksums: Every release includes a checksums.txt file with SHA256 hashes of all binaries.

# Linux / macOS (sha256sum)
sha256sum -c checksums.txt --ignore-missing

# macOS (alternative with shasum)
shasum -a 256 -c checksums.txt --ignore-missing

# Windows PowerShell
Get-FileHash .\deepscanbot_windows_amd64.exe -Algorithm SHA256

Attestation: GitHub signed attestations are generated for every release. Verify provenance with the GitHub CLI:

gh attestation verify deepscanbot_darwin_amd64 --owner mindfiredigital

The Homebrew tap is automatically managed using a dedicated branch in this repository. No separate repository is required.

1. Source files live in the homebrew-tap/ directory on the main branch
2. GoReleaser generates the cask file during releases: homebrew-tap/Casks/deepscanbot.rb
3. GitHub Actions automatically syncs the cask to the homebrew-tap branch
4. Homebrew taps the homebrew-tap branch for installations

DeepScanBot/
├── main branch
│   ├── homebrew-tap/
│   │   ├── Casks/              # Source cask files (generated by GoReleaser)
│   │   │   └── deepscanbot.rb
│   │   ├── README.md           # Tap documentation
│   │   └── .gitignore          # Ignores generated files
│   ├── .goreleaser.yml         # Configures cask generation
│   └── .github/workflows/
│       └── homebrew-sync.yml   # Auto-syncs to homebrew-tap branch
│
└── homebrew-tap branch (dedicated for Homebrew)
    ├── Casks/
    │   └── deepscanbot.rb      # Only cask files (auto-synced)
    └── README.md

The .github/workflows/homebrew-sync.yml workflow automatically:

• Triggers on every GitHub release
• Extracts the generated cask from homebrew-tap/Casks/deepscanbot.rb
• Commits and pushes to the homebrew-tap branch
• Makes the cask available via: brew tap mindfiredigital/DeepScanBot homebrew-tap

To test the Homebrew tap locally:

# Clone the repository
git clone https://github.com/mindfiredigital/DeepScanBot.git
cd DeepScanBot

# Checkout the homebrew-tap branch
git checkout homebrew-tap

# View the cask file
cat Casks/deepscanbot.rb

# Test the tap (from another directory)
brew tap mindfiredigital/DeepScanBot homebrew-tap
brew install deepscanbot
brew test deepscanbot

Cask not found after release:

1. Verify the release completed successfully
2. Check the homebrew-tap branch was updated: git ls-remote origin homebrew-tap
3. Wait a few minutes for GitHub to propagate the branch
4. Run brew update and try again

Installation fails:

1. Check the cask file syntax: brew audit --cask deepscanbot
2. Verify the download URL is accessible
3. Check SHA256 checksums match the release artifacts

# Crawl a website with default settings
deepscanbot scan https://example.com

# Crawl with custom depth and concurrency
deepscanbot scan https://example.com depth=3 concurrency=10

# Output as JSON
deepscanbot scan https://example.com --json

> **Note:** The `--json` global flag enables JSON-formatted output for commands that produce structured output (`scan`, `version`, `doctor`, and `help`).

# Use a proxy
deepscanbot scan https://example.com proxy=http://127.0.0.1:8080

# Enable sitemap discovery
deepscanbot scan https://example.com sitemap=true

# Resume a previous crawl
deepscanbot scan https://example.com resume=true output=crawler_results

# Show version
deepscanbot --version

## CLI Commands

DeepScanBot provides several commands to help you work with the tool effectively.

### scan

Crawl and analyze a website. This is the main command for web crawling.

```bash
# Basic scan
deepscanbot scan https://example.com

# Scan with options
deepscanbot scan https://example.com depth=3 concurrency=10

# Scan with JSON output
deepscanbot scan https://example.com --json

Show the installed version of DeepScanBot.

# Show version
deepscanbot version
deepscanbot --version

# Show version in JSON format
deepscanbot version --json

Verify your DeepScanBot installation and environment. This command checks that the binary is properly installed and the environment is configured correctly.

# Run diagnostics
deepscanbot doctor

# Run diagnostics with JSON output
deepscanbot doctor --json

Manage CLI configuration settings. Currently, all settings must be specified via command-line flags.

# View configuration help
deepscanbot config

Generate shell completion scripts for Bash, Zsh, Fish, or PowerShell. This enables tab completion for DeepScanBot commands.

# Generate bash completion
deepscanbot completion bash

# Generate zsh completion
deepscanbot completion zsh

# Generate fish completion
deepscanbot completion fish

# Generate PowerShell completion
deepscanbot completion powershell

Setup instructions:

# Bash - Add to ~/.bashrc or ~/.bash_profile
mkdir -p ~/.local/share/bash-completion/completions
deepscanbot completion bash > ~/.local/share/bash-completion/completions/deepscanbot

# Zsh - Add to ~/.zshrc
deepscanbot completion zsh > "${fpath[1]}/_deepscanbot"

# Fish - Add to ~/.config/fish/completions/
deepscanbot completion fish > ~/.config/fish/completions/deepscanbot.fish

Show help for any command.

# Show general help
deepscanbot help
deepscanbot --help

# Show help for a specific command
deepscanbot help scan
deepscanbot scan --help

deepscanbot doctor

# Generate shell completion
deepscanbot completion bash

DeepScanBot supports a consistent --json flag across all commands that return structured data. When enabled, all output is written as valid JSON to stdout, while progress messages and diagnostics are sent to stderr.

DeepScanBot is designed to run reliably in CI/CD pipelines, scripts, and AI agents without ever hanging on user input.

Add --no-input to any command to disable all interactive prompts. If required input is missing, the CLI fails immediately with a clear error message instead of waiting.

# Run scan in non-interactive mode
deepscanbot scan https://example.com --no-input --force

# Check version non-interactively
deepscanbot --no-input --version --json

# Run doctor in CI
deepscanbot --no-input doctor

The --force flag allows overwriting existing output files without prompting. In non-interactive mode, the scan command refuses to overwrite an existing output file unless --force is explicitly passed.

# Safe for CI/CD — overwrites output if it exists
deepscanbot scan https://example.com --no-input --force

# Will fail in non-interactive mode without --force
deepscanbot scan https://example.com --no-input
# Error: Output file "crawler_results.txt" already exists.
# Hint: Pass --force to overwrite or use output=<filename>.

When stdin is not connected to a terminal (e.g., piped input, CI runners), the CLI automatically detects the non-TTY environment and never waits for user input. The --no-input flag provides an explicit override for cases where TTY detection is insufficient.

# Always use --no-input and --force for automated runs
deepscanbot scan https://example.com --no-input --force --json

# Use exit codes to check success
if deepscanbot scan https://example.com --no-input --force depth=0; then
  echo "Scan completed"
else
  echo "Scan failed with exit code $?"
fi

DeepScanBot uses standardized exit codes to make CLI failures predictable for scripts, CI/CD pipelines, and AI agents. Every command returns a consistent exit code that tells you exactly what happened.

# Check exit code in a script
deepscanbot scan https://example.com
exit_code=$?
echo "Exit code: $exit_code"

# Conditional execution based on exit code
if deepscanbot scan https://example.com depth=0; then
  echo "Scan succeeded"
else
  echo "Scan failed with exit code $?"
fi

All errors include:

• What went wrong — a clear description of the problem
• Why it happened — the root cause when possible
• How to fix it — an actionable hint with an example

$ deepscanbot scan ftp://example.com
Error: Invalid URL: "ftp://example.com" must be an absolute http:// or https:// URL.
Hint: Example: https://example.com

$ deepscanbot scan http://example.com output=
Error: Output filename must not be empty.
Hint: Use output=<filename> with a non-empty value.

# Scan with JSON output
deepscanbot scan https://example.com --json

# Version with JSON output
deepscanbot --version --json

# Doctor with JSON output
deepscanbot doctor --json

$ deepscanbot scan https://example.com depth=0 --json
{
  "status": "success",
  "data": {
    "start_url": "https://example.com",
    "output_file": "crawler_results.json",
    "started_at": "2024-01-15T10:30:00Z",
    "finished_at": "2024-01-15T10:30:05Z",
    "duration_ms": 5000,
    "summary": {
      "total": 1,
      "passed": 1,
      "failed": 0,
      "skipped": 0,
      "discovered": 0,
      "max_depth": 0
    },
    "urls": [
      {
        "url": "https://example.com",
        "depth": 0,
        "status_code": 200,
        "content_type": "text/html",
        "result": "passed"
      }
    ],
    "skipped": null
  },
  "meta": {
    "timestamp": "2024-01-15T10:30:05Z",
    "command": "scan",
    "duration_ms": 5000
  }
}

$ deepscanbot --version --json
{
  "status": "success",
  "data": {
    "version": "1.0.0",
    "name": "DeepScanBot CLI"
  },
  "meta": {
    "timestamp": "2024-01-15T10:30:00Z",
    "command": "version",
    "duration_ms": 0
  }
}

$ deepscanbot doctor --json
{
  "status": "success",
  "data": {
    "installed": true,
    "executable": true,
    "configured": true,
    "checks_passed": 3,
    "message": "All checks passed!"
  },
  "meta": {
    "timestamp": "2024-01-15T10:30:00Z",
    "command": "doctor",
    "duration_ms": 0
  }
}

$ deepscanbot scan invalid-url --json
{
  "status": "error",
  "error": {
    "message": "invalid URL \"invalid-url\": must be an absolute http:// or https:// URL",
    "code": "invalid_url"
  },
  "meta": {
    "timestamp": "2024-01-15T10:30:00Z",
    "command": "scan",
    "duration_ms": 0
  }
}

• Valid JSON Only: stdout contains only valid JSON when --json is enabled
• Separate Streams: Progress messages, warnings, and logs are written to stderr
• Consistent Format: All commands use the same JSON response structure
• Backward Compatible: Existing behavior is preserved when --json is not specified
• Extensible: The centralized output formatting makes it easy to add new output formats in the future

All JSON responses follow this structure:

{
  "status": "success | error",
  "data": { ... },        // Present on success
  "error": {              // Present on error
    "message": "Error description",
    "code": "error_code"
  },
  "meta": {
    "timestamp": "ISO8601 timestamp",
    "command": "command_name",
    "duration_ms": 1234
  }
}

The JSON output mode is designed for easy integration with scripts and tools:

# Parse with jq
deepscanbot scan https://example.com --json | jq '.data.summary'

# Extract specific fields
deepscanbot --version --json | jq -r '.data.version'

# Check command success
if deepscanbot scan https://example.com --json | jq -e '.status == "success"'; then
  echo "Scan completed successfully"
fi

DeepScanBot uses a modern command-based CLI structure similar to git, docker, and kubectl.

deepscanbot <command> [options]

deepscanbot scan <url> [key=value ...]

Options are specified as key=value pairs after the URL.

| ignore-robots | Ignore robots.txt restrictions | false | | cross-domain | Follow links to other hosts | false | | retries | Number of retry attempts | 0 | | retry-backoff | Base retry backoff duration (e.g. 500ms, 2s) | 1s | | delay | Politeness delay between requests to same host | 0 | | sitemap | Discover URLs from /sitemap.xml | false | | resume | Load existing output and avoid recrawling | false |

Note on Output Files: The output filename is specified without an extension. DeepScanBot automatically appends the appropriate extension based on the output format:

• Text format (default): crawler_results.txt
• JSON format (--json): crawler_results.json Note on Timeout: The default timeout is 2 seconds, which balances speed and reliability for most websites. Increase this value (for example, timeout=10 or timeout=30) when crawling slow websites or large pages that take longer to respond. For very slow sites, consider timeout=60 or higher.

deepscanbot scan https://example.com

deepscanbot scan https://docs.example.com \
  depth=5 \
  concurrency=20 \
  host-concurrency=5 \
  timeout=10 \
  delay=200ms \
  retries=3 \
  retry-backoff=1s \
  --json \
  sitemap=true \
  cross-domain=true \
  content-types="text/html application/pdf" \
  output=scan_results

The release process is fully automated via GitHub Actions.

Git Tag (v1.0.0)
      ↓
GitHub Action (release.yml)
      ↓
GoReleaser Build
      ↓
dist/ (binaries + checksums)
      ↓
Copy to npm package
      ↓
Update package.json version
      ↓
npm publish
      ↓
npm install -g @mindfiredigital/deepscanbot
      ↓
[Async] Sync homebrew-tap branch (may take a few minutes)
      ↓
brew update && brew install deepscanbot

Note: Homebrew tap synchronization happens asynchronously after the release. If the cask is not immediately available, wait a few minutes for the homebrew-tap branch to update, then run brew update before installing.

1. Create a Git tag:

git tag -a v1.0.0 -m "Release v1.0.0"
git push origin v1.0.0

2. The GitHub Action will automatically:

   • Build binaries for all platforms via GoReleaser
   • Generate SHA256 checksums
   • Copy binaries into the npm package
   • Update package.json version from the Git tag
   • Publish to the npm registry
   • Create a GitHub Release with artifacts

3. Users can then install:

npm install -g @mindfiredigital/deepscanbot

This project follows Semantic Versioning:

• MAJOR (1.x.x): Breaking changes
• MINOR (x.1.x): New features, backward compatible
• PATCH (x.x.1): Bug fixes, backward compatible

Pre-release versions use the -next tag on npm:

npm install -g @mindfiredigital/deepscanbot@next

The npm publish configuration comes from environment variables:

# npmjs (default)
npm publish

# GitHub Packages
NPM_REGISTRY=https://npm.pkg.github.com/ npm publish

# Verdaccio (local)
NPM_REGISTRY=http://localhost:4873/ npm publish

# Dry run (verify without publishing)
npm publish --dry-run

For testing purposes, you can create snapshot releases:

# Create a snapshot build
goreleaser release --snapshot --clean

# Copy binaries and update version
bash scripts/copy-to-npm.sh
VERSION=0.0.0-snapshot node scripts/sync-version.js

# Dry run publish
npm publish --dry-run

project/
│
├── apps/
│   └── cli/              # Go CLI application
│       ├── main.go
│       └── tests/
│
├── packages/
│   ├── crawler/          # Web crawling logic
│   ├── exitcode/         # Standardized exit codes and error handling
│   ├── noinput/          # Non-interactive mode and TTY detection
│   ├── fetcher/          # HTTP fetching
│   ├── logger/           # Logging utilities
│   ├── output/           # Output formatting (JSON, human-readable, command tree)
│   ├── parser/           # HTML parsing
│   ├── storage/          # Output storage
│   └── types/            # Shared types
│
├── dist/                 # Pre-built binaries (generated)
├── bin/                  # Installed binary (generated by postinstall)
├── scripts/              # Helper scripts
│   └── prepublish-check.js
│
├── .github/
│   └── workflows/
│       ├── ci.yml             # CI workflow
│       └── release.yml        # Release workflow
│       └── release-docs.yml   # Release docs workflow
│
├── .goreleaser.yml       # GoReleaser configuration
├── package.json          # npm package configuration
├── postinstall.js        # npm post-install script
└── README.md             # This file

Problem: npm install -g @mindfiredigital/deepscanbot fails

Solutions:

• Ensure you have Node.js 18+ installed: node --version
• Check npm permissions: npm config get prefix
• Try with sudo (Unix): sudo npm install -g @mindfiredigital/deepscanbot
• Clear npm cache: npm cache clean --force

Problem: "Unsupported platform" error during installation

Solutions:

• Verify your OS and architecture: node -e "console.log(process.platform, process.arch)"
• Supported platforms: macOS (x64, arm64), Linux (x64, arm64), Windows (x64)
• Ensure you're using a 64-bit version of Node.js

Problem: "Command not found" after installation

Solutions:

• Check npm global bin directory is in your PATH: npm bin -g
• Add to PATH: export PATH=$(npm bin -g):$PATH
• On Windows, restart your terminal after installation

Problem: No results or empty output

Solutions:

• Verify the URL is accessible: curl -I https://example.com
• Increase timeout: deepscanbot scan <url> timeout=10
• Disable TLS verification for testing: deepscanbot scan <url> insecure=true
• Check if robots.txt is blocking: deepscanbot scan <url> ignore-robots=true

Problem: Too many requests or being blocked

Solutions:

• Reduce concurrency: deepscanbot scan <url> concurrency=2
• Add crawl delay: deepscanbot scan <url> delay=1s
• Use a proxy: deepscanbot scan <url> proxy=http://proxy.example.com:8080

Runs on every push and pull request to main:

• Builds Go code
• Runs tests with race detection
• Runs golangci-lint
• Checks Go formatting
• Verifies GoReleaser configuration
• Cross-platform build check
• npm package validation

Triggered by pushing a Git tag matching v*:

• Builds binaries via GoReleaser
• Copies binaries to npm package
• Updates package.json version
• Generates SHA256 checksums
• Verifies binaries
• Publishes to npm registry
• Uploads release artifacts
• Syncs Homebrew tap to homebrew-tap branch

Triggered after every GitHub release:

• Extracts generated cask from homebrew-tap/Casks/deepscanbot.rb
• Commits cask to dedicated homebrew-tap branch
• Pushes branch to make it available via Homebrew

Please see CONTRIBUTING.md for contribution guidelines.

This project is licensed under the MIT License - see LICENSE.md for details.