🔨 List all IP ranges from: Google (Cloud & GoogleBot), Bing (Bingbot), Amazon (AWS), Microsoft, Oracle (Cloud), GitHub, Facebook (Meta), OpenAI (GPTBot) and other with daily updates.

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

A crawler that crawls search engine! 😎 Usable for collecting site with dorks and wildcards. Also provides output in web interface with more than 3 API endpoints!

Continuously-updated public IP ranges for major cloud providers, SaaS services, and bots — one txt file per provider, refreshed every 4 hours via GitHub Actions. Drop-in for firewalls, allowlists, and geo blocks.

Turn raw server/CDN access logs into verified-crawler analytics: parse many log formats, verify Googlebot/Bingbot by IP (not user-agent), enrich with DuckDB, and analyze crawl budget, crawl waste, and crawl traps with SQL.

A Bash script that fetches and parses JSON-based IP range data for trusted search engine bots (Googlebot, Bingbot, and others), ideal for use with ModSecurity and other web application firewalls and web servers.

Interactive crawler IP intelligence dashboard for search, AI, and user-triggered fetchers.

Sitenize googlebot, yandexbot, bingbot gibi kullanıcılar geldiğinde göstebileceğiniz veya göstermek istemediğiniz şeyleri ayarlayabileceğiniz basit bir php kodudur.

Fetch and inspect any URL as Googlebot, Bingbot or a real browser, directly from your Umbraco site. Built for SEO debugging, diagnosing hacked/cloaked pages and inspecting redirects.