| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
| Name | Name | Last commit date | ||
|---|---|---|---|---|
一个高效、可靠的 MySQL 数据库变更捕获(CDC)工具
实时监控数据库变更,支持高并发 webhook 分发
pikachu 是一个基于 Go 语言开发的高效 MySQL 数据库变更捕获(CDC)工具。它通过解析 MySQL 的 binlog 日志来实时捕获数据库表的变更事件(插入、更新、删除),并将这些变更通过 webhook 的方式发送到指定的回调地址。
| 指标 | 数值 | 说明 |
|---|---|---|
| 事件处理延迟 | < 10ms | P99 延迟,从 binlog 到 webhook 发送 |
| 吞吐量 | 10,000+ events/sec | 单实例处理能力 |
| Webhook 成功率 | > 99.9% | 包含重试机制的整体成功率 |
| 内存占用 | < 100MB | 基础运行内存(不含事件队列) |
| CPU 使用率 | < 5% | 正常负载下的 CPU 占用 |
| 并发处理 | 50+ workers | 可配置的 webhook 并发数 |
graph TB
A[MySQL Database] -->|Binlog Events| B[Monitor Component]
B --> C[Event Queue<br/>Size: 10,000]
C --> D[Dispatcher Worker Pool<br/>Workers: 20-50]
D --> E[Webhook Callbacks<br/>Retry Logic]
B --> F[Schema Cache<br/>Table Metadata]
D --> G[URL Pre-builder<br/>Performance Opt]
subgraph "pikachu Core Components"
B
C
D
F
G
end
subgraph "Monitoring & Health"
H[HTTP Server<br/>Port: 8080]
I[Metrics Collector<br/>Performance Data]
J[Health Checks<br/>System Status]
end
subgraph "External Services"
K[Config Files<br/>config.yaml<br/>tasks.yaml]
L[Target APIs<br/>Webhook URLs]
M[Monitoring Systems<br/>Prometheus etc.]
end
K --> B
K --> C
K --> D
B --> H
B --> I
D --> J
E --> L
I --> M
H --> M
# 克隆仓库
git clone https://github.com/tiyee/pikachu.git
cd pikachu
# 编译应用
go build -o pikachu .
# 或使用 Makefile(推荐)
make build# 下载对应平台的二进制文件
wget https://github.com/tiyee/pikachu/releases/latest/download/pikachu-linux-amd64.tar.gz
# 解压
tar -xzf pikachu-linux-amd64.tar.gz
# 赋予执行权限
chmod +x pikachu# 拉取镜像
docker pull pikachu:latest
# 或使用 Docker Compose(推荐)
docker-compose up -d# 数据库配置
database:
host: "localhost"
port: 3306
user: "root"
password: "password"
database: "test_db"
server_id: 100 # 唯一标识,避免与主从复制冲突
charset: "utf8mb4" # 可选,默认 utf8mb4
# 日志配置
log:
level: "info" # debug, info, warn, error, fatal, panic
format: "text" # text, json
# HTTP 服务器配置
server:
enabled: true # 是否启用健康检查服务器
port: 8080 # 服务器端口
path: "/health" # 健康检查路径
# 分发器配置 (性能优化)
dispatcher:
worker_count: 20 # 工作协程数量 (推荐: CPU核心数 * 2)
queue_size: 1000 # 队列大小 (支持突发流量)
timeout: 30s # HTTP请求超时
max_retries: 3 # 最大重试次数
retry_base_delay: 5s # 重试基础延迟 (最小3s)
max_connections: 100 # 最大连接数
# 监控器配置
monitor:
event_queue_size: 10000 # 事件队列大小 (高负载优化)
event_queue_timeout: 2s # 事件队列超时时间 (快速响应)
# 可选:回调主机地址(用于相对路径的回调URL)
callback_host: "http://localhost:3000"tasks:
# 基础示例:监控用户表所有变更
- task_id: "user_monitor"
name: "用户表变更监控"
table_name: "users"
events: ["insert", "update", "delete"]
callback_url: "/webhook/user" # 相对路径
# 高级示例:只监控订单表的插入和更新
- task_id: "order_monitor"
name: "订单表变更监控"
table_name: "orders"
events: ["insert", "update"] # 不监控删除事件
callback_url: "https://api.example.com/webhook/order" # 绝对路径
# 特殊表名示例:MySQL关键字表名
- task_id: "keyword_table_monitor"
name: "关键字表名监控"
table_name: "order" # 'order' 是MySQL关键字,系统自动处理
events: ["insert", "update", "delete"]
callback_url: "/webhook/order"
# 复杂表名示例:特殊字符和数字开头
- task_id: "complex_table_monitor"
name: "复杂表名监控"
table_name: "2024_user-activity_log" # 包含连字符和数字开头
events: ["insert"]
callback_url: "/webhook/activity"
# 生产环境示例:外部API回调
- task_id: "production_sync"
name: "生产环境数据同步"
table_name: "sync_data"
events: ["update"]
callback_url: "https://external-api.company.com/v1/sync"开发环境 (config.dev.yaml):
log:
level: "debug"
format: "text"
dispatcher:
worker_count: 2
queue_size: 50
monitor:
event_queue_size: 100生产环境 (config.prod.yaml):
log:
level: "warn"
format: "json"
dispatcher:
worker_count: 10
queue_size: 500
timeout: 60s
max_retries: 5
monitor:
event_queue_size: 2000# 使用默认配置
./pikachu -config config.yaml
# 使用测试环境配置
./pikachu -config config.test.yaml
# 使用生产环境配置
./pikachu -config config.prod.yaml确保 config.yaml 和 tasks.yaml 文件已正确配置。
docker-compose up -d| 字段 | 类型 | 必填 | 说明 |
|---|---|---|---|
| host | string | 是 | MySQL 主机地址 |
| port | int | 是 | MySQL 端口 |
| user | string | 是 | MySQL 用户名 |
| password | string | 是 | MySQL 密码 |
| database | string | 是 | 数据库名称 |
| server_id | uint32 | 是 | 用于 binlog 同步的唯一 server ID |
| charset | string | 否 | 字符集,默认为 utf8mb4 |
| 字段 | 类型 | 必填 | 说明 |
|---|---|---|---|
| level | string | 否 | 日志级别:debug, info, warn, error, fatal, panic (默认: info) |
| format | string | 否 | 日志格式:text, json (默认: text) |
| 字段 | 类型 | 必填 | 说明 |
|---|---|---|---|
| enabled | bool | 否 | 是否启用健康检查服务器 (默认: false) |
| port | int | 否 | 服务器端口 (默认: 8080) |
| path | string | 否 | 健康检查路径 (默认: /health) |
| 字段 | 类型 | 必填 | 说明 |
|---|---|---|---|
| worker_count | int | 否 | 工作协程数量 (默认: 5) |
| queue_size | int | 否 | 队列大小 (默认: 100) |
| timeout | duration | 否 | HTTP请求超时时间 (默认: 30s) |
| max_retries | int | 否 | 最大重试次数 (默认: 3) |
| retry_base_delay | duration | 否 | 重试基础延迟 (默认: 10s,最小: 3s*) |
*注意: 如果设置了 max_retries > 0,则 retry_base_delay 不能小于 3 秒,以避免对目标服务造成过大压力。
| 字段 | 类型 | 必填 | 说明 |
|---|---|---|---|
| event_queue_size | int | 否 | 事件队列大小 (默认: 1000) |
| event_queue_timeout | duration | 否 | 事件队列超时时间 (默认: 5s) |
| 字段 | 类型 | 必填 | 说明 |
|---|---|---|---|
| task_id | string | 是 | 任务唯一标识 |
| name | string | 是 | 任务名称 |
| table_name | string | 是 | 要监控的表名(支持MySQL关键字) |
| events | []string | 是 | 要监控的事件类型 (insert/update/delete) |
| callback_url | string | 是 | webhook 回调地址(支持相对路径和绝对路径) |
| 字段 | 类型 | 必填 | 说明 |
|---|---|---|---|
| callback_host | string | 否 | 回调主机地址,用于拼接相对路径的回调URL |
Pikachu 支持配置文件分离,便于多环境部署:
主配置文件:
任务配置文件:
生产环境特点:
测试环境特点:
MySQL 用户需要以下权限:
确保 MySQL 服务器已正确配置:
发送到回调地址的数据格式如下:
{
"primary_id": 1,
"event": "insert",
"table": "users",
"data": {
"id": 1,
"name": "John Doe",
"email": "john@example.com"
},
"timestamp": "2023-01-01T12:00:00Z"
}根据不同事件类型,数据格式略有不同:
Pikachu 提供了完整的 HTTP 监控端点:
端点: GET http://<host>:<port>/health
响应示例:
{
"status": "UP",
"monitor_running": true,
"dispatcher_running": true,
"event_queue_size": 0,
"last_event_time": "2023-05-15T10:30:45Z",
"uptime": "2h45m30s",
"version": "v1.0.0"
}状态说明:
端点: GET http://<host>:<port>/metrics
响应示例:
{
"system": {
"goroutines": 15,
"memory_alloc": "2.5MB",
"memory_total": "15.2MB",
"gc_cycles": 42
},
"monitor": {
"status": "running",
"tables_monitored": 5,
"total_events_processed": 10250,
"events_per_second": 12.5,
"last_event_time": "2023-05-15T10:30:45Z",
"binlog_position": {
"file": "mysql-bin.000123",
"position": 456789
}
},
"dispatcher": {
"status": "running",
"workers_active": 3,
"workers_total": 5,
"queue_size": 0,
"queue_capacity": 100,
"webhooks_sent": 10245,
"webhooks_failed": 5,
"success_rate": 99.95,
"avg_response_time": "125ms"
},
"tasks": [
{
"task_id": "user_monitor",
"table_name": "users",
"events_processed": 5230,
"last_processed": "2023-05-15T10:30:42Z",
"status": "active"
},
{
"task_id": "order_monitor",
"table_name": "orders",
"events_processed": 5020,
"last_processed": "2023-05-15T10:30:45Z",
"status": "active"
}
]
}| 状态码 | 说明 |
|---|---|
| 200 | 请求成功 |
| 400 | 请求参数错误 |
| 404 | 端点不存在 |
| 500 | 服务器内部错误 |
| 503 | 服务不可用 |
程序使用结构化日志记录关键操作和错误信息:
pikachu 自动处理各种特殊表名,包括:
- task_id: "order_monitor"
table_name: "order" # 'order' 是MySQL关键字,系统自动处理- task_id: "special_table_monitor"
table_name: "my-table" # 包含连字符,系统自动处理- task_id: "numeric_table_monitor"
table_name: "2024_orders" # 以数字开头,系统自动处理系统会自动为所有表名添加反引号,确保SQL语句的正确性,无需用户手动处理。
pikachu 采用多阶段构建策略:
方式一:Docker Compose(推荐)
# 1. 克隆项目
git clone https://github.com/tiyee/pikachu.git
cd pikachu
# 2. 配置环境变量
cp config-example.yaml config.yaml
cp tasks-example.yaml tasks.yaml
# 3. 编辑配置文件
vim config.yaml # 配置数据库连接等信息
vim tasks.yaml # 配置监控任务
# 4. 启动服务
docker-compose up -d
# 5. 查看日志
docker-compose logs -f pikachu方式二:单独使用 Docker
# 1. 构建镜像
docker build -t pikachu:latest .
# 2. 创建数据卷
docker volume create pikachu-logs
docker volume create pikachu-config
# 3. 运行容器
docker run -d \
--name pikachu \
-p 8080:8080 \
-v $(pwd)/config.yaml:/app/config.yaml:ro \
-v $(pwd)/tasks.yaml:/app/tasks.yaml:ro \
-v pikachu-logs:/app/logs \
pikachu:latestDocker Compose 生产配置:
version: '3.8'
services:
pikachu:
image: pikachu:latest
container_name: pikachu-prod
restart: unless-stopped
# 环境变量
environment:
- TZ=Asia/Shanghai
# 端口映射
ports:
- "8080:8080"
# 卷挂载
volumes:
- ./config.prod.yaml:/app/config.yaml:ro
- ./tasks.yaml:/app/tasks.yaml:ro
- /data/logs/pikachu:/app/logs
- /etc/localtime:/etc/localtime:ro
# 资源限制
deploy:
resources:
limits:
memory: 512M
cpus: '0.5'
reservations:
memory: 128M
cpus: '0.1'
# 健康检查
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8080/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
# 网络配置
networks:
- pikachu-network
# 日志配置
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
networks:
pikachu-network:
driver: bridgeDeployment 配置:
apiVersion: apps/v1
kind: Deployment
metadata:
name: pikachu
namespace: monitoring
labels:
app: pikachu
spec:
replicas: 2
selector:
matchLabels:
app: pikachu
template:
metadata:
labels:
app: pikachu
spec:
containers:
- name: pikachu
image: pikachu:latest
ports:
- containerPort: 8080
name: http
env:
- name: TZ
value: "Asia/Shanghai"
volumeMounts:
- name: config
mountPath: /app/config.yaml
subPath: config.yaml
readOnly: true
- name: config
mountPath: /app/tasks.yaml
subPath: tasks.yaml
readOnly: true
- name: logs
mountPath: /app/logs
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
volumes:
- name: config
configMap:
name: pikachu-config
- name: logs
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: pikachu-service
namespace: monitoring
spec:
selector:
app: pikachu
ports:
- protocol: TCP
port: 80
targetPort: 8080
name: http
type: ClusterIP开发环境:
log:
level: "debug"
format: "text"
dispatcher:
worker_count: 2
queue_size: 50
timeout: 10s
monitor:
event_queue_size: 100生产环境:
log:
level: "warn"
format: "json"
dispatcher:
worker_count: 10-20 # 根据 CPU 核心数调整
queue_size: 500-1000 # 根据内存容量调整
timeout: 60s
max_retries: 5
monitor:
event_queue_size: 2000-5000# 高并发场景配置
dispatcher:
worker_count: 50 # 更多工作协程
queue_size: 2000 # 更大的队列
timeout: 120s # 更长的超时时间
max_retries: 10 # 更多重试次数
retry_base_delay: 30s # 更长的重试间隔
monitor:
event_queue_size: 10000 # 更大的事件队列关键指标:
cp config.yaml config.yaml.backup提取任务配置 从现有的 config.yaml 中复制 tasks 部分到新的 tasks.yaml 文件
更新主配置文件 从 config.yaml 中移除 tasks 部分
验证配置
./pikachu -config config.yaml新版本保持向后兼容,如果 tasks.yaml 不存在,系统会尝试从主配置文件中加载任务配置。
go test ./...go test -bench=. ./...go build -ldflags="-s -w" -o pikachu .# 用户服务 -> 订单服务 数据同步
tasks:
- task_id: "user_sync_to_order"
name: "用户信息同步到订单服务"
table_name: "users"
events: ["update"] # 只同步用户信息变更
callback_url: "https://order-service.internal/api/user-updates"
- task_id: "profile_sync_to_notification"
name: "用户资料同步到通知服务"
table_name: "user_profiles"
events: ["insert", "update"]
callback_url: "/api/sync/user-profile" # 相对路径,使用 callback_host# 商品表变更 -> Elasticsearch 索引更新
tasks:
- task_id: "product_index_update"
name: "商品搜索引擎索引更新"
table_name: "products"
events: ["insert", "update", "delete"]
callback_url: "https://search-service.internal/index/product"
- task_id: "category_index_update"
name: "分类索引更新"
table_name: "product_categories"
events: ["insert", "update", "delete"]
callback_url: "https://search-service.internal/index/category"# 敏感操作审计日志
tasks:
- task_id: "financial_audit"
name: "财务操作审计"
table_name: "financial_transactions"
events: ["insert", "update", "delete"]
callback_url: "https://audit-service.internal/log/financial"
- task_id: "user_action_audit"
name: "用户操作审计"
table_name: "user_action_logs"
events: ["insert"]
callback_url: "https://audit-service.internal/log/user-actions"# 数据变更 -> Redis 缓存失效
tasks:
- task_id: "cache_invalidation"
name: "缓存失效通知"
table_name: "user_preferences"
events: ["update", "delete"]
callback_url: "https://cache-service.internal/invalidate/user"
- task_id: "product_cache_invalidation"
name: "商品缓存失效"
table_name: "products"
events: ["update", "delete"]
callback_url: "https://cache-service.internal/invalidate/product"# 实时通知 -> WebSocket 服务
tasks:
- task_id: "realtime_notification"
name: "实时数据推送"
table_name: "notifications"
events: ["insert"]
callback_url: "https://websocket-service.internal/push/notification"
- task_id: "order_status_update"
name: "订单状态实时推送"
table_name: "order_status_history"
events: ["insert"]
callback_url: "https://websocket-service.internal/push/order-status"# OLTP -> OLAP 数据同步
tasks:
- task_id: "data_warehouse_sync"
name: "数据仓库同步"
table_name: "sales_transactions"
events: ["insert", "update"]
callback_url: "https://data-warehouse.internal/api/sync/sales"
- task_id: "analytics_sync"
name: "分析数据同步"
table_name: "user_behavior_events"
events: ["insert"]
callback_url: "https://analytics-service.internal/api/events"# 业务流程自动化触发
tasks:
- task_id: "order_workflow"
name: "订单工作流触发"
table_name: "orders"
events: ["insert", "update"]
callback_url: "https://workflow-service.internal/trigger/order-process"
- task_id: "inventory_restock"
name: "库存补货触发"
table_name: "inventory"
events: ["update"]
callback_url: "https://inventory-service.internal/trigger/restock"我们欢迎所有形式的贡献!请遵循以下步骤:
本项目采用 MIT License 开源协议。
⭐ 如果这个项目对您有帮助,请给我们一个 Star!
Made with ❤️ by tiyee
| Back | FazBrowse Home | New Git URL |