FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

qs3c/E-commerce-cli: CDP-powered e-commerce product research CLI for human-paced browser collection, competitor price comparison & Excel reports. 基于真实 Chrome 的电商调研、同行比价与 Excel 报告工具。JD.com supported; Taobao, Pinduoduo, Amazon & more planned. · GitHub

Repository files navigation

E-commerce CLI

用真实可见的 Chrome 完成电商商品调研、同类商品比价与 Excel 报告生成。

Research products, compare market prices, and generate Excel reports through a real, visible Chrome browser.

中文 · English

🌐 项目官网 / Project website: https://qs3c.github.io/E-commerce-cli/


中文

ecom 是一个面向电商运营、选品团队、市场研究人员与 AI Agent 的本地电商调研 CLI。输入关键词后,它会通过 Chrome DevTools Protocol(CDP)连接本机的可见 Chrome,模拟正常用户的输入、点击和滚动,采集搜索结果与商品详情,并将数据整理为可恢复的批次文件和便于决策的 Excel 报告。

当前 MVP 已支持 京东(JD.com)。淘宝、天猫、拼多多、当当、Amazon、eBay 等国内外电商平台已列入 Roadmap,将通过可扩展的站点适配器逐步接入。

Important

本项目强调低频、串行、可见且可人工接管的浏览器交互,能够减少无头、高并发采集常见的异常行为特征,但 CDP 或人类式节奏不代表“不可检测”,也不能保证不触发平台验证。请遵守目标网站条款、robots 规则和当地法律,仅采集你有权访问的数据。

为什么使用 E-commerce CLI?

  • 自动商品调研:按关键词批量收集标题、价格、店铺、品牌、评价、促销、库存配送和规格等信息。
  • 同行比价与市场洞察:查看最低价、中位价、平均价、最高价、价格分布、品牌 Top 10、店铺 Top 10 与自营比例。
  • 一键 Excel 报告:生成包含“调研概览”和“商品明细”的 .xlsx 工作簿,适合选品、采购、定价和竞品分析。
  • 真实浏览器交互:通过 CDP 驱动本机可见 Chrome,使用页面输入、点击、滚轮滚动和正常页面跳转,而非直接调用私有接口。
  • 断点恢复:冻结首次搜索队列;遇到人工验证、访问限制或中断后,可从检查点继续,不重复已完成商品。
  • Agent 友好:所有主要命令支持稳定的单行 JSON 输出,stdout 与诊断日志分离,便于脚本、工作流和 AI Agent 集成。
  • 本地优先:浏览器 Profile、认证状态和调研数据均保留在本机;批次目录不复制 Cookie、Local Storage、请求头或账号资料。

工作流程

flowchart LR
    A["输入商品关键词"] --> B["可见 Chrome + 本机 CDP"]
    B --> C["逐字输入、点击、分段滚动"]
    C --> D["串行采集搜索结果与商品详情"]
    D --> E["JSONL / CSV 可恢复批次"]
    E --> F["价格分析与 Excel 调研报告"]
    C -. "登录或验证" .-> G["暂停并等待人工处理"]
    G -. "resume" .-> D
Loading

当前可采集的数据

类别 字段
商品身份 商品 ID、搜索排名、标题、canonical 商品链接、主图
价格与销售 展示价格、精确金额、评价数、促销信息
商家 店铺名称、店铺链接、自营标记
商品详情 品牌、规格参数、库存、配送地区与配送信息
质量与追踪 完整/部分采集状态、采集时间、来源页面、商品级错误

单次任务可串行处理 1–50 个商品。页面中缺失的非关键字段会记录为 null 或“部分采集”,而不是臆造数据。

Excel 调研报告

完成采集后,可显式生成专业的 Excel 市场调研报告:

  • 调研概览:批次状态、样本规模、错误数、价格摘要、五档价格分布、品牌/店铺 Top 10、自营比例和价格分布图。
  • 商品明细:主图、标题、价格、品牌、店铺、自营、评价数、促销、库存配送、规格摘要、商品链接与采集状态。
  • 精确与安全:人民币金额按“分”精确统计;文本经过公式注入防护;临时工作簿验证成功后才原子发布。
  • 失败隔离:单张图片失败不会中断整份报告;报告生成失败不会改写原始 JSON/JSONL/CSV,也不会覆盖已有的有效报告。

Excel 导出依赖可选的 OfficeCLI 1.0.139+,采集功能本身不依赖 Microsoft Office 或 OfficeCLI。详细说明见 Excel 报告文档

更稳健的浏览器交互与安全边界

项目没有把“反检测”建立在隐身补丁或指纹伪装上,而是尽量贴近低频、可观察、可中断的正常浏览过程:

  • 使用 puppeteer-core 通过仅监听 127.0.0.1 的 CDP 端口连接 可见 Chrome
  • 使用独立 Chrome Profile,由用户在浏览器内亲自登录,不读取日常 Chrome Profile。
  • 逐字输入关键词、分段滚动、商品间随机停顿,并在若干商品后进行更长休息。
  • 商品详情始终串行采集,不进行高并发页面操作或批量私有 API 请求。
  • 发现登录、安全验证或访问限制时立即写入检查点并停止,等待用户处理。
  • 不提供无头模式、验证码绕过、指纹伪装、请求拦截、Cookie 导出或账号信息采集。

这些设计有助于减少不必要的风控触发并保护账号与会话数据,但无法消除平台检测风险。平台规则、页面结构和风控策略可能随时变化。

数据与报告侧也包含额外防护:

  • 批次目录只保存页面可见的商品研究数据和运行诊断,不保存浏览器认证材料。
  • CSV/Excel 文本经过公式注入防护,避免网页内容被表格软件解释为公式。
  • 报告缩略图下载不携带 Cookie、认证头、Referer 或浏览器状态,并拒绝本机、内网、保留地址及不安全图片。
  • 原始批次和报告均采用校验与原子写入策略,降低中断造成文件损坏的风险。

快速开始

环境要求

  • Windows
  • Node.js 22.12.0+
  • Google Chrome
  • 本机 127.0.0.1 可使用一个 CDP 端口(默认 53471)

安装

npm install --global e-commerce-cli
ecom doctor

如需从源码安装:

git clone https://github.com/qs3c/E-commerce-cli.git
cd E-commerce-cli
npm install
npm run build
npm install --global .

也可不全局安装,将下文的 ecom 替换为 node dist/cli/index.js。

登录京东

ecom login jd

命令会打开可见 Chrome 后立即返回。请在浏览器中人工完成登录;后续任务复用项目专用 Profile:

%USERPROFILE%\.e-commerce-cli\browser-data

采集与生成报告

# 采集最多 20 个“机械键盘”搜索结果及商品详情
ecom collect jd "机械键盘" --limit 20

# 查看批次状态
ecom status jd-20260826-103500-a1b2

# 从冻结队列继续被中断的任务
ecom resume jd-20260826-103500-a1b2

# 用户确认后生成 Excel 报告(需要 OfficeCLI 1.0.139+)
ecom report excel jd-20260826-103500-a1b2

JSON 输出与自动化集成

doctor、collect、status、resume 和 report excel 都支持 --json:

ecom collect jd "机械键盘" --limit 3 --json

JSON 模式下 stdout 只输出一行成功或失败 envelope,进度与诊断只写入 stderr。因此脚本或 Agent 可以直接解析结果,无需清洗日志。采集命令仅返回建议的 Excel 后续动作,不会在未确认时自动调用 OfficeCLI。

批次文件

每个批次保存在:

%USERPROFILE%\.e-commerce-cli\batches\<batch-id>\
文件 用途
manifest.json 批次状态、冻结搜索队列、进度、运行环境和产物路径
products.jsonl 每行一个通过 Zod Schema 校验的商品记录
products.csv 带 UTF-8 BOM、按排名排序且防公式注入的表格数据
errors.jsonl 商品级警告、失败和任务级错误
report.xlsx 可选的 Excel 调研报告

平台支持与 Roadmap

平台 状态 计划方向
京东 / JD.com ✅ 已支持(MVP) 关键词搜索、商品详情、断点恢复、CSV/JSONL、Excel 报告
淘宝 / Taobao、天猫 / Tmall 🗺️ 计划中 商品搜索、店铺与价格研究
拼多多 / Pinduoduo 🗺️ 计划中 商品搜索、拼单价与店铺信息
当当 / Dangdang 🗺️ 计划中 图书与综合商品调研
苏宁易购、唯品会、1688 🗺️ 长期计划 国内多平台选品与价格比较
Amazon、eBay、Walmart 🗺️ 长期计划 海外零售商品研究与竞品比价
AliExpress、Shopee、Lazada 🗺️ 长期计划 跨境电商与区域市场研究

Roadmap 不代表交付承诺或确定日期。我们会优先依据社区需求、平台可测试性和合规边界逐步实现;欢迎通过 Issue 告诉我们你最需要的平台和数据字段。

技术架构

  • TypeScript + Node.js:严格类型的 CLI 与领域模型。
  • Chrome DevTools Protocol + Puppeteer Core:连接真实可见 Chrome,不捆绑另一个浏览器。
  • 站点适配器:搜索、详情与页面状态识别按平台隔离,方便后续扩展新的电商网站。
  • Zod Schema:所有落盘记录先验证,避免静默写入结构不完整的数据。
  • 可恢复任务引擎:冻结搜索快照、记录逐商品状态并验证恢复现场。
  • OfficeCLI:可选地构建、校验并原子发布 Excel 工作簿。

Chrome 配置

程序按以下顺序查找 Chrome:

  1. ECOM_CHROME_PATH
  2. %LOCALAPPDATA%\Google\Chrome\Application\chrome.exe
  3. %PROGRAMFILES%\Google\Chrome\Application\chrome.exe
  4. %PROGRAMFILES(X86)%\Google\Chrome\Application\chrome.exe

可覆盖 Chrome 路径与本机 CDP 端口:

$env:ECOM_CHROME_PATH = 'C:\Program Files\Google\Chrome\Application\chrome.exe'
$env:ECOM_BROWSER_REMOTE_DEBUGGING_PORT = '53471'

开发与验证

npm run check

该命令依次执行 TypeScript 类型检查、全部 Vitest 测试和生产构建。

参与贡献

欢迎提交 Issue、适配器建议、页面选择器更新、测试夹具和 Pull Request。新增平台时,请保留项目现有原则:可见浏览器、低频串行操作、遇验证停止、不复制认证数据,并为页面解析与异常状态补充脱敏测试夹具。

如果这个项目对你的电商调研、竞品分析、选品或价格监控工作有帮助,欢迎点一个 ⭐,也欢迎分享你的使用场景。


English

ecom is a local e-commerce research CLI for marketplace operators, sourcing teams, market researchers, and AI agents. Give it a product keyword and it connects to a visible local Chrome through the Chrome DevTools Protocol (CDP), performs human-paced page interactions, collects search results and product details, and turns the data into resumable batch artifacts and decision-ready Excel reports.

The current MVP supports JD.com. Taobao, Tmall, Pinduoduo, Dangdang, Amazon, eBay, and other domestic and international marketplaces are on the roadmap and will be added through extensible site adapters.

Important

This project favors low-frequency, serial, visible, and human-supervised browser interaction. That reduces behavioral anomalies commonly associated with headless or high-concurrency collection, but CDP and human-like pacing do not make automation undetectable and cannot guarantee that a platform will not request verification. Follow each website's terms, robots rules, and applicable laws, and collect only data you are authorized to access.

Why E-commerce CLI?

  • Automated product research: collect titles, prices, stores, brands, review counts, promotions, stock, delivery details, and specifications by keyword.
  • Competitor price comparison and market insight: analyze minimum, median, average, and maximum prices, price distribution, top brands, top stores, and first-party seller ratio.
  • One-command Excel reports: build an .xlsx workbook with Research Overview and Product Details sheets for sourcing, pricing, assortment planning, and competitor analysis.
  • Real-browser interaction: drive visible local Chrome through CDP using page input, clicks, wheel scrolling, and normal navigation instead of private APIs.
  • Resumable jobs: freeze the initial result queue and continue from a checkpoint after manual verification, access limits, or interruption without reprocessing completed products.
  • Agent-friendly output: every main command supports stable single-line JSON, with stdout separated from progress and diagnostics for scripts, workflows, and AI agents.
  • Local-first privacy: browser profiles, authentication state, and research data remain local; batch folders never copy cookies, Local Storage, request headers, or account details.

Workflow

flowchart LR
    A["Product keyword"] --> B["Visible Chrome + local CDP"]
    B --> C["Typing, clicks, and segmented scrolling"]
    C --> D["Serial search and product-detail collection"]
    D --> E["Resumable JSONL / CSV batch"]
    E --> F["Price analysis and Excel report"]
    C -. "Login or verification" .-> G["Pause for human action"]
    G -. "resume" .-> D
Loading

Data Collected Today

Category Fields
Product identity Product ID, search rank, title, canonical product URL, main image
Price and sales Display price, exact amount, review count, promotions
Merchant Store name, store URL, first-party seller flag
Product details Brand, specifications, stock, delivery region, delivery details
Quality and traceability Complete/partial status, capture time, source pages, per-product errors

Each job processes 1–50 products serially. Missing optional page fields are recorded as null or partial data instead of being invented.

Excel Research Reports

After collection, explicitly generate a professional market-research workbook:

  • Research Overview: batch status, sample sizes, errors, price summary, five price buckets, top 10 brands/stores, first-party seller ratio, and price distribution chart.
  • Product Details: image, title, price, brand, store, first-party flag, review count, promotions, stock and delivery, specification summary, product link, and capture status.
  • Precision and safety: CNY calculations preserve exact cents, text is protected against spreadsheet formula injection, and a temporary workbook is validated before atomic publication.
  • Failure isolation: one failed image does not fail the workbook; report errors do not modify source JSON/JSONL/CSV or overwrite an existing valid report.

Excel export uses optional OfficeCLI 1.0.139+. Product collection itself requires neither Microsoft Office nor OfficeCLI. See the Excel report guide for details.

Safer Browser Interaction and Security Boundaries

Instead of relying on stealth patches or fingerprint spoofing, the project follows a low-frequency, observable, and interruptible browsing model:

  • puppeteer-core connects to visible Chrome through a CDP port bound only to 127.0.0.1.
  • A dedicated Chrome profile is used; the user logs in manually without exposing their everyday Chrome profile.
  • Keywords are typed progressively, pages are scrolled in segments, product actions use randomized pauses, and longer rests occur periodically.
  • Product details are collected serially—never through high-concurrency page actions or bulk private API calls.
  • Login pages, security verification, and access limits create a checkpoint and stop the job for human action.
  • No headless mode, CAPTCHA bypass, fingerprint spoofing, request interception, cookie export, or account-data collection is provided.

These choices can reduce unnecessary risk-control triggers and protect account/session data, but they cannot eliminate platform detection risk. Platform rules, page structures, and risk-control systems can change at any time.

Additional data and report protections include:

  • Batch folders contain only visible product-research data and diagnostics, never browser authentication material.
  • CSV and Excel text is protected against formula injection.
  • Report thumbnails are downloaded without cookies, authorization headers, Referer, or browser state; local, private, reserved, and unsafe image targets are rejected.
  • Raw batches and reports use validation and atomic writes to reduce corruption after interruption.

Quick Start

Requirements

  • Windows
  • Node.js 22.12.0+
  • Google Chrome
  • An available local CDP port on 127.0.0.1 (default: 53471)

Install

npm install --global e-commerce-cli
ecom doctor

To install from source instead:

git clone https://github.com/qs3c/E-commerce-cli.git
cd E-commerce-cli
npm install
npm run build
npm install --global .

Without a global install, replace ecom below with node dist/cli/index.js.

Sign In to JD.com

ecom login jd

The command opens visible Chrome and returns immediately. Sign in manually in the browser. Future jobs reuse the project-specific profile at:

%USERPROFILE%\.e-commerce-cli\browser-data

Collect and Report

# Collect up to 20 mechanical-keyboard results and product details
ecom collect jd "mechanical keyboard" --limit 20

# Inspect batch status
ecom status jd-20260826-103500-a1b2

# Continue an interrupted frozen queue
ecom resume jd-20260826-103500-a1b2

# Generate Excel after user confirmation (OfficeCLI 1.0.139+ required)
ecom report excel jd-20260826-103500-a1b2

JSON Output and Automation

doctor, collect, status, resume, and report excel support --json:

ecom collect jd "mechanical keyboard" --limit 3 --json

In JSON mode, stdout contains exactly one success or error envelope; progress and diagnostics go to stderr. Scripts and agents can parse results without filtering logs. Collection only suggests a follow-up Excel action and never invokes OfficeCLI without confirmation.

Batch Artifacts

Each batch is stored under:

%USERPROFILE%\.e-commerce-cli\batches\<batch-id>\
File Purpose
manifest.json Batch state, frozen result queue, progress, runtime, and artifact paths
products.jsonl One Zod-validated product record per line
products.csv UTF-8 BOM, rank-sorted, formula-injection-safe tabular data
errors.jsonl Per-product warnings/failures and job-level errors
report.xlsx Optional Excel research report

Platform Support and Roadmap

Marketplace Status Planned scope
JD.com ✅ Supported (MVP) Keyword search, details, resume, CSV/JSONL, Excel reports
Taobao and Tmall 🗺️ Planned Product search, store research, and price analysis
Pinduoduo 🗺️ Planned Product search, group pricing, and merchant information
Dangdang 🗺️ Planned Book and general-product research
Suning, Vipshop, and 1688 🗺️ Long-term Multi-marketplace sourcing and price comparison in China
Amazon, eBay, and Walmart 🗺️ Long-term International retail research and competitor pricing
AliExpress, Shopee, and Lazada 🗺️ Long-term Cross-border and regional marketplace research

The roadmap is not a delivery commitment or schedule. Priorities will reflect community demand, testability, and compliance boundaries. Open an Issue to tell us which marketplace and data fields matter most to you.

Architecture

  • TypeScript + Node.js for a strongly typed CLI and domain model.
  • Chrome DevTools Protocol + Puppeteer Core to connect to real, visible Chrome without bundling another browser.
  • Site adapters to isolate search, detail extraction, and page-state detection for each marketplace.
  • Zod schemas to validate every persisted record before it reaches disk.
  • Resumable task engine to freeze search snapshots, record per-product progress, and validate browser state before resuming.
  • OfficeCLI to optionally build, validate, and atomically publish Excel workbooks.

Chrome Configuration

Chrome is located in this order:

  1. ECOM_CHROME_PATH
  2. %LOCALAPPDATA%\Google\Chrome\Application\chrome.exe
  3. %PROGRAMFILES%\Google\Chrome\Application\chrome.exe
  4. %PROGRAMFILES(X86)%\Google\Chrome\Application\chrome.exe

Override the executable or local CDP port when needed:

$env:ECOM_CHROME_PATH = 'C:\Program Files\Google\Chrome\Application\chrome.exe'
$env:ECOM_BROWSER_REMOTE_DEBUGGING_PORT = '53471'

Development and Verification

npm run check

This runs TypeScript type checking, the complete Vitest suite, and a production build.

Contributing

Issues, adapter proposals, selector updates, sanitized fixtures, and pull requests are welcome. New marketplace adapters should preserve the project's existing principles: visible browsing, low-frequency serial interactions, stop-on-verification behavior, no authentication-data copying, and sanitized fixtures for parsing and exceptional states.

If E-commerce CLI helps with your product research, competitor analysis, sourcing, or price-monitoring workflow, consider giving the repository a ⭐ and sharing your use case.

About

CDP-powered e-commerce product research CLI for human-paced browser collection, competitor price comparison & Excel reports. 基于真实 Chrome 的电商调研、同行比价与 Excel 报告工具。JD.com supported; Taobao, Pinduoduo, Amazon & more planned.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages


Back | FazBrowse Home | New Git URL