| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Atelier turns a published paper into a FindSim experiment definition. Given a PDF and a figure or panel to target, the pipeline reads the paper, builds the experiment description, digitizes the plotted data, and writes a FindSim record.
| Stage | Module | Output |
|---|---|---|
| 1. Paper to JSON | core/Paper2Json.py | paper.json |
| 2. Experiment builder | experimentBuilder.py | experiment.json, findsim_preview.json |
| 3. Data builder | core/data_builder.py | data.json, extracted_data.csv |
| 4. Model builder (optional) | core/model_builder.py | experimentMap.json |
| 5. Fold | fold.py | final findsim_preview.json |
pipeline.py is the single orchestrator that runs all of them.
Python 3.11 or newer (developed on 3.14).
git clone https://github.com/<your-org>/FSW-AI.git
cd FSW-AI
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtEvery stage uses one LLM backend, set with --backend:
| --backend | Requirement |
|---|---|
| cc (default) | The Claude Code CLI on your PATH, logged in |
| api | export ANTHROPIC_API_KEY=... |
| openai | export OPENAI_API_KEY=... |
Use --model to override the model for the chosen backend.
The pipeline needs two things at minimum:
Two optional inputs enable the model builder stage. Supply both or neither:
cd FSW-AI/pipeline
python pipeline.py \
--pdf /path/to/Abrams1991.pdf \
--target "Figure 2:A" \
--output-dir out/ \
--backend cc--target is repeatable. It accepts "Figure 2" for a whole figure or "Figure 2:A" for one panel. Omit it entirely to process every figure the paper contains.
A manifest is a JSON list of targets. PDFs are resolved as <pdf-dir>/<paper>.pdf.
[
{"paper": "Abrams1991", "figure": "Figure 2", "panel": null},
{"paper": "Alano2010", "figure": "Figure 3", "panel": "B"},
{"paper": "Fujiwara2014", "figure": "Figure 3", "panel": "C"}
]python pipeline.py \
--manifest manifests/list1.json \
--pdf-dir /path/to/pdfs \
--output-dir out/ \
--backend api \
--parallel 4Example manifests are in FSW-AI/pipeline/manifests/. To build one from a directory of ground truth filenames such as Abrams1991_Fig2.json:
python manifest_from_groundtruth.py --groundtruth-dir /path/to/Expts --output targets.jsonSize --parallel to your API rate limit, not your CPU count. Every worker draws from the same account wide budget.
One directory per paper, one subdirectory per figure or panel:
out/
└── Abrams1991/
├── paper.json # stage 1
├── figures/ # extracted figure images
└── figure2A/
├── experiment.json # stage 2
├── data.json # stage 3
├── extracted_data.csv # digitized points
├── extracted_plot.png # digitizer overlay
├── comparison_plot.png # digitized vs original
├── experimentMap.json # stage 4, only with --sim-model/--sim-map
└── findsim_preview.json # final FindSim record
| Flag | Effect |
|---|---|
| --reuse-paper-json | Skip stage 1 and reuse an existing paper.json |
| --reuse-experiment-json | Also reuse experiment.json and rerun from stage 3 |
| --force-redigitize | Ignore a cached extracted_data.json and call the digitizer again |
| --limit N | Process only the first N papers of a manifest |
| -v | DEBUG level logging |
Run python pipeline.py --help for the full list.
If a FindSim checkout sits beside this repository, its FindSim-Schema.json is found automatically. Otherwise point at it directly:
export FINDSIM_SCHEMA=/path/to/FindSim/FindSim-Schema.jsonWithout it, folding falls back to the Pydantic mirror in core/FindSim_schema.py.
| Script | Purpose |
|---|---|
| data_extractor.py | Digitize a single figure image or PDF, no paper context needed |
| pdf_annotator.py | Flask UI for reviewing and correcting paper.json |
| inspect_experiment.py | Print a readable summary of an experiment.json |
| plot_result.py | Plot digitized data against the source figure |
cd FSW-AI/pipeline
pytest -m unitUnit tests mock every LLM call. Tests marked integration need a live backend, and those marked docling run real PDF layout extraction. Tests that compare against FindSim itself skip unless FINDSIM_DIR points at a checkout.
separate_panels_by_whitespace.py # standalone panel splitter, imported by the pipeline WIP docs/ # flowchart and curation notes FSW-AI/pipeline/ ├── pipeline.py # main orchestrator ├── experimentBuilder.py # stage 2 CLI ├── dataBuilder.py # stage 3 CLI ├── modelBuilder.py # stage 4 CLI ├── fold.py # stage 5, merges data into the FindSim record ├── core/ # library modules for every stage ├── agents/ # LLM prompts, one CLAUDE.md per agent ├── manifests/ # example target lists └── templates/ # pdf_annotator UI template
The algorithms here are built by the authors, and the code is generated by the authors with the help of LLMs, specifically Claude's Sonnet 4.5 and Sonnet 5.0.
| Back | FazBrowse Home | New Git URL |