…rget_meta, packaging, CI (#4)
* Wire up request-schema validation in scrape/scrape_async
- scrape() and scrape_async() now accept ScrapeRequest | Mapping[str, Any]
- Added _to_payload() helper: model_dump(by_alias=True) for Pydantic models, dict() for mappings
- Added _validate() method using jsonschema, raises decodo.errors.ValidationError on failure
- scrape_batch() accepts BatchRequest | Mapping[str, Any] but skips validation (mirrors TS)
- Constructor default changed to BundledSchema.shared (mirrors TS)
- Removed now-unnecessary type: ignore[arg-type] comments in tests
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Codegen: emit target_meta as plain dicts, add BatchRequest batch params
- generate_targets.py: removed generated TargetMeta pydantic class; target_meta is
now emitted as dict[str, dict[str, Any]] (plain dict literals, matching TS behavior)
- For each target, generate a *BatchParams class identical to *Params except url/query
are typed list[str] | None instead of str | None
- BatchRequest is now a proper discriminated union of all *BatchParams classes (was
wrongly aliased to ScrapeRequest)
- Regenerated src/decodo/generated/targets.py and request_schemas.py from IR
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix __init__ exports: drop phantom TargetTargetParams, add batch params
- Removed TargetTargetParams from __all__ (never existed, caused ImportError)
- Added all 52 *BatchParams classes and BatchRequest to imports and __all__
- GoogleSearchBatchParams and BatchRequest now publicly accessible for examples
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add py.typed marker and jsonschema runtime dep; remove codegen console script
- src/decodo/py.typed: empty marker file so mypy and PEP 561 consumers find types
- pyproject.toml: added jsonschema>=4.0 to runtime dependencies (scrape() uses it)
- pyproject.toml: added types-jsonschema>=4.0 to dev deps for mypy
- pyproject.toml: added [tool.setuptools.package-data] to ship py.typed in wheel
- pyproject.toml: removed [project.scripts] decodo-codegen entry (footgun: runs
full network fetch + file overwrite on any invocation including --help; maintainers
run via python -m decodo.codegen.codegen instead)
- pyproject.toml: added ^build/ to mypy exclude to avoid duplicate-module error
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* CI: run pytest; fix publish workflow checkout permission
- .github/workflows/test.yml: new workflow running pytest on push+PR, Python 3.12,
installs .[dev]; mirrors lint.yml style
- .github/workflows/worklfow.yml: added contents: read permission so actions/checkout
can access the private repo (id-token: write alone zeros contents, causing
'Repository not found' on checkout)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix README snippets and broken batch example
README.md:
- All DecodoClient(...) calls now use DecodoConfig(web_scraping_api=WebScrapingApiConfig(token=...))
- Quick start and all API snippets use typed Params objects (GoogleSearchParams etc.)
- Batch snippet uses GoogleSearchBatchParams with query as list
- Error handling snippet fixed to use typed params
- Documented that token is base64-encoded user:password from dashboard
- Added git install fallback (pip install from GitHub) until PyPI publish
- Removed git conflict marker line (>>>>>>> 03b68da)
examples/web_scraping_api/batch/google_search_batch.py:
- Import GoogleSearchBatchParams instead of GoogleSearchParams
- Removed duplicate print('Polling for results...') before the loop
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Return type generation from json
* Add message if imports missing, run codegen
* Add codegen in ci/cd
* Update readme
* Add codegen
* Update readme about outdir
* Update codegen dynamic imports
* Update target validation
* Remove friction from pip installs for codegen and bubundled schema issues after install
* Fix client side confusing warning error
* Fix mypy ir path issue
---------
Co-authored-by: Donatas Kasparavicius <donatas.kasparavicius@oxylabs.io>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: julka <julius.tvarijonavcius@decodo.com>