FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat(python): add layers map for separate per-layer dependency sets by nitishagar · Pull Request #13671 · serverless/serverless · GitHub

Repository navigation

feat(python): add layers map for separate per-layer dependency sets - #13671

Open
nitishagar wants to merge 2 commits into
serverless:mainfrom
nitishagar:fix/13384-python-separate-layers
Open

nitishagar wants to merge 2 commits into
serverless:mainfrom
nitishagar:fix/13384-python-separate-layers

Conversation

nitishagar commented Jul 2, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown

Summary

  • Adds custom.pythonRequirements.layers — a named map where each entry specifies its own requirements file, enabling independent dependency sets across multiple Lambda layers.
  • Each named entry installs, zips, and registers its own service.layers[<name>] entry; the core compiler emits <Name>LambdaLayer + <Name>LambdaLayerQualifiedArn with no changes to core code.
  • The existing single-layer (layer: true) path is untouched; layer and layers coexist independently.

Key design decisions

  • Map shape (not array): aligns with service.layers which is already a named map; makes Ref: <Name>LambdaLayer natural.
  • package.artifact explicit pointer: satisfies provider.js, get-lambda-layer-artifact-path.js, and package-service.js without relying on naming-convention coincidence.
  • Deferred registration: accumulator is Object.assign-ed onto service.layers only after all layers build successfully — no half-registered state on failure (INV-8).
  • Pre-flight name validation: reserved name pythonRequirements and collisions with existing service layers are rejected before any install work begins (INV-3).

Test plan

  • npm run test:unit -w @serverless/framework -- test/unit/lib/plugins/python — 12 new unit tests cover all invariants (registration, defaults, overrides, empty map, reserved name, collision, mid-loop failure atomicity, single-layer coexistence)
  • Full unit suite: npm run test:unit -w @serverless/framework — 1887 tests, all pass
  • Integration fixture packages/sf-core/tests/python/tests/layers_multi/ verifies end-to-end packaging with a real Python toolchain (best-effort; not required for CI without Python)
  • npm run prettier:fix && npm run lint:fix — clean

Closes #13384

Summary by CodeRabbit

  • New Features
    • Added support for packaging multiple Python dependency layers from separate requirements files.
    • Layer settings can be configured per named layer, including names, descriptions, runtimes, architectures, and function attachments.
  • Bug Fixes
    • Improved validation for layer definitions, including clearer errors for invalid requirements files, reserved names, and naming collisions.
    • Preserved compatibility with the existing single-layer configuration.
  • Documentation
    • Expanded the Python Lambda Layer guide with multiple-layer configuration guidance and examples.

Mmarzex commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues
✅ Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

coderabbitai Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info ⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 61535768-f87c-4e73-898d-e21cf514e164

📥 Commits

Reviewing files that changed from the base of the PR and between 60f04e1 and fb232ca.

📒 Files selected for processing (2)
  • packages/serverless/lib/plugins/python/lib/pip.js
  • packages/serverless/test/unit/lib/plugins/python/pip.test.js

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.


📝 Walkthrough

Walkthrough

This PR adds support for multiple named Lambda layers from separate Python requirements files through custom.pythonRequirements.layers. It adds validation, per-layer installation and packaging, tests, documentation, and an integration fixture.

Changes

Named Python Lambda Layers

Layer / File(s) Summary
Options sanitization, schema, and activation predicate
packages/serverless/lib/plugins/python/index.js
Invalid layers values are removed. The schema defines per-layer properties. Python layer-only activation supports both layer and layers.
Per-file requirements installation helper
packages/serverless/lib/plugins/python/lib/pip.js
Adds and exports installRequirementsForFile, which validates requirements files, installs dependencies, and handles per-layer caching and vendor libraries.
Named layer validation and packaging
packages/serverless/lib/plugins/python/lib/layer.js
Validates names, creates per-layer archives under python/, registers named layers, and preserves the existing single-layer path.
Unit tests for installation and layer logic
packages/serverless/test/unit/lib/plugins/python/layer.test.js, packages/serverless/test/unit/lib/plugins/python/pip.test.js
Tests requirements-file validation, caching, layer registration, metadata overrides, name conflicts, failures, and single-layer compatibility.
Integration fixture and documentation
packages/sf-core/tests/python/test.js, packages/sf-core/tests/python/tests/layers_multi/*, docs/sf/providers/aws/guide/python.md
Adds a two-layer service fixture, integration assertions for resources and archive contents, and documentation for multiple layers.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Merge Risk: ⚪ Minimal · up to fb232

This adds named Python dependency layers while preserving the existing single-layer configuration. Invalid inputs and layer-name conflicts are handled before packaging, and the supplied tests cover packaging output and failure paths; no merge-blocking risk is identified.

Sequence Diagram(s)

sequenceDiagram
  participant ServerlessConfig
  participant PythonPlugin
  participant RequirementsInstaller
  participant LambdaLayers
  ServerlessConfig->>PythonPlugin: custom.pythonRequirements.layers
  PythonPlugin->>RequirementsInstaller: install each requirements file
  RequirementsInstaller-->>PythonPlugin: per-layer working directory
  PythonPlugin->>PythonPlugin: create per-layer zip archive
  PythonPlugin->>LambdaLayers: register named layer resources
Loading

Suggested reviewers: czubocha

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: adding a Python requirements layers map for separate per-layer dependency sets.
Linked Issues check ✅ Passed The changes satisfy issue #13384 by adding named Python requirement layers, building each layer from its own requirements file, registering separate Lambda layer resources, and supporting independent …
Out of Scope Changes check ✅ Passed The implementation, schema updates, documentation, unit tests, integration test, and fixtures directly support the named per-layer dependency objective. No unrelated code changes are identified.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1 ⚔️ Resolve merge conflicts 💡
  • Resolve merge conflict in branch fix/13384-python-separate-layers
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Actionable comments posted: 4

🧹 Nitpick comments (2)
docs/sf/providers/aws/guide/python.md (1)

342-358: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Minor: combined example block breaks from surrounding doc convention.

The preceding "Lambda Layer" section splits custom and functions config into separate YAML blocks; this new example merges both into one, which is a small stylistic inconsistency.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/sf/providers/aws/guide/python.md` around lines 342 - 358, The new Python
layer example in the AWS guide breaks the existing documentation style by
merging the `custom` and `functions` sections into a single YAML block. Update
the example near the `pythonRequirements` and `api` configuration so it matches
the surrounding “Lambda Layer” convention: keep the `custom` configuration and
the `functions` configuration as separate YAML snippets while preserving the
same `PydanticLambdaLayer` and `WebLambdaLayer` references.
packages/serverless/lib/plugins/python/index.js (1)

127-132: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Invalid options.layers is silently dropped without any warning.

When custom.pythonRequirements.layers is misconfigured (not an object, or an array), it's deleted with no log message. Elsewhere in this same getter, misconfiguration (dockerImage+dockerFile) throws, and docker-related misconfig triggers this.warningLogged/this.log.warning. A user who accidentally sets layers to the wrong shape will see no layers built and no explanation why.

💡 Suggested fix: warn on invalid shape
     if (
       options.layers != null &&
       (typeof options.layers !== 'object' || Array.isArray(options.layers))
     ) {
+      if (this.log) {
+        this.log.warning(
+          'custom.pythonRequirements.layers must be an object map of layer definitions; ignoring invalid value.',
+        )
+      } else {
+        this.serverless.cli.log(
+          'WARNING: custom.pythonRequirements.layers must be an object map of layer definitions; ignoring invalid value.',
+        )
+      }
       delete options.layers
     }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/serverless/lib/plugins/python/index.js` around lines 127 - 132, The
invalid options.layers handling in pythonRequirements should not silently delete
misconfigured values; update the getter in the Python plugin to emit a warning
before removing layers when it is not an object or is an array. Follow the
existing validation patterns used nearby in this same getter, such as the
dockerImage/dockerFile conflict and docker-related warning logic, and use
this.warningLogged or this.log.warning so users know their
custom.pythonRequirements.layers setting was ignored.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/serverless/lib/plugins/python/lib/layer.js`:
- Around line 126-131: The empty-requirements fast path in the layer zip flow
only checks requirementsTxtPath with fse.existsSync, so a zero-byte
requirements.txt still falls through and gets packaged. Update the branch in the
layer.js logic that builds the rootZip/writeZip path to treat an empty
requirements.txt the same as a missing one by checking file size as well as
existence, so the JSZip clean empty zip path is used whenever the generated
requirements file has no content.
- Around line 112-165: The symlink creation in zipNamedLayerRequirements is not
awaited, so failures can be swallowed and the layer zip may never be created;
update the async flow to await the file operation and let errors fail the build.
Apply the same fix in zipRequirements as well, since it has the same un-awaited
fse.symlink pattern. Use the existing zipCachePath/targetZipPath branch logic
and keep the Windows copySync path unchanged.

In `@packages/serverless/lib/plugins/python/lib/pip.js`:
- Around line 1067-1079: The empty requirements flow is inconsistent between
installRequirementsForFile and zipNamedLayerRequirements, causing an empty
requirements.txt to be packaged instead of treated as “no reqs.” Update
installRequirementsForFile to signal the empty-file case more explicitly (for
example by returning null/undefined) or update zipNamedLayerRequirements to
check the requirements file size in addition to existsSync before zipping. Make
the behavior consistent using the existing symbols generateRequirementsFile,
installRequirementsForFile, and zipNamedLayerRequirements so an empty layer
produces a clean empty python/ artifact without requirements.txt.
- Around line 1052-1065: Add an explicit guard in the Python requirements layer
flow before `path.isAbsolute(requirementsFile)` is called, since
`requirementsFile` may be missing or non-string when schema validation is
skipped. Update the logic in `pip.js` around the `absRequirementsFile`
resolution to first validate `requirementsFile` and throw a `ServerlessError`
with a clear message and the existing
`PYTHON_REQUIREMENTS_LAYER_REQUIREMENTS_FILE_INVALID` code, rather than allowing
a native `TypeError` to escape. Use the existing `requirementsFile` and
`layerName` handling in this block to keep the error consistent with the current
file-existence check.

---

Nitpick comments:
In `@docs/sf/providers/aws/guide/python.md`:
- Around line 342-358: The new Python layer example in the AWS guide breaks the
existing documentation style by merging the `custom` and `functions` sections
into a single YAML block. Update the example near the `pythonRequirements` and
`api` configuration so it matches the surrounding “Lambda Layer” convention:
keep the `custom` configuration and the `functions` configuration as separate
YAML snippets while preserving the same `PydanticLambdaLayer` and
`WebLambdaLayer` references.

In `@packages/serverless/lib/plugins/python/index.js`:
- Around line 127-132: The invalid options.layers handling in pythonRequirements
should not silently delete misconfigured values; update the getter in the Python
plugin to emit a warning before removing layers when it is not an object or is
an array. Follow the existing validation patterns used nearby in this same
getter, such as the dockerImage/dockerFile conflict and docker-related warning
logic, and use this.warningLogged or this.log.warning so users know their
custom.pythonRequirements.layers setting was ignored.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info ⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5ef6c132-10c6-430e-92b1-c80848a65d3e

📥 Commits

Reviewing files that changed from the base of the PR and between b5ff282 and 8df2d23cbf5fce325c4fe4e11607dd005ef35993.

📒 Files selected for processing (10)
  • docs/sf/providers/aws/guide/python.md
  • packages/serverless/lib/plugins/python/index.js
  • packages/serverless/lib/plugins/python/lib/layer.js
  • packages/serverless/lib/plugins/python/lib/pip.js
  • packages/serverless/test/unit/lib/plugins/python/layer.test.js
  • packages/serverless/test/unit/lib/plugins/python/pip.test.js
  • packages/sf-core/tests/python/test.js
  • packages/sf-core/tests/python/tests/layers_multi/requirements/pydantic.txt
  • packages/sf-core/tests/python/tests/layers_multi/requirements/ulid.txt
  • packages/sf-core/tests/python/tests/layers_multi/serverless.yml

Add `custom.pythonRequirements.layers` — a named map where each entry
specifies its own requirements file.  For each entry the plugin:

1. Installs the explicit requirements file into a per-layer working
   directory via a new `installRequirementsForFile` helper in pip.js.
2. Zips the installed packages under a `python/` prefix to
   `.serverless/pythonRequirements-<name>.zip`.
3. Registers `service.layers[<name>]` with `package.artifact` set so
   the core layer compiler emits `<Name>LambdaLayer` and
   `<Name>LambdaLayerQualifiedArn` without further changes.

Functions attach a layer with `Ref: <Name>LambdaLayer`.

Invariants preserved:
- The existing single-layer (`layer: true`) path is untouched.
- `layer` and `layers` coexist independently.
- The reserved name `pythonRequirements` and collisions with existing
  service layers are rejected before any install work begins.
- Registration is deferred to a post-loop `Object.assign` so a
  mid-loop failure never leaves a half-registered state.

Adds a JSON schema for `custom.pythonRequirements.layers` (via
`defineCustomProperties`) and extends the function-less-service hook
guard to keep hooks alive for `layers`-only services.

Closes serverless#13384
nitishagar force-pushed the fix/13384-python-separate-layers branch from 8df2d23 to 60f04e1 Compare July 2, 2026 03:50
Per review: installRequirementsForFile now rejects a missing/empty
requirementsFile with the friendly PYTHON_REQUIREMENTS_LAYER_REQUIREMENTS_FILE_INVALID
error instead of the raw TypeError path.isAbsolute throws, matching the
adjacent does-not-exist check.
nitishagar force-pushed the fix/13384-python-separate-layers branch from aa231f6 to fb232ca Compare September 6, 2026 04:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Separate dependency for each layer using a separate requirements.txt file

2 participants


Back | FazBrowse Home | New Git URL