Integrate upstream changes by joe4dev · Pull Request #23 · localstack/lambda-runtime-init · GitHub

Motivation

The official AWS lambda-runtime-init received some upstream updates (typically merged in big blobs from their internal repository). We should update our customized localstack version to support the latest features (e.g., response streaming) and updates (e.g., the new Golang version). Beyond smaller doc updates, the main updates were:

Jun 8, 2023 chore(deps): Update to Golang 1.19 (#95) ⇒ Easy deps upgrade to Go 1.19
Apr 27, 2023 feat: Pull upstream changes 2023/04 (#87) ⇒ Major changes to the init and internal API (6,845 additions and 2,726 deletions)

Background

Internal docs: https://www.notion.so/localstack/Update-RIE-with-upstream-changes-for-response-streaming-18a303b503c54f958cfbee5841ef5402?pvs=4

Changes

Integrates the upstream changes from the AWS lambda-runtime-init repo and adjusts the API accordingly (see documented commit log for details).

👉 Custom LocalStack changes git diff 8b003ee53ffe758979cf64f43f6ae58363efcc2d..integrate-upstream-changes

Testing

Test against the LocalStack test suite.

🧪 The localstack branch integrate-lambda-init-upstream-changes contains some improved tests for the request_id propagation issue. For example: tests.aws.services.lambda_.test_lambda.TestRequestIdHandling.test_request_id_invoke

I tested other categories including:

tests.aws.services.lambda_.test_lambda.TestLambdaBaseFeatures
tests.aws.services.lambda_.test_lambda.TestRequestIdHandling

TODO

Fix failing snapshot test due to uuid not being replaced in the test tests.aws.services.lambda_.test_lambda.TestLambdaFeatures.test_invocation_with_qualifier
Resolve TODOs (mostly coding out some logs, error handling etc)
Rebase after merging the permission PR Fix filesystem permission parity #22 => Merged the localstack branch back as a quickfix because rebasing again is terrible with so many conflicts 😢

Removing the trailing whitespace in the README.md to improve the formatting and readability of the file.

Add Shield.io badges for Go version and GitHub release. Useful for displaying important information in a visually appealing and easily accessible way at a glance. Updated the license badge to standard Shield.io format. As it appears it was incorrect pointing to the badge for 'aws-sam-local'.

Add content table to README.md, making it more convenient for users to find the information they need and enabling them to quickly jump to specific sections.

Add markup to environment variable for better browsing

The content for some bullet points are not correctly indented or spaced. This causes text and code blocks to not be displayed properly, causing confusion for customers. This change indents the content of the bullet points so it is now correctly displayed increasing readability.

* Add missing backslash in docker command * Remove stray "`" from command * Covert code snippet to code block for consistency * Remove extraneous newline * Fix invalid italics and use "_" for consistency * Change (invalid) italics to header Make it consistent with the preceding section that says: "To build the emulator into your image" Also, the stray whitespace before the final asterisk was incorrect. * Add syntax highlighting to code blocks * Add header to content table

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.1.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.1.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…am-changes Integrate upstream changes. Notable conflicts: * Makefile: simply upgrade Go version to 1.19 from upstream and use LocalStack changes * go.mod: merge dependency updates from upstream and keep LocalStack changes * go.sum: re-generate * lambda/rapi/server.go: new interop import from upstream (?), otherwise not really conflicts just API updates and by-1-line-neighboring updates

Removed https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-f3273d2200ae8dda95046a3aa7643050f008af08f96b1359d8386682dd3bb5f9L69

Removed https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-2c1a36d379bd5ed6e893749912ac8473cc46dddf5765024d46b44da2eb56348dL192

Added https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-2c1a36d379bd5ed6e893749912ac8473cc46dddf5765024d46b44da2eb56348dR212

Added https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-2c1a36d379bd5ed6e893749912ac8473cc46dddf5765024d46b44da2eb56348dR183

Added https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-4f3b0e186aa0135ea4a0df9ea408382232a52ada5d8776b78f3391eb8e910890R152

Interface changes inspired by adaptations in MockInteropServer https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-ca7d03af9ea519754c0f107f0f63d2bbfbc18381703c4e226129cff807e9b719L29

Removed https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-af70cbe1459e50292214935e938de51f504915babf7cb657e7418ae7fb4e0109L153

Changed https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-4f3b0e186aa0135ea4a0df9ea408382232a52ada5d8776b78f3391eb8e910890R83

It's rather hard to get through everything, but our changes look good. Let's see if the LS tests pass :)

Fixed the error reporting to use the new dedicated callback for SendInitErrorResponse

Other issues discovered during testing:

It appears the test TestLambdaRuntimesCommon.test_introspection_invoke fails for for the runtimes python3_7, java8, java8_al2, java11, java17, go1_x, provided, provided_al2 due the two missing environment variables PWD=/var/task and SHLVL=0. These are part of CustomerEnvironmentVariables but just not available anymore for some runtimes 🙃 I also updated the snapshots but nothing has changed at real AWS.
go1.x runtime tests fail for me locally but that might be an issue with the ARM build. Both the previous init and these changes yield an error 2023-10-17 21:20:08 time="2023-10-17T19:20:08Z" level=warning msg="Process 19(bootstrap) exited: Runtime exited with error: signal: trace/breakpoint trap" func="go.amzn.com/lambda/core.(*Watchdog).GoWait.func1" file="/home/runner/work/lambda-runtime-init/lambda-runtime-init/lambda/core/watchdog.go:67"
Several more tests fail on macOS, most of them due to port in use when using multiple Lambda functions. Including: test_lambda_different_iam_keys_environment, test_runtime_introspection_arm (due to different debug env), test_lambda_concurrency_block, test_provisioned_concurrency, test_reserved_concurrency_async_queue, test_lambda_versions_with_code_changes, test_lambda_alias_moving, test_alias_routingconfig

It's rather hard to get through everything, but our changes look good. Let's see if the LS tests pass :)

Yeah, I think we just need to release an imperfect "-dev" version to get a good overview from a LocalStack CI run. (we could consider short-lived feature-branch artifacts in the future if velocity becomes a challenge)

Callum Bodels and others added 23 commits January 4, 2023 10:19

docs: removing trailing whitespace (#76 )

2aea6aa

Removing the trailing whitespace in the README.md to improve the formatting and readability of the file.

docs: adding content table (#78 )

4c56a13

Add content table to README.md, making it more convenient for users to find the information they need and enabling them to quickly jump to specific sections.

docs: Display environment variable as code snippet

ee52c16

Add markup to environment variable for better browsing

feat: Pull upstream changes 2023/04 (#87 )

c1cf0c5

chore(deps): Update to Golang 1.19 (#95 )

bf7e248

Remove unknown field CorrelationID for Invoke

8b003ee

Removed https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-f3273d2200ae8dda95046a3aa7643050f008af08f96b1359d8386682dd3bb5f9L69

Remove unknown field CorrelationID for Init

c6ac472

Removed https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-2c1a36d379bd5ed6e893749912ac8473cc46dddf5765024d46b44da2eb56348dL192

Add new sandbox init parameters

49a6bb0

Added https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-2c1a36d379bd5ed6e893749912ac8473cc46dddf5765024d46b44da2eb56348dR212

Add bootstrap to InitHandler parameters

75f706b

Added https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-2c1a36d379bd5ed6e893749912ac8473cc46dddf5765024d46b44da2eb56348dR183

Add new runtimeDomainRoot to NewBootstrapSingleCmd

a413742

Added https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-4f3b0e186aa0135ea4a0df9ea408382232a52ada5d8776b78f3391eb8e910890R152

Fix new interop server interface (WIP)

0d0a067

Interface changes inspired by adaptations in MockInteropServer https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-ca7d03af9ea519754c0f107f0f63d2bbfbc18381703c4e226129cff807e9b719L29

Remove SetTailLogOutput

35e77fb

Removed https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-af70cbe1459e50292214935e938de51f504915babf7cb657e7418ae7fb4e0109L153

Use new way of getting the default interop server

c4e9547

Changed https://github.com/aws/aws-lambda-runtime-interface-emulator/pull/87/files#diff-4f3b0e186aa0135ea4a0df9ea408382232a52ada5d8776b78f3391eb8e910890R83

Bump Golang Docker image for delve build

691651c

Clarify ARM debugging limitations

1e9ef9d

Fix sanbox creation nil pointer with new interop initialization

80f6683

Add status ready to LocalStack after await initialized (WIP)

ed9ea3e

Add custom LocalStack LogsEgressAPI using logCollector

6b9d9a6

joe4dev requested review from dfangl and dominikschubert October 16, 2023 08:41

joe4dev assigned joe4dev and dfangl Oct 16, 2023

joe4dev added 3 commits October 16, 2023 12:09

Improve logging for runtime init initialization

9fabdc9

Merge branch 'localstack' into integrate-upstream-changes

b273ca6

Fix missing request metadata in reserve call

95ee607

Add changelog entry for awsutil

cee0c69

joe4dev marked this pull request as ready for review October 17, 2023 14:48

joe4dev assigned dominikschubert Oct 17, 2023

dfangl approved these changes Oct 17, 2023

View reviewed changes

Fix init error payload reporting

37c4634

joe4dev merged commit 0064e07 into localstack Oct 18, 2023

joe4dev deleted the integrate-upstream-changes branch October 18, 2023 09:04

joe4dev mentioned this pull request Oct 18, 2023

Integrate lambda init upstream changes localstack/localstack#9378

Merged

1 task

joe4dev restored the integrate-upstream-changes branch October 18, 2023 12:54

joe4dev mentioned this pull request Oct 18, 2023

Integrate upstream changes v2 #24

Closed

3 tasks

joe4dev deleted the integrate-upstream-changes branch October 18, 2023 14:02

Integrate upstream changes by joe4dev · Pull Request #23 · localstack/lambda-runtime-init · GitHub

Uh oh!

Integrate upstream changes - #23

Integrate upstream changes#23
joe4dev merged 28 commits into
localstackfrom
integrate-upstream-changes

joe4dev commented Oct 16, 2023 •
edited
Loading

Uh oh!

Uh oh!

dfangl left a comment

Uh oh!

joe4dev commented Oct 17, 2023

Uh oh!

joe4dev commented Oct 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Integrate upstream changes by joe4dev · Pull Request #23 · localstack/lambda-runtime-init · GitHub

Uh oh!

Integrate upstream changes - #23

Integrate upstream changes#23joe4dev merged 28 commits intolocalstacklocalstack/lambda-runtime-init:localstackfrom integrate-upstream-changeslocalstack/lambda-runtime-init:integrate-upstream-changesCopy head branch name to clipboard

Conversation

joe4dev commented Oct 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Background

Changes

Testing

TODO

Uh oh!

dfangl left a comment

Choose a reason for hiding this comment

Uh oh!

joe4dev commented Oct 17, 2023

Uh oh!

joe4dev commented Oct 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Integrate upstream changes#23
joe4dev merged 28 commits into
localstackfrom
integrate-upstream-changes

joe4dev commented Oct 16, 2023 •
edited
Loading