I struggle to see how this API can ever stabilize - what one can do when the heap is near the heap limit is very limited, and in particular if you do too much in the callback to incur another allocation, the program would be in a very unstable state. Our current callbacks only do very limited things within the C++, allowing users to do arbitrary things in JS sounds like an API that invites bugs...I think a safer option would be to only write a profile when it reaches the limit, though even then I am not sure how well the profiler works when the heap is about to reach the limit...

I'm also wary on adding a JS callback to be invoked at new heap limit. This callback also takes a heap profile parameter which is taken on a big heap, sounds like exacerbating the problem it wants to fix.

This API is not intended to make JS safe near OOM. It closes a blind spot in the sampling heap profiler: today, if a process starts startHeapProfile() and reaches a fatal OOM, the accumulated sampling data is lost because stopHeapProfile() is never called so the API provides a best-effort chance to retrieve that data before the process terminates.
The samples are collected continuously, so the near-limit event does not take a full heap snapshot or walk every heap object. However profiler->GetAllocationProfile() still materializes the sampled call tree and may allocate, so capture itself is also best-effort.

Why would the application want to save the profile in JS memory when it's likely not relevent for the application itself that may already crash soon? I think a file-backed API would achieve the same thing without making the users waste the remaining memory with (arguably very noisy) diagnostic data.

@joyeecheung Yes I agree with your proposal! I changed the approach to a file-backed one. please feel free to review it

Codecov Report

❌ Patch coverage is 49.24242% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.12%. Comparing base (984e46f) to head (63db010).
⚠️ Report is 13 commits behind head on main.

@@            Coverage Diff             @@
##             main   #64676      +/-   ##
==========================================
- Coverage   90.12%   90.12%   -0.01%     
==========================================
  Files         751      751              
  Lines      253629   253767     +138     
  Branches    47789    47808      +19     
==========================================
+ Hits       228575   228697     +122     
- Misses      16305    16312       +7     
- Partials     8749     8758       +9     

... and 34 files with indirect coverage changes

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@legendecas The sampler's live set isn't monotonic, so profile N+1 isn't a superset of profile N (with the default flags kSamplingNoFlags) a major GC always runs between two near-heap-limit callbacks, since those are the last-resort GC paths. So each profile drops whatever died since the previous one, which is often the spike that triggered it.

limit here is a budget on how many times we do expensive allocating work on a near-OOM path.

CI: https://ci.nodejs.org/job/node-test-pull-request/76449/

Would you mind rebasing the commits and squash them? CI is not really good at handling merge commits.

CI: https://ci.nodejs.org/job/node-test-pull-request/76479/