FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Fix for ForEach-Object -Parallel perf problem with many runspaces by PaulHigin · Pull Request #10455 · PowerShell/PowerShell · GitHub

Fix for ForEach-Object -Parallel perf problem with many runspaces - #10455

Merged
Aditya Patwardhan (adityapatwardhan) merged 3 commits into
PowerShell:masterfrom
PaulHigin:foreach_parallel_perf_fix
Sep 5, 2019
Merged

Fix for ForEach-Object -Parallel perf problem with many runspaces#10455
Aditya Patwardhan (adityapatwardhan) merged 3 commits into
PowerShell:masterfrom
PaulHigin:foreach_parallel_perf_fix

Conversation

Copy link
Copy Markdown
Contributor

PR Summary

This is a fix for the ForEach-Object perf Issue: #10450.

measure-command { 1..254 | foreach -Parallel {ping "192.168.0.$_" -n 1 | where {$_ -match "ttl="}} -ThrottleLimit 300 }

# Time for ThreadJob and ForEach -Parallel -AsJob is around 8 seconds
# Time for ForEach -Parallel is around 60 seconds

PR Context

The problem was that the number of concurrent runspaces was being limited to around 40-50, even though the ThrottleLimit was set to 300. It turns out it was the Cmdlet input processing thread that was causing the bottleneck. I don't know what the conflict is while spinning up runspaces, but the fix is to now do this on a dedicated thread similar to how it currently works for -AsJob.

PR Checklist

Copy link
Copy Markdown
Member

Paul Higinbotham (@PaulHigin)

The change is causing a lot memory allocations as compared to before.

Scenario: measure-command { 1..10kb | ForEach-Object -Parallel { }}

Before change: 105 MB in 2.6 minutes
After change: 10,200 MB in 2.6 minutes

Dongbo Wang (daxian-dbw) dismissed Aditya Patwardhan (adityapatwardhan)’s stale review August 30, 2019 23:31

Dismiss Aditya's review given his new finding regarding the PR.

Copy link
Copy Markdown
Contributor Author

Aditya Patwardhan (@adityapatwardhan)
I am not too surprised at this, as this change increases the processing rate of creating/running scripts. Even though the default Throttle limit is 5, since the script block is empty the code is basically creating and disposing threads/runspaces as fast as it can, 10000 times. So I am not surprised that memory consumption is large.

As long as the CLR eventually collects the memory, I think we are Ok.

I have run tests that loop for hours, detecting handle and working set usage. So far I see the usage plateau after a while so there seems to be no resource leak. However, the CLR and GC is getting a lot of exercise!

Rain Sallow (/u/ta11ow) (vexx32) commented Sep 4, 2019
edited
Loading

Copy link
Copy Markdown
Collaborator

Speed is awesome, but I don't think most folks will agree that... what, 10 gigabytes(?) of RAM usage for a task that is literally doing no work is something they'll want to work with regularly if at all.

Are currently available alternatives this excessive in terms of memory usage? (e.g., PoshRSJobs?)

Copy link
Copy Markdown
Contributor Author

Rain Sallow (/u/ta11ow) (@vexx32)

measure-command { 1..10kb | ForEach-Object -Parallel { }}

The code is useless and is essentially creating 100s of thousands of managed objects and releasing them as fast as possible. It is no surprise that memory usage peaks until GC can deal with it. This is more about managed code and the CLR than about PowerShell. You can do the same thing by writing silly C# code.

Aditya Patwardhan (adityapatwardhan) added the CL-Performance Indicates that a PR should be marked as a performance improvement in the Change Log label Sep 5, 2019
Aditya Patwardhan (adityapatwardhan) merged commit cfdbd71 into PowerShell:master Sep 5, 2019

Copy link
Copy Markdown

🎉v7.0.0-preview.4 has been released which incorporates this pull request.:tada:

Handy links:

Thatgfsj (Thatgfsj) pushed a commit to Thatgfsj/PowerShell that referenced this pull request Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CL-Performance Indicates that a PR should be marked as a performance improvement in the Change Log

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants


Back | FazBrowse Home | New Git URL