FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat: pregel memory optimizations by SemyonSinchenko · Pull Request #847 · graphframes/graphframes · GitHub

/ graphframes Public

feat: pregel memory optimizations - #847

Merged
SemyonSinchenko merged 5 commits into
graphframes:mainfrom
SemyonSinchenko:835-required-edge-columns
Jun 11, 2026
ConversationCommits (5)ChecksFiles changed
Merged

feat: pregel memory optimizations#847
SemyonSinchenko merged 5 commits into
graphframes:mainfrom
SemyonSinchenko:835-required-edge-columns

Conversation

SemyonSinchenko commented Jun 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

Edge columns are not persisted by default anymore.

Why are the changes needed?

x2 less peak memory usage + small benefit on local benchmarks:

This PR

[info] Result "org.graphframes.benchmarks.LabelPropagationBenchmark.benchmarkLabelPropagation":
[info]   40.889 ±(99.9%) 14.652 s/op [Average]
[info]   (min, avg, max) = (39.969, 40.889, 41.450), stdev = 0.803
[info]   CI (99.9%): [26.237, 55.540] (assumes normal distribution)

Main

[info] Result "org.graphframes.benchmarks.LabelPropagationBenchmark.benchmarkLabelPropagation":
[info]   41.504 ±(99.9%) 12.715 s/op [Average]
[info]   (min, avg, max) = (40.712, 41.504, 42.025), stdev = 0.697
[info]   CI (99.9%): [28.789, 54.219] (assumes normal distribution)

Close #835

P.S. I do not see this as a breaking change. It was not documented anywhere that edges are persisted by default. I see it as a small change that will benefit 99% of users and only ~1% that used edge-columns in Pregel should do a small update.

SemyonSinchenko commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

PySpark bindings and docs update after pre-approve

SemyonSinchenko self-assigned this Jun 9, 2026
SemyonSinchenko merged commit 10ecd74 into graphframes:main Jun 11, 2026
8 checks passed
SemyonSinchenko deleted the 835-required-edge-columns branch June 11, 2026 21:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

james-willis james-willis approved these changes

Assignees

SemyonSinchenko

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat: requiredEdgeColumns API in Pregel

2 participants

Back | FazBrowse Home | New Git URL