FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

doc: Pregel Tutorial by rjurney · Pull Request #809 · graphframes/graphframes · GitHub

doc: Pregel Tutorial - #809

Open
rjurney wants to merge 163 commits into
graphframes:mainfrom
rjurney:rjurney/pypi-tutorials
Open

doc: Pregel Tutorial#809
rjurney wants to merge 163 commits into
graphframes:mainfrom
rjurney:rjurney/pypi-tutorials

Conversation

rjurney commented Mar 13, 2026

Copy link
Copy Markdown
Collaborator
  1. New, big, fancy, super, duper Pregel tutorial
  2. Moved Stack Exchange data content from Network Motif Finding Tutorial into Data Setup tutorial. Refer to from both motif and Pregel tutorials.
  3. Point at new tutorial(s) from list of tutorials.
  4. New network motif and Pregel tutorial Jupyter notebooks
  5. Some other minor changes...

…s.txt and split out requirements-dev.txt. Version bumps.
rjurney added 14 commits July 30, 2026 13:56
Add a 'Choosing a Termination Condition' subsection to 'How Fast Does It
Converge?' in the Pregel tutorial: a table of the three conditions, the
cost of the two dynamic ones (each adds a Spark action per superstep),
and per-algorithm guidance for when each applies.

Also fix the user guide's vertex-voting bullet, which named
stopIfAllNonActiveVertices (the actual API is
setStopIfAllNonActiveVertices) and had an unclosed paren.
Switch generate_diagrams.py from the PhantomJS-based mmdc converter to
mermaidx, which bundles its own JS engine and needs no browser or Node
install. The new subgraph syntax separates the node id from the title, so
restore the G numbers in the graphlets overview titles ("G1 single directed
edge") that the bare-title syntax had provided implicitly.

Regenerate all motif and Pregel diagrams with the new renderer, and wire the
SVG versions into the motif tutorial in place of the older PNG figures.
Move the SparkSession setup, path resolution, and the repartition/checkpoint/
cache load of Nodes.parquet and Edges.parquet into 03-data-setup.md, which
now ends with nodes_df, edges_df, and g ready to use. The motif and Pregel
tutorials drop their duplicated copies of that boilerplate and state up front
that they continue from those objects, keeping only the imports each one adds.

Correct the prerequisites while here: Java 21 and Spark 4.1.3, matching the
versions the tutorials were actually written against.
Add a stat.corr call to the reputation propagation example and report the
result: authority and ViewCount correlate at only 0.07, so state that number
rather than implying the two usually track each other. Widen the truncate
width so the question titles are readable, and bold the point that joins grow
quadratically per hop while Pregel just needs a larger setMaxIter.
Replace the "the implementation would use..." hand-wave with a working
average-answer-score-per-tag algorithm, in both the tutorial and pregel.py.
It reverses the Tags edges so both hops travel with sendMsgToDst, sends a
struct carrying score and count together, and unpacks it with two
withVertexColumn definitions - which also demonstrates the vertex-vs-triplet
expression contexts.

Add two diagrams: one mapping each design question to its Pregel paper
concept and GraphFrames API call, one tracing scores from Answer to Question
to Tag. Note in the text that the four questions restate the components of
the original Pregel model, and add the vertex-centric literature behind that
framing to Further Reading - the McCune survey, Tian et al. on "think like a
vertex", and Colyer's walkthrough of the Pregel paper.

Switch figcaptions to white for legibility on the dark docs theme.
…rial assets in black and isort. Add mispelling exceptions.
Comment thread python/pyproject.toml
[tool.black]
line-length = 100
target-version = ["py39"]
include = ["graphframes"]

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

The default behavior is to just check Python files. This configuration cleared that and had it checking non-Python files from the tutorial's data.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 25 out of 47 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

python/graphframes/tutorials/download.py:21

  • DEFAULT_DATA_DIR defaults to a path inside the installed package (Path(__file__).parent / "data"). In many environments site-packages is read-only, so graphframes stackexchange ... can fail with a permissions error unless users always pass --data-dir.
    .pre-commit-config.yaml:40
  • The codespell hook exclude regex references python/tutorials/, but this repository’s tutorials live under python/graphframes/tutorials/ (there is no python/tutorials/ directory). As a result, codespell will not skip the intended tutorial content.
        exclude: ^(graphx|python/tutorials/)

docs/src/03-tutorials/03-data-setup.md:129

  • This comment says the spark.sql.caseSensitive setting is needed to let Id and id coexist, but the tutorial data generation script renames Id to StackId and drops Id. The comment is misleading for readers following the documented workflow.
# Lets the Id:(Stack Overflow int) and id:(GraphFrames ULID) coexist
spark.conf.set("spark.sql.caseSensitive", True)

Comment on lines 1 to 16
import click

from graphframes.tutorials import download


@click.group()
def cli():
"""GraphFrames CLI: a collection of commands for graphframes."""
pass


cli.add_command(download.stackexchange)


def main():
# Lazy-import tutorials.download to avoid requiring py7zr/requests
# at import time — those are only in the 'tutorials' optional extra.
from graphframes.tutorials import download

cli.add_command(download.stackexchange)
cli()
rjurney added 4 commits August 6, 2026 10:07
Clarified the purpose of Pregel in relation to built-in algorithms and emphasized its general-purpose nature for iterative graph algorithms.
Removed unnecessary line break and improved clarity on Pregel's synchronization and vertex function paradigm.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 24 out of 46 changed files in this pull request and generated no new comments.

Suppressed comments (7)

docs/src/helium/custom.css:16

  • figcaption is forced to white (#fff). With Helium’s default light theme, this makes captions unreadable and hurts contrast/accessibility. Prefer inheriting the theme’s text color (or using a theme variable) instead of hard-coding white.
figcaption {
    display: block;
    text-align: center;
    font-size: 0.875em;
    color: #fff;
    font-style: italic;

python/graphframes/tutorials/pregel.py:15

  • The Spark 3.5.x usage examples hard-code GraphFrames 0.11.0, but other tutorial docs use 0.12.1. Align the version string to avoid confusion.
    python/graphframes/tutorials/motif.py:12
  • These header comments hard-code GraphFrames 0.11.0, but the Data Setup tutorial examples use 0.12.1. Align the version so readers don’t mix artifacts.
    .pre-commit-config.yaml:40
  • The codespell hook excludes python/tutorials/, but this repository’s tutorial code lives under python/graphframes/tutorials/ (so the exclude currently matches nothing and won’t prevent codespell noise in the intended directory). Update the regex to the correct path.
      - id: codespell
        exclude: ^(graphx|python/tutorials/)

python/graphframes/tutorials/pregel.py:10

  • The Spark 4.x usage examples hard-code GraphFrames 0.11.0, but the Data Setup tutorial uses 0.12.1. Keeping these in sync avoids users mixing incompatible versions.

This issue also appears on line 12 of the same file.
python/graphframes/tutorials/motif.py:7

  • These header comments hard-code GraphFrames 0.11.0, but the Data Setup tutorial examples use 0.12.1. Align the version so readers don’t mix artifacts.

This issue also appears on line 9 of the same file.
docs/src/03-tutorials/02-motif-tutorial.md:41

  • This tutorial command uses GraphFrames 0.11.0, while the Data Setup tutorial uses 0.12.1. Align the version string so readers don’t accidentally run mismatched artifacts.
spark-submit --packages io.graphframes:graphframes-spark4_2.13:0.11.0 python/graphframes/tutorials/motif.py

SemyonSinchenko left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

~LGTM overall from the context point of view.

It looks like adding notebooks to the python/graphframes breaks the build:

ValueError: '/var/home/sem/github/graphframes/docs/src/img/4-node-directed-graphlets.png' is not in the subpath of '/var/home/sem/github/graphframes/python' OR one path is relative and the other is absolute.

The only way to fix it I see is moving all the ipynb away from the python/graphframes. It makes sense because we barely wants to publish notebooks as a part of the sdist / wheel

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants


Back | FazBrowse Home | New Git URL