Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide by rjurney · Pull Request #578 · graphframes/graphframes · GitHub

What changes were proposed in this pull request?

Why are the changes needed?

…s.txt and split out requirements-dev.txt. Version bumps.

…ney/build-upgrades

+ graphframes core

+ spark should be compatible with GF version

I would point people to installation section of docs

Why not just lit(1)?

It is better to wrap chains to () instead of using line breaks imo. And in previous snippets you were using ()

I was expecting you will highlight the whole API... Early stopping conditions especially. For example, PageRank may be run for a fixed num of iterations, but also until convergence based on tolerance factor. I was thinking you put it to the end, but you didn't. Can you add a couple of words about different early stopping strategies and maybe a simple example?

That is the docs section, jfyi: https://graphframes.io/04-user-guide/10-pregel.html#termination-conditions

Why not F.mode?

And there is a small tricky story :)
Deterministic versus not. To pass LDBC tests you need to run mode with deterministic=True (was added in spark 4.x)

Actually that is the reason why GF implementation does not use mode (tldr we will switch to mode after spark 3.5.x EOL)

…y/pregel-tutorial

…mes into rjurney/pregel-tutorial

…ock results

…ake sure it all works. Update codespell for Sergey Brin.

… problem with pandas/numpy version 2.5.1 being required by Python >= 3.10

…ontext lessons Present the original broken LPA code as a teaching device: explain why Pregel.src() fails outside message expressions, why aggMsgs forbids nested aggregates (mode(collect_list()) -> F.mode(Pregel.msg())), and why withVertexColumn must own the label column. Add verified corrected implementation with real Stack Exchange output and a log2-bucketed text histogram of community sizes with log10-scaled bars.

…-weighted PageRank Replace simulated Id % 3 node types with the graph's real Type column and fix the type weight to use Pregel.src("Type") in the message expression (triplet context). Add human-readable names via coalesce over DisplayName/ Title/TagName/Body, verified output tables, and a new subsection explaining the User teleport-floor result: edges point from users, so rank never flows to them - edge direction determines which node types accumulate importance. Add "suppor" to codespell ignore list: it appears in verbatim truncated Spark output quoted in the tutorial.

…y/pyproject-numpy # Conflicts: # python/poetry.lock # python/pyproject.toml

…mes into rjurney/pregel-tutorial

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…mes into rjurney/pregel-tutorial

Pull request overview

This PR adds end-to-end tutorial material for GraphFrames’ Pregel API and a Neo4j ↔ GraphFrames integration workflow, including new Python tutorial scripts/CLI commands and new documentation pages to guide users through data preparation, loading into Neo4j, running GraphFrames algorithms, and writing results back.

Changes:

Add a new Neo4j integration tutorial (docs + Python tutorial scripts + graphframes neo4j ... CLI).
Add a new Pregel tutorial to the docs and update spellcheck configuration accordingly.
Update Python packaging (Poetry) to include runtime/tutorial dependencies needed by the new tutorial code.

Reviewed changes

Copilot reviewed 13 out of 18 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
python/tests/test_neo4j_tutorial.py	Adds a Neo4j tutorial “validation” script (currently placed under tests).
python/pyproject.toml	Adds/updates Python dependencies for runtime and tutorial/dev groups.
python/poetry.lock	Updates locked Python dependency set to match pyproject changes.
python/graphframes/tutorials/stackexchange.py	Adjusts data path resolution and writes additional Parquet outputs for Neo4j loading.
python/graphframes/tutorials/neo4j/loaders.py	Adds APOC Parquet loading helpers and loader registry for node/edge types.
python/graphframes/tutorials/neo4j/load.py	Adds a Spark-based Neo4j loading script (connector-based alternative).
python/graphframes/tutorials/neo4j/docker.py	Adds helpers to start/stop/manage a Neo4j Docker container configured for APOC.
python/graphframes/tutorials/neo4j/connected_components.py	Adds a script to read from Neo4j, run Connected Components, and write results back.
python/graphframes/tutorials/neo4j/cli.py	Adds graphframes neo4j CLI group for container management, loading, and status.
python/graphframes/tutorials/neo4j/init.py	Exposes the Neo4j CLI entry point from the tutorials package.
python/graphframes/console.py	Registers the new Neo4j CLI command in the main graphframes CLI.
docs/src/03-tutorials/04-neo4j-integration.md	Adds detailed Neo4j integration tutorial documentation.
docs/src/03-tutorials/03-pregel-tutorial.md	Adds a comprehensive Pregel tutorial page.
.codespellrc	Updates codespell ignore list to accommodate tutorial text.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

rjurney added 30 commits February 15, 2025 20:58

Converted tests to pytest. Build a Python package. Update requirement…

f4e9cdb

…s.txt and split out requirements-dev.txt. Version bumps.

Restore Python .gitignore

c256244

Extra newline removed

6c3df0b

Merge branch 'master' of github.com:graphframes/graphframes into rjur…

b2838d2

…ney/build-upgrades

Added VERSION file set to 0.8.5

caf5091

isort; fiex edgesDF variable name.

7cfa2d1

Merge branch 'master' of github.com:graphframes/graphframes into rjur…

2ca9a15

…ney/build-upgrades

Back out Dockerfile changes

a8bf0be

Back out version change in build.sbt

54a942d

Backout changes to config and run-tests

8b0e346

Back out pytest conversion

46c2b93

Back out version changes to make nose tests pass

18b5da0

Remove changes to requirements

8eca097

Put nose back in requirements.txt

277c06f

Remove version bump to version.sbt

b55ee48

Remove packages related to testing

f8a8fd9

Remove old setup.py / setup.cfg

bc2cb36

New pyproject.toml and poetry.lock

728be33

Short README for Python package, poetry won't allow a ../README.md path

3cea1a8

Remove requirements files in favor of pyproject.toml

87cc975

Try to poetrize CI build

6f84a5a

pyspark min 3.4

9a8eef0

Local python README in pyproject.toml

75ecd99

Trying to remove he working folder to debug scala issue

80231d0

Set Python working directory again

2a9170b

Accidental newline

3de2263

Install Python for test...

4662717

Run tests from python/ folder

1b7b9f8

Try running tests from python/

58da493

poetry run the unit tests

9f4aa24

SemyonSinchenko reviewed Dec 10, 2025

View reviewed changes

rjurney added 4 commits December 16, 2025 21:14

Merge branch 'graphframes:main' into rjurney/pregel-tutorial

95c34a9

Merge branch 'main' of github.com:graphframes/graphframes into rjurne…

b5350f2

…y/pregel-tutorial

Lots of work making a Neo4j integration tutorial work.

7d9a82f

Merge branch 'rjurney/pregel-tutorial' of github.com:rjurney/graphfra…

02fc409

…mes into rjurney/pregel-tutorial

rjurney changed the title Create a Pregel tutorial and add Pregel API to user guide Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide Dec 25, 2025

rjurney added 11 commits July 23, 2026 14:15

Merged main, ajusted pyproject increase of neo4j version and poetry l…

a84d0d9

…ock results

Updates to latest version in code and doing complete run-through to m…

efe5a20

…ake sure it all works. Update codespell for Sergey Brin.

Update pyproject.toml to add pre-commit as a dev dependency and fix a…

80f3afd

… problem with pandas/numpy version 2.5.1 being required by Python >= 3.10

git pull rjurney/pyproject-numpy # And resolve conflicts

6fc828f

Merge branch 'main' of github.com:graphframes/graphframes into rjurne…

ee7ba1d

…y/pyproject-numpy # Conflicts: # python/poetry.lock # python/pyproject.toml

Merge and fix changes

7eec98e

Add grpcio ^1.83.0 as a dev dependency and relock

04ba023

Removed grpcio

82a4339

Merge branch 'rjurney/pyproject-numpy' of github.com:rjurney/graphfra…

24322a3

…mes into rjurney/pregel-tutorial

rjurney mentioned this pull request Jul 28, 2026

Add pre-commit and Python >= 3.12 numpy dependencies to Poetry #877

Closed

rjurney and others added 4 commits July 28, 2026 13:59

Potential fix for pull request finding

38da664

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Merge branch 'rjurney/pyproject-numpy' of github.com:rjurney/graphfra…

0f74041

…mes into rjurney/pregel-tutorial

New poetry lock from previous commit

e8ed209

Merge poetry.lock

d17a2db

rjurney mentioned this pull request Jul 28, 2026

doc: Pregel Tutorial #809

Open

Fix stale Spark 3.x reference

cfae3ec

rjurney requested a review from Copilot July 28, 2026 21:58

Copilot started reviewing on behalf of rjurney July 28, 2026 21:58 View session

Bump version required for this tutorial to 0.11.0

2ed3e88

Copilot AI reviewed Jul 28, 2026

View reviewed changes

+pre-commit 4.6.1 dev dependency

dbfe9a3

+                      F.coalesce(Pregel.msg(), Pregel.src("label"))) \
+                  .sendMsgToDst(Pregel.src("label")) \
+                  .sendMsgToSrc(Pregel.dst("label")) \
+                  .aggMsgs(F.expr("mode(collect_list(msg))")) \

+              # Compute the default data directory relative to this package
+              _PACKAGE_DIR = Path(__file__).parent.parent  # tutorials/
+              DEFAULT_DATA_DIR = str(_PACKAGE_DIR / "data")
+              from graphframes.tutorials.neo4j.docker import (
+                  CONTAINER_NAME,
+                  DEFAULT_PASSWORD,
+                  check_docker,
+                  container_exists,
+                  remove_container,
+                  start_container,
+                  wait_for_ready,
+              )
+              from graphframes.tutorials.neo4j.loaders import (
+                  EDGE_LOADERS,
+                  NODE_LOADERS,
+                  clear_database,
+                  create_indexes,
+                  get_parquet_file_uris,
+              )

+              """
+              Test script for Neo4j integration tutorial.
+              Tests the data processing logic without requiring Neo4j.
+              """
+              import pyspark.sql.functions as F
+              from graphframes import GraphFrame
+              from pyspark.sql import SparkSession, DataFrame
+              print("=" * 80)
+              print("NEO4J TUTORIAL - CODE VALIDATION")
+              print("=" * 80)

+                  spark-submit \\
+                    --packages org.neo4j:neo4j-connector-apache-spark_2.12:5.3.1_for_spark_3 \\
+                    --driver-memory 4g \\
+                    --executor-memory 4g \\
+                    python/graphframes/tutorials/neo4j_load.py

+                  spark-submit \\
+                    --packages org.neo4j:neo4j-connector-apache-spark_2.12:5.3.1_for_spark_3 \\
+                    --driver-memory 4g \\
+                    --executor-memory 4g \\
+                    python/graphframes/tutorials/neo4j_connected_components.py

Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide by rjurney · Pull Request #578 · graphframes/graphframes · GitHub

Uh oh!

Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide - #578

Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide#578
rjurney wants to merge 107 commits into
graphframes:mainfrom
rjurney:rjurney/pregel-tutorial

rjurney commented Apr 15, 2025

Uh oh!

SemyonSinchenko Dec 10, 2025 •
edited
Loading

Uh oh!

Uh oh!

SemyonSinchenko Dec 10, 2025 •
edited
Loading

Uh oh!

Uh oh!

SemyonSinchenko Dec 10, 2025

Uh oh!

SemyonSinchenko Dec 10, 2025

Uh oh!

SemyonSinchenko Dec 10, 2025

Uh oh!

SemyonSinchenko Dec 10, 2025

Uh oh!

SemyonSinchenko Dec 10, 2025

Uh oh!

SemyonSinchenko Dec 10, 2025

Uh oh!

SemyonSinchenko Dec 10, 2025

Uh oh!

SemyonSinchenko Dec 10, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		Before starting this tutorial, ensure you have:

		- GraphFrames installed: `pip install graphframes-py`

+              Before starting this tutorial, ensure you have:
+              - **GraphFrames installed**: `pip install graphframes-py`
+              - **Apache Spark 3.x**: Compatible with your Python version

                   </figure>
               </center>
+              ## Prerequisites

+              # Create a GraphFrame to get access to AggregateMessages API
+              g: GraphFrame = GraphFrame(nodes_df, edges_df)
+              msgToDst = AM.src["start_degree"]

+              g_simple = GraphFrame(vertices_simple, edges_simple)
+              # Calculate in-degree using Pregel API
+              pregel_result = g_simple.pregel \

+              # +---+-------+---------+
+              ```
+              ### Understanding the Pregel API

		@@ -10,6 +11,7 @@ def cli():


		cli.add_command(download.stackexchange)
		cli.add_command(neo4j)

+              |                                              name|            pagerank|
+              +--------------------------------------------------+--------------------+
+              |                          TeX processing for Stats|  0.1450891970426848|
+              |What typographic support is available to suppor...| 0.14112968883288293|

+              #   brin     - Sergey Brin, co-founder of Google (PageRank co-inventor)
+              #   suppor   - truncated word in verbatim Spark output in the Pregel tutorial
+              ignore-words-list = mis,te,ser,fof,mye,protocall,alledges,afterall,jurney,brin,suppor

Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide by rjurney · Pull Request #578 · graphframes/graphframes · GitHub

Uh oh!

Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide - #578

Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide#578rjurney wants to merge 107 commits intographframes:maingraphframes/graphframes:mainfrom rjurney:rjurney/pregel-tutorialrjurney/graphframes:rjurney/pregel-tutorialCopy head branch name to clipboard

Conversation

rjurney commented Apr 15, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Uh oh!

SemyonSinchenko Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SemyonSinchenko Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide#578
rjurney wants to merge 107 commits into
graphframes:mainfrom
rjurney:rjurney/pregel-tutorial

SemyonSinchenko Dec 10, 2025 •
edited
Loading

SemyonSinchenko Dec 10, 2025 •
edited
Loading