| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
…s.txt and split out requirements-dev.txt. Version bumps.
…ney/build-upgrades
…ney/build-upgrades
Add a 'Choosing a Termination Condition' subsection to 'How Fast Does It Converge?' in the Pregel tutorial: a table of the three conditions, the cost of the two dynamic ones (each adds a Spark action per superstep), and per-algorithm guidance for when each applies. Also fix the user guide's vertex-voting bullet, which named stopIfAllNonActiveVertices (the actual API is setStopIfAllNonActiveVertices) and had an unclosed paren.
Switch generate_diagrams.py from the PhantomJS-based mmdc converter to
mermaidx, which bundles its own JS engine and needs no browser or Node
install. The new subgraph syntax separates the node id from the title, so
restore the G numbers in the graphlets overview titles ("G1 single directed
edge") that the bare-title syntax had provided implicitly.
Regenerate all motif and Pregel diagrams with the new renderer, and wire the
SVG versions into the motif tutorial in place of the older PNG figures.
Move the SparkSession setup, path resolution, and the repartition/checkpoint/ cache load of Nodes.parquet and Edges.parquet into 03-data-setup.md, which now ends with nodes_df, edges_df, and g ready to use. The motif and Pregel tutorials drop their duplicated copies of that boilerplate and state up front that they continue from those objects, keeping only the imports each one adds. Correct the prerequisites while here: Java 21 and Spark 4.1.3, matching the versions the tutorials were actually written against.
Add a stat.corr call to the reputation propagation example and report the result: authority and ViewCount correlate at only 0.07, so state that number rather than implying the two usually track each other. Widen the truncate width so the question titles are readable, and bold the point that joins grow quadratically per hop while Pregel just needs a larger setMaxIter.
Replace the "the implementation would use..." hand-wave with a working average-answer-score-per-tag algorithm, in both the tutorial and pregel.py. It reverses the Tags edges so both hops travel with sendMsgToDst, sends a struct carrying score and count together, and unpacks it with two withVertexColumn definitions - which also demonstrates the vertex-vs-triplet expression contexts. Add two diagrams: one mapping each design question to its Pregel paper concept and GraphFrames API call, one tracing scores from Answer to Question to Tag. Note in the text that the four questions restate the components of the original Pregel model, and add the vertex-centric literature behind that framing to Further Reading - the McCune survey, Tian et al. on "think like a vertex", and Colyer's walkthrough of the Pregel paper. Switch figcaptions to white for legibility on the dark docs theme.
…rial assets in black and isort. Add mispelling exceptions.
| [tool.black] | ||
| line-length = 100 | ||
| target-version = ["py39"] | ||
| include = ["graphframes"] |
There was a problem hiding this comment.
The default behavior is to just check Python files. This configuration cleared that and had it checking non-Python files from the tutorial's data.
Sorry, something went wrong.
There was a problem hiding this comment.
Copilot reviewed 25 out of 47 changed files in this pull request and generated 1 comment.
Suppressed comments (3)python/graphframes/tutorials/download.py:21
exclude: ^(graphx|python/tutorials/)
docs/src/03-tutorials/03-data-setup.md:129
# Lets the Id:(Stack Overflow int) and id:(GraphFrames ULID) coexist
spark.conf.set("spark.sql.caseSensitive", True)
Sorry, something went wrong.
| import click | ||
|
|
||
| from graphframes.tutorials import download | ||
|
|
||
|
|
||
| @click.group() | ||
| def cli(): | ||
| """GraphFrames CLI: a collection of commands for graphframes.""" | ||
| pass | ||
|
|
||
|
|
||
| cli.add_command(download.stackexchange) | ||
|
|
||
|
|
||
| def main(): | ||
| # Lazy-import tutorials.download to avoid requiring py7zr/requests | ||
| # at import time — those are only in the 'tutorials' optional extra. | ||
| from graphframes.tutorials import download | ||
|
|
||
| cli.add_command(download.stackexchange) | ||
| cli() |
Clarified the purpose of Pregel in relation to built-in algorithms and emphasized its general-purpose nature for iterative graph algorithms.
Removed unnecessary line break and improved clarity on Pregel's synchronization and vertex function paradigm.
There was a problem hiding this comment.
Copilot reviewed 24 out of 46 changed files in this pull request and generated no new comments.
Suppressed comments (7)docs/src/helium/custom.css:16
figcaption {
display: block;
text-align: center;
font-size: 0.875em;
color: #fff;
font-style: italic;
python/graphframes/tutorials/pregel.py:15
- id: codespell
exclude: ^(graphx|python/tutorials/)
python/graphframes/tutorials/pregel.py:10
This issue also appears on line 12 of the same file.
python/graphframes/tutorials/motif.py:7
This issue also appears on line 9 of the same file.
docs/src/03-tutorials/02-motif-tutorial.md:41
spark-submit --packages io.graphframes:graphframes-spark4_2.13:0.11.0 python/graphframes/tutorials/motif.py
Sorry, something went wrong.
There was a problem hiding this comment.
~LGTM overall from the context point of view.
It looks like adding notebooks to the python/graphframes breaks the build:
ValueError: '/var/home/sem/github/graphframes/docs/src/img/4-node-directed-graphlets.png' is not in the subpath of '/var/home/sem/github/graphframes/python' OR one path is relative and the other is absolute.
The only way to fix it I see is moving all the ipynb away from the python/graphframes. It makes sense because we barely wants to publish notebooks as a part of the sdist / wheel
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.