FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

[codex] Parallelize incremental parse step by Extrodox · Pull Request #1052 · CodeGraphContext/CodeGraphContext · GitHub

[codex] Parallelize incremental parse step - #1052

Draft
Extrodox wants to merge 3 commits into
CodeGraphContext:mainfrom
Extrodox:codex/incremental-index-parallel-parse
Draft

[codex] Parallelize incremental parse step#1052
Extrodox wants to merge 3 commits into
CodeGraphContext:mainfrom
Extrodox:codex/incremental-index-parallel-parse

Conversation

Extrodox commented Jun 2, 2026

Copy link
Copy Markdown

Summary

  • Reuse the changed file parse result from update_file_in_graph() instead of parsing the same file again during incremental relinking.
  • Parse remaining affected files with a bounded ThreadPoolExecutor driven by PARALLEL_WORKERS.
  • Keep graph writes and relationship relinking serialized to avoid backend lock contention.
  • Add regression coverage for the worker-count config and changed-file parse reuse.

Why

Incremental indexing was effectively serial for affected-file parsing, and the changed file could be parsed twice in one update pass. This makes large affected sets slower than necessary even when parsing work is CPU-bound and independent per file.

Validation

  • PYTHONPATH=src /tmp/cgc-pr-venv/bin/python -m pytest tests/unit/tools/test_graph_builder_perf_fixes.py::TestWatcherIncrementalHandleModification::test_incremental_parse_workers_uses_parallel_workers_config tests/unit/tools/test_graph_builder_perf_fixes.py::TestWatcherIncrementalHandleModification::test_handle_modification_reuses_changed_file_parse_result -q
  • PYTHONPATH=src /tmp/cgc-pr-venv/bin/python -m compileall -q src/codegraphcontext/core/watcher.py

Note: the default Homebrew Python in this shell is externally managed and lacked project dependencies, so validation used a temporary venv under /tmp/cgc-pr-venv.

vercel Bot commented Jun 2, 2026

Copy link
Copy Markdown

@Extrodox is attempting to deploy a commit to the shashankss1205's projects Team on Vercel.

A member of the Team first needs to authorize it.

Copy link
Copy Markdown
Collaborator

Overlaps merged #1071 watcher sync and conflicts with current watcher.py.

# Conflicts:
#	src/codegraphcontext/core/watcher.py
main added _update_lock/_timers_lock after this branch was cut (CodeGraphContext#1424,
serialising concurrent watcher updates). Tests that build the handler via
__new__ must set them, as the neighbouring tests in this file already do.

Copy link
Copy Markdown
Collaborator

Rebased onto current main and pushed to your branch — it was 64 days stale and watcher.py has had 15 commits since you branched from v0.4.13. Suite is green (1111 passed, 7 skipped). Two things I had to resolve:

  1. The conflict in _handle_modification. Kept your side — capturing the return of update_file_in_graph to reuse the parse. I checked that still works: graph_builder.py:1015/1019 returns file_data, so the reuse optimisation is valid against current main.
  2. Your new test failed with AttributeError: 'RepositoryEventHandler' object has no attribute '_update_lock'. main added _update_lock/_timers_lock in fix(watcher): handle both endpoints of a rename; serialise concurrent updates #1424 after you branched, and tests building the handler via __new__ must supply them — the neighbouring tests in that file already do. Added the same two lines.

Why I have not merged it

The parallel parsing hits an open segfault bug — #1370.

That issue reports non-deterministic Fatal Python error: Segmentation fault when multiple worker threads call tree_sitter_manager.execute_query() concurrently on different languages. Its traceback shows three threads inside execute_query() at once (TypeScript, Python, HTML).

Your ThreadPoolExecutor runs graph_builder.parse_file across affected files, and in a mixed-language repo the affected set spans languages — the exact condition. And execute_query is still unsynchronised on main:

# utils/tree_sitter_manager.py:322
def execute_query(language, query_string, node):
    query = Query(language, query_string)      # per call, no lock
    cursor = QueryCursor(query)
    res = cursor.captures(node)

tree_sitter_manager locks its language cache (_cache_lock) but nothing guards query execution.

So merging this would move a known crash from cgc index into cgc watch, where it is worse: watch runs unattended for hours, and a segfault takes the whole process down mid-session.

What would unblock it

Either is fine, and either would be a genuinely valuable PR on its own:

The rest of the PR is good — reusing the already-parsed changed file is a clear win regardless, and if you wanted to split that out as its own PR it would merge immediately. Keeping this open as a draft.

Copy link
Copy Markdown
Collaborator

#1370 is now fixed#1593 merged, adding a process-wide lock around tree-sitter query construction and execution.

That was the blocker on this PR. Measured cost of the lock is ~1% (3.48s vs 3.45s median over 120 parses on 8 threads), because only the query step is serialised — parser.parse and all Python-level work stay parallel, so your ThreadPoolExecutor still buys what it was meant to buy.

One caveat I want to be straight about: I could not reproduce the segfault locally, so #1593 is a guard against the reported crash rather than one I demonstrated. It is not proof your parallelism is now safe — it removes the specific unsynchronised path the report pointed at.

Your branch is already rebased onto main and green. Since it is still marked draft, I have left it for you to mark ready. Worth re-running your own workload against it first if you have the mixed-language repo that motivated the change.

Copy link
Copy Markdown
Collaborator

This is still marked as a draft, so I've left it alone — flagging that it's now conflict-free against main if you'd like to mark it ready for review. Parallelising the incremental parse step is worth having; happy to review once you take it out of draft.

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog tasks

Development

Successfully merging this pull request may close these issues.

2 participants


Back | FazBrowse Home | New Git URL