| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
Benchmarked on a generated 20,000-row x 40-col xlsx (mixed shared-string, numeric, and empty cells). Median full-parse time dropped ~1.68x (1.85s -> 1.11s) and allocations dropped ~3x (28.6M -> ~9M), with byte-identical output verified on both default- and prefixed-namespace files across rows, simple_rows, and rows_with_meta_data. Changes, in order of impact: - Resolve the namespace prefix once via a `namespace_resolved` flag instead of re-checking `node.namespaces` on every node. Worksheets use a default namespace, so the old `prefix.empty?` guard never latched and allocated a namespaces hash for every node in the stream. This is the bulk of the wall-clock win. - Hoist `node.name`/`node.node_type` into locals (each was read up to 4x per node in the if/elsif chain) and reorder branches so the hottest nodes (<v>/<c>) are tested first. - Use `node.attribute_hash` instead of `node.attributes` for cell and row nodes. `Reader#attributes` is `attribute_hash.merge(namespaces)`, so it built and merged a namespaces hash on every call; we only need the element's own attributes. This is an allocation/GC win (drops Hash#merge and the per-node namespaces hash) with negligible wall-clock change. Safe because xlsx row/cell elements never declare their own namespaces (the namespace hash is empty for them), so the result is identical -- confirmed byte-for-byte including the self-closing-row + meta-data path. - In fill_in_empty_cells, drop the redundant `.to_a` on the column range and use `delete_suffix` instead of `gsub` to strip the row number. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the rows_generator optimization. Fetch the three cell
attributes with individual node.attribute('t'/'s'/'r') C calls instead of
building a per-cell attribute_hash and indexing it. With hundreds of
thousands of cells the per-cell Hash allocation dominates, so three cheap
lookups are both faster and leaner.
Same 20,000-row x 40-col benchmark, vs the attribute_hash version:
| Cell-attr approach | Median | Allocations |
|------------------------|---------|-------------|
| node.attribute_hash | ~1.10s | 9,226,711 |
| node.attribute() x3 | ~1.08s | 7,626,711 |
=> ~3-4% faster, 1.6M (~17%) fewer allocations (deterministic: ~640K
cells x the per-cell hash no longer built).
Output byte-identical across rows, simple_rows, and rows_with_meta_data on
both default- and prefixed-namespace files. Suite green (47 examples).
The row-opener node keeps attribute_hash (only ~one per row, and it
preserves the exact meta-data row hash).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`Reader#namespaces` and `Reader#attributes` both go through `xmlTextReaderExpand()`, which materialises the subtree under the current node. The prefix detection runs on every node until it latches, so it runs on the worksheet root -- where "the subtree under the current node" is the entire document. One call therefore builds a DOM of the whole worksheet and the streaming parse is lost before the first row is yielded. The cost is ~25x the size of the sheet XML, in native memory: 1.9 MB -> 79 MB, 7.4 MB -> 220 MB, 22.7 MB -> 592 MB, 44.8 MB -> 1120 MB. Being native, it appears in neither `GC.stat` nor allocation profiles, which is why it survived several rounds of allocation-focused tuning: on a 44.8 MB sheet, pythonicrubyist#138 alone cuts wall clock in half but leaves RSS at 1149 MB, and pythonicrubyist#132 brings RSS down to 113 MB but doubles wall clock by calling `namespace_uri` on every node and keeping `attributes` on every cell. `Reader#prefix` and `Reader#namespace_uri` map to `xmlTextReaderConstPrefix` and `xmlTextReaderConstNamespaceUri`, which read the current node only. Taking the prefix from the first node that is *in* the spreadsheetml namespace, rather than from the namespaces declared *on* it, also handles a worksheet whose root is in another namespace while its rows are prefixed. Measured on a real 57895-row FEC export (44.8 MB of sheet XML, 5 MB zipped), on top of pythonicrubyist#138: 1149 MB -> 113 MB RSS and 13.2 s -> 8.7 s, with byte-identical rows. Claude-Session: https://claude.ai/code/session_01BNG66yvb43CB5x3mAWY64h
| Back | FazBrowse Home | New Git URL |
After noticing that this method uses a lot of memory/takes a lot of time in the Rails app at work, I used Claude to optimize this and then verify that the output was identical from the released gem vs this change on a number of different files.
This should hopefully help resolve some long-standing issues with the performance/memory usage of the gem.
Benchmarked on a generated 20,000-row x 40-col xlsx (mixed shared-string, numeric, and empty cells). Median full-parse time dropped ~1.72x (1.85s -> ~1.08s) and allocations dropped ~3.75x (28.6M -> 7.63M), with byte-identical output verified on both default- and prefixed-namespace files across rows, simple_rows, and rows_with_meta_data (including the self-closing-row + meta-data path).
Changes, in order of impact: