Permalink
Comparing changes
Choose two branches to see what’s changed or to start a new pull request.
If you need to, you can also or
learn more about diff comparisons.
Open a pull request
Create a new pull request by comparing changes across two branches. If you need to, you can also .
Learn more about diff comparisons here.
base repository:
culturecode/spatial_features
base:
master
...
head repository:
culturecode/spatial_features
compare:
batch-feature-saves
Checking mergeability…
Don’t worry, you can still create the pull request.
- 5 commits
- 11 files changed
- 2 contributors
Commits on Aug 18, 2026
-
perf: Resolve a KML geometry's Placemark by walking its parents
`each_record` read a geometry's name and metadata from `feature.ancestors('Placemark').first`. `Nokogiri::XML::Node#ancestors` answers a selector by walking to the document root and searching the whole document from there, then scanning the results for each ancestor, and it caches nothing. Called once per geometry, that costs geometries x document size. A KML where each Placemark holds a single geometry never shows it; one exported from design software, where a Placemark is a `<MultiGeometry>` of hundreds of faces, turns a few thousand Placemarks into six figures of geometries and spends minutes there. Instead of asking Nokogiri for the ancestors matching a selector, `enclosing_placemark` walks the parent chain and stops at the first Placemark. The answer is the same, since `ancestors` returns matches nearest-first and only the first was ever read. On documents built by repeating the `test.kml` fixture's Placemark, 20,000 geometries go from 14.44s to 0.0155s. The walk stops at whatever no longer responds to `parent`, which is how a geometry with no Placemark ancestor returns nil. `Nokogiri::XML::Document` is that node, and it responds to `name` but not `parent`. Adds `kml_file_with_multi_geometry_placemarks.kml`, built by `fixtures:build` from the same squares as the other KML fixtures, covering that every part of a MultiGeometry takes the name and metadata of the Placemark holding it. Its fourth example asserts the document is not searched again once parsed, which fails against the previous implementation. Closes #66 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Configuration menuCopy the full SHA5eb9091 View commit details -
perf: Read a Placemark's geometry from the Placemark down
Reading a geometry's Placemark, however cheaply, is work repeated for every geometry, and it makes the Placemark's metadata repeated work too: `extract_metadata` parses the CDATA table in a `<description>` into a hash once per geometry, so a Placemark holding several hundred parts parsed the same description several hundred times. Instead of walking up from each geometry to its Placemark, `each_record` now starts at each Placemark and reads the geometry below it. The Placemark's metadata, name and image paths are read once and shared by its parts, and no geometry ever asks what it belongs to. Measured on a 33.8 MB CAD-derived KML holding 76,190 geometries in 1,233 Placemarks, with the per-geometry database round trip stubbed out so the figures cover the traversal alone: ancestors + metadata per geometry 902.02s (before this branch) parent walk + metadata per geometry 10.76s (previous commit) Placemark-first 2.86s Eliminating the upward search is worth 891s of that and eliminating the repeated metadata 7.9s, so the second is small in absolute terms while still being most of what was left. Geometry outside any Placemark still imports with no name and no metadata, which iterating Placemarks alone would silently drop. A second pass matches it with one XPath over the document rather than by asking each element for its ancestors. `kmz_file_features_without_placemarks.kmz` covers it, and fails when that pass is removed. Each part takes its own copy of the Placemark's metadata, since a hash shared between features would be one object behind several records. Image paths are read once per Placemark rather than once per part, because `images_from_metadata` removes the key it reads, so calling it per part would leave every part after the first without images. Iteration order changes. It was every Polygon in the document, then every LineString, then every Point; it is now each Placemark's geometry together in document order, then any geometry outside a Placemark. Nothing asserts on feature sequence, `features_hash` is an MD5 of the source bytes rather than of what the importer emits, and `mvt_sql` orders by id at query time. Feature ids do land in a different order. Closes #66 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Configuration menuCopy the full SHA2e9cd35 View commit details -
fix: Read a nested Placemark's geometry once rather than twice
Reading a Placemark's geometry with `placemark.css(...)` matches everything below it, including the geometry of a Placemark nested inside it. Both Placemarks are iterated, so the inner geometry was read twice and imported as two features. KML 2.2 does not allow the nesting, but a document that does it produced duplicates rather than being read the way the previous implementation read it, which took each geometry's nearest enclosing Placemark and so claimed it once. Instead of every Placemark claiming all the geometry below it, `geometries_in` scopes the match to elements whose nearest enclosing Placemark is that one. Scoping is expressed by depth, since XPath 1.0 has no node-identity operator and cannot ask whether an ancestor is a particular node. The scoped path counts a candidate's ancestors, so it costs an upward walk per element. Measured on a document holding 76,190 geometries in 1,233 Placemarks, it takes 0.330s against 0.045s for the plain selector, and this branch exists to take that class of work out of the loop. It is therefore used only where it is needed: `//Placemark//Placemark` answers once per document whether anything nests, which costs 0.021s on a 32 MB document, and a document that does not nest keeps the plain selector. Adds `kml_file_with_nested_placemarks.kml`, built by `fixtures:build` from the same squares as the other KML fixtures. Reading it yields two features rather than three, and yields three when the scoping is removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Configuration menuCopy the full SHA3047dd9 View commit details -
perf: Read the geometry validity of a batch of features in one query
Saving a feature asks the database whether its geometry is valid, and asks separately for every feature. On an import of any size that is the largest single cost after parsing: profiling one record's import showed the validity query at 18.9% of all database time, behind only the inserts themselves, and the save loop as a whole at 69.7% of wall clock. Instead of asking once per feature, `::precompute_geometry_validation` asks for a batch in one query and hands each record its own answer, which `valid?` then reads in place of querying. Saving is otherwise untouched and still runs every callback, so nothing about what is written changes. A record whose geometry is repaired discards the batch's answer, because `make_valid` replaces the geometry the answer was about. Those records fall back to asking for themselves, which is what they did before. The batch is sent as a literal list of geometries, so what bounds it is the size of that statement rather than the record count: a file of few but very large geometries reaches the ceiling first. At 500 the largest statement measured over a 136,769-feature import was 0.39 MB, against 0.05 MB before, and peak memory rose 2.3%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Configuration menuCopy the full SHA76534ea View commit details -
perf: Parse a batch of KML geometry elements in one query
`geom_from_kml` sent one `ST_GeomFromKML` per geometry element, each wrapped in its own savepoint so that an element PostGIS could not read cost only itself. A KML holding six figures of geometries therefore spent six figures of round trips there, which profiling put at 11.8% of database time with the savepoints adding most of another 8.6%. Instead of one query per element, elements are held until there are enough to parse together and then read in a single query. The savepoint moves with them: a batch holding an element PostGIS rejects fails as a whole, so it is caught and re-read one element at a time, and only that element is lost. That is the behaviour the per-element savepoints provided, at one round trip per batch rather than one per element. Order is unchanged, since elements are held and yielded in the order they were read, and an element with no coordinates is still dropped before it is held rather than after. Building the features for a 136,769-feature import goes from 39.4s to 15.2s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Configuration menuCopy the full SHA6a80bf1 View commit details
Loading
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff master...batch-feature-saves