FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat(reconciler)!: add async commits and incremental lists by owenthcarey · Pull Request #107 · pythonnative/pythonnative · GitHub

Repository navigation

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension .cpp  (2) .js  (4) .json  (3) .kt  (34) .md  (12) .mjs  (1) .py  (44) .swift  (28) .yaml  (3) All 9 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
2 changes: 1 addition & 1 deletion docs/api/component-properties.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -640,7 +640,7 @@ pn.FlatList(data=items, render_item=render_fn, key_extractor=key_fn,
## SectionList

```python
pn.SectionList(sections=[{"title": "A", "data": ["Apple"]}],
pn.SectionList(sections=[pn.Section(key="a", title="A", data=["Apple"])],
render_item=lambda item, i, s: pn.Text(item),
render_section_header=lambda section, s: pn.Text(section["title"]),
sticky_section_headers=True, inverted=False,
Expand Down
14 changes: 14 additions & 0 deletions docs/api/components.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,20 @@ For the visual and layout properties accepted by each component's
- "!^_"
- "!^(ContentSizeEvent|ImageLoadEvent|KeyPressEvent|LayoutEvent|ScrollEvent|SelectionEvent|WebNavigationEvent)$"

## List data and records

`ListData` publishes explicit keyed edits to `FlatList`. `Section` gives each
section a stable identity, and `ViewableItem` describes a visible item. See the
[list guide](../guides/lists.md) for ownership, batching, and performance details.

::: pythonnative.list_data
options:
show_root_heading: false
show_root_toc_entry: false
members: [ListData, Section, ViewableItem]
members_order: source
filters: ["!^_"]

## Typed event payloads

Frozen dataclasses delivered to `on_layout`, `on_scroll` (and the drag
Expand Down
21 changes: 16 additions & 5 deletions docs/concepts/architecture.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,12 @@ choice for shared snapshots.

Swift and Kotlin component managers create widgets, apply props, measure native
content, handle commands, and release resources. Yoga owns geometry. The platform
UI thread owns every widget mutation and input callback. The bridge releases
Python's execution lock while native applies synchronous requests.
UI thread owns every widget mutation and input callback. A worker performs the
blocking native commit crossing while the application
loop remains available for input and asyncio tasks. One transaction mounts at a
time; queued transactions prepare against the latest acknowledged revision.
Refs, effects, and queued native events publish after acknowledgement. State
writes during mounting coalesce into the following render.

Navigation presents logical screen roots through a nested UIKit navigation
controller or Android fragments. Pushing a screen doesn't create another Python
Expand All @@ -59,12 +63,16 @@ footers, empty states, grouped grids, and sections use the same ownership model.

## Contracts and tooling

Protocol 3 validates commits before mutation and acknowledges exact revisions.
Protocol 4 validates commits before mutation and acknowledges exact revisions.
Events carry application and revision identities. Controlled inputs additionally
acknowledge native edit revisions to avoid overwriting newer typing. Native
animation graphs perform frame updates independently of Python callbacks.

The SDK compiles dataclass props and protocol methods into portable contracts.
The SDK compiles dataclass props and protocol methods into portable contracts
and executable Swift/Kotlin predicates. Native prop wrappers cache decoded
fields. Yoga applies touched styles directly and returns geometry only for
observed refs and layout callbacks. `ListData` feeds revisioned keyed patches;
ordinary sequences use a scanning adapter to that same protocol.
Generated artifacts are checked in and tested for drift. Target-wheel plugin
metadata is read as data, without importing a mobile binary on the development
machine. Resources and registration code are staged into the native libraries.
Expand All @@ -77,7 +85,10 @@ run on a device. Test every supported deployment target.
Fast Refresh preserves compatible component state. Changes to hook order or
custom-hook signatures remount affected instances. Changes to helper classes or
services remount the application to avoid retaining instances of old definitions.
Native contract changes require rebuilding the dev client.
Native contract changes require rebuilding the dev client. Development errors
use a host-owned overlay with a traceback and reload/dismiss controls. It can
report a poisoned renderer and reload a fresh surface without rendering an
additional Python error tree.

Python phase timings and bridge work counters can be captured in Chrome trace
format. See [Profiling](../guides/dev-workflow.md#profiling) for collection and
Expand Down
43 changes: 37 additions & 6 deletions docs/concepts/bridge.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,10 @@ in `src/pythonnative/native/android`. App templates stage and embed those librar

## Application and UI threads

The application has an ordinary asyncio loop on its own thread. Calls into UIKit
and Android synchronously marshal native operations to the platform UI thread.
The application has an ordinary asyncio loop on its own thread. A commit's
blocking transport call runs on a worker. Native JSON decoding
happens there before operations are queued to the platform UI thread. Structural
preflight, widget mutation, and native measurement retain UI-thread ownership.
On iOS, the ctypes transport releases the GIL during this crossing. Native input
is queued back to Python rather than waiting for a Python handler on the UI thread.

Expand All @@ -28,7 +30,7 @@ A surface commit has this shape:

```json
{
"version": 3,
"version": 4,
"application": "unique-application-id",
"surface": 1,
"revision": 1,
Expand Down Expand Up @@ -61,6 +63,31 @@ A failed surface rejects further incremental updates. A new application identity
and a complete remount establish a clean surface; replaying a partial commit
isn't a recovery strategy. The current app host uses one surface.

The application loop keeps running while acknowledgement is pending. A reconciler
queues state writes instead of rendering against an uncommitted tree. Shared
backend transactions serialize; preparation uses the acknowledged revision when
a transaction reaches the front. Refs, effects, and native events wait for
publication. Cancelling a caller's wait doesn't cancel native mounting.

### Incremental list packets

`VirtualList.dataset` contains `base`, `revision`, and `changes`. Each revision
advances by one. There may be one dataset patch per list per commit.

| Change | Payload |
| --- | --- |
| `reset` | array of `[key, item_revision, extent, sticky]` records |
| `i` | insertion index, row record |
| `u` | replacement row record with the same key |
| `d` | key to remove |
| `m` | key, final index |

A reset must be first. Keys are unique nonempty strings; extents are finite and
positive. Validation stages edits before publishing them. Updating metadata
preserves the order index; structural edits rebuild indexes as needed. Native
window events include the active sticky header. Application scroll events keep
the `ScrollEvent` shape and are emitted only when subscribed.

## Events and controlled input

Events carry `application`, `surface`, `revision`, `sequence`, and an `args` list.
Expand All @@ -84,12 +111,16 @@ core, and the browser preview uses the corresponding Yoga WebAssembly package.

The application sends styles as props. Native leaf managers measure text,
controls, and images next to their widgets. When the viewport is known, a commit
includes a `layout` object with `roots`, `width`, and `height`. Its acknowledgment
includes the resulting frames, so mutation and required layout share one bridge
includes a `layout` object with `roots`, `width`, `height`, and `selective: true`.
Its acknowledgment includes changed frames for `_pn_layout` observers (refs or
`on_layout`), so mutation and required layout share one bridge
crossing. `Layout.compute` remains available for viewport changes without
mutations. Both return frames with `application`, `surface`, and `revision`.
Geometry from another surface or revision is ignored. Drawing-only prop updates
skip a separate request. Yoga calculations reuse unchanged constraints, and
skip a separate request. Changed styles update existing Yoga nodes; removal
restores the property default or surviving shorthand. Diagnostic raw layout
requests may omit `selective` to collect all changed frames. Yoga calculations
reuse unchanged constraints, and
frame collection visits only subtrees with new layout. Python doesn't make one
measurement RPC per leaf.

Expand Down
19 changes: 11 additions & 8 deletions docs/examples/lists.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,10 @@ def TodoList():
),
pn.FlatList(
data=todos,
render_item=lambda t: TodoRow(
render_item=lambda t, index: TodoRow(
todo=t, on_toggle=toggle, on_delete=delete, key=t["id"]
),
key_extractor=lambda t: t["id"],
key_extractor=lambda t, index: t["id"],
style={"flex": 1},
),
)
Expand All @@ -85,10 +85,9 @@ def TodoList():
## Why keys matter

The reconciler matches children by `key` first and by position only as
a fallback. Without keys, a delete of the first row would update the
remaining rows in place (showing the wrong text briefly) before
unmounting the last row. With keys, identity flows from the data, so
"todo 2" stays mounted as "todo 2" even when its index shifts.
a fallback. Position-based keys can transfer a row's local state to another
item after a deletion. With stable keys, "todo 2" keeps its identity even
when its index shifts, as long as it remains in the mounted window.

The `key_extractor` on `FlatList` and the `key=` on the rendered row
both come from the same `todo["id"]`. When in doubt: use a stable
Expand All @@ -97,15 +96,19 @@ identifier that's part of the data, not the position.
## Performance notes

- `FlatList` lazily mounts only the rows that are visible (or near
visible); off-screen rows are represented by spacers and the window
shifts as the user scrolls. For long lists it scales much better
visible); the native list owns scrolling, item geometry, and recycling.
For long lists it scales much better
than wrapping a `Column` in a `ScrollView`.
- Avoid recomputing `data` on every render. If you derive it from
another piece of state, wrap it in
[`use_memo`][pythonnative.use_memo].
- For row callbacks that you pass deeply, consider
[`use_callback`][pythonnative.use_callback] to keep references
stable.
- Ordinary sequences are convenient for small, derived collections. For large
collections with frequent edits, use [`ListData`][pythonnative.ListData] to
publish keyed changes without scanning every item. See the
[incremental list guide](../guides/lists.md).

## Sorting and filtering

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/custom-native-components.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ import com.pythonnative.runtime.components.TypedComponentManager
import com.pythonnative.runtime.components.ViewStyler
import org.json.JSONObject

class BadgeManager : TypedComponentManager<BadgeProps>({ values, partial -> BadgeProps(values, partial) }) {
class BadgeManager : TypedComponentManager<BadgeProps>({ values, partial, validated -> BadgeProps(values, partial, validated) }) {
override fun createView(context: Context, tag: Long, props: JSONObject): View =
TextView(context).apply {
gravity = Gravity.CENTER
Expand Down
22 changes: 18 additions & 4 deletions docs/guides/dev-workflow.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -237,10 +237,24 @@ PN_PROFILE=/tmp/pythonnative-trace.json pn preview
```

Collection retains the latest 10,000 timing events and exports the trace at
normal process exit. The trace includes render, layout, and commit phases,
plus counters for rendered components and bridge operations and bytes. This
command captures the preview's Python process; it doesn't configure an
environment variable inside a connected mobile app.
normal process exit. The trace identifies rendered components and separates
Python preparation, acknowledgement, and native decode, queue, preflight,
mutation, and layout durations. Counters cover bridge bytes/operations, list
snapshot rows and patches, and mounted views. `Profiler.summary()` reports
retained-sample p50/p95/max durations, work counters, and gauges.

Setting `PN_PROFILE` here captures the preview's Python process; it doesn't
configure an environment variable inside a connected mobile app.

Run the reproducible headless work benchmark from the repository root:

```bash
python scripts/benchmark-rendering.py --rows 100000 --edits 20 > benchmark.json
```

It compares ordinary sequence replacement with `ListData`, asserts bounded
metadata and mounted-view work, and emits JSON. Timings include profiling and
aren't device frame-rate measurements or CI thresholds.

For an explicit capture in a headless test:

Expand Down
89 changes: 73 additions & 16 deletions docs/guides/lists.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

[`FlatList`][pythonnative.FlatList] and
[`SectionList`][pythonnative.SectionList] are the two list components
shipped with PythonNative. Both are **virtualized**: only the rows
shipped with PythonNative. Sections use frozen `Section(key=..., data=..., title=...)` records; section keys
are unique and independent of their positions. Both lists are **virtualized**: only the rows
inside (and just beyond) the viewport are mounted as native views.
On mobile, `RecyclerView` (Android) and `UICollectionView` (iOS) own physical
cell recycling. Rows remain ordinary keyed components in the application's
Expand Down Expand Up @@ -35,6 +36,64 @@ def Big():
The list never holds 10,000 native views; only the window around the
viewport ever exists.

## Incremental data

Use `ListData[T]` for frequently edited lists. It owns stable string keys and
notifies mounted lists when items change. Keep it in a repository or create it
with `use_memo` on the application thread:

```python
from dataclasses import dataclass, replace
import pythonnative as pn

@dataclass(frozen=True)
class Message:
id: str
title: str


def render_message(message: Message, index: int):
return pn.Text(message.title)


@pn.component
def Messages():
messages = pn.use_memo(
lambda: pn.ListData([Message("a", "Hello")], key=lambda message: message.id),
[],
)

def edit():
new_key = f"message-{messages.revision}"
with messages.batch():
messages.update("a", replace(messages.get("a"), title="Edited"))
messages.append(Message(new_key, "New message"))
messages.move(new_key, 0)

return pn.Column(
pn.Button("Edit messages", on_press=edit),
pn.FlatList(data=messages, render_item=render_message, style={"flex": 1}),
style={"flex": 1},
)
```

Omit `key_extractor` when passing `ListData`; the source owns its keys. Mutate
it on its creating thread. `update(key, replacement)` preserves identity;
`insert(index, item)`, `append(item)`, `remove(key)`, `move(key, final_index)`,
and `clear()` publish explicit edits. `batch()` combines notifications, including
nested batches. It doesn't roll back successful edits if its body raises.

A one-item update does constant metadata work and sends one row record, even
with 100,000 items. Initial mounting and expired history require a full metadata
snapshot. Structural edits shift indexed arrays, so insertion and reordering
aren't constant-time operations. The bounded journal defaults to 1,024 changes;
each mounted list tracks its own committed revision.

The incremental path supports ordinary single-column lists with fixed or
estimated heights. Grids and `get_item_height` use the sequence adapter below.
Changing rendering callbacks or list decorations also invalidates the snapshot.
Keep callbacks stable with module-level functions or `use_callback`.

## Data snapshots

Lists accept a `Sequence` and cache keys and row metadata separately from the
Expand Down Expand Up @@ -129,7 +188,7 @@ pn.FlatList(
```

`on_viewable_items_changed` reports the set of visible rows whenever
it changes, as a list of `{"index", "key", "item"}` dicts; on a
it changes, as a list of frozen `ViewableItem` records with `.index`, `.key`, and `.item`; on a
`SectionList` the list covers items only, with flat indices.
`on_scroll` receives a [`ScrollEvent`][pythonnative.ScrollEvent] as the
list scrolls.
Expand Down Expand Up @@ -186,21 +245,21 @@ pn.FlatList(
## Section lists

[`SectionList`][pythonnative.SectionList] flattens an iterable of
`{"title": ..., "data": [...]}` sections into a single virtualized
`Section` records into a single virtualized
list, dispatching to either `render_section_header` or `render_item`
depending on the row's kind.

```python
sections = [
{"title": "A", "data": ["Apple", "Avocado"]},
{"title": "B", "data": ["Banana", "Blueberry"]},
pn.Section(key="a", title="A", data=["Apple", "Avocado"]),
pn.Section(key="b", title="B", data=["Banana", "Blueberry"]),
]

pn.SectionList(
sections=sections,
item_height=44,
section_header_height=32,
render_section_header=lambda s, _: pn.Text(s["title"]),
render_section_header=lambda s, _: pn.Text(s.title),
render_item=lambda item, _i, _s: pn.Text(item),
)
```
Expand All @@ -210,29 +269,27 @@ rows work exactly as in `FlatList` (exact via `get_item_height`, or
estimated and measured). `inverted`, `item_separator`, and
`on_viewable_items_changed` work as on `FlatList`.

`sticky_section_headers=True` keeps the current section's header
pinned at the top of the list while its items scroll by. Python
renders the overlay (through `render_section_header`, absolutely
positioned over the list) from the first visible native row and hides
it exactly while the real header is itself at the top, so the two
never show at once:
`sticky_section_headers=True` keeps the current section's header pinned at the
top of the native list and lets the next header push it away. Native layout
owns this movement, including while Python is busy. Python retains the active
header in its bounded mounted window. No application `on_scroll` handler is
needed.

```python
pn.SectionList(
sections=sections,
sticky_section_headers=True,
section_header_height=32,
render_section_header=lambda s, _: pn.Text(s["title"], style={"background_color": "#FFF"}),
render_section_header=lambda s, _: pn.Text(s.title, style={"background_color": "#FFF"}),
render_item=lambda item, _i, _s: pn.Text(item),
)
```

Give the header an opaque background so rows don't show through the
overlay.
Give the header an opaque background so rows don't show through it.

## Performance notes

- Always provide a stable `key_extractor` so rows that stay inside
- Use `ListData` keys or a stable `key_extractor` so rows that stay inside
the window refresh in place rather than tearing down and rebuilding
their subtree as the window shifts.
- Provide real extents (`item_height` / `get_item_height`) when you
Expand Down
Loading
Loading

Back | FazBrowse Home | New Git URL