FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Add a device writer for the standardized Harp file format by bruno-f-cruz · Pull Request #52 · harp-tech/python · GitHub

Add a device writer for the standardized Harp file format - #52

Open
bruno-f-cruz wants to merge 1 commit into
mainfrom
feat-add-device-writer
Open

Add a device writer for the standardized Harp file format#52
bruno-f-cruz wants to merge 1 commit into
mainfrom
feat-add-device-writer

Conversation

bruno-f-cruz commented Aug 27, 2026
edited
Loading

Copy link
Copy Markdown
Member

Add a device writer for the standardized Harp file format

harp.data could read a de-multiplexed dataset folder but nothing in the python client could produce one, so a recording had to come from Bonsai or from a script the user wrote against Device.subscribe_all. DeviceWriter and attach_writer close that loop, writing the layout harp-tech/protocol#69 defines and DatasetReader already reads:

with serial.open_device(behavior, port="COM3") as device:
    with client.attach_writer(device, "session.harp"):
        input("Recording. Press Enter to stop.\n")

The writer owns its subscription, so leaving the block detaches from the device and closes the files. The device outlives the recording, and several recordings can be made over one open device.

What a recording contains

Each message is appended to the file of its own register as the complete Harp frame it arrived as, header and checksum included. The specification prose says the payload is what is saved, but the reference C# MessageWriter writes input.MessageBytes, and parse_to_dataframe reads frames rather than bare payloads, so the implementation is what this follows. A file is then a run of frames that needs nothing else to be decoded.

A register's file is created the first time a message for it arrives, so a folder holds exactly the registers that were seen rather than every register the schema declares. DatasetReader already distinguishes those two cases through contents, so a register with no data reads as an empty DataFrame with the right columns either way.

All three message types are recorded by default, where subscribe_all defaults to Event alone. A device answers reads and writes with Read and Write messages, and the register dump the specification recommends under "Logging the device's initial configuration" arrives as a burst of Read. Capturing only Event would drop the configuration the session ran under. Requesting that dump stays the caller's to do, since it writes to OperationControl and so changes device state (see example).

DEVICE_METADATA

A folder that cannot be decoded later is not worth writing, so the device.yml goes in beside the binaries. That means the schema has to travel with the device module rather than with whatever file the caller happened to read.

DeviceModuleLike gains DEVICE_METADATA: bytes, alongside DEVICE_NAME, WHO_AM_I and REGISTER_MAP, and create_device_module fills it from the text it was given. A generated package declares it the same way, which tests/device/expected_device.py pins.

It is bytes rather than a stream. A BytesIO on a module is a single shared handle with a read position, so the first reader exhausts it and every later one sees nothing, and every consumer has to remember to seek. Bytes have no position, so reading is not a one-shot and BytesIO(module.DEVICE_METADATA) still gets a file-like object where one is wanted.

Because the member is required, a module carrying no metadata is broken rather than a folder to write undescribed, and the writer raises instead of silently producing one. There is no opt-out: the copy is not optional, and neither is the module.

This also means that the static code generator upstream MUST honor this interface and include the device.yml

Naming files

Only the address varies over a recording, so a formatter is given an address and returns a name relative to the folder:

stamp = datetime.now().strftime("%Y%m%dT%H%M%S")
attach_writer(device, "session.harp", formatter=lambda address: f"Behavior_{address}_{stamp}.bin")

The device name and anything else a layout needs are bound when the formatter is built, which is what the default does with DEVICE_NAME.

Notable details

Writes are serialized under one lock and may come from any thread, and the frames of one register keep the order they were written in. write raises on a closed writer, while the subscription path drops a message already in flight at close, since there is nothing wrong with a recording having ended first and a raise there would surface as a logged handler traceback on every normal close.

overwrite is off by default, so the first write to a path already on disk raises FileExistsError rather than half-overwriting an existing recording. The folder and its device.yml are written eagerly at construction, so a path that cannot be written fails before any message is taken; a register file is opened on first use, so that failure arrives later in the run.

attach_writer refuses a device opened without a module, since the module is what names the files and describes the folder.

tests/device/test_writer.py covers the layout, the schema copy and its failures, overwrite, the lifecycle including concurrent writers, attach_writer against a scripted transport, and two round trips back through DatasetReader and open_dataset. docs/examples/record-dataset.md joins the examples in the navigation, and shows the register dump in place.

Closes #50

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Example dumping live session to disk in ingestable format

1 participant


Back | FazBrowse Home | New Git URL