FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Add a device writer for the standardized Harp file format by bruno-f-cruz · Pull Request #52 · harp-tech/python · GitHub

Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension .md  (5) .py  (11) .yml  (1) All 3 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
1 change: 1 addition & 0 deletions docs/api/data.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
::: harp.data.read
::: harp.data.DatasetReader
::: harp.data.default_file_resolver
::: harp.data.FileNameResolver
::: harp.data.parse_to_dataframe
::: harp.data.payload_to_dataframe
::: harp.data.to_file
Expand Down
3 changes: 3 additions & 0 deletions docs/api/device.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
---

::: harp.device.client.Device
::: harp.device.client.attach_writer
::: harp.device.client.DeviceWriter
::: harp.device.client.DeviceError
::: harp.device.client.Subscription
::: harp.device.client.EventHandler
Expand All @@ -14,4 +16,5 @@
::: harp.device.schema.ConverterContext
::: harp.device.schema.DeviceModule
::: harp.device.schema.DeviceModuleLike
::: harp.device.schema.DEVICE_SCHEMA_FILENAME
::: harp.device.core
15 changes: 15 additions & 0 deletions docs/examples/record-dataset.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Record a dataset folder

`harp.device.client.attach_writer` records everything a device emits into a **de-multiplexed dataset folder**, the layout [Read a dataset folder](dataset.md) reads back: one binary file per register, named `<DeviceName>_<address>.bin`, next to a copy of the `device.yml` the module was built from.

Each message is appended to the file of its own register as the complete Harp frame it arrived as, header and checksum included, which is what the reference C# writer records. The file of a register is created the first time a message for it arrives, so the folder holds exactly the registers that were seen.

The writer owns its subscription, so leaving the `with` block both detaches it from the device and closes the files. All three message types are recorded by default: the register dump a device performs on request arrives as `Read` messages, so recording only `Event` would drop the configuration the session ran under.

{% include-markdown "includes/serial-port.md" %}

<!--codeinclude-->
```python
[](./record_dataset.py)
```
<!--/codeinclude-->
35 changes: 35 additions & 0 deletions docs/examples/record_dataset.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
from pathlib import Path

from harp import data
from harp import serial
from harp.device import client, core, schema

SERIAL_PORT = "/dev/ttyUSB0" # or "COMx" in Windows, where "x" is the serial port number

behavior = schema.create_device_module(Path("device.yml").read_bytes())

with serial.open_device(behavior, port=SERIAL_PORT) as device:
# The writer owns its subscription, so leaving this block detaches it from the
# device and closes the files. `device.yml` is copied into the folder as it opens.
with client.attach_writer(device, "session.harp") as writer:
# Ask the device to report every register, so the folder records the
# configuration the session ran under and not only its events. Do this once the
# writer is attached, or the reply burst is missed.
device.write(
core.OperationControl,
core.OperationControlPayload(
operation_mode=core.OperationMode.ACTIVE,
dump_registers=True,
heartbeat=core.EnableFlag.ENABLED,
mute_replies=False,
operation_led=core.EnableFlag.ENABLED,
visual_indicators=core.EnableFlag.ENABLED,
),
)

input("Recording. Press Enter to stop.\n")
print("recorded:", {address: path.name for address, path in writer.paths.items()})

# The folder carries its own schema, so it reads back without a module in hand.
reader = data.open_dataset("session.harp")
print(reader.contents)
1 change: 1 addition & 0 deletions mkdocs.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ nav:
- Read and write registers: examples/read-write-registers.md
- Subscribe to events: examples/events.md
- Read a dataset folder: examples/dataset.md
- Record a dataset folder: examples/record-dataset.md
- Read a single register file: examples/register-file.md
- Registers from a schema: examples/registers-from-schema.md
- Guides:
Expand Down
2 changes: 2 additions & 0 deletions src/packages/harp-data/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ The `<DeviceName>` prefix comes from the `DEVICE_NAME` declared by the device mo

When a device module declaring an identity is supplied and the folder carries a `device.yml`, their `whoAmI` values are checked against each other. Reusing a module across sessions and opening the wrong folder then fails on construction rather than decoding the files against the wrong register map. Pass `validate=False` to turn off every check the reader performs, so a folder whose `device.yml` is damaged can be read with a module obtained elsewhere.

Recordings are produced by `harp.device.client.attach_writer`, on the other side of the same file format. See the [harp-device README](https://github.com/harp-tech/python/tree/main/src/packages/harp-device).

## Read a single register file

`parse_to_dataframe` takes a register and a source, either a path, bytes, or an open binary file, and returns one row per frame:
Expand Down
3 changes: 2 additions & 1 deletion src/packages/harp-data/src/harp/data/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from ._dataset import DatasetReader, default_file_resolver, open_dataset
from ._dataset import DatasetReader, FileNameResolver, default_file_resolver, open_dataset
from ._read import read
from ._reader import REFERENCE_EPOCH, parse_to_dataframe, payload_to_dataframe
from ._write import to_buffer, to_file
Expand All @@ -12,5 +12,6 @@
"DatasetReader",
"open_dataset",
"default_file_resolver",
"FileNameResolver",
"REFERENCE_EPOCH",
]
6 changes: 2 additions & 4 deletions src/packages/harp-data/src/harp/data/_dataset.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

import pandas as pd
from harp.device.schema import (
DEVICE_SCHEMA_FILENAME,
DeviceModule,
DeviceModuleLike,
create_device_module,
Expand All @@ -22,9 +23,6 @@

FileNameResolver = Callable[[Path, str], Mapping[int, list[Path]]]

DEVICE_SCHEMA_FILENAME = "device.yml"
"""Default filename of the device schema looked up inside a dataset folder."""


def default_file_resolver(root: Path, name: str) -> dict[int, list[Path]]:
"""Harp file format resolver: map address -> sorted ``<name>_<address>...`` files."""
Expand Down Expand Up @@ -70,7 +68,7 @@ class DatasetReader(Generic[M]):

File resolution defaults to the Harp file format: ``<name>_<address>.bin`` and,
when a register was logged as several ``<name>_<address>_<suffix>.bin`` chunks,
they are concatenated in filename order. Pass ``resolver`` (a :data:`FileResolver`)
they are concatenated in filename order. Pass ``resolver`` (a :data:`FileNameResolver`)
to support an alternative on-disk layout.

``epoch`` anchors the time index of every read to absolute time, so one dataset is
Expand Down
4 changes: 4 additions & 0 deletions src/packages/harp-device/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,18 @@ A device is described by a module. Downstream, often generated, packages record
```python
from harp.device.core import REGISTER_MAP as _CORE_REGISTER_MAP

DEVICE_NAME: str = "Behavior"
WHO_AM_I: int = 1216
DEVICE_METADATA: bytes = ... # the device.yml the package was built from
REGISTER_MAP = {**_CORE_REGISTER_MAP, 32: DigitalInputState, ...}
```

This is the same structure `create_device_module` builds from a schema, so a device reads the same way whether it was generated ahead of time or compiled at runtime. A `WHO_AM_I` of `0` marks an unregistered device, used while a device is in development or outside the official registry, and identity checks are skipped for it.

A device module names only what its schema declares, the registers beside the enums and payload classes built from them. The core registers and any core mask reused by the schema have a single definition, in `harp.device.core`, and are accessed from there rather than through the device module. The core register set is not a device, so it carries no `WHO_AM_I`. `REGISTER_MAP` covers the complete device address space, including both core and application registers.

`DEVICE_METADATA` is the `device.yml` the module was built from, as bytes, so the schema travels with the module and a recording can carry a copy of it without the original file being at hand. `DeviceWriter` is what usually reads it, copying it into the folder it records.

Pass the module to `Device`, or to `open_device`, to validate identity on open:

```python
Expand Down
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
"""Talking to a Harp device: the device itself, its transport and the framer."""
"""Talking to a Harp device: the device itself, its transport, the framer, and the
writer that records what it emits."""

from ._device import Device, DeviceError, EventHandler, Subscription
from ._framer import HarpFramer
from ._transport import ITransport, TransportError
from ._writer import DeviceWriter, attach_writer

__all__ = [
"Device",
Expand All @@ -12,4 +14,6 @@
"HarpFramer",
"ITransport",
"TransportError",
"DeviceWriter",
"attach_writer",
]
Loading

Back | FazBrowse Home | New Git URL