(artificially low, as some tests are run in a spun off docker container that isn't currently counted see PR #82 )

A controller for reading waveform data from a rabbitmq queue and processing it.

See architecture overview diagram

Up-to-date Docker and Docker Compose. We have seen config bugs when using old Docker Compose versions, such as that packaged with recent Ubuntu LTS. Docker Compose v5.0.1 and Docker 29.1.5 are known to work.

Follow the emap development instructions configure and deploy a version of EMAP. To run a local version you'll need to set

  fake_uds:
    enable_fake_uds: true
  uds:
    UDS_JDBC_URL: jdbc:postgresql://fakeuds:5432/fakeuds

and configure and synthetic waveform generator

waveform:
  enable_waveform: true
  enable_waveform_generator: true
  CORE_WAVEFORM_RETENTION_HOURS: 24
  WAVEFORM_HL7_SOURCE_ADDRESS_ALLOW_LIST: ALL
  WAVEFORM_HL7_TEST_DUMP_FILE: ""
  WAVEFORM_HL7_SAVE_DIRECTORY: "/waveform-saved-messages"
  WAVEFORM_SYNTHETIC_NUM_PATIENTS: 2
  WAVEFORM_SYNTHETIC_WARP_FACTOR:1
  WAVEFORM_SYNTHETIC_START_DATETIME: "2024-01-02T12:00:00Z"
  WAVEFORM_SYNTHETIC_END_DATETIME: "2024-01-03T12:00:00Z"

Once configured you can start it with

emap docker up -d

Create a root directory for your installation of the waveform-controller project, separate to the Emap project root.

├── PIXL (repo root of the PIXL repo)
├── config (config files for the waveform project)
├── waveform-controller (repo root for this repo)
└── waveform-export (bind mounted by the containers, this is the main working directory for the waveform project)

Clone this repo (waveform-controller) and PIXL, both inside your root directory.

Inside the PIXL repo, checkout the commit that we have pinned in workflow file.

If on a system that has access to sensitive data, disable push remotes on all cloned repos as follows:

git remote set-url --push origin no_push.example.com

Set up the config files as follows:

mkdir config
cp waveform-controller/config.EXAMPLE/controller.env.EXAMPLE config/controller.env
cp waveform-controller/config.EXAMPLE/exporter.env.EXAMPLE config/exporter.env
cp waveform-controller/config.EXAMPLE/hasher.env.EXAMPLE config/hasher.env

From the new config files, remove the comments telling you not to put secrets in it, as instructed.

The config files contain documentation in comments, but some are further discussed here.

See azure and hasher setup to configure the hasher.

INSTANCE_NAME in exporter.env should always be set to a non-empty, globally unique value. This is used to identify the instance of the exporter when the files are uploaded to the FTPS server. This avoids production data being mixed up with synthetic data. The INSTANCE_NAME isn't used as part of file paths on the system where the instance is running: it's assumed that only one instance will be using each filesystem location, so this would be unnecessary. It's only used on the FTPS server. Example names:

• yourname-dev for your dev instance
• production for production ONLY

When deploying a new version of this code, you should diff the .EXAMPLE file against its live version, eg. by running vimdiff waveform-controller/config.EXAMPLE/controller.env.EXAMPLE config/controller.env.

This checks if any config options have been added/removed from the .EXAMPLE, and thus should be added/removed from the live file.

If it doesn't already exist you should create a directory named waveform-export in the parent directory to store the saved waveform messages.

mkdir waveform-export

Build and start the hasher, controller and exporter with docker.

cd waveform-controller
docker compose build
docker compose up -d

For more complex deployment scenarios, such as where there is existing data you need to preserve, see the more advanced deployment doc

Running the controller will save (to ../waveform-export) waveform messages matched to Contact Serial Number (CSN) as csv files, each containing data for one calender day, as YYYY-MM-DD.CSN.sourceName.units.csv

Each row of the csv will contain

csn, mrn, units, samplingRate, observationTime, waveformData

See debugging doc to investigate further

See developing docs