C++ WebRTC Bridge for Unreal Engine PixelStreaming

Unreal Engine released PixelStreaming with its 4.27 version. It enables remote visualization for small-frontend devices.

Synavis enables the coupling of simulation and ML tools to the Unreal Engine by leveraging PixelStreaming as data source.

See our video:

I am currently migrating the Synavis Plugin to accomodate dynamic workflows. Read more about it in the news post and in the living doc.

• DataConnector and MediaReceiver: Connect DL Frameworks and Simulations to UE and interact with the scene
• Synavis: Setup connections via a bridged network (like a TURN server) via port relays. This is intended for HPC systems but might be used together with Putty to bridge to client PCs
• Syanvis Signalling Server: Load Handling and Connection Management
• PySynavis: Python Coupling of all revelant functionality

• Automatic Connection setup and infrastructure-focussed handling is in works
• Decoding is in works, with UE allowing the parsing of software-based encoders, we aim to support these primarily, as they are not dependent on a specific GPU.

• Our course "Virtual Worlds for Machine Learning" provided some in-depth information on the framework and is the newest tutorial. Contact us for more information.
• The wiki contains all current information and is being edited more frequently than this repository: Synavis Wiki

I would greatly appreciate help with this project. It is an integral part of my thesis, but not the main focus of it. I am working in Berlin time and will respond during work hours when I can spare time.

I would like to acknowledge funding provided by the German government to the Gauss Centre for Supercomputing via the InHPC-DE project (01—H17001).

Dirk Norbert Helmrich, Felix Maximilian Bauer, Mona Giraud, Andrea Schnepf, Jens Henrik Göbbert, Hanno Scharr, Ebba Þora Hvannberg, Morris Riedel, A Scalable Pipeline to Create Synthetic Datasets from Functional-Structural Plant Models for Deep Learning, in silico Plants, 2023;, diad022, https://doi.org/10.1093/insilicoplants/diad022

Baker et al. "Adapting Agricultural Virtual Environments in Game Engines to Improve HPC Accessibility" 6th Nordic e-Infrastructure Collaboration Conference, NeIC 2024, Tallinn, Estonia, May 27–29, 2024, Communications in Computer and Information Science, vol 2398. Springer: 10.1007/978-3-031-86240-3_11

Dirk N Baker, Mona Giraud, Jens Henrik Göbbert, Hanno Scharr, Morris Riedel, Ebba Thóra Hvannberg, Andrea Schnepf, Virtual World Coupling with Photosynthesis Evaluation for Synthetic Data Production, in silico Plants, 2025;, diaf018, https://doi.org/10.1093/insilicoplants/diaf018