Leandro Lucarella edited this page Oct 17, 2022
·
1 revision
Design
Why Python?
It's a very widely used and known language
It is very easy to prototype with because of the dynamic
Flexibility: is a dynamically type language, which makes writing small scripts and prototyping very fast and convenient but it provides optional type checking we consider essential for a big system or support library
Very good support for data science / machine learning
Why an actor and pub/sub architecture?
We start with some assumptions:
We want something that is event driven
We want it to be future proof for distribution
We want to use Python
The first assumption implies that calculations have to work concurrently, otherwise obviously one calculation waiting for an event would block everything.
So the cheapest thing we can get for concurrency is tasks (AKA fibers, green threads).
If we want communication between those tasks we need to share data somehow. Using channels might on a first glimpse look like some overhead, but that comes with a clear design advantage, because suddenly you can break down the design of the whole system in modules that communicate with each other and we can draw modules with arrows between them. This is a more modern, scalable and flexible way to design a system rather object oriented modeling, UML stuff etc. It is basically an embedded micro-services architecture.
So that leaves us with Python, because that's it what makes us think about possible performance problems here. We gained some experience using channels for inter-task communication with Rust and those are damn fast. So there should be no problem here.
At the end, if and only if we are impacting performance by the design choices we take, then there is plenty of low hanging fruit to improve that situation, e.g.:
Use alternatives to the (default) CPhython interpreter, e.g. cython or pypy
Rewrite critical code paths in a system language (like Rust) and expose python bindings