Stateful timeseries - #1980
Conversation
Sorry, something went wrong.
* Document in DEBUG.md
* for virtual feeds
* Use HTTP API to update tariff data inputs
* More documentation about using stateful timeseries for costs and standing charges * Use PHP DateInterval specifications for standing charge process
* Tests on PHP7.4 failing due to use of named parameters
* Keep the test happy even though precedence of the operators is explicitly defined in https://www.php.net/manual/en/language.operators.precedence.php
|
Hello @townsen Thanks for this pull request, appreciate your interest in furthering emoncms development! I'd be really interested in some more detail on how you are using this, are you using this with flat tariffs that only change e.g once a year? Do you use custom dashboards to visualise or are you able to use the emoncms apps? |
Sorry, something went wrong.
|
Hi trystan, 
The results can then be plotted on a graph showing my total energy bill and breaking it down into its 4 constituent parts, like so: 
Another use I have for these processors is to graph the duty cycle of my underfloor heating - which is obviously either on or off and turns on or off at irregular times. Again this goes to a variable timeseries and by using the 'state' process I can producesa graph that shows the state of the UFH over time. The 'step' option on graphing can do a similar thing but if the transition occurred before the current time window then the step graph will not show the correct results, nor will it show the state extending into the future.. There are some issues with the conversion of variable times into fixed times which I discuss in the notes, but in practice they can be ignored. To fix those would require a more complex approach (apportioning a value that changes in the middle of a fixed interval) which I don't have the energy to try and implement right now. I hope this helps. |
Sorry, something went wrong.
|
Sorry for the delay @townsen thanks for coming back to me with the above. In many ways I think your solution to this is elegant and clearly functional - but if I'm honest Im currently of the mind that I would like to ultimately deprecate PHPTimeSeries and limit Emoncms to PHPFina fixed interval only (removing all the other engines as well). I also try to avoid virtual feeds unless for simple quick calculations . The reason for this is that once data is in PHPFina format - which matches the way we collect all our data at fixed interval - access and post processing of that data becomes very efficient (post processing being something we're doing more and more e.g for the MyHeatpump and the new MyElectricFlow dashboards) - and so I want to explore if there are any alternative approaches that we could use to meet your same requirements before merging this and increasing project scope. One option could be just to record the tariff information to PHPFIna at hourly resolution, I appreciate in many ways that's overkill in terms of disk storage but I think it would be possible to work out a solution that solved it this way. Another - which is probably the way I would do it personally is to build a custom emoncms app with a nice UI interface to enter new tariff change points - you could do quite a lot more in an app such as calculate summaries/totals for longer periods etc. This could be a custom app for your own use rather than included in the public repo though you could also share it if any others have the same requirements.. Do you think either of these could be a solution to your requirements? |
Sorry, something went wrong.
|
Well, my first response is that some data naturally occurs in the TimeSeries format and not Fina - so it would be a mistake to deprecate it. Recording it as an hourly fixed interval might be fine for tariffs, but it wouldn’t be fine for recording duty cycles - time a heatpump spends on and off for example. Moving it out to an App would work but it’s hard to see how that could be leveraged back into the core.
I don’t have an issue with the ‘processing load’ being more (and here I drop into ‘mark my words son’ mode!) in my 40+ years of programming I’ve seen this situation play out several times: it doesn’t end well for faking the data and avoiding the processing, the processing always becomes largely irrelevant :)
What is a problem right now is the lack of the right interface between the consumers of data (graphing) and the source of data (the feeds). You can see that in order to get the stateful timeseries request back into the engines I have to add extra parameters to the call(s), which aren’t relevant to other sources. As I was shoe-horning this into the code I felt a re-factor coming on but wasn’t going to undertake that if there was no chance it would ever get merged. Your comment about ‘increasing project scope’ was anticipated…
If you are thinking of streamlining the codebase, refactoring and removing stuff then possibly the time is right for a V2 plan. I used to love PHP (and Perl before that - I met the great Larry Wall once upon a time) but it wouldn’t be my first choice for a project today. You could use this as an opportunity to re-platform it. There are numerous time series databases around today that weren’t there when you started, and GUI tools that can query them. None that I’ve seen have the fit-for-purpose that EmonCMS is built for but could be leveraged. I’d be happy to be part of that effort as I’m just now retired…
If that’s not a flyer then I’ll probably keep this PR current for my own use, I’ve actually been using it for years keeping it up-to-date through some of your refactoring and refining. However, one thing I would suggest is to adopt my improvements to the binary search code, as I was coding that section I did fix some minor logic bugs.
… On 13 Jun 2026, at 19:19, Trystan Lea ***@***.***> wrote:
TrystanLea
left a comment
(emoncms/emoncms#1980)
<#1980 (comment)>
Sorry for the delay @townsen <https://github.com/townsen> thanks for coming back to me with the above.
In many ways I think your solution to this is elegant and clearly functional - but if I'm honest Im currently of the mind that I would like to ultimately deprecate PHPTimeSeries and limit Emoncms to PHPFina fixed interval only (removing all the other engines as well). I also try to avoid virtual feeds unless for simple quick calculations . The reason for this is that once data is in PHPFina format - which matches the way we collect all our data at fixed interval - access and post processing of that data becomes very efficient (post processing being something we're doing more and more e.g for the MyHeatpump and the new MyElectricFlow dashboards) - and so I want to explore if there are any alternative approaches that we could use to meet your same requirements before merging this and increasing project scope.
One option could be just to record the tariff information to PHPFIna at hourly resolution, I appreciate in many ways that's overkill in terms of disk storage but I think it would be possible to work out a solution that solved it this way.
Another - which is probably the way I would do it personally is to build a custom emoncms app with a nice UI interface to enter new tariff change points - you could do quite a lot more in an app such as calculate summaries/totals for longer periods etc. This could be a custom app for your own use rather than included in the public repo though you could also share it if any others have the same requirements..
Do you think either of these could be a solution to your requirements?
—
Reply to this email directly, view it on GitHub <#1980?email_source=notifications&email_token=AKQLRVWRBCCYXVL7MTSWFXT47WLLNA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRZHEZTQMRTGY32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4699382367>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKQLRVW4ZNMNFXCOAP5VPVL47WLLNAVCNFSNUABDKJSXA33TNF2G64TZHM3DEMZTHAYDIO2JONZXKZJ3GM4DOMRUGU4DMNRTUF3AE>.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AKQLRVRMDT44WKPTQ7ELMR347WLLNA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRZHEZTQMRTGY32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AKQLRVRW7NVOS5VYDZTUZQ347WLLNA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINRZHEZTQMRTGY32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today!
You are receiving this because you were mentioned.
|
Sorry, something went wrong.
Making a PHPTimeSeries feed Stateful
The TimeSeries feeds in EmonCMS contain datapoints at irregular time intervals. When used
for graphing the values of those feeds at times within these intervals is usually 'null'
(unless by chance a datapoint exists within the interval). However when such a
TimeSeries represents the state of a system or a tariff cost the value at any point in
time is the most recent datapoint, even if that falls before the interval.
To help solve these two related problems we introduce four new Virtual Feed processes.
See the file STATEFUL_TIMESERIES.md for a full description
With these features you can create feeds that represent point-in-time tariff changes and graph your energy cost over time including standing charges and accurately reflected costs over long periods. There is a small limitation discussed in the document which would require more significant changes to alleviate. This is a start.