Previously we ran into a bug with Orbit 2 where the client gave no indication that issues were with a particular deployment of the service component in Kubernetes. The observed behavior was that calls would attempt to call the actor but never return and never log an error. The resolution was to delete the deployment of the Orbit 2 server and re-deploy it. This experience brought up the fact that there is insufficient information in logs and metrics to indicate a deployment is in a bad state, why the server is not instantiating an actor, the client isn’t returning, etc. I believe an improvement is needed here in order to greatly reduce the MTTR for these types of issues.
Reactions are currently unavailable
Previously we ran into a bug with Orbit 2 where the client gave no indication that issues were with a particular deployment of the service component in Kubernetes. The observed behavior was that calls would attempt to call the actor but never return and never log an error. The resolution was to delete the deployment of the Orbit 2 server and re-deploy it. This experience brought up the fact that there is insufficient information in logs and metrics to indicate a deployment is in a bad state, why the server is not instantiating an actor, the client isn’t returning, etc. I believe an improvement is needed here in order to greatly reduce the MTTR for these types of issues.