FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat(go): Implement metrics and tracing for http and grpc servers by luisazofracabify · Pull Request #5925 · feast-dev/feast · GitHub

10 changes: 10 additions & 0 deletions go.mod
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,15 @@ require (
github.com/aws/aws-sdk-go-v2/config v1.29.14
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.43.3
github.com/aws/aws-sdk-go-v2/service/s3 v1.79.3
github.com/cabify/gotoprom v1.1.0
github.com/ghodss/yaml v1.0.0
github.com/go-sql-driver/mysql v1.8.1
github.com/golang/protobuf v1.5.4
github.com/google/uuid v1.6.0
github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus v1.1.0
github.com/mattn/go-sqlite3 v1.14.23
github.com/pkg/errors v0.9.1
github.com/prometheus/client_golang v1.23.2
github.com/redis/go-redis/v9 v9.6.1
github.com/roberson-io/mmh3 v0.0.0-20190729202758-fdfce3ba6225
github.com/rs/zerolog v1.33.0
Expand Down Expand Up @@ -64,6 +67,7 @@ require (
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.30.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.33.19 // indirect
github.com/aws/smithy-go v1.22.2 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cncf/xds/go v0.0.0-20250501225837-2ac532fd4443 // indirect
Expand All @@ -81,6 +85,7 @@ require (
github.com/google/s2a-go v0.1.9 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.7 // indirect
github.com/googleapis/gax-go/v2 v2.15.0 // indirect
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.1.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.2 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
Expand All @@ -93,9 +98,13 @@ require (
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/procfs v0.16.1 // indirect
github.com/spiffe/go-spiffe/v2 v2.5.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/zeebo/errs v1.4.0 // indirect
Expand All @@ -107,6 +116,7 @@ require (
go.opentelemetry.io/otel/metric v1.38.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.38.0 // indirect
go.opentelemetry.io/proto/otlp v1.7.1 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
golang.org/x/crypto v0.45.0 // indirect
golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 // indirect
golang.org/x/mod v0.29.0 // indirect
Expand Down
70 changes: 61 additions & 9 deletions go.sum

Large diffs are not rendered by default.

19 changes: 16 additions & 3 deletions go/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,27 @@ To build and run the Go Feature Server locally, create a feature_store.yaml file

```bash
go build -o feast-go ./go/main.go
# start the http server
./feast-go --type=http --port=8080
# start the http server (metrics on port 9090 by default)
./feast-go --type=http --port=8080 --metrics-port=9090
# or start the gRPC server
#./feast-go --type=grpc --port=[your-choice]
#./feast-go --type=grpc --port=[your-choice] --metrics-port=9091
```
## Prometheus Metrics
The server exposes Prometheus metrics at the `/metrics` endpoint on a dedicated port (default `:9090`).
- **HTTP Mode**: Metrics server runs on port `9090` (configurable via `-metrics-port`).
- **gRPC Mode**: Metrics server runs on port `9090` (configurable via `-metrics-port`).

Key metrics include:
- `http_request_duration_seconds`: Histogram of response latency.
- `http_requests_total`: Counter of HTTP requests by status, method, and path.
- Standard Go and Process metrics.

A `/health` endpoint is available on the main application port (default `:8080`) for readiness probes.


## OTEL based observability
The OS level env variable `ENABLE_OTEL_TRACING=="true"/"false"` (string type) is used to enable/disable this service (with Tracing only).
You can also configure the service name using `OTEL_SERVICE_NAME` env variable (defaults to "FeastGoFeatureServer").

The default exporter URL is "http://localhost:4318". The default schema of sending data to collector is **HTTP**. Please refer the following two docs about the configuration of the OTEL exporter:
1. https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/
Expand Down
66 changes: 66 additions & 0 deletions go/internal/feast/metrics/metrics.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package metrics

import (
"reflect"
"time"

"github.com/cabify/gotoprom"
"github.com/cabify/gotoprom/prometheusvanilla"
"github.com/prometheus/client_golang/prometheus"
)

var HttpMetrics struct {
Duration func(HttpLabels) TimeHistogram `name:"http_request_duration_seconds" help:"Time taken to serve HTTP requests" buckets:".005,.01,.025,.05,.1,.25,.5,1,2.5,5,10"`

RequestsTotal func(HttpLabels) prometheus.Counter `name:"http_requests_total" help:"Total number of HTTP requests"`
}

type HttpLabels struct {
Method string `label:"method"`
Status int `label:"status"`
Path string `label:"path"`
}

func init() {
gotoprom.MustAddBuilder(TimeHistogramType, RegisterTimeHistogram)
gotoprom.MustInit(&HttpMetrics, "feast")
}

var (
TimeHistogramType = reflect.TypeOf((*TimeHistogram)(nil)).Elem()
)


func RegisterTimeHistogram(name, help, namespace string, labelNames []string, tag reflect.StructTag) (func(prometheus.Labels) interface{}, prometheus.Collector, error) {
f, collector, err := prometheusvanilla.BuildHistogram(name, help, namespace, labelNames, tag)
if err != nil {
return nil, nil, err
}

return func(labels prometheus.Labels) interface{} {
return timeHistogramAdapter{Histogram: f(labels).(prometheus.Histogram)}
}, collector, nil
}

// TimeHistogram offers the basic prometheus.Histogram functionality
type TimeHistogram interface {
prometheus.Histogram
// Duration observes the duration in seconds
Duration(duration time.Duration)
// Since observes the duration in seconds since the time point provided
Since(time.Time)
}

type timeHistogramAdapter struct {
prometheus.Histogram
}

// Duration observes the duration in seconds
func (to timeHistogramAdapter) Duration(duration time.Duration) {
to.Observe(duration.Seconds())
}

// Since observes the duration in seconds since the time point provided
func (to timeHistogramAdapter) Since(duration time.Time) {
to.Duration(time.Since(duration))
}
51 changes: 49 additions & 2 deletions go/internal/feast/server/http_server.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ import (
prototypes "github.com/feast-dev/feast/go/protos/feast/types"
"github.com/feast-dev/feast/go/types"
"github.com/rs/zerolog/log"

"github.com/feast-dev/feast/go/internal/feast/metrics"
)

type httpServer struct {
Expand Down Expand Up @@ -335,10 +337,55 @@ func recoverMiddleware(next http.Handler) http.Handler {
})
}

type statusWriter struct {
http.ResponseWriter
status int
}

func (w *statusWriter) WriteHeader(status int) {
if w.status == 0 {
w.status = status
}
w.ResponseWriter.WriteHeader(status)
}

func (w *statusWriter) Write(b []byte) (int, error) {
if w.status == 0 {
w.status = 200
}
n, err := w.ResponseWriter.Write(b)
return n, err
}

func metricsMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t0 := time.Now()
sw := &statusWriter{ResponseWriter: w}
next.ServeHTTP(sw, r)
duration := time.Since(t0)

if sw.status == 0 {
sw.status = 200
}

metrics.HttpMetrics.Duration(metrics.HttpLabels{
Method: r.Method,
Status: sw.status,
Path: r.URL.Path,
}).Duration(duration)

metrics.HttpMetrics.RequestsTotal(metrics.HttpLabels{
Method: r.Method,
Status: sw.status,
Path: r.URL.Path,
}).Inc()
})
}

func (s *httpServer) Serve(host string, port int) error {
mux := http.NewServeMux()
mux.Handle("/get-online-features", recoverMiddleware(http.HandlerFunc(s.getOnlineFeatures)))
mux.HandleFunc("/health", healthCheckHandler)
mux.Handle("/get-online-features", metricsMiddleware(recoverMiddleware(http.HandlerFunc(s.getOnlineFeatures))))
mux.Handle("/health", metricsMiddleware(http.HandlerFunc(healthCheckHandler)))
s.server = &http.Server{Addr: fmt.Sprintf("%s:%d", host, port), Handler: mux, ReadTimeout: 5 * time.Second, WriteTimeout: 10 * time.Second, IdleTimeout: 15 * time.Second}
err := s.server.ListenAndServe()
// Don't return the error if it's caused by graceful shutdown using Stop()
Expand Down
Loading
Loading

Back | FazBrowse Home | New Git URL