[ Web Proxy ]
URL:
Viewing: https://cloud.google.com/stackdriver/docs/managed-prometheus/hpa [Back]  [Original]
Horizontal pod autoscaling (HPA)  |  Google Cloud Observability  |  Google Cloud Documentation Skip to main content
Google Cloud Documentation [Google Cloud Documentation]
Technology areas Cross-product tools
Search:
Console
[]
Start free
Overview Guides Reference Resources
Google Cloud Documentation [Google Cloud Documentation]
Send feedback

Horizontal pod autoscaling (HPA) Stay organized with collections Save and categorize content based on your preferences.

This document describes how to enable horizontal pod autoscaling (HPA) for Google Cloud Managed Service for Prometheus. You can enable HPA by doing one of the following:

You can't use the Stackdriver Adapter and the Prometheus Adapter together in the same cluster because their resource definitions overlap, as described in Troubleshooting. We recommend choosing only one solution for HPA.

Use KEDA

KEDA (Kubernetes Event-driven Autoscaling) is the most recently released autoscaler that uses Prometheus metrics, and it is becoming a favored solution within the Prometheus community.

To get started, see the KEDA documentation for integrating with Google Cloud Managed Service for Prometheus.

Use the Custom Metrics Stackdriver Adapter

The Custom Metrics Stackdriver Adapter supports querying metrics from Managed Service for Prometheus starting with version v0.13.1 of the adapter.

To set up an example HPA configuration using the Custom Metrics Stackdriver Adapter, do the following:

  1. Set up managed collection in your cluster.

  2. Install Custom Metrics Stackdriver Adapter in your cluster.

    kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml

  3. Deploy an example Prometheus metrics exporter and an HPA resource:

    kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/examples/prometheus-to-sd/custom-metrics-prometheus-sd.yaml

    This command deploys an exporter application that emits the metric foo and an HPA resource. The HPA scales this application up to 5 replicas to achieve the target value for the metric foo.

  4. If you use Workload Identity Federation for GKE, you must also grant the Monitoring Viewer role to the service account the adapter runs under. Skip this step if you do not have Workload Identity Federation for GKE enabled on your Kubernetes cluster.

    export PROJECT_NUMBER=$(gcloud projects describe PROJECT_ID --format 'get(projectNumber)')
gcloud projects add-iam-policy-binding projects/PROJECT_ID \
  --role roles/monitoring.viewer \
  --member=principal://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/custom-metrics/sa/custom-metrics-stackdriver-adapter

  5. Define a PodMonitoring resource by placing the following configuration in a file named podmonitoring.yaml.

    apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: prom-example
spec:
  selector:
    matchLabels:
      run: custom-metric-prometheus-sd
  endpoints:
  - port: 8080
    interval: 30s

  6. Deploy the new PodMonitoring resource:

    kubectl -n default apply -f podmonitoring.yaml

    Within a couple of minutes, Managed Service for Prometheus processes the metrics scraped from the exporter and stores them in Cloud Monitoring using a long-form name. Prometheus metrics are stored with the following conventions:

    • The prefix prometheus.googleapis.com.
    • This suffix is usually one of gauge, counter, summary, or histogram, although untyped metrics might have the unknown or unknown:counter suffix. To verify the suffix, look up the metric in Cloud Monitoring by using Metrics Explorer.

  7. Update the deployed HPA to query the metric from Cloud Monitoring. The metric foo is ingested as prometheus.googleapis.com/foo/gauge. To make the metric queryable by the deployed HorizontalPodAutoscaler resource, you use the long-form name in the deployed HPA, but you have to modify it by replacing the all forward slashes (/) with the pipe character (|): prometheus.googleapis.com|foo|gauge. For more information, see the Metrics available from Stackdriver section of the Custom Metrics Stackdriver Adapter repository.

    1. Update the deployed HPA by running the following command:

      kubectl edit hpa custom-metric-prometheus-sd

    2. Change the value of the pods.metric.name field from foo to prometheus.googleapis.com|foo|gauge. The spec section should look like the following:

      spec:
   maxReplicas: 5
   metrics:
   - pods:
       metric:
         name: prometheus.googleapis.com|foo|gauge
       target:
         averageValue: "20"
         type: AverageValue
     type: Pods
   minReplicas: 1

    In this example, the HPA configuration looks for the average value of the metric prometheus.googleapis.com/foo/gauge to be 20. Because the Deployment sets the value of the metric is 40, the HPA controller increases the number of pods up to the value of the maxReplicas (5) field to try to reduce the average value of the metric across all pods to 20.

    The HPA query is scoped to the namespace and cluster in which the HPA resource is installed, so identical metrics in other clusters and namespaces don't affect your autoscaling.

  8. To watch the workload scale up, run the following command:

    kubectl get hpa custom-metric-prometheus-sd --watch

    The value of the REPLICAS field changes from 1 to 5.

    NAME                          REFERENCE                                TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
custom-metric-prometheus-sd   Deployment/custom-metric-prometheus-sd   40/20          1         5         5          *

  9. To scale down the deployment, update the target metric value to be higher than the exported metric value. In this example, the Deployment sets the value of the prometheus.googleapis.com/foo/gauge metric to 40. If you set the target value to a number that is higher than 40, then the deployment will scale down.

    For example, use kubectl edit to change the value of the pods.target.averageValue field in the HPA configuration from 20 to 100.

    kubectl edit hpa custom-metric-prometheus-sd

    Modify the spec section to match the following:

    spec:
  maxReplicas: 5
  metrics:
  - pods:
      metric:
        name: prometheus.googleapis.com|foo|gauge
      target:
        averageValue: "100"
        type: AverageValue
  type: Pods
  minReplicas: 1

  10. To watch the workload scale down, run the following command:

    kubectl get hpa custom-metric-prometheus-sd --watch

    The value of the REPLICAS field changes from 5 to 1. By design, this happens more slowly than when scaling the number of pods up:

    NAME                          REFERENCE                                TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
custom-metric-prometheus-sd   Deployment/custom-metric-prometheus-sd   40/100          1         5         1          *

  11. To clean up the deployed example, run the following commands:

    kubectl delete -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml
kubectl delete -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/examples/prometheus-to-sd/custom-metrics-prometheus-sd.yaml
kubectl delete podmonitoring/prom-example

For more information, see the Prometheus example in the Custom Metrics Stackdriver Adapter repository, or see Scaling an application.

Use the Prometheus Adapter

Existing prometheus-adapter configs can be used to autoscale with only a few changes. Configuring prometheus-adapter to scale using Managed Service for Prometheus has two additional restrictions compared to scaling using upstream Prometheus:

As data can take slightly longer to be available within Managed Service for Prometheus compared to upstream Prometheus, configuring overly-eager autoscaling logic might cause unwanted behavior. Although there is no guarantee on data freshness, data is typically available to query 3-7 seconds after it is sent to Managed Service for Prometheus, excluding any network latency.

All queries issued by prometheus-adapter are global in scope. This means that if you have applications in two namespaces that emit identically named metrics, an HPA configuration using that metric scales using data from both applications. We recommend always using namespace or cluster filters in your PromQL to avoid scaling using incorrect data.

To set up an example HPA configuration using prometheus-adapter and managed collection, use the following steps:

  1. Set up managed collection in your cluster.
  2. Deploy the Prometheus frontend UI proxy in your cluster. If you use Workload Identity Federation for GKE, you must also configure and authorize a service account.
  3. Deploy the manifests in the examples/hpa/ directory within the prometheus-engine repo:
    • example-app.yaml: An example deployment and service that emits metrics.
    • pod-monitoring.yaml: A resource that configures scraping the example metrics.
    • hpa.yaml: The HPA resource that configures scaling for your workload.

  4. Make sure prometheus-adapter is installed in your cluster. This can be done by deploying the example install manifest to your cluster. This manifest is configured to:

    • Query a frontend proxy deployed in the default namespace.
    • Issue PromQL to calculate and return the http_requests_per_second metric from the example deployment.
    Note: The http_requests_per_second metric won't be available until load is generated against the example application.

  5. Run the following commands, each in a separate terminal session:

    1. Generate HTTP load against the prometheus-example-app service: 
      kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://prometheus-example-app; done"
    2. Watch the horizontal pod autoscaler: 
      kubectl get hpa prometheus-example-app --watch
    3. Watch the workload scale up: 
      kubectl get po -lapp.kubernetes.io/name=prometheus-example-app --watch

  6. Stop HTTP load generation by using Ctrl+C and watch the workload scale back down.

Troubleshooting

Custom Metrics Stackdriver Adapter uses resource definitions with the same names as those in the Prometheus Adapter, prometheus-adapter. This overlap in names means that running more than one adapter in the same cluster causes errors.

Installing the Prometheus Adapter in a cluster that previously had the Custom Metrics Stackdriver Adapter installed might throw errors such as FailedGetObjectMetric due to colliding names. To resolve this, you might have to delete the v1beta1.external.metrics.k8s.io, v1beta1.custom.metrics.k8s.io, and v1beta2.custom.metrics.k8s.io apiservices previously registered by the Custom Metrics Adapter.

Troubleshooting tips:

Send feedback

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2026-08-12 UTC.

Need to tell us more? [[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2026-08-12 UTC."],[],[]]
Web Proxy Viewer  |  New URL  |  Original Page