Autoscale vLLM model serving with GPU metrics Stay organized with collections Save and categorize content based on your preferences.

Download and upload model weights to Cloud Storage

Download model weights from Hugging Face and transfer them to a Cloud Storage bucket to make them available for model serving:

1. Install the Hugging Face CLI:

   pip install -U "huggingface_hub[cli]"
   

2. Download the model weights locally using the Hugging Face CLI:

   export HF_TOKEN="HF_TOKEN"
   export LOCAL_DIR="/tmp/$MODEL_NAME"
   HF_HOME=/tmp/huggingface python -m huggingface_hub.cli.hf download google/$MODEL_NAME --token $HF_TOKEN --local-dir=$LOCAL_DIR
   

   Replace HF_TOKEN with your Hugging Face user access token. The token should begin with hf_ followed by 35 random alphanumeric characters (for example, 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).

3. Create a Cloud Storage bucket and copy the downloaded weights:

   gcloud storage buckets create gs://$BUCKET_NAME \
       --project=$PROJECT_ID \
       --location=$REGION \
       --uniform-bucket-level-access
   
   gcloud storage cp -r $LOCAL_DIR gs://$BUCKET_NAME/
   

Push the vLLM container image to Artifact Registry

Pull your model serving container images and push them to a repository in Artifact Registry:

1. Create a Docker repository in Artifact Registry:

   gcloud artifacts repositories create $REPO_NAME \
       --repository-format=docker \
       --location=$REGION \
       --description="vLLM Docker Images"
   

2. Authenticate local Docker daemon with the registry:

   gcloud auth configure-docker ${REGION}-docker.pkg.dev
   

3. Pull the vLLM image, tag it, and push it to Artifact Registry:

   docker pull vllm/vllm-openai:v0.6.3
   docker tag vllm/vllm-openai:v0.6.3 ${REGION}-docker.pkg.dev/$PROJECT_ID/$REPO_NAME/vllm-openai:v0.6.3
   docker push ${REGION}-docker.pkg.dev/$PROJECT_ID/$REPO_NAME/vllm-openai:v0.6.3
   

Deploy the CREMA autoscaler service

Configure service account roles and parameter manifests for CREMA before deploying the autoscaler service.

Create a custom service account

Create a custom service account with minimum permissions required to use the provisioned resources. This service account acts as the identity for the autoscaler. Run the following command to create the CREMA service account:

export CREMA_SA="crema-autoscaler@${PROJECT_ID}.iam.gserviceaccount.com"

gcloud iam service-accounts create crema-autoscaler \
    --description="Service account for Cloud Run CREMA to read metrics and scale workloads" \
    --display-name="CREMA Autoscaler System"

Grant additional permissions to your custom service account

To scale the service, grant the following permissions on the custom service account:

1. Grant your CREMA service account permission to read from the Parameter Manager:

   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member="serviceAccount:$CREMA_SA" \
       --role="roles/parametermanager.parameterViewer"
   

2. Grant your CREMA service account permission to scale the service:

   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member="serviceAccount:$CREMA_SA" \
       --role="roles/run.developer"
   

3. Grant your CREMA service account the service account user role:

   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member="serviceAccount:$CREMA_SA" \
       --role="roles/iam.serviceAccountUser"
   

4. Grant your CREMA service account permission to view metrics:

   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member="serviceAccount:$CREMA_SA" \
       --role="roles/monitoring.viewer"
   

Create and register the CREMA configuration

Define scaling thresholds and rules in a CREMA configuration manifest and register it in Parameter Manager:

1. Save the following configuration as my-crema-config.yaml. This configuration triggers scaling when the number of running requests (vllm:num_requests_running) exceeds 2:

   apiVersion: crema/v1
   kind: CremaConfig
   spec:
     pollingInterval: 15
     triggerAuthentications:
       - metadata:
           name: adc-trigger-auth
         spec:
           podIdentity:
             provider: gcp
     scaledObjects:
       - spec:
           scaleTargetRef:
             name: projects/PROJECT_ID/locations/us-central1/services/vllm-service
           minReplicaCount: 1
           maxReplicaCount: 5
           triggers:
             - type: prometheus
               authenticationRef:
                 name: adc-trigger-auth
               metadata:
                 serverAddress: https://monitoring.googleapis.com/v1/projects/PROJECT_ID/location/global/prometheus
                 metric: vllm:num_requests_running
                 query: sum(vllm:num_requests_running)
                 threshold: '2'
   

2. Register the configuration file in Parameter Manager:

   gcloud parametermanager parameters create crema-config \
       --location=global \
       --parameter-format=YAML
   
   gcloud parametermanager parameters versions create 1 \
       --location=global \
       --parameter=crema-config \
       --payload-data-from-file=my-crema-config.yaml
   

Deploy the CREMA service

Deploy the CREMA image as an internal background service on Cloud Run:

gcloud run deploy $CREMA_SERVICE_NAME \
    --image=us-central1-docker.pkg.dev/cloud-run-oss-images/crema-v1/autoscaler:1.0 \
    --region=$REGION \
    --service-account="$CREMA_SA" \
    --no-allow-unauthenticated \
    --no-cpu-throttling \
    --cpu=1 \
    --memory=1Gi \
    --min-instances=1 \
    --max-instances=1 \
    --ingress=internal \
    --base-image=us-central1-docker.pkg.dev/serverless-runtimes/google-24/runtimes/java25 \
    --set-env-vars="CREMA_CONFIG=projects/$PROJECT_ID/locations/global/parameters/crema-config/versions/1,OUTPUT_SCALER_METRICS=True"

Configure vLLM service permissions

Grant the default Compute Engine service account permissions to export metrics and read model weights from Cloud Storage:

1. Retrieve your project number:

   export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')
   

2. Grant your service account permission to write metrics:

   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member="serviceAccount:$PROJECT_NUMBER-compute@developer.gserviceaccount.com" \
       --role="roles/monitoring.metricWriter"
   

3. Grant your service account permission to read model weights from Cloud Storage:

   gcloud projects add-iam-policy-binding $PROJECT_ID \
       --member="serviceAccount:$PROJECT_NUMBER-compute@developer.gserviceaccount.com" \
       --role="roles/storage.objectViewer"
   

Deploy vLLM service with OpenTelemetry sidecar

Deploy your primary vLLM serving container to Cloud Run with model weights mounted from Cloud Storage. Because deploying multi-container services with sidecars on Cloud Run requires a declarative YAML service specification, you configure the primary vLLM engine alongside an OpenTelemetry sidecar collector to scrape and export vLLM metrics:

1. Save the following multi-container deployment specification as vllm-service.yaml:

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: vllm-service
     labels:
       cloud.googleapis.com/location: us-central1
     annotations:
       run.googleapis.com/scalingMode: manual
       run.googleapis.com/manualInstanceCount: "1"
   spec:
     template:
       metadata:
         annotations:
           run.googleapis.com/execution-environment: gen2
           run.googleapis.com/cpu-throttling: "false"
           run.googleapis.com/gpu-zonal-redundancy-disabled: "true"
           autoscaling.knative.dev/minScale: "1"
       spec:
         containerConcurrency: 80
         nodeSelector:
           run.googleapis.com/accelerator: nvidia-l4
         volumes:
           - name: gcs-volume
             csi:
               driver: gcsfuse.run.googleapis.com
               volumeAttributes:
                 bucketName: my-vllm-models-PROJECT_ID
         containers:
           # Primary container: vLLM serving engine
           - name: vllm-container
             image: us-central1-docker.pkg.dev/PROJECT_ID/vllm-repo/vllm-openai:latest
             ports:
               - containerPort: 8080
             resources:
               limits:
                 cpu: "4"
                 memory: 16Gi
                 nvidia.com/gpu: "1"
             args:
               - "--model"
               - "/gcs/gemma-2-2b-it"
               - "--port"
               - "8080"
               - "--max-model-len"
               - "2048"
               - "--chat-template"
               - "{% for msg in messages %}{{ msg['content'] }}{% endfor %}"
             volumeMounts:
               - name: gcs-volume
                 mountPath: /gcs
             startupProbe:
               httpGet:
                 path: /health
                 port: 8080
               periodSeconds: 10
               failureThreshold: 24
   
           # Sidecar container: OpenTelemetry Collector
           - name: otel-collector
             image: otel/opentelemetry-collector-contrib:latest
             resources:
               limits:
                 cpu: "1"
                 memory: 1Gi
             args:
               - |
                 --config=yaml:
                 receivers:
                   prometheus:
                     config:
                       scrape_configs:
                         - job_name: 'vllm'
                           scrape_interval: 10s
                           metrics_path: '/metrics'
                           static_configs:
                             - targets: ['localhost:8080']
                 processors:
                   resourcedetection:
                     detectors: [gcp]
                     timeout: 2s
                   transform:
                     metric_statements:
                       - context: datapoint
                         statements:
                           - set(attributes["exported_location"], attributes["location"])
                           - delete_key(attributes, "location")
                           - set(attributes["exported_cluster"], attributes["cluster"])
                           - delete_key(attributes, "cluster")
                           - set(attributes["exported_namespace"], attributes["namespace"])
                           - delete_key(attributes, "namespace")
                           - set(attributes["exported_job"], attributes["job"])
                           - delete_key(attributes, "job")
                           - set(attributes["exported_instance"], attributes["instance"])
                           - delete_key(attributes, "instance")
                 exporters:
                   googlemanagedprometheus:
                 service:
                   pipelines:
                     metrics:
                       receivers: [prometheus]
                       processors: [resourcedetection, transform]
                       exporters: [googlemanagedprometheus]
   

2. Replace the existing service configuration with the multi-container manifest:

   gcloud run services replace vllm-service.yaml
   

Check CREMA service logs

1. In the Google Cloud console, navigate to the Cloud Run page.
2. Select your crema-service.

3. Click the Logs tab and verify that metric polling cycles are active:

   [INFO] [METRIC-PROVIDER] Starting metric collection cycle
   [INFO] [METRIC-PROVIDER] Successfully fetched scaled object metrics ...
   [INFO] [METRIC-PROVIDER] Sending scale request ...
   [INFO] [SCALER] Received ScaleRequest ...
   [INFO] [SCALER] Current instances ...
   [INFO] [SCALER] Recommended instances ...
   

Run a load test

To test autoscaling, run a load testing script to send concurrent requests to the vLLM service:

1. In your working directory, create a file named load-test.sh and add the following code:

   #!/bin/bash
   
   export SERVICE_URL=$(gcloud run services describe $VLLM_SERVICE_NAME --region $REGION --format='value(status.url)')
   
   echo "Launching 5 parallel heavy requests to trigger autoscaling..."
   
   for i in {1..5}; do
       curl -s -X POST "${SERVICE_URL}/v1/chat/completions" \
           -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
           -H "Content-Type: application/json" \
           -d "{
               \"model\": \"/gcs/${MODEL_NAME}\",
               \"messages\": [{\"role\": \"user\", \"content\": \"Write an exceptionally long, detailed, and exhaustive essay about the entire history of the universe from the Big Bang to the modern day.\"}]
           }" > /dev/null &
   done
   
   echo "All 5 requests dispatched. Waiting for requests to complete..."
   wait
   echo "Done."
   

2. Make the script executable and run the load test:

   chmod +x load-test.sh
   ./load-test.sh
   

3. Re-check the crema-service logs and Cloud Run metrics dashboard to verify that the recommended instance count scales up in response to the queued requests.

Explore vLLM metrics in Cloud Monitoring

After running the load test, explore how traffic affects the model serving metrics in Cloud Monitoring:

1. In the Google Cloud console, go to the Metrics Explorer page in Cloud Monitoring.

   Go to Metrics Explorer

2. Click Select a metric.

3. Expand Prometheus Target > Vllm and select any of the available metrics ending in /gauge. For example, select prometheus/vllm:num_requests_running/gauge to view the active request count during the load test.

As described in the vLLM production metrics documentation, additional vLLM metrics exported to Cloud Monitoring include:

• prometheus/vllm:num_requests_waiting/gauge: The number of requests waiting in the queue to be processed by the vLLM engine.
• prometheus/vllm:num_requests_running/gauge: The number of requests executing in model batches.
• prometheus/vllm:gpu_cache_usage_perc/gauge: The percentage of GPU KV-cache memory utilized.
• prometheus/vllm:num_requests_swapped/gauge: The number of requests whose KV cache was swapped to host CPU memory due to memory pressure.

Although this tutorial scales based on vllm:num_requests_running, you can use any of these vLLM metrics in your CREMA configuration to customize autoscaling rules for your workloads based on queue size, KV-cache utilization, or request swapping.

Unlike standard HTTP request concurrency metrics that treat all requests equally, vLLM's internal metrics account for the dynamic GPU memory footprint of different prompt lengths. Scaling on vllm:num_requests_running helps you scale proactively based on genuine GPU load. This maintains an active capacity buffer before the server is forced to queue requests into vllm:num_requests_waiting, protecting users from severe Time To First Token (TTFT) latency spikes.