[ Web Proxy ]
URL:
Viewing: https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting [Back]  [Original]
Troubleshooting Managed Service for Prometheus  |  Google Cloud Observability  |  Google Cloud Documentation Skip to main content
Google Cloud Documentation [Google Cloud Documentation]
Technology areas Cross-product tools
Search:
Console
[]
Start free
Overview Guides Reference Resources
Google Cloud Documentation [Google Cloud Documentation]
Send feedback

Troubleshooting Managed Service for Prometheus Stay organized with collections Save and categorize content based on your preferences.

This document describes some problems you might encounter when using Google Cloud Managed Service for Prometheus and provides information on diagnosing and resolving the problems.

You configured Managed Service for Prometheus but are not seeing any metric data in Grafana or the Prometheus UI. At a high level, the cause might be either of the following:

To determine whether the problem is on the ingestion side or the query side, try querying data by using the Metrics Explorer PromQL tab in the Google Cloud console. This page is guaranteed not to have any issues with read permissions or Grafana settings.

To view this page, do the following:

  1. Use the Google Cloud console project picker to select the project for which you are not seeing data.

  2. In the Google Cloud console, go to the  Metrics explorer page:

    Go to Metrics explorer

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  3. In the toolbar of the query-builder pane, select the button whose name is  PromQL.

  4. Enter the following query into the editor, and then click Run query:

    up

If you query the up metric and see results, then the problem is on the query side. For information on resolving these problems, see Query-side problems.

If you query the up metric and do not see any results, then the problem is on the ingestion side. For information on resolving these problems, see Ingestion-side problems.

A firewall can also cause ingestion and query problems; for more information, see Firewalls.

The Cloud Monitoring Metrics Management page provides information that can help you control the amount you spend on billable metrics without affecting observability. The Metrics Management page reports the following information:

You can also use the Metrics Management page to exclude unneeded metrics, eliminating the cost of ingesting them.

To view the Metrics Management page, do the following:

  1. In the Google Cloud console, go to the  Metrics management page:

    Go to Metrics management

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  2. In the toolbar, select your time window. By default, the Metrics Management page displays information about the metrics collected in the previous one day.

For more information about the Metrics Management page, see View and manage metric usage.

Query-side problems

The cause of most query-side problems is one of the following:

Start by doing the following:

If the problem still persists, then consider the following possibilities:

Misconfigured or mistyped secrets

If you see any of the following, then you might have a missing or mistyped secret:

If you are using the data source syncer to authenticate and configure Grafana, try the following to resolve these errors:

  1. Verify that you have chosen the correct Grafana API endpoint, Grafana data source UID, and Grafana API token. You can inspect the variables in the CronJob by running the command kubectl describe cronjob datasource-syncer.

  2. Verify that you have set the data source syncer's project ID to the same metrics scope or project that your service account has credentials for.

  3. Verify that your Grafana service account has the "Admin" role and that your API token has not expired.

  4. Verify that your service account has the Monitoring Viewer role for the chosen project ID.

  5. Verify that there are no errors in the logs for the data source syncer Job by running kubectl logs job.batch/datasource-syncer-init. This command has to be run immediately after applying the datasource-syncer.yaml file.

  6. If using Workload Identity Federation for GKE, verify that you have not mistyped the account key or credentials, and verify that you have bound it to the correct namespace.

If you are using the legacy frontend UI proxy, try the following to resolve these errors:

  1. Verify that you have set the frontend UI's project ID to the same metrics scope or project that your service account has credentials for.

  2. Verify the project ID you've specified for any --query.project-id flags.

  3. Verify that your service account has the Monitoring Viewer role for the chosen project ID.

  4. Verify you have set the correct project ID when deploying the frontend UI and did not leave it set to the literal string PROJECT_ID.

  5. If using Workload Identity, verify that you have not mistyped the account key or credentials, and verify that you have bound it to the correct namespace.

  6. If mounting your own secret, make sure the secret is present:

    kubectl get secret gmp-test-sa -o json | jq '.data | keys'

  7. Verify that the secret is correctly mounted:

    kubectl get deploy frontend -o json | jq .spec.template.spec.volumes

kubectl get deploy frontend -o json | jq .spec.template.spec.containers[].volumeMounts

  8. Make sure the secret is passed correctly to the container: 

    kubectl get deploy frontend -o json | jq .spec.template.spec.containers[].args

Incorrect HTTP method for Grafana

If you see the following API error from Grafana, then Grafana is configured to send a POST request instead of a GET request:

To resolve this issue, configure Grafana to use a GET request by following the instructions in Configure a data source.

Timeouts on large or long-running queries

If you see the following error in Grafana, then your default query timeout is too low:

Managed Service for Prometheus does not time out until a query exceeds 120 seconds, while Grafana times out after 30 seconds by default. To fix this, raise the timeouts in Grafana to 120 seconds by following the instructions in Configure a data source.

Label-validation errors

If you see one of the following errors in Grafana, then you might be using an unsupported endpoint:

Managed Service for Prometheus supports the /api/v1/$label/values endpoint only for the __name__ label. This limitation causes queries using the label_values($label) variable in Grafana to fail.

Instead, use the label_values($metric, $label) form. This query is recommended because it constrains the returned label values by metric, which prevents retrieval of values not related to the dashboard's contents. This query calls a supported endpoint for Prometheus.

For more information about supported endpoints, see API compatibility.

Quota exceeded

If you see the following error, then you have exceeded your read quota for the Cloud Monitoring API:

To resolve this issue, submit a request to increase your read quota for the Monitoring API. For assistance, contact Google Cloud Support. For more information about quotas, see the Cloud Quotas documentation.

Metrics from multiple projects

If you want to view metrics from multiple Google Cloud projects, you don't have to configure multiple data source syncers or create multiple data sources in Grafana.

Instead, create a Cloud Monitoring metrics scope in one Google Cloud project — the scoping project — that contains the projects you want to monitor. When you configure the Grafana data source with a scoping project, you get access to the data from all projects in the metrics scope. For more information, see Queries and metrics scopes.

No monitored resource type specified

If you see the following error, then you need to specify a monitored resource type when using PromQL to query a Google Cloud system metric:

You can specify a monitored resource type by filtering using the monitored_resource label. For more information about identifying and choosing a valid monitored resource type, see Specifying a monitored resource type.

Counter, histogram, and summary raw values not matching between the collector UI and the Google Cloud console

You might notice a difference between the values in the local collector Prometheus UI and the Google Cloud Google Cloud console when querying the raw value of cumulative Prometheus metrics, including counters, histograms, and summaries. This behavior is expected.

Monarch requires start timestamps, but Prometheus doesn't have start timestamps. Managed Service for Prometheus generates start timestamps by skipping the first ingested point in any time series and converting it into a start timestamp. Subsequent points have the value of the initial skipped point subtracted from their value to ensure rates are correct. This causes a persistent deficit in the raw value of those points.

The difference between the number in the collector UI and the number in the Google Cloud console is equal to the first value recorded in the collector UI, which is expected because the system skips that initial value, and subtracts it from subsequent points.

This is acceptable because there's no production need for running a query for raw values for cumulative metrics; all useful queries require a rate() function or the like, in which case the difference over any time horizon is identical between the two UIs. Cumulative metrics only ever increase, so you can't set an alert on a raw query as a time series only ever hits a threshold one time. All useful alerts and charts look at the change or the rate of change in the value.

The collector only holds about 10 minutes of data locally. Discrepancies in raw cumulative values might also arise due to a reset happening before the 10 minute horizon. To rule out this possibility, try setting only a 10 minute query lookback period when comparing the collector UI to the Google Cloud console.

Discrepancies can also be caused by having multiple worker threads in your application, each with a /metrics endpoint. If your application spins up multiple threads, you have to put the Prometheus client library in multiprocess mode. For more information, see the documentation for using multiprocess mode in Prometheus' Python client library.

Missing counter data or broken histograms

The most common signal of this problem is seeing no data or seeing data gaps when querying a plain counter metric (for example, a PromQL query of metric_name_foo). You can confirm this if data appears after you add a rate function to your query (for example, rate(metric_name_foo[5m])).

You might also notice that your samples ingested has risen sharply without any major change in scrape volume or that new metrics are being created with "unknown" or "unknown:counter" suffixes in Cloud Monitoring.

You might also notice that histogram operations, such as the quantile() function, don't work as expected.

These issues occur when a metric is collected without a Prometheus metric TYPE. As Monarch is strongly typed, Managed Service for Prometheus accounts for untyped metrics suffixing them with "unknown" and ingesting them twice, once as a gauge and once as a counter. The query engine then chooses whether to query the underlying gauge or counter metric based on what query functions you use.

While this heuristic usually works quite well, it can lead to issues such as strange results when querying a raw "unknown:counter" metric. Also, as histograms are specifically typed objects in Monarch, ingesting the three required histogram metrics as individual counter metrics causes histogram functions to not work. As "unknown"-typed metrics are ingested twice, not setting a TYPE doubles your samples ingested.

Common reasons why TYPE might not be set include:

To resolve this issue, do the following:

You can also create a metric-exclusion rule within Metrics Management to prevent any "unknown"-suffixed metrics from being ingested by using the regular expression prometheus.googleapis.com/.+/unknown.*. If you don't fix the underlying issue before installing this rule, you might prevent wanted metric data from being ingested.

Grafana data not persisted after pod restart

If your data appears to vanish from Grafana after a pod restart but is visible in Cloud Monitoring, then you are using Grafana to query the local Prometheus instance instead of Managed Service for Prometheus.

For information about configuring Grafana to use the managed service as a data source, see Grafana.

Inconsistent query or alert rule results that automatically fix themselves

You might notice a pattern where queries over recent windows, such as queries run by recording or alerting rules, return unexplainable spikes in data. When you investigate the spike by running the query in Grafana or Metrics Explorer, you might see that the spike has disappeared and the data looks normal again.

This behavior might happen more often if any of the following are true:

There are a few possible causes for temporary spikes in these kinds of queries:

To account for these possible root causes, you should ensure your queries, rules, and alerts follow these best practices:

These recommendations are mostly relevant if you are seeing inconsistent query results when querying recent data. If you see this issue happening when querying data that's over 25 hours old, then there might be a technical issue with Monarch. If this happens, contact Cloud Customer Care so we can investigate.

Importing Grafana dashboards

For information about using and troubleshooting the dashboard importer, see Import Grafana dashboards into Cloud Monitoring.

For information about problems with the conversion of the dashboard contents, see the importer's README file.

Ingestion-side problems

Ingestion-side problems can be related to either collection or rule evaluation. Start by looking at the error logs for managed collection. You can run the following commands:

kubectl logs -f -n gmp-system -lapp.kubernetes.io/part-of=gmp

kubectl logs -f -n gmp-system -lapp.kubernetes.io/name=collector -c prometheus

On GKE Autopilot clusters, you can run the following commands:

kubectl logs -f -n gke-gmp-system -lapp.kubernetes.io/part-of=gmp

kubectl logs -f -n gke-gmp-system -lapp.kubernetes.io/name=collector -c prometheus

The target status feature can help you debug your scrape target. For more information, see target status information.

Endpoint status is missing or too old

If you have enabled the target status feature but one or more of your PodMonitoring or ClusterPodMonitoring resources are missing the Status.Endpoint Statuses field or value, then you might have one of the following problems:

Similar problems can also cause the Status.Endpoint Statuses.Last Update Time field to have value older than a few minutes plus your scrape interval.

To resolve this issue, start by checking that the Kubernetes pods associated with your scrape endpoint are running. If your Kubernetes pods are running, the label selectors match, and you can manually access the scrape endpoints (typically by visiting the /metrics endpoint), then check whether the Managed Service for Prometheus collectors are running.

Collectors fraction is less than 1

If you have enabled the target status feature, then you get status information about your resources. The Status.Endpoint Statuses.Collectors Fraction value of your PodMonitoring or ClusterPodMonitoring resources represents the fraction of collectors, expressed from 0 to 1, that are reachable. For example, a value of 0.5 indicates that 50% of your collectors are reachable, while a value of 1 indicates that 100% of your collectors are reachable.

If the Collectors Fraction field has a value other than 1, then one or more collectors are unreachable, and metrics in any of those nodes are possibly not being scraped. Ensure that all collectors are running and reachable over the cluster network. You can view the status of collector pods with the following command:

kubectl -n gmp-system get pods --selector="app.kubernetes.io/name=collector"

On GKE Autopilot clusters, this command looks slightly different:

kubectl -n gke-gmp-system get pods --selector="app.kubernetes.io/name=collector"

You can investigate individual collector pods (for example, a collector pod named collector-12345) with the following command:

kubectl -n gmp-system describe pods/collector-12345

On GKE Autopilot clusters, run the following command:

kubectl -n gke-gmp-system describe pods/collector-12345

If collectors are not healthy, see GKE workload troubleshooting.

If the collectors are healthy, then check the operator logs. To check the operator logs, first run the following command to find the operator pod name:

kubectl -n gmp-system get pods --selector="app.kubernetes.io/name=gmp-collector"

On GKE Autopilot clusters, run the following command:

kubectl -n gke-gmp-system get pods --selector="app.kubernetes.io/name=gmp-collector"

Then, check the operator logs (for example, an operator pod named gmp-operator-12345) with the following command:

kubectl -n gmp-system logs pods/gmp-operator-12345

On GKE Autopilot clusters, run the following command:

kubectl -n gke-gmp-system logs pods/gmp-operator-12345

Unhealthy targets

If you have enabled the target status feature, but one or more of your PodMonitoring or ClusterPodMonitoring resources has the Status.Endpoint Statuses.Unhealthy Targets field with the value other than 0, then the collector cannot scrape one or more of your targets.

View the Sample Groups field, which groups targets by error message, and find the Last Error field. The Last Error field comes from Prometheus and tells you why the target was unable to be scraped. To resolve this issue, using the sample targets as a reference, check whether your scrape endpoints are running.

Unauthorized scrape endpoint

If you see one of the following errors and your scrape target requires authorization, then your collector is either not set up to use the correct authorization type or is using the incorrect authorization payload:

To resolve this issue, see Configuring an authorized scrape endpoint.

Quota exceeded

If you see the following error, then you have exceeded your ingestion quota for the Cloud Monitoring API:

This error is most commonly seen when first bringing up the managed service. The default quota exhausts at 100,000 samples per second ingested.

To resolve this issue, submit a request to increase your ingestion quota for the Monitoring API. For assistance, contact Google Cloud Support. For more information about quotas, see the Cloud Quotas documentation.

Missing permission on the node's default service account

If you see one of the following errors, then the default service account on the node might be missing permissions:

Managed collection and the managed rule evaluator in Managed Service for Prometheus both use the default service account on the node. This account is created with all the necessary permissions, but customers sometimes manually remove the Monitoring permissions. This removal causes collection and rule evaluation to fail.

To verify the permissions of the service account, do one of the following:

It's also possible that the data source syncer or the Prometheus UI has been configured to look at the wrong project. For information about verifying that you are querying the intended metrics scope, see Change the queried project.

Misconfigured service account

If you see one of the following error messages, then the service account used by the collector does not have the correct permissions:

To verify that your service account has the correct permissions, do the following:

  1. In the Google Cloud console, go to the IAM page:

    Go to IAM

    If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.

  2. Identify the service account name in the list of principals. Verify that the name of the service account is correctly spelled. Then click Edit.

  3. Select the Role field, then click Currently used and search for the Monitoring Metric Writer or the Monitoring Editor role. If the service account doesn't have one of these roles, then grant the service account the role Monitoring Metric Writer (roles/monitoring.metricWriter).

If you are running on non-GKE Kubernetes, then you must explicitly pass credentials to both the collector and the rule evaluator. You must repeat the credentials in both the rules and collection sections. For more information, see Provide credentials explicitly (for collection) or Provide credentials explicitly (for rules).

Service accounts are often scoped to a single Google Cloud project. Using one service account to write metric data for multiple projects — for example, when one managed rule evaluator is querying a multi-project metrics scope — can cause this permission error. If you are using the default service account, consider configuring a dedicated service account so that you can safely add the monitoring.timeSeries.create permission for several projects. If you can't grant this permission, then you can use metric relabeling to rewrite the project_id label to another name. The project ID then defaults to the Google Cloud project in which your Prometheus server or rule evaluator is running.

Invalid scrape configuration

If you see the following error, then your PodMonitoring or ClusterPodMonitoring is improperly formed:

To solve this, make sure your custom resource is properly formed according to the specification.

Metric paths with HTTP query parameters aren't scraped

You are trying to send a metric by using a path field that includes query parameters to Managed Service for Prometheus, but the metric isn't scraped. For example, your scrape configuration might include the following:

    path: /metrics/detailed?family=queue_metrics&family=queue_consumer_count

The reason the metric isn't scraped is that Prometheus URL-encodes the question-mark (?) character as %3F, so the data is sent to /metrics/detailed%3Ffamily=queue_metrics&family=queue_consumer_count instead.

To fix this problem, use the params field. For example, if the metric is /metrics/detailed?family=queue_metrics&family=queue_consumer_count, then set up the scrape configuration as follows:

    path: /metrics/detailed
    params:
      family: ['queue_metrics', 'queue_consumer_count']

Admission webhook unable to parse or invalid HTTP client config

On versions of Managed Service for Prometheus earlier than 0.12, you might see an error similar to the following, which is related to secret injection in the non-default namespace:

To solve this issue, upgrade to version 0.12 or later.

Problems with scrape intervals and timeouts

When using Managed Service for Prometheus, the scrape timeout can't be greater than the scrape interval. To check your logs for this problem, run the following command:

kubectl -n gmp-system logs ds/collector prometheus

On GKE Autopilot clusters, run the following command:

kubectl -n gke-gmp-system logs ds/collector prometheus

Look for this message:

To resolve this issue, set the value of the scrape interval equal to or greater than the value of the scrape timeout.

Missing TYPE on metric

If you see the following error, then the metric is missing type information:

To verify that missing type information is the problem, check the /metrics output of the exporting application. If there is no line like the following, then the type information is missing:

# TYPE {metric_name} <type>

Certain libraries, such as those from VictoriaMetrics older than version 1.28.0, intentionally drop the type information. These libraries are not supported by Managed Service for Prometheus.

Time-series collisions

If you see one of the following errors, you might have more than one collector attempting to write to the same time series:

The most common causes and solutions follow:

A less common cause of time-series collisions is using a scrape interval shorter than 5 seconds. The minimum scrape interval supported by Managed Service for Prometheus is 5 seconds.

Exceeding the limit on the number of labels

If you see the following error, then you might have too many labels defined for one of your metrics:

This error usually occurs when you rapidly change the definition of the metric so that one metric name effectively has multiple independent sets of label keys over the whole lifetime of your metric. The Cloud Monitoring imposes a limit on number of labels for each metric; for more information see the limits for user-defined metrics.

Note: The number of labels (also called label names, label keys, or dimensions) is different than cardinality. Cardinality refers to the number of combinations of unique label values across all labels.

There are three steps to resolve this problem:

  1. Identify why a given metric has too many or frequently changing labels.

  2. Address the source of the problem, which might involve adjusting your PodMonitoring's relabeling rules, changing the exporter, or fixing your instrumentation.

  3. Delete the metric descriptor for this metric (which incurs data loss), so it can be recreated with a smaller, more stable set of labels. You can use the metricDescriptors.delete method to do so.

The most common sources of the problem are:

Rate limits on creating and updating metrics and labels

If you see the following error, then you have hit the per-minute rate limit on creating new metrics and adding new metric labels to existing metrics:

This rate limit is usually only hit when first integrating with Managed Service for Prometheus, for example when you migrate an existing, mature Prometheus deployment to use self-deployed collection. This is not a rate limit on ingesting data points. This rate limit only applies when creating never-before-seen metrics or when adding new labels to existing metrics.

This quota is fixed, but any issues should automatically resolve as new metrics and metric labels get created up to the per-minute limit.

Limits on the number of metric descriptors

If you see the following error, then you have hit the quota limit for the number of metric descriptors within a single Google Cloud project:

By default, this limit is set to 25,000. Although this quota can be lifted by request if your metrics are well-formed, it is far more likely that you hit this limit because you are ingesting malformed metric names into the system.

Prometheus has a dimensional data model where information such as cluster or namespace name should get encoded as a label value. When dimensional information is instead embedded in the metric name itself, then the number of metric descriptors increases indefinitely. In addition, because in this scenario labels are not properly used, it becomes much more difficult to query and aggregate data across clusters, namespaces, or services.

Neither Cloud Monitoring nor Managed Service for Prometheus supports non-dimensional metrics, such as those formatted for StatsD or Graphite. While most Prometheus exporters are configured correctly out-of-the-box, certain exporters, such as the StatsD exporter, the Vault exporter, or the Envoy Proxy that comes with Istio, must be explicitly configured to use labels instead of embedding information in the metric name. Examples of malformed metric names include:

To confirm this issue, do the following:

  1. Within Google Cloud console, select the Google Cloud project that is linked to the error.

  2. In the Google Cloud console, go to the  Metrics management page:

    Go to Metrics management

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  3. Confirm that the sum of Active plus Inactive metrics is over 25,000. In most situations, you should see a large number of Inactive metrics.
  4. Select "Inactive" in the Quick Filters panel, page through the list, and look for patterns.
  5. Select "Active" in the Quick Filters panel, sort by Samples billable volume descending, page through the list, and look for patterns.
  6. Sort by Samples billable volume ascending, page through the list, and look for patterns.

Alternatively, you can confirm this issue by using Metrics Explorer:

  1. Within Google Cloud console, select the Google Cloud project that is linked to the error.

  2. In the Google Cloud console, go to the  Metrics explorer page:

    Go to Metrics explorer

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  3. In the query builder, click select a metric, then clear the "Active" checkbox.
  4. Type "prometheus" into the search bar.
  5. Look for any patterns in the names of metrics.

Once you have identified the patterns that indicate malformed metrics, you can mitigate the issue by fixing the exporter at the source and then deleting the offending metric descriptors.

To prevent this issue from happening again, you must first configure the relevant exporter to no longer emit malformed metrics. We recommend consulting the documentation for your exporter for help. You can confirm you have fixed the problem by manually visiting the /metrics endpoint and inspecting the exported metric names.

You can then free up your quota by deleting the malformed metrics using the projects.metricDescriptors.delete method. To more easily iterate through the list of malformed metrics, we provide a Golang script you can use. This script accepts a regular expression that can identify your malformed metrics and deletes any metric descriptors that match the pattern. As metric deletion is irreversible, we strongly recommend first running the script using dry run mode.

Some metrics are missing for short-running targets

Google Cloud Managed Service for Prometheus is deployed and there are no configuration errors; however, some metrics are missing.

Determine the deployment that generates the partially missing metrics. If the deployment is a Google Kubernetes Engine' CronJob, then determine how long the job typically runs:

  1. Find the cron job deployment yaml file and find the status, which is list at the end of the file. The status in this example shows that the job ran for one minute:

      status:
    lastScheduleTime: "2024-04-03T16:20:00Z"
    lastSuccessfulTime: "2024-04-03T16:21:07Z"

  2. If the run time is less than five minutes, then the job isn't running long enough for the metric data to be consistently scraped.

    To resolve this situation, try the following:

    • Configure the job to ensure that it doesn't exit until at least five minutes have elapsed since the job started.

    • Configure the job to detect whether metrics have been scraped before exiting. This capability requires library support.

    • Consider creating a log based distribution-valued metric instead of collecting metric data. This approach is suggested when data is published at a low rate. For more information, see Log-based metrics.

  3. If the run time is longer than five minutes or if it is inconsistent, then see the Unhealthy targets section of this document.

Problems with collection from exporters

If your metrics from an exporter are not being ingested, check the following:

Collector Out Of Memory (OOM) errors

If you are using managed collection and encountering Out Of Memory (OOM) errors on your collectors, then consider enabling vertical pod autoscaling.

Operator Out Of Memory (OOM) errors

If you are using managed collection and encountering Out Of Memory (OOM) errors on your operator, then consider disabling target status feature. The target status feature can cause operator performance issues in larger clusters.

Too many time series or increased 503 responses and context deadline exceeded errors, especially during peak load

You might be encountering this issue if you see the following error message:

"Context deadline exceeded" is a generic 503 error returned from Monarch for any ingestion-side problem that doesn't have a specific cause. A very small number of "context deadline exceeded" errors is expected with normal use of the system.

However, you might notice a pattern where "context deadline exceeded" errors increase and materially impact your data ingestion. One potential root cause is that you might be incorrectly setting target labels. This is more likely if the following are true:

If these points are true, it's likely that your exporter has misconfigured the resource labels in a way that conflicts with Monarch's requirements.

Monarch scales by storing related data together in a target. A target for Managed Service for Prometheus is defined by the prometheus_target resource type and the project_id, location, cluster, namespace, job, and instance labels. For more information about these labels and defaulting behavior, see Reserved labels in Managed Collection or Reserved labels in Self-deployed collection.

Of these labels, instance is the lowest-level target field and is therefore most important to get right. Efficiently storing and querying metrics in Monarch requires relatively small, diverse targets, ideally around the size of a typical VM or a container. When running Managed Service for Prometheus in typical scenarios, the open-source default behavior built into the collector usually picks good values for the job and instance labels, which is why this topic is not covered elsewhere in the documentation.

However, the default logic might fail when you are running an exporter that reports metrics on behalf of other resources in your cluster, such as the statsd_exporter. Instead of setting the value of instance to the IP:port of the resource that emits the metric, the value of instance gets set to the IP:port of the statsd_exporter itself. The issue can be compounded by the job label, as instead of relating to the metric package or service, it also lacks diversity by being set to statsd-exporter.

When this happens, all metrics that come from this exporter within a given cluster and namespace get written into the same Monarch target. As this target gets larger, writes begin failing, and you see increased "Context deadline exceeded" 503 errors.

You can get verification that this is happening to you by contacting Cloud Customer Care and asking them to check the "Monarch Quarantiner hospitalization logs". Include any known values for the six reserved labels in your ticket. Make sure to report the Google Cloud project that is sending the data, not the Google Cloud project of your metrics scope.

To fix this issue, you have to change your collection pipeline to use more diverse target labels. Some potential strategies, listed in order of effectiveness, include:

Duplicate buckets within a histogram point

You might be encountering this issue if you see the following error message:

This is caused by having two histogram points within your exporter that have the exact same set of labels. This could happen for the following reasons:

Prometheus spec allows out-of-order time series within a /metrics endpoint. The collector scrapes out-of-order time series, re-orders them, merges them into a single metric based on the labels, and sends the point.

When there are histograms with duplicative labels in a single /metrics endpoint, the resulting merged histogram ends up with two buckets that have the same le value. This error is retuned by the API because bucket le values must be unique and in increasing order.

To fix this, make sure that all histogram points have a unique set of labels in your exporter.

No errors and no metrics

If you are using managed collection, you don't see any errors, but data is not appearing in Cloud Monitoring, then the most likely cause is that your metric exporters or scrape configurations are not configured correctly. Managed Service for Prometheus does not send any time series data unless you first apply a valid scrape configuration.

To identify whether this is the cause, try deploying the example application and example PodMonitoring resource. If you now see the up metric (it may take a few minutes), then the problem is with your scrape configuration or exporter.

The root cause could be any number of things. We recommend checking the following:

Firewalls

A firewall can cause both ingestion and query problems. Your firewall must be configured to permit both POST and GET requests to the Monitoring API service, monitoring.googleapis.com, to allow ingestion and queries.

Error about concurrent edits

The error message "Too many concurrent edits to the project configuration" is usually transient, resolving after a few minutes. It is usually caused by removing a relabeling rule that affects many different metrics. The removal causes the formation of a queue of updates to the metric descriptors in your project. The error goes away when the queue is processed.

For more information, see Limits on creating and updating metrics and labels.

Queries blocked and cancelled by Monarch

If you see the following error, then you have hit the internal limit for the number of concurrent queries that can be run for any given project:

To protect against abuse, the system enforces a hard limit on the number of queries from one project that can run concurrently within Monarch. With typical Prometheus usage, queries should be quick and this limit should never be reached.

You might hit this limit if you are issuing a lot of concurrent queries that run for a longer-than-expected time. Queries requesting more than 25 hours of data are usually slower to execute than queries requesting less than 25 hours of data, and the longer the query lookback, the slower the query is expected to be.

Typically this issue is triggered by running lots of long-lookback rules in an inefficient way. For example, you might have many rules that run once every minute and request a 4-week rate. If each of these rules takes a long time to run, it might eventually cause a backup of queries waiting to run for your project, which then causes Monarch to throttle queries.

To resolve this issue, you need to increase the evaluation interval of your long-lookback rules so that they're not running every 1 minute. Running a query for a 4-week rate every 1 minute is unnecessary; there are 40,320 minutes in 4 weeks, so each minute gives you almost no additional signal (your data changes at most by 1/40,320th). Using a 1 hour evaluation interval should be sufficient for a query that requests a 4-week rate.

Once you resolve the bottleneck caused by inefficient long-running queries executing too frequently, this issue should resolve itself.

Incompatible value types

If you see the following error upon ingestion or query, then you have a value type incompatibility in your metrics:

You might see this error upon ingestion, as Monarch does not support writing DOUBLE-typed data to INT64-typed metrics nor does it support writing INT64-typed data to DOUBLE-typed metrics. You also might see this error when querying using a multi-project metrics scope, as Monarch cannot union DOUBLE-typed metrics in one project with INT64-typed metrics in another project.

This error only happens when you have OpenTelemetry collectors reporting data, and it is more likely to happen if you have both OpenTelemetry (using the googlemanagedprometheus exporter) and Prometheus reporting data for the same metric as commonly happens for the target_info metric.

The cause is likely one of the following:

This problem can be solved by forcing all metric value types to DOUBLE by doing the following:

  1. Reconfigure your OpenTelemetry collectors to force all metrics to be a DOUBLE by enabling the feature-gate exporter.googlemanagedprometheus.intToDouble flag.
  2. Delete all INT64 metric descriptors and let them get recreated as a DOUBLE. You can use the delete_metric_descriptors.go script to automate this.

Following these steps deletes all data that is stored as an INT64 metric. There is no alternative to deleting the INT64 metrics that fully solves this problem.

Send feedback

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2026-08-12 UTC.

Need to tell us more? [[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2026-08-12 UTC."],[],[]]
Web Proxy Viewer  |  New URL  |  Original Page