Run a computation-based evaluation pipeline Stay organized with collections Save and categorize content based on your preferences.

REST

To create a model evaluation job, send a POST request by using the pipelineJobs method.

Before using any of the request data, make the following replacements:

• PROJECT_ID: The Google Cloud project that runs the pipeline components.
• PIPELINEJOB_DISPLAYNAME: A display name for the pipelineJob.
• LOCATION: The region to run the pipeline components. Currently, only us-central1 is supported.
• DATASET_URI: The Cloud Storage URI of your reference dataset. You can specify one or multiple URIs. This parameter supports wildcards. To learn more about this parameter, see InputConfig.
• OUTPUT_DIR: The Cloud Storage URI to store evaluation output.
• MODEL_NAME: Specify a publisher model or a tuned model resource as follows:
  • Publisher model: publishers/google/models/MODEL@MODEL_VERSION

    Example: publishers/google/models/text-bison@002

  • Tuned model: projects/PROJECT_NUMBER/locations/LOCATION/models/ENDPOINT_ID

    Example: projects/123456789012/locations/us-central1/models/1234567890123456789

  The evaluation job doesn't impact any existing deployments of the model or their resources.

• EVALUATION_TASK: The task that you want to evaluate the model on. The evaluation job computes a set of metrics relevant to that specific task. Acceptable values include the following:
  • summarization
  • question-answering
  • text-generation
  • classification
• INSTANCES_FORMAT: The format of your dataset. Currently, only jsonl is supported. To learn more about this parameter, see InputConfig.
• PREDICTIONS_FORMAT: The format of the evaluation output. Currently, only jsonl is supported. To learn more about this parameter, see InputConfig.
• MACHINE_TYPE: (Optional) The machine type for running the evaluation job. The default value is e2-highmem-16. For a list of supported machine types, see Machine types.
• SERVICE_ACCOUNT: (Optional) The service account to use for running the evaluation job. To learn how to create a custom service account, see Configure a service account with granular permissions. If unspecified, the Gemini Enterprise Agent Platform Custom Code Service Agent is used.
• NETWORK: (Optional) The fully qualified name of the Compute Engine network to peer the evaluatiuon job to. The format of the network name is projects/PROJECT_NUMBER/global/networks/NETWORK_NAME. If you specify this field, you need to have a VPC Network Peering for Gemini Enterprise Agent Platform. If left unspecified, the evaluation job is not peered with any network.
• KEY_NAME: (Optional) The name of the customer-managed encryption key (CMEK). If configured, resources created by the evaluation job is encrypted using the provided encryption key. The format of the key name is projects/PROJECT_ID/locations/REGION/keyRings/KEY_RING/cryptoKeys/KEY. The key needs to be in the same region as the evaluation job.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/pipelineJobs

Request JSON body:

{
  "displayName": "PIPELINEJOB_DISPLAYNAME",
  "runtimeConfig": {
    "gcsOutputDirectory": "gs://OUTPUT_DIR",
    "parameterValues": {
      "project": "PROJECT_ID",
      "location": "LOCATION",
      "batch_predict_gcs_source_uris": ["gs://DATASET_URI"],
      "batch_predict_gcs_destination_output_uri": "gs://OUTPUT_DIR",
      "model_name": "MODEL_NAME",
      "evaluation_task": "EVALUATION_TASK",
      "batch_predict_instances_format": "INSTANCES_FORMAT",
      "batch_predict_predictions_format: "PREDICTIONS_FORMAT",
      "machine_type": "MACHINE_TYPE",
      "service_account": "SERVICE_ACCOUNT",
      "network": "NETWORK",
      "encryption_spec_key_name": "KEY_NAME"
    }
  },
  "templateUri": "https://us-kfp.pkg.dev/vertex-evaluation/pipeline-templates/evaluation-llm-text-generation-pipeline/1.0.1"
}

To send your request, choose one of these options:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/pipelineJobs"

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/pipelineJobs" | Select-Object -Expand Content

You should receive a JSON response similar to the following. Note that pipelineSpec has been truncated to save space.

Response

......
.....
 "state": "PIPELINE_STATE_PENDING",
  "labels": {
    "vertex-ai-pipelines-run-billing-id": "1234567890123456789"
  },
  "runtimeConfig": {
    "gcsOutputDirectory": "gs://my-evaluation-bucket/output",
    "parameterValues": {
      "project": "my-project",
      "location": "us-central1",
      "batch_predict_gcs_source_uris": [
        "gs://my-evaluation-bucket/reference-datasets/eval_data.jsonl"
      ],
      "batch_predict_gcs_destination_output_uri": "gs://my-evaluation-bucket/output",
      "model_name": "publishers/google/models/text-bison@002"
    }
  },
  "serviceAccount": "123456789012-compute@developer.gserviceaccount.com",
  "templateUri": "https://us-kfp.pkg.dev/vertex-evaluation/pipeline-templates/evaluation-llm-text-generation-pipeline/1.0.1",
  "templateMetadata": {
    "version": "sha256:d4c0d665533f6b360eb474111aa5e00f000fb8eac298d367e831f3520b21cb1a"
  }
}

Example curl command

PROJECT_ID=myproject
REGION=us-central1
MODEL_NAME=publishers/google/models/text-bison@002
TEST_DATASET_URI=gs://my-gcs-bucket-uri/dataset.jsonl
OUTPUT_DIR=gs://my-gcs-bucket-uri/output

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
"https://${REGION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${REGION}/pipelineJobs" -d \
$'{
  "displayName": "evaluation-llm-text-generation-pipeline",
  "runtimeConfig": {
    "gcsOutputDirectory": "'${OUTPUT_DIR}'",
    "parameterValues": {
      "project": "'${PROJECT_ID}'",
      "location": "'${REGION}'",
      "batch_predict_gcs_source_uris": ["'${TEST_DATASET_URI}'"],
      "batch_predict_gcs_destination_output_uri": "'${OUTPUT_DIR}'",
      "model_name": "'${MODEL_NAME}'",
    }
  },
  "templateUri": "https://us-kfp.pkg.dev/vertex-evaluation/pipeline-templates/evaluation-llm-text-generation-pipeline/1.0.1"
}'

Python

Before trying this sample, follow the Python setup instructions in the Agent Platform quickstart using client libraries.

To authenticate to Agent Platform, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

import os

import vertexai
from vertexai.evaluation import EvalResult, EvalTask

# TODO (Developer) Set environment variables
PROJECT_ID = os.getenv("GOOGLE_CLOUD_PROJECT")
LOCATION_ID = os.getenv("LOCATION_ID", "us-central1")
MODEL_NAME = os.getenv("MODEL_NAME", "gemini-2.5-flash")


def evaluate_model() -> EvalResult:
    """Evaluate the performance of a generative AI model."""

    vertexai.init(project=PROJECT_ID, location=LOCATION_ID)

    # Dataset URI containing input prompts and ground truth labels
    dataset_uri = "gs://cloud-samples-data/ai-platform/generative_ai/llm_classification_bp_input_prompts_with_ground_truth.jsonl"

    metric_column_mapping = {"reference": "ground_truth"}

    # Define evaluation task
    eval_task = EvalTask(
        dataset=dataset_uri,
        metrics=["exact_match"],
        experiment="gemini-classification-eval",
        metric_column_mapping=metric_column_mapping,
    )

    # Define a prompt template so the generative Gemini model produces formatted labels
    prompt_template = (
        "Classify the following text into exactly one category from "
        "[nature, news, sports, health, startups]. Return only the category name:\n\n{prompt}"
    )

    # Evaluate using a modern Gemini model
    eval_result = eval_task.evaluate(
        model=MODEL_NAME,
        prompt_template=prompt_template,
    )

    print("=== SUMMARY METRICS ===")
    print(eval_result.summary_metrics)

    print("\n=== METRICS TABLE SAMPLE ===")
    print(eval_result.metrics_table.head())

    return eval_result

Console

To create a model evaluation job by using the Google Cloud console, perform the following steps:

1. In the Google Cloud console, go to the Gemini Enterprise Agent Platform Model Registry page.

   Go to Gemini Enterprise Agent Platform Model Registry

2. Click the name of the model that you want to evaluate.
3. In the Evaluate tab, click Create evaluation and configure as follows:
• Objective: Select the task that you want to evaluate.
• Target column or field: (Classification only) Enter the target column for prediction. Example: ground_truth.
• Source path: Enter or select the URI of your evaluation dataset.
• Output format: Enter the format of the evaluation output. Currently, only jsonl is supported.
• Cloud Storage path: Enter or select the URI to store evaluation output.
• Class names: (Classification only) Enter the list of possible class names.
• Number of compute nodes: Enter the number of compute nodes to run the evaluation job.
• Machine type: Select a machine type to use for running the evaluation job.
4. Click Start evaluation