Experiment Quick Start

Experiments and feature flags use the Amplitude Experiment SDK or REST API to communicate with Amplitude Experiment.

Setting up an experiment is a multi-stage process with these procedures:

1. Install and set up SDKs and deployments.
2. Create feature flags.
3. Select the deployment and create a payload.
4. Create variations and send payloads.
5. Design experiments.

Before you begin

Before you start using experiments:

• Confirm you have developer write access to the Experiment SDK.
• Confirm you have access to the Experiment feature.
• Confirm you have developer access to the application where you integrate your feature flags.

Set up the SDK

Install the Amplitude SDK with the Experiment client. For example:

npm install @amplitude/analytics-browser @amplitude/experiment-js-client

import * as amplitude from "@amplitude/analytics-browser";
import { Experiment } from "@amplitude/experiment-js-client";
amplitude.init("AMPLITUDE_API_KEY");
const experiment = Experiment.initialize("DEPLOYMENT_API_KEY");
await experiment.start();

Set up a deployment

Experiment uses the same projects as Amplitude Analytics. As a best practice, create one project for each product and each environment. Because flags, experiments, and deployments only exist within a single project, you must duplicate these objects across projects within the same product.

In Amplitude Experiment, a deployment serves a group of flags or experiments for use in an application. Each project has a deployment using the project API key as the deployment key, available by default. On creation, Experiment assigns a randomly generated deployment key to each deployment. Experiment uses the deployment key to identify the deployment and authorize requests to the evaluation servers.

Deployments belong to Amplitude Analytics projects, and a project can have multiple deployments. Amplitude recommends that you name deployments after the platform (client-side) or service (server-side) to which Experiment serves variants (for example: android, ios, web). The default project API key deployment is useful for getting started. Use explicit deployments for each platform or service in larger organizations or teams that may share the same Amplitude project across multiple platforms for the same application. Each deployment receives a unique key for use in your application.

To create a deployment

1. Go to Experiments > Deployments.
2. Select Create Deployment.
3. Select the project you want from the dropdown list.
4. Name your deployment.
5. Select whether your deployment is for Client or Server.
6. Select Create a Deployment. For full details, go to Configure your experiment.

Create a new flag

A flag lets you enable or disable a function or feature in your product without deploying new code each time. Flags drive both experiments and feature rollouts. Flags work well for launching experiments and ending them after you collect enough data, and for rolling out new features (and rolling them back, if needed).

To create a new feature flag

1. Go to Experiment > Feature Flags.
2. Select Create Feature Flag.
3. In the Create Flag section, select the project you want from the dropdown list, then give your flag a name. Amplitude Experiment generates the flag key from the name you choose. The flag key is an identifier for the flag used in your codebase.
4. Specify the evaluation mode for your experiment. Select either Remote or Local.
5. Specify the bucketing unit you want to use for this experiment.

6. Select Create. Experiment opens a blank template for your flag.
7. Choose the deployment you want from the Deployment dropdown menu.
8. (Optional) Select Advanced Settings to modify the bucketing salt options.

Create variations

A variant exists within a flag or an experiment, and represents a variable experience for a user. Variants comprise the A/B changes you want to test. All feature flags must contain at least one variant. You can add as many variants as you want to a flag.

To add a variant

1. Go to your Experiment > Feature Flags and select your flag.
2. In the Variants section, select the Plus icon to create a variant.
3. Enter the name, value, and description of your variant.
4. Select Apply.

{
  "layout": "cards",
  "titlePosition": "above",
  "gradient": false,
  "showDescription": true,
  "cardCount": 3
}

Add targeting to the flag

In the Assignment section, define the user segments you want to experience your new feature. Defining a user segment limits your rollout to users in specific geographical locations, certain demographic groups, or who meet certain usage thresholds in your product (for example, power users). For more information on segmenting, go to Define your audience.

To add targeting

1. Specify the percentage of users who receive the variant.
2. To define a user segment, go to the Rule Based User Segments section and select Segment 1. Then follow the same steps you use to build a user segment in Amplitude Analytics.

Finalize the flag

After you set up the flag, associate it with a deployment, set up your variants or payloads, and target your users, finalize the feature flag. Finalizing the flag activates the flag and makes it available.

To finalize a feature flag

1. Go to your feature flag.
2. Select Activate flag.

Design the experiment

You can create an experiment directly or convert an existing flag to an experiment.

When designing your experiment:

• Set metrics for the experiment.
• Set up any further variations and payloads.

Adding goals (or metrics) lets you track the success rate of your experiment. All experiments should have at least one metric. Tell Amplitude Experiment what you want your primary metric to be, and define any secondary metrics. The primary metric determines whether Amplitude accepts or rejects your hypothesis, and therefore whether your experiment succeeded or failed.

To add metrics

1. Open your experiment and go to the Metrics section.
2. Select your recommended metric from the Metric dropdown or create a custom metric.
3. Select the metric type using "should" or "should not":
   • "Should": A Success metric states the goal changes by the goal amount and direction.
   • "Should not": A Guardrail metric states the goal doesnt change by the goal amount and direction.
4. Specify whether you expect the metric to increase or decrease.
5. (Optional) Set the minimally acceptable goal for the experiment, also known as the minimum detectable effect. The minimum detectable effect is the minimum difference between the control and the variant for the experiment to count as a positive result.
6. To add secondary metrics, select Add Metric and repeat this process for each additional metric you want to include.

To add more variations and payloads

To create more variations and payloads, repeat the steps in Create variations in your flag.

Start your experiment

After you finish designing your experiment, select Start Experiment to begin.

Code examples

The following code examples show the code for a feature flag and a JSON payload.

import { useState, useEffect } from 'react';
import { getBlogLayoutFlag } from '../services/featureFlags'; // Adjust to wherever you fetch your Amplitude flag
import type { BlogPost } from '../types';

type LayoutFlag = {
  layout: 'cards' | 'list' | 'carousel';
  titlePosition: 'above' | 'below' | 'center';
  gradient: boolean;
  showDescription: boolean;
  cardCount: number;
};

export default function BlogPostLayoutClient({ posts }: { posts: BlogPost[] }) {
  const [layoutFlag, setLayoutFlag] = useState<LayoutFlag | null>(null);

  useEffect(() => {
    getBlogLayoutFlag().then((flag) => {
      console.log(':magic_wand: Received Flag from Amplitude:', flag);
      if (flag) {
        setLayoutFlag(flag);
      } else {
        console.log(':warning: No flag returned, falling back to default layout');
        setLayoutFlag({
          layout: 'cards',
          titlePosition: 'above',
          gradient: false,
          showDescription: true,
          cardCount: 3,
        });
      }
    });
  }, []);

  if (!layoutFlag) {
    // You might render a loader here
    return null;
  }

  // Render your posts according to layoutFlag...
  return (
    <div>
      {/* e.g. layoutFlag.layout === 'cards' ? <CardGrid posts={posts} /> : ... */}
    </div>
  );
}