A high-performance E2E testing framework for Flutter mobile apps. Write tests in plain English, execute with sub-50ms command round-trips via direct widget-tree access — no UI automation layer, no WebDriver overhead.

test "user can log in"
  @smoke
  open the app
  wait until "Sign In" appears
  type "user@example.com" into the "Email" field
  type "secret" into the "Password" field
  tap "Sign In"
  see "Dashboard"

• How It Works
• Installation
• ProbeScript Language
• Composite Tests
• Annotation-driven Tests
• CLI Commands
• Studio (Beta Preview)
• MCP Server
• Visual Regression
• AI-Powered Visual Assertions
• Test Recording
• CI/CD Integration
• Cloud Device Farms
• Migrating from Other Frameworks
• VS Code Extension
• Configuration (probe.yaml)
• Self-Healing Selectors
• Repository Structure
• Documentation
• License

FlutterProbe has two components that communicate over WebSocket + JSON-RPC 2.0:

┌──────────────┐     WebSocket / JSON-RPC 2.0      ┌─────────────────┐
│  probe CLI   │ ──────────────────────────────────▶│  ProbeAgent     │
│  (Go)        │   localhost:48686                   │  (Dart, on      │
│              │   tap, type, see, wait, swipe       │   device)       │
│  Parses .probe│  screenshot, dump_tree             │                 │
│  Manages devs │  One-time token auth               │  Walks widget   │
│  Reports     │                                     │  tree directly  │
└──────────────┘                                    └─────────────────┘

• Android: CLI → adb forward → WebSocket → ProbeAgent
• iOS Simulator: CLI → localhost directly (simulator shares host loopback)

The ProbeAgent is a Dart package you add to your Flutter app as a dev dependency. It runs a WebSocket server inside your app and executes commands against the live widget tree — no flutter_test, no TestWidgetsFlutterBinding, no external driver.

brew tap AlphaWaveSystems/tap
brew install probe

Upgrades are handled automatically with brew upgrade probe.

Pre-built binaries for Linux, macOS (Intel + Apple Silicon), and Windows are attached to every GitHub Release.

# Linux (amd64)
curl -Lo probe https://github.com/AlphaWaveSystems/flutter-probe/releases/latest/download/probe-linux-amd64
chmod +x probe && sudo mv probe /usr/local/bin/

# macOS Apple Silicon
curl -Lo probe https://github.com/AlphaWaveSystems/flutter-probe/releases/latest/download/probe-darwin-arm64
chmod +x probe && sudo mv probe /usr/local/bin/

# macOS Intel
curl -Lo probe https://github.com/AlphaWaveSystems/flutter-probe/releases/latest/download/probe-darwin-amd64
chmod +x probe && sudo mv probe /usr/local/bin/

git clone https://github.com/AlphaWaveSystems/flutter-probe.git
cd flutter-probe
make build        # → bin/probe
make install      # → $GOPATH/bin/probe (optional)

Requirements: Go 1.26+, Dart 3.3+ / Flutter 3.19+ (tested up to 3.41), ADB (Android), Xcode (iOS)

WiFi (recommended) — stable, zero connection drops:

# Build with WiFi enabled
flutter build ios --profile --dart-define=PROBE_AGENT=true --dart-define=PROBE_WIFI=true

# Run tests (find token in app console: PROBE_TOKEN=...)
probe test tests/ --host <device-ip> --token <probe-token>

USB — requires libimobiledevice (may experience USB-C drops):

brew install libimobiledevice  # provides iproxy, idevicesyslog, idevice_id

Tip: WiFi is recommended over USB-C. USB-C cables switch between charging and data modes, causing intermittent connection drops. See FAQ for details.

In your app's pubspec.yaml:

dev_dependencies:
  flutter_probe_agent:
    path: /path/to/flutter-probe/probe_agent

In your main.dart, start the agent before runApp:

import 'package:flutter_probe_agent/flutter_probe_agent.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await ProbeAgent.start();   // starts WebSocket server on port 48686
  runApp(MyApp());
}

cd your-flutter-app
probe init          # creates probe.yaml and tests/ scaffold
probe test tests/   # run all tests

Tests are written in .probe files using natural English with Python-style indentation.

test "checkout flow"
  @smoke @checkout
  open the app
  wait until "Cart" appears
  tap "Checkout"
  type "John Doe" into the "Name" field
  type "john@example.com" into the "Email" field
  tap "Place Order"
  see "Order confirmed"
  take a screenshot called "order_confirmation"

tap "Sign In"                   # by visible text
tap #login_button               # by ValueKey
tap "Submit" in "LoginForm"     # positional (widget inside a parent)
tap on the 1st "ListItem"       # ordinal

# tests/recipes/auth.probe
recipe "sign in" (email, password)
  open the app
  wait until "Sign In" appears
  type <email> into the "Email" field
  type <password> into the "Password" field
  tap "Sign In"

# tests/login.probe
use "tests/recipes/auth.probe"

test "user can access dashboard"
  sign in "user@example.com" and "secret"
  see "Dashboard"

before all
  open the app
  tap "Accept Terms"

before each
  see "Home"

after each
  take a screenshot called "after"

on failure
  take a screenshot called "failure"
  dump the widget tree

after all
  take a screenshot called "suite_final"

test "login with <email>"
  open the app
  type <email> into the "Email" field
  tap "Continue"
  see <result>
  Examples:
    | email              | result      |
    | "user@example.com" | "Dashboard" |
    | "bad@example.com"  | "Error"     |

Load data from CSV files:

with examples from "fixtures/users.csv"

type "<random.email>" into "Email"
type "<random.name>" into "Name"
type "<random.phone>" into "Phone"
type "<random.number(1,100)>" into "Age"

Make real API requests from tests:

call POST "https://api.example.com/seed" with body "{\"env\":\"test\"}"
call GET "https://api.example.com/health"

copy "user@test.com" to clipboard
paste from clipboard
set location 37.7749, -122.4194
travel to                                 # simulate GPS movement through waypoints
  37.7749, -122.4194
  37.7849, -122.4094
over 10 seconds
add media "fixtures/photo.jpg"            # seed the camera roll/gallery
kill the app
open the app
verify external browser opened
open link "myapp://profile/42" in the app  # deep link via OS intent handling

Reach outside the Flutter widget tree into native, OS-owned UI — pickers, share sheets — matched by uiautomator's text or resource-id:

tap native "Choose from Gallery"
see native "IMG_0001.jpg"
type native "wifi" into "Search settings"

take screenshot "checkout_page"                # capture
compare screenshot "checkout_page"             # baseline on first run, diff after
compare screenshot "total_price" of "Total"    # crop the comparison to one widget

if "Onboarding" is visible
  tap "Skip"
repeat 3 times
  swipe up
  wait 1 seconds
retry 3 times            # re-runs the block on failure, stops at first success
  tap "Submit"
  see "Success"
tap "Rate this app" optional   # attempt, but don't fail the test if it errors

clear app data          # wipe storage, relaunch, reconnect
restart the app         # force-stop, relaunch, reconnect (data preserved)

Composite tests coordinate multiple devices simultaneously in a single .probe file. Designed for real-time features: chat, push notifications, multiplayer, sync.

composite test "alice sends bob a message"
  devices
    A: iPhone 15 Simulator
    B: Pixel 9 Emulator

  A:
    open app
    tap "Login"
    type "alice@example.com" in "email"
    tap "Continue"

  B:
    open app
    tap "Login"
    type "bob@example.com" in "email"
    tap "Continue"

  sync "both logged in"

  A:
    tap "New Message"
    type "Hello Bob" in "compose"
    tap "Send"

  B:
    wait until "Hello Bob" appears
    see "Hello Bob"

How it works: FlutterProbe launches one goroutine per device. Steps tagged A: run concurrently with B: steps. sync "label" is a barrier — all goroutines block until the last one arrives, then all proceed together.

Failure semantics: if one device fails, the shared context is cancelled immediately, all barriers are aborted, and other devices stop at their next step. The failing device is reported as FAIL; others as CANCELLED.

N devices: add more aliases to devices: and write matching step blocks — no limit.

Run composite tests by mapping aliases to real devices:

# WiFi (physical devices or cross-machine)
probe test tests/ --composite-device "A=192.168.1.10:48686/token1" \
                  --composite-device "B=192.168.1.11:48686/token2"

# Local simulators (by UDID)
probe test tests/ --composite-device "A=A1B2C3D4-..." \
                  --composite-device "B=E5F6G7H8-..."

# Android
probe test tests/ --composite-device "A=emulator-5554" \
                  --composite-device "B=emulator-5556"

Or pin them in probe.yaml:

composite:
  devices:
    A: "192.168.1.10:48686/my-token"
    B: "00008030-001A34E40258002E"

Composite tests without configured devices are reported as SKIPPED, so single-device pipelines are unaffected. Full reference: Composite Tests wiki.

allow permission "camera"
deny permission "notifications"
grant all permissions
revoke all permissions

run dart:
  final version = await PackageInfo.fromPlatform();
  print('App version: ${version.version}');

Co-locate .probe tests with the Flutter widgets they exercise. Two Dart packages handle this:

• flutter_probe_annotation — @ProbeSuite, @ProbeTest, @ProbeRecipe, @ProbeCompositeTest decorators plus a fully type-checked step DSL (all 31 ProbeScript verbs, all 6 selector kinds, hooks, loops, conditionals, recipes, examples, multi-device composite tests).
• flutter_probe_gen — a build_runner builder that reads the annotations and emits matching .probe files into tests/generated/.

Add to your Flutter app's pubspec.yaml:

dependencies:
  flutter_probe_annotation: ^0.9.6
  flutter_probe_agent: ^0.9.6

dev_dependencies:
  flutter_probe_gen: ^0.9.6
  build_runner: ^2.15.0

Annotate any screen class:

import 'package:flutter_probe_annotation/flutter_probe_annotation.dart';

@ProbeSuite(
  beforeEach: [Open()],
  tests: [
    ProbeTest('user can log in', tags: ['smoke'], steps: [
      Tap(id: 'email_field'),
      Type('alice@example.com'),
      Tap(id: 'password_field'),
      Type('hunter2'),
      Tap(text: 'Sign In'),
      WaitUntil.appears('Dashboard'),
      See('Dashboard'),
    ]),
  ],
)
class LoginScreen extends StatelessWidget { /* … */ }

Run the builder:

dart run build_runner build
probe test tests/        # picks up tests/generated/login_screen.probe

Test definitions are now type-checked by flutter analyze — a misspelt step name is a compile error rather than a runtime surprise. Selectors stay in sync with widget code because they live in the same file. The generated .probe file goes through the same parser, agent, and reporter as a hand-written one.

v0.9.6 completes the annotation surface: full composite-test DSL (@ProbeCompositeTest, Device, OnDevice, Sync), id/selector-based See/DontSee, WaitUntil.idAppears, and composable state + containing + matching assertions. Plus fixes for two emitter bugs (Mock paths and See suffix dropping).

v0.9.7 adds biometric authentication testing — enroll biometric, biometric match, biometric no match steps (and matching EnrollBiometric() / BiometricMatch() / BiometricNoMatch() annotation classes) drive Face ID / Touch ID / fingerprint flows on iOS Simulator and Android emulator. Skipped on physical devices.

v0.9.8 fixes biometric no-match on iOS 26+ simulator where notifyutil no-match notifications no longer resolve LAContext.evaluatePolicy. The CLI now delivers results via probe.biometric_signal; use awaitBiometricResult() from flutter_probe_agent instead of local_auth.authenticate() in PROBE_AGENT builds. Also adds a port-range fallback: the agent auto-tries ports 48686–48695 and logs PROBE_PORT_BUSY=N (another probe agent is running) when a collision is with a sibling agent.

Full reference: flutterprobe.dev/probescript/annotations (or docs/wiki/Annotations.md on GitHub).

v0.6.0 ships FlutterProbe Studio — a cross-platform desktop app for visual ProbeScript test authoring with an embedded device view, live widget-tree inspector, and in-process test execution. Built with Wails 2.12; macOS / Windows / Linux.

# From the repo
cd studio
go install github.com/wailsapp/wails/v2/cmd/wails@latest
wails build
# → studio/build/bin/flutter-probe-studio.app (macOS)

Studio binaries also ship as part of every GitHub release. See the dedicated Studio docs for screenshots, architecture, system requirements, and known limitations.

probe-mcp is a standalone binary that exposes all FlutterProbe capabilities to AI agents (Claude Desktop, Cursor, any MCP-compatible client) via 18 tools:

Every CLI feature is accessible from MCP. Key capabilities an agent can use:

• Composite tests — write_test supports the full composite test syntax; run_tests accepts a composite_devices parameter ("A=host:port/token B=udid")
• Structured results — pass --format json in run_tests's flags to get machine-readable output, then call get_report or generate_report for HTML
• Parallel/shard — --parallel or --shard 1/3 via flags
• WiFi physical devices — --host <ip> --token <tok> via flags
• Android + iOS shutdown — shutdown_device accepts udid (iOS) or serial (Android)
• Pre-write validation — write_test parses the content before writing; syntax errors are returned immediately without creating the file
• Interaction recording — record captures gestures and returns a .probe file

Each release publishes a one-click .mcpb extension for Claude Desktop. No brew install, no JSON config, no PATH setup.

1. Download the bundle for your platform from the latest release:
   • flutter-probe-darwin-arm64.mcpb — Apple Silicon Macs
   • flutter-probe-darwin-amd64.mcpb — Intel Macs
   • flutter-probe-linux-amd64.mcpb — Linux x86_64
   • flutter-probe-win32-amd64.mcpb — Windows x86_64
2. In Claude Desktop, open Settings → Extensions → Install Extension, pick the .mcpb file.
3. When prompted, select your Flutter project directory (the folder that contains probe.yaml and tests/).
4. Done — all 18 tools appear in Claude.

The bundle ships the probe-mcp binary inside the extension; auto-updates and lifecycle are managed by Claude Desktop.

For Cursor or any other MCP-compatible client, install the binary and point your client at it:

{
  "mcpServers": {
    "flutter-probe": {
      "command": "probe-mcp"
    }
  }
}

The legacy probe mcp-server subcommand still works but prints a deprecation notice. Full setup guide for Claude Desktop, Cursor, and other MCP clients: MCP Server docs.

Capture baseline screenshots and compare on every run:

test "main screen looks correct"
  open the app
  wait 2 seconds
  take a screenshot called "main_screen"   # first run: saves baseline
                                            # subsequent runs: compares

Configure sensitivity in probe.yaml:

visual:
  threshold: 0.5    # max % of pixels allowed to differ
  pixel_delta: 8    # per-pixel color tolerance (0–255)

For checks that are hard to express with a plain selector — "does this screen look right," dynamic content, generated charts:

test "checkout screen looks right"
  open the app
  see "checkout total, tax, and shipping are visible and add up correctly" with ai

For a generic "does this screen look broken" smoke check — no specific claim to write, just cut-off/overlapping/mis-centered elements:

  assert no visual defects with ai

Unlike see ... with ai, with ai is mandatory here — there's no non-AI equivalent of "assert no visual defects."

To read a specific piece of text off the screen into a variable — an OTP code, a dynamically-generated ID — for use later in the same test:

read "the 6-digit OTP code" with ai into otp
type <otp> into the "Code" field

Same with ai-mandatory rule as assert no visual defects with ai. If the requested text isn't visible, the step fails with a clear error rather than storing an empty/wrong value.

This requires a provider you configure yourself in probe.yaml — there is no default provider and nothing is sent anywhere unless you set this up. Unlike tools that route screenshots through a vendor-operated cloud service, FlutterProbe calls the provider you pick directly, with your own key:

ai:
  provider: anthropic                # openai | anthropic | local — required, no default
  api_key: ${ANTHROPIC_API_KEY}      # your own key; never sent to a FlutterProbe-operated service
  redact:                            # black out these widgets before any screenshot leaves the device
    - selector: "#credit_card_field"
    - selector: "Account Balance"

For nothing to leave the device/host at all — not even to a BYO-key cloud vendor — point provider: local at any OpenAI-compatible local server (e.g. Ollama):

ai:
  provider: local
  endpoint: http://localhost:11434/v1  # OpenAI-compatible base URL — required for provider: local
  model: llava                         # required — no default; must be a vision-capable model your server has loaded
  timeout: 90s                         # default: 60s. Large local reasoning models can take longer than that per call.

ai.timeout applies to every provider, but matters most for local: confirmed against a real LM Studio instance running a 31B reasoning model, where assert no visual defects with ai (a richer prompt than a plain see ... with ai) consistently took longer than the 60s default — raise it if you hit context deadline exceeded with a large local model.

This is currently the only "fully local" option FlutterProbe ships. Native on-device model support (Apple Intelligence on iOS/macOS, Gemini Nano on Android) is not implemented — those are in-app, entitlement-gated platform SDKs a CLI process has no route to, a materially larger effort than this feature, tracked separately rather than folded in silently.

with ai cannot be negated, and fails fast with a clear error at parse time — before any device connects — if ai: isn't configured. See docs/prd/ai-visual-assertions-prd.md and docs/research/maestro-ai-assertions-investigation.md for the design rationale.

Capture real interactions and generate ProbeScript automatically:

probe record --device emulator-5554 --output tests/recorded.probe

Tap, swipe, and type in your app — FlutterProbe writes the .probe file as you go.

No need to clone this repo in your own CI pipelines. Download the pre-built binary directly from GitHub Releases:

# .github/workflows/e2e.yml
jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install FlutterProbe
        run: |
          curl -Lo probe https://github.com/AlphaWaveSystems/flutter-probe/releases/latest/download/probe-linux-amd64
          chmod +x probe
          sudo mv probe /usr/local/bin/

      - name: Start Android emulator
        uses: reactivecircus/android-emulator-runner@v2
        with:
          api-level: 33
          script: |
            probe test tests/ \
              --device emulator-5554 \
              --format junit \
              -o results.xml \
              --timeout 60s -v -y

      - name: Upload test results
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results
          path: results.xml

Pin to a specific version for reproducible builds:

- name: Install FlutterProbe
  run: |
    curl -Lo probe https://github.com/AlphaWaveSystems/flutter-probe/releases/latest/download/probe-linux-amd64
    chmod +x probe && sudo mv probe /usr/local/bin/

Generate a portable HTML report from JSON output:

probe test tests/ --format json -o reports/results.json
probe report --input reports/results.json -o reports/report.html --open

Run tests on real devices without managing your own device lab. Bring your own account:

probe test tests/ \
  --cloud-provider browserstack \
  --cloud-device "Google Pixel 7-13.0" \
  --cloud-app app-release.apk \
  --cloud-key YOUR_KEY \
  --cloud-secret YOUR_SECRET

probe-convert translates tests from 7 formats at 100% construct coverage:

probe-convert tests/maestro/          # Maestro YAML
probe-convert tests/features/         # Gherkin / Cucumber
probe-convert tests/robot/            # Robot Framework
probe-convert tests/detox/            # Detox (JS/TS)
probe-convert tests/appium/           # Appium (Python, Java, JS)

Install from source (vscode/) for:

• Syntax highlighting for .probe files
• Code completion for all ProbeScript commands, selectors, tags, and recipes
• CodeLens Run buttons above every test block
• Run by tag, run with options, auto-lint on save
• Session Manager — configure and run across multiple devices simultaneously
• Test Explorer sidebar
• Integrated recording and Studio launcher

project:
  app: com.example.myapp

defaults:
  platform: android
  timeout: 30s
  screenshots: true
  video: false
  retry: 0

agent:
  port: 48686
  dial_timeout: 30s
  token_timeout: 30s

device:
  boot_timeout: 120s

# Composite test device aliases (optional — can also use --composite-device flag)
composite:
  devices:
    A: "192.168.1.10:48686/my-token"   # WiFi: host:port/token
    B: "00008030-001A34E40258002E"       # iOS simulator UDID
    C: "emulator-5554"                   # Android serial

visual:
  threshold: 0.5
  pixel_delta: 8

tools:
  adb: /path/to/adb          # optional override
  flutter: /path/to/flutter  # optional override

ai:
  api_key: sk-ant-...        # for probe generate and self-healing
  model: claude-sonnet-4-6

When a selector fails, FlutterProbe automatically tries to find a replacement using:

1. Fuzzy text match — Levenshtein similarity against visible text
2. Partial key match — Substring match on ValueKey
3. Semantic match — Accessibility label similarity
4. LLM-assisted — Claude API suggestion with confidence score (requires ai.api_key)

cmd/probe/              CLI entry point
internal/
  cli/                  Cobra command implementations
  parser/               ProbeScript lexer + parser (AST)
  runner/               Test orchestration + reporting
  probelink/            JSON-RPC 2.0 WebSocket client
  device/               ADB integration (Android)
  ios/                  xcrun simctl integration (iOS)
  cloud/                Device farm provider integrations
  ai/                   Self-healing + AI test generation
  visual/               Screenshot visual regression
  plugin/               YAML-defined custom commands
  report/               HTML report generation
probe_agent/            Dart package (runs on-device)
tools/probe-convert/    Multi-format test converter
vscode/                 VS Code extension
website/                Documentation site (Starlight/Astro)
tests/                  E2E test suites and health checks

Full documentation: alphawavesystems.github.io/flutter-probe

• Installation Guide
• Quick Start
• ProbeScript Syntax
• CLI Reference
• CI/CD Integration
• Configuration Reference
• Cloud Providers
• Visual Regression

On Android, the app needs ~5 seconds to boot after restart. Always add wait 5 seconds after restart the app. On iOS simulators, the navigation stack may persist across restarts — use pushNamedAndRemoveUntil in your Flutter app instead of pushReplacementNamed.

Use text selectors (tap "Settings") instead of ID selectors (tap #nav_settings) for ListTile navigation. The framework finds the widget by key but the tap may not hit the interactive area. Text selectors target the Text widget which is always within the tappable zone.

Add wait 1 seconds or wait 2 seconds after every navigation tap to let the page transition complete before asserting. The widget tree needs time to rebuild after a route push.

Don't use restart the app inside before each hooks — it creates WebSocket reconnection issues. Instead, put restart the app + wait 5 seconds inside each test body for full isolation.

Wrap variables in quotes: see "<expected>" not see <expected>. The <variable> syntax requires the enclosing quotes to be treated as a text selector.

FlutterProbe uses am start -n {package}/.MainActivity for Android. Ensure your app's AndroidManifest.xml has MainActivity as the launcher activity. This is the default for Flutter apps created with flutter create.

Use composite test. Write device-tagged step blocks (A:, B:) and use sync "label" as a barrier. See Composite Tests above.

Add --composite-device ALIAS=SPEC flags (one per alias) or set composite.devices in probe.yaml. WiFi mode is recommended: --composite-device "A=192.168.1.10:48686/token". The spec format is host:port/token for WiFi, a simulator UDID, or an ADB serial.

Yes. test "..." and composite test "..." coexist in any .probe file. Regular tests run on the primary device; composite tests run multi-device. If composite devices are not configured, only the composite tests are skipped — regular tests run normally.

# Local: auto-discover all connected devices
probe test tests/ --parallel

# CI: split files across matrix jobs (each job = 1 emulator)
probe test tests/ --shard 1/3 --device emulator-5554

Use matrix sharding — 3 parallel CI jobs each running 1/3 of the test files:

strategy:
  matrix:
    shard: ["1/3", "2/3", "3/3"]
steps:
  - run: probe test tests/ --shard ${{ matrix.shard }} -v -y

BrowserStack, Sauce Labs, AWS Device Farm, LambdaTest (interactive via WebSocket relay), and Firebase Test Lab (batch mode only). All relay-compatible providers support --parallel for concurrent multi-device execution.

1. Run with -v for step-by-step output: probe test tests/my_test.probe -v The runner prints → step before each step runs and ✓/✗ step (Xs) after. For steps that take longer than 5 seconds a ⏱ progress tick is emitted every 5 seconds, and a ⚠ warning appears at 80% of the timeout — so you see a slow or stuck step well before it fails. Even without -v, a faint status line shows the current step on a TTY.
2. Check failure screenshots in reports/screenshots/
3. Add pause or take a screenshot called "debug" at the point of failure
4. Use dump the widget tree to inspect what's on screen

Flutter 3.19+ with Dart 3.3+. The ProbeAgent uses package:flutter/services.dart APIs that require these versions.

Business Source License 1.1 — free for all use except competing commercial hosted testing services. Converts to Apache 2.0 after 4 years per release.

Copyright © 2026 Alpha Wave Systems S.A. de C.V.