Benchmark Dragonfly, Redis, and Valkey on real AWS infrastructure with one command, or drive a long-lived benchmark environment across many runs.

dfbench provisions an isolated VPC plus two (or three) EC2 instances via a single CloudFormation stack, runs memtier_benchmark (or dfly_bench for the master-replica scenario) over SSH, produces JSON/CSV results you can diff between runs, and tears the whole thing down with one call.

• Two modes
• Commands at a glance
• Prerequisites
• Install
• 30-second quickstart
• Stateful lifecycle
• Per-command benchmark
• Replication scenario
• Command reference
• Layout on disk
• Result file schemas
• Security notes
• Troubleshooting
• FAQ
• Further reading

All three modes share the same CloudFormation template, the same SSH key material, and the same results schemas, so files produced by any mode can be fed into dfbench compare.

One-liner per subcommand; jump into the full section for flags and examples.

The "mutates" commands take an advisory POSIX flock on ~/.dfbench/runs/<id>/.lock; the "read-only" commands never block, so info / list / ssh --print / template work mid-benchmark.

• Go 1.26 (repository is github.com/dragonflydb/benchmarking, module requires Go 1.26).
• AWS credentials on the current shell (profile, env vars, or SSO).
• Permission to create the AWS resources listed below.
• A clock: a typical quickstart run takes 10-15 minutes end to end.

Cost. Two c7g.4xlarge instances in us-east-1 cost around $1.16/hr; a quickstart session is roughly 15-30 minutes. The approval screen prints a live hourly estimate from pricing/pricing.go so you can sanity-check before you hit y.

IAM actions required. Any policy granting the actions below is sufficient; AdministratorAccess is the simplest choice for a demo.

• cloudformation:CreateStack, DeleteStack, DescribeStacks, DescribeStackEvents, UpdateStack
• ec2:* on the resources the template manages (VPC, subnet, IGW, route table, SG, key pair, placement group, instance)
• ssm:GetParameters on /aws/service/canonical/ubuntu/server/22.04/*
• sts:GetCallerIdentity (used only for the pre-flight auth check)

No dfbench-specific IAM role is created; the credentials in your shell are the credentials CloudFormation will see.

From the repo root:

make dfbench
# -> build/dfbench

This builds the single binary build/dfbench. The make all target includes dfbench alongside dfcloud, dfadmin, dfctl, dfk, and dftest so a fresh checkout picks it up automatically.

./build/dfbench quickstart \
  --engines dragonfly,redis,valkey \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64

What happens:

1. A 4096-bit SSH key pair is generated locally at ~/.dfbench/runs/<id>/ssh/.
2. A CloudFormation template is rendered, saved to disk, and printed as a plain-English plan plus $/hr estimate.
3. You approve with y. The stack is created with OnFailure=DELETE (automatic rollback on any failure).
4. memtier_benchmark is built on the client EC2; Dragonfly, Redis, and Valkey are installed on the server EC2.
5. Each engine is started, warmed up, benchmarked for --trials trials (default 3), stopped, and 30-second cooldown before the next.
6. A comparison table is printed to stdout and results.json is saved under ~/.dfbench/runs/<id>/.
7. The stack is deleted. Pass --skip-destroy to keep it running.

Skip the approval prompt with --yes or by exporting DFBENCH_NO_CONFIRM=1.

Use the stateful flow when you want to iterate. Each step below maps to one subcommand.

flowchart LR
  U[User] --> S["dfbench setup --id NAME"]
  S --> Plan[Render CFN + print plan + cost]
  Plan --> Approve["Prompt y/N"]
  Approve -->|yes| Create[CreateStack]
  Create --> Wait[Wait for CREATE_COMPLETE]
  Wait --> InstMem[Install memtier on client]
  InstMem --> InstEng[Pre-install engines on server and replicas]
  InstEng --> Ready["status=ready"]
  Ready --> R["dfbench run --id NAME"]
  R --> Trials[Start engine, warmup, trials, stop]
  Trials --> Save[Append BenchmarkRun to state.json]
  Save --> Ready
  Ready --> RR["dfbench run-replication --id NAME"]
  RR --> RepFlow[Master + replica + sustained load]
  RepFlow --> Ready
  Ready --> I["dfbench info / ssh / list"]
  Ready --> D["dfbench destroy --id NAME"] --> Gone[DeleteStack]

dfbench setup \
  --id perf-may \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64

Prints the plan, asks for confirmation, provisions the stack, and pre-installs every engine you select via --engines (default: dragonfly,redis,valkey). Use --dry-run to stop after the plan, or --yes to skip the prompt.

Everything from this point lives under ~/.dfbench/runs/perf-may/.

dfbench run --id perf-may \
  --engines dragonfly,redis,valkey \
  --memtier-threads 8 --memtier-clients 50 \
  --memtier-requests 1000000 --memtier-pipeline 30 \
  --trials 3

Each invocation:

• Writes artifacts under ~/.dfbench/runs/perf-may/benchmarks/<timestamp>/.
• Appends a BenchmarkRun record to the setup's state.json.
• Restores status to ready on exit (including on Ctrl+C) so the setup stays usable.

• dfbench list - all known setups with status and age.
• dfbench info --id perf-may - detailed status, instance IPs, SSH cheat-sheet, installed engines, past benchmarks, running cost.
• dfbench ssh --id perf-may --target server - interactive shell on the server instance. Use --print to print the exact ssh -i ... ubuntu@<ip> command instead of running it.

dfbench destroy --id perf-may

Deletes the CloudFormation stack and removes the local SSH key files. Idempotent; if the stack is already gone (manual cleanup, prior rollback) the command still succeeds.

Use this scenario when you want a per-command performance profile instead of a single mixed SET:GET number: each invocation runs one dedicated memtier_benchmark test for one Redis command, per engine, using memtier's arbitrary-command mode (--command="SET __key__ __data__"). Benchmark commands one at a time; each run produces its own results file.

dfbench setup \
  --id percmd \
  --server-instance m7g.8xlarge --server-arch arm64 \
  --client-instance c6gn.8xlarge --client-arch arm64

dfbench run-commands --id percmd \
  --engines dragonfly,redis,valkey \
  --command SET --dragonfly_num_shards -1 \
  --memtier-threads 32 --memtier-clients 5 --memtier-pipeline 20

Repeat with the next command whenever you're ready - the setup stays provisioned between runs (--dragonfly_num_shards is required whenever dragonfly is among the engines; -1 auto-derives the tuned value):

dfbench run-commands --id percmd --command GET --dragonfly_num_shards -1
dfbench run-commands --id percmd --command LPUSH --dragonfly_num_shards -1
dfbench run-commands --id percmd --command LRANGE --dragonfly_num_shards -1

For each engine the scenario:

1. FLUSHALL, then preloads the keyspace if the command reads existing data. Preloads match the data type: GET/EXISTS/DEL get plain strings, LRANGE/LPOP get 100-element lists, SMEMBERS gets 10-member sets, and so on. Preloads use memtier's parallel key pattern so the whole key range is covered exactly once per item.
2. Runs a discarded warmup pass (--warmup-time, default 10s).
3. Runs --trials measured trials of --test-time seconds each. Mutating commands (SET, INCR, LPUSH, ...) get a fresh flush + preload before every trial; read-only commands (GET, LRANGE, ...) keep their preloaded keyspace across trials.

Supported commands: strings (SET, GET, MSET, MGET, INCR, APPEND), keyspace ops (EXISTS, EXPIRE, TTL, DEL), lists (LPUSH, RPUSH, LPOP, RPOP, LRANGE), hashes (HSET, HGET, HGETALL), sets (SADD, SISMEMBER, SMEMBERS), sorted sets (ZADD, ZSCORE, ZRANGE), and PING.

Draining commands (DEL, LPOP, RPOP) can empty their preloaded keyspace mid-trial; their numbers include miss-path operations and are marked approximate in the results.

Compare two runs of the same command (e.g. before/after an engine upgrade):

dfbench compare \
  ~/.dfbench/runs/percmd/commands/20260706-120000/results.json \
  ~/.dfbench/runs/percmd/commands/20260707-093000/results.json

Auto-detects the per-command shape (via the top-level scenario: "commands" tag) and diffs median throughput, p50, p99, p99.9, and avg latency per engine, with the same exit-code contract as the other comparators.

To drive Dragonfly at its optimal single-node configuration (the setup that reaches ~10M QPS for SET on a c7g.8xlarge, data-bound rather than network-bound), combine server network tuning with Dragonfly's shard/connection-affinity flags.

Provision with --tune-network so net_tune.sh pins the NIC IRQs and RPS/XPS steering to the top CPUs (e.g. 16-31 on a 32-vCPU box), leaving the low CPUs (0-15) free for shards:

dfbench setup --id maxqps --tune-network \
  --server-instance c7g.8xlarge --server-arch arm64 \
  --client-instance c7gn.8xlarge --client-arch arm64 \
  --engines dragonfly

If you plan to run a Dragonfly build compiled against a newer glibc (e.g. one that needs GLIBC_2.38), add --ubuntu-version 24.04 so the server launches with a matching OS. The default is Ubuntu 22.04 (glibc 2.35), which runs the published releases but not builds linked against 2.38:

dfbench setup --id maxqps --tune-network --ubuntu-version 24.04 \
  --server-instance m7g.8xlarge --server-arch arm64 \
  --client-instance c6gn.8xlarge --client-arch arm64 \
  --engines dragonfly

Then benchmark. --dragonfly_num_shards must be stated explicitly when benchmarking dragonfly (pass -1 for the auto-derived tuned value); --dragonfly_conn_use_incoming_cpu and --redis_pin_network / --valkey_pin_network default to the tuned configuration:

dfbench run-commands --id maxqps --engines dragonfly \
  --command SET --memtier-threads 32 --memtier-clients 5 \
  --memtier-pipeline 35 --memtier-data-size 128 \
  --dragonfly_num_shards -1 \
  --key-maximum 100000000 --test-time 300 \
  --trials 1 --warmup-time 0 --distinct-client-seed

This launches Dragonfly with --conn_use_incoming_cpu and --num_shards=$(VCPUS - 2*NUM_RX_QUEUES) (16 on a 32-vCPU/8-queue box). Validate on the server with htop: the low CPUs should be ~100% green (shards, userspace) and the high CPUs mixed green/red (network I/O). Ensure the client is not the bottleneck - the result should be data-bound (well under the NIC line rate).

Notes and caveats:

• --num_shards is coupled to the NIC RX-queue count; the auto value only equals 16 on 32-vCPU/8-queue instances, which is why it is derived on the server rather than hardcoded.
• Redis/Valkey pinning (--redis_pin_network / --valkey_pin_network) is best-effort: taskset constrains the process cpuset (CPU 0 plus the I/O CPUs), but the engines offer no per-thread affinity guarantee. Pair with --redis_io_threads 8 / --valkey_io_threads 8 (or 16).
• Comparing binaries (e.g. latest vs an older tag): keep the tuning, shard, and memtier settings byte-identical between runs so any delta is attributable to the binary. setup installs the latest release; use run-commands --dragonfly_version to reinstall a specific build on the same tuned setup right before a run, then diff the result files:

# Baseline: pin an older release.
dfbench run-commands --id maxqps --engines dragonfly --command SET \
  --dragonfly_version v1.31.0 --dragonfly_num_shards -1 \
  --memtier-pipeline 35 --memtier-data-size 128 --test-time 300 --trials 1

# Candidate: latest release (or a nightly/custom tarball URL).
dfbench run-commands --id maxqps --engines dragonfly --command SET \
  --dragonfly_version latest --dragonfly_num_shards -1 \
  --memtier-pipeline 35 --memtier-data-size 128 --test-time 300 --trials 1

--dragonfly_version accepts a release tag (v1.31.0), latest, a full https:// URL to a dragonfly-<arch>.tar.gz tarball, or a path to a local file. Drop an unreleased binary someone hands you into the repo and point at it directly; dfbench uploads it to the server and installs it (raw binary or .tar.gz, extracted automatically):

# Local binary dropped into the repo (path is relative to your CWD).
dfbench run-commands --id maxqps --engines dragonfly --command SET \
  --dragonfly_version ./bin/dragonfly --dragonfly_num_shards -1 \
  --memtier-pipeline 35 --memtier-data-size 128 --test-time 300 --trials 1

The uploaded binary must match the server's CPU architecture (arm64 for the c7g/c7gn instances above). The installed version is recorded per-engine in results.json.

Use this scenario to measure Dragonfly replication: sync duration under a given dataset size, sustained RPS and p99 latency during sync, and the effect of replica-side flags like --replication_stream_compression=zstd.

dfbench setup \
  --id rep-demo \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64 \
  --replicas 1

--replica-instance and --replica-arch default to the server's values, so the simplest form is what is shown above. Setup pre-installs Dragonfly on the replica(s) concurrently (up to four at a time), so subsequent run-replication invocations pay zero install cost.

dfbench run-replication --id rep-demo \
  --cases qps=100000,qps=50000,maxload \
  --test-time 30 \
  --preload-keys 500000 \
  --value-size 4096 \
  --ratio 1:0 \
  --master-flags "--maxmemory=25gb" \
  --replica-flags "--maxmemory=25gb --replication_stream_compression=zstd"

For each case (one row of the QPS sweep) the scenario:

1. Starts Dragonfly on the server as master.
2. FLUSHALL and preloads the keyspace from the client using dfly_bench for 30s.
3. Sets a READY=true sentinel on the master and starts the replica with --replicaof=<master-private-ip>:6379.
4. Waits for the Transitioned into stable sync log marker and records sync_duration_ms.
5. Drives sustained load for --test-time seconds at the target QPS while a background goroutine polls INFO replication every 2s; a drop in connected_slaves from 1 to 0 ends the case early with termination_reason=replica_disconnect.
6. FLUSHALL ASYNC and kills both processes so the next case starts clean.

dfbench compare \
  ~/.dfbench/runs/rep-demo/replications/20260408-143217/results.json \
  ~/.dfbench/runs/rep-demo/replications/20260408-152103/results.json

Auto-detects that the files are replication results (via the top-level scenario: "replication" tag) and prints a per-case table:

• median_rps (higher is better)
• median_p99_us (lower is better)
• sync_duration_ms (lower is better)

Exit codes: 0 (within tolerance), 1 (any metric regressed beyond --regression-pct, default 5%), 2 (input error: unreadable file or mismatched shapes).

The flag choices in bench/replication/dflybench.go mirror playground/replication-bench/benchmark.py exactly so dfbench numbers are comparable with the upstream Python prototype:

• Preload: --proactor_threads=4 -c=15 --pipeline=10 --qps=0 --test_time=30. 60 concurrent writers, random keys over [0, --preload-keys). --preload-keys names the keyspace range (the prototype's --key_maximum), not the number of keys actually written.
• Load: --proactor_threads=1 -c=2 --pipeline=5 --qps=<case QPS>. Two connections is the crucial bit: dfly_bench's --qps is a per-connection cap, so a qps=100000 case schedules 200k ops/s total. maxload emits --qps=0 (explicitly "no cap"); omitting the flag would silently pick up dfly_bench's default of 20 req/sec.

Every subcommand runs under an advisory POSIX flock on ~/.dfbench/runs/<id>/.lock to prevent concurrent mutations on the same setup.

Provision a stateful benchmark environment. Persists state at ~/.dfbench/runs/<id>/state.json and refuses to overwrite an existing setup unless it was previously destroyed.

Key flags (all optional except --id is auto-generated from a timestamp if omitted):

Example - provision a two-replica arm64 fleet in eu-west-1, but stop after rendering the plan so you can review it before committing any AWS spend:

dfbench setup \
  --id perf-eu \
  --region eu-west-1 \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64 \
  --replicas 2 \
  --dry-run

The rendered CloudFormation template is saved at ~/.dfbench/runs/perf-eu/stack.yaml. Drop the --dry-run and rerun once you're happy.

Execute a memtier_benchmark benchmark against an existing setup. Requires --id; every engine listed via --engines must already be installed (see dfbench info). The per-run memtier flags below are not persisted in state, so pass them on every invocation.

Example - dragonfly-only, no warmup, distinct-client-seed ON, stdout as JSON redirected to a file for scripting:

dfbench run --id perf-may \
  --engines dragonfly \
  --memtier-threads 8 --memtier-clients 100 \
  --memtier-requests 5000000 --memtier-pipeline 50 \
  --memtier-ratio 1:4 \
  --warmup-requests 0 \
  --distinct-client-seed \
  --trials 5 \
  --output json > /tmp/perf-may-2026-04-08.json

Progress banners go to stderr when --output is json or csv, so the redirected file is a single well-formed payload.

Benchmark one Redis command against an existing setup. Requires --id and either --command (a built-in command) or --command-template (a custom one); writes to ~/.dfbench/runs/<id>/commands/<timestamp>/. Run it once per command you want to profile.

Example - quick single-engine, single-trial smoke run on GET with a small keyspace (not representative numbers; use the defaults for real measurements):

dfbench run-commands --id percmd \
  --engines dragonfly \
  --command GET --dragonfly_num_shards -1 \
  --key-maximum 1000000 --test-time 15 --trials 1

To benchmark a command that isn't in the built-in registry, replace --command <name> with --command-template (the two are mutually exclusive). The template uses memtier's arbitrary-command form with the same __key__ / __data__ placeholders as the built-in specs (__data__ is filled from --memtier-data-size).

Example - a mutating custom command (SETEX) and a read command (GETRANGE) preloaded with plain strings:

dfbench run-commands --id percmd --engines dragonfly --dragonfly_num_shards -1 \
  --command-template "SETEX __key__ 3600 __data__" --command-name SETEX --command-mutates

dfbench run-commands --id percmd --engines dragonfly --dragonfly_num_shards -1 \
  --command-template "GETRANGE __key__ 0 50" --command-name GETRANGE \
  --preload-template "SET __key__ __data__"

Budget note: one run is roughly (engines) x (trials x test-time + warmup + preload + pauses); the default 3 engines x 3 x 300s lands around 50-60 minutes per command. Preloading a 100M keyspace for read commands adds roughly 20-60s per engine depending on its write throughput.

Memory note: size --key-maximum to the smallest engine memory you benchmark. 100M x 256B assumes a ~128GB server; on smaller instances lower it proportionally or the engines will evict or OOM during preload.

The dfly_bench counterpart to run-commands: it drives the same per-command flush/preload/warmup/trials cycle, but with Dragonfly's own load generator instead of memtier. dfly_bench is built from the dragonflydb/dragonfly source and installed on the client at dfbench setup time (alongside memtier); override the build with --dfly-bench-ref. Results use the same schema as run-commands (tagged tool="dfly_bench") and land in ~/.dfbench/runs/<id>/commands/<timestamp>/, so a memtier file and a dfly_bench file for the same command diff with dfbench compare.

Because dfly_bench uses its own key prefix (--key_prefix, default key:), read commands (GET, ...) are preloaded by dfly_bench itself (sequential distribution, one connection per thread covering the whole --key-maximum range) rather than reusing a memtier preload.

Flag naming: client load generator vs. server engine. The prefix tells you what a flag tunes:

• --memtier-* (in run-commands) and --dfly-bench-* (in run-commands-dfly) configure the client-side load generator (memtier_benchmark and dfly_bench respectively). The two commands are deliberately kept separate so it's always explicit which tool is driving load.
• --dragonfly_*, --redis_*, --valkey_* configure the server engine under test and behave identically in both commands. In particular, --dragonfly_num_shards is Dragonfly server tuning, not a client flag.

So --dfly-bench-threads sets dfly_bench's --proactor_threads on the client, while --dragonfly_num_shards sets Dragonfly's shard count on the server; they are unrelated.

The --dfly-bench-* flags were previously named --dfly-* (--dfly-threads, --dfly-conns, --dfly-pipeline, --dfly-data-size). Those spellings still work as hidden, deprecated aliases (they print a deprecation notice); prefer the --dfly-bench-* names.

dfly_bench's --json_out_file writes a memtier-compatible latency report, so the same parser produces the throughput / p50 / p99 / p99.9 / avg columns for both tools.

dfbench run-commands-dfly --id percmd \
  --engines dragonfly \
  --command GET --dragonfly_num_shards -1 \
  --key-maximum 1000000 --test-time 15 --trials 1

Execute the Dragonfly master/replica benchmark scenario against an existing setup that has at least one replica. Writes to ~/.dfbench/runs/<id>/replications/<timestamp>/.

Example - compression A/B test. Run the same single-case sweep twice against the same setup, once with zstd replication-stream compression enabled and once without, then diff the two JSON files:

dfbench run-replication --id rep-demo \
  --cases qps=80000 --test-time 60 \
  --preload-keys 500000 --value-size 4096 --ratio 1:0 \
  --master-flags  "--maxmemory=25gb" \
  --replica-flags "--maxmemory=25gb" \
  --output-dir ~/.dfbench/runs/rep-demo/replications/nocomp/

dfbench run-replication --id rep-demo \
  --cases qps=80000 --test-time 60 \
  --preload-keys 500000 --value-size 4096 --ratio 1:0 \
  --master-flags  "--maxmemory=25gb" \
  --replica-flags "--maxmemory=25gb --replication_stream_compression=zstd" \
  --output-dir ~/.dfbench/runs/rep-demo/replications/zstd/

dfbench compare \
  ~/.dfbench/runs/rep-demo/replications/nocomp/results.json \
  ~/.dfbench/runs/rep-demo/replications/zstd/results.json

One-shot: provision, benchmark, tear down. Accepts the union of setup and run flags, plus --skip-destroy to leave the stack running. Good for CI and demos; use the stateful flow for iteration.

Example - Dragonfly-only, five trials, keep the stack up after the run so you can SSH in and poke around (useful when numbers look surprising):

./build/dfbench quickstart \
  --engines dragonfly \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64 \
  --trials 5 \
  --skip-destroy \
  --yes

Remember to follow up with dfbench list and dfbench destroy --id <ts> once you're done - quickstart auto-generates the ID from the current timestamp.

dfbench compare <baseline.json> <candidate.json>

Auto-detects the shape of each file (memtier when the top-level has engines; replication when it has scenario: "replication" or cases; per-command when it has scenario: "commands" or commands) and emits a per-metric diff table or JSON payload.

Exit codes: 0 (within tolerance), 1 (regression), 2 (input error).

Memtier example - diff two dfbench run results with a tighter 2% tolerance:

dfbench compare \
  ~/.dfbench/runs/perf-may/benchmarks/20260408-143217/results.json \
  ~/.dfbench/runs/perf-may/benchmarks/20260409-091802/results.json \
  --regression-pct 2.0

Replication example with JSON output, suitable for piping into jq or a downstream dashboard:

dfbench compare baseline.json candidate.json --format json \
  | jq '.cases[] | select(.regressions != [])'

CI wiring - branch on the exit code so a flaky run doesn't silently pass:

dfbench compare baseline.json candidate.json
case $? in
  0) echo "ok: candidate within tolerance" ;;
  1) echo "REGRESSION; see output above"; exit 1 ;;
  2) echo "input error (missing / mismatched files)"; exit 2 ;;
esac

All six are thin wrappers around the local state file or the CloudFormation API; none of them re-run any benchmark work. Examples are copy-pasteable against a real setup called perf-may.

Read-only dump from state.json. No AWS calls, no side effects. Shows status, instance metadata (public + private IPs), SSH cheat-sheet, installed-engine inventory, past benchmark history, running cost estimate, and next-step hints.

dfbench info --id perf-may

Sample output (truncated):

Setup: perf-may
  Status:       ready
  Created:      2026-04-08T14:30:11Z (17m ago)
  Region:       us-east-1
  Stack:        dfbench-perf-may
  AWS Account:  123456789012

Instances:
  server:  c7g.4xlarge arm64   public=3.88.12.4   private=10.0.1.42
  client:  c7g.4xlarge arm64   public=3.88.12.17  private=10.0.1.88
  Est. cost:    $1.156/hr  (running ~17m -> ~$0.33 so far)

SSH:
  Key:    /Users/ops/.dfbench/runs/perf-may/ssh/id_rsa
  Server: ssh -i /Users/ops/.dfbench/runs/perf-may/ssh/id_rsa ubuntu@3.88.12.4
  Client: ssh -i /Users/ops/.dfbench/runs/perf-may/ssh/id_rsa ubuntu@3.88.12.17

Engines (pre-installed on server):
  NAME       VERSION              STATUS
  dragonfly  dragonfly v1.27.0    installed
  redis      Redis server v=7.4   installed
  valkey     Valkey server v=7.2  installed

Next steps:
  dfbench run     --id perf-may --engines dragonfly,redis,valkey
  dfbench ssh     --id perf-may --target server
  dfbench destroy --id perf-may

Every setup with local state, newest first. By default destroyed rows are hidden so you see only actionable setups.

dfbench list

ID         STATUS  REGION     SERVER               CLIENT               AGE
perf-may   ready   us-east-1  c7g.4xlarge arm64    c7g.4xlarge arm64    17m
rep-demo   ready   us-east-1  c7g.4xlarge arm64    c7g.4xlarge arm64    2h14m

Add --all to include destroyed setups (handy for audit / recovering an ID from a week-old state file):

dfbench list --all

Interactive shell via syscall.Exec - dfbench replaces itself with /usr/bin/ssh, so your terminal becomes the session. --target defaults to server; valid values are server, client, and replica-N (0-based).

dfbench ssh --id perf-may                    # defaults to --target server
dfbench ssh --id rep-demo --target replica-0

Use --print when you want the command as a string (for scripts, Wiki pages, or a second terminal tab):

dfbench ssh --id perf-may --target client --print
# -> ssh -i /Users/ops/.dfbench/runs/perf-may/ssh/id_rsa -o StrictHostKeyChecking=no \
#        -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR ubuntu@3.88.12.17

Delete the CloudFormation stack and the local SSH key material. Asks for confirmation unless --yes. Idempotent: a second invocation against an already-deleted stack succeeds with a short note.

dfbench destroy --id perf-may --yes
# [1/2] Deleting CloudFormation stack dfbench-perf-may...
# [2/2] Removing local SSH key files...
# Done.

dfbench destroy --id perf-may --yes
# setup "perf-may" not found (run 'dfbench list' to see available setups)

Render the CloudFormation YAML dfbench would apply for a given flag set, without calling AWS. Convenient for security review and for piping into aws cloudformation validate-template. Uses --run-id (not --id) because no state file is involved.

dfbench template \
  --run-id preview \
  --server-instance c7g.4xlarge --server-arch arm64 \
  --client-instance c7g.4xlarge --client-arch arm64 \
  --replicas 1 \
  --out /tmp/dfbench-preview.yaml

aws cloudformation validate-template \
  --template-body file:///tmp/dfbench-preview.yaml

The embedded public key is a harmless placeholder; pass --public-key ~/.ssh/mykey.pub if you want the preview to reflect real key material.

dfbench version
# dfbench v0.3.0 (commit abc1234, built 2026-04-08)

Every setup gets its own self-describing directory under ~/.dfbench/runs/:

~/.dfbench/runs/<id>/
  state.json                        lifecycle + AWS resources + engine + benchmark history
  stack.yaml                        CloudFormation template dfbench applied (saved before any AWS call)
  .lock                             POSIX flock; setup/run/destroy acquire it exclusively
  ssh/
    id_rsa                          4096-bit RSA private key, mode 0600, never leaves the laptop
    id_rsa.pub                      public half, embedded in the CFN template as the EC2 key pair
  benchmarks/
    20260408-143217/
      config.json                   memtier flags for this particular run
      results.json                  aggregated + per-trial stats
      memtier/
        dragonfly-t1.json           raw memtier_benchmark JSON per (engine, trial)
        dragonfly-t2.json
        ...
  commands/
    20260706-120000/
      results.json                  CommandRunResults: per-command x per-engine aggregates
      results.csv                   one row per (command, engine, trial) + median rows
      memtier/
        dragonfly-SET-t1.json       raw memtier_benchmark JSON per (engine, command, trial)
        ...
  replications/
    20260408-143217/
      results.json                  ReplicationRunResults (see below)
      benchmark_results.csv         upstream playground schema: Run_Name, Time_Seconds, RPS_Now, p99_Latency_us

Who writes what:

Both commands produce self-contained JSON: every input flag, every instance type, every per-trial number is captured so compare can reconstruct what the run looked like months later.

{
  "run_id": "20260408-143217",
  "created_at": "2026-04-08T14:32:17Z",
  "completed_at": "2026-04-08T14:41:05Z",
  "status": "completed",
  "region": "us-east-1",
  "aws_account": "123456789012",
  "server_instance": "c7g.4xlarge",
  "client_instance": "c7g.4xlarge",
  "memtier": {
    "threads": 8, "clients": 50, "requests": 1000000,
    "ratio": "1:10", "data_size": 256, "pipeline": 30,
    "distinct_client_seed": false
  },
  "warmup_requests": 100000,
  "trials": 3,
  "engines": [
    {
      "engine": "dragonfly",
      "version": "dragonfly v1.27.0",
      "trials": [
        {"trial": 1, "throughput": 2646502, "p50": 4.6, "p99": 7.1, "p999": 9.8, "avg_latency": 4.72}
      ],
      "throughput": {"median": 2646502, "min": 2601000, "max": 2672000, "stddev": 35400},
      "p50": {...}, "p99": {...}, "p999": {...}, "avg_latency": {...}
    }
  ]
}

Schema defined in bench/output.go (RunResults) and bench/result.go (AggregateResult, TrialResult, Stats). The aggregation is median-centred because a single outlier trial (cold cache, noisy neighbour) should not drown out the signal from the other two.

{
  "scenario": "replication",
  "run_id": "20260408-143217",
  "setup_id": "rep-demo",
  "created_at": "...",
  "completed_at": "...",
  "region": "us-east-1",
  "master_instance": "c7g.4xlarge",
  "replica_instance": "c7g.4xlarge",
  "client_instance": "c7g.4xlarge",
  "dfly_bench_ref": "main",
  "cases": [
    {
      "run_name": "qps=100000",
      "preload_args": "dfly_bench --h=... --ratio=1:0 ...",
      "load_args":    "dfly_bench --h=... --ratio=1:0 ...",
      "master_flags":  ["--maxmemory=25gb"],
      "replica_flags": ["--maxmemory=25gb", "--replication_stream_compression=zstd"],
      "series": [
        {"t_sec": 1, "rps": 198500, "p99_us": 2100},
        {"t_sec": 2, "rps": 199800, "p99_us": 2050}
      ],
      "sync_duration_ms": 3427,
      "termination_reason": "timeout"
    }
  ]
}

Schema defined in bench/replication/results.go (ReplicationRunResults, ReplicationCase, RepSample). termination_reason is one of timeout, replica_disconnect, error, ctx_cancelled.

Alongside the JSON, a benchmark_results.csv with Run_Name,Time_Seconds,RPS_Now,p99_Latency_us is written so the upstream playground compare.py drops in unchanged.

{
  "scenario": "commands",
  "run_id": "20260706-120000",
  "setup_id": "percmd",
  "created_at": "...",
  "completed_at": "...",
  "status": "completed",
  "region": "us-east-1",
  "server_instance": "m7g.8xlarge",
  "client_instance": "c6gn.8xlarge",
  "memtier": {
    "threads": 32, "clients": 5, "pipeline": 20, "data_size": 256,
    "key_maximum": 100000000, "test_time_sec": 300, "warmup_time_sec": 10,
    "distinct_client_seed": true
  },
  "trials": 3,
  "commands": [
    {
      "command": "SET",
      "template": "SET __key__ __data__",
      "engines": [
        {
          "engine": "dragonfly",
          "version": "dragonfly v1.27.0",
          "trials": [
            {"trial": 1, "throughput": 2646502, "p50": 4.6, "p99": 7.1, "p999": 9.8, "avg_latency": 4.72}
          ],
          "throughput": {"median": 2646502, "min": 2601000, "max": 2672000, "stddev": 35400},
          "p50": {}, "p99": {}, "p999": {}, "avg_latency": {}
        }
      ]
    }
  ]
}

Schema defined in bench/commands/results.go (CommandRunResults, CommandResult); the per-engine aggregates reuse AggregateResult/Stats from bench/result.go. The commands array holds the single command the run measured (the array shape keeps the compare tooling uniform). Draining commands (DEL, LPOP, RPOP) additionally carry "approximate": true because their keyspace can empty mid-trial. A results.csv with one row per (engine, trial) plus median rows is written alongside.

• The SSH private key is 4096-bit RSA, generated locally inside ~/.dfbench/runs/<id>/ssh/id_rsa. The file is opened with O_CREATE|O_WRONLY|O_TRUNC, 0o600 so there is no window where the unencrypted bytes sit at the umask default of 0o644. See remote/sshkey.go.
• Only the public half (id_rsa.pub) ships to AWS, as AWS::EC2::KeyPair.PublicKeyMaterial inside the CloudFormation template.
• SSH port 22 is open to 0.0.0.0/0 on the VPC's security group. Acceptable for a short-lived, ephemeral benchmark fleet; not acceptable for long-lived hosts. If you keep setups around, narrow the rule by hand or tear them down with dfbench destroy.
• The Redis port (6379) is reachable only from inside the security group (intra-SG rule with IpProtocol: -1). The operator's laptop cannot hit it directly; replication RESP is tunneled through SSH (see remote/ssh.go DialTunnel and bench/replication/orchestrator.go DialVia).
• StrictHostKeyChecking=no and UserKnownHostsFile=/dev/null are deliberate: every EC2 instance dfbench talks to was minted seconds earlier by the same process, so TOFU on an ephemeral host key would be pure ceremony. See the comment in remote/ssh.go near InsecureIgnoreHostKey.
• destroy removes both key files on successful teardown. A failed teardown leaves them in place so you can retry.

Why CloudFormation and not raw EC2 API calls? The template is a single artifact you can diff, review, validate with aws cloudformation validate-template, and tear down atomically with DeleteStack. OnFailure=DELETE replaces dfbench's old hand-rolled cleanup registry: any resource that got created before a failure is rolled back server-side, no local LIFO bookkeeping needed.

Why a cluster placement group? It colocates the server, client, and replica on the same physical rack, giving sub-millisecond inter-node RTT. That's what makes benchmark numbers comparable across engines rather than swamped by AZ-level network jitter. The trade-off is the burstable/a1/mac instance-family restriction (see Troubleshooting).

Why does the client talk to the server over the private IP? Lower latency, no cross-AZ or NAT gateway charges, and the DB port never appears on the public internet.

Why does the operator laptop talk to everything over the public IP? No bastion overhead for a short-lived fleet, and the SG rule is already SSH-from-0.0.0.0/0.

Why do replication RESP calls tunnel through SSH? Master and replica listen on VPC-private addresses that the laptop cannot reach directly. Instead of adding an ingress rule for the operator's IP, dfbench uses DialTunnel (which sets up a TCP forwarding over the existing SSH session) to make the EC2 instance act as a bastion for RESP frames.

Why build dfly_bench from source on the client? Dragonfly does not publish dfly_bench as a release artifact. We pin to --dfly-ref because its stdout format has changed historically and the scrape regexes in bench/replication/dflybench.go are coupled to a known schema.

Why --preload-keys is not the number of keys written. It is the keyspace range (dfly_bench --key_maximum). Preload is time-boxed at 30 seconds matching the upstream playground, so at 4KB values and a small instance the DB ends up partially populated. That is intentional: it is the shape replication has to cope with under sustained load.

Why two connections (-c=2) during load? dfly_bench's --qps is a per-connection cap. qps=100000 -c=2 schedules 200k ops/s in total, which matches the upstream baseline line-for-line.

Why FLUSHALL ASYNC? Synchronous FLUSHALL after a write-heavy run can block for tens of seconds waiting for shard deletion. Replica teardown happens immediately afterwards, so we do not need the sync guarantee; ASYNC schedules and acks right away.