This crate provides the Rust-native Kind e2e harness and shared primitives used by the separate real-cluster fault-test runner.

The harness is intentionally separated from the main operator crate so e2e-only dependencies stay scoped to the e2e/ manifest while still being validated by make e2e-check and the default make pre-commit path. It is driven through the reduced live entrypoints e2e-live-create, e2e-live-run, e2e-live-update, and e2e-live-delete.

The harness is split into five top-level domains:

• manifests/: e2e-owned static manifests such as the dedicated Kind config.
• framework/: reusable infrastructure primitives.
• fault/: real-cluster fault-test orchestration and fault-specific helpers.
• cases/: release test-case inventory grouped by product boundary.
• tests/: executable suite entrypoints; live tests are ignored by default and run only through explicit Make targets.

e2e/
  Cargo.toml
  FAULT_TESTING.md     package-local fault-test operations manual
  scripts/
    fault-test.sh      guarded real-cluster fault-test orchestration
  manifests/
    kind-rustfs-e2e.yaml  dedicated 1 control-plane + 3 worker Kind cluster
  src/
    lib.rs
    bin/rustfs-e2e.rs  Makefile-internal helper for live workflow steps
    fault/
      config.rs          real-cluster fault-test configuration and safety checks
      scenarios.rs       fault scenario catalog
      plan.rs            fault plan expansion for one or more fault injections
      runner.rs          destructive fault-test orchestration
      fixture.rs         fault namespace ownership and real-cluster Tenant fixture
      backends/          Chaos Mesh and host-side fault backends
      workload.rs        S3 workload generation and execution
      history.rs         workload operation recorder
      checker.rs         committed-object correctness checker
    framework/
      config.rs          dedicated Kind e2e configuration
      command.rs         safe subprocess wrapper for kind/docker/kubectl
      kind.rs            Kind cluster lifecycle and host mount preparation
      kubectl.rs         kubectl command construction boundary
      live.rs            live-run guardrails and context safety
      tools.rs           local host tool inventory
      kube_client.rs     kube-rs client boundary
      console_client.rs  reqwest Console API boundary
      wait.rs            timeout/polling helpers and Tenant Ready wait
      artifacts.rs       failure artifact collection boundary
      port_forward.rs    kubectl port-forward boundary
      images.rs          operator/console/rustfs image set boundary
      resources.rs       namespace/Secret/Tenant apply boundary
      storage.rs         local StorageClass/PV preparation boundary
      assertions.rs      Kubernetes and Tenant status assertions
      tenant_factory.rs  Kind-local and real-cluster Tenant templates
    cases/
      smoke.rs           install and readiness checks
      operator.rs        Tenant status and observed-generation checks
      console.rs         Console API health/readiness/OpenAPI checks
  tests/
    smoke.rs             ignored live smoke entrypoints
    operator.rs          ignored live Operator assertion
    console.rs           ignored live Console API assertion
    faults.rs            real-cluster destructive fault-injection suite with scenario-selected runners; not part of e2e case inventory

1. framework::command is the only layer that should execute host commands directly.
2. framework::kubectl is the shell/Kubernetes YAML boundary and must always pin --context.
3. framework::kube_client is the typed Kubernetes API boundary.
4. framework::console_client is the HTTP boundary for Console API tests.
5. framework::storage owns Kind local PV setup; framework::resources owns shared namespace/Secret/Tenant lifecycle.
6. framework::live owns live-run opt-in and dedicated-context checks.
7. cases/* should describe behavior and call framework helpers; avoid shell details there.
8. Kind e2e cases remain in cases/*; real-cluster fault tests are intentionally excluded from that inventory.
9. Destructive real-cluster fault tests are documented only in FAULT_TESTING.md.

Default configuration targets a dedicated Kind cluster:

cluster:          rustfs-e2e
context:          kind-rustfs-e2e
operator ns:      rustfs-system
test namespace:   rustfs-e2e-smoke
tenant name:      e2e-tenant
console URL:      http://127.0.0.1:19090
rustfs image:      rustfs/rustfs:1.0.0
storage class:    local-storage
PV count:         12
kind config:      e2e/manifests/kind-rustfs-e2e.yaml

Set RUSTFS_E2E_SERVER_IMAGE to exercise a branch or custom RustFS image. The harness writes operator.rustfs.com/runtime-default-image-ack with that exact reference so the Tenant can use the Operator's RuntimeDefault seccomp default. A mutable override can drift while the annotation stays unchanged; use a digest when reproducibility matters. The built-in default does not receive the annotation, so the smoke suite continues to test the normal implicit-default path.

Live tests are #[ignore] and run through the reduced Make workflow. The Makefile injects RUSTFS_E2E_LIVE=1 internally, so the common flow does not need the environment prefix:

make e2e-live-run

The harness refuses to run live tests unless the active Kubernetes context matches the configured dedicated Kind context.

make e2e-check

This runs e2e formatting, non-live tests, and clippy. Live tests remain #[ignore] and require the live commands below.

• make e2e-live-create: Creates the dedicated live environment: builds the e2e image, removes old kind-rustfs-e2e, cleans dedicated storage, then performs create + image load.
• make e2e-live-run: Runs all live suites (smoke/operator/console) in an existing live environment.
• make e2e-live-update: Rebuilds the e2e image and updates it into the live environment (load + rollout). Prerequisite: control-plane components must already be deployed (usually after make e2e-live-run).
• make e2e-live-delete: Deletes the live cluster and cleans dedicated storage at /tmp/rustfs-e2e-storage-{1,2,3}.

Image builds use Docker host network internally to avoid local bridge DNS resolution issues for npm/crates registries; the exposed user entry points remain only these four commands.

Recommended workflow:

# Initial setup
make e2e-live-create

# Run all suites once (deploys control plane and creates tenant)
make e2e-live-run

# Rebuild image and restart deployment after code changes
make e2e-live-update

# Run suites again after rollout
make e2e-live-run

# Clean up
make e2e-live-delete