A tool for testing Ceph locally using nested rootless podman containers

ceph-devstack is a tool that can deploy and manage containerized versions of teuthology and its associated services, to test Ceph (or just teuthology) on your local machine. It lets you avoid:

• Accessing Ceph's Sepia lab
• Needing dedicated storage devices to test Ceph OSDs

Basically, the goal is that you can test your Ceph branch locally using containers as storage test nodes.

It is currently under active development and has not yet had a formal release.

☑︎ CentOS 9.Stream should work out of the box

☑︎ CentOS 8.Stream mostly works - but has not yet passed a Ceph test

☐ A recent Fedora should work but has not been tested

☒ Ubuntu does not currently ship a new enough podman

☒ MacOS will require special effort to support since podman operations are done inside a VM

• A supported operating system
• podman 4.0+ using the crun runtime.
  • On CentOS 8, modify /etc/containers/containers.conf to set the runtime
• Linux kernel 5.12+, or 4.15+ and fuse-overlayfs
• cgroup v2
  • On CentOS 8, see ./docs/cgroup_v2.md
• With podman <5.0, podman's DNS plugin, from the podman-plugins package
• A user account that has sudo access and also is a member of the disk group
• The following sysctl settings:
  • fs.aio-max-nr=1048576
  • kernel.pid_max=4194304
• If using SELinux in enforcing mode:
  • setsebool -P container_manage_cgroup=true
  • setsebool -P container_use_devices=true

ceph-devstack doctor will check the above and report any issues along with suggested remedies; its --fix flag will apply them for you.

sudo usermod -a -G disk $(whoami)  # and re-login afterward
git clone https://github.com/ceph/teuthology/
cd teuthology && ./bootstrap
python3 -m venv venv
source ./venv/bin/activate
python3 -m pip install git+https://github.com/zmc/ceph-devstack.git

ceph-devstack 's default configuration is here. It can be extended by placing a file at ~/.config/ceph-devstack/config.toml or by using the --config-file flag.

ceph-devstack config dump will output the current configuration.

To back loop devices with slices of a real NVMe device instead of sparse files, configure [block_pool] in your config. The pool is shared across ceph_node and all testnode_* containers. Each loop device uses that container's loop_device_size for its backing region on the pool parent.

[block_pool]
parent = "/dev/nvme0n1p1"   # or a dedicated whole disk
allow_enroll = true         # first run only; unset afterward

[containers.ceph_node]
loop_device_size = "5G"

[containers.testnode]
loop_device_size = "5G"

Requirements:

• Your user must be in the disk group and able to read/write the parent device directly (enrollment writes a tail marker to the raw device).
• The parent must be empty, or already enrolled by ceph-devstack (tail marker at the end of the device).

ceph-devstack block-pool status shows current allocations.

Each container picks host loop devices at create time: it needs loop_device_count devices of loop_device_size, reuses any existing backing files for that container name, then takes the lowest free loop numbers after scanning what is already attached or claimed in the stack's disk_images directory.

If block_pool.json is lost but the on-disk marker remains, delete the state file and start again; the pool reclaims the marker without reformatting the device.

To run a single-container local Ceph cluster instead of teuthology:

ceph-devstack --stack ceph pull
ceph-devstack --stack ceph create
ceph-devstack --stack ceph start
podman logs -f ceph_node
podman exec ceph_node ceph -c /var/lib/ceph-devstack/cluster/ceph.conf -s

The dashboard listens on port 8080 by default (admin / admin). Set dashboard_show_password = true under [containers.ceph_node] to print the password in the container log.

With image_builder = "binary-patch" (the default), the build-ceph stack:

1. Compiles Ceph binaries via src/script/build-with-container.py
2. Patches the compiled binaries into a base container image using src/script/cpatch.py, producing a local image (default localhost/ceph-cpatch:main)

Point the ceph stack at that image:

[containers.ceph_builder]
repo = "~/src/ceph"
# image_builder = "binary-patch"  # default

[containers.ceph_node]
image = "localhost/ceph-cpatch:main"

Then run the two stacks manually:

ceph-devstack --stack build-ceph start
ceph-devstack --stack ceph start

Or use depends to combine them (see below). This mode is faster for iterative development since it compiles only the changed C++ targets and patches them into an existing image rather than rebuilding RPM packages.

With image_builder = "package-build" under [containers.ceph_builder] (RPM distros such as centos9 only), the build-ceph stack:

1. Builds packages via src/script/build-with-container.py
2. Indexes the RPM tree (createrepo) and serves it briefly over HTTP on the build host (inside the Podman machine on macOS)
3. Runs upstream container/build.sh with CUSTOM_CEPH_REPO_URL pointing at a generated yum .repo file, then tags the result as a local image (default localhost/ceph-devstack:main)

Point the ceph stack at that image:

[containers.ceph_builder]
repo = "~/src/ceph"
image_builder = "package-build"
build_distro = "centos9"

[containers.ceph_node]
image = "localhost/ceph-devstack:main"
image_builder = "package-build"

Then run the two stacks manually:

ceph-devstack --stack build-ceph start
ceph-devstack --stack ceph start

Or use depends to combine them into a single command. Add this to your user config (~/.config/ceph-devstack/config.toml):

[stacks.ceph]
depends = ["build-ceph"]

Then ceph-devstack --stack ceph start will automatically run the build-ceph stack first, compiling Ceph and producing the runtime image before deploying the cluster. stop and remove do not cascade into dependency stacks, so build caches are preserved.

The teuthology stack includes two services that bridge locally-built artifacts into test jobs:

• package_repo — an HTTP server exposing built RPMs at http://package_repo:8080/rpm/ on the ceph-devstack network
• registry — an OCI distribution registry at registry:5000 for locally-built container images

Both are included in stacks.teuthology.services by default. When present and local_artifacts = true is set on the stack, ceph-devstack:

1. Mounts the RPM output directory from build-ceph into package_repo
2. Configures testnodes to trust registry:5000 as an insecure registry
3. Deep-merges local artifact settings into teuthology's .teuthology.yaml so that install tasks pull packages from the local repo and cephadm tasks pull images from the local registry

First, configure build-ceph to compile from your local checkout (~/.config/ceph-devstack/config.toml):

[containers.ceph_builder]
repo = "~/src/ceph"
image_builder = "package-build"
build_distro = "centos9"

[stacks.teuthology]
depends = ["build-ceph"]
local_artifacts = true

Then start the teuthology stack. The depends declaration causes build-ceph to run first, compiling packages and producing a container image:

ceph-devstack --stack teuthology start

After the build completes, package_repo immediately begins serving the RPMs. To also make the built container image available, push it into the registry:

podman tag localhost/ceph-devstack:main localhost:5000/ceph:main
podman push --tls-verify=false localhost:5000/ceph:main

Now schedule a test suite that exercises local artifacts:

export TEUTHOLOGY_SUITE=orch:cephadm:smoke-small
ceph-devstack --stack teuthology start

Check that the package repo is serving RPMs:

podman exec package_repo ls /packages/rpm/

Check that the registry has your image:

podman exec registry wget -qO- http://localhost:5000/v2/_catalog

If you prefer to run stacks manually rather than using depends:

ceph-devstack --stack build-ceph start
podman tag localhost/ceph-devstack:main localhost:5000/ceph:main
podman push --tls-verify=false localhost:5000/ceph:main
ceph-devstack --stack teuthology start

If you don't need local artifacts (e.g. testing against upstream packages), the simplest approach is to set local_artifacts = false (the default):

[stacks.teuthology]
local_artifacts = false

This prevents the .teuthology.yaml merge while keeping the services available for manual use. To also stop the containers from being created, remove package_repo and registry from your teuthology stack config:

[stacks.teuthology]
services = [
    "postgres",
    "paddles",
    "beanstalk",
    "pulpito",
    "teuthology",
    "testnode",
    "archive",
]

When these services are absent, testnodes will not receive the insecure registry config and teuthology's config will not be modified.

As an example, the following configuration will use a local image for paddles with the tag TEST; it will also create ten testnode containers; and will build its teuthology container from the git repo at ~/src/teuthology:

containers:
  paddles:
    image: localhost/paddles:TEST
  testnode:
    count: 10
  teuthology:
    repo: ~/src/teuthology

By default, pre-built container images are pulled from quay.io/ceph-infra. The images can be overridden via the config file. It's also possible to build images from on-disk git repositories.

First, you'll want to pull all the images:

ceph-devstack pull

Optional: If building any images from repos:

ceph-devstack build

Next, you can start the containers with:

ceph-devstack start

Once everything is started, a message similar to this will be logged:

View test results at http://smithi065.front.sepia.ceph.com:8081/

This link points to the running Pulpito instance. Test archives are also stored in the --data-dir (default: ~/.local/share/ceph-devstack).

To watch teuthology's output, you can:

podman logs -f teuthology

If you want testnode containers to be replaced as they are stopped and destroyed, you can:

ceph-devstack watch

When finished, this command removes all the resources that were created:

ceph-devstack remove

By default, we run the teuthology:no-ceph suite to self-test teuthology. If we wanted to test Ceph itself, we could use the orch:cephadm:smoke-small suite:

export TEUTHOLOGY_SUITE=orch:cephadm:smoke-small

It's possible to skip the automatic suite-scheduling behavior:

export TEUTHOLOGY_SUITE=none

If you need to use "real" testnodes and have access to a lab, there are a few additonal steps to take. We will use the Sepia lab as an example below:

To give the teuthology container access to your SSH private key (via podman secret):

export SSH_PRIVKEY_PATH=$HOME/.ssh/id_rsa

To lock machines from the lab:

ssh teuthology.front.sepia.ceph.com
~/teuthology/virtualenv/bin/teuthology-lock \
  --lock-many 1 \
  --machine-type smithi \
  --desc "teuthology dev testing"

Once you have your machines locked, you need to provide a list of their hostnames and their machine type:

export TEUTHOLOGY_TESTNODES="smithiXXX.front.sepia.ceph.com,smithiYYY.front.sepia.ceph.com"
export TEUTHOLOGY_MACHINE_TYPE="smithi"

1. First fork the repo if you have not done so.
2. Clone your forked repo

git clone https://github.com/<user-name>/ceph-devstack

3. Setup the remote repo as upstream (this will prevent creating additional branches)

git remote add upstream https://github.com/zmc/ceph-devstack

4. Create virtual env in the root directory of ceph-devstack & install python dependencies

python3 -m venv venv
./venv/bin/pip3 install -e .

5. Activate venv

source venv/bin/activate

6. Run doctor command to check & fix the dependencies that you need for ceph-devstack

ceph-devstack -v doctor --fix

7. Build, Create and Start the all containers in ceph-devstack

ceph-devstack -v build
ceph-devstack -v create
ceph-devstack -v start

8. Test the containers by waiting for teuthology to finish and print the logs

ceph-devstack wait teuthology
podman logs -f teuthology