kvm: apply rbd_default_data_pool when creating volumes from templates on RBD by bhouse-nexthop · Pull Request #13361 · apache/cloudstack · GitHub

Description

RBD erasure-coded (EC) pool support (#9808) taught the KVM agent to honor the rbd_default_data_pool storage-pool detail, but only in two places:

KVMPhysicalDisk.RBDStringBuilder() — the qemu-img URI builder
createPhysicalDisk() — blank volume creation (routes RBD creates to the QemuImg path when a data pool is set)

It was not applied to createDiskFromTemplateOnRBD(), which creates ROOT volumes from a template using rados-java (rbd.clone() / rbd.create()) directly. Those calls build a Rados connection that sets mon_host, key and client_mount_timeout but never rbd_default_data_pool, so the resulting image is created without a data pool.

Net effect: every volume cloned from a template onto an EC-backed primary storage has all of its data objects written to the replicated metadata pool instead of the erasure-coded data pool. This silently defeats EC and consumes ~3x raw space for those volumes. Blank data disks on the same pool are correct, which makes the inconsistency easy to miss.

This was observed in production: of the VMs on an EC primary storage, the template-derived ROOT volumes had no data_pool (rbd info shows no data_pool: line and lacks the data-pool feature), while blank DATADISKs and the template base images themselves were correct.

Fix

In createDiskFromTemplateOnRBD, read the destination pool's rbd_default_data_pool detail once and, when present, confSet it on the Rados connection before connect() — in both the same-cluster clone/copy branch and the cross-cluster copy branch. librbd then uses it as the default data pool when the new image is created, so template-derived volumes get data_pool set, exactly like blank volumes already do. This mirrors how RBDStringBuilder injects the same key for the qemu-img path.

String dataPool = (destDetails == null) ? null : destDetails.get(KVMPhysicalDisk.RBD_DEFAULT_DATA_POOL);
...
if (dataPool != null) {
    r.confSet(KVMPhysicalDisk.RBD_DEFAULT_DATA_POOL, dataPool);
}
r.connect();

No behavior change for non-EC pools (dataPool == null → no-op).

Types of changes

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Root cause reproduced live: deploying a VM from a template onto an EC primary storage produced a ROOT clone with no data_pool (parent base snapshot in the metadata pool); the data-pool feature was absent.
Confirmed the working paths (template seed via qemu-img, blank DATADISK create) already set the data pool, isolating the gap to the rados-java clone/create path.
Code-path traced to Add support to RBD erasure code pools #9808, which did not touch createDiskFromTemplateOnRBD.

… on RBD RBD erasure-coded pool support (apache#9808) added handling of the rbd_default_data_pool storage-pool detail to RBDStringBuilder (qemu-img path) and to createPhysicalDisk (blank volumes), but not to createDiskFromTemplateOnRBD. As a result, ROOT volumes created from a template via rados-java rbd.clone()/rbd.create() are created without a data pool: all of their data objects land in the (replicated) metadata pool instead of the erasure-coded data pool, defeating the point of EC and consuming ~3x raw space. Set rbd_default_data_pool on the Rados connection (before connect) in both the same-cluster clone/copy branch and the cross-cluster copy branch of createDiskFromTemplateOnRBD, using the destination pool's detail. librbd then honors it as the default data pool when the new image is created, so template-derived volumes get data_pool set, the same way blank volumes already do. Signed-off-by: Brad House <bhouse@nexthop.ai>

@blueorangutan package

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Codecov Report

❌ Patch coverage is 0% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.94%. Comparing base (10d3e11) to head (c664fb9).
⚠️ Report is 19 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
.../hypervisor/kvm/storage/LibvirtStorageAdaptor.java	0.00%	8 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #13361      +/-   ##
============================================
+ Coverage     17.91%   17.94%   +0.02%     
- Complexity    16109    16146      +37     
============================================
  Files          5928     5928              
  Lines        534964   535182     +218     
  Branches      65463    65497      +34     
============================================
+ Hits          95865    96019     +154     
- Misses       428191   428234      +43     
- Partials      10908    10929      +21

Flag	Coverage Δ
uitests	4.02% <ø> (-0.01%)	⬇️
unittests	19.01% <0.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18262

@blueorangutan test

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[SF] Trillian test result (tid-16344)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 50505 seconds
Marvin logs: [archive removed]
Smoke tests completed. 149 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

clgtm

@vladimirpetrov @sureshanaparti please make sure this gets into 4.22.2

Signed-off-by: Brad House <bhouse@nexthop.ai>

@blueorangutan package

code lgtm, I will apply the change, and run some tests to make sure no regression

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19193

I didn't see any regressions in my testing. That said, I don't use RBD erasure code pools, so my testing doesn't cover that case.

@bhouse-nexthop Have you tested this? If so, could you share the "rbd info" for the root and data volumes?

@BryanMLima Would you mind reviewing and testing this? You added support for RBD erasure code pools in #9808.

Sure.

I just reverted to the official 4.22.1.1 release in my lab, and created an instance on the EC pool:

root@node1:~# rbd info vm-images-ec-meta/54c2ce78-822f-401f-91fb-653a436e560d
rbd image '54c2ce78-822f-401f-91fb-653a436e560d':
	size 20 GiB in 5120 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 15009df80d35f7
	block_name_prefix: rbd_data.15009df80d35f7
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	op_features:
	flags:
	create_timestamp: Fri Sep 11 11:55:44 2026
	access_timestamp: Fri Sep 11 12:04:08 2026
	modify_timestamp: Fri Sep 11 12:07:33 2026
	parent: vm-images-ec-meta/ba450ca9-8259-4351-90f3-234932d860f2@cloudstack-base-snap
	overlap: 3.5 GiB

See no data_pool entry.

I'll follow up after I re-upgrade and show a new instance there.

And with fix deployed:

root@node1:~# rbd info vm-images-ec-meta/9c5c2996-666a-4814-848d-05ac5f26f722
rbd image '9c5c2996-666a-4814-848d-05ac5f26f722':
	size 20 GiB in 5120 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 1558c5c328f8fa
	data_pool: vm-images-ec
	block_name_prefix: rbd_data.17.1558c5c328f8fa
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
	op_features:
	flags:
	create_timestamp: Fri Sep 11 12:31:04 2026
	access_timestamp: Fri Sep 11 12:31:04 2026
	modify_timestamp: Fri Sep 11 12:31:04 2026
	parent: vm-images-ec-meta/ba450ca9-8259-4351-90f3-234932d860f2@cloudstack-base-snap
	overlap: 3.5 GiB

Note the data_pool entry

And with fix deployed:

root@node1:~# rbd info vm-images-ec-meta/9c5c2996-666a-4814-848d-05ac5f26f722
rbd image '9c5c2996-666a-4814-848d-05ac5f26f722':
	size 20 GiB in 5120 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 1558c5c328f8fa
	data_pool: vm-images-ec
	block_name_prefix: rbd_data.17.1558c5c328f8fa
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
	op_features:
	flags:
	create_timestamp: Fri Sep 11 12:31:04 2026
	access_timestamp: Fri Sep 11 12:31:04 2026
	modify_timestamp: Fri Sep 11 12:31:04 2026
	parent: vm-images-ec-meta/ba450ca9-8259-4351-90f3-234932d860f2@cloudstack-base-snap
	overlap: 3.5 GiB

Note the data_pool entry

cool, I will verify too

And with fix deployed:

root@node1:~# rbd info vm-images-ec-meta/9c5c2996-666a-4814-848d-05ac5f26f722
rbd image '9c5c2996-666a-4814-848d-05ac5f26f722':
	size 20 GiB in 5120 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 1558c5c328f8fa
	data_pool: vm-images-ec
	block_name_prefix: rbd_data.17.1558c5c328f8fa
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
	op_features:
	flags:
	create_timestamp: Fri Sep 11 12:31:04 2026
	access_timestamp: Fri Sep 11 12:31:04 2026
	modify_timestamp: Fri Sep 11 12:31:04 2026
	parent: vm-images-ec-meta/ba450ca9-8259-4351-90f3-234932d860f2@cloudstack-base-snap
	overlap: 3.5 GiB

Note the data_pool entry

cool, I will verify too

tested ok

# rbd info ecpool-metadata/70ad5892-d489-42bf-8215-56dabd36eda8
...
data_pool: ecpool
...
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
...

merging on approvals and manual/smoke test

… on RBD (apache#13361) RBD erasure-coded pool support (apache#9808) added handling of the rbd_default_data_pool storage-pool detail to RBDStringBuilder (qemu-img path) and to createPhysicalDisk (blank volumes), but not to createDiskFromTemplateOnRBD. As a result, ROOT volumes created from a template via rados-java rbd.clone()/rbd.create() are created without a data pool: all of their data objects land in the (replicated) metadata pool instead of the erasure-coded data pool, defeating the point of EC and consuming ~3x raw space. Set rbd_default_data_pool on the Rados connection (before connect) in both the same-cluster clone/copy branch and the cross-cluster copy branch of createDiskFromTemplateOnRBD, using the destination pool's detail. librbd then honors it as the default data pool when the new image is created, so template-derived volumes get data_pool set, the same way blank volumes already do.

* origin/main: (156 commits) Upgrade Netris SDK version to 1.1.0 since 24.0 (apache#14365) feature: instance boot group and readiness rules (apache#13907) dont show unathorized notification for prelogin (apache#14054) .github: Upgrade docker/login-action to fix ASF Allowlist failure (apache#14276) virtual router: Add route-maps to BGP peers for Routed Mode (apache#9964) Add pre-commit hook to validate dependabot.yml (apache#14107) Add the ASF Allowlist check workflow (apache#14113) Revert "VMware to KVM: cold (VDDK) and warm (CBT) migration into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes and Linstor root/data volumes (apache#13656)" VMware to KVM: cold (VDDK) and warm (CBT) migration into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes and Linstor root/data volumes (apache#13656) engine-schema: upgrade path for 24.0.0 (apache#14033) xenserver/xcpng: do not bypass secondary storage when copy volumes between pools (apache#13750) backport: Incremental NAS backup support for KVM (apache#13074) (apache#13877) storage: KVM - enable RBD/Ceph volume encryption support (apache#13556) backup: remove the powered-off precondition for all backup providers (apache#14094) Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper (apache#14006) kvm: apply rbd_default_data_pool when creating volumes from templates on RBD (apache#13361) server: copy template details to vm instance details when import VM (apache#12793) marvin: use pycryptodome instead of PyCrypt (apache#13594) Packaging: accept normalized Marvin archive names (apache#13793) SharedFS: fix NPE when accessing systemvm templates for unsupported hypervisors (apache#13930) ...

Hi @weizhouapache !
Thank you for this fix!
I faced this problem with 4.22.1.0 and I backported your patch to this version.
Its working now, but its possible to inject the data_pool on already created volumes? I have several disks in production, and it would be nice apply this change on them.

Hi @weizhouapache ! Thank you for this fix! I faced this problem with 4.22.1.0 and I backported your patch to this version. Its working now, but its possible to inject the data_pool on already created volumes? I have several disks in production, and it would be nice apply this change on them.

This only affects creation of rbd images as the data pool is stored on those. you've got to do a migration using normal ceph utilities for the data pool unfortunately. We had to do that ourselves for a few hundred images, which does require a restart and a few minute outage per vm :/

boring-cyborg Bot added the component:kvm label Jun 6, 2026

github-actions Bot mentioned this pull request Jun 6, 2026

[repo-status] Daily Status Report – June 6, 2026 #13364

Closed

DaanHoogland approved these changes Jun 18, 2026

View reviewed changes

DaanHoogland added the status:needs-review label Jun 18, 2026

weizhouapache added this to the 4.22.2 milestone Jun 29, 2026

DaanHoogland requested review from sureshanaparti and vladimirpetrov August 10, 2026 11:39

DaanHoogland added this to CloudStack Testing Aug 10, 2026

github-project-automation Bot moved this to Backlog in CloudStack Testing Aug 10, 2026

DaanHoogland moved this from Backlog to Ready in CloudStack Testing Sep 1, 2026

DaanHoogland added status:needs-testing and removed status:needs-review labels Sep 1, 2026

DaanHoogland assigned nvazquez Sep 1, 2026

nvazquez assigned weizhouapache and unassigned nvazquez Sep 7, 2026

weizhouapache added the component:ceph label Sep 8, 2026

Merge branch '4.22' into fix/kvm-rbd-ec-datapool-template-clone

c664fb9

Signed-off-by: Brad House <bhouse@nexthop.ai>

weizhouapache reviewed Sep 10, 2026

View reviewed changes

bhouse-nexthop force-pushed the fix/kvm-rbd-ec-datapool-template-clone branch from 5283d5a to c664fb9 Compare September 11, 2026 01:26

weizhouapache assigned BryanMLima Sep 11, 2026

weizhouapache approved these changes Sep 14, 2026

View reviewed changes

weizhouapache merged commit e38b7a7 into apache:4.22 Sep 14, 2026
25 of 26 checks passed

github-project-automation Bot moved this from Ready to Done in CloudStack Testing Sep 14, 2026

winterhazel removed the status:needs-testing label Sep 23, 2026

kvm: apply rbd_default_data_pool when creating volumes from templates on RBD by bhouse-nexthop · Pull Request #13361 · apache/cloudstack · GitHub

Repository navigation

kvm: apply rbd_default_data_pool when creating volumes from templates on RBD - #13361

Conversation

bhouse-nexthop commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Fix

Types of changes

How Has This Been Tested?

Uh oh!

DaanHoogland commented Jun 15, 2026

Uh oh!

blueorangutan commented Jun 15, 2026

Uh oh!

codecov Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Jun 15, 2026

Uh oh!

DaanHoogland commented Jun 17, 2026

Uh oh!

blueorangutan commented Jun 17, 2026

Uh oh!

blueorangutan commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

bhouse-nexthop commented Aug 6, 2026

Uh oh!

weizhouapache commented Sep 10, 2026

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Sep 10, 2026

Uh oh!

blueorangutan commented Sep 10, 2026

Uh oh!

weizhouapache commented Sep 11, 2026

Uh oh!

bhouse-nexthop commented Sep 11, 2026

Uh oh!

bhouse-nexthop commented Sep 11, 2026

Uh oh!

weizhouapache commented Sep 12, 2026

Uh oh!

weizhouapache commented Sep 14, 2026

Uh oh!

weizhouapache commented Sep 14, 2026

Uh oh!

Uh oh!

EduFrazao commented Oct 14, 2026

Uh oh!

bhouse-nexthop commented Oct 14, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

bhouse-nexthop commented Jun 6, 2026 •
edited
Loading

codecov Bot commented Jun 15, 2026 •
edited
Loading

blueorangutan commented Jun 18, 2026 •
edited
Loading