FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Add support to RBD erasure code pools by BryanMLima · Pull Request #9808 · apache/cloudstack · GitHub

  1. apache
  2. cloudstack

Repository navigation

Add support to RBD erasure code pools - #9808

Merged
JoaoJandre merged 6 commits into
apache:mainfrom
scclouds:add-support-to-erasure-code-rbd-pools
Apr 2, 2025
ConversationCommits (6)ChecksFiles changed
Merged

Add support to RBD erasure code pools#9808
JoaoJandre merged 6 commits into
apache:mainfrom
scclouds:add-support-to-erasure-code-rbd-pools

Conversation

BryanMLima commented Oct 16, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

This PR adds support for Ceph erasure code pools, allowing users to specify the data pool required for using this type of Ceph pool.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Screenshots (if appropriate):

Details of the data pool in the UI

How Has This Been Tested?

The following tests were executed to verify this patch; tests were executed on replicated pools as well to check if this patch would interfere in it.

  • Deploy of VM in a Ceph replicated pool
  • Deploy of VM in a Ceph erasure code pool
  • Volume resize in a Ceph replicated pool
  • Volume resize in a Ceph erasure code pool
  • Live migration of VM with volume allocated in a Ceph replicated pool
  • Live migration of VM with volume allocated in a Ceph erasure code pool
  • Attach, mount and utilization of volume in a Ceph replicated pool
  • Attach, mount and utilization of volume in a Ceph erasure code pool
  • Template creation from a volume in a Ceph replicated pool
  • Deploy of VM from the template based on volume in a Ceph replicated pool
  • Template creation from a volume in a Ceph erasure code pool
  • Deploy of VM from the template based on volume in a Ceph erasure code pool
  • Volume migration from/to Ceph replicated pool to/from NFS
  • Volume migration from/to Ceph replicated pool to/from iSCSI (SharedMountPoint)
  • Volume migration from/to Ceph erasure code pool to/from NFS
  • Volume migration from/to Ceph erasure code pool to/from iSCSI (SharedMountPoint)
  • Volume migration from/to Ceph erasure code pool to/from Ceph replicated pool
  • Snapshot of a volume from Ceph replicated pool
  • Revert snapshot of a volume from Ceph replicated pool
  • Snapshot of a volume from Ceph erasure code pool
  • Revert snapshot of a volume from Ceph erasure code pool

How did you try to break this feature and the system with this change?

I tried to migrate volumes from different type of pool types (NFS, iSCSI and RBD) to each other, verifying that the migration was successful.

codecov Bot commented Oct 16, 2024 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 6.42202% with 102 lines in your changes missing coverage. Please review.

Project coverage is 16.18%. Comparing base (81e052c) to head (fce6e14).
Report is 360 commits behind head on main.

Files with missing lines Patch % Lines
...hypervisor/kvm/storage/StoragePoolInformation.java 0.00% 35 Missing ⚠️
.../hypervisor/kvm/storage/LibvirtStorageAdaptor.java 0.00% 18 Missing ⚠️
...ain/java/com/cloud/api/query/QueryManagerImpl.java 0.00% 16 Missing ⚠️
.../hypervisor/kvm/storage/KVMStoragePoolManager.java 0.00% 10 Missing ⚠️
...e/cloudstack/api/response/StoragePoolResponse.java 0.00% 6 Missing ⚠️
...torage/datastore/provider/DefaultHostListener.java 0.00% 4 Missing ⚠️
...oud/hypervisor/kvm/storage/LibvirtStoragePool.java 0.00% 4 Missing ⚠️
.../cloud/hypervisor/kvm/storage/KVMPhysicalDisk.java 70.00% 1 Missing and 2 partials ⚠️
...ud/hypervisor/kvm/storage/KVMStorageProcessor.java 0.00% 2 Missing ⚠️
...CreatePrivateTemplateFromVolumeCommandWrapper.java 0.00% 1 Missing ⚠️
... and 3 more
Additional details and impacted files 
@@             Coverage Diff              @@
##               main    #9808      +/-   ##
============================================
- Coverage     16.18%   16.18%   -0.01%     
+ Complexity    13046    13045       -1     
============================================
  Files          5645     5646       +1     
  Lines        494795   494841      +46     
  Branches      59955    59963       +8     
============================================
+ Hits          80065    80069       +4     
- Misses       405897   405937      +40     
- Partials       8833     8835       +2
Flag Coverage Δ
uitests 4.01% <ø> (-0.01%) ⬇️
unittests 17.03% <6.42%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

BryanMLima commented Oct 16, 2024

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

BryanMLima requested review from DaanHoogland, JoaoJandre and sureshanaparti and removed request for DaanHoogland October 16, 2024 16:41

blueorangutan commented Oct 16, 2024

Copy link
Copy Markdown

@BryanMLima a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan commented Oct 16, 2024

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11365

DaanHoogland approved these changes Oct 28, 2024
View reviewed changes

DaanHoogland left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

clgtm, but needs testing

JoaoJandre approved these changes Oct 29, 2024
View reviewed changes

JoaoJandre left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM, did not test it. Only left a small nitpick

Comment thread ...ns/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java Resolved
slavkap assigned slavkap and unassigned slavkap Oct 30, 2024
slavkap self-requested a review October 30, 2024 10:35
JoaoJandre approved these changes Nov 26, 2024
View reviewed changes

JoaoJandre left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM, did not test it

github-actions Bot commented Jan 8, 2025

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

blueorangutan commented Mar 20, 2025

Copy link
Copy Markdown

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

DaanHoogland commented Mar 20, 2025

Copy link
Copy Markdown
Contributor

@blueorangutan test

blueorangutan commented Mar 20, 2025

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan commented Mar 20, 2025

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12848

DaanHoogland commented Mar 24, 2025

Copy link
Copy Markdown
Contributor

@blueorangutan test

blueorangutan commented Mar 24, 2025

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan commented Mar 25, 2025

Copy link
Copy Markdown

[SF] Trillian test result (tid-12807)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 69190 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9808-t12807-kvm-ol8.zip
Smoke tests completed. 135 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestPrivateGwACL>:setup Error 0.00 test_privategw_acl.py
test_01_create_template Error 1.15 test_templates.py
test_CreateTemplateWithDuplicateName Error 1.14 test_templates.py
test_02_create_template_with_checksum_sha1 Error 65.61 test_templates.py
test_03_create_template_with_checksum_sha256 Error 65.65 test_templates.py
test_04_create_template_with_checksum_md5 Error 65.64 test_templates.py
test_05_create_template_with_no_checksum Error 65.62 test_templates.py
ContextSuite context=TestTemplates>:setup Error 270.58 test_templates.py
ContextSuite context=TestISOUsage>:setup Error 0.00 test_usage.py
test_01_snapshot_usage Error 24.76 test_usage.py
test_06_download_detached_volume Error 314.07 test_volumes.py
test_13_migrate_volume_and_change_offering Error 124.18 test_volumes.py
ContextSuite context=TestIpv6Vpc>:setup Error 0.00 test_vpc_ipv6.py
ContextSuite context=TestVPCRedundancy>:setup Error 0.00 test_vpc_redundant.py

JoaoJandre commented Mar 31, 2025

Copy link
Copy Markdown
Contributor

@blueorangutan package

blueorangutan commented Mar 31, 2025

Copy link
Copy Markdown

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan commented Mar 31, 2025

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12934

JoaoJandre commented Apr 1, 2025

Copy link
Copy Markdown
Contributor

@DaanHoogland could we try the CI again?

DaanHoogland commented Apr 1, 2025

Copy link
Copy Markdown
Contributor

@blueorangutan test

blueorangutan commented Apr 1, 2025

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan commented Apr 2, 2025

Copy link
Copy Markdown

[SF] Trillian test result (tid-12877)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52440 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9808-t12877-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

JoaoJandre merged commit cb4848b into apache:main Apr 2, 2025
dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Jun 19, 2025
Add support to RBD erasure code pools (apache#9808)
e17f5fc 
* Readd filename string on qemuimg create

* Remove empty object on the data pool details of storage pools with no data pool

* Only use the method createPhysicalDiskByLibVirt with RBD when the pool is of erasure code type. Also added javadoc for createPhysicalDisk method

* Change literal '/' string to File.separator

* Add support for erasure code pools

* Fix null on putAll
bhouse-nexthop mentioned this pull request Jun 6, 2026
Closed
1 task done
bhouse-nexthop pushed a commit to bhouse-nexthop/cloudstack that referenced this pull request Jun 6, 2026
kvm: apply rbd_default_data_pool when creating volumes from templates…
aaa2a61 
… on RBD

RBD erasure-coded pool support (apache#9808) added handling of the
rbd_default_data_pool storage-pool detail to RBDStringBuilder (qemu-img
path) and to createPhysicalDisk (blank volumes), but not to
createDiskFromTemplateOnRBD. As a result, ROOT volumes created from a
template via rados-java rbd.clone()/rbd.create() are created without a
data pool: all of their data objects land in the (replicated) metadata
pool instead of the erasure-coded data pool, defeating the point of EC
and consuming ~3x raw space.

Set rbd_default_data_pool on the Rados connection (before connect) in
both the same-cluster clone/copy branch and the cross-cluster copy
branch of createDiskFromTemplateOnRBD, using the destination pool's
detail. librbd then honors it as the default data pool when the new
image is created, so template-derived volumes get data_pool set, the
same way blank volumes already do.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bhouse-nexthop mentioned this pull request Jun 6, 2026
Merged
1 task done
kiranchavala mentioned this pull request Jul 27, 2026
Open
bhouse-nexthop pushed a commit to bhouse-nexthop/cloudstack that referenced this pull request Sep 10, 2026
kvm: apply rbd_default_data_pool when creating volumes from templates…
23b61e2 
… on RBD

RBD erasure-coded pool support (apache#9808) added handling of the
rbd_default_data_pool storage-pool detail to RBDStringBuilder (qemu-img
path) and to createPhysicalDisk (blank volumes), but not to
createDiskFromTemplateOnRBD. As a result, ROOT volumes created from a
template via rados-java rbd.clone()/rbd.create() are created without a
data pool: all of their data objects land in the (replicated) metadata
pool instead of the erasure-coded data pool, defeating the point of EC
and consuming ~3x raw space.

Set rbd_default_data_pool on the Rados connection (before connect) in
both the same-cluster clone/copy branch and the cross-cluster copy
branch of createDiskFromTemplateOnRBD, using the destination pool's
detail. librbd then honors it as the default data pool when the new
image is created, so template-derived volumes get data_pool set, the
same way blank volumes already do.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bhouse-nexthop pushed a commit to bhouse-nexthop/cloudstack that referenced this pull request Sep 11, 2026
kvm: apply rbd_default_data_pool when creating volumes from templates…
298b5bc 
… on RBD

RBD erasure-coded pool support (apache#9808) added handling of the
rbd_default_data_pool storage-pool detail to RBDStringBuilder (qemu-img
path) and to createPhysicalDisk (blank volumes), but not to
createDiskFromTemplateOnRBD. As a result, ROOT volumes created from a
template via rados-java rbd.clone()/rbd.create() are created without a
data pool: all of their data objects land in the (replicated) metadata
pool instead of the erasure-coded data pool, defeating the point of EC
and consuming ~3x raw space.

Set rbd_default_data_pool on the Rados connection (before connect) in
both the same-cluster clone/copy branch and the cross-cluster copy
branch of createDiskFromTemplateOnRBD, using the destination pool's
detail. librbd then honors it as the default data pool when the new
image is created, so template-derived volumes get data_pool set, the
same way blank volumes already do.

Signed-off-by: Brad House <bhouse@nexthop.ai>
bhouse-nexthop pushed a commit to bhouse-nexthop/cloudstack that referenced this pull request Sep 11, 2026
kvm: apply rbd_default_data_pool when creating volumes from templates…
69c16a3 
… on RBD

RBD erasure-coded pool support (apache#9808) added handling of the
rbd_default_data_pool storage-pool detail to RBDStringBuilder (qemu-img
path) and to createPhysicalDisk (blank volumes), but not to
createDiskFromTemplateOnRBD. As a result, ROOT volumes created from a
template via rados-java rbd.clone()/rbd.create() are created without a
data pool: all of their data objects land in the (replicated) metadata
pool instead of the erasure-coded data pool, defeating the point of EC
and consuming ~3x raw space.

Set rbd_default_data_pool on the Rados connection (before connect) in
both the same-cluster clone/copy branch and the cross-cluster copy
branch of createDiskFromTemplateOnRBD, using the destination pool's
detail. librbd then honors it as the default data pool when the new
image is created, so template-derived volumes get data_pool set, the
same way blank volumes already do.

Signed-off-by: Brad House <bhouse@nexthop.ai>
bhouse-nexthop pushed a commit to bhouse-nexthop/cloudstack that referenced this pull request Sep 11, 2026
kvm: apply rbd_default_data_pool when creating volumes from templates…
3d74d5a 
… on RBD

RBD erasure-coded pool support (apache#9808) added handling of the
rbd_default_data_pool storage-pool detail to RBDStringBuilder (qemu-img
path) and to createPhysicalDisk (blank volumes), but not to
createDiskFromTemplateOnRBD. As a result, ROOT volumes created from a
template via rados-java rbd.clone()/rbd.create() are created without a
data pool: all of their data objects land in the (replicated) metadata
pool instead of the erasure-coded data pool, defeating the point of EC
and consuming ~3x raw space.

Set rbd_default_data_pool on the Rados connection (before connect) in
both the same-cluster clone/copy branch and the cross-cluster copy
branch of createDiskFromTemplateOnRBD, using the destination pool's
detail. librbd then honors it as the default data pool when the new
image is created, so template-derived volumes get data_pool set, the
same way blank volumes already do.

Signed-off-by: Brad House <bhouse@nexthop.ai>
weizhouapache pushed a commit that referenced this pull request Sep 14, 2026
kvm: apply rbd_default_data_pool when creating volumes from templates…
e38b7a7 
… on RBD (#13361)

RBD erasure-coded pool support (#9808) added handling of the
rbd_default_data_pool storage-pool detail to RBDStringBuilder (qemu-img
path) and to createPhysicalDisk (blank volumes), but not to
createDiskFromTemplateOnRBD. As a result, ROOT volumes created from a
template via rados-java rbd.clone()/rbd.create() are created without a
data pool: all of their data objects land in the (replicated) metadata
pool instead of the erasure-coded data pool, defeating the point of EC
and consuming ~3x raw space.

Set rbd_default_data_pool on the Rados connection (before connect) in
both the same-cluster clone/copy branch and the cross-cluster copy
branch of createDiskFromTemplateOnRBD, using the destination pool's
detail. librbd then honors it as the default data pool when the new
image is created, so template-derived volumes get data_pool set, the
same way blank volumes already do.
weizhouapache pushed a commit to weizhouapache/cloudstack that referenced this pull request Oct 12, 2026
kvm: apply rbd_default_data_pool when creating volumes from templates…
72b39c5 
… on RBD (apache#13361)

RBD erasure-coded pool support (apache#9808) added handling of the
rbd_default_data_pool storage-pool detail to RBDStringBuilder (qemu-img
path) and to createPhysicalDisk (blank volumes), but not to
createDiskFromTemplateOnRBD. As a result, ROOT volumes created from a
template via rados-java rbd.clone()/rbd.create() are created without a
data pool: all of their data objects land in the (replicated) metadata
pool instead of the erasure-coded data pool, defeating the point of EC
and consuming ~3x raw space.

Set rbd_default_data_pool on the Rados connection (before connect) in
both the same-cluster clone/copy branch and the cross-cluster copy
branch of createDiskFromTemplateOnRBD, using the destination pool's
detail. librbd then honors it as the default data pool when the new
image is created, so template-derived volumes get data_pool set, the
same way blank volumes already do.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

weizhouapache weizhouapache left review comments
DaanHoogland DaanHoogland approved these changes
winterhazel winterhazel approved these changes
JoaoJandre JoaoJandre approved these changes
sureshanaparti Awaiting requested review from sureshanaparti
slavkap Awaiting requested review from slavkap slavkap is a code owner

Assignees

No one assigned

Labels

component:api component:iscsi component:kvm component:primary-storage component:storage component:UI type:improvement

Projects

No open projects
Apache CloudStack 4.21.0
Status: Done

Milestone

4.21.0

Development

Successfully merging this pull request may close these issues.

8 participants

Back | FazBrowse Home | New Git URL