Repository navigation
Incorrect Disk usage is reported when using Ceph erasure code pool #13718
Description
problem
Incorrect Disk usage is reported when a Ceph storage
versions
ACS 4.22.1
ceph version
ceph version 20.2.2 (0fcffee29411e3a38036764817b6e1afc59741cc) tentacle (stable - RelWithDebInfo)
Ubuntu 24.0.4 kvm host
The steps to reproduce the bug
CloudStack added support for ceph erasure code pool in the 4.21 release
github.com/apache/cloudstack/pull/9808
Steps to reproduce the issue
- Create a ceph-storage with Erasure code setting
ceph osd erasure-code-profile set cloudstack-ec k=2 m=1 crush-failure-domain=host ceph osd pool create cloudstack-ec 64 erasure cloudstack-ec ceph osd pool set cloudstack-ec allow_ec_overwrites true ceph osd pool application enable cloudstack-ec rbd ceph osd pool create cloudstack-ec-metadata 32 replicated ceph osd pool set cloudstack-ec-metadata size 3 rbd pool init cloudstack-ec-metadata
- Add the storage pool in cloudstack
createStoragePool scope=zone zoneid=<ZONE_ID> hypervisor=KVM \ name=MyCephECCluster provider=DefaultPrimary \ url=rbd://cloudstack:<RADOS_SECRET>@192.168.55.11/cloudstack-ec-metadata \ details[0].key=dataPool details[0].value=cloudstack-ec \ tags=RBD-EC
- Check the disk usage from ceph side
root@ceph1:/home/ubuntu# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 150 GiB 150 GiB 200 MiB 200 MiB 0.13 TOTAL 150 GiB 150 GiB 200 MiB 200 MiB 0.13 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cloudstack-ec 3 64 8 KiB 1 12 KiB 0 95 GiB cloudstack-ec-metadata 4 32 7.9 KiB 3 36 KiB 0 47 GiB
- Check the Disk used size from Cloudstack
-
Deploy a data disk of size lets say 5gb on the ceph storage pool
-
Attach it to a running vm
-
Populate some data on the data disk
dd if=/dev/urandom of=/dev/vdb bs=1M count=2048 status=progress oflag=direct
- Check the ceph disk usage
root@ceph1:/home/ubuntu# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 150 GiB 147 GiB 3.2 GiB 3.2 GiB 2.13 TOTAL 150 GiB 147 GiB 3.2 GiB 3.2 GiB 2.13 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 577 KiB 2 1.7 MiB 0 46 GiB cloudstack 2 64 6.8 KiB 3 32 KiB 0 46 GiB cloudstack-ec 3 64 2.0 GiB 513 3.0 GiB 2.11 93 GiB cloudstack-ec-metadata 4 32 9.1 KiB 6 62 KiB 0 46 GiB
- Check the cloudstack storage used
Cloudstack is reporting the used size from the replicated metadata pool and not the actual erasure code pool
What to do about it?
CloudStack should report the correct disk size used from the erasure -code pool and not the replicated metadata stats
Activity
kiranchavala commented on Jul 27, 2026
Based on the discussion in
#5500
CloudStack gets the used bytes information from libvirt.
wido commented on Jul 27, 2026
That is correct. CloudStack leverages Libvirt for this. If this is broken or wrong in Libvirt, it has to be fixed there. There isn't much CloudStack can do here.
I've set the priority to low as this is outside CloudStack's scope and assigned it to myself.
Will probably close this one soon(ish).
kiranchavala commented on Jul 27, 2026
@wido Wondering if we should consider it as an enhancement
Bypassing libvirt entirely and querying Ceph directly via restful api to get the usage stats of a erasure code pool or a ceph storage pool
curl -k -u admin: https://<>:8443/api/pool
The usage reporting in cloudstack could also get affected because of querying the data from libvirt
wido commented on Jul 27, 2026
I doubt if that's a good idea. Technically, yes, it could be done, but I wonder if we can get this to work. Ceph is implemented in the DefaultPrimartStorageDriver and this relies on Libvirt. If we want to do this cleanly we should write a separate Ceph driver for CloudStack, a plugin.
Maybe Libvirt can be improved, getting a patch into Libvirt might be better and more future proof.
DaanHoogland commented on Jul 28, 2026
this sounds like we want storage extensions now ;)
wido commented on Jul 28, 2026
Maybe @DaanHoogland ! But at this moment I don't see this being changed. Somebody has to invest time into this to get it fixed.