server, plugin: enhance storage stats for IOPS by shwstppr · Pull Request #10034 · apache/cloudstack · GitHub

Description

Adds framework layer change to allow retrieving and storing IOPS stats for storage pools. Custom PrimaryStoreDriver can implement method - getStorageIopsStats for returning IOPS stats. Existing method getUsedIops can also be overridden by such plugins when only used IOPS is returned. For testing purpose, implementation has been added for simulator hypervisor plugin to return capacity and used IOPS for a pool. For local storage pool, implementation has been added using iostat to return currently used IOPS. StoragePoolResponse class has been updated to return IOPS values which allows showing IOPS values in UI for different storage pool related views and APIs.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

How Has This Been Tested?

With local storage (used iops returned by listStoragePools API):

(localcloud) 🐱 > list storagepools scope=HOST
{
  "count": 1,
  "storagepool": [
    {
      "clusterid": "3c21973f-de72-4196-af20-a8d13bd043a2",
      "clustername": "p1-c1",
      "created": "2024-12-03T06:31:52+0000",
      "disksizeallocated": 0,
      "disksizetotal": 20386414592,
      "disksizeused": 10423214080,
      "hasannotations": false,
      "hypervisor": "KVM",
      "id": "124992d7-8710-4c58-bafd-24efa7b79570",
      "ipaddress": "10.0.32.118",
      "name": "ol8.localdomain-local-124992d7",
      "overprovisionfactor": "2.0",
      "path": "/var/lib/libvirt/images",
      "podid": "42791654-5ace-4f1e-89e4-62bc2c3e6396",
      "podname": "Pod1",
      "provider": "DefaultPrimary",
      "scope": "HOST",
      "state": "Up",
      "storagecapabilities": {
        "VOLUME_SNAPSHOT_QUIESCEVM": "false"
      },
      "type": "Filesystem",
      "usediops": 2600140,
      "zoneid": "7fd98beb-22f0-4fb7-9740-8ec6290c7500",
      "zonename": "pr488-t11753-kvm-ol8"
    }
  ]
}

How did you try to break this feature and the system with this change?

Adds framework layer change to allow retrieving and storing IOPS stats for storage pools. Custom `PrimaryStoreDriver` can implement method - `getStorageIopsStats` for returning IOPS stats. Existing method `getUsedIops` can also be overridden by such plugins when only used IOPS is returned. For testing purpose, implementation has been added for simulator hypervisor plugin to return capacity and used IOPS for a pool. For local storage pool, implementation has been added using iostat to return currently used IOPS. StoragePoolResponse class has been updated to return IOPS values which allows showing IOPS values in UI for different storage pool related views and APIs. Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

@blueorangutan package

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 11707

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Codecov Report

Attention: Patch coverage is 62.98343% with 67 lines in your changes missing coverage. Please review.

Project coverage is 16.05%. Comparing base (9231c1c) to head (56e8660).
Report is 28 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...oud/hypervisor/kvm/storage/LibvirtStoragePool.java	20.00%	12 Missing ⚠️
...src/main/java/com/cloud/server/StatsCollector.java	50.00%	7 Missing and 4 partials ⚠️
...e/cloudstack/api/response/StoragePoolResponse.java	0.00%	6 Missing ⚠️
...cloudstack/storage/datastore/db/StoragePoolVO.java	0.00%	6 Missing ⚠️
...m/cloud/hypervisor/kvm/storage/KVMStoragePool.java	0.00%	6 Missing ⚠️
...ava/com/cloud/simulator/dao/MockVolumeDaoImpl.java	0.00%	6 Missing ⚠️
...java/com/cloud/api/query/vo/StoragePoolJoinVO.java	0.00%	6 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java	64.28%	5 Missing ⚠️
.../subsystem/api/storage/PrimaryDataStoreDriver.java	0.00%	3 Missing ⚠️
...om/cloud/agent/manager/MockStorageManagerImpl.java	0.00%	3 Missing ⚠️
... and 2 more

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20   #10034      +/-   ##
============================================
+ Coverage     16.03%   16.05%   +0.01%     
- Complexity    12815    12840      +25     
============================================
  Files          5637     5637              
  Lines        493501   493650     +149     
  Branches      59829    59838       +9     
============================================
+ Hits          79130    79231     +101     
- Misses       405595   405636      +41     
- Partials       8776     8783       +7

Flag	Coverage Δ
uitests	4.02% <ø> (-0.01%)	⬇️
unittests	16.89% <62.98%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

@blueorangutan package

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

clgtm

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11711

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@blueorangutan package

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11822

@blueorangutan test

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[SF] Trillian test result (tid-11920)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51214 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10034-t11920-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

@blueorangutan package

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11921

@blueorangutan test matrix

@shwstppr a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests

[SF] Trillian test result (tid-11996)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51526 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10034-t11996-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[SF] Trillian test result (tid-11997)
Environment: kvm-ubuntu22 (x2), Advanced Networking with Mgmt server u22
Total time taken: 54551 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10034-t11997-kvm-ubuntu22.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_oobm_multiple_mgmt_server_ownership	Failure	31.79	test_outofbandmanagement.py
test_hostha_enable_ha_when_host_disabled	Error	3.09	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	Error	302.18	test_hostha_kvm.py

[SF] Trillian test result (tid-11998)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 56601 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10034-t11998-vmware-70u3.zip
Smoke tests completed. 133 look OK, 8 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_DeployVmAntiAffinityGroup_in_project	Error	131.25	test_affinity_groups_projects.py
test_DeployVmAntiAffinityGroup	Error	45.95	test_affinity_groups.py
test_03_deploy_and_scale_kubernetes_cluster	Failure	48.20	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	Failure	0.08	test_kubernetes_clusters.py
test_01_deployVMInSharedNetwork	Error	155.31	test_network.py
test_03_nic_multiple_vmware	Error	490.75	test_nic.py
test_01_non_strict_host_anti_affinity	Failure	158.26	test_nonstrict_affinity_group.py
test_02_non_strict_host_affinity	Error	100.98	test_nonstrict_affinity_group.py
test_02_restore_vm_with_disk_offering	Error	62.13	test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_size	Error	50.88	test_restore_vm.py
ContextSuite context=TestMigrateVMStrictTags>:setup	Error	0.00	test_vm_strict_host_tags.py

[SF] Trillian test result (tid-11999)
Environment: xcpng82 (x2), Advanced Networking with Mgmt server ol9
Total time taken: 76324 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10034-t11999-xcpng82.zip
Smoke tests completed. 134 look OK, 7 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_condensed_drs_algorithm	Failure	173.93	test_cluster_drs.py
test_02_balanced_drs_algorithm	Failure	184.03	test_cluster_drs.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	Error	499.49	test_internal_lb.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	Error	7.40	test_network.py
test_01_non_strict_host_anti_affinity	Error	223.62	test_nonstrict_affinity_group.py
test_02_non_strict_host_affinity	Error	108.97	test_nonstrict_affinity_group.py
test_02_create_volume	Error	5.26	test_resource_names.py
test_05_scale_vm_dont_allow_disk_offering_change	Failure	70.52	test_scale_vm.py
test_01_volume_usage	Error	96.88	test_usage.py

Test failures are unrelated to change and look similar to what we have in the test matrix for health check PR. Re-running

@blueorangutan package

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11962

@blueorangutan test matrix

@shwstppr a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests

[SF] Trillian test result (tid-12023)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 54713 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10034-t12023-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_05_vmschedule_test_e2e	Failure	362.07	test_vm_schedule.py

[SF] Trillian test result (tid-12024)
Environment: kvm-ubuntu22 (x2), Advanced Networking with Mgmt server u22
Total time taken: 56540 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10034-t12024-kvm-ubuntu22.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_oobm_multiple_mgmt_server_ownership	Failure	31.78	test_outofbandmanagement.py

[SF] Trillian test result (tid-12025)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 63014 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10034-t12025-vmware-70u3.zip
Smoke tests completed. 138 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_deployVMInSharedNetwork	Error	157.32	test_network.py
test_02_restore_vm_with_disk_offering	Error	61.28	test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_size	Error	55.11	test_restore_vm.py
test_02_list_cpvm_vm	Failure	0.04	test_ssvm.py
test_04_cpvm_internals	Failure	0.04	test_ssvm.py
test_12_destroy_cpvm	Error	6.28	test_ssvm.py

Merging based on the manual testing and LGTMs
#10034 (review)

Test failures are unrelated and can be seen in the health check PR

* 4.20: merge errors fixed Restrict the migration of volumes attached to VMs in Starting state (#9725) server, plugin: enhance storage stats for IOPS (#10034) Introducing granular command timeouts global setting (#9659) Improve logging to include more identifiable information (#9873)

Adds framework layer change to allow retrieving and storing IOPS stats for storage pools. Custom `PrimaryStoreDriver` can implement method - `getStorageIopsStats` for returning IOPS stats. Existing method `getUsedIops` can also be overridden by such plugins when only used IOPS is returned. For testing purpose, implementation has been added for simulator hypervisor plugin to return capacity and used IOPS for a pool. For local storage pool, implementation has been added using iostat to return currently used IOPS. StoragePoolResponse class has been updated to return IOPS values which allows showing IOPS values in UI for different storage pool related views and APIs. Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Addresses: shapeblue/cloudstack-apple#490 Adds framework layer change to allow retrieving and storing IOPS stats for storage pools. Custom `PrimaryStoreDriver` can implement method - `getStorageIopsStats` for returning IOPS stats. Existing method `getUsedIops` can also be overridden by such plugins when only used IOPS is returned. For testing purpose, implementation has been added for simulator hypervisor plugin to return capacity and used IOPS for a pool. For local storage pool, implementation has been added using iostat to return currently used IOPS. StoragePoolResponse class has been updated to return IOPS values which allows showing IOPS values in UI for different storage pool related views and APIs. Schema changes: [fr83-schema-changes.sql.txt](https://github.com/user-attachments/files/18006838/fr83-schema-changes.sql.txt) Upstream PR: apache#10034 --------- Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

boring-cyborg Bot added component:api component:kvm component:management-server component:packaging component:simulator component:storage component:UI distro:ubuntu labels Dec 4, 2024

address comments

a96e02a

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr added 2 commits December 4, 2024 16:01

fix tests

5b2c4aa

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

unused import

383f48b

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

change

b9531f0

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

sureshanaparti approved these changes Dec 4, 2024

View reviewed changes

build fix

77dd307

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

github-actions Bot added the status:has-conflicts label Dec 4, 2024

yadvr added this to the 4.20.1 milestone Dec 4, 2024

Merge branch '4.20' into add-storagestats-iops-4.20

adee70d

github-actions Bot removed the status:has-conflicts label Dec 5, 2024

more tests

edb3846

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr merged commit bd488c4 into apache:4.20 Jan 7, 2025

shwstppr deleted the add-storagestats-iops-4.20 branch January 7, 2025 11:47

kiranchavala mentioned this pull request Jan 16, 2025

Should Cloudstack consider Minops and Maxiops for a vm deployment #10199

Closed

Pearl1594 added this to ACS 4.20.1 Mar 17, 2025

Pearl1594 moved this to Done in ACS 4.20.1 Mar 17, 2025

server, plugin: enhance storage stats for IOPS by shwstppr · Pull Request #10034 · apache/cloudstack · GitHub

Repository navigation

server, plugin: enhance storage stats for IOPS - #10034

server, plugin: enhance storage stats for IOPS#10034shwstppr merged 12 commits intoapache:4.20apache/cloudstack:4.20from shapeblue:add-storagestats-iops-4.20shapeblue/cloudstack:add-storagestats-iops-4.20Copy head branch name to clipboard

Conversation

shwstppr commented Dec 4, 2024

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

shwstppr commented Dec 4, 2024

Uh oh!

blueorangutan commented Dec 4, 2024

Uh oh!

blueorangutan commented Dec 4, 2024

Uh oh!

codecov Bot commented Dec 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

shwstppr commented Dec 4, 2024

Uh oh!

blueorangutan commented Dec 4, 2024

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Dec 4, 2024

Uh oh!

github-actions Bot commented Dec 4, 2024

Uh oh!

shwstppr commented Dec 5, 2024

Uh oh!

blueorangutan commented Dec 5, 2024

Uh oh!

blueorangutan commented Dec 16, 2024

Uh oh!

shwstppr commented Dec 16, 2024

Uh oh!

blueorangutan commented Dec 16, 2024

Uh oh!

blueorangutan commented Dec 17, 2024

Uh oh!

shwstppr commented Dec 27, 2024

Uh oh!

blueorangutan commented Dec 27, 2024

Uh oh!

blueorangutan commented Dec 27, 2024

Uh oh!

shwstppr commented Dec 28, 2024

Uh oh!

blueorangutan commented Dec 28, 2024

Uh oh!

blueorangutan commented Dec 28, 2024

Uh oh!

blueorangutan commented Dec 28, 2024

Uh oh!

blueorangutan commented Dec 28, 2024

Uh oh!

blueorangutan commented Dec 29, 2024

Uh oh!

shwstppr commented Dec 31, 2024

Uh oh!

blueorangutan commented Dec 31, 2024

Uh oh!

blueorangutan commented Dec 31, 2024

Uh oh!

shwstppr commented Jan 2, 2025

Uh oh!

blueorangutan commented Jan 2, 2025

Uh oh!

blueorangutan commented Jan 2, 2025

Uh oh!

blueorangutan commented Jan 2, 2025

Uh oh!

server, plugin: enhance storage stats for IOPS#10034
shwstppr merged 12 commits into
apache:4.20from
shapeblue:add-storagestats-iops-4.20

codecov Bot commented Dec 4, 2024 •
edited
Loading