Flexible tags for hosts and storage pools by JoaoJandre · Pull Request #7489 · apache/cloudstack · GitHub

Description

Currently when setting tags for a resource, a host for example, offerings with those tags will be directed towards that resource. However, offerings without any tags may also be directed towards it. Making it so even after tagging a resource, with the intention of making it exclusive to certain types of offerings, this exclusiveness might be ignored.

Furthermore, the current tag system only allows the user to inform a simple list of tags, without the possibility to create more complex rules, like checking if the offering has certain pairs of tags.

To address the described problems, we propose to create a new feature to allow users to use JS to create more complex tagging rules in the following APIs:

updateHost
createStoragePool
updateStoragePool

For this, we will use the JS interpreter introduced in #5909. We will introduce a new preset variable, called "tags", which can be used to refer to the offering's tags list that will be paired with the resource. A simple example of a rule using this preset variable is:
tags[0] == cereal && tags[1] == milk

When setting a resource with this tag rule, only offerings whose first tag is cereal and second tag is milk will be accepted, offerings without tags will not be accepted. Although, this simple rule does not care for the rest of the offering's tags, so an offering with a third tag will also be accepted, no matter what the tag is.

To allow users to keep using the current tagging system without any hassle, we propose to introduce a new parameter to the affected APIs to inform if ACS should treat them the classic way, or to try to interpret it a rule in JS.

In order to support this behavior, we need to store which way the tags should be interpreted by ACS. We propose to add a new column called is_tag_a_rule in the following tables:

host_tags
storage_pool_tags

Furthermore, those same tables will have their tag column type changed from varchar(255) to text, to allow rules that have more than 255 characters.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Unit tests were created, and some were updated. Moreover, manual tests were performed:
Regarding creating and updating storage pools (SP)

No.	Test	Result	Expected behavior? (Y/N)
1	Create zoneWide SP with flexible tag	SP successfully created	Y
2	Create clusterWide SP with flexible tag	SP successfully created	Y
3	Create SP datastore Cluster (vmware) with flexible tag	SP successfully created	Y
4	Update SP (all three types) with flexible tag	Success	Y

With the following situation in the environment:
Storage 1 with rule tags[0] == 'milk' and Storage 2 with rule tags[0] == 'cereal'

No.	Test	Result	Expected behavior? (Y/N)
1	Create volume using offering tagged milk	Volume allocated to storage 1	Y
2	Create volume using offering tagged cereal	Volume allocated to storage 2	Y
3	Create volume using untagged offering	Error trying to allocate volume	Y
3	Create volume using offering tagged banana	Error trying to allocate volume	Y

With the following situation in the environment:
Storage 1 with rule tags[0] == 'milk' and Storage 2 with tag cereal

No.	Test	Result	Expected behavior? (Y/N)
1	Create volume using offering tagged milk	Volume allocated to storage 1	Y
2	Create volume using offering tagged cereal	Volume allocated to storage 2	Y
3	Create volume using untagged offering	Volume allocated to storage 2	Y
4	With the volume of test 1, resize the volume, without changing the offering	Success	Y
5	With the volume of test 1, resize the volume, changing the offering to another with the same tag	Success	Y
6	With the volume of test 1, resize the volume, changing the offering to another one with a different tag	Exception	Y
7	With the volume of test 2, tests 4, 5 and 6 were redone	Same results	Y

Regarding Hosts:

No.	Test	Result	Expected behavior? (Y/N)
1	Removing a host and making that host rediscover mgmt	Success	Y
2	Call the updateHost API to update a host tag to a flexible tag	Changed tag	Y

Regarding deploying VMs:
With the following situation in the environment:
Host 1 with rule tags[0] == 'milk', Host 2 and Host 3 with rule tags[0] == cereal

No.	Test	Result	Expected behavior? (Y/N)
1	VM Deploy with offering tagged milk	VM allocated to host 1	Y
2	VM Deploy with offering tagged cereal	VM allocated to host 2/3	Y
3	VM Deploy with untagged offering	Exception	Y
4	Vm Deploy with offering tagged banana	Exception	Y

With the following situation in the environment:
Host 1 with rule tags[0] == 'milk', Host 2 and Host 3 with tag cereal

No.	Test	Result	Expected behavior? (Y/N)
1	VM Deploy with offering tagged milk	VM allocated to host 1	Y
2	VM Deploy with offering tagged cereal	VM allocated to host 2/3	Y
3	VM Deploy with untagged offering	VM allocated to host 2/3	Y
4	VM Deploy with offering tagged banana	Exception	Y

Regarding quota activation rules:

No.	Test	Result	Expected behavior? (Y/N)
1	Create tariff using activation rule value.host.isTagARule	rule activated correctly	Y

@JoaoJandre , I haven´t got my head around how exactly this would work yet but it seems promising.
@andrijapanicsb Can you have a look?

@JoaoJandre could a global setting and its use in the deployment planner be enough to make the deployment and related lifecycles are strictly or non-strictly allowed the by deployment planner?

Codecov Report

Attention: 135 lines in your changes are missing coverage. Please review.

Comparison is base (c7ed4ca) 18.33% compared to head (76e4aae) 29.17%.
Report is 5 commits behind head on main.

Files	Patch %	Lines
.../src/main/java/com/cloud/host/dao/HostDaoImpl.java	53.84%	20 Missing and 4 partials ⚠️
...loudstack/utils/jsinterpreter/TagAsRuleHelper.java	0.00%	16 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java	13.33%	13 Missing ⚠️
.../cloud/configuration/ConfigurationManagerImpl.java	40.00%	10 Missing and 2 partials ⚠️
...om/cloud/api/query/dao/StoragePoolJoinDaoImpl.java	78.37%	6 Missing and 2 partials ⚠️
.../agent/manager/allocator/impl/RandomAllocator.java	0.00%	7 Missing ⚠️
.../main/java/com/cloud/storage/StoragePoolTagVO.java	45.45%	6 Missing ⚠️
.../com/cloud/storage/dao/StoragePoolTagsDaoImpl.java	40.00%	6 Missing ⚠️
...che/cloudstack/metrics/PrometheusExporterImpl.java	0.00%	6 Missing ⚠️
...gent/manager/allocator/impl/FirstFitAllocator.java	20.00%	3 Missing and 1 partial ⚠️
... and 16 more

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #7489       +/-   ##
=============================================
+ Coverage     18.33%   29.17%   +10.84%     
- Complexity    17808    31127    +13319     
=============================================
  Files          5059     5195      +136     
  Lines        344296   366930    +22634     
  Branches      49577    53735     +4158     
=============================================
+ Hits          63132   107065    +43933     
+ Misses       272093   245223    -26870     
- Partials       9071    14642     +5571

Flag	Coverage Δ
simulator-marvin-tests	25.16% <41.56%> (+5.60%)	⬆️
uitests	4.45% <ø> (-0.04%)	⬇️
unit-tests	14.82% <28.32%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@JoaoJandre could a global setting and its use in the deployment planner be enough to make the deployment and related lifecycles are strictly or non-strictly allowed the by deployment planner?

This would be the simplest solution; however, it would be limited as well. A setting like that would either make all hosts accept or not accept offerings without tags (the same with storage pools). The solution proposed in this PR makes the hosts and storage tags much more flexible; for instance, you could have hosts that accept untagged offerings, while others do not. Furthermore, being able to write a rule in JS opens space for operator creativity, and they would have a much finer control/understanding over their cloud environment.

Also, the current behavior is kept; therefore, the use of tag as rules is optional.

Sounds interesting - as long as we keep the original behaviour a default (backward compatibility), I think this would be an interesting feature. It does add complexity (as with anything else), so proper documentation should be in place.

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

Sounds interesting - as long as we keep the original behaviour a default (backward compatibility), I think this would be an interesting feature. It does add complexity (as with anything else), so proper documentation should be in place.

@JoaoJandre , can you reply this comment, please? As I read this the old behaviour is intact without change. But is there any risk that a user may think they use an old fashion tag but end up creating a dynamic one?

Sounds interesting - as long as we keep the original behaviour a default (backward compatibility), I think this would be an interesting feature. It does add complexity (as with anything else), so proper documentation should be in place.

@JoaoJandre , can you reply this comment, please? As I read this the old behaviour is intact without change. But is there any risk that a user may think they use an old fashion tag but end up creating a dynamic one?

Sure, as I stated on #7489 (comment), the current behavior is kept; therefore, the use of tag as rules is optional. Users have to explicitly inform the isTagARule parameter as true when calling the APIs to make the tags be interpreted as rules.

@blueorangutan package

@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

good code , but needs good functional verification

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6180

@blueorangutan test matrix

@DaanHoogland a [SF] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

[SF] Trillian test result (tid-6627)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41078 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7489-t6627-xenserver-71.zip
Smoke tests completed. 110 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[SF] Trillian test result (tid-6628)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 41071 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7489-t6628-vmware-67u3.zip
Smoke tests completed. 85 look OK, 1 have errors, 24 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_arping_in_ssvm	Failure	60.66	test_diagnostics.py
test_09_arping_in_cpvm	Failure	60.75	test_diagnostics.py
all_test_safe_shutdown	Skipped	---	test_safe_shutdown.py
all_test_metrics_api	Skipped	---	test_metrics_api.py
all_test_outofbandmanagement	Skipped	---	test_outofbandmanagement.py
all_test_outofbandmanagement_nestedplugin	Skipped	---	test_outofbandmanagement_nestedplugin.py
all_test_routers_iptables_default_policy	Skipped	---	test_routers_iptables_default_policy.py
all_test_secondary_storage	Skipped	---	test_secondary_storage.py
all_test_service_offerings	Skipped	---	test_service_offerings.py
all_test_storage_policy	Skipped	---	test_storage_policy.py
all_test_templates	Skipped	---	test_templates.py
all_test_update_security_group	Skipped	---	test_update_security_group.py
all_test_usage_events	Skipped	---	test_usage_events.py
all_test_vm_autoscaling	Skipped	---	test_vm_autoscaling.py
all_test_vm_deployment_planner	Skipped	---	test_vm_deployment_planner.py
all_test_vm_life_cycle	Skipped	---	test_vm_life_cycle.py
all_test_vm_lifecycle_unmanage_import	Skipped	---	test_vm_lifecycle_unmanage_import.py
all_test_vm_snapshot_kvm	Skipped	---	test_vm_snapshot_kvm.py
all_test_vm_snapshots	Skipped	---	test_vm_snapshots.py
all_test_volumes	Skipped	---	test_volumes.py
all_test_vpc_ipv6	Skipped	---	test_vpc_ipv6.py
all_test_vpc_redundant	Skipped	---	test_vpc_redundant.py
all_test_vpc_router_nics	Skipped	---	test_vpc_router_nics.py
all_test_vpc_vpn	Skipped	---	test_vpc_vpn.py
all_test_host_maintenance	Skipped	---	test_host_maintenance.py
all_test_hostha_kvm	Skipped	---	test_hostha_kvm.py

[SF] Trillian test result (tid-6629)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44219 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7489-t6629-kvm-centos7.zip
Smoke tests completed. 110 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

@JoaoJandre can you solve the conflicts?

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@JoaoJandre can you solve the conflicts?

@DaanHoogland I guess the conflicts solved themselves :D

@JoaoJandre do we need to move storage_pool_view to the view file now as per #7417?

@shwstppr done

@blueorangutan package

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan test

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

from the backend:

Smoke tests completed. 117 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_migrate_vm	Error	42.83	test_vm_life_cycle.py

github api rate limit hit, but I think we are ok with this one, @shwstppr.

@blueorangutan package

@GutoVeronezi a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7860

@shwstppr @GutoVeronezi are we good with this one?

@shwstppr @GutoVeronezi are we good with this one?

LGTM, nothing more from my side.

@DaanHoogland I'm good as it already has the required approvals.

@JoaoJandre should we add some notes in the documentation to highlight the new behaviour?

@DaanHoogland I'm good as it already has the required approvals.

@JoaoJandre should we add some notes in the documentation to highlight the new behaviour?

@JoaoJandre I added an issue for documentation and assigned you (#8283 )

merging

@DaanHoogland I'm good as it already has the required approvals.
@JoaoJandre should we add some notes in the documentation to highlight the new behaviour?

@JoaoJandre I added an issue for documentation and assigned you (#8283 )

merging

@DaanHoogland Okay, I'll add it to my tasks and do it in the following days.

Co-authored-by: João Jandre <joao@scclouds.com.br>

Flexible tags for hosts and storage pools

d5fefa8

boring-cyborg Bot added component:api component:local-storage component:quota component:storage labels May 2, 2023

fix sql

4669e6f

fix storage pool listing

edb401f

vladimirpetrov mentioned this pull request May 23, 2023

Console proxy doesn't work since the beginning of May #7550

Closed

github-actions Bot added the status:has-conflicts label May 25, 2023

Merge remote-tracking branch 'origin/main' into flexible-tags

01fa877

github-actions Bot removed the status:has-conflicts label May 26, 2023

Fix zonewide storagepool listing with ruleTags

c6581f0

DaanHoogland approved these changes May 31, 2023

View reviewed changes

github-actions Bot added the status:has-conflicts label Jun 1, 2023

github-actions Bot added the status:has-conflicts label Nov 17, 2023

Merge remote-tracking branch 'origin/main' into flexible-tags

f923536

github-actions Bot removed the status:has-conflicts label Nov 17, 2023

shwstppr reviewed Nov 17, 2023

View reviewed changes

move views to their proper files

76e4aae

shwstppr closed this Nov 21, 2023

shwstppr reopened this Nov 21, 2023

DaanHoogland mentioned this pull request Nov 30, 2023

documentation for flexible tags #8283

Closed

DaanHoogland merged commit 26b01f6 into apache:main Nov 30, 2023

Pearl1594 mentioned this pull request Dec 5, 2023

API: Fix listing zones and stoage pools #8311

Closed

13 tasks

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Dec 6, 2023

Flexible tags for hosts and storage pools (apache#7489 )

a53d53a

Co-authored-by: João Jandre <joao@scclouds.com.br>

BryanMLima mentioned this pull request Feb 5, 2024

Fix allocation of VMs with multiple clusters #8611

Merged

11 tasks

weizhouapache mentioned this pull request Nov 29, 2024

Resize volume: add pool capacity disablethreshold for resize and allow volume auto migration #9761

Merged

14 tasks

Flexible tags for hosts and storage pools by JoaoJandre · Pull Request #7489 · apache/cloudstack · GitHub

Uh oh!

Flexible tags for hosts and storage pools - #7489

Flexible tags for hosts and storage pools#7489DaanHoogland merged 19 commits intoapache:mainapache/cloudstack:mainfrom scclouds:flexible-tagsscclouds/cloudstack:flexible-tagsCopy head branch name to clipboard

Conversation

JoaoJandre commented May 2, 2023

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

DaanHoogland commented May 8, 2023

Uh oh!

yadvr commented May 8, 2023

Uh oh!

codecov Bot commented May 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JoaoJandre commented May 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrijapanicsb commented May 9, 2023

Uh oh!

github-actions Bot commented May 25, 2023

Uh oh!

DaanHoogland commented May 30, 2023

Uh oh!

JoaoJandre commented May 30, 2023

Uh oh!

DaanHoogland commented May 31, 2023

Uh oh!

blueorangutan commented May 31, 2023

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented May 31, 2023

Uh oh!

DaanHoogland commented May 31, 2023

Uh oh!

blueorangutan commented May 31, 2023

Uh oh!

blueorangutan commented May 31, 2023

Uh oh!

blueorangutan commented May 31, 2023

Uh oh!

blueorangutan commented May 31, 2023

Uh oh!

DaanHoogland commented Nov 17, 2023

Uh oh!

github-actions Bot commented Nov 17, 2023

Uh oh!

JoaoJandre commented Nov 17, 2023

Uh oh!

shwstppr left a comment

Choose a reason for hiding this comment

Uh oh!

JoaoJandre commented Nov 17, 2023

Uh oh!

shwstppr commented Nov 17, 2023

Uh oh!

blueorangutan commented Nov 17, 2023

Uh oh!

DaanHoogland commented Nov 18, 2023

Uh oh!

blueorangutan commented Nov 18, 2023

Uh oh!

DaanHoogland commented Nov 21, 2023

Uh oh!

GutoVeronezi commented Nov 28, 2023

Uh oh!

blueorangutan commented Nov 28, 2023

Uh oh!

blueorangutan commented Nov 28, 2023

Uh oh!

DaanHoogland commented Nov 29, 2023

Flexible tags for hosts and storage pools#7489
DaanHoogland merged 19 commits into
apache:mainfrom
scclouds:flexible-tags

codecov Bot commented May 8, 2023 •
edited
Loading

JoaoJandre commented May 8, 2023 •
edited
Loading