CKS: open the cluster public IP on VPC tiers when the VPC has the Firewall service by adestis-mf · Pull Request #14670 · apache/cloudstack · GitHub

Description

This PR fixes CKS clusters on VPC tiers whose VPC uses the Firewall service for public IPs
(added in #12706). In such a VPC the VPC virtual router drops ingress traffic to a public IP
unless a firewall rule allows it. CKS only created network ACL items, port forwarding and the
API load balancer rule on VPC tiers. The management server could not reach the nodes (SSH 2222+,
API 6443), the control node could not reach its own API through the public IP
(controlPlaneEndpoint), and cluster creation failed after the start timeout.

When the VPC virtual router provides the Firewall service of the cluster's VPC, CKS now creates
the same ingress firewall rules as on isolated networks:

create: SSH range and 6443 on the cluster public IP (setupKubernetesClusterVpcTierRules),
and the etcd SSH ports (setupKubernetesEtcdNetworkRules);
scale: the SSH firewall rule is replaced to match the new node count
(scaleKubernetesClusterVpcTierRules), as for isolated networks;
add/remove node: the per-node SSH firewall rule is created and revoked
(KubernetesClusterAddWorker, KubernetesClusterRemoveWorker).

In a VPC the firewall rules are created after the load balancing and port forwarding
rules. Applying firewall rules on a VPC public IP does not associate the IP with the VR
(IpAddressManagerImpl.applyRules skips it when the rule has no network). If the firewall rules
come first, the later LB/PF rules do not associate it either and the IP never reaches the VR
(#14669). On isolated networks the order stays as it was.

Firewall rules on VPC public IPs are stored with vpc_id and without network_id, so the
per-node lookup now goes by public IP and ports for VPC tiers
(findIngressFirewallRuleForPorts); isolated networks keep using findByNetworkIdAndPorts.
On cluster deletion the rules are revoked together with the release of the cluster public IP
(cleanupIpResources → revokeFirewallRulesForIp), so the destroy worker is unchanged.

The check (isVpcFirewallEnabled) is the one that sets vpc_firewall_enabled on the VR:
VpcServiceMapDao.canProviderSupportServiceInVpc(vpcId, Firewall, VPCVirtualRouter). VPCs
without the Firewall service, isolated networks and the NSX/Netris paths behave as before.

FirewallRulesDao is now injected once in KubernetesClusterActionWorker instead of separately
in three subclasses, so that the shared lookup can use it.

This targets main: the Firewall service for VPC public IPs is not in 4.22, and there is no
4.23 branch. The commit also applies cleanly on top of the 4.23.0.0 tag.

Fixes: #14668

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Unit tests (new): KubernetesClusterActionWorkerTest covers isVpcFirewallEnabled,
isPublicIpFirewallRequired and findIngressFirewallRuleForPorts for isolated networks and VPC
tiers with and without the Firewall service; KubernetesClusterResourceModifierActionWorkerTest
checks that setupKubernetesClusterVpcTierRules creates the firewall rules only when the VPC
firewall is enabled, and only after the LB and SSH port forwarding rules.

mvn -pl plugins/integrations/kubernetes-service -am test -Dtest='**/kubernetes/**/*Test' \
    -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false

Result on JDK 11 / Maven 3.10: BUILD SUCCESS, 141 tests in the kubernetes-service module,
0 failures, 0 errors; checkstyle 0 violations.

Live, on 4.23.0.0 (KVM) without the patch, adding by hand the firewall rules the patch creates
(same VPC with the Firewall service, same tier, v1.35.4, 1 control + 1 worker):

Run	Firewall rules	Result
1	none, then added after 30 min	Error after 3600 s: first kubeadm init had failed, control node never recovers
2	within 7 s of the IP, before LB/PF	Error: IP never configured on the VR (#14669)
3	about 10 s after CKS created LB and PF	Running in under 3 min; scale-up to 2 workers Running

Deleting each cluster released the public IP and left no firewall rule behind. The patched
build itself was not deployed to a live management server.

How did you try to break this feature and the system with this change?

Isolated networks: the same calls in the same order as before; the lookup still uses
findByNetworkIdAndPorts.
VPC without the Firewall service: no firewall call is made, only ACL, PF and LB as before.
Order in a VPC: firewall rules before LB/PF break the IP association (run 2 above), so the
patch creates them afterwards; a unit test pins the order.
A missing firewall rule during node revert/removal is now skipped with a warning instead of a
NullPointerException.
Out of scope: the source CIDR of the rules stays 0.0.0.0/0 as on isolated networks
(CKS: make the source CIDR of the auto-provisioned node SSH firewall rules configurable #13970, CKS: add an editable per-cluster management-access CIDR policy #14212); removeSshFirewallRule still matches by port, as before. On scale the VPC
path goes through the same removeSshFirewallRule, so the fix for clusters with separate etcd
nodes in kubernetes: prefer the node SSH rule when scaling a cluster with etcd nodes #14165 (CKS: scale-down with separate etcd revokes wrong SSH rules and fails (4.22.1.1) #14159) applies to VPC tiers as well once it is merged forward.

…ewall service Since apache#12706 a VPC offering can provide the Firewall service for public IPs. The VPC virtual router then drops ingress traffic to a public IP that has no firewall rule. CKS only created network ACL items, port forwarding and the API load balancer rule on VPC tiers, so the management server could not reach the nodes (SSH 2222+, API 6443), the control node could not reach its own API through the public IP, and cluster creation failed after the start timeout. When the VPC virtual router provides the Firewall service of the cluster's VPC (the check that sets vpc_firewall_enabled on the VR), CKS now creates the same ingress firewall rules as on isolated networks on create (including etcd nodes), scale and node add/remove. In a VPC the firewall rules are created after the load balancing and port forwarding rules: applying firewall rules on a VPC public IP does not associate the IP with the VR, and if they come first the IP is never associated. Firewall rules on VPC public IPs have no network_id, so per-node rules are looked up by public IP and ports. On delete the rules go with the release of the cluster public IP. Isolated networks, VPCs without the Firewall service and the NSX/Netris paths are unchanged.

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
Be sure to read the CloudStack Coding Conventions.
Apache CloudStack is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@cloudstack.apache.org (https://cloudstack.apache.org/mailing-lists.html)
Slack: https://apachecloudstack.slack.com/

🟡 Changes recommended

Scaling a firewall-enabled VPC cluster after importing a worker can fail because unreconciled per-node SSH rules conflict with the replacement port range.

1 open issue

Imported worker SSH rule conflicts with VPC scale-up · New

What changed in this PR

This PR fixes CKS connectivity on VPC tiers whose virtual router provides the Firewall service, extending public-IP firewall handling across cluster lifecycle operations.

Changes:

Opens API and node SSH ports after load-balancing and port-forwarding rules establish VPC IP association.
Updates scaling and node add/remove paths, with shared firewall lookup by public IP for VPC tiers.
Adds tests for firewall-service detection, rule lookup, and provisioning order.

File	Description
plugins/integrations/kubernetes-service/src/test/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorkerTest.java	Tests conditional VPC firewall creation and rule ordering.
plugins/integrations/kubernetes-service/src/test/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorkerTest.java	Tests firewall detection and network/public-IP lookups.
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterStartWorker.java	Opens etcd SSH ports after VPC port forwarding.
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java	Rebuilds VPC SSH firewall rules during scaling.
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorker.java	Adds API and SSH firewall rules during VPC setup.
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterRemoveWorker.java	Revokes per-node VPC firewall rules with missing-rule handling.
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterAddWorker.java	Creates and rolls back per-node VPC firewall rules.
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java	Centralizes firewall DAO injection, detection, and lookup helpers.

Review effort: Balanced

Thanks, the scenario is valid: the imported node's single-port SSH rule overlaps the rebuilt
range, and detectRulesConflict rejects it (same protocol and source CIDR).

Reading the code, isolated networks on main behave the same way: the add-node path creates the
same per-node rule
(KubernetesClusterAddWorker.java#L229),
and the isolated scale path also revokes a single rule before recreating the range
(KubernetesClusterScaleWorker.java#L135-L149).
This PR mirrors that handling for VPC tiers, so the case is not new, it just becomes reachable on
VPC tiers with the Firewall service.

I'd prefer to keep this PR to the missing rules and fix the import-then-scale case for both
network types in #14673, since it also touches removeSshFirewallRule, which
#14165 is changing. If you'd rather have it here, I'll add it to this PR.

@adestis-mf , you opened three issues;

I'd prefer to keep this PR to the missing rules and fix the import-then-scale case for both
network types in #14673, since it also touches removeSshFirewallRule, which
#14165 is changing. If you'd rather have it here, I'll add it to this PR.

are you sugesting to solve all three in this PR?

No, this PR only fixes #14668: the missing firewall rules for CKS clusters on VPC tiers when the VPC
has the Firewall service. The other two are separate:

VPC firewall: a firewall rule on a new VPC public IP before any PF/LB rule prevents the IP from being configured on the VR #14669 is not specific to CKS. It turned out to be the same problem as CreatePortForwardingRuleCmd on a VPC tier doesn't trigger ipassoc ( firewall and conserve mode on in vpc offering) #14146 (once a public IP of
a VPC tier has a firewall rule, further public IPs of the tier are no longer added to the router)
and is fixed in VPC: associate tier public IPs when a public IP of the tier has firewall rules #14681, which only changes VpcVirtualRouterElement.
CKS: scaling up a cluster with imported nodes fails on the SSH firewall rules #14673 (scaling up a cluster with imported nodes) affects isolated networks and VPC tiers
alike and touches removeSshFirewallRule, which kubernetes: prefer the node SSH rule when scaling a cluster with etcd nodes #14165 changes, so I would keep it for a separate
PR. If you prefer to have it here, I can add it to this PR.

Thanks for adding #14669 and #14673 to the 24.0 milestone.

An update on the import-then-scale case: #14673 turned out to be a duplicate of #12633, and a fix is proposed in #14683. It targets 4.22, where the bug exists for isolated networks, and it leaves removeSshFirewallRule unchanged, so it does not conflict with #14165.

The VPC tier path of this PR is not in 4.22. Once #14683 is merged forward to main, I'll call the same method (removeExternalNodeSshRules) in scaleKubernetesClusterVpcTierRules here, so that VPC tiers with the Firewall service are covered as well.

A note on a dependency, found while looking into #14146 (root cause and reproduction in my comment
there; #14669 is the same problem): once a public IP of a VPC tier has a firewall rule, as the
cluster IP has with this change, further public IPs of the same tier are not added to the router
when they get port forwarding or load balancing rules. This applies to any public IP, so it would
also apply to a second cluster in the same tier or to a LoadBalancer service of the CloudStack
Kubernetes Provider on its own IP; I have not tested it with CKS. It is independent of this PR and
fixed in #14681. The order in this PR (firewall rules after the load balancing and port forwarding
rules) keeps the first cluster of a tier working on versions without that fix.

@blueorangutan package

@RosiKyu a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 20005

Copilot AI balanced review requested due to automatic review settings November 10, 2026 22:46

boring-cyborg Bot added the component:kubernetes label Nov 10, 2026

Copilot started reviewing on behalf of adestis-mf November 10, 2026 22:46

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

adestis-mf mentioned this pull request Nov 10, 2026

CKS: scaling up a cluster with imported nodes fails on the SSH firewall rules #14673

Closed

nvazquez added this to the 24.0 milestone Nov 11, 2026

nvazquez added type:bug component:cks labels Nov 11, 2026

nvazquez requested review from DaanHoogland, Damans227, Pearl1594, RosiKyu and kiranchavala November 11, 2026 00:51

RosiKyu requested a review from nvazquez November 11, 2026 11:56

This was referenced Nov 11, 2026

LoadBalancer services in a VPC with the Firewall service get no firewall rule apache/cloudstack-kubernetes-provider#114

Open

VPC: associate tier public IPs when a public IP of the tier has firewall rules #14681

Open

adestis-mf mentioned this pull request Nov 12, 2026

CKS: fix scaling a cluster with external nodes #14683

Open

2 of 14 tasks

CKS: open the cluster public IP on VPC tiers when the VPC has the Firewall service by adestis-mf · Pull Request #14670 · apache/cloudstack · GitHub

Repository navigation

CKS: open the cluster public IP on VPC tiers when the VPC has the Firewall service - #14670

CKS: open the cluster public IP on VPC tiers when the VPC has the Firewall service#14670
adestis-mf wants to merge 1 commit into
apache:mainfrom
adestis-mf:cks-vpc-firewall-rules

adestis-mf commented Nov 10, 2026

Uh oh!

boring-cyborg Bot commented Nov 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

adestis-mf Nov 10, 2026

Uh oh!

DaanHoogland Nov 11, 2026

Uh oh!

adestis-mf Nov 11, 2026

Uh oh!

adestis-mf Nov 12, 2026

Uh oh!

adestis-mf commented Nov 11, 2026

Uh oh!

RosiKyu commented Nov 11, 2026

Uh oh!

blueorangutan commented Nov 11, 2026

Uh oh!

blueorangutan commented Nov 11, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

                       }
+                      if (vpcFirewallEnabled) {
+                          // After the port forwarding rules, as when the cluster is created
+                          addFirewallRulesForNodes(publicIp, clusterVMIds.size());

CKS: open the cluster public IP on VPC tiers when the VPC has the Firewall service by adestis-mf · Pull Request #14670 · apache/cloudstack · GitHub

Repository navigation

CKS: open the cluster public IP on VPC tiers when the VPC has the Firewall service - #14670

CKS: open the cluster public IP on VPC tiers when the VPC has the Firewall service#14670adestis-mf wants to merge 1 commit intoapache:mainapache/cloudstack:mainfrom adestis-mf:cks-vpc-firewall-rulesadestis-mf/cloudstack:cks-vpc-firewall-rulesCopy head branch name to clipboard

Conversation

adestis-mf commented Nov 10, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

boring-cyborg Bot commented Nov 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

adestis-mf Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

DaanHoogland Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

adestis-mf Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

adestis-mf Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

adestis-mf commented Nov 11, 2026

Uh oh!

RosiKyu commented Nov 11, 2026

Uh oh!

blueorangutan commented Nov 11, 2026

Uh oh!

blueorangutan commented Nov 11, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

CKS: open the cluster public IP on VPC tiers when the VPC has the Firewall service#14670
adestis-mf wants to merge 1 commit into
apache:mainfrom
adestis-mf:cks-vpc-firewall-rules