[New feature] Load balancer customization (haproxy-based) by weizhouapache · Pull Request #4141 · apache/cloudstack · GitHub

Description

As discussed in mailing list, create this PR for haproxy config customization.

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/VR+haproxy+customization+in+CloudStack

Fixes #3789

It supports (or will support)

Basic haproxy configurations:
haproxy statistics uri, auth, enable/disable
global maxconn and maxpipes
Basic configurations on rule
timeout connection, client ,server globally and per rule
http, httpalive per rule
maxconn, fullconn per rule
maxconn, minconn, maxqueue per server in rule
advanced features
transparent load balancer
SSL offloading
http2 support
Variable SSL configurations

TODO:

UI change on primate
unit tests

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

@andrijapanicsb @svenvogel @rhtyd @DaanHoogland @onitake
This PR is ready for review and testing !

Any suggestions and comments are welcome !

@weizhouapache Can you provide test packages or should we build the PR ourselves?
I haven't done a custom CloudStack build in a while, I'm not sure if I even have the necessary build/packaging environment any more...

@weizhouapache Can you provide test packages or should we build the PR ourselves?
I haven't done a custom CloudStack build in a while, I'm not sure if I even have the necessary build/packaging environment any more...

@onitake we can provide the packages. could you please tell me what os version you use ?

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos7 ✔debian. JID-1606

@weizhouapache will this require a new systemvmtemplate?

@weizhouapache will this require a new systemvmtemplate?

@rhtyd it requires haproxy 1.8+ for http2 support. As I remember haproxy 1.8 is installed in debian10 systemvm template. No other changes is needed in systemvm template

@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-2382)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50655 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4141-t2382-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_loadbalance.py
Intermittent failure detected: /marvin/tests/smoke/test_network.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 80 look OK, 3 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_create_lb_rule_src_nat	Error	0.10	test_loadbalance.py
test_02_create_lb_rule_non_nat	Error	0.04	test_loadbalance.py
test_assign_and_removal_lb	Error	0.06	test_loadbalance.py
test_delete_account	Error	79.75	test_network.py
test_reboot_router	Error	179.46	test_network.py
test_releaseIP	Error	60.47	test_network.py
test_network_rules_acquired_public_ip_3_Load_Balancer_Rule	Error	3.23	test_network.py
test_01_lb_usage	Error	0.05	test_usage.py

@weizhouapache can you look at failures? Thanks

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@weizhouapache can you look at failures? Thanks

@rhtyd I will do

@weizhouapache can you look at failures? Thanks

@rhtyd pushed a commit to fix it. can you re-kick off the tests ?

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos7 ✔debian. JID-1752

It's a nice feature, is it possible to have it in 4.15 ?

@weizhouapache a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 419

We could use org.apache.commons.lang3.ArrayUtils here:

...
if (ArrayUtils.isEmpty(lbConfigs)) {
...

@GutoVeronezi org.apache.commons.lang3.ArrayUtils package don't have isEmpty method which accepts List object. I have to override it like:

...
ArrayUtils.isEmpty(new List[]{lbConfigs})
...

It imposes unnecessary conversion. What do you think?

We could use org.apache.commons.lang3.BooleanUtils here:

...
return BooleanUtils.toBoolean(forced);
...

We could implement a ternary here.

We could use org.apache.commons.lang3.BooleanUtils here:

...
return BooleanUtils.toBoolean(forced);
...

As config is not referenced outside the try block, we could declare it inside.

This code seems repeated, but changing parameters; We could extract it to a method.

This code seems repeated, but changing parameters; We could extract it to a method or create an addAndNotNull to SearchCriteria.

We could use org.apache.commons.lang3.ObjectUtils here:

...
if (ObjectUtils.anyNotNull(id, networkId...
...

We could use org.apache.commons.lang3.ObjectUtils here:

...
ObjectUtils.anyNotNull(...
...

This code seems repeated, but changing parameters; We could extract it to a method.

There is a lot of code addition, but none unit testing or javadoc. IMHO we should add they.

@ravening @soreana
could you have a look at @GutoVeronezi 's remarks ?

@weizhouapache Sure! I asked Neven to make a room for it.

…he-main

@soreana can you resolve that confict, please?

…nezi.

ping @soreana can you please fix the conflicting files.

Ping @weizhouapache I think some more work may be needed on this as now Debian 11 has moved to haproxy 2.x, should this be targeted for next milestone. I worry it may not be stable within two weeks, ie current tentative RC date.

Ping @weizhouapache I think some more work may be needed on this as now Debian 11 has moved to haproxy 2.x, should this be targeted for next milestone. I worry it may not be stable within two weeks, ie current tentative RC date.

@rhtyd it is ok to me.
@ravening @soreana what do you think ?

Ping @weizhouapache I think some more work may be needed on this as now Debian 11 has moved to haproxy 2.x, should this be targeted for next milestone. I worry it may not be stable within two weeks, ie current tentative RC date.

@rhtyd it is ok to me.
@ravening @soreana what do you think ?

@rhtyd @weizhouapache yes lets move it out of this milestone as this will not make it in time

@rhtyd @weizhouapache Sure let's do this.

@weizhouapache @rhtyd I have created a new pr for this in #5799

@weizhouapache @rhtyd I have created a new pr for this in #5799

@ravening
great. closing this ticket then.

Testing: ``` (local) mgt01 > replace loadbalancerconfigs networkid=bfdf8e5a-86bc-4dcf-9279-10482ffcb8ed scope=Network config[0].global.maxconn=5002 config[0].global.maxpipes=5003 { "accountid": "e3a80e2d-0977-11eb-8a7f-069b900033a6", "cmd": "org.apache.cloudstack.api.command.user.loadbalancer.ReplaceLoadBalancerConfigsCmd", "completed": "2020-11-10T14:42:40+0000", "created": "2020-11-10T14:42:28+0000", "jobid": "0ea21420-6c48-40c1-9f5f-3fddceac1b0d", "jobprocstatus": 0, "jobresult": { "success": true }, "jobresultcode": 0, "jobresulttype": "object", "jobstatus": 1, "userid": "e3a8b95f-0977-11eb-8a7f-069b900033a6" } ```

…var/log/cloud.log

ustcweizhou force-pushed the 4.15-lbconfigs branch from 025d214 to 1447ac4 Compare July 20, 2020 19:29

weizhouapache closed this Jul 21, 2020

weizhouapache reopened this Jul 21, 2020

weizhouapache changed the title [WIP] Load balancer customization (haproxy-based) [New feature] Load balancer customization (haproxy-based) Jul 21, 2020

weizhouapache marked this pull request as ready for review July 21, 2020 07:51

ravening mentioned this pull request Jul 28, 2020

use X-forward-for address instead of the load balancer ip. #4209

Closed

DaanHoogland added component:virtual-router type:new-feature type:enhancement labels Jul 29, 2020

yadvr mentioned this pull request Aug 12, 2020

Niche feature configuration is always prominent in UI (SSL Offload) #3056

Closed

GutoVeronezi reviewed Jun 29, 2021

View reviewed changes

Sina Kashipazha added 2 commits July 12, 2021 12:28

Merge branch 'main' of https://github.com/apache/cloudstack into apac…

444d37a

…he-main

Merge branch 'apache-main' into 4.15-lbconfigs

317ecbb

Refactored the code, regarding the code change suggetions by GutoVero…

f5e5d0c

…nezi.

soreana force-pushed the 4.15-lbconfigs branch from 2ff1691 to f5e5d0c Compare July 21, 2021 14:50

weizhouapache modified the milestones: 4.16.0.0, 4.17.0.0 Sep 14, 2021

ravening mentioned this pull request Dec 21, 2021

HAproxy based load balancer customization #5799

Closed

2 of 12 tasks

weizhouapache closed this Dec 21, 2021

yadvr removed this from the 4.17.0.0 milestone Mar 24, 2022

GutoVeronezi mentioned this pull request Jun 21, 2022

Add new parameter to createLoadBalancerRule API #6460

Merged

2 of 12 tasks

CharlesQueiroz mentioned this pull request May 1, 2023

HAproxy based load balancer Customisation #7484

Closed

2 of 7 tasks

weizhouapache pushed a commit to weizhouapache/cloudstack that referenced this pull request Jun 14, 2025

apache#4141 LB config: Fix spring-server-core-managers-context.xml

68a0422

weizhouapache pushed a commit to weizhouapache/cloudstack that referenced this pull request Jun 14, 2025

apache#4141 : return empty list if config is null

090bc56

weizhouapache pushed a commit to weizhouapache/cloudstack that referenced this pull request Jun 14, 2025

apache#4141 Ignore case for protocol name in LB rules

3995455

weizhouapache pushed a commit to weizhouapache/cloudstack that referenced this pull request Jun 14, 2025

apache#4141 LB config: fix many 'mkdir -p /etc/ssl/cloudstack/' in /…

1a8f64d

…var/log/cloud.log

+                      if (lbMaxConn != null) {
+                          loadBalancingData.append(",lb.maxconn=").append(lbMaxConn);
+                      }
+                      if (lbFullConn != null) {
+                          loadBalancingData.append(",lb.fullconn=").append(lbFullConn);
+                      }
+                      if (lbTimeoutConnect != null) {
+                          loadBalancingData.append(",lb.timeout.connect=").append(lbTimeoutConnect);
+                      }
+                      if (lbTimeoutServer != null) {
+                          loadBalancingData.append(",lb.timeout.server=").append(lbTimeoutServer);
+                      }
+                      if (lbTimeoutClient != null) {
+                          loadBalancingData.append(",lb.timeout.client=").append(lbTimeoutClient);
+                      }
+                      if (lbBackendHttps != null) {
+                          loadBalancingData.append(",lb.backend.https=").append(lbBackendHttps);
+                      }
+                      if (lbHttp2 != null) {
+                          loadBalancingData.append(",http2=").append(lbHttp2);
+                      }
+                      if (serverMaxconn != null) {
+                          loadBalancingData.append(",server.maxconn=").append(serverMaxconn);
+                      }
+                      if (serverMinconn != null) {
+                          loadBalancingData.append(",server.minconn=").append(serverMinconn);
+                      }
+                      if (serverMaxqueue != null) {
+                          loadBalancingData.append(",server.maxqueue=").append(serverMaxqueue);
+                      }

+                      if (id != null) {
+                          sc.addAnd("id", SearchCriteria.Op.EQ, id);
+                      }
+                      if (scope != null) {
+                          sc.addAnd("scope", SearchCriteria.Op.EQ, scope);
+                      }
+                      if (networkId != null) {
+                          sc.addAnd("networkId", SearchCriteria.Op.EQ, networkId);
+                      }
+                      if (vpcId != null) {
+                          sc.addAnd("vpcId", SearchCriteria.Op.EQ, vpcId);
+                      }
+                      if (loadBalancerId != null) {
+                          sc.addAnd("loadBalancerId", SearchCriteria.Op.EQ, loadBalancerId);
+                      }
+                      if (name != null) {
+                          sc.addAnd("name", SearchCriteria.Op.EQ, name);
+                      }

+                      if (scope == Scope.Network) {
+                          if (networkId == null) {
+                              if (listAll) {
+                                  return;
+                              }
+                              throw new InvalidParameterValueException("networkId is required");
+                          }
+                          if (vpcId != null || loadBalancerId != null) {
+                              throw new InvalidParameterValueException("vpcId and loadBalancerId should be null if scope is Network");
+                          }
+                          if (networkId == null) {
+                              throw new InvalidParameterValueException("networkId is required");
+                          }
+                          NetworkVO network = _networkDao.findById(networkId);
+                          if (network == null) {
+                              throw new InvalidParameterValueException("Cannot find network by id " + networkId);
+                          }
+                          // Perform permission check
+                          _accountMgr.checkAccess(caller, null, true, network);
+                          if (network.getVpcId() != null) {
+                              throw new InvalidParameterValueException("network " + network.getName() + " is a VPC tier, please add LB configs to VPC instead");
+                          }
+                      } else if (scope == Scope.Vpc) {
+                          if (vpcId == null) {
+                              if (listAll) {
+                                  return;
+                              }
+                              throw new InvalidParameterValueException("vpcId is required");
+                          }
+                          if (networkId != null || loadBalancerId != null) {
+                              throw new InvalidParameterValueException("networkId and loadBalancerId should be null if scope is Vpc");
+                          }
+                          VpcVO vpc = _vpcDao.findById(vpcId);
+                          if (vpc == null) {
+                              throw new InvalidParameterValueException("Cannot find vpc by id " + vpcId);
+                          }
+                          // Perform permission check
+                          _accountMgr.checkAccess(caller, null, true, vpc);
+                      } else if (scope == Scope.LoadBalancerRule) {
+                          if (loadBalancerId == null) {
+                              if (listAll) {
+                                  return;
+                              }
+                              throw new InvalidParameterValueException("loadBalancerId is required");
+                          }
+                          if (networkId != null || vpcId != null) {
+                              throw new InvalidParameterValueException("networkId and vpcId should be null if scope is LoadBalancerRule");
+                          }
+                          LoadBalancerVO rule = _lbDao.findById(loadBalancerId);
+                          if (rule == null) {
+                              throw new InvalidParameterValueException("Cannot find load balancer rule by id " + loadBalancerId);
+                          }
+                          if (networkId != null) {
+                              // Perform permission check
+                              checkPermission(Scope.Network, rule.getNetworkId(), null, null);
+                          }
+                      }

+                  }
+                  public void setLbConfigs(List<? extends LoadBalancerConfig> lbConfigs) {
+                      if (lbConfigs == null || lbConfigs.size() == 0) {

                   }
+                  public boolean isForced() {
+                      return (forced != null) ? forced : false;

+                      if (lb == null) {
+                          return null;
+                      }
+                      return lb.getNetworkId();

+                      LoadBalancerConfig config = null;
+                      try {
+                          config = _entityMgr.findById(LoadBalancerConfig.class, getEntityId());

+                          sc.addAnd("name", SearchCriteria.Op.EQ, name);
+                      }
+                      List<LoadBalancerConfigVO> configs = new ArrayList<LoadBalancerConfigVO>();
+                      if (id != null || networkId != null || vpcId != null || loadBalancerId != null) {

[New feature] Load balancer customization (haproxy-based) by weizhouapache · Pull Request #4141 · apache/cloudstack · GitHub

Repository navigation

[New feature] Load balancer customization (haproxy-based) - #4141

[New feature] Load balancer customization (haproxy-based)#4141weizhouapache wants to merge 83 commits intoapache:mainapache/cloudstack:mainfrom ustcweizhou:4.15-lbconfigsustcweizhou/cloudstack:4.15-lbconfigsCopy head branch name to clipboard

Conversation

weizhouapache commented Jun 10, 2020 • edited by DaanHoogland Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

weizhouapache commented Jul 21, 2020

Uh oh!

onitake commented Jul 21, 2020

Uh oh!

weizhouapache commented Jul 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr commented Jul 27, 2020

Uh oh!

blueorangutan commented Jul 27, 2020

Uh oh!

blueorangutan commented Jul 27, 2020

Uh oh!

yadvr commented Aug 5, 2020

Uh oh!

weizhouapache commented Aug 6, 2020

Uh oh!

DaanHoogland commented Aug 11, 2020

Uh oh!

blueorangutan commented Aug 11, 2020

Uh oh!

blueorangutan commented Aug 13, 2020

Uh oh!

yadvr commented Aug 13, 2020

Uh oh!

yadvr commented Aug 13, 2020

Uh oh!

blueorangutan commented Aug 13, 2020

Uh oh!

weizhouapache commented Aug 13, 2020

Uh oh!

weizhouapache commented Aug 13, 2020

Uh oh!

yadvr commented Aug 14, 2020

Uh oh!

blueorangutan commented Aug 14, 2020

Uh oh!

blueorangutan commented Aug 14, 2020

Uh oh!

luhaijiao commented Aug 24, 2020

Uh oh!

blueorangutan commented Jun 29, 2021

Uh oh!

blueorangutan commented Jun 29, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

soreana Jul 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

[New feature] Load balancer customization (haproxy-based)#4141
weizhouapache wants to merge 83 commits into
apache:mainfrom
ustcweizhou:4.15-lbconfigs

weizhouapache commented Jun 10, 2020 •
edited by DaanHoogland
Loading

weizhouapache commented Jul 21, 2020 •
edited
Loading

soreana Jul 12, 2021 •
edited
Loading