Fix the deletion of S3 buckets by bernardodemarco · Pull Request #14678 · apache/cloudstack · GitHub

Description

PR #10140 introduced the allocated_size column to the cloud.object_store table, which is expected to hold the sum of all bucket quota, in bytes. To initialize the new column, the following query was introduced:

cloudstack/engine/schema/src/main/resources/META-INF/db/schema-42040to42100.sql

Lines 742 to 749 in f8f3848

    
           UPDATE `cloud`.`object_store` 
        
           JOIN ( 
        
               SELECT object_store_id, SUM(quota) AS total_quota 
        
               FROM `cloud`.`bucket` 
        
               WHERE removed IS NULL 
        
               GROUP BY object_store_id 
        
           ) buckets_quota_sum_view ON `object_store`.id = buckets_quota_sum_view.object_store_id 
        
           SET `object_store`.allocated_size = buckets_quota_sum_view.total_quota;

However, bucket quotas are stored in GiB, not bytes, and the query does not convert them. As a result, Object Storage providers in environments upgraded from versions prior to 4.21 have an allocated_size that is smaller than expected. When a bucket is deleted, its quota (converted to bytes) is subtracted from this value, which becomes negative. Since the column is unsigned, the update fails and the bucket cannot be deleted:

Impossibility of deleting buckets

2026-11-10 16:10:07,939 ERROR [o.a.c.s.d.d.ObjectStoreDaoImpl] (qtp1404565079-3736:[ctx-3a285296, ctx-ce4f3b14]) (logid:78d41388) DB Exception on: HikariProxyPreparedStatement@1008598898 wrapping com.mysql.cj.jdbc.ServerPreparedStatement[21]: UPDATE object_store SET object_store.allocated_size=-536870912000 WHERE object_store.id = 2 com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Out of range value for column 'allocated_size' at row 1
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:104)
	at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:555)
	at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:339)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1320)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:994)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)

Therefore, this PR recalculates the allocated size in bytes for each Object Storage provider. It targets the 4.22.2.0 version. A follow-up PR targeting main will apply the same fix to environments already running 4.23.0.0.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

BLOCKER
Critical
Major
Minor
Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Executed the normalization query locally and verified it recalculates correctly the allocated size for each Object Storage provider.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 18.32%. Comparing base (b56bbc2) to head (74017f5).

Additional details and impacted files

@@            Coverage Diff            @@
##               4.22   #14678   +/-   ##
=========================================
  Coverage     18.32%   18.32%           
- Complexity    16644    16647    +3     
=========================================
  Files          5939     5939           
  Lines        537787   537787           
  Branches      65932    65932           
=========================================
+ Hits          98544    98553    +9     
+ Misses       428033   428023   -10     
- Partials      11210    11211    +1

Flag	Coverage Δ
uitests	4.08% <ø> (ø)
unittests	19.41% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueorangutan package

@vladimirpetrov a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 20019

Recalculate the allocated size in bytes for each Object Storage provider

74017f5

bernardodemarco added this to the 4.22.2 milestone Nov 11, 2026

bernardodemarco requested review from hsato03 and winterhazel November 11, 2026 13:02

bernardodemarco mentioned this pull request Nov 11, 2026

Backport #14678 to main: Fix the deletion of S3 buckets #14679

Open

2 of 12 tasks

Fix the deletion of S3 buckets by bernardodemarco · Pull Request #14678 · apache/cloudstack · GitHub

Repository navigation

Fix the deletion of S3 buckets - #14678

Fix the deletion of S3 buckets#14678
bernardodemarco wants to merge 1 commit into
apache:4.22from
scclouds:fix-object-store-allocated-size

bernardodemarco commented Nov 11, 2026

Uh oh!

codecov Bot commented Nov 11, 2026 •
edited
Loading

Uh oh!

Uh oh!

vladimirpetrov commented Nov 12, 2026

Uh oh!

blueorangutan commented Nov 12, 2026

Uh oh!

blueorangutan commented Nov 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	UPDATE `cloud`.`object_store`
	JOIN (
	SELECT object_store_id, SUM(quota) AS total_quota
	FROM `cloud`.`bucket`
	WHERE removed IS NULL
	GROUP BY object_store_id
	) buckets_quota_sum_view ON `object_store`.id = buckets_quota_sum_view.object_store_id
	SET `object_store`.allocated_size = buckets_quota_sum_view.total_quota;

Fix the deletion of S3 buckets by bernardodemarco · Pull Request #14678 · apache/cloudstack · GitHub

Repository navigation

Fix the deletion of S3 buckets - #14678

Fix the deletion of S3 buckets#14678bernardodemarco wants to merge 1 commit intoapache:4.22apache/cloudstack:4.22from scclouds:fix-object-store-allocated-sizescclouds/cloudstack:fix-object-store-allocated-sizeCopy head branch name to clipboard

Conversation

bernardodemarco commented Nov 11, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

codecov Bot commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vladimirpetrov commented Nov 12, 2026

Uh oh!

blueorangutan commented Nov 12, 2026

Uh oh!

blueorangutan commented Nov 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Fix the deletion of S3 buckets#14678
bernardodemarco wants to merge 1 commit into
apache:4.22from
scclouds:fix-object-store-allocated-size

codecov Bot commented Nov 11, 2026 •
edited
Loading