FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Adding new Dell EMC ECS Object Storage Plugin for CloudStack by mhkadhum · Pull Request #12124 · apache/cloudstack · GitHub

  1. apache
  2. cloudstack

Repository navigation

Adding new Dell EMC ECS Object Storage Plugin for CloudStack - #12124

Open
mhkadhum wants to merge 27 commits into
apache:mainfrom
mhkadhum:main
ConversationCommits (27)ChecksFiles changed
Open

Adding new Dell EMC ECS Object Storage Plugin for CloudStack#12124
mhkadhum wants to merge 27 commits into
apache:mainfrom
mhkadhum:main

Conversation

mhkadhum commented Nov 24, 2025

Copy link
Copy Markdown

Description

This PR adds support for a Dell EMC ECS S3 object storage plugin for Apache CloudStack. ECS is a software-defined object storage platform that supports both object and file-system protocols, with a focus on scalable and reliable object storage. We have been using ECS in production for three years, and this work extends CloudStack’s existing MinIO plugin to provide similar functionality for ECS.

The implementation supports the full lifecycle of S3-compatible buckets on ECS, including user provisioning, bucket creation, policy management, versioning, creation-time encryption, and integration with CloudStack’s S3 Browser. All functionality has been tested in a lab environment on Ubuntu using a CloudStack development setup based on the official installation guidelines.

Key architectural differences from MinIO:

Management API Integration
ECS requires use of the ECS Management API (port 4443, or 443 when fronted by HAProxy). CloudStack authenticates with management-user credentials to perform bucket and user operations.

Namespace Requirements
A dedicated ECS namespace is required for CloudStack-managed buckets. Multiple namespaces allow different CloudStack environments to share the same ECS cluster.

S3 Endpoints (Public and Private)
ECS exposes S3 services on ports 9020/9021 (non-TLS/TLS). In our deployment, these are routed through HAProxy and exposed externally on port 443. The Public URL is displayed to CloudStack users, while the Private URL is used internally.

TLS Handling
The “Allow Insecure HTTPS” option controls whether CloudStack accepts untrusted certificates when communicating with the ECS Management API.

User Provisioning Workflow
When a CloudStack user creates their first bucket, CloudStack provisions a corresponding ECS object user using the CloudStack UUID with a cs- prefix. ECS generates access and secret keys once, which CloudStack securely stores and reuses for subsequent bucket operations.

Bucket features:

  • Encryption: Supported only during bucket creation; CloudStack hides encryption in the update view.
  • Bucket Policy: Supports Private and Public configurations.
  • Versioning: Fully supported through the S3 API rather than the Management API.
  • Object Lock: Not supported in this release; the UI hides the option and API calls return a clear error.

Bucket modification supports quota changes, versioning updates, and policy changes. Encryption is excluded because ECS does not allow changing it after creation.

CloudStack’s S3 Browser supports upload, download, delete, listing, and prefix filtering through the ECS S3 endpoint. ECS prevents deletion of non-empty buckets, and CloudStack surfaces these errors accordingly.

We welcome review and feedback. The development fork is available here:
https://github.com/mhkadhum/cloudstack

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

The plugin was tested in an Ubuntu-based CloudStack development environment created by cloning the CloudStack source and following the official installation instructions. Testing included:

  • Management API authentication and namespace operations
  • Object-user provisioning and credential handling
  • Bucket creation, deletion, versioning, quota updates, and policy changes
  • S3 Browser operations (upload, download, delete, listing, prefix navigation)
  • Error handling for invalid configurations and unsupported Object Lock requests
  • TLS and insecure-TLS scenarios
  • HAProxy-fronted deployments for both Management API and S3 endpoints

boring-cyborg Bot commented Nov 24, 2025

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

jbampton reviewed Nov 24, 2025
View reviewed changes

jbampton left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

You have failing tests for pre-commit and license checks.

Have a quick read up on pre-commit it really is a series of basic checks we run:

https://github.com/apache/cloudstack/blob/main/PRE-COMMIT.md

Also we normally add the ASF license header to all files or add an exclude

codecov Bot commented Nov 24, 2025 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 810 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.08%. Comparing base (856d83a) to head (3781560).
⚠️ Report is 496 commits behind head on main.

Files with missing lines Patch % Lines
...age/datastore/driver/EcsObjectStoreDriverImpl.java 0.00% 634 Missing ⚠️
...tastore/lifecycle/EcsObjectStoreLifeCycleImpl.java 0.00% 84 Missing ⚠️
...udstack/storage/datastore/driver/EcsXmlParser.java 0.00% 39 Missing ⚠️
.../cloudstack/storage/datastore/driver/EcsUtils.java 0.00% 31 Missing ⚠️
...datastore/provider/EcsObjectStoreProviderImpl.java 0.00% 13 Missing ⚠️
...he/cloudstack/storage/datastore/driver/EcsCfg.java 0.00% 7 Missing ⚠️
...age/datastore/driver/EcsUnauthorizedException.java 0.00% 2 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               main   #12124      +/-   ##
============================================
+ Coverage     18.01%   18.08%   +0.06%     
- Complexity    16607    16753     +146     
============================================
  Files          6029     6044      +15     
  Lines        542154   543636    +1482     
  Branches      66451    66646     +195     
============================================
+ Hits          97681    98290     +609     
- Misses       433457   434305     +848     
- Partials      11016    11041      +25
Flag Coverage Δ
uitests 3.51% <ø> (-0.01%) ⬇️
unittests 19.24% <0.00%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sureshanaparti commented Nov 25, 2025

Copy link
Copy Markdown
Contributor

@mhkadhum can you check the build errors?

boring-cyborg Bot added component:marvin Python Warning... Python code Ahead! labels Nov 26, 2025

mhkadhum commented Nov 26, 2025

Copy link
Copy Markdown
Author

Thanks for the feedback!

ive added the missing license headers and ran pre-commit locally, and pushed an updated commit

DaanHoogland reviewed Nov 27, 2025
View reviewed changes
Comment thread tools/marvin/setup.py Resolved

DaanHoogland commented Nov 28, 2025

Copy link
Copy Markdown
Contributor

looks generally good @mhkadhum , thanks for the contribution. Will you be maintaining this? As it is a 3rd party component and it may require hardware or licenses not available to the project.

mhkadhum commented Nov 28, 2025

Copy link
Copy Markdown
Author

looks generally good @mhkadhum , thanks for the contribution. Will you be maintaining this? As it is a 3rd party component and it may require hardware or licenses not available to the project.

Yes, I will be the maintainer for the plugin. I have made a new commit to fix the previous build. Could you kindly review it

DaanHoogland reviewed Nov 28, 2025
View reviewed changes

DaanHoogland left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I have not found any oddities in the code, but one remark (in two parts ;) ) Feel free to ignore but I think it will help in the future.

Comment thread ...main/java/org/apache/cloudstack/storage/datastore/lifecycle/EcsObjectStoreLifeCycleImpl.java Resolved
Comment thread ...s/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java Resolved
DaanHoogland requested a review from abh1sar December 3, 2025 06:58

winterhazel commented Jun 9, 2026

Copy link
Copy Markdown
Member

@mhkadhum I was not able to review the new changes yet. The test failure here in GitHub does not seem related to them. We can try running the integration tests with a real hypervisor to check.

winterhazel commented Jun 9, 2026

Copy link
Copy Markdown
Member

@DaanHoogland @weizhouapache @nvazquez can we run the CI here?

github-actions Bot mentioned this pull request Jun 9, 2026
Closed

mhkadhum commented Jun 12, 2026

Copy link
Copy Markdown
Author

@DaanHoogland @weizhouapache @nvazquez can we run the CI here?

Hi @DaanHoogland
Can u check this please 🙂

winterhazel commented Jun 12, 2026

Copy link
Copy Markdown
Member

@DaanHoogland @weizhouapache @nvazquez can we run the CI here?

Hi @DaanHoogland Can u check this please 🙂

Also pinging @Pearl1594

Pearl1594 commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

@blueorangutan test

blueorangutan commented Jun 12, 2026

Copy link
Copy Markdown

@Pearl1594 a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

weizhouapache approved these changes Jun 12, 2026
View reviewed changes

weizhouapache left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

java code lgtm

github-actions Bot mentioned this pull request Jun 12, 2026
Closed

blueorangutan commented Jun 13, 2026

Copy link
Copy Markdown

[SF] Trillian test result (tid-16299)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 49044 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12124-t16299-kvm-ol8.zip
Smoke tests completed. 151 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

github-actions Bot mentioned this pull request Jun 13, 2026
Closed

mhkadhum commented Jun 16, 2026

Copy link
Copy Markdown
Author

Hello @winterhazel @DaanHoogland @Pearl1594

The test result looks good.

[SF] Trillian test result (tid-16299) Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8 Total time taken: 49044 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12124-t16299-kvm-ol8.zip Smoke tests completed. 151 look OK, 0 have errors, 0 did not run Only failed and skipped tests results shown below:

Test Result Time (s) Test File

i am not sure how to resolve these two :/

mhkadhum commented Jun 17, 2026

Copy link
Copy Markdown
Author

Hello @winterhazel @DaanHoogland @Pearl1594

The test result looks good.

[SF] Trillian test result (tid-16299) Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8 Total time taken: 49044 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12124-t16299-kvm-ol8.zip Smoke tests completed. 151 look OK, 0 have errors, 0 did not run Only failed and skipped tests results shown below:
Test Result Time (s) Test File

i am not sure how to resolve these two :/

@DaanHoogland

winterhazel commented Jun 17, 2026

Copy link
Copy Markdown
Member

Hello @winterhazel @DaanHoogland @Pearl1594

The test result looks good.

[SF] Trillian test result (tid-16299) Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8 Total time taken: 49044 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12124-t16299-kvm-ol8.zip Smoke tests completed. 151 look OK, 0 have errors, 0 did not run Only failed and skipped tests results shown below:
Test Result Time (s) Test File

i am not sure how to resolve these two :/

@mhkadhum the first failure is caused by the lack of unit tests (coverage under 18%) for these changes. Passing this check is not a requirement for merging, but of course it would be good to have tests to prevent future regressions, if you would like to add some. The second one does not seem related to your changes, but I triggered a rerun.

winterhazel commented Jun 17, 2026

Copy link
Copy Markdown
Member

@mhkadhum the failure is an issue with Codecov, not with your changes:

==> Verifying GPG signature integrity
 -> Downloading https://cli.codecov.io/latest/linux/codecov.SHA256SUM
 -> Downloading https://cli.codecov.io/latest/linux/codecov.SHA256SUM.sig
 
gpg: Signature made Tue Apr 21 19:28:03 2026 UTC
gpg:                using RSA key 27034E7FDB850E0BBC2C62FF806BB28AED779869
gpg: Can't check signature: No public key
==> Could not verify signature. Please contact Codecov if problem continues
    Exiting...
Error: Process completed with exit code 1.

mhkadhum commented Jun 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

@winterhazel

@mhkadhum the failure is an issue with Codecov, not with your changes:

==> Verifying GPG signature integrity
 -> Downloading https://cli.codecov.io/latest/linux/codecov.SHA256SUM
 -> Downloading https://cli.codecov.io/latest/linux/codecov.SHA256SUM.sig
 
gpg: Signature made Tue Apr 21 19:28:03 2026 UTC
gpg:                using RSA key 27034E7FDB850E0BBC2C62FF806BB28AED779869
gpg: Can't check signature: No public key
==> Could not verify signature. Please contact Codecov if problem continues
    Exiting...
Error: Process completed with exit code 1.

Great, I think we are ready to merge ?
regarding the unit test they can be added later as an improvement

winterhazel commented Jun 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

@winterhazel

@mhkadhum the failure is an issue with Codecov, not with your changes:

==> Verifying GPG signature integrity
 -> Downloading https://cli.codecov.io/latest/linux/codecov.SHA256SUM
 -> Downloading https://cli.codecov.io/latest/linux/codecov.SHA256SUM.sig
 
gpg: Signature made Tue Apr 21 19:28:03 2026 UTC
gpg:                using RSA key 27034E7FDB850E0BBC2C62FF806BB28AED779869
gpg: Can't check signature: No public key
==> Could not verify signature. Please contact Codecov if problem continues
    Exiting...
Error: Process completed with exit code 1.

Great, I think we are ready to merge ? regarding the unit test they can be added later as an improvement

@mhkadhum I think there were some issues while adding your changes on the latest commits. Some unrelated labels were modified or removed from en.json. Also, it seems that some of @abh1sar's suggestions were reverted as well. Could you have a look?

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

weizhouapache commented Jun 30, 2026

Copy link
Copy Markdown
Member

@mhkadhum
could you fix the conflict ?

weizhouapache commented Jun 30, 2026

Copy link
Copy Markdown
Member

@abh1sar
can you review it ? thanks

mhkadhum commented Jul 1, 2026

Copy link
Copy Markdown
Author

@abh1sar can you review it ? thanks

Hello

I'm quite busy with other commitments this week and won't be able to make progress on this until next week.

thank u

DaanHoogland requested changes Jul 2, 2026
View reviewed changes
Comment thread ui/public/locales/en.json

DaanHoogland Jul 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I am pretty sure that the removals in this file are invalid. The additions seem ecs related, but the removals are not.

weizhouapache commented Jul 8, 2026

Copy link
Copy Markdown
Member

moving to 4.24.0

weizhouapache modified the milestones: 4.23.0, 4.24.0 Jul 8, 2026
DaanHoogland moved this from Backlog to conflict/waiting for author in CloudStack Testing Aug 31, 2026

nvazquez commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Hi @mhkadhum can you please address the conflicts?

This branch has not been deployed

No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

DaanHoogland DaanHoogland requested changes
weizhouapache weizhouapache approved these changes
jbampton Awaiting requested review from jbampton
sureshanaparti Awaiting requested review from sureshanaparti
abh1sar Awaiting requested review from abh1sar
winterhazel Awaiting requested review from winterhazel

Assignees

No one assigned

Labels

3rd-party:ecs component:build component:marvin component:object-storage Object Storage component:storage component:UI has-3rdparty-components Python Warning... Python code Ahead! status:has-conflicts status:needs-review

Projects

CloudStack Testing
Status: conflict/waiting

Milestone

24.0

Development

Successfully merging this pull request may close these issues.

10 participants

Back | FazBrowse Home | New Git URL