Add support for Ceph RGW Object Store by wido · Pull Request #8389 · apache/cloudstack · GitHub

Description

This PR adds support for the Ceph RGW as an Object Store Driver in CloudStack

Types of changes

New feature (non-breaking change which adds functionality)

Screenshots (if appropriate):

How Has This Been Tested?

I have a local Ceph environment (IPv6 only) running with the RADOS Gateway (S3 server) running against which I tested this PR.

It also has SSL enabled so it could verify that this works as expected.

How did you try to break this feature and the system with this change?

I manually tried to create and remove buckets and that worked as expected.

Codecov Report

Attention: Patch coverage is 13.02083% with 334 lines in your changes missing coverage. Please review.

Project coverage is 15.64%. Comparing base (72d0546) to head (a448992).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...ge/datastore/driver/CephObjectStoreDriverImpl.java	13.08%	185 Missing and 1 partial ⚠️
...astore/lifecycle/CephObjectStoreLifeCycleImpl.java	0.00%	57 Missing ⚠️
...e/datastore/driver/MinIOObjectStoreDriverImpl.java	11.11%	16 Missing ⚠️
...loudstack/storage/object/BucketApiServiceImpl.java	0.00%	16 Missing ⚠️
...atastore/provider/CephObjectStoreProviderImpl.java	40.00%	15 Missing ⚠️
...src/main/java/com/cloud/agent/api/to/BucketTO.java	22.22%	14 Missing ⚠️
...oudstack/storage/object/store/ObjectStoreImpl.java	0.00%	14 Missing ⚠️
...tastore/driver/SimulatorObjectStoreDriverImpl.java	0.00%	11 Missing ⚠️
...hema/src/main/java/com/cloud/storage/BucketVO.java	66.66%	3 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8389      +/-   ##
============================================
+ Coverage     15.57%   15.64%   +0.06%     
- Complexity    12056    12118      +62     
============================================
  Files          5506     5539      +33     
  Lines        482997   485358    +2361     
  Branches      59483    58983     -500     
============================================
+ Hits          75236    75941     +705     
- Misses       399450   401043    +1593     
- Partials       8311     8374      +63

Flag	Coverage Δ
uitests	4.12% <ø> (-0.05%)	⬇️
unittests	16.44% <13.02%> (+0.08%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

clgtm

@blueorangutan package

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 8356

@wido I'll help kick tests again - can you check for build failures?

@blueorangutan package

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 8524

@blueorangutan package

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 8548

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

@blueorangutan package

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10984

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

@blueorangutan package

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10987

Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 10989

@wido :

11:03:41 [ERROR] /jenkins/workspace/acs-deb-pkg-builder/plugins/storage/object/ceph/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/CephObjectStoreLifeCycleImpl.java:[30,23] error: package org.apache.log4j does not exist
11:03:41 [ERROR] /jenkins/workspace/acs-deb-pkg-builder/plugins/storage/object/ceph/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/CephObjectStoreLifeCycleImpl.java:[41,25] error: cannot find symbol
11:03:41   symbol:   class Logger
11:03:41   location: class CephObjectStoreLifeCycleImpl

strange this error would only happen on the debian build and the el build is successful, but there you have it ???

Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 10989

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Fixed some issue, tested LGTM but some cases don't work will raise a separate issue for @wido to explore further:

Bucket creation fails when any of the options are specified or these options aren't supported: quota, versioning & encryption, bucket policy
During Bucket creation, locking feature/option is ignored
User quota not enforced/configured
Bucket policy fails and throws exception:

Error while updating bucket. Error while updating bucket: admin-test-bucket. SignatureDoesNotMatch (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx00000c2e2357a96c47d4c-0066d97a79-42778b-default; S3 Extended Request ID: 42778b-default-default; Proxy: null)

UI browser doesn't work (due to cors issue, couldn't figure out how to configure radosgw for that)

@blueorangutan package

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

LGTM, tested this plugins with Ceph Reef rgw, doesn't support all options but good to merge this.

Testing notes:

Setup ceph cluster & rgw service:

Next, was able to add this as object storage pool in cloudstack using url & api/secret credentials by creating a system 'admin' user in the rgw:

Following screenshots show, when cloudstack bucket is created, the relevant user is created in rgw & the bucket:

Was able to create bucket without any options or bucket policy, but then was able to use S3 client like Cyberduck to create/configure access permissions on the bucket & upload/download objects/files in it:

Bucket policies don't work via cloudstack UI though and throw exception:

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

@blueorangutan package

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Merging this based on manual QA with limitations and followup logged at #9641 (cc @wido), reviews, earlier smoketests & GH build/smoketests.

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11001

This feature adds support for Ceph's RADOS Gateway (RGW) support for the Object Store feature of CloudStack. The RGW of Ceph is Amazon S3 compliant and is therefor an easy and straigforward implementation of basic S3 features. Existing Ceph environments can have the RGW added as an additional feature to a cluster already providing RBD (Block Device) to a CloudStack environment. Introduce the BucketTO to pass to the drivers. This replaces just passing the bucket's name. Some upcoming drivers require more information then just the bucket name to perform their actions, for example they require the access and secret key which belong to the account of this bucket. This is leftover code from a long time ago and this validation test has nu influence on the end result on how a URL will be used afterwards. We should support hosts pointing to an IPv6(-only) address out of the box. For the code it does not matter if it's IPv4 or IPv6. This is the admin's choice. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com> Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

wido added component:ceph component:object-storage Object Storage labels Dec 20, 2023

wido requested a review from kishankavala December 20, 2023 16:22

boring-cyborg Bot added component:api component:build component:storage component:UI labels Dec 20, 2023

DaanHoogland requested a review from vishesh92 December 21, 2023 08:16

github-actions Bot added the status:has-conflicts label Jan 9, 2024

wido force-pushed the ceph-object-store branch from 209332f to 7eb5966 Compare January 9, 2024 15:13

github-actions Bot removed the status:has-conflicts label Jan 9, 2024

DaanHoogland added this to the 4.20.0.0 milestone Jan 10, 2024

vishesh92 approved these changes Jan 17, 2024

View reviewed changes

yadvr requested review from harikrishna-patnala and sureshanaparti February 5, 2024 09:01

yadvr closed this Feb 5, 2024

yadvr reopened this Feb 5, 2024

weizhouapache reviewed Feb 5, 2024

View reviewed changes

Comment thread plugins/storage/object/ceph/pom.xml Resolved

wido force-pushed the ceph-object-store branch from 7eb5966 to a0b0471 Compare February 6, 2024 10:11

yadvr added 2 commits September 5, 2024 12:53

fix failing unit test

04905c5

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

remove unnecessary stubbing

2f4ac5f

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

get latest bucketTO in case credentials are changed/updated

c86eb5a

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

fix user creation and bucket creation issue on rgw

9d5c436

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr approved these changes Sep 5, 2024

View reviewed changes

yadvr mentioned this pull request Sep 5, 2024

Ceph RADOS-GW object storage plugin issue #9641

Closed

yadvr closed this Sep 5, 2024

yadvr reopened this Sep 5, 2024

apache deleted a comment from blueorangutan Sep 5, 2024

yadvr added 2 commits September 5, 2024 17:03

Merge remote-tracking branch 'origin/main' into ceph-object-store

e5a7405

fix logger build issue

3d58be1

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

fix unit test :(

a448992

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr merged commit c3f0d14 into apache:main Sep 5, 2024

Add support for Ceph RGW Object Store by wido · Pull Request #8389 · apache/cloudstack · GitHub

Repository navigation

Add support for Ceph RGW Object Store - #8389

Add support for Ceph RGW Object Store#8389yadvr merged 18 commits intoapache:mainapache/cloudstack:mainfrom wido:ceph-object-storewido/cloudstack:ceph-object-storeCopy head branch name to clipboard

Conversation

wido commented Dec 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov Bot commented Dec 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Jan 9, 2024

Uh oh!

vishesh92 left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Jan 18, 2024

Uh oh!

blueorangutan commented Jan 18, 2024

Uh oh!

blueorangutan commented Jan 18, 2024

Uh oh!

yadvr commented Feb 5, 2024

Uh oh!

blueorangutan commented Feb 5, 2024

Uh oh!

blueorangutan commented Feb 5, 2024

Uh oh!

Uh oh!

weizhouapache commented Feb 6, 2024

Uh oh!

blueorangutan commented Feb 6, 2024

Uh oh!

blueorangutan commented Feb 6, 2024

Uh oh!

yadvr commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

DaanHoogland commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

DaanHoogland commented Sep 5, 2024

Uh oh!

yadvr commented Sep 5, 2024

Uh oh!

yadvr commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr commented Sep 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

yadvr commented Sep 5, 2024

Uh oh!

blueorangutan commented Sep 5, 2024

Uh oh!

Reviewers

Uh oh!

Assignees

Add support for Ceph RGW Object Store#8389
yadvr merged 18 commits into
apache:mainfrom
wido:ceph-object-store

wido commented Dec 20, 2023 •
edited
Loading

codecov Bot commented Dec 20, 2023 •
edited
Loading

yadvr commented Sep 5, 2024 •
edited
Loading