[BREAKING] Refactor: Provide a general storage abstraction layer by 0oshowero0 · Pull Request #66 · TransferQueue/TransferQueue · GitHub

Provide TransferQueueStorageManager abs class, acts as a general entry point for storage layer
Implement AsyncTransferQueueStorageSimpleUnitManager class to manage TransferQueueSimpleUnit
Disentangle the complex storage related operations from TransferQueueClient, provide a general Client interface

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

TODO: UT

Pull Request Overview

This PR introduces a general storage abstraction layer for the Transfer Queue system, refactoring storage operations from the client to dedicated storage managers. The key goal is to decouple complex storage-related operations from the client and provide a cleaner architecture.

Abstract base class TransferQueueStorageManager provides common functionality for controller notifications
New AsyncTransferQueueStorageSimpleUnitManager manages storage units and implements the storage interface
Refactored client to use storage managers instead of direct storage unit communication

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
transfer_queue/storage.py	Introduces storage manager abstraction and implementation
transfer_queue/metadata.py	Removes storage-specific fields from metadata classes
transfer_queue/controller.py	Removes storage unit management and mapping logic
transfer_queue/client.py	Refactors to use storage manager instead of direct storage operations
recipe/simple_use_case/async_demo.py	Updates demo to use new storage manager initialization pattern

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

TODO:

Now TransferQueueClient may not be able to be passed through Ray (not serializable). Need to investigate if there is a simple way to solve it.
TransferQueueClient needs a single controller_infos, while TransferQueueStorageManager requires all controller_infos. Need to reconcile.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

…abstract

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

…abstract

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

refactor: renew README.md, fix import error, and fix errors in sync_demo.py

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

…abstract

…#2) * feat: enhance handshake process with timeout and retransmission logic Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com> * fix: issues in comments Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com> * fix: fix issues in comments Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com> * Update transfer_queue/storage.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update transfer_queue/storage.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com> Co-authored-by: Han Zhenyu 韩振宇 <o0shower0o@outlook.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

#3) * docs: improve docstrings for clarity and consistency across storage.py Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com> * docs: update examples in build_storage_meta_groups and _register_servers for clarity Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com> --------- Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com>

…StorageManager (#8) Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com>

…#5) * docs: improve docstrings for clarity and consistency across client.py * Update transfer_queue/client.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix comments * fix comments --------- Co-authored-by: liuximeng <13073314+liuximeng18772102439@user.noreply.gitee.com> Co-authored-by: Han Zhenyu 韩振宇 <o0shower0o@outlook.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com>

* Enable Controller UT * fix codecheck * fix codecheck --------- Co-authored-by: liuximeng <13073314+liuximeng18772102439@user.noreply.gitee.com>

…impleStorageManager (#7) * feat: update tests for SimpleStorageUnit and add new tests for AsyncSimpleStorageManager Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com> * refactor: simplify mock operations and adjust performance test parameters for faster execution Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com> --------- Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com>

* Handle TODOs in the demo * fix codecheck * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: liuximeng <13073314+liuximeng18772102439@user.noreply.gitee.com> Co-authored-by: Han Zhenyu 韩振宇 <o0shower0o@outlook.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

Pull Request Overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

The .items() call returns tuples of (key, value), but the code treats controller_info as if it's the value only. This will cause incorrect type checking. Should iterate over .values() instead or destructure the tuple properly.

Suggested change

      
                            for controller_info in raw_controller_infos.items():
          
                            for controller_info in raw_controller_infos.values():

Docstring mentions looking up storage node addresses but the implementation no longer does this after refactoring. The docstring should be updated to reflect that this method now only creates metadata without storage location mapping.

Method check_current_step_consumption has async signature but the implementation is just pass. If this is intentional placeholder, add a TODO comment or docstring explaining when it will be implemented. The similar method check_current_step_production has the same issue.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

0oshowero0 added 3 commits October 11, 2025 18:45

refactor storage manager abstraction

832a7de

fix

cc02634

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

replace origin files

ac21c16

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

0oshowero0 requested review from LLLLxmmm, baymax591, Copilot, ji-huazhong and jianjunzhong October 13, 2025 07:24

Copilot AI reviewed Oct 13, 2025

View reviewed changes

0oshowero0 added 14 commits October 14, 2025 14:15

fix

d6a41e9

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

fix conflict with PR67

947d977

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

fix conflict with PR67

eff9d03

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

apply pre-commit check

391c902

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

provide StorageManagerFactory

83ca41f

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

Merge remote-tracking branch 'upstream/dev' into han/unified_storage_…

8e2115f

…abstract

update README

2dff556

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

simplify class name

0783a1e

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

update

7491ea2

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

update

a29ad74

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

add DeepWiki and Github star stats

f41e4cd

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

Merge remote-tracking branch 'upstream/dev' into dev

cc43ffe

Merge remote-tracking branch 'upstream/dev' into han/unified_storage_…

f4398aa

…abstract

simplify some codes & add TODO

85df853

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

This was referenced Oct 21, 2025

[Roadmap] A general storage abstraction layer #72

Open

Shared-memory via ray.put()/ray.get() vs socket IPC for data transfer #71

Closed

jianjunzhong and others added 3 commits October 21, 2025 16:38

Merge pull request #1 from jianjunzhong/refactor/1021

53e1e4d

refactor: renew README.md, fix import error, and fix errors in sync_demo.py

update progress

f60aac0

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

Merge remote-tracking branch 'upstream/dev' into dev

1d355f0

0oshowero0 and others added 6 commits October 21, 2025 17:02

fix typo

e537cc0

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

fix typo

0cb3655

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

Merge remote-tracking branch 'upstream/dev' into dev

44f4592

Merge branch 'dev' into han/unified_storage_abstract

e4fc730

Merge remote-tracking branch 'upstream/dev' into han/unified_storage_…

a389569

…abstract

jianjunzhong mentioned this pull request Oct 22, 2025

[Enhancement] Improve robustness of handshake logic in _connect_to_controller() #78

Closed

jianjunzhong and others added 13 commits October 23, 2025 14:32

refactor: remove unnecessary global_indexes assignment in AsyncSimple…

7e166fe

…StorageManager (#8) Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com>

docs: streamline docstrings and comments for clarity in metadata.py (#6 )

cf7143b

Signed-off-by: jianjunzhong <jianjunzhong@foxmail.com>

Enable Controller UT (#9 )

a9d5bbf

* Enable Controller UT * fix codecheck * fix codecheck --------- Co-authored-by: liuximeng <13073314+liuximeng18772102439@user.noreply.gitee.com>

fix client ut

46ce00a

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

update test dependencies

4c2ac4f

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

update

c07c47c

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

fix

4c065d5

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

Merge branch 'dev' into han/unified_storage_abstract

d507514

apply pre-commit check

a9028f9

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

0oshowero0 requested a review from Copilot October 25, 2025 06:34

Copilot AI reviewed Oct 25, 2025

View reviewed changes

0oshowero0 and others added 3 commits October 25, 2025 14:36

Update transfer_queue/metadata.py

0b8e663

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

update docstring

3ba55f2

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

fix

8bebd60

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>

0oshowero0 merged commit fc92f03 into TransferQueue:dev Oct 25, 2025
2 checks passed

0oshowero0 deleted the han/unified_storage_abstract branch October 29, 2025 06:13

0oshowero0 mentioned this pull request Nov 18, 2025

[recipe, data] feat: TransferQueue - Support managing multiple data partitions for Train/Val/Test in controller verl-project/verl#4175

Merged

7 tasks

+                          elif isinstance(self.controller_infos, dict):
+                              raw_controller_infos = self.controller_infos
+                              self.controller_infos = {}
+                              for controller_info in raw_controller_infos.items():

[BREAKING] Refactor: Provide a general storage abstraction layer by 0oshowero0 · Pull Request #66 · TransferQueue/TransferQueue · GitHub

Uh oh!

[BREAKING] Refactor: Provide a general storage abstraction layer - #66

[BREAKING] Refactor: Provide a general storage abstraction layer#66
0oshowero0 merged 42 commits into
TransferQueue:devfrom
0oshowero0:han/unified_storage_abstract

0oshowero0 commented Oct 13, 2025

Uh oh!

0oshowero0 commented Oct 13, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0oshowero0 commented Oct 13, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Oct 25, 2025

Uh oh!

Uh oh!

Copilot AI Oct 25, 2025

Uh oh!

Copilot AI Oct 25, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	for controller_info in raw_controller_infos.items():
	for controller_info in raw_controller_infos.values():

		@@ -407,8 +368,6 @@ def _generate_batch_meta(
		Generate BatchMeta by resolving storage locations for given global indexes.

		For each global index, looks up the corresponding storage node address using:

-                              body={"local_indexes": local_indexes},
-                          )
+                  @dynamic_socket(socket_name="request_handle_socket")
+                  async def check_current_step_consumption(self, task_name: str, global_step: int):

[BREAKING] Refactor: Provide a general storage abstraction layer by 0oshowero0 · Pull Request #66 · TransferQueue/TransferQueue · GitHub

Uh oh!

[BREAKING] Refactor: Provide a general storage abstraction layer - #66

[BREAKING] Refactor: Provide a general storage abstraction layer#660oshowero0 merged 42 commits intoTransferQueue:devTransferQueue/TransferQueue:devfrom 0oshowero0:han/unified_storage_abstractCopy head branch name to clipboard

Conversation

0oshowero0 commented Oct 13, 2025

Uh oh!

0oshowero0 commented Oct 13, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0oshowero0 commented Oct 13, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[BREAKING] Refactor: Provide a general storage abstraction layer#66
0oshowero0 merged 42 commits into
TransferQueue:devfrom
0oshowero0:han/unified_storage_abstract