Backgrounds

In many ML use cases, we need to manage three or more parts of data, including training sets, validation sets, and test sets. This requires the data system to provide different partitions for these data, preventing users from retrieving unintended data.

Currently, TransferQueueController manages all data samples together and thus does not support this usage. In our recipe in verl, we temporarily work around this issue by initializing another set of TransferQueues for _validate. This adds extra complexity to usage, and we will soon provide official support for data partitions.

Solution

We will add another parameter data_partition to the get_meta & put functions in TransferQueueClient. By specifying this parameter, users can logically divide data into different partitions.

TODO:

• Add logical partition in TransferQueueController PR#98 @LLLLxmmm
• Add data_partition for get_meta & put functions in TransferQueueClient PR#98 @LLLLxmmm