Multimodal

In this section, we will walk through how to use DocArray to process multiple data modalities in tandem.

Model your data

If you work with multiple modalities at the same time, most likely they stand in some relation with each other. DocArray allows you to model your data and these relationships.

Define a schema

Suppose you want to model a page of a newspaper that contains a main text, an image URL, a corresponding tensor as well as a description. You can model this example in the following way:

from docarray import BaseDoc
from docarray.typing import ImageTorchTensor, ImageUrl


class Page(BaseDoc):
    main_text: str
    img_url: ImageUrl = None
    img_description: str = None
    img_tensor: ImageTorchTensor = None

Instantiate an object

After extending BaseDoc and defining your schema, you can instantiate an object with your actual data.

page = Page(
    main_text='Hello world',
    img_url='https://github.com/docarray/docarray/blob/main/docs/assets/favicon.png?raw=true',
    img_description='This is the image of an apple',
)

page.img_tensor = page.img_url.load()

page.summary()

Output

 Page : 8f39674 ...

 Attribute                     Value                                         

 main_text: str                Hello world                                   
 img_url: ImageUrl             https://github.com/docarray/docarray/blob/ma 
                               ... (length: 90)                              
 img_description: str          This is DocArray                              
 img_tensor: ImageTorchTensor  ImageTorchTensor of shape (320, 320, 3),      
                               dtype: torch.uint8

Access data

After instantiation, each modality can be accessed directly from the Page object:

print(page.main_text)
print(page.img_url)
print(page.img_description)
print(page.img_tensor)

Output

Hello world
https://github.com/docarray/docarray/blob/main/docs/assets/favicon.png?raw=true
This is DocArray
ImageTorchTensor([[[0, 0, 0],
                   [0, 0, 0],
                   [0, 0, 0],
                   ...,
                   [0, 0, 0]]])

Nested data

If the data you want to model requires a more complex structure, nesting your attributes may be a good solution.

For this example, let's try to define a schema to represent a newspaper. The newspaper should consist of a cover page, any number of following pages, and some metadata. Further, each page contains a main text and can contain an image and an image description.

To implement this you can add a Newspaper class to the previous implementation. The newspaper has a required cover_page attribute of type Page as well as a pages attribute, which is a DocList of Pages.

from docarray import BaseDoc, DocList
from docarray.typing import ImageTorchTensor, ImageUrl


class Page(BaseDoc):
    main_text: str
    img_url: ImageUrl = None
    img_description: str = None
    img_tensor: ImageTorchTensor = None


class Newspaper(BaseDoc):
    cover: Page
    pages: DocList[Page] = None
    metadata: dict = None

You can instantiate this more complex Newspaper object the same way as before:

cover_page = Page(
    main_text='DocArray Daily',
    img_url='https://github.com/docarray/docarray/blob/main/docs/assets/favicon.png',
)

pages = DocList[Page](
    [
        Page(
            main_text='Hello world',
            img_url='https://github.com/docarray/docarray/blob/main/docs/assets/favicon.png',
            img_description='This is the image of an apple',
        ),
        Page(main_text='Second page'),
        Page(main_text='Third page'),
    ]
)

docarray_daily = Newspaper(
    cover=cover_page,
    pages=pages,
    metadata={'author': 'DocArray and friends', 'issue': '0.30.0'},
)

docarray_daily.summary()

Output

 Newspaper : 63189f7 ...

 Attribute       Value                                                       

 metadata: dict  {'author': 'DocArray and friends', 'issue': '0.0.3 ... }    
                 (length: 2)                                                 

  cover: Page
     Page : ca164e3 ...
       
        Attribute          Value                                            
       
        main_text: str     DocArray Daily                                   
        img_url: ImageUrl  https://github.com/docarray/docarray/blob/main/ 
                           ... (length: 81)                                 
       
  pages: DocList[Page]
      Page : 64ed19c ...
       
        Attribute             Value                                         
       
        main_text: str        Hello world                                   
        img_url: ImageUrl     https://github.com/docarray/docarray/blob/ma 
                              ... (length: 81)                              
        img_description: str  DocArray logoooo                              
       
      Page : 4bd7e45 ...
       
        Attribute            Value          
       
        main_text: str       Second page    
       
     ... 1 more Page documents