3D Mesh

DocArray supports many different modalities including 3D Mesh. This section will show you how to load and handle 3D data using DocArray.

A 3D mesh is the structural build of a 3D model consisting of polygons. Most 3D meshes are created via professional software packages, such as commercial suites like Unity, or the open-source Blender.

pip install "docarray[mesh]"

Vertices and Faces representation

A 3D mesh can be represented by its vertices and faces:

• Vertices are points in a 3D space, represented as a tensor of shape (n_points, 3).
• Faces are triangular surfaces that are defined by three points in 3D space, corresponding to the three vertices of a triangle. They can be represented as a tensor of shape (n_faces, 3). Each number in that tensor refers to an index of a vertex in the tensor of vertices.

Load vertices and faces

First, let's define our class MyMesh3D, which extends BaseDoc and provides attributes to store our 3D data:

• The mesh_url attribute of type Mesh3DUrl.
• The optional tensors attribute, of type VerticesAndFaces
• The VerticesAndFaces class has the attributes vertices and faces, both of type AnyTensor. This especially comes in handy later when we want to display our 3D mesh.

But for now, let's create a MyMesh3D instance with a URL to a remote .obj file:

from typing import Optional

from docarray import BaseDoc
from docarray.documents.mesh.vertices_and_faces import VerticesAndFaces
from docarray.typing import Mesh3DUrl


class MyMesh3D(BaseDoc):
    mesh_url: Mesh3DUrl
    tensors: Optional[VerticesAndFaces] = None


doc = MyMesh3D(mesh_url="https://people.sc.fsu.edu/~jburkardt/data/obj/al.obj")

To load the vertices and faces information, you can call .load() on the Mesh3DUrl instance. This will return a VerticesAndFaces object:

doc.tensors = doc.mesh_url.load()
doc.summary()

 MyMesh3D : 9d8c26f ...

 Attribute            Value                                                  

 mesh_url: Mesh3DUrl  https://people.sc.fsu.edu/~jburkardt/data/obj/al.o ... 
                      (length: 52)                                           

  tensors: VerticesAndFaces
      VerticesAndFaces : 8cae4c4 ...
        
         Attribute          Value                                            
        
         vertices: NdArray  NdArray of shape (3980, 3), dtype: float64       
         faces: NdArray     NdArray of shape (7152, 3), dtype: int64         
        

Display 3D mesh in notebook

You can display your 3D mesh interactively from its URL as well as a VerticesAndFaces instance, by calling .display() on either one. The latter will always display without color, whereas the display from the URL will show with color if this information is included in the file content.

doc.mesh_url.display()

Point cloud representation

A point cloud is a representation of a 3D mesh. It is made by repeatedly and uniformly sampling points within the surface of the 3D body. Compared to the mesh representation, the point cloud is a fixed size ndarray and hence easier for deep learning algorithms to handle.

Load point cloud

In DocArray, loading a point cloud from a PointCloud3DUrl instance will return a PointsAndColors instance. Such an object has a points attribute containing the information about the points in 3D space as well as an optional colors attribute.

First, let's define our class MyPointCloud, which extends BaseDoc and provides attributes to store the point cloud information:

from typing import Optional

from docarray import BaseDoc
from docarray.documents.point_cloud.points_and_colors import PointsAndColors
from docarray.typing import PointCloud3DUrl


class MyPointCloud(BaseDoc):
    url: PointCloud3DUrl
    tensors: Optional[PointsAndColors] = None


doc = MyPointCloud(url="https://people.sc.fsu.edu/~jburkardt/data/obj/al.obj")

Next, we can load a point cloud of size samples by simply calling .load() on the PointCloud3DUrl instance:

doc.tensors = doc.url.load(samples=1000)
doc.summary()

 MyPointCloud : a63374d ...

 Attribute                         Value                                     

 url: PointCloud3DUrl              https://people.sc.fsu.edu/~jburkardt/dat 
                                   ... (length: 52)                          

  tensors: PointsAndColors
      PointsAndColors : 70ae175 ...
        
         Attribute        Value                                              
        
         points: NdArray  NdArray of shape (1000, 3), dtype: float64         
        

Display 3D point cloud in notebook

You can display your point cloud and interact with it from its URL as well as from a PointsAndColors instance. The first will always display without color, whereas the display from PointsAndColors will show with color if .colors is not None.

doc.url.display()

Getting started - Predefined documents

To get started and play around with the 3D modalities, DocArray provides the predefined documents Mesh3D and PointCloud3D, which includes all of the previously mentioned functionalities.

Mesh3D

The Mesh3D class provides a Mesh3DUrl field and VerticesAndFaces field.

class Mesh3D(BaseDoc):
    url: Optional[Mesh3DUrl] = None
    tensors: Optional[VerticesAndFaces] = None
    embedding: Optional[AnyEmbedding] = None
    bytes_: Optional[bytes] = None

PointCloud3D

class PointCloud3D(BaseDoc):
    url: Optional[PointCloud3DUrl] = None
    tensors: Optional[PointsAndColors] = None
    embedding: Optional[AnyEmbedding] = None
    bytes_: Optional[bytes] = None

You can use them directly, extend or compose them:

from docarray import BaseDoc
from docarray.documents import Mesh3D, PointCloud3D


class My3DObject(BaseDoc):
    title: str
    mesh: Mesh3D
    pc: PointCloud3D


obj_file = 'https://people.sc.fsu.edu/~jburkardt/data/obj/al.obj'

doc = My3DObject(
    title='My first 3D object!',
    mesh=Mesh3D(url=obj_file),
    pc=PointCloud3D(url=obj_file),
)

doc.mesh.tensors = doc.mesh.url.load()
doc.pc.tensors = doc.pc.url.load(samples=100)