FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

chore(docs): sync up README from docarray · docarray/docarray.github.io@63a43d9 · GitHub

/ docarray.github.io Public

Commit 63a43d9

Browse files
dev-bot@jina.ai
authored and
dev-bot@jina.ai
committed
chore(docs): sync up README from docarray
Signed-off-by: dev-bot@jina.ai <Jina Dev Bot>
1 parent 177b5b3 commit 63a43d9

1 file changed

Lines changed: 73 additions & 72 deletions

File tree

‎README.md‎

Lines changed: 73 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -12,49 +12,56 @@
1212
<a href="https://discord.gg/WaMp6PVPgR"><img src="https://dcbadge.vercel.app/api/server/WaMp6PVPgR?theme=default-inverted&style=flat-square"></a>
1313
</p>
1414

15-
> ⬆️ **DocArray v2**: This readme is for the second version of DocArray (starting at 0.30). If you want to use the older
16-
> version (prior to 0.30) check out the [docarray-v1-fixes](https://github.com/docarray/docarray/tree/docarray-v1-fixes) branch
15+
> **Note**
16+
> The README you're currently viewing is for DocArray>0.30, which introduces some significant changes from DocArray 0.21. If you wish to continue using the older DocArray <=0.21, ensure you install it via `pip install docarray==0.21`. Refer to its [codebase](https://github.com/docarray/docarray/tree/v0.21.0), [documentation](https://docarray.jina.ai), and [its hot-fixes branch](https://github.com/docarray/docarray/tree/docarray-v1-fixes) for more information.
1717
18-
DocArray is a library for **representing, sending and storing multi-modal data**, perfect for **Machine Learning applications**.
1918

20-
With DocArray you can:
19+
DocArray is a Python library expertly crafted for the [representation](#represent), [transmission](#send), [storage](#store), and [retrieval](#retrieve) of multimodal data. Tailored for the development of multimodal AI applications, its design guarantees seamless integration with the extensive Python and machine learning ecosystems. As of January 2022, DocArray is openly distributed under the [Apache License 2.0](https://github.com/docarray/docarray/blob/main/LICENSE) and currently enjoys the status of a sandbox project within the [LF AI & Data Foundation](https://lfaidata.foundation/).
2120

22-
1. [**Represent data**](#represent)
23-
2. [**Send data**](#send)
24-
3. [**Store data**](#store)
2521

26-
DocArray handles your data while integrating seamlessly with the rest of your **Python and ML ecosystem**:
2722

28-
- :fire: Native compatibility for **[NumPy](https://github.com/numpy/numpy)**, **[PyTorch](https://github.com/pytorch/pytorch)** and **[TensorFlow](https://github.com/tensorflow/tensorflow)**, including for **model training use cases**
29-
- :zap: Built on **[Pydantic](https://github.com/pydantic/pydantic)** and out-of-the-box compatible with **[FastAPI](https://github.com/tiangolo/fastapi/)** and **[Jina](https://github.com/jina-ai/jina/)**
30-
- :package: Support for vector databases like **[Weaviate](https://weaviate.io/), [Qdrant](https://qdrant.tech/), [ElasticSearch](https://www.elastic.co/de/elasticsearch/)** and **[HNSWLib](https://github.com/nmslib/hnswlib)**
31-
- :chains: Send data as JSON over **HTTP** or as **[Protobuf](https://protobuf.dev/)** over **[gRPC](https://grpc.io/)**
23+
- :fire: Offers native support for **[NumPy](https://github.com/numpy/numpy)**, **[PyTorch](https://github.com/pytorch/pytorch)**, and **[TensorFlow](https://github.com/tensorflow/tensorflow)**, catering specifically to **model training scenarios**.
24+
- :zap: Based on **[Pydantic](https://github.com/pydantic/pydantic)**, and instantly compatible with web and microservice frameworks like **[FastAPI](https://github.com/tiangolo/fastapi/)** and **[Jina](https://github.com/jina-ai/jina/)**.
25+
- :package: Provides support for vector databases such as **[Weaviate](https://weaviate.io/), [Qdrant](https://qdrant.tech/), [ElasticSearch](https://www.elastic.co/de/elasticsearch/)**, and **[HNSWLib](https://github.com/nmslib/hnswlib)**.
26+
- :chains: Allows data transmission as JSON over **HTTP** or as **[Protobuf](https://protobuf.dev/)** over **[gRPC](https://grpc.io/)**.
3227

33-
> :bulb: **Where are you coming from?** Based on your use case and background, there are different ways to understand DocArray:
34-
>
35-
> - [Coming from pure PyTorch or TensorFlow](#coming-from-pytorch)
36-
> - [Coming from Pydantic](#coming-from-pydantic)
37-
> - [Coming from FastAPI](#coming-from-fastapi)
38-
> - [Coming from a vector database](#coming-from-vector-database)
39-
> - [Coming from Langchain](#coming-from-langchain)
28+
## Installation
29+
30+
To install DocArray from the CLI, run the following command:
31+
32+
```shell
33+
pip install -U docarray
34+
```
35+
36+
> **Note**
37+
> To use DocArray <=0.21, make sure you install via `pip install docarray==0.21` and check out its [codebase](https://github.com/docarray/docarray/tree/v0.21.0) and [docs](https://docarray.jina.ai) and [its hot-fixes branch](https://github.com/docarray/docarray/tree/docarray-v1-fixes).
38+
39+
## Get Started
40+
New to DocArray? Depending on your use case and background, there are multiple ways to learn about DocArray:
41+
42+
- [Coming from pure PyTorch or TensorFlow](#coming-from-pytorch)
43+
- [Coming from Pydantic](#coming-from-pydantic)
44+
- [Coming from FastAPI](#coming-from-fastapi)
45+
- [Coming from a vector database](#coming-from-vector-database)
46+
- [Coming from Langchain](#coming-from-langchain)
4047

41-
DocArray has been distributed under the open-source [Apache License 2.0](https://github.com/docarray/docarray/blob/main/LICENSE) since January 2022. It is currently a sandbox project under [LF AI & Data Foundation](https://lfaidata.foundation/).
4248

4349
## Represent
4450

45-
DocArray allows you to **represent your data**, in an ML-native way.
51+
DocArray empowers you to **represent your data** in a manner that is inherently attuned to machine learning.
4652

47-
This is useful for different use cases:
53+
This is particularly beneficial for various scenarios:
4854

49-
- :running: You are **training a model**: There are tensors of different shapes and sizes flying around, representing different _things_, and you want to keep a straight head about them.
50-
- :cloud: You are **serving a model**: For example through FastAPI, and you want to specify your API endpoints.
51-
- :card_index_dividers: You are **parsing data**: For later use in your ML or data science applications.
55+
- :running: You are **training a model**: You're dealing with tensors of varying shapes and sizes, each signifying different elements. You desire a method to logically organize them.
56+
- :cloud: You are **serving a model**: Let's say through FastAPI, and you wish to define your API endpoints precisely.
57+
- :card_index_dividers: You are **parsing data**: Perhaps for future deployment in your machine learning or data science projects.
5258

53-
> :bulb: **Coming from Pydantic?** You should be happy to hear
54-
> that DocArray is built on top of, and is fully compatible with, Pydantic!
55-
> Also, we have a [dedicated section](#coming-from-pydantic) just for you!
59+
> :bulb: **Familiar with Pydantic?** You'll be pleased to learn
60+
> that DocArray is not only constructed atop Pydantic but also maintains complete compatibility with it!
61+
> Furthermore, we have a [specific section](#coming-from-pydantic) dedicated to your needs!
62+
63+
In essence, DocArray facilitates data representation in a way that mirrors Python dataclasses, with machine learning being an integral component:
5664

57-
Put simply, DocArray lets you represent your data in a dataclass-like way, with ML as a first class citizen:
5865

5966
```python
6067
from docarray import BaseDoc
@@ -256,21 +263,22 @@ assert isinstance(dl_2, DocList)
256263

257264
## Send
258265

259-
DocArray allows you to **send your data** in an ML-native way.
266+
DocArray facilitates the **transmission of your data** in a manner inherently compatible with machine learning.
267+
268+
This includes native support for **Protobuf and gRPC**, along with **HTTP** and serialization to JSON, JSONSchema, Base64, and Bytes.
260269

261-
This means there is native support for **Protobuf and gRPC**, on top of **HTTP** and serialization to JSON, JSONSchema, Base64, and Bytes.
270+
This feature proves beneficial for several scenarios:
262271

263-
This is useful for different use cases:
272+
- :cloud: You are **serving a model**, perhaps through frameworks like **[Jina](https://github.com/jina-ai/jina/)** or **[FastAPI](https://github.com/tiangolo/fastapi/)**
273+
- :spider_web: You are **distributing your model** across multiple machines and need an efficient means of transmitting your data between nodes
274+
- :gear: You are architecting a **microservice** environment and require a method for data transmission between microservices
264275

265-
- :cloud: You are **serving a model**, for example through **[Jina](https://github.com/jina-ai/jina/)** or **[FastAPI](https://github.com/tiangolo/fastapi/)**
266-
- :spider_web: You are **distributing your model** across machines and need to send your data between nodes
267-
- :gear: You are building a **microservice** architecture and need to send your data between microservices
276+
> :bulb: **Are you familiar with FastAPI?** You'll be delighted to learn
277+
> that DocArray maintains full compatibility with FastAPI!
278+
> Plus, we have a [dedicated section](#coming-from-fastapi) specifically for you!
268279
269-
> :bulb: **Coming from FastAPI?** You should be happy to hear
270-
> that DocArray is fully compatible with FastAPI!
271-
> Also, we have a [dedicated section](#coming-from-fastapi) just for you!
280+
When it comes to data transmission, serialization is a crucial step. Let's delve into how DocArray streamlines this process:
272281

273-
Whenever you want to send your data, you need to serialize it, so let's take a look at how that works with DocArray:
274282

275283
```python
276284
from docarray import BaseDoc
@@ -305,18 +313,14 @@ Of course, serialization is not all you need. So check out how DocArray integrat
305313

306314
## Store
307315

308-
Once you've modelled your data, and maybe sent it around, usually you want to **store it** somewhere.
309-
DocArray has you covered!
316+
After modeling and possibly distributing your data, you'll typically want to **store it** somewhere. That's where DocArray steps in!
310317

311-
**Document Stores** let you, well, store your Documents, locally or remotely, all with the same user interface:
318+
**Document Stores** provide a seamless way to, as the name suggests, store your Documents. Be it locally or remotely, you can do it all through the same user interface:
312319

313-
- :cd: **On disk** as a file in your local file system
320+
- :cd: **On disk**, as a file in your local filesystem
314321
- :bucket: On **[AWS S3](https://aws.amazon.com/de/s3/)**
315322
- :cloud: On **[Jina AI Cloud](https://cloud.jina.ai/)**
316323

317-
<details markdown="1">
318-
<summary>See Document Store usage</summary>
319-
320324
The Document Store interface lets you push and pull Documents to and from multiple data sources, all with the same user interface.
321325

322326
For example, let's see how that works with on-disk storage:
@@ -334,7 +338,8 @@ docs.push('file://simple_docs')
334338

335339
docs_pull = DocList[SimpleDoc].pull('file://simple_docs')
336340
```
337-
</details>
341+
342+
## Retrieve
338343

339344
**Document Indexes** let you index your Documents in a **vector database** for efficient similarity-based retrieval.
340345

@@ -346,9 +351,6 @@ This is useful for:
346351

347352
Currently, Document Indexes support **[Weaviate](https://weaviate.io/)**, **[Qdrant](https://qdrant.tech/)**, **[ElasticSearch](https://www.elastic.co/)**, and **[HNSWLib](https://github.com/nmslib/hnswlib)**, with more to come!
348353

349-
<details markdown="1">
350-
<summary>See Document Index usage</summary>
351-
352354
The Document Index interface lets you index and retrieve Documents from multiple vector databases, all with the same user interface.
353355

354356
It supports ANN vector search, text search, filtering, and hybrid search.
@@ -391,18 +393,21 @@ query = dl[0]
391393
results, scores = index.find(query, limit=10, search_field='embedding')
392394
```
393395

394-
</details>
396+
397+
---
398+
399+
## Learn DocArray
395400

396401
Depending on your background and use case, there are different ways for you to understand DocArray.
397402

398-
## Coming from old DocArray
403+
### Coming from DocArray <=0.21
399404

400405
<details markdown="1">
401406
<summary>Click to expand</summary>
402407

403408
If you are using DocArray version 0.30.0 or lower, you will be familiar with its [dataclass API](https://docarray.jina.ai/fundamentals/dataclass/).
404409

405-
_DocArray v2 is that idea, taken seriously._ Every document is created through a dataclass-like interface,
410+
_DocArray >=0.30 is that idea, taken seriously._ Every document is created through a dataclass-like interface,
406411
courtesy of [Pydantic](https://pydantic-docs.helpmanual.io/usage/models/).
407412

408413
This gives the following advantages:
@@ -420,7 +425,7 @@ For now, Document Indexes support **[Weaviate](https://weaviate.io/)**, **[Qdran
420425

421426
</details>
422427

423-
## Coming from Pydantic
428+
### Coming from Pydantic
424429

425430
<details markdown="1">
426431
<summary>Click to expand</summary>
@@ -497,7 +502,7 @@ except Exception as e:
497502

498503
</details>
499504

500-
## Coming from PyTorch
505+
### Coming from PyTorch
501506

502507
<details markdown="1">
503508
<summary>Click to expand</summary>
@@ -511,7 +516,7 @@ It offers you several advantages:
511516
- **Go directly to deployment**, by re-using your data model as a [FastAPI](https://fastapi.tiangolo.com/) or [Jina](https://github.com/jina-ai/jina) API schema
512517
- Connect model components between **microservices**, using Protobuf and gRPC
513518

514-
DocArray can be used directly inside ML models to handle and represent multi-modal data.
519+
DocArray can be used directly inside ML models to handle and represent multimodaldata.
515520
This allows you to reason about your data using DocArray's abstractions deep inside of `nn.Module`,
516521
and provides a FastAPI-compatible schema that eases the transition between model training and model serving.
517522

@@ -609,7 +614,7 @@ schema definition (see [below](#coming-from-fastapi)). Everything is handled in
609614
</details>
610615

611616

612-
## Coming from TensorFlow
617+
### Coming from TensorFlow
613618

614619
<details markdown="1">
615620
<summary>Click to expand</summary>
@@ -657,7 +662,7 @@ class MyPodcastModel(tf.keras.Model):
657662

658663
</details>
659664

660-
## Coming from FastAPI
665+
### Coming from FastAPI
661666

662667
<details markdown="1">
663668
<summary>Click to expand</summary>
@@ -680,6 +685,7 @@ from docarray import BaseDoc
680685
from docarray.documents import ImageDoc
681686
from docarray.typing import NdArray
682687

688+
683689
class InputDoc(BaseDoc):
684690
img: ImageDoc
685691
text: str
@@ -692,29 +698,32 @@ class OutputDoc(BaseDoc):
692698

693699
app = FastAPI()
694700

701+
695702
def model_img(img: ImageTensor) -> NdArray:
696703
return np.zeros((100, 1))
697704

705+
698706
def model_text(text: str) -> NdArray:
699707
return np.zeros((100, 1))
700708

709+
701710
@app.post("/embed/", response_model=OutputDoc, response_class=DocArrayResponse)
702711
async def create_item(doc: InputDoc) -> OutputDoc:
703712
doc = OutputDoc(
704713
embedding_clip=model_img(doc.img.tensor), embedding_bert=model_text(doc.text)
705714
)
706715
return doc
707716

717+
708718
async with AsyncClient(app=app, base_url="http://test") as ac:
709719
response = await ac.post("/embed/", data=input_doc.json())
710-
711720
```
712721

713722
Just like a vanilla Pydantic model!
714723

715724
</details>
716725

717-
## Coming from a vector database
726+
### Coming from a vector database
718727

719728
<details markdown="1">
720729
<summary>Click to expand</summary>
@@ -770,14 +779,14 @@ Currently, DocArray supports the following vector databases:
770779

771780
An integration of [OpenSearch](https://opensearch.org/) is currently in progress.
772781

773-
Legacy versions of DocArray also support [Redis](https://redis.io/) and [Milvus](https://milvus.io/), but these are not yet supported in the current version.
782+
DocArray <=0.21 also support [Redis](https://redis.io/) and [Milvus](https://milvus.io/), but these are not yet supported in the current version.
774783

775784
Of course this is only one of the things that DocArray can do, so we encourage you to check out the rest of this readme!
776785

777786
</details>
778787

779788

780-
## Coming from Langchain
789+
### Coming from Langchain
781790

782791
<details markdown="1">
783792
<summary>Click to expand</summary>
@@ -835,7 +844,6 @@ db = InMemoryExactNNIndex[MovieDoc](docs)
835844

836845
3. Finally, initialize a retriever and integrate it into your chain!
837846
```python
838-
839847
from langchain.chat_models import ChatOpenAI
840848
from langchain.chains import ConversationalRetrievalChain
841849
from langchain.retrievers import DocArrayRetriever
@@ -859,20 +867,13 @@ Both are user-friendly and are best suited to small to medium-sized datasets.
859867

860868
</details>
861869

862-
## Installation
863-
864-
To install DocArray from the CLI, run the following command:
865-
866-
```shell
867-
pip install -U docarray
868-
```
869870

870871
## See also
871872

872873
- [Documentation](https://docs.docarray.org)
874+
- [DocArray<=0.21 documentation](https://docarray.jina.ai/)
873875
- [Join our Discord server](https://discord.gg/WaMp6PVPgR)
874876
- [Donation to Linux Foundation AI&Data blog post](https://jina.ai/news/donate-docarray-lf-for-inclusive-standard-multimodal-data-model/)
875-
- ["Legacy" DocArray github page](https://github.com/docarray/docarray/tree/docarray-v1-fixes)
876-
- ["Legacy" DocArray documentation](https://docarray.jina.ai/)
877+
877878

878879
> DocArray is a trademark of LF AI Projects, LLC

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL