Apache OpenDAL Python binding#

Installation#

1

pip install opendal

Local Usage#

Developer must set two required arguments to work with files locally: - scheme: which should be specified as fs - root: where OpenDAL considers the root of the directory for operations will be.

For example in the following operator: opendal.Operator("fs", root="/foo")

OpenDAL considers /foo to be the root of all paths, and that means that we can access paths inside of /foo without specifying anything else. If /foo happens to contain the file baz.txt, we can simply call .list("/")

We can see this in the following example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from opendal import Operator
from pathlib import Path
import os


os.makedirs("foo", exist_ok=True)

file = Path("foo", "bar.txt")
file.touch()

op = Operator("fs", root=Path("foo").name)
for file in op.list("/"):
    print(file)

When running we get the following output:

1
2

/
bar.txt

If we want full access to our file system, we can specify a root of "/", but note that for any operations you will always need to specify the full path to a file or directory.

Reading#

There are two ways to read data using OpenDAL. One way is to use the read method on an operator:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

from opendal import Operator
from pathlib import Path
import os


os.makedirs("foo", exist_ok=True)

file = Path("foo", "bar.txt")
file.touch()
file.write_text("baz")


op = Operator("fs", root=Path("foo").name)

data = op.read(file.name)
print(data)

Yields the following:

1

b'baz'

Note that the output is bytes, but we can get it as a string by simply calling .decode() on the data we read. All reads with OpenDAL return bytes.

Now lets use the open method on an operator:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

from opendal import Operator
from pathlib import Path
import os


os.makedirs("foo", exist_ok=True)

file = Path("foo", "bar.txt")
file.touch()
file.write_text("baz")


op = Operator("fs", root=Path("foo").name)

with op.open(file.name, "rb") as f:
    print(f.read())

This again yields

1

b'baz'

Again, note that all reads with OpenDAL return bytes, so specifying a mode of "r" will raise an exception.

Writing#

Now that we know how to read data, let's replace the Pathlib code above with OpenDAL using the write method:

1
2
3
4
5
6
7
8
9

from opendal import Operator
from pathlib import Path


op = Operator("fs", root=Path("foo").name)

op.write("baz.txt", "my amazing data".encode())

print(op.read("baz.txt").decode())

This yields the following:

1

my amazing data

And again, but using the open method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

from opendal import Operator
from pathlib import Path


op = Operator("fs", root=Path("foo").name)

with op.open("baz.txt", "wb") as f:
    f.write("my amazing data".encode())

print(op.read("baz.txt").decode())

This again yields:

1

my amazing data

Again, note that all writing happens in bytes and a mode of "w" will raise an exception.

Async#

OpenDAL supports async operation on all operator methods. One can simply replace the Operator with an AsyncOperator and await on method calls. The below example illustrates this behavior:

Standard API:

1
2
3
4
5
6

import opendal

op = opendal.Operator("fs", root="/tmp")
op.write("test.txt", b"Hello World")
print(op.read("test.txt"))
print(op.stat("test.txt").content_length)

Async API equivalent:

1
2
3
4
5
6
7
8

import asyncio

async def main():
    op = opendal.AsyncOperator("fs", root="/tmp")
    await op.write("test.txt", b"Hello World")
    print(await op.read("test.txt"))

asyncio.run(main())