pyarrow.fs.AzureFileSystem#

class pyarrow.fs.AzureFileSystem(account_name, account_key=None, *, blob_storage_authority=None, blob_storage_scheme=None, client_id=None, client_secret=None, dfs_storage_authority=None, dfs_storage_scheme=None, sas_token=None, tenant_id=None)#

Bases: FileSystem

Azure Blob Storage backed FileSystem implementation

This implementation supports flat namespace and hierarchical namespace (HNS) a.k.a. Data Lake Gen2 storage accounts. HNS will be automatically detected and HNS specific features will be used when they provide a performance advantage. Azurite emulator is also supported. Note: / is the only supported delimiter.

The storage account is considered the root of the filesystem. When enabled, containers will be created or deleted during relevant directory operations. Obviously, this also requires authentication with the additional permissions.

By default DefaultAzureCredential is used for authentication. This means it will try several types of authentication and go with the first one that works. If any authentication parameters are provided when initialising the FileSystem, they will be used instead of the default credential.

Parameters:

account_namestr

Azure Blob Storage account name. This is the globally unique identifier for the storage account.

account_keystr, default None

Account key of the storage account. If sas_token and account_key are None the default credential will be used. The parameters account_key and sas_token are mutually exclusive.

blob_storage_authoritystr, default None

hostname[:port] of the Blob Service. Defaults to .blob.core.windows.net. Useful for connecting to a local emulator, like Azurite.

blob_storage_schemestr, default None

Either http or https. Defaults to https. Useful for connecting to a local emulator, like Azurite.

client_idstr, default None

The client ID (Application ID) for Azure Active Directory authentication. Its interpretation depends on the credential type being used:

For ClientSecretCredential: It is the Application (client) ID of your registered Azure AD application (Service Principal). It must be provided together with tenant_id and client_secret to use ClientSecretCredential.
For ManagedIdentityCredential: It is the client ID of a specific user-assigned managed identity. This is only necessary if you are using a user-assigned managed identity and need to explicitly specify which one (e.g., if the resource has multiple user-assigned identities). For system-assigned managed identities, this parameter is typically not required.

client_secretstr, default None

Client secret for Azure Active Directory authentication. Must be provided together with tenant_id and client_id to use ClientSecretCredential.

dfs_storage_authoritystr, default None

hostname[:port] of the Data Lake Gen 2 Service. Defaults to .dfs.core.windows.net. Useful for connecting to a local emulator, like Azurite.

dfs_storage_schemestr, default None

Either http or https. Defaults to https. Useful for connecting to a local emulator, like Azurite.

sas_tokenstr, default None

SAS token for the storage account, used as an alternative to account_key. If sas_token and account_key are None the default credential will be used. The parameters account_key and sas_token are mutually exclusive.

tenant_idstr, default None

Tenant ID for Azure Active Directory authentication. Must be provided together with client_id and client_secret to use ClientSecretCredential.

Examples

>>> from pyarrow import fs
>>> azure_fs = fs.AzureFileSystem(account_name='myaccount')
>>> azurite_fs = fs.AzureFileSystem(
...     account_name='devstoreaccount1',
...     account_key='Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==',
...     blob_storage_authority='127.0.0.1:10000',
...     dfs_storage_authority='127.0.0.1:10000',
...     blob_storage_scheme='http',
...     dfs_storage_scheme='http',
... )

For usage of the methods see examples for LocalFileSystem().

__init__(*args, **kwargs)#

Methods

`__init__`(args,*kwargs)
`copy_file`(self,src,dest)	Copy a file.
`create_dir`(self,path,*,boolrecursive=True)	Create a directory and subdirectories.
`delete_dir`(self,path)	Delete a directory and its contents, recursively.
`delete_dir_contents`(self,path,*,...)	Delete a directory's contents, recursively.
`delete_file`(self,path)	Delete a file.
`equals`(self,FileSystemother)
`from_uri`(uri)	Create a new FileSystem from URI or Path.
`get_file_info`(self,paths_or_selector)	Get info for the given files.
`move`(self,src,dest)	Move / rename a file or directory.
`normalize_path`(self,path)	Normalize filesystem path.
`open_append_stream`(self,path[,...])	Open an output stream for appending.
`open_input_file`(self,path)	Open an input file for random access reading.
`open_input_stream`(self,path[,compression,...])	Open an input stream for sequential reading.
`open_output_stream`(self,path[,...])	Open an output stream for sequential writing.

Attributes

type_name

The filesystem's type name.

copy_file(self, src, dest)#

Copy a file.

If the destination exists and is a directory, an error is returned. Otherwise, it is replaced.

Parameters:

srcstr: The path of the file to be copied from.
deststr: The destination path where the file is copied to.

Examples

>>> local.copy_file(path,
...                 local_path + '/pyarrow-fs-example_copy.dat')

Inspect the file info:

>>> local.get_file_info(local_path + '/pyarrow-fs-example_copy.dat')
<FileInfo for '/.../pyarrow-fs-example_copy.dat': type=FileType.File, size=4>
>>> local.get_file_info(path)
<FileInfo for '/.../pyarrow-fs-example.dat': type=FileType.File, size=4>

create_dir(self, path, *, bool recursive=True)#

Create a directory and subdirectories.

This function succeeds if the directory already exists.

Parameters:

pathstr: The path of the new directory.
recursivebool, default True: Create nested directories as well.

delete_dir(self, path)#

Delete a directory and its contents, recursively.

Parameters:

pathstr: The path of the directory to be deleted.

delete_dir_contents(self, path, *, bool accept_root_dir=False, bool missing_dir_ok=False)#

Delete a directorys contents, recursively.

Like delete_dir, but doesnt delete the directory itself.

Parameters:

pathstr: The path of the directory to be deleted.
accept_root_dirbool, default False: Allow deleting the root directorys contents (if path is empty or /)
missing_dir_okbool, default False: If False then an error is raised if path does not exist

delete_file(self, path)#

Delete a file.

Parameters:

pathstr: The path of the file to be deleted.

equals(self, FileSystem other)#

Parameters:

otherpyarrow.fs.FileSystem

Returns:

bool

static from_uri(uri)#

Create a new FileSystem from URI or Path.

Recognized URI schemes are file, mock, s3fs, gs, gcs, hdfs and viewfs. In addition, the argument can be a pathlib.Path object, or a string describing an absolute local path.

Parameters:

uristr: URI-based path, for example: file:///some/local/path.

Returns:

tuple of (FileSystem, str path): With (filesystem, path) tuple where path is the abstract path inside the FileSystem instance.

Examples

Create a new FileSystem subclass from a URI:

>>> uri = f'file:///{local_path}/pyarrow-fs-example.dat'
>>> local_new, path_new = fs.FileSystem.from_uri(uri)
>>> local_new
<pyarrow._fs.LocalFileSystem object at ...
>>> path_new
'/.../pyarrow-fs-example.dat'

Or from a s3 bucket:

>>> fs.FileSystem.from_uri("s3://usgs-landsat/collection02/")
(<pyarrow._s3fs.S3FileSystem object at ...>, 'usgs-landsat/collection02')

Or from an fsspec+ URI:

>>> fs.FileSystem.from_uri("fsspec+memory:///path/to/file")
(<pyarrow._fs.PyFileSystem object at ...>, '/path/to/file')

get_file_info(self, paths_or_selector)#

Get info for the given files.

Any symlink is automatically dereferenced, recursively. A non-existing or unreachable file returns a FileStat object and has a FileType of value NotFound. An exception indicates a truly exceptional condition (low-level I/O error, etc.).

Parameters:

paths_or_selectorFileSelector, path-like or list of path-likes: Either a selector object, a path-like object or a list of path-like objects. The selectors base directory will not be part of the results, even if it exists. If it doesnt exist, use allow_not_found.

Returns:

FileInfo or list of FileInfo: Single FileInfo object is returned for a single path, otherwise a list of FileInfo objects is returned.

Examples

>>> local
<pyarrow._fs.LocalFileSystem object at ...>
>>> local.get_file_info(f"/{local_path}/pyarrow-fs-example.dat")
<FileInfo for '/.../pyarrow-fs-example.dat': type=FileType.File, size=4>

move(self, src, dest)#

Move / rename a file or directory.

If the destination exists: - if it is a non-empty directory, an error is returned - otherwise, if it has the same type as the source, it is replaced - otherwise, behavior is unspecified (implementation-dependent).

Parameters:

srcstr: The path of the file or the directory to be moved.
deststr: The destination path where the file or directory is moved to.

Examples

Create a new folder with a file:

>>> local.create_dir('/tmp/other_dir')
>>> local.copy_file(path,'/tmp/move_example.dat')

Move the file:

>>> local.move('/tmp/move_example.dat',
...            '/tmp/other_dir/move_example_2.dat')

Inspect the file info:

>>> local.get_file_info('/tmp/other_dir/move_example_2.dat')
<FileInfo for '/tmp/other_dir/move_example_2.dat': type=FileType.File, size=4>
>>> local.get_file_info('/tmp/move_example.dat')
<FileInfo for '/tmp/move_example.dat': type=FileType.NotFound>

Delete the folder: >>> local.delete_dir(/tmp/other_dir)

normalize_path(self, path)#

Normalize filesystem path.

Parameters:

pathstr: The path to normalize

Returns:

normalized_pathstr: The normalized path

open_append_stream(self, path, compression='detect', buffer_size=None, metadata=None)#

Open an output stream for appending.

If the target doesnt exist, a new empty file is created.

Note

Some filesystem implementations do not support efficient appending to an existing file, in which case this method will raise NotImplementedError. Consider writing to multiple files (using e.g. the dataset layer) instead.

Parameters:

pathstr: The source to open for writing.
compressionstr optional, default detect: The compression algorithm to use for on-the-fly compression. If detect and source is a file path, then compression will be chosen based on the file extension. If None, no compression will be applied. Otherwise, a well-known algorithm name must be supplied (e.g. gzip).
buffer_sizeint optional, default None: If None or 0, no buffering will happen. Otherwise the size of the temporary write buffer.
metadatadict optional, default None: If not None, a mapping of string keys to string values. Some filesystems support storing metadata along the file (such as Content-Type). Unsupported metadata keys will be ignored.

Returns:

streamNativeFile

Examples

Append new data to a FileSystem subclass with nonempty file:

>>> with local.open_append_stream(path) as f:
...     f.write(b'+newly added')
12

Print out the content to the file:

>>> with local.open_input_file(path) as f:
...     print(f.readall())
b'data+newly added'

open_input_file(self, path)#

Open an input file for random access reading.

Parameters:

pathstr: The source to open for reading.

Returns:

streamNativeFile

Examples

Print the data from the file with open_input_file():

>>> with local.open_input_file(path) as f:
...     print(f.readall())
b'data'

open_input_stream(self, path, compression='detect', buffer_size=None)#

Open an input stream for sequential reading.

Parameters:

pathstr: The source to open for reading.
compressionstr optional, default detect: The compression algorithm to use for on-the-fly decompression. If detect and source is a file path, then compression will be chosen based on the file extension. If None, no compression will be applied. Otherwise, a well-known algorithm name must be supplied (e.g. gzip).
buffer_sizeint optional, default None: If None or 0, no buffering will happen. Otherwise the size of the temporary read buffer.

Returns:

streamNativeFile

Examples

Print the data from the file with open_input_stream():

>>> with local.open_input_stream(path) as f:
...     print(f.readall())
b'data'

open_output_stream(self, path, compression='detect', buffer_size=None, metadata=None)#

Open an output stream for sequential writing.

If the target already exists, existing data is truncated.

Parameters:

pathstr: The source to open for writing.
compressionstr optional, default detect: The compression algorithm to use for on-the-fly compression. If detect and source is a file path, then compression will be chosen based on the file extension. If None, no compression will be applied. Otherwise, a well-known algorithm name must be supplied (e.g. gzip).
buffer_sizeint optional, default None: If None or 0, no buffering will happen. Otherwise the size of the temporary write buffer.
metadatadict optional, default None: If not None, a mapping of string keys to string values. Some filesystems support storing metadata along the file (such as Content-Type). Unsupported metadata keys will be ignored.

Returns:

streamNativeFile

Examples

>>> local = fs.LocalFileSystem()
>>> with local.open_output_stream(path) as stream:
...     stream.write(b'data')
4

type_name#: The filesystems type name.

pyarrow.fs.SubTreeFileSystem

pyarrow.fs.PyFileSystem