pyarrow.Codec#

class pyarrow.Codec(str compression, compression_level=None)#

Bases: _Weakrefable

Compression codec.

Parameters:

compressionstr

Type of compression codec to initialize, valid values are: gzip, bz2, brotli, lz4 (or lz4_frame), lz4_raw, zstd and snappy.

compression_levelint, None

Optional parameter specifying how aggressively to compress. The possible ranges and effect of this parameter depend on the specific codec chosen. Higher values compress more but typically use more resources (CPU/RAM). Some codecs support negative values.

gzip

The compression_level maps to the memlevel parameter of deflateInit2. Higher levels use more RAM but are faster and should have higher compression ratios.

bz2

The compression level maps to the blockSize100k parameter of the BZ2_bzCompressInit function. Higher levels use more RAM but are faster and should have higher compression ratios.

brotli

The compression level maps to the BROTLI_PARAM_QUALITY parameter. Higher values are slower and should have higher compression ratios.

lz4/lz4_frame/lz4_raw

The compression level parameter is not supported and must be None

zstd

The compression level maps to the compressionLevel parameter of ZSTD_initCStream. Negative values are supported. Higher values are slower and should have higher compression ratios.

snappy

The compression level parameter is not supported and must be None

Raises:

ValueError

If invalid compression value is passed.

Examples

>>> import pyarrow as pa
>>> pa.Codec.is_available('gzip')
True
>>> codec = pa.Codec('gzip')
>>> codec.name
'gzip'
>>> codec.compression_level
9

__init__(*args, **kwargs)#

Methods

__init__(*args,**kwargs)
compress(self,buf[,asbytes,memory_pool])	Compress data from buffer-like object.
decompress(self,buf[,decompressed_size,...])	Decompress data from buffer-like object.
default_compression_level(strcompression)	Returns the compression level that Arrow will use for the codec if None is specified.
detect(path)	Detect and instantiate compression codec based on file extension.
is_available(strcompression)	Returns whether the compression support has been built and enabled.
maximum_compression_level(strcompression)	Returns the largest valid value for the compression level
minimum_compression_level(strcompression)	Returns the smallest valid value for the compression level
supports_compression_level(strcompression)	Returns true if the compression level parameter is supported for the given codec.

Attributes

compression_level	Returns the compression level parameter of the codec
name	Returns the name of the codec

compress(self, buf, asbytes=False, memory_pool=None)#

Compress data from buffer-like object.

Parameters:

bufpyarrow.Buffer, bytes, or other object supporting buffer protocol

asbytesbool, default False

Return result as Python bytes object, otherwise Buffer

memory_poolMemoryPool, default None

Memory pool to use for buffer allocations, if any

Returns:

compressedpyarrow.Buffer or bytes (if asbytes=True)

compression_level#

Returns the compression level parameter of the codec

decompress(self, buf, decompressed_size=None, asbytes=False, memory_pool=None)#

Decompress data from buffer-like object.

Parameters:

bufpyarrow.Buffer, bytes, or memoryview-compatible object

decompressed_sizeint, default None

Size of the decompressed result

asbytesbool, default False

Return result as Python bytes object, otherwise Buffer

memory_poolMemoryPool, default None

Memory pool to use for buffer allocations, if any.

Returns:

uncompressedpyarrow.Buffer or bytes (if asbytes=True)

static default_compression_level(str compression)#

Returns the compression level that Arrow will use for the codec if None is specified.

Parameters:

compressionstr

Type of compression codec, refer to Codec docstring for a list of supported ones.

static detect(path)#

Detect and instantiate compression codec based on file extension.

Parameters:

pathstr, path-like

File-path to detect compression from.

Returns:

Codec

Raises:

TypeError

If the passed value is not path-like.

ValueError

If the compression cant be detected from the path.

static is_available(str compression)#

Returns whether the compression support has been built and enabled.

Parameters:

compressionstr

Type of compression codec, refer to Codec docstring for a list of supported ones.

Returns:

bool

static maximum_compression_level(str compression)#

Returns the largest valid value for the compression level

Parameters:

compressionstr

Type of compression codec, refer to Codec docstring for a list of supported ones.

static minimum_compression_level(str compression)#

Returns the smallest valid value for the compression level

Parameters:

compressionstr

Type of compression codec, refer to Codec docstring for a list of supported ones.

name#

Returns the name of the codec

static supports_compression_level(str compression)#

Returns true if the compression level parameter is supported for the given codec.

Parameters:

compressionstr

Type of compression codec, refer to Codec docstring for a list of supported ones.