Section Navigation

Dataset#

Factory functions#

`dataset`(source[,schema,format,...])	Open a dataset.
`parquet_dataset`(metadata_path[,schema,...])	Create a FileSystemDataset from a _metadata file created via pyarrow.parquet.write_metadata.
`partitioning`([schema,field_names,flavor,...])	Specify a partitioning scheme.
`field`(*name_or_index)	Reference a column of the dataset.
`scalar`(value)	Expression representing a scalar value.
`write_dataset`(data,base_dir,*[,...])	Write a dataset to a given format and partitioning.

Classes#

`FileFormat`()
`CsvFileFormat`(ParseOptionsparse_options=None)	FileFormat for CSV files.
`CsvFragmentScanOptions`(...)	Scan-specific options for CSV fragments.
`IpcFileFormat`()
`JsonFileFormat`([default_fragment_scan_options])	FileFormat for JSON files.
`ParquetFileFormat`([read_options,...])	FileFormat for Parquet
`ParquetReadOptions`([dictionary_columns,...])	Parquet format specific options for reading.
`ParquetFragmentScanOptions`(...[,...])	Scan-specific options for Parquet fragments.
`ParquetFileFragment`	A Fragment representing a parquet file.
`OrcFileFormat`()
`Partitioning`()
`PartitioningFactory`()
`DirectoryPartitioning`(Schemaschema[,...])	A Partitioning based on a specified Schema.
`HivePartitioning`(Schemaschema[,...])	A Partitioning for "/$key=$value/" nested directories as found in Apache Hive.
`FilenamePartitioning`(Schemaschema[,...])	A Partitioning based on a specified Schema.
`Dataset`()	Collection of data fragments and potentially child datasets.
`FileSystemDataset`(fragments,Schemaschema,...)	A Dataset of file fragments.
`FileSystemFactoryOptions`([...])	Influences the discovery of filesystem paths.
`FileSystemDatasetFactory`(...)	Create a DatasetFactory from a list of paths with schema inspection.
`UnionDataset`(Schemaschema,children)	A Dataset wrapping child datasets.
`Fragment`()	Fragment of data from a Dataset.
`FragmentScanOptions`()	Scan options specific to a particular fragment and scan operation.
`TaggedRecordBatch`(record_batch,fragment)	A combination of a record batch and the fragment it came from.
`Scanner`()	A materialized scan operation with context and options bound.
`Expression`()	A logical expression to be evaluated against some input.
`InMemoryDataset`(source,Schemaschema=None)	A Dataset wrapping in-memory data.
`WrittenFile`(path,metadata,size)	Metadata information about files written as part of a dataset write operation

Helper functions#

get_partition_keys(...)

Extract partition keys (equality constraints between a field and a scalar) from an expression as a dict mapping the field's name to its value.

pyarrow.fs.S3LogLevel

pyarrow.dataset.dataset