Section Navigation

User Guide#

High-Level Overview
- The physical layer
- The one-dimensional layer
- The two-dimensional layer
- The compute layer
- The IO layer
- The Inter-Process Communication (IPC) layer
- The file formats layer
- The devices layer
- The filesystem layer
Memory Management
- Buffers
- Memory Pools
- Devices
  - Device-Agnostic Programming
- Memory Profiling
Arrays
- Building an array
- Size Limitations and Recommendations
- Chunked Arrays
- Slicing
- FromJSONString Helpers
Data Types
- Creating data types
- Type Traits
- Visitor Pattern
Tabular Data
- Fields
- Schemas
- Tables
- Record Batches
Compute Functions
- The generic Compute API
- Implicit casts
  - Common numeric type
- Available functions
The Gandiva Expression Compiler
- Expression, Projector and Filter
- External Functions Development
  - Gandiva Expression, Projector, and Filter
    - Building Expressions
    - Projectors and Filters
  - Gandiva External Functions Development Guide
Acero: A C++ streaming execution engine
- Acero Overview
- Acero Users Guide
- Using Acero with Substrait
  - Substrait Conformance
Input / output and filesystems
- Reading binary data
- Writing binary data
- Filesystems
- Defining new filesystems
Reading and writing the Arrow IPC format
- Reading IPC streams and files
  - Synchronous reading
  - Event-driven reading
- Writing IPC streams and files
- Configuring
Reading and Writing ORC files
- Supported ORC features
  - Data types
  - Compression
- Reading ORC Files
  - ORCFileReader
- Writing ORC Files
  - ORCFileWriter
Reading and writing Parquet files
- Reading Parquet files
  - FileReader
    - Performance and Memory Efficiency
  - StreamReader
- Writing Parquet files
- Supported Parquet features
Reading and Writing CSV files
- Reading CSV files
- Writing CSV files
- Column names
- Column selection
  - Interaction with column names
- Data types
- Write Options
- Performance
Reading JSON files
- TableReader
- StreamingReader
- Data types
Tabular Datasets
- Reading Datasets
- Filtering data
- Projecting columns
- Reading and writing partitioned data
  - Different partitioning schemes
  - Partitioning performance considerations
- Reading from other data sources
  - Reading in-memory data
  - Reading from cloud storage
- A note on transactions & ACID guarantees
- Full Example
Arrow Flight RPC
- Writing a Flight Service
- Using the Flight Client
- Cancellation and Timeouts
- Enabling TLS
- Enabling Authentication
- Custom Middleware
- Best practices
Security Considerations
- API parameter validity
- Ingesting untrusted data
  - IPC and Parquet readers
  - CSV reader
Debugging code using Arrow
- GDB extension for Arrow C++
Thread Management
- Thread Pools
- Asynchronous Utilities
  - Future
OpenTelemetry
- Creating a Build with OpenTelemetry Tracing Enabled
- Exporting Tracing Information
- Visualizing Traces with Jaeger UI
Environment Variables

Arrow Datasets

High-Level Overview