Python Data Source#

`DataSource.name`()	Returns a string represents the format name of this data source.
`DataSource.reader`(schema)	Returns a `DataSourceReader` instance for reading data.
`DataSource.schema`()	Returns the schema of the data source.
`DataSource.streamReader`(schema)	Returns a `DataSourceStreamReader` instance for reading streaming data.
`DataSource.writer`(schema,overwrite)	Returns a `DataSourceWriter` instance for writing data.
`DataSourceReader.partitions`()	Returns an iterator of partitions for this data source.
`DataSourceReader.pushFilters`(filters)	Called with the list of filters that can be pushed down to the data source.
`DataSourceReader.read`(partition)	Generates data for a given partition and returns an iterator of tuples or rows.
`DataSourceRegistration.register`(dataSource)	Register a Python user-defined data source.
`DataSourceStreamReader.commit`(end)	Informs the source that Spark has completed processing all data for offsets less than or equal to end and will only request offsets greater than end in the future.
`DataSourceStreamReader.getDefaultReadLimit`()	Returns the read limits potentially passed to the data source through options when creating the data source.
`DataSourceStreamReader.initialOffset`()	Return the initial offset of the streaming data source.
`DataSourceStreamReader.latestOffset`(start,limit)	Returns the most recent offset available given a read limit.
`DataSourceStreamReader.partitions`(start,end)	Returns a list of InputPartition given the start and end offsets.
`DataSourceStreamReader.read`(partition)	Generates data for a given partition and returns an iterator of tuples or rows.
`DataSourceStreamReader.reportLatestOffset`()	Returns the most recent offset available.
`DataSourceStreamReader.stop`()	Stop this source and free any resources it has allocated.
`DataSourceWriter.abort`(messages)	Aborts this writing job due to task failures.
`DataSourceWriter.commit`(messages)	Commits this writing job with a list of commit messages.
`DataSourceWriter.write`(iterator)	Writes data into the data source.

pyspark.sql.protobuf.functions.to_protobuf

pyspark.sql.datasource.DataSource.name