org.apache.spark.sql.sources.v2.reader

Interface SupportsScanUnsafeRow

All Superinterfaces:

DataSourceV2Reader

@InterfaceStability.Unstable
public interface SupportsScanUnsafeRow
extends DataSourceV2Reader

A mix-in interface for DataSourceV2Reader. Data source readers can implement this interface to output UnsafeRow directly and avoid the row copy at Spark side. This is an experimental and unstable interface, as UnsafeRow is not public and may get changed in the future Spark versions.

Method Summary

Modifier and Type	Method and Description
default java.util.List<ReadTask<Row>>	createReadTasks() Returns a list of read tasks.
java.util.List<ReadTask<org.apache.spark.sql.catalyst.expressions.UnsafeRow>>	createUnsafeRowReadTasks() Similar to DataSourceV2Reader.createReadTasks(), but returns data in unsafe row format.

Methods inherited from interface org.apache.spark.sql.sources.v2.reader.DataSourceV2Reader

readSchema

Method Detail

createReadTasks

default java.util.List<ReadTask<Row>> createReadTasks()

Description copied from interface: DataSourceV2Reader

Returns a list of read tasks. Each task is responsible for outputting data for one RDD partition. That means the number of tasks returned here is same as the number of RDD partitions this scan outputs. Note that, this may not be a full scan if the data source reader mixes in other optimization interfaces like column pruning, filter push-down, etc. These optimizations are applied before Spark issues the scan request. If this method fails (by throwing an exception), the action would fail and no Spark job was submitted.

Specified by:

createReadTasks in interface DataSourceV2Reader

createUnsafeRowReadTasks

java.util.List<ReadTask<org.apache.spark.sql.catalyst.expressions.UnsafeRow>> createUnsafeRowReadTasks()

Similar to DataSourceV2Reader.createReadTasks(), but returns data in unsafe row format.