Autonomy Software C++ 24.5.1
Welcome to the Autonomy Software repository of the Mars Rover Design Team (MRDT) at Missouri University of Science and Technology (Missouri S&T)! API reference contains the source code and other resources for the development of the autonomy software for our Mars rover. The Autonomy Software project aims to compete in the University Rover Challenge (URC) by demonstrating advanced autonomous capabilities and robust navigation algorithms.
Loading...
Searching...
No Matches
Public Member Functions | Static Public Member Functions | Static Public Attributes | Protected Attributes | List of all members
duckdb::MultiFileReader Struct Reference

The MultiFileReader class provides a set of helper methods to handle scanning from multiple files. More...

Collaboration diagram for duckdb::MultiFileReader:
Collaboration graph
[legend]

Public Member Functions

virtual DUCKDB_API vector< string > ParsePaths (const Value &input)
 Parse a Value containing 1 or more paths into a vector of paths. Note: no expansion is performed here.
 
virtual DUCKDB_API shared_ptr< MultiFileList > CreateFileList (ClientContext &context, const vector< string > &paths, const FileGlobInput &glob_input=FileGlobOptions::DISALLOW_EMPTY)
 Create a MultiFileList from a vector of paths. Any globs will be expanded using the default filesystem.
 
DUCKDB_API shared_ptr< MultiFileList > CreateFileList (ClientContext &context, const Value &input, const FileGlobInput &glob_input=FileGlobOptions::DISALLOW_EMPTY)
 Shorthand for ParsePaths + CreateFileList.
 
virtual DUCKDB_API bool ParseOption (const string &key, const Value &val, MultiFileOptions &options, ClientContext &context)
 Parse the named parameters of a multi-file reader.
 
virtual DUCKDB_API unique_ptr< MultiFileList > ComplexFilterPushdown (ClientContext &context, MultiFileList &files, const MultiFileOptions &options, MultiFilePushdownInfo &info, vector< unique_ptr< Expression > > &filters)
 Perform filter pushdown into the MultiFileList. Returns a new MultiFileList if filters were pushed down.
 
virtual DUCKDB_API unique_ptr< MultiFileList > DynamicFilterPushdown (ClientContext &context, const MultiFileList &files, const MultiFileOptions &options, const vector< string > &names, const vector< LogicalType > &types, const vector< column_t > &column_ids, TableFilterSet &filters)
 
virtual DUCKDB_API bool Bind (MultiFileOptions &options, MultiFileList &files, vector< LogicalType > &return_types, vector< string > &names, MultiFileReaderBindData &bind_data)
 
virtual DUCKDB_API void BindOptions (MultiFileOptions &options, MultiFileList &files, vector< LogicalType > &return_types, vector< string > &names, MultiFileReaderBindData &bind_data)
 Bind the options of the multi-file reader, potentially emitting any extra columns that are required.
 
virtual DUCKDB_API unique_ptr< MultiFileReaderGlobalState > InitializeGlobalState (ClientContext &context, const MultiFileOptions &file_options, const MultiFileReaderBindData &bind_data, const MultiFileList &file_list, const vector< MultiFileColumnDefinition > &global_columns, const vector< ColumnIndex > &global_column_ids)
 Initialize global state used by the MultiFileReader.
 
virtual DUCKDB_API void FinalizeBind (MultiFileReaderData &reader_data, const MultiFileOptions &file_options, const MultiFileReaderBindData &options, const vector< MultiFileColumnDefinition > &global_columns, const vector< ColumnIndex > &global_column_ids, ClientContext &context, optional_ptr< MultiFileReaderGlobalState > global_state)
 
virtual DUCKDB_API ReaderInitializeType CreateMapping (ClientContext &context, MultiFileReaderData &reader_data, const vector< MultiFileColumnDefinition > &global_columns, const vector< ColumnIndex > &global_column_ids, optional_ptr< TableFilterSet > filters, MultiFileList &multi_file_list, const MultiFileReaderBindData &bind_data, const virtual_column_map_t &virtual_columns, MultiFileColumnMappingMode mapping_mode)
 Create all required mappings from the global types/names to the file-local types/names.
 
virtual DUCKDB_API ReaderInitializeType CreateMapping (ClientContext &context, MultiFileReaderData &reader_data, const vector< MultiFileColumnDefinition > &global_columns, const vector< ColumnIndex > &global_column_ids, optional_ptr< TableFilterSet > filters, MultiFileList &multi_file_list, const MultiFileReaderBindData &bind_data, const virtual_column_map_t &virtual_columns)
 
virtual DUCKDB_API void FinalizeChunk (ClientContext &context, const MultiFileBindData &bind_data, BaseFileReader &reader, const MultiFileReaderData &reader_data, DataChunk &input_chunk, DataChunk &output_chunk, ExpressionExecutor &executor, optional_ptr< MultiFileReaderGlobalState > global_state)
 Finalize the reading of a chunk - applying any constants that are required.
 
virtual DUCKDB_API void GetPartitionData (ClientContext &context, const MultiFileReaderBindData &bind_data, const MultiFileReaderData &reader_data, optional_ptr< MultiFileReaderGlobalState > global_state, const OperatorPartitionInfo &partition_info, OperatorPartitionData &partition_data)
 Fetch the partition data for the current chunk.
 
MultiFileReaderBindData BindUnionReader (ClientContext &context, vector< LogicalType > &return_types, vector< string > &names, MultiFileList &files, MultiFileBindData &result, BaseFileReaderOptions &options, MultiFileOptions &file_options)
 
MultiFileReaderBindData BindReader (ClientContext &context, vector< LogicalType > &return_types, vector< string > &names, MultiFileList &files, MultiFileBindData &result, BaseFileReaderOptions &options, MultiFileOptions &file_options)
 
virtual DUCKDB_API ReaderInitializeType InitializeReader (MultiFileReaderData &reader_data, const MultiFileBindData &bind_data, const vector< MultiFileColumnDefinition > &global_columns, const vector< ColumnIndex > &global_column_ids, optional_ptr< TableFilterSet > table_filters, ClientContext &context, MultiFileGlobalState &gstate)
 
virtual DUCKDB_API shared_ptr< BaseFileReader > CreateReader (ClientContext &context, GlobalTableFunctionState &gstate, BaseUnionData &union_data, const MultiFileBindData &bind_data)
 
virtual DUCKDB_API shared_ptr< BaseFileReader > CreateReader (ClientContext &context, GlobalTableFunctionState &gstate, const OpenFileInfo &file, idx_t file_idx, const MultiFileBindData &bind_data)
 
virtual DUCKDB_API shared_ptr< BaseFileReader > CreateReader (ClientContext &context, const OpenFileInfo &file, BaseFileReaderOptions &options, const MultiFileOptions &file_options, MultiFileReaderInterface &interface)
 
virtual DUCKDB_API TablePartitionInfo GetPartitionInfo (ClientContext &context, const MultiFileReaderBindData &bind_data, TableFunctionPartitionInput &input)
 Get partition info.
 
virtual DUCKDB_API unique_ptr< Expression > GetConstantVirtualColumn (MultiFileReaderData &reader_data, idx_t column_id, const LogicalType &type)
 
virtual DUCKDB_API unique_ptr< Expression > GetVirtualColumnExpression (ClientContext &context, MultiFileReaderData &reader_data, const vector< MultiFileColumnDefinition > &local_columns, idx_t &column_id, const LogicalType &type, MultiFileLocalIndex local_index, optional_ptr< MultiFileColumnDefinition > &global_column_reference)
 Gets an expression to evaluate the given virtual column.
 
virtual DUCKDB_API unique_ptr< MultiFileReader > Copy () const
 
virtual DUCKDB_API FileGlobInput GetGlobInput (MultiFileReaderInterface &interface)
 
template<class TARGET >
TARGET & Cast ()
 
template<class TARGET >
const TARGET & Cast () const
 

Static Public Member Functions

static DUCKDB_API unique_ptr< MultiFileReader > Create (const TableFunction &table_function)
 Create a MultiFileReader for a specific TableFunction, using its function name for errors.
 
static DUCKDB_API unique_ptr< MultiFileReader > CreateDefault (const string &function_name="")
 Create a default MultiFileReader, function_name is used for errors.
 
static Value CreateValueFromFileList (const vector< string > &files)
 Create a LIST Value from a vector of strings (list of file paths)
 
static DUCKDB_API void AddParameters (TableFunction &table_function)
 Add the parameters for multi-file readers (e.g. union_by_name, filename) to a table function.
 
static DUCKDB_API TableFunctionSet CreateFunctionSet (TableFunction table_function)
 Creates a table function set from a single reader function (including e.g. list parameters, etc)
 
static DUCKDB_API void GetVirtualColumns (ClientContext &context, MultiFileReaderBindData &bind_data, virtual_column_map_t &result)
 
static void PruneReaders (MultiFileBindData &data, MultiFileList &file_list)
 

Static Public Attributes

static constexpr column_t COLUMN_IDENTIFIER_FILENAME = UINT64_C(9223372036854775808)
 
static constexpr column_t COLUMN_IDENTIFIER_FILE_ROW_NUMBER = UINT64_C(9223372036854775809)
 
static constexpr column_t COLUMN_IDENTIFIER_FILE_INDEX = UINT64_C(9223372036854775810)
 
static constexpr int32_t ORDINAL_FIELD_ID = 2147483645
 
static constexpr int32_t FILENAME_FIELD_ID = 2147483646
 
static constexpr int32_t DELETE_FILE_PATH_FIELD_ID = 2147483546
 
static constexpr int32_t DELETE_POS_FIELD_ID = 2147483545
 
static constexpr int32_t ROW_ID_FIELD_ID = 2147483540
 
static constexpr int32_t LAST_UPDATED_SEQUENCE_NUMBER_ID = 2147483539
 

Protected Attributes

string function_name
 Used in errors to report which function is using this MultiFileReader.
 

Detailed Description

The MultiFileReader class provides a set of helper methods to handle scanning from multiple files.

Member Function Documentation

◆ Bind()

virtual DUCKDB_API bool duckdb::MultiFileReader::Bind ( MultiFileOptions &  options,
MultiFileList &  files,
vector< LogicalType > &  return_types,
vector< string > &  names,
MultiFileReaderBindData &  bind_data 
)
virtual

Try to use the MultiFileReader for binding. Returns true if a bind could be made, returns false if the MultiFileReader can not perform the bind and binding should be performed on 1 or more files in the MultiFileList directly.

◆ FinalizeBind()

virtual DUCKDB_API void duckdb::MultiFileReader::FinalizeBind ( MultiFileReaderData &  reader_data,
const MultiFileOptions &  file_options,
const MultiFileReaderBindData &  options,
const vector< MultiFileColumnDefinition > &  global_columns,
const vector< ColumnIndex > &  global_column_ids,
ClientContext &  context,
optional_ptr< MultiFileReaderGlobalState >  global_state 
)
virtual

Finalize the bind phase of the multi-file reader after we know (1) the required (output) columns, and (2) the pushed down table filters

◆ GetConstantVirtualColumn()

virtual DUCKDB_API unique_ptr< Expression > duckdb::MultiFileReader::GetConstantVirtualColumn ( MultiFileReaderData &  reader_data,
idx_t  column_id,
const LogicalType &  type 
)
virtual

Get a constant expression for the given virtual column, if this can be handled by the MultiFileReader This expression is constant for the entire file

◆ Cast() [1/2]

template<class TARGET >
TARGET & duckdb::MultiFileReader::Cast ( )
inline
80819 {
80820 DynamicCastCheck<TARGET>(this);
80821 return reinterpret_cast<TARGET &>(*this);
80822 }

◆ Cast() [2/2]

template<class TARGET >
const TARGET & duckdb::MultiFileReader::Cast ( ) const
inline
80824 {
80825 DynamicCastCheck<TARGET>(this);
80826 return reinterpret_cast<const TARGET &>(*this);
80827 }
The documentation for this struct was generated from the following file: