duckdb::HivePartitionedColumnData Class Reference

Inheritance diagram for duckdb::HivePartitionedColumnData:

Collaboration diagram for duckdb::HivePartitionedColumnData:

Public Member Functions
	HivePartitionedColumnData (ClientContext &context, vector< LogicalType > types, vector< idx_t > partition_by_cols, shared_ptr< GlobalHivePartitionState > global_state=nullptr)
void	ComputePartitionIndices (PartitionedColumnDataAppendState &state, DataChunk &input) override
std::map< idx_t, const HivePartitionKey * >	GetReverseMap ()
	Reverse lookup map to reconstruct keys from a partition id.
Public Member Functions inherited from duckdb::PartitionedColumnData
unique_ptr< PartitionedColumnData >	CreateShared ()
void	InitializeAppendState (PartitionedColumnDataAppendState &state) const
	Initializes a local state for parallel partitioning that can be merged into this PartitionedColumnData.
void	Append (PartitionedColumnDataAppendState &state, DataChunk &input)
	Appends a DataChunk to this PartitionedColumnData.
void	FlushAppendState (PartitionedColumnDataAppendState &state)
	Flushes any remaining data in the append state into this PartitionedColumnData.
void	Combine (PartitionedColumnData &other)
	Combine another PartitionedColumnData into this PartitionedColumnData.
vector< unique_ptr< ColumnDataCollection > > &	GetPartitions ()
	Get the partitions in this PartitionedColumnData.
template<class TARGET >
TARGET &	Cast ()
template<class TARGET >
const TARGET &	Cast () const

Protected Member Functions
idx_t	RegisterNewPartition (HivePartitionKey key, PartitionedColumnDataAppendState &state)
	Register a newly discovered partition.
void	AddNewPartition (HivePartitionKey key, idx_t partition_id, PartitionedColumnDataAppendState &state)
	Add a new partition with the given partition id.
Protected Member Functions inherited from duckdb::PartitionedColumnData
virtual idx_t	BufferSize () const
	Size of the buffers in the append states for this type of partitioning (default 128)
virtual void	InitializeAppendStateInternal (PartitionedColumnDataAppendState &state) const
	Initialize a PartitionedColumnDataAppendState for this type of partitioning (optional)
virtual idx_t	MaxPartitionIndex () const
	Maximum partition index (optional)
	PartitionedColumnData (PartitionedColumnDataType type, ClientContext &context, vector< LogicalType > types)
	PartitionedColumnData can only be instantiated by derived classes.
	PartitionedColumnData (const PartitionedColumnData &other)
idx_t	HalfBufferSize () const
	If the buffer is half full, we append to the partition.
void	CreateAllocator ()
	Create a new shared allocator.
bool	UseFixedSizeMap () const
	Whether to use fixed size map or regular map.
void	BuildPartitionSel (PartitionedColumnDataAppendState &state, const idx_t append_count) const
template<bool fixed>
void	AppendInternal (PartitionedColumnDataAppendState &state, DataChunk &input)
	Appends a DataChunk to this PartitionedColumnData.
unique_ptr< ColumnDataCollection >	CreatePartitionCollection (idx_t partition_index) const
	Create a collection for a specific a partition.
unique_ptr< DataChunk >	CreatePartitionBuffer () const
	Create a DataChunk used for buffering appends to the partition.

Protected Attributes
shared_ptr< GlobalHivePartitionState >	global_state
	Shared HivePartitionedColumnData should always have a global state to allow parallel key discovery.
hive_partition_map_t	local_partition_map
	Thread-local copy of the partition map.
vector< idx_t >	group_by_columns
	The columns that make up the key.
Vector	hashes_v
	Thread-local pre-allocated vector for hashes.
vector< HivePartitionKey >	keys
	Thread-local pre-allocated HivePartitionKeys.
Protected Attributes inherited from duckdb::PartitionedColumnData
PartitionedColumnDataType	type
ClientContext &	context
vector< LogicalType >	types
mutex	lock
shared_ptr< PartitionColumnDataAllocators >	allocators
vector< unique_ptr< ColumnDataCollection > >	partitions

Private Member Functions
void	InitializeKeys ()

Additional Inherited Members
Static Protected Member Functions inherited from duckdb::PartitionedColumnData
template<bool fixed>
static void	BuildPartitionSel (PartitionedColumnDataAppendState &state, const idx_t append_count)

Constructor & Destructor Documentation

HivePartitionedColumnData()

duckdb::HivePartitionedColumnData::HivePartitionedColumnData	(	ClientContext &	context,
		vector< LogicalType >	types,
		vector< idx_t >	partition_by_cols,
		shared_ptr< GlobalHivePartitionState >	global_state = nullptr
	)

    : PartitionedColumnData(PartitionedColumnDataType::HIVE, context, std::move(types)),
      global_state(std::move(global_state)), group_by_columns(std::move(partition_by_cols)),
      hashes_v(LogicalType::HASH) {
    InitializeKeys();
    CreateAllocator();
}

Member Function Documentation

ComputePartitionIndices()

void duckdb::HivePartitionedColumnData::ComputePartitionIndices ( PartitionedColumnDataAppendState &  state, DataChunk &  input  )

overridevirtual

Compute the partition indices for this type of partitioning for the input DataChunk and store them in the partition_data of the local state. If this type creates partitions on the fly (for, e.g., hive), this function is also in charge of creating new partitions and mapping the input data to a partition index

Reimplemented from duckdb::PartitionedColumnData.

                                                                                                                 {
    const auto count = input.size();
 
    input.Hash(group_by_columns, hashes_v);
    hashes_v.Flatten(count);
 
    for (idx_t col_idx = 0; col_idx < group_by_columns.size(); col_idx++) {
        auto &group_by_col = input.data[group_by_columns[col_idx]];
        GetHivePartitionValuesTypeSwitch(group_by_col, keys, col_idx, count);
    }
 
    const auto hashes = FlatVector::GetData<hash_t>(hashes_v);
    const auto partition_indices = FlatVector::GetData<idx_t>(state.partition_indices);
    for (idx_t i = 0; i < count; i++) {
        auto &key = keys[i];
        key.hash = hashes[i];
        auto lookup = local_partition_map.find(key);
        if (lookup == local_partition_map.end()) {
            idx_t new_partition_id = RegisterNewPartition(key, state);
            partition_indices[i] = new_partition_id;
        } else {
            partition_indices[i] = lookup->second;
        }
    }
}

Here is the call graph for this function:

GetReverseMap()

std::map< idx_t, const HivePartitionKey * > duckdb::HivePartitionedColumnData::GetReverseMap

(

)

Reverse lookup map to reconstruct keys from a partition id.

                                                                                 {
    std::map<idx_t, const HivePartitionKey *> ret;
    for (const auto &pair : local_partition_map) {
        ret[pair.second] = &(pair.first);
    }
    return ret;
}

RegisterNewPartition()

idx_t duckdb::HivePartitionedColumnData::RegisterNewPartition ( HivePartitionKey  key, PartitionedColumnDataAppendState &  state  )

protected

Register a newly discovered partition.

                                                                                                                   {
    idx_t partition_id;
    if (global_state) {
        // Synchronize Global state with our local state with the newly discovered partition
        unique_lock<mutex> lck_gstate(global_state->lock);
 
        // Insert into global map, or return partition if already present
        auto res = global_state->partition_map.emplace(std::make_pair(key, global_state->partition_map.size()));
        partition_id = res.first->second;
    } else {
        partition_id = local_partition_map.size();
    }
    AddNewPartition(std::move(key), partition_id, state);
    return partition_id;
}

Here is the call graph for this function:

Here is the caller graph for this function:

AddNewPartition()

void duckdb::HivePartitionedColumnData::AddNewPartition ( HivePartitionKey  key, idx_t  partition_id, PartitionedColumnDataAppendState &  state  )

protected

Add a new partition with the given partition id.

                                                                                         {
    local_partition_map.emplace(std::move(key), partition_id);
 
    if (state.partition_append_states.size() <= partition_id) {
        state.partition_append_states.resize(partition_id + 1);
        state.partition_buffers.resize(partition_id + 1);
        partitions.resize(partition_id + 1);
    }
    state.partition_append_states[partition_id] = make_uniq<ColumnDataAppendState>();
    state.partition_buffers[partition_id] = CreatePartitionBuffer();
    partitions[partition_id] = CreatePartitionCollection(0);
    partitions[partition_id]->InitializeAppend(*state.partition_append_states[partition_id]);
}

Here is the call graph for this function:

Here is the caller graph for this function:

InitializeKeys()

void duckdb::HivePartitionedColumnData::InitializeKeys ( )

private

                                               {
    keys.resize(STANDARD_VECTOR_SIZE);
    for (idx_t i = 0; i < STANDARD_VECTOR_SIZE; i++) {
        keys[i].values.resize(group_by_columns.size());
    }
}

---

The documentation for this class was generated from the following files:

• external/duckdb/duckdb.hpp
• external/duckdb/duckdb.cpp