Class star::StarDataset¶

ClassList > star > StarDataset

A cloud-optimized binary key-value store for serializable data types. More...

#include <stards.h>

Inherits the following classes: std::enable_shared_from_this< StarDataset >

Public Types¶

Type	Name
typedef std::vector< std::string >::const_iterator	const_iterator
typedef std::vector< std::string >::iterator	iterator

Public Attributes¶

Type	Name
std::vector< char > *	m_capture_image = `nullptr`
ColdStorage	m_cold
std::vector< char >	m_compress_buffer
StarConfig	m_config
std::vector< ValueVariant >	m_data_storage
FileHeader	m_file_header
FileMode	m_file_mode
std::string	m_filename
bool	m_flushed = `false`
bool	m_header_dirty = `false`
size_t	m_header_size = `0`
HotStorage	m_hot
KeyRegistry	m_key_registry
std::unordered_map< std::string, size_t >	m_key_to_index
std::vector< std::unordered_map< uint16_t, size_t > >	m_layer_metadata_indices
std::vector< bool >	m_layer_metadata_loaded
LayerMetadataRegistry	m_layer_metadata_registry
std::map< std::string, std::vector< uint64_t > >	m_layer_presence
std::vector< char >	m_memory_source
std::unordered_map< size_t, DataType >	m_metadata_dtypes
bool	m_metadata_loaded = `false`
std::unordered_map< size_t, std::vector< size_t > >	m_metadata_shapes
std::shared_mutex	m_mutex
OpenOptions	m_open_options
FilePathInfo	m_path_info
std::unique_ptr< RangeReader >	m_reader
std::unique_ptr< S3Credentials >	m_s3_credentials
std::vector< char >	m_serialize_buffer
std::unique_ptr< ThreadPool >	m_thread_pool
MetadataAccessor	meta

Public Functions¶

Type	Name
	StarDataset (const std::string & fname, FileMode mode, const StarConfig * config, const OpenOptions & open_options={}, std::vector< char > * memory_source=nullptr) Constructor with compression options.
	StarDataset (const StarDataset &) = delete
	StarDataset (StarDataset &&) = delete
size_t	array_length (const std::string & key) const Length of a stored array's first (outermost) dimension.
iterator	begin () Returns an iterator to the beginning of the index.
const_iterator	begin () const Returns a const iterator to the beginning of the index.
size_t	calculateHeaderSize () Calculates the size of the header based on current index.
const_iterator	cbegin () const Returns a const iterator to the beginning of the index.
const_iterator	cend () const Returns a const iterator to the end of the index.
void	close () Flush all pending writes to disk.
std::vector< char >	compress_single_block (const std::vector< char > & uncompressed, CompressionAlgorithm compression) Compress a single block (helper for layer metadata).
bool	contains (const std::string & key) const Check whether a key exists in the dataset.
std::shared_ptr< LayerView >	create_layer (const std::string & layer_name) Create new layer and return view.
ValueVariant	decode_array_bytes (const std::vector< char > & compressed_data, const std::vector< BlockInfo > & blocks, CompressionAlgorithm compression, DataType dtype, const std::vector< size_t > & shape, ThreadPool * thread_pool)
ValueVariant	decode_numeric_blocks_into_ndarray (const std::vector< char > & compressed_data, const std::vector< BlockInfo > & blocks, CompressionAlgorithm compression, DataType dtype, const std::vector< size_t > & shape, ThreadPool * thread_pool) Decode one array entry's compressed bytes into a ValueVariant.
std::vector< char >	decompress_single_block (const std::vector< char > & compressed, CompressionAlgorithm compression) Decompress a single block (helper for layer metadata).
std::string	deserializeKey (std::istream & is) Deserializes a key from the input stream.
ValueVariant	deserialize_typed_value (std::istream & is, DataType dtype, const std::vector< size_t > & shape, size_t data_len) Deserializes typed value from stream.
ValueVariant	deserialize_typed_value_bytes (const char * data, size_t data_len, DataType dtype, const std::vector< size_t > & shape, const ByteUnshuffleSpec & unshuffle={}) Zero-stream variant of deserialize_typed_value() for the array load hot path: deserialize directly from a contiguous byte buffer.
DataType	dtype_of (const std::string & key) const Get the element data type of a stored array.
iterator	end () Returns an iterator to the end of the index.
const_iterator	end () const Returns a const iterator to the end of the index.
void	ensure_layer_metadata_loaded (size_t layer_idx) Loads metadata block from file if not already loaded (supports HTTP/remote URLs).
void	flush () Writes all data to the file (including metadata block).
void	flush_quiet () Flush without throwing on read-only (silently skips instead).
NDArray< T >	get (const std::string & key) Get array from storage.
std::vector< std::string >	get_all_keys () Gets all keys (metadata block + separate arrays).
const FileHeader &	get_file_header () const Get file header with version information.
std::string	get_filename () const Get current filename.
std::shared_ptr< LayerView >	get_layer (const std::string & layer_name) Get existing layer view.
size_t	get_metadata_count () const Gets count of entries in metadata block.
std::vector< std::string >	get_metadata_keys () const Gets list of keys in metadata block.
NDArray< T >	get_slice (const std::string & key, const std::vector< Slice > & slices) Gets n-dimensional slice of an array.
bool	has_layer (const std::string & layer_name) const Check if layer exists.
bool	has_metadata_block () const Checks if file has metadata block.
bool	is_metadata_loaded () const Checks if metadata block has been loaded.
bool	is_read_only () const Check if file is in read-only mode.
bool	is_sliceable (const std::string & key) const Checks if an array supports slicing.
bool	key_in_layer (const std::string & key, const std::string & layer_name) const Check if key exists in specific layer using bit-mask (O(1)).
bool	layer_inheritance () const
std::vector< std::string >	list_layers () const Get list of all layer names.
void	loadIndex () Loads the index from the file.
void	load_all_metadata () Load all layer metadata blocks at once (v3 format).
void	load_entry (size_t idx) Load entry from disk into memory.
void	load_layer_metadata (const std::string & layer_name) Load a specific layer's metadata block (v3 format).
void	load_metadata_block ()
const OpenOptions &	open_options () const Read-time options (e.g. layer inheritance) for this dataset.
StarDataset &	operator= (const StarDataset &) = delete
StarDataset &	operator= (StarDataset &&) = delete
void	parse_layer_metadata_block (size_t layer_idx, const std::vector< char > & decompressed) Parse a layer's metadata block (STARMeta format).
void	prefetch (const std::vector< std::string > & keys) Load several arrays into cache in parallel (a read-ahead hint).
void	print_header () const
void	put (const std::string & key, NDArray< T > && value) Store array as separate compressed array (not in metadata block).
void	put (const std::string & key, const NDArray< T > & value) Store array as separate compressed array (const lvalue overload).
std::vector< char >	read_range (size_t position, size_t len) Read exactly [position, position+len) from the backing file via the persistent reader. Returns the bytes actually read.
RangeReader &	reader () Return the dataset's persistent byte-range reader, creating it on first use. One reader (= one reused connection for remote files) is shared by every read path — the index load, layer-metadata loads, and array-block reads — so remote reads never open a new connection or issue a HEAD per read.
void	save_to (const std::string & target_path)
std::vector< char >	serialize_array_data (const std::vector< T > & data) const Serialize an NDArray 's element data into a byte buffer for block storage, in the SAME wire formatdeserialize_typed_value() reads.
std::vector< char >	serialize_array_data (const std::vector< T > & data, CompressionAlgorithm codec, size_t block_size) const Serialize array data, applying the byte-shuffle prefilter if the codec is a _SHUFFLE variant.*
void	serialize_layer_metadata_block (std::ostream & os, const std::string & layer_name) Serializes a single layer's metadata in STARMeta format (v1).
void	serialize_metadata_block (std::ostream & os) const Serializes metadata block to stream (OLD FORMAT - deprecated).
std::enable_if< std::is_same< T, std::string >::value, void >::type	serialize_metadata_value (std::ostream & os, const std::vector< T > & data) const Helper to serialize metadata value (string specialization).
std::enable_if<!std::is_same< T, std::string >::value, void >::type	serialize_metadata_value (std::ostream & os, const std::vector< T > & data) const Helper to serialize metadata value (numeric types).
void	set_layer_inheritance (bool on)
void	set_layer_presence (const std::string & layer_name, const std::string & key, bool present) Set layer presence for a data array key.
void	set_open_options (const OpenOptions & opts)
std::vector< size_t >	shape_of (const std::string & key) const Full shape (all dimensions) of a stored data array.
size_t	size () const Removes a key-value pair from the store.
bool	useThreading (size_t num_blocks, size_t data_size) const Check if threading should be used based on workload.
std::vector< char >	write_bytes () Serialize the dataset to an in-memory byte buffer.
	~StarDataset () Destructor - writes all pending changes to disk (RAII).

Public Static Functions¶

Type	Name
std::shared_ptr< StarDataset >	create (const std::string & filename, const StarConfig & config=StarConfig()) Create a new Star dataset file.
std::shared_ptr< StarDataset >	open (const std::string & filename, FileMode mode=FileMode::READ_WRITE, const OpenOptions & opts={}) Open an existing Star dataset file.
std::shared_ptr< StarDataset >	open (const std::string & filename, const std::string & mode_str, const OpenOptions & opts={}) Open an existing Star dataset file (string mode overload).
std::shared_ptr< StarDataset >	open_bytes (std::vector< char > bytes, const OpenOptions & opts={}) Open a dataset from an in-memory byte buffer.
std::shared_ptr< StarDataset >	open_bytes (const void * data, size_t size, const OpenOptions & opts={}) Convenience overload taking a raw pointer + length (e.g. from a C buffer, Python bytes, or a mapped region).

Detailed Description¶

This class implements a binary key-value store that can persist data to disk in a cloud-optimized format. It uses a single file with an index section followed by data for efficient cloud storage and retrieval. Large arrays are chunked for better performance over networks.

Public Types Documentation¶

typedef const_iterator¶

using star::StarDataset::const_iterator = std::vector<std::string>::const_iterator;

typedef iterator¶

using star::StarDataset::iterator = std::vector<std::string>::iterator;

Public Attributes Documentation¶

variable m_capture_image¶

std::vector<char>* star::StarDataset::m_capture_image;

variable m_cold¶

ColdStorage star::StarDataset::m_cold;

variable m_compress_buffer¶

std::vector<char> star::StarDataset::m_compress_buffer;

variable m_config¶

StarConfig star::StarDataset::m_config;

variable m_data_storage¶

std::vector<ValueVariant> star::StarDataset::m_data_storage;

variable m_file_header¶

FileHeader star::StarDataset::m_file_header;

variable m_file_mode¶

FileMode star::StarDataset::m_file_mode;

variable m_filename¶

std::string star::StarDataset::m_filename;

variable m_flushed¶

bool star::StarDataset::m_flushed;

variable m_header_dirty¶

bool star::StarDataset::m_header_dirty;

variable m_header_size¶

size_t star::StarDataset::m_header_size;

variable m_hot¶

HotStorage star::StarDataset::m_hot;

variable m_key_registry¶

KeyRegistry star::StarDataset::m_key_registry;

variable m_key_to_index¶

std::unordered_map<std::string, size_t> star::StarDataset::m_key_to_index;

variable m_layer_metadata_indices¶

std::vector<std::unordered_map<uint16_t, size_t> > star::StarDataset::m_layer_metadata_indices;

variable m_layer_metadata_loaded¶

std::vector<bool> star::StarDataset::m_layer_metadata_loaded;

variable m_layer_metadata_registry¶

LayerMetadataRegistry star::StarDataset::m_layer_metadata_registry;

variable m_layer_presence¶

std::map<std::string, std::vector<uint64_t> > star::StarDataset::m_layer_presence;

variable m_memory_source¶

std::vector<char> star::StarDataset::m_memory_source;

variable m_metadata_dtypes¶

std::unordered_map<size_t, DataType> star::StarDataset::m_metadata_dtypes;

variable m_metadata_loaded¶

bool star::StarDataset::m_metadata_loaded;

variable m_metadata_shapes¶

std::unordered_map<size_t, std::vector<size_t> > star::StarDataset::m_metadata_shapes;

variable m_mutex¶

std::shared_mutex star::StarDataset::m_mutex;

variable m_open_options¶

OpenOptions star::StarDataset::m_open_options;

variable m_path_info¶

FilePathInfo star::StarDataset::m_path_info;

variable m_reader¶

std::unique_ptr<RangeReader> star::StarDataset::m_reader;

variable m_s3_credentials¶

std::unique_ptr<S3Credentials> star::StarDataset::m_s3_credentials;

variable m_serialize_buffer¶

std::vector<char> star::StarDataset::m_serialize_buffer;

variable m_thread_pool¶

std::unique_ptr<ThreadPool> star::StarDataset::m_thread_pool;

variable meta¶

MetadataAccessor star::StarDataset::meta;

Public Functions Documentation¶

function StarDataset [1/3]¶

Constructor with compression options.

inline star::StarDataset::StarDataset (
    const std::string & fname,
    FileMode mode,
    const StarConfig * config,
    const OpenOptions & open_options={},
    std::vector< char > * memory_source=nullptr
)

Parameters:

fname Filename to use for storage
mode File open mode (READ_WRITE or READ_ONLY)
config Optional configuration (nullptr to load from file)

NOTE: Constructor is public to enable std::make_shared, but you should use the static factory methods create() and open() instead.

function StarDataset [2/3]¶

star::StarDataset::StarDataset (
    const StarDataset &
) = delete

function StarDataset [3/3]¶

star::StarDataset::StarDataset (
    StarDataset &&
) = delete

function array_length¶

Length of a stored array's first (outermost) dimension.

inline size_t star::StarDataset::array_length (
    const std::string & key
) const

Read from the index that open() loads up front (m_cold.shapes) — NO block data is fetched, so this is cheap even for a multi-GB array or a remote (/vsicurl, /vsis3) source, and lets a caller size a streaming read (get_slice windows) before pulling any values. For a 1-D column this is the element count; for N-D it is shape[0] (the row count). Scalars report 1.

Parameters:

key Array key (must be a separately-stored data array, not metadata)

Returns:

Length of dimension 0

Exception:

std::runtime_error if the key is not a stored data array

function begin [1/2]¶

Returns an iterator to the beginning of the index.

inline iterator star::StarDataset::begin ()

Returns:

Iterator to the first key-value pair

function begin [2/2]¶

Returns a const iterator to the beginning of the index.

inline const_iterator star::StarDataset::begin () const

Returns:

Const iterator to the first key-value pair

function calculateHeaderSize¶

Calculates the size of the header based on current index.

inline size_t star::StarDataset::calculateHeaderSize ()

Returns:

Size of the header in bytes

function cbegin¶

Returns a const iterator to the beginning of the index.

inline const_iterator star::StarDataset::cbegin () const

Returns:

Const iterator to the first key-value pair

function cend¶

Returns a const iterator to the end of the index.

inline const_iterator star::StarDataset::cend () const

Returns:

Const iterator to one past the last key-value pair

function close¶

Flush all pending writes to disk.

inline void star::StarDataset::close ()

This is automatically called on context manager exit or object destruction. You can call this manually to ensure data is persisted without closing the dataset.

function compress_single_block¶

Compress a single block (helper for layer metadata).

inline std::vector< char > star::StarDataset::compress_single_block (
    const std::vector< char > & uncompressed,
    CompressionAlgorithm compression
)

Parameters:

uncompressed Uncompressed data
compression Compression algorithm

Returns:

Compressed data

function contains¶

Check whether a key exists in the dataset.

inline bool star::StarDataset::contains (
    const std::string & key
) const

Returns true if key names either a stored array (array namespace) or a metadata-block value (metadata namespace). Layer-prefixed internal keys are matched as-is. Use this for membership tests; it does not load any data.

Parameters:

key Key to look up

Returns:

true if the key exists in either namespace

function create_layer¶

Create new layer and return view.

inline std::shared_ptr< LayerView > star::StarDataset::create_layer (
    const std::string & layer_name
)

Parameters:

layer_name Name of new layer

Returns:

Shared pointer to LayerView

Exception:

std::runtime_error if layer already exists

function decode_array_bytes¶

inline ValueVariant star::StarDataset::decode_array_bytes (
    const std::vector< char > & compressed_data,
    const std::vector< BlockInfo > & blocks,
    CompressionAlgorithm compression,
    DataType dtype,
    const std::vector< size_t > & shape,
    ThreadPool * thread_pool
)

function decode_numeric_blocks_into_ndarray¶

Decode one array entry's compressed bytes into a ValueVariant.

inline ValueVariant star::StarDataset::decode_numeric_blocks_into_ndarray (
    const std::vector< char > & compressed_data,
    const std::vector< BlockInfo > & blocks,
    CompressionAlgorithm compression,
    DataType dtype,
    const std::vector< size_t > & shape,
    ThreadPool * thread_pool
)

This is the CPU half of an array load (everything after the ranged read): decompress the blocks, reverse any byte-shuffle prefilter, and materialize the NDArray. It is a pure function of its arguments — it reads no shared mutable state and writes none — so it is safe to run concurrently on thread-pool workers, which is what prefetch() relies on to decode many columns in parallel.

thread_pool controls only INTRA-array block parallelism inside decompressBlocks(). A batch caller that is already parallelizing ACROSS arrays MUST pass nullptr here: a pool worker that enqueued to and waited on the same pool could deadlock. The single-array load path (load_entry) passes the pool so a lone large array still decompresses its blocks in parallel.

One-pass decode for the fast path: build the NDArray<T> and decompress every block directly into its storage.

Only reached for fixed-width numeric arrays with no byte-shuffle prefilter, where the decompressed block bytes are exactly the element bytes. Produces the identical NDArray the general (decompressBlocks + memcpy) path would, without the intermediate full-size buffer or the second pass. Pure function of its arguments — safe to run on pool workers, same as decode_array_bytes().

function decompress_single_block¶

Decompress a single block (helper for layer metadata).

inline std::vector< char > star::StarDataset::decompress_single_block (
    const std::vector< char > & compressed,
    CompressionAlgorithm compression
)

Parameters:

compressed Compressed data
compression Compression algorithm

Returns:

Decompressed data

function deserializeKey¶

Deserializes a key from the input stream.

inline std::string star::StarDataset::deserializeKey (
    std::istream & is
)

Parameters:

is Input stream

Returns:

Deserialized key

function deserialize_typed_value¶

Deserializes typed value from stream.

inline ValueVariant star::StarDataset::deserialize_typed_value (
    std::istream & is,
    DataType dtype,
    const std::vector< size_t > & shape,
    size_t data_len
)

Parameters:

is Input stream
dtype Data type
shape Array shape
data_len Data length in bytes

Returns:

ValueVariant containing the deserialized value

function deserialize_typed_value_bytes¶

Zero-stream variant of deserialize_typed_value() for the array load hot path: deserialize directly from a contiguous byte buffer.

inline ValueVariant star::StarDataset::deserialize_typed_value_bytes (
    const char * data,
    size_t data_len,
    DataType dtype,
    const std::vector< size_t > & shape,
    const ByteUnshuffleSpec & unshuffle={}
)

The stream version copies the buffer INTO a std::stringstream and then reads it back OUT into the NDArray — two extra full-buffer passes plus stream overhead. For fixed-width numeric arrays (the common case) the wire format is just raw native-endian bytes, so a single std::memcpy reproduces the exact same result an order of magnitude faster. Strings are variable-width (length-prefixed) and rare on this path, so they delegate to the stream version unchanged — behavior is identical for every dtype.

When unshuffle.active, the incoming bytes are byte-planes from a shuffle codec: instead of a plain memcpy we run byte_unshuffle straight INTO the NDArray storage. This fuses the transpose with the fill, eliminating the scratch buffer and the extra full-buffer pass the caller would otherwise do (decompress → scratch → unshuffle → scratch2 → memcpy → NDArray becomes decompress → scratch → unshuffle → NDArray). The default (active == false) is a byte-identical plain memcpy, so non-shuffle codecs are unchanged.

function dtype_of¶

Get the element data type of a stored array.

inline DataType star::StarDataset::dtype_of (
    const std::string & key
) const

Looks up the array namespace (values stored via put()/put<T>()). This lets callers dispatch on the concrete type without probing every get<T>() overload. For metadata-namespace values use meta.get(key)->dtype instead.

Parameters:

key Array key

Returns:

DataType of the stored array

Exception:

std::runtime_error if the key is not an array (not found in array storage)

function end [1/2]¶

Returns an iterator to the end of the index.

inline iterator star::StarDataset::end ()

Returns:

Iterator to one past the last key-value pair

function end [2/2]¶

Returns a const iterator to the end of the index.

inline const_iterator star::StarDataset::end () const

Returns:

Iterator to one past the last key-value pair

function ensure_layer_metadata_loaded¶

Loads metadata block from file if not already loaded (supports HTTP/remote URLs).

inline void star::StarDataset::ensure_layer_metadata_loaded (
    size_t layer_idx
)

Ensure a specific layer's metadata is loaded (v3 format)

Parameters:

layer_idx Index of the layer to load

function flush¶

Writes all data to the file (including metadata block).

inline void star::StarDataset::flush ()

function flush_quiet¶

Flush without throwing on read-only (silently skips instead).

inline void star::StarDataset::flush_quiet ()

Used by close() and the destructor, where flushing is best-effort cleanup rather than an explicit persist request — a read-only dataset must be destructible without raising.

function get¶

Get array from storage.

template<typename T>
inline NDArray < T > star::StarDataset::get (
    const std::string & key
)

Retrieves array from either metadata block or separate storage.

Parameters:

key Array key

Returns:

NDArray

function get_all_keys¶

Gets all keys (metadata block + separate arrays).

inline std::vector< std::string > star::StarDataset::get_all_keys ()

Returns:

Vector of all keys in the store

function get_file_header¶

Get file header with version information.

inline const FileHeader & star::StarDataset::get_file_header () const

Returns:

Reference to FileHeader

function get_filename¶

Get current filename.

inline std::string star::StarDataset::get_filename () const

Returns:

Filename

function get_layer¶

Get existing layer view.

inline std::shared_ptr< LayerView > star::StarDataset::get_layer (
    const std::string & layer_name
)

Parameters:

layer_name Name of the layer

Returns:

Shared pointer to LayerView

Exception:

std::runtime_error if layer doesn't exist

function get_metadata_count¶

Gets count of entries in metadata block.

inline size_t star::StarDataset::get_metadata_count () const

Returns:

Number of metadata entries (excludes layer-prefixed internal keys)

function get_metadata_keys¶

Gets list of keys in metadata block.

inline std::vector< std::string > star::StarDataset::get_metadata_keys () const

Returns:

Vector of key names (excludes layer-prefixed internal keys)

function get_slice¶

Gets n-dimensional slice of an array.

template<typename T>
inline NDArray < T > star::StarDataset::get_slice (
    const std::string & key,
    const std::vector< Slice > & slices
)

IMPORTANT: This function only works with arrays stored as separate compressed arrays with block structure. Arrays stored in the metadata block cannot be sliced and must be accessed as complete units using meta.get() instead.

Parameters:

key Array key in store (must be separately stored with blocks)
slices Vector of slices, one per dimension Empty = entire dimension, unfilled dimensions = full slice

Returns:

NDArray containing the requested slice

Exception:

std::runtime_error if key not found or stored in metadata block

Examples: // 1D: elements 1000-2000 (step defaults to 1) get_slice<double>("large_timeseries", {{1000, 2000}})

// 2D: rows 10-20, all columns get_slice<float>("image_data", {{10, 20}, {0, width}})

// Using helper functions for clarity get_slice<float>("matrix", {slice_range(10, 20), slice_all(width)})

// 3D: hyperslab get_slice<uint16_t>("volume", {{0, 10}, {5, 15}, {0, depth}})

// For small arrays in metadata block, use meta.get() instead: auto small_array = store.meta.get("small_data");

function has_layer¶

Check if layer exists.

inline bool star::StarDataset::has_layer (
    const std::string & layer_name
) const

Parameters:

layer_name Layer name to check

Returns:

true if layer exists, false otherwise

function has_metadata_block¶

Checks if file has metadata block.

inline bool star::StarDataset::has_metadata_block () const

Returns:

True if metadata block exists

function is_metadata_loaded¶

Checks if metadata block has been loaded.

inline bool star::StarDataset::is_metadata_loaded () const

Returns:

True if loaded

function is_read_only¶

Check if file is in read-only mode.

inline bool star::StarDataset::is_read_only () const

Returns:

True if read-only

function is_sliceable¶

Checks if an array supports slicing.

inline bool star::StarDataset::is_sliceable (
    const std::string & key
) const

Arrays stored in the metadata block cannot be sliced - they must be accessed as complete units. Only separately stored arrays with block compression support efficient slicing.

Parameters:

key Array key to check

Returns:

True if array supports get_slice(), false if only meta.get() works

Example: if (store.is_sliceable("large_data")) { auto slice = store.get_slice<double>("large_data", {{0, 1000}}); } else { auto full = store.meta.get("small_data"); }

function key_in_layer¶

Check if key exists in specific layer using bit-mask (O(1)).

inline bool star::StarDataset::key_in_layer (
    const std::string & key,
    const std::string & layer_name
) const

Parameters:

key Key to check
layer_name Layer name

Returns:

true if key exists in layer, false otherwise

function layer_inheritance¶

inline bool star::StarDataset::layer_inheritance () const

function list_layers¶

Get list of all layer names.

inline std::vector< std::string > star::StarDataset::list_layers () const

Returns:

Vector of layer names

function loadIndex¶

Loads the index from the file.

inline void star::StarDataset::loadIndex ()

function load_all_metadata¶

Load all layer metadata blocks at once (v3 format).

inline void star::StarDataset::load_all_metadata ()

function load_entry¶

Load entry from disk into memory.

inline void star::StarDataset::load_entry (
    size_t idx
)

Parameters:

idx Index in SoA arrays

function load_layer_metadata¶

Load a specific layer's metadata block (v3 format).

inline void star::StarDataset::load_layer_metadata (
    const std::string & layer_name
)

Parameters:

layer_name Name of the layer to load

function load_metadata_block¶

inline void star::StarDataset::load_metadata_block ()

function open_options¶

Read-time options (e.g. layer inheritance) for this dataset.

inline const OpenOptions & star::StarDataset::open_options () const

These affect only how the in-memory dataset resolves reads; nothing is persisted. They are safe to change at any time, including on a read-only dataset, and take effect immediately for existing and future LayerViews.

function operator=¶

StarDataset & star::StarDataset::operator= (
    const StarDataset &
) = delete

function operator=¶

StarDataset & star::StarDataset::operator= (
    StarDataset &&
) = delete

function parse_layer_metadata_block¶

Parse a layer's metadata block (STARMeta format).

inline void star::StarDataset::parse_layer_metadata_block (
    size_t layer_idx,
    const std::vector< char > & decompressed
)

Parameters:

layer_idx Layer index
decompressed Decompressed block data

function prefetch¶

Load several arrays into cache in parallel (a read-ahead hint).

inline void star::StarDataset::prefetch (
    const std::vector< std::string > & keys
)

A single get()/get_slice() reads one array end-to-end: the read pipeline (ranged I/O -> decompress -> unshuffle -> materialize) runs serially, and a caller reading N columns pays that latency N times back-to-back — only the block decompression WITHIN one array is threaded. prefetch() overlaps the whole pipeline ACROSS arrays:

I/O stays SERIAL on the dataset's single persistent reader (one reused connection; remote reads issue exactly one GET per array, same as today — no request amplification), but
each array's CPU work (decompress + unshuffle + build NDArray) is dispatched to the thread pool the moment its bytes arrive, so array k+1's read overlaps array k's decode and multiple decodes run at once.

After prefetch() returns, the named arrays are cached, so subsequent get<T>()/get_slice<T>() calls just copy out — this is the batch "read many columns" path. Results are byte-identical to loading each key individually; this only changes WHEN/where the work runs. Keys that are unknown throw (same contract as get); keys already loaded or stored in the metadata block are handled on the normal serial path. With threading disabled (num_threads == 1) it degrades to loading each key in turn.

Thread-safety: holds the dataset write lock for the whole batch; the decode tasks are pure (they read only stable per-entry metadata we do not mutate here and own their input buffers), and they pass nullptr for the pool so a pool worker never waits on the same pool (no nested-parallel deadlock).

function print_header¶

inline void star::StarDataset::print_header () const

function put [1/2]¶

Store array as separate compressed array (not in metadata block).

template<typename T>
inline void star::StarDataset::put (
    const std::string & key,
    NDArray < T > && value
)

Use this for large arrays that need slicing support. Small arrays should use meta.put() instead.

Parameters:

key Array key
value NDArray to store

function put [2/2]¶

Store array as separate compressed array (const lvalue overload).

template<typename T>
inline void star::StarDataset::put (
    const std::string & key,
    const NDArray < T > & value
)

function read_range¶

Read exactly [position, position+len) from the backing file via the persistent reader. Returns the bytes actually read.

inline std::vector< char > star::StarDataset::read_range (
    size_t position,
    size_t len
)

function reader¶

Return the dataset's persistent byte-range reader, creating it on first use. One reader (= one reused connection for remote files) is shared by every read path — the index load, layer-metadata loads, and array-block reads — so remote reads never open a new connection or issue a HEAD per read.

inline RangeReader & star::StarDataset::reader ()

For small REMOTE files, this also triggers the whole-file prefetch (OpenOptions::prefetch_whole_below_bytes): the object is fetched once and all subsequent reads are served from memory.

function save_to¶

inline void star::StarDataset::save_to (
    const std::string & target_path
)

function serialize_array_data [1/2]¶

Serialize an NDArray 's element data into a byte buffer for block storage, in the SAME wire formatdeserialize_typed_value() reads.

template<typename T>
inline std::vector< char > star::StarDataset::serialize_array_data (
    const std::vector< T > & data
) const

Numeric types are raw contiguous bytes; std::string arrays are length-prefixed (uint32 total length, then per-element uint32 length + bytes). Using this everywhere fixes the bug where string arrays were memcpy'd as raw std::string OBJECTS (pointers/SSO) and came back empty after reload. For numeric types the bytes are identical to the old memcpy.

function serialize_array_data [2/2]¶

Serialize array data, applying the byte-shuffle prefilter if the codec is a *_SHUFFLE variant.

template<typename T>
inline std::vector< char > star::StarDataset::serialize_array_data (
    const std::vector< T > & data,
    CompressionAlgorithm codec,
    size_t block_size
) const

Shuffle is only valid for fixed-width numeric elements laid out as raw contiguous bytes; std::string arrays are length-prefixed and variable width, so they are never shuffled (they store fine under the base codec).

Two shuffle layouts are produced depending on the codec: * global (GZIP_SHUFFLE/LZ4_SHUFFLE): shuffle the whole buffer at once. * per-block (GZIP_SHUFFLE_BLOCK/LZ4_SHUFFLE_BLOCK): shuffle each block_size chunk independently, so it lines up with the compression blocks that compressBlocksBuffered() cuts at the same boundaries. Each block is then self-contained and sliceable.

function serialize_layer_metadata_block¶

Serializes a single layer's metadata in STARMeta format (v1).

inline void star::StarDataset::serialize_layer_metadata_block (
    std::ostream & os,
    const std::string & layer_name
)

Parameters:

os Output stream
layer_name Layer name

function serialize_metadata_block¶

Serializes metadata block to stream (OLD FORMAT - deprecated).

inline void star::StarDataset::serialize_metadata_block (
    std::ostream & os
) const

Parameters:

os Output stream

function serialize_metadata_value [1/2]¶

Helper to serialize metadata value (string specialization).

template<typename T>
inline std::enable_if< std::is_same< T, std::string >::value, void >::type star::StarDataset::serialize_metadata_value (
    std::ostream & os,
    const std::vector< T > & data
) const

function serialize_metadata_value [2/2]¶

Helper to serialize metadata value (numeric types).

template<typename T>
inline std::enable_if<!std::is_same< T, std::string >::value, void >::type star::StarDataset::serialize_metadata_value (
    std::ostream & os,
    const std::vector< T > & data
) const

function set_layer_inheritance¶

inline void star::StarDataset::set_layer_inheritance (
    bool on
)

function set_layer_presence¶

Set layer presence for a data array key.

inline void star::StarDataset::set_layer_presence (
    const std::string & layer_name,
    const std::string & key,
    bool present
)

Parameters:

layer_name Layer name
key Data array key
present Whether the key is present in this layer

function set_open_options¶

inline void star::StarDataset::set_open_options (
    const OpenOptions & opts
)

function shape_of¶

Full shape (all dimensions) of a stored data array.

inline std::vector< size_t > star::StarDataset::shape_of (
    const std::string & key
) const

Metadata-only: the shape is read from the index loaded at open() time, so this issues NO data read (no decompression, no extra network request for remote datasets). Complements dtype_of()/array_length(); use get<T>() only when the element data itself is needed.

Parameters:

key Array key in store

Returns:

Dimension sizes (empty for a scalar)

Exception:

std::runtime_error if the key is not a stored data array

function size¶

Removes a key-value pair from the store.

inline size_t star::StarDataset::size () const

Parameters:

key Key to remove

Returns the number of key-value pairs in the store

Returns:

Number of key-value pairs

function useThreading¶

Check if threading should be used based on workload.

inline bool star::StarDataset::useThreading (
    size_t num_blocks,
    size_t data_size
) const

Parameters:

num_blocks Number of blocks to process
data_size Total data size in bytes

Returns:

true if threading should be used, false otherwise

function write_bytes¶

Serialize the dataset to an in-memory byte buffer.

inline std::vector< char > star::StarDataset::write_bytes ()

The byte-array counterpart of save_to(): returns a complete .stards image (the exact bytes that would be written to a file) instead of writing to a path. Works on any dataset — including read-only ones and datasets opened with open_bytes() — since it never touches the source file. Round-trips with open_bytes(): open_bytes (ds-> write_bytes() ) reconstructs the dataset.

Returns:

A complete .stards image as a byte array.

function ~StarDataset¶

Destructor - writes all pending changes to disk (RAII).

inline star::StarDataset::~StarDataset ()

Public Static Functions Documentation¶

function create¶

Create a new Star dataset file.

static inline std::shared_ptr< StarDataset > star::StarDataset::create (
    const std::string & filename,
    const StarConfig & config=StarConfig ()
)

Creates a new file with the specified configuration. If the file already exists, it will be overwritten. The file will be created on the first flush() or when the object is destroyed.

Parameters:

filename Path to create (local, s3://... / /vsis3/...)
config Configuration for compression, block sizes, metadata

Returns:

New StarDataset instance

function open [1/2]¶

Open an existing Star dataset file.

static inline std::shared_ptr< StarDataset > star::StarDataset::open (
    const std::string & filename,
    FileMode mode=FileMode::READ_WRITE,
    const OpenOptions & opts={}
)

Opens an existing file and reads its configuration from the file header. The configuration used when the file was created is preserved.

If mode is READ_WRITE and file doesn't exist, it will be created. If mode is READ_ONLY and file doesn't exist, an error is thrown.

Parameters:

filename Path to open (local, s3:// or /vsis3/, https:// or /vsicurl/)
mode FileMode enum (READ_WRITE/READ_ONLY)

Returns:

Opened StarDataset instance

Exception:

std::runtime_error if file doesn't exist in READ_ONLY mode or is corrupt

function open [2/2]¶

Open an existing Star dataset file (string mode overload).

static inline std::shared_ptr< StarDataset > star::StarDataset::open (
    const std::string & filename,
    const std::string & mode_str,
    const OpenOptions & opts={}
)

Parameters:

filename Path to open (local, s3:// or /vsis3/, https:// or /vsicurl/)
mode_str String mode ("r", "w", "rw", "a")

Returns:

Opened StarDataset instance

Exception:

std::runtime_error if file doesn't exist or is invalid

function open_bytes [1/2]¶

Open a dataset from an in-memory byte buffer.

static inline std::shared_ptr< StarDataset > star::StarDataset::open_bytes (
    std::vector< char > bytes,
    const OpenOptions & opts={}
)

Mirrors open(), but the source is a byte array holding a complete .stards image (e.g. bytes received over a socket or pulled from a database) instead of a path. The dataset is READ_ONLY — there is no backing file to flush to; use write_bytes() to serialize modifications back out to a new byte array.

Parameters:

bytes A complete .stards image.
opts Read-time options (e.g. layer_inheritance).

Returns:

Opened StarDataset backed by the provided bytes.

Exception:

std::runtime_error if the bytes are not a valid STAR image.

function open_bytes [2/2]¶

Convenience overload taking a raw pointer + length (e.g. from a C buffer, Python bytes, or a mapped region).

static inline std::shared_ptr< StarDataset > star::StarDataset::open_bytes (
    const void * data,
    size_t size,
    const OpenOptions & opts={}
)

The documentation for this class was generated from the following file StarDS/include/stards.h