Namespace star¶

Namespaces¶

Type	Name
namespace	logger
namespace	s3crypto
namespace	shuffle_detail

Classes¶

Type	Name
class	AWSConfigParser AWS Configuration file parser.
class	AWSTokenCache AWS SSO Token Cache reader.
class	AWSV4Signer AWS Signature Version 4 signer.
struct	BlockInfo Metadata for a single compressed block.
struct	BlockMap Maps logical elements to physical blocks.
struct	ByteUnshuffleSpec Describes an in-place byte-unshuffle to fuse into an array's fill.
struct	ColdStorage Cold storage - infrequently accessed data.
struct	ExtractionPlan Describes how to extract elements from blocks.
struct	FileHeader File header structure (31 bytes fixed size).
struct	FilePathInfo
struct	HotStorage Hot storage - frequently accessed data (cache-friendly).
class	HttpRangeReader
struct	IndexEntry Index entry with block compression support and shape information.
struct	KeyRegistry Global key registry using data-oriented design (Structure of Arrays).
class	LayerMetadataAccessor Metadata accessor for a specific layer with inheritance.
struct	LayerMetadataRegistry Per-layer metadata registry using data-oriented design (Structure of Arrays).
class	LayerView Lightweight view into a specific layer with inheritance from base.
class	LocalRangeReader
class	MemoryRangeReader
class	MetadataAccessor Accessor for metadata operations.
struct	MetadataValue Type-erased wrapper for metadata values.
class	NDArray <typename T> Modern n-dimensional array class with xtensor-style API.
struct	OpenOptions Read-time options for StarDataset::open() .
class	RangeReader
struct	S3Credentials AWS Credentials with resolution chain.
struct	S3EndpointConfig S3 endpoint resolution (default AWS, or an override for S3-compatible services such as MinIO and for local testing).
class	S3RangeReader
class	S3Writer S3 writer for uploading objects.
struct	Slice Describes a slice along one dimension (Python-style slicing) Plain struct - no methods except helpers, just data.
struct	SliceSpec Complete slice specification for n-dimensional array.
struct	StarConfig Configuration for metadata block optimization.
class	StarDataset A cloud-optimized binary key-value store for serializable data types.
class	ThreadPool Simple thread pool for parallel block operations.
struct	TypeToDataType <typename T>
struct	TypeToDataType< double > <>
struct	TypeToDataType< float > <>
struct	TypeToDataType< int16_t > <>
struct	TypeToDataType< int32_t > <>
struct	TypeToDataType< int64_t > <>
struct	TypeToDataType< int8_t > <>
struct	TypeToDataType< std::string > <>
struct	TypeToDataType< uint16_t > <>
struct	TypeToDataType< uint32_t > <>
struct	TypeToDataType< uint64_t > <>
struct	TypeToDataType< uint8_t > <>

Public Types¶

Type	Name
enum uint8_t	CompressionAlgorithm
enum uint8_t	DataType
enum	FileMode
enum	StorageClass Storage classification for values (DEPRECATED - use StorageLocation).
enum	StorageLocation Storage location state for unified storage model.
typedef std::variant< NDArray< int8_t >, NDArray< int16_t >, NDArray< int32_t >, NDArray< int64_t >, NDArray< uint8_t >, NDArray< uint16_t >, NDArray< uint32_t >, NDArray< uint64_t >, NDArray< float >, NDArray< double >, NDArray< std::string > >	ValueVariant

Public Attributes¶

Type	Name
const char *	MAGIC_STRING = `"STARDS"`
const size_t	MAGIC_STRING_LENGTH = `6`
const std::string	PROJECT_NAME = `"STARDS " STAR\_VERSION\_STRING`
size_t	g_min_blocks_for_threading = `4`
size_t	g_min_bytes_for_threading = `256 \* 1024`
std::atomic< uint64_t >	g_network_request_count = `{0}`
size_t	g_num_threads = `0`

Public Functions¶

Type	Name
CompressionAlgorithm	base_compression (CompressionAlgorithm c) The underlying block codec for a (possibly shuffle-prefiltered) algorithm.
void	byte_shuffle (const char * in, char * out, size_t count, size_t elem_size) Byte-shuffle: reorder `count` elements of`elem_size` bytes so that byte plane 0 of every element comes first, then plane 1, etc.
void	byte_shuffle_blocked (const char * in, char * out, size_t data_size, size_t elem_size, size_t block_size) Per-block byte-shuffle: apply byte_shuffle() independently within each `block_size` -byte chunk of a`data_size` -byte buffer.
void	byte_unshuffle (const char * in, char * out, size_t count, size_t elem_size) Inverse of byte_shuffle(): reassemble byte planes back into elements.
void	byte_unshuffle_blocked (const char * in, char * out, size_t data_size, size_t elem_size, size_t block_size) Inverse of byte_shuffle_blocked(): un-shuffle each `block_size` chunk.
std::pair< std::vector< char >, std::vector< BlockInfo > >	compressBlocks (const char * data, size_t data_size, CompressionAlgorithm algorithm, size_t block_size) Compresses data in blocks and returns block metadata (legacy version).
std::vector< BlockInfo >	compressBlocksBuffered (const char * data, size_t data_size, CompressionAlgorithm algorithm, size_t block_size, std::vector< char > & compressed_output, std::vector< char > & temp_buffer, ThreadPool * thread_pool=nullptr) Compresses data in blocks using pre-allocated buffer.
size_t	datatype_size (DataType dtype) Get element size in bytes for a DataType.
const char *	datatype_to_string (DataType dtype) Get string representation of DataType.
std::vector< char >	decompressBlocks (const std::vector< char > & compressed_data, const std::vector< BlockInfo > & blocks, CompressionAlgorithm algorithm, const std::vector< size_t > & block_indices={}, ThreadPool * thread_pool=nullptr) Decompresses specific blocks from compressed data.
void	decompressBlocksInto (const std::vector< char > & compressed_data, const std::vector< BlockInfo > & blocks, CompressionAlgorithm algorithm, char * dst, size_t dst_capacity, ThreadPool * thread_pool=nullptr) One-pass sibling of decompressBlocks(): decompress every block straight into a caller-owned destination buffer instead of allocating and returning a std::vector.
size_t	estimateCompressedSize (const char * data, size_t data_size, CompressionAlgorithm algorithm, size_t block_size) Estimate compressed size using deflateBound without actually compressing.
std::string	getLibraryVersion () Get library version string.
uint64_t	getNetworkRequestCount ()
size_t	getNumThreads () Get current thread count setting.
std::string	getS3Region () Get S3 region from environment or default.
uint64_t	hash_key (const std::string & key) Hash function for keys in global key registry.
FilePathInfo	parseFilePath (const std::string & filename) Parse a file path and determine its type (local, HTTP, or S3).
FileMode	parseModeString (const std::string & mode_str)
UInt	read_le (std::istream & is)
uint16_t	read_u16 (std::istream & is)
uint32_t	read_u32 (std::istream & is)
uint64_t	read_u64 (std::istream & is)
uint8_t	read_u8 (std::istream & is)
void	resetNetworkRequestCount ()
void	setMinBlocksForThreading (size_t min_blocks) Set minimum blocks threshold for using threading.
void	setMinBytesForThreading (size_t min_bytes) Set minimum data size threshold for using threading.
void	setNumThreads (size_t num_threads) Set number of threads for parallel operations (all datasets).
Slice	slice_all (size_t dim_size)
Slice	slice_range (size_t start, size_t stop)
CURLcode	star_curl_perform (CURL * handle)
bool	uses_block_shuffle (CompressionAlgorithm c) Whether the shuffle prefilter is applied PER BLOCK (self-contained blocks). Such arrays are sliceable.
bool	uses_global_shuffle (CompressionAlgorithm c) Whether the shuffle prefilter is applied across the WHOLE array (legacy layout). Such arrays are not sliceable.
bool	uses_shuffle (CompressionAlgorithm c) Whether a compression algorithm uses the byte-shuffle prefilter (either the legacy global variant or the per-block variant).
void	write_le (std::ostream & os, UInt value)
void	write_u16 (std::ostream & os, uint16_t v)
void	write_u32 (std::ostream & os, uint32_t v)
void	write_u64 (std::ostream & os, uint64_t v)
void	write_u8 (std::ostream & os, uint8_t v)

Public Static Functions¶

Type	Name
DataType	extract_dtype_from_variant (const ValueVariant & var)
std::vector< size_t >	extract_shape_from_variant (const ValueVariant & var)

Public Types Documentation¶

enum CompressionAlgorithm¶

enum star::CompressionAlgorithm {
    NONE = 0,
    GZIP = 1,
    ZSTD = 2,
    LZ4 = 3,
    GZIP_SHUFFLE = 4,
    LZ4_SHUFFLE = 5,
    GZIP_SHUFFLE_BLOCK = 6,
    LZ4_SHUFFLE_BLOCK = 7
};

enum DataType¶

enum star::DataType {
    INT8 = 0,
    INT16 = 1,
    INT32 = 2,
    INT64 = 3,
    UINT8 = 4,
    UINT16 = 5,
    UINT32 = 6,
    UINT64 = 7,
    FLOAT32 = 8,
    FLOAT64 = 9,
    STRING = 10
};

enum FileMode¶

enum star::FileMode {
    READ_WRITE,
    READ_ONLY
};

enum StorageClass¶

Storage classification for values (DEPRECATED - use StorageLocation).

enum star::StorageClass {
    METADATA_BLOCK,
    SEPARATE_ARRAY,
    FORCE_SEPARATE
};

enum StorageLocation¶

Storage location state for unified storage model.

enum star::StorageLocation {
    PENDING,
    PERSISTED,
    CACHED
};

typedef ValueVariant¶

using star::ValueVariant =  std::variant<
    NDArray<int8_t>, NDArray<int16_t>, NDArray<int32_t>, NDArray<int64_t>,
    NDArray<uint8_t>, NDArray<uint16_t>, NDArray<uint32_t>, NDArray<uint64_t>,
    NDArray<float>, NDArray<double>,
    NDArray<std::string>
>;

Public Attributes Documentation¶

variable MAGIC_STRING¶

const char* star::MAGIC_STRING;

variable MAGIC_STRING_LENGTH¶

const size_t star::MAGIC_STRING_LENGTH;

variable PROJECT_NAME¶

const std::string star::PROJECT_NAME;

variable g_min_blocks_for_threading¶

size_t star::g_min_blocks_for_threading;

variable g_min_bytes_for_threading¶

size_t star::g_min_bytes_for_threading;

variable g_network_request_count¶

std::atomic<uint64_t> star::g_network_request_count;

variable g_num_threads¶

size_t star::g_num_threads;

Public Functions Documentation¶

function base_compression¶

The underlying block codec for a (possibly shuffle-prefiltered) algorithm.

inline CompressionAlgorithm star::base_compression (
    CompressionAlgorithm c
)

The block (de)compressor only understands the base codecs; the shuffle variants differ only in a byte-reordering prefilter applied around it.

function byte_shuffle¶

Byte-shuffle: reorder count elements ofelem_size bytes so that byte plane 0 of every element comes first, then plane 1, etc.

inline void star::byte_shuffle (
    const char * in,
    char * out,
    size_t count,
    size_t elem_size
)

Splits an array-of-structs byte layout into a struct-of-byte-planes layout. A no-op for elem_size <= 1. out must hold count * elem_size bytes. SIMD-accelerated on NEON for elem_size {2,4,8}; scalar everywhere else (the scalar tail below is bit-identical to the historical loop).

function byte_shuffle_blocked¶

Per-block byte-shuffle: apply byte_shuffle() independently within each block_size -byte chunk of adata_size -byte buffer.

inline void star::byte_shuffle_blocked (
    const char * in,
    char * out,
    size_t data_size,
    size_t elem_size,
    size_t block_size
)

This is the prefilter for the BLOCK-shuffle codecs. Because the buffer is later cut into blocks at the SAME block_size boundaries by compressBlocksBuffered(), each compression block ends up holding exactly one self-contained shuffled chunk, so it can be un-shuffled on its own (enabling slicing) — unlike the global variant, whose byte planes span the whole array.

Each chunk is shuffled over its whole-element prefix (chunk_size / elem_size elements); any trailing bytes that don't form a complete element (only possible when block_size is not a multiple of elem_size) are copied through verbatim, and byte_unshuffle_blocked() reverses this exactly. out must hold data_size bytes.

function byte_unshuffle¶

Inverse of byte_shuffle(): reassemble byte planes back into elements.

inline void star::byte_unshuffle (
    const char * in,
    char * out,
    size_t count,
    size_t elem_size
)

This is the read hot path. SIMD-accelerated for elem_size {2,4,8} on NEON (arm64) and SSE2 (x86-64); scalar everywhere else. The scalar tail is bit-identical to the historical loop, so decoded bytes never change.

function byte_unshuffle_blocked¶

Inverse of byte_shuffle_blocked(): un-shuffle each block_size chunk.

inline void star::byte_unshuffle_blocked (
    const char * in,
    char * out,
    size_t data_size,
    size_t elem_size,
    size_t block_size
)

function compressBlocks¶

Compresses data in blocks and returns block metadata (legacy version).

inline std::pair< std::vector< char >, std::vector< BlockInfo > > star::compressBlocks (
    const char * data,
    size_t data_size,
    CompressionAlgorithm algorithm,
    size_t block_size
)

Parameters:

data Raw uncompressed data
data_size Size of raw data
algorithm Compression algorithm to use
block_size Size of each uncompressed block

Returns:

Pair of compressed data and block metadata

function compressBlocksBuffered¶

Compresses data in blocks using pre-allocated buffer.

inline std::vector< BlockInfo > star::compressBlocksBuffered (
    const char * data,
    size_t data_size,
    CompressionAlgorithm algorithm,
    size_t block_size,
    std::vector< char > & compressed_output,
    std::vector< char > & temp_buffer,
    ThreadPool * thread_pool=nullptr
)

Parameters:

data Raw uncompressed data
data_size Size of raw data
algorithm Compression algorithm to use
block_size Size of each uncompressed block
compressed_output Pre-allocated output buffer (will be cleared and reused)
temp_buffer Pre-allocated temporary buffer for compression (will be resized as needed)
thread_pool Optional thread pool for parallel compression (nullptr = single-threaded)

Returns:

Block metadata

function datatype_size¶

Get element size in bytes for a DataType.

inline size_t star::datatype_size (
    DataType dtype
)

Parameters:

dtype DataType to query

Returns:

Size in bytes (0 for variable-length types like STRING)

function datatype_to_string¶

Get string representation of DataType.

inline const char * star::datatype_to_string (
    DataType dtype
)

Parameters:

dtype DataType to convert

Returns:

String representation

function decompressBlocks¶

Decompresses specific blocks from compressed data.

inline std::vector< char > star::decompressBlocks (
    const std::vector< char > & compressed_data,
    const std::vector< BlockInfo > & blocks,
    CompressionAlgorithm algorithm,
    const std::vector< size_t > & block_indices={},
    ThreadPool * thread_pool=nullptr
)

Parameters:

compressed_data Full compressed data
blocks Block metadata
algorithm Compression algorithm used
block_indices Which blocks to decompress (empty = all)
thread_pool Optional thread pool for parallel decompression (nullptr = single-threaded)

Returns:

Decompressed data

function decompressBlocksInto¶

One-pass sibling of decompressBlocks(): decompress every block straight into a caller-owned destination buffer instead of allocating and returning a std::vector.

inline void star::decompressBlocksInto (
    const std::vector< char > & compressed_data,
    const std::vector< BlockInfo > & blocks,
    CompressionAlgorithm algorithm,
    char * dst,
    size_t dst_capacity,
    ThreadPool * thread_pool=nullptr
)

decompressBlocks() allocates a full-size buffer, decodes into it, and returns it — the caller then copies that buffer into its final home (e.g. an NDArray), so every byte is written twice and a whole extra buffer is allocated. When the on-disk bytes ARE the final element bytes (fixed-width numeric arrays with no byte-shuffle prefilter to undo), that intermediate buffer is pure overhead: this function decodes each block directly to dst + <block offset>, giving a single pass over the data with no scratch allocation.

The per-codec decode calls are identical to decompressBlocks() (same LZ4/zlib entry points, same block layout, same threading gate) so the produced bytes are bit-for-bit the same; only the destination differs. It intentionally does NOT support a block-index subset (the fused path is always a whole-array read) — the slicing paths keep using decompressBlocks(). Left as a separate function rather than refactoring decompressBlocks() to delegate, so the existing return-a-vector path is untouched.

Parameters:

dst Destination buffer; must hold at least the sum of block uncompressed sizes.
dst_capacity Size of dst in bytes (checked; a larger declared shape leaves the trailing bytes untouched, matching the partial memcpy the general path performs).

function estimateCompressedSize¶

Estimate compressed size using deflateBound without actually compressing.

inline size_t star::estimateCompressedSize (
    const char * data,
    size_t data_size,
    CompressionAlgorithm algorithm,
    size_t block_size
)

This provides an upper bound on the compressed size, which is useful for pre-calculating file positions before actual compression. The estimate is typically 10-30% larger than actual compressed size.

Parameters:

data Raw uncompressed data
data_size Size of raw data
algorithm Compression algorithm to use
block_size Size of each uncompressed block

Returns:

Estimated upper bound on compressed size

function getLibraryVersion¶

Get library version string.

inline std::string star::getLibraryVersion ()

Returns:

Version string (e.g., "1.0.0")

function getNetworkRequestCount¶

inline uint64_t star::getNetworkRequestCount ()

function getNumThreads¶

Get current thread count setting.

inline size_t star::getNumThreads ()

function getS3Region¶

Get S3 region from environment or default.

inline std::string star::getS3Region ()

function hash_key¶

Hash function for keys in global key registry.

inline uint64_t star::hash_key (
    const std::string & key
)

function parseFilePath¶

Parse a file path and determine its type (local, HTTP, or S3).

inline FilePathInfo star::parseFilePath (
    const std::string & filename
)

Recognizes both the GDAL virtual-filesystem prefixes and the equivalent plain URL/URI forms, so either works interchangeably: * S3: "/vsis3/bucket/key" OR "s3://bucket/key" * HTTP: "/vsicurl/https://host/path" OR "https://host/path" / "http://..." * anything else -> a local filesystem path. The /vsi* prefixes stay supported for GDAL compatibility; the plain forms are the natural way to name a remote object and map to the same handling.

Defined unconditionally: StarDataset::open()/ctor call it for every path, and the LOCAL branch (the common case) has no S3/curl dependency. Only the S3 region-resolution needs AWSConfigParser, so that part is gated on ENABLE_S3; without S3 support an S3 path is rejected with a clear error.

function parseModeString¶

inline FileMode star::parseModeString (
    const std::string & mode_str
)

function read_le¶

template<typename UInt>
inline UInt star::read_le (
    std::istream & is
)

function read_u16¶

inline uint16_t star::read_u16 (
    std::istream & is
)

function read_u32¶

inline uint32_t star::read_u32 (
    std::istream & is
)

function read_u64¶

inline uint64_t star::read_u64 (
    std::istream & is
)

function read_u8¶

inline uint8_t star::read_u8 (
    std::istream & is
)

function resetNetworkRequestCount¶

inline void star::resetNetworkRequestCount ()

function setMinBlocksForThreading¶

Set minimum blocks threshold for using threading.

inline void star::setMinBlocksForThreading (
    size_t min_blocks
)

Parameters:

min_blocks Minimum number of blocks (default: 4)

function setMinBytesForThreading¶

Set minimum data size threshold for using threading.

inline void star::setMinBytesForThreading (
    size_t min_bytes
)

Parameters:

min_bytes Minimum data size in bytes (default: 256KB)

function setNumThreads¶

Set number of threads for parallel operations (all datasets).

inline void star::setNumThreads (
    size_t num_threads
)

Parameters:

num_threads Number of threads (0 = auto-detect, 1 = single-threaded)

function slice_all¶

inline Slice star::slice_all (
    size_t dim_size
)

function slice_range¶

inline Slice star::slice_range (
    size_t start,
    size_t stop
)

function star_curl_perform¶

inline CURLcode star::star_curl_perform (
    CURL * handle
)

function uses_block_shuffle¶

Whether the shuffle prefilter is applied PER BLOCK (self-contained blocks). Such arrays are sliceable.

inline bool star::uses_block_shuffle (
    CompressionAlgorithm c
)

function uses_global_shuffle¶

Whether the shuffle prefilter is applied across the WHOLE array (legacy layout). Such arrays are not sliceable.

inline bool star::uses_global_shuffle (
    CompressionAlgorithm c
)

function uses_shuffle¶

Whether a compression algorithm uses the byte-shuffle prefilter (either the legacy global variant or the per-block variant).

inline bool star::uses_shuffle (
    CompressionAlgorithm c
)

function write_le¶

template<typename UInt>
inline void star::write_le (
    std::ostream & os,
    UInt value
)

function write_u16¶

inline void star::write_u16 (
    std::ostream & os,
    uint16_t v
)

function write_u32¶

inline void star::write_u32 (
    std::ostream & os,
    uint32_t v
)

function write_u64¶

inline void star::write_u64 (
    std::ostream & os,
    uint64_t v
)

function write_u8¶

inline void star::write_u8 (
    std::ostream & os,
    uint8_t v
)

Public Static Functions Documentation¶

function extract_dtype_from_variant¶

static inline DataType star::extract_dtype_from_variant (
    const ValueVariant & var
)

function extract_shape_from_variant¶

static inline std::vector< size_t > star::extract_shape_from_variant (
    const ValueVariant & var
)

The documentation for this class was generated from the following file StarDS/include/stards.h