HDF5 VOL Data Mapping

Navigate back: Main / HDF5 User Guide

---

The HDF5 Data Mapping can only be used with the HDF5 VOL connectors that implement map objects. The native HDF5 library does not support this feature.

HDF5 Map Object

Introduction

While the HDF5 data model provides a flexible way to store structured data, some applications require a more general mechanism for indexing information with arbitrary keys. HDF5 Map objects address this need by providing application-defined key-value stores where key-value pairs can be added, retrieved by key, and iterated over.

Map objects are available only through VOL connectors that implement the map interface. The native HDF5 file format does not support map objects. VOL connectors with map support include DAOS and other storage systems optimized for key-value operations.

See also

VOL Mapping (H5M) Reference Manual

Map Object Life Cycle

Map objects follow a well-defined life cycle from creation through cleanup:

1. Creation

   • Create a new map with H5Mcreate() or H5Mcreate_anon():
     • H5Mcreate() creates a map and links it into the file hierarchy with a specified name
     • H5Mcreate_anon() creates an anonymous map that must be linked with H5Olink before closing
   • Specify the key datatype and value datatype during creation - these define how keys and values are stored in the map
   • Optionally provide creation and access property lists (MCPL and MAPL) to control map behavior
   • The function returns a map identifier (hid_t) used for all subsequent operations

2. Opening

   • Open an existing map with H5Mopen() by providing its location and name
   • Optionally specify a map access property list (MAPL) to control access behavior
   • Multiple opens of the same map are allowed and each returns a separate identifier

3. Data Operations

   • Adding/Updating: Use H5Mput() to add new key-value pairs or update existing values
     • Specify memory datatypes for the key and value being written
     • If the key already exists, its value is updated
     • If the key is new, a new key-value pair is added
   • Retrieving: Use H5Mget() to retrieve a value by its key
     • Provide a buffer to receive the value
     • Specify memory datatypes for key and value
     • Type conversion is performed if memory and stored datatypes differ
   • Checking Existence: Use H5Mexists() to check if a key exists without retrieving its value
   • Deleting: Use H5Mdelete() to remove a key-value pair from the map
   • Iterating: Use H5Miterate() or H5Miterate_by_name() to process all key-value pairs
     • Provide a callback function that is invoked for each key-value pair
     • Iteration order is determined by the VOL connector implementation
     • Can start iteration from a specific index position

4. Query Operations

   • H5Mget_count() retrieves the number of key-value pairs stored in the map
   • H5Mget_key_type() returns the datatype used for keys
   • H5Mget_val_type() returns the datatype used for values
   • H5Mget_create_plist() retrieves the map creation property list (MCPL)
   • H5Mget_access_plist() retrieves the map access property list (MAPL)

5. Closing
   • Close the map with H5Mclose() when finished
   • All map identifiers should be closed before closing the file
   • Closing a map does not delete it from the file
   • The library maintains reference counts and will not actually close the map until all identifiers are closed

Map Property Lists

Map objects use two types of property lists that fit into the HDF5 property list class hierarchy (see Property List Classes):

• Map Creation Property List (MCPL) - H5P_MAP_CREATE

  • Controls properties that are set when a map is created
  • These properties are permanent characteristics of the map
  • Inherits from the Object Creation Property List (OCPL) class
  • Can be retrieved later with H5Mget_create_plist()
  • Default MCPL: H5P_MAP_CREATE_DEFAULT or H5P_DEFAULT

• Map Access Property List (MAPL) - H5P_MAP_ACCESS
  • Controls properties for accessing an existing map
  • These properties affect how the map is opened and accessed
  • Settings can vary between different opens of the same map
  • Can be retrieved with H5Mget_access_plist()
  • Default MAPL: H5P_MAP_ACCESS_DEFAULT or H5P_DEFAULT

These property list classes follow the same inheritance hierarchy as other HDF5 objects. The MCPL inherits properties relevant to object creation (like character encoding for names), while the MAPL controls access-specific settings.

Key and Value Datatypes

Map objects require two datatypes:

• Key Datatype: Defines how keys are stored in the map

  • Can be any valid HDF5 datatype (integer, float, string, compound, etc.)
  • Variable-length strings (H5T_C_S1 with H5T_VARIABLE size) are commonly used for keys
  • Fixed at map creation time

• Value Datatype: Defines how values are stored in the map
  • Can be any valid HDF5 datatype
  • All values in a map must have the same datatype
  • Fixed at map creation time

During H5Mput() and H5Mget() operations, memory datatypes can differ from the stored datatypes. The HDF5 library will perform type conversion as needed, similar to dataset I/O operations.

Example Usage

The following example demonstrates creating a map, adding key-value pairs, and retrieving values. Note that this example requires a VOL connector with map support (e.g., DAOS). See HDF5 Virtual Object Layer (VOL) for VOL connector configuration details.

hid_t file_id, map_id, vls_type_id, fapl_id;
const char *names[2] = {"Alice", "Bob"};
uint64_t IDs[2] = {25385486, 34873275};
uint64_t val_out;
herr_t ret;
 
// Setup file access property list with VOL connector that supports maps
// (Configuration depends on specific VOL connector - see connector documentation)
fapl_id = H5Pcreate(H5P_FILE_ACCESS);
// ... configure VOL connector in fapl_id ...
 
// Create variable-length string datatype for keys
vls_type_id = H5Tcopy(H5T_C_S1);
H5Tset_size(vls_type_id, H5T_VARIABLE);
 
// Create file
file_id = H5Fcreate("file.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl_id);
 
// Create map with string keys and uint64 values
map_id = H5Mcreate(file_id, "map", vls_type_id, H5T_NATIVE_UINT64,
                   H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
 
// Add key-value pairs
H5Mput(map_id, vls_type_id, &names[0], H5T_NATIVE_UINT64, &IDs[0], H5P_DEFAULT);
H5Mput(map_id, vls_type_id, &names[1], H5T_NATIVE_UINT64, &IDs[1], H5P_DEFAULT);
 
// Retrieve a value by key
ret = H5Mget(map_id, vls_type_id, &names[0], H5T_NATIVE_UINT64, &val_out, H5P_DEFAULT);
if(ret < 0 || val_out != IDs[0]) {
    fprintf(stderr, "Failed to retrieve correct value from map\n");
    goto error;
}
 
// Close map and other objects
H5Mclose(map_id);
H5Tclose(vls_type_id);
H5Pclose(fapl_id);
H5Fclose(file_id);
return 0;
 
error:
    H5E_BEGIN_TRY {
        H5Mclose(map_id);
        H5Tclose(vls_type_id);
        H5Pclose(fapl_id);
        H5Fclose(file_id);
    } H5E_END_TRY;
    return -1;

Important Notes

• VOL Connector Requirement: Map functionality is only available through VOL connectors that implement the map interface. The native HDF5 file format does not support maps.

• Experimental API: The H5M interface is experimental and subject to change in future releases. Application code using maps should be prepared for potential API modifications.

• Key Uniqueness: Each key in a map must be unique. Calling H5Mput() with an existing key will update that key's value rather than creating a duplicate entry.

• Iteration Order: The order in which key-value pairs are returned during iteration is determined by the VOL connector implementation and may not be predictable.

• Type Conversion: As with datasets, the library performs datatype conversion between memory and storage representations. Ensure memory buffers are sized appropriately for the memory datatype specified.

• Asynchronous Operations: Several map operations have asynchronous variants (e.g., H5Mcreate_async, H5Mopen_async) for use with the asynchronous I/O interface.

Previous Chapter The HDF5 Event Set Interface - Next Chapter HDF5 References

---

Navigate back: Main / HDF5 User Guide