Utility functions

Introduction

When storing experimental data in HDF5 files, some extra operations may be useful to process them on a low level. The operations provided by this module mangle HDF5 files directly (through h5py), without relying on higher-level functionality. In turn, some of the functionality may be useful for more advanced stuff.

Note that all functions rely on a specific file format, specifically on the format which is written by PuPy.RobotCollector, with experiments in groups and sensor data in seperate datasets within the experiment group. On this ground, too short experiments can be removed (remove_init_only_groups()) or files merged together (h5_merge_experiments()). When data is split up between two files, they can easily be put together by H5CombinedFile.

Reference

HDPy.inout.remove_init_only_groups(pth, init_steps)[source]

Remove groups from HDF5 data files, which

  1. Are empty (0 members)
  2. Have collected less than init_steps epochs
HDPy.inout.h5_reorder(pth)[source]

Rearrange the experiments in pth such that the experiment indices are in the range [0,N], without missing ones. No order of the experiments is guaranteed.

HDPy.inout.h5_merge_experiments(pth0, pth1, trg=None)[source]

Merge groups of the HDF5 files pth0 and pth1. If trg is given, a new file will be created. Otherwise the data is merged into pth0.

HDPy.inout.remove_boundary_groups(pth)[source]

Remove the first and last experiment with respect to webots restart/revert in pth. The boundaries are determined through the init_step group. This method is to save possibly corrupted experimental data files, due to webots’ memory issues. To work properly, the groups must not be altered before this method, e.g. by remove_init_only_groups().

class HDPy.inout.H5CombinedFile(pth_main, pth_additional)[source]

Combine two HDF5 files which have the same groups on the root level but different datasets within these groups. The files are packed together such that they can be handled as if a single file was present.

pth_main
Path to the first HDF5 file. If a dataset is available in both files, the one from this file will be used.
pth_additional
Path to the second HDF5 file.
class HDPy.inout.H5CombinedGroup(grp0, grp1)[source]

Combine two related HDF5 groups which store different datasets and present them as a single group. Instances to this class are typically exclusively created through H5CombinedFile.

grp0
Group of the first file. If a dataset is present in both groups, the one from this group will be used.
grp1
Group of the second file.

Table Of Contents

Previous topic

Reinforcement Learning

Next topic

Analysis

This Page