deepdrivemd.data.stream.aggregator_reader

Classes

`AdiosReader`(fn, config, stream_name, variables)	This class is used to read the next N steps from an adios stream.
`StreamContactMapVariable`(name, dtype, structure)	Implementation of StreamVariable that handles contact maps: unpack bits to 1D array, convert 1D array to 2D array.
`StreamScalarVariable`(name, dtype, structure)	Implementation of StreamVariable that handles scalar variables.
`StreamVariable`(name, dtype, structure)	This class is used to read a variable from BP file.
`Streams`(files, variables[, config, ...])	The class keeps lastN steps from each aggregator.

class deepdrivemd.data.stream.aggregator_reader.AdiosReader(fn: str, config: pathlib.Path, stream_name: str, variables: List[deepdrivemd.data.stream.aggregator_reader.StreamVariable])

This class is used to read the next N steps from an adios stream.

adios

Type: adios2.adios2.ADIOS

io

Type: adios2.adios2.IO

stream

Type: adios2.adios2.Engine

__del__(): Destructor: clean the adios resources.

__init__(fn: str, config: pathlib.Path, stream_name: str, variables: List[deepdrivemd.data.stream.aggregator_reader.StreamVariable])

Initialize AdiosReader object.

Parameters

fn (str) – file name of bp file or sst socket (without sst extension)
config (Path) – path to adios.xml file
stream_name (str) – name of a stream in adios.xml file

next(N: int) → Dict[str, Union[numpy.array, str, int, float]]

Read the next N steps of all variables.

Parameters: N (int) – read that many steps
Returns: Dict[str, Union[np.array, str, int, float]] – values for different variables whose names are used as keys

class deepdrivemd.data.stream.aggregator_reader.StreamContactMapVariable(name: str, dtype: type, structure: deepdrivemd.data.stream.enumerations.DataStructure)

Implementation of StreamVariable that handles contact maps: unpack bits to 1D array, convert 1D array to 2D array.

next(ARW)

Get the variable value for the next time step and append it to total.

Parameters: ARW (AdiosStreamStepRW) – low level object for reading data from ADIOS stream (BP file or SST stream)

class deepdrivemd.data.stream.aggregator_reader.StreamScalarVariable(name: str, dtype: type, structure: deepdrivemd.data.stream.enumerations.DataStructure)

Implementation of StreamVariable that handles scalar variables.

next(ARW)

Get the variable value for the next time step and append it to total.

Parameters: ARW (AdiosStreamStepRW) – low level object for reading data from ADIOS stream (BP file or SST stream)

class deepdrivemd.data.stream.aggregator_reader.StreamVariable(name: str, dtype: type, structure: deepdrivemd.data.stream.enumerations.DataStructure)

This class is used to read a variable from BP file.

name

variable name in adios file

Type: str

dtype

variable type, for example, np.uint8

Type: type

structure

enumeration: array, scalar, string

Type: DataStructure

total

list of variable values for different steps

Type: List

__init__(name: str, dtype: type, structure: deepdrivemd.data.stream.enumerations.DataStructure)

Parameters

name (str) – variable name in adios file
dtype (type) – variable type, for example, np.uint8
structure (DataStructure) – structure type: array, scalar, string

next(ARW: deepdrivemd.data.stream.adios_utils.AdiosStreamStepRW)

Get the variable value for the next time step and append it to total.

Parameters: ARW (AdiosStreamStepRW) – low level object for reading data from ADIOS stream (BP file or SST stream)

class deepdrivemd.data.stream.aggregator_reader.Streams(files: List[str], variables: List[deepdrivemd.data.stream.aggregator_reader.StreamVariable], config: pathlib.Path = PosixPath('../aggregate/adios.xml'), stream_name: str = 'AdiosOutput', lastN: int = 2000, batch: int = 10000)

The class keeps lastN steps from each aggregator.

readers

a dictionary of AdiosReader indexed by the corresponding adios file name

Type: Dict[str, AdiosReader]

positions

Type: Dict[str, np.ndarray]

md5

Type: Dict[str, str]

steps

Type: Dict[str, np.ndarray]

rmsds

Type: Dict[str, np.ndarray]

cm

Type: Dict[str, np.ndarray]

velocities

Type: Dict[str, np.ndarray]

lastN

keep that many last steps from each aggregator

Type: int

batch

up to how many steps to read from each adios file at a time

Type: int

__init__(files: List[str], variables: List[deepdrivemd.data.stream.aggregator_reader.StreamVariable], config: pathlib.Path = PosixPath('../aggregate/adios.xml'), stream_name: str = 'AdiosOutput', lastN: int = 2000, batch: int = 10000)

Initialize Streams object.

Parameters

files (List[str]) – adios files from each aggregator,
variables (List[StreamVariable]) – list of variables to read from the aggegator file
config (Path) – adios xml file for the files,
stream_name (str) – corresponding stream name in adios.xml
lastN (int) – number of last steps to keep from each adios file
batch (int) – up to how many steps to read from each adios file at a time (call of next())

next() → Dict[str, Union[numpy.array, str, int, float]]

Provide lastN steps from each aggregator.

Returns: Dict[str, Union[np.array, int, float, str]] – values for the the variables whose names are used as keys