deepdrivemd.aggregation.stream.aggregator

Functions

aggregate(cfg, connections, ...)

Read adios streams from a subset of simulations handled by this aggregator and write them to adios file to be used by machine learning and outlier search.

connect_to_input(cfg, bpfiles)

Open adios streams for reading.

find_input(cfg)

Find adios streams to which simulations write.
deepdrivemd.aggregation.stream.aggregator.aggregate(cfg: deepdrivemd.aggregation.stream.config.StreamAggregation, connections: Dict[int, Tuple[adios2.adios2.ADIOS, adios2.adios2.IO, adios2.adios2.Engine]], aggregator_stream: adios2.adios2.Engine, aggregator_stream_4ml: adios2.adios2.Engine)

Read adios streams from a subset of simulations handled by this aggregator and write them to adios file to be used by machine learning and outlier search.

Parameters

  • cfg (StreamAggregation)

  • connections (Dict[int, Tuple[adios2.adios2.ADIOS, adios2.adios2.IO, adios2.adios2.Engine]]) – key - task id, value - a tuple of adios objects

  • aggregator_stream (adios2.adios2.Engine) – an adios stream of aggregated file to write to.

Note

If we do not need to save the data for postproduction, we can get rid of the aggregated adios file and replace it by SST stream.

deepdrivemd.aggregation.stream.aggregator.connect_to_input(cfg: deepdrivemd.aggregation.stream.config.StreamAggregation, bpfiles: List[pathlib.Path]) → Dict[int, Tuple[adios2.adios2.ADIOS, adios2.adios2.IO, adios2.adios2.Engine]]

Open adios streams for reading.

Parameters

  • cfg (StreamAggregation)

  • bpfiles (List[Path])

Returns

Dict[int, Tuple[adios2.adios2.ADIOS, adios2.adios2.IO, adios2.adios2.Engine]] – key - simulation task id, value - tuple of the corresponding adios objects.

deepdrivemd.aggregation.stream.aggregator.find_input(cfg: deepdrivemd.aggregation.stream.config.StreamAggregation) → List[str]

Find adios streams to which simulations write.

Parameters

cfg (StreamAggregation)

Returns

List[str] – a list of sst files associated with simulations