shm#
Distributed Shared Memory Utility for Dataloader
- fkat.utils.shm.load(path: Path) Any[source]#
Load serialized object with out-of-band data from path based on zero-copy shared memory.
- Parameters:
path (pathlib.Path) – Folder used to save serialized data with serialize(). Usually a folder /dev/shm
- Returns:
Raw deserialized data
- fkat.utils.shm.save(obj: Any, path: Optional[Path] = None) Path[source]#
Serialize obj with out-of-band data to path for zero-copy shared memory usage.
If the object to be serialized itself, or the objects it uses for data storage (such as numpy arrays) implement the the pickle protocol version 5 pickle.PickleBuffer type in __reduce_ex__, then this function can store these buffers out-of-band as files in path so that they subsequently be re-used for zero-copy sharing accross processes.
- Parameters:
obj (object) – Object to serialize. For example a PyArrow Table, a Pandas Dataframe or any type that relies on NumPy to store the binary data.
path (pathlib.Path, optional) – Empty folder used to save serialized data. Usually a folder in /dev/shm
- Returns:
pathlib.Path where the data was serialized