Intermediate result storage¶
Sometimes, specially with big payloads, alternative transfer methods can be considered, since transferring huge data streams using messages can be quite expensive.
There are multiple approaches to solve this issue, mostly revolving around transfering data via external means, some of them will be explained here.
External data stores¶
Using a centralized broker where all processes can concurrently store and retrieve data can be considered, specially when distributing actors over the network.
Some in-memory databases (such as memcached or redis) are specially good at storing temporary data, but even traditional databases or dedicated blob storage services (such as min.io) could be used, enabling those resources to be accessed on actors accross the network.
When talking about local-only actors, tempfile.NamedTemporaryFile could be also a good option when used alongside RAM-based virtual filesystems like tmpfs. Regular filesystems should be only considered for objects being forwarded across many actors since even the fastest block device is still slower than regular multiprocessing mechanisms.
Integrating with external services is a whole big subject on its own and its outside this documentation scope.