Python Multiprocessing Pitfalls
It is not much a suprise not getting things right the first time. It serves as a validation. Correctness. Things to know and look forward to finally get it done and done well. In this excerpt are personal notes exploratory to the subject of multiprocessing in Python. A lot of pretty interesting stuff. Also a lot of time spent experimenting, reading, and making mistakes. So here’s an attempt to avoid making them again.
- Pool of worker processes
- Expecting too much from proxies
Pool of worker processes
Spawns worker processes ready to do some work. Not much to do with structure but for the series of steps like in a recipe.
Expecting too much from proxies
Shared access. Not exclusive rights.
A manager provides a convenient way for processes to gain access and manipulate objects. Managers also allow support for arbitrary object types and not limited to
Few of the supported types are
dict among others which you may already be using from multiprocessing module directly (
- Mix use of proxy and non-proxy objects.
# Case 1: Nested proxies
# Case 2: Nested proxies with non-proxy objects (list, dict, set, etc) and how to notify
- Comparison of proxy types
Note The proxy types in multiprocessing do nothing to support comparisons by value. So, for instance, we have: …
🐋 hello there! If you enjoy this, a
Or you can also ...Buy me a tea