Python Multiprocessing Pitfalls

It is not much a suprise not getting things right the first time. It serves as a validation. Correctness. Things to know and look forward to finally get it done and done well. In this excerpt are personal notes exploratory to the subject of multiprocessing in Python. A lot of pretty interesting stuff. Also a lot of time spent experimenting, reading, and making mistakes. So here’s an attempt to avoid making them again.

Overview:

• Pool of worker processes
• Expecting too much from proxies

Pool of worker processes

Spawns worker processes ready to do some work. Not much to do with structure but for the series of steps like in a recipe.

Expecting too much from proxies

Shared access. Not exclusive rights.

A manager provides a convenient way for processes to gain access and manipulate objects. Managers also allow support for arbitrary object types and not limited to ctypes. Few of the supported types are list and dict among others which you may already be using from multiprocessing module directly (Lock, Semaphore, Queue, Value, Array).

1. Mix use of proxy and non-proxy objects.

# Case 1: Nested proxies
# Case 2: Nested proxies with non-proxy objects (list, dict, set, etc) and how to notify

2. Comparison of proxy types

Note The proxy types in multiprocessing do nothing to support comparisons by value. So, for instance, we have: …