This document discusses various methods for communication between processes in Python, including client-server architectures, message queues, serialization, multiprocessing pipes, forking, and memory-mapped files. It provides code examples and performance results for different serialization formats like JSON, msgpack, and pickle. Memory usage is also compared for multiprocessing pipes, forking, and memory mapping for inter-process communication.

1. COMMUNICATION BETWEEN
PROCESSES
LVIV.PY #5

2. LVIV.PY #5
ABOUT
▸ Myself: Yehor Nazarkin
▸ Occupation: Python Software Engineer at DataRobot LLC
▸ Follow up: @nimnull, github.com/nimnull

3. LVIV.PY #5
IPC
▸ Client-server
▸ Signals
▸ Message queue
▸ Data Storage
SERVER
CLIENT
CLIENT
BUS
SERVER

4. LVIV.PY #5
SERIALISATION
▸ CPU/memory aligned
▸ Takes some time
▸ Variadic performance for different data

5. LVIV.PY #5
SERIALISATION

6. LVIV.PY #5
SERIALISATION
▸ Tested for json, ujson, msgpack, pickle
▸ first test — huge attributes number, small attribute size
json: 16.32 ms
ujson: 2.89 ms
msgpack: 1.40 ms
pickle: 1.49 ms

7. LVIV.PY #5
SERIALISATION

8. LVIV.PY #5
SERIALISATION
▸ Tested for json, ujson, msgpack, pickle
▸ second test — small attributes number, huge attribute size
json: 1.59 ms
ujson: 1.91 ms
msgpack: 1.22 ms
pickle: 1.20 ms

9. LVIV.PY #5
ZMQ (WHATEVER MQ)

10. LVIV.PY #5
ZMQ (WHATEVER MQ)

11. LVIV.PY #5
ZMQ (WHATEVER MQ)
▸ Matrix 1000x1000, float, ~7.68Mb
▸ Mean exchange time ~ 15.61ms
▸ Client-server
▸ Serialisation
▸ Buffer I/O
▸ Netwok latency

12. LVIV.PY #5
MULTIPROCESSING PIPES

13. LVIV.PY #5
MULTIPROCESSING PIPES
PID Command USS
# — parent process
20363 python multiproc_test.py 13368 <- keep matrix
# -- child process
20367 python multiproc_test.py 1484 <- no matrix
20367 python multiproc_test.py 10056 <- matrix passed
▸ Pipe — parent connection, child connection
▸ Memory overhead on copying data

14. LVIV.PY #5
OS.FORK()

15. LVIV.PY #5
OS.FORK()
PID Command USS PSS RSS
28181 python fork_test.py 4724 15836 29792
28182 python fork_test.py 836 11701 23560 <-- before copy on write
28182 python fork_test.py 8756 15853 23944 <-- after copy on write
▸ Copy-on-write benefits.
▸ One-way channel.

16. LVIV.PY #5
MEMORY-MAPPED FILES

17. LVIV.PY #5
MEMORY-MAPPED FILES
PID Command USS PSS RSS
16737 python mmap_test.py 26540 26914 29968 <-- before mmap write
16737 python mmap_test.py 8096 19782 34200 <-- after mmap write
16795 python mmap_test.py 5484 16920 29656 <-- child processing
▸ Limitations — string/byte ndarray type (numpy)
▸ Value data types (no refcounts, etc)
▸ Serialisation

18. LVIV.PY #5
SOURCES
▸ https://github.com/nimnull/lviv.py-5

19. LVIV.PY #5
QUESTIONS?
▸ Source: https://github.com/nimnull/lviv.py-5
▸ Follow me: @nimnull