Download as PDF, PPTX
Uploaded byAlessandra Bilardi
Parallelize data processing - 2023-10-24
The document discusses the parallelization of data processing with a focus on performance challenges and results using various libraries like pandas, Dask, and Modin on different datasets. It compares execution times, highlights best practices for optimizing performance, and emphasizes the importance of vectorization and multiprocessing. Resources such as links to benchmarking tools and related works are provided to support the findings.
More Related Content
Similar to Parallelize data processing - 2023-10-24
Parallelize data processing - 2023-10-24
- 1. Parallelize data processing @PyDataVenice#13 #Meetup #PyData
- 2. Alessandra Bilardi Data &Automation Specialist ● AWS User Group Venezia member ● Coderdojo member ● PyData Venezia member alessandra.bilardi@gmail.com @abilardi bilardi
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20. Solution Goal: performance Challenge: on 4 CPU& 8GB RAM ● try.sh ○ for each dataset ■ run.py ● run.py ○ for each library ■ importlib ■ main() ● main() ○ for each method ■ timeit before/after ■ print timing ■ gc.collect()
- 21. Datasets - rows Goal: executiontiming Challenge: resources ● 10 ● 50 ● 100 ● 500 ● 1.000 ● 5.000 ● 10.000 ● 50.000 ● 100.000 ● 500.000 ● 1.000.000 ● 5.000.000 ● 10.000.000
- 22. Libraries Goal: learning curve Challenge: the samebehaviours ● concurrent ● dask ● joblib ○ joblib on dask ● modin ○ modin on dask ● multiprocessing ● multiprocesspandas ● pandarallel ● pandas ● parallelize ● pyspark ● swifter
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32. Results $ grep "hlp.get_pd_sample()"* concurrent_actions.py: sample = hlp.get_pd_sample() joblib_actions.py: sample = hlp.get_pd_sample() joblibdask_actions.py: sample = hlp.get_pd_sample() multiprocessing_actions.py: sample = hlp.get_pd_sample() multiprocesspandas_actions.py: sample = hlp.get_pd_sample() pandarallel_actions.py: sample = hlp.get_pd_sample() pandas_actions.py:sample = hlp.get_pd_sample() parallelize_actions.py: sample = hlp.get_pd_sample() swifter_actions.py: sample = hlp.get_pd_sample()
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45.
- 46.
- 47.
- 48. Segmenti casuali dicodice o è qualcosa di più ..
- 49.
- 50.
- 51.
- 52.
- 53.
- 54. Oddities Goal: performance Challenge: on 4 CPU& 8GB RAM ● pandas takes longer ○ than libs who use it ● pandas generate more ○ pids & threads than pyspark ● modin generate more ○ pids & threads than others ● modin works better if ..
- 55. Best practies Goal: performance Challenge: on 4CPU & 8GB RAM ● vectorialization ○ better than apply ● multiprocessing ● pay attention ○ df.copy() ○ groupby / sum / apply
- 56. Thanks for listening. @PyDataVenice#13 #Meetup #PyData