This document discusses using Python, Pandas, and Spark 2.0 for data analysis. It covers loading CSV and other data formats into Pandas and Spark DataFrames, using Parquet for efficient storage, and transferring data between Pandas and Spark for hybrid processing using CPUs and SSDs. The last part discusses new features in Spark 2.0 like SQL support on DataFrames and improved Python integration.

1. Python, Pandas, Spark 2.0
Sky

3. •
• Python 2000
(**)
• db tech showcase MongoDB
•
• FB: Ryuji Tamagawa
• Twitter : tamagawa_ryuji

5. 2017

6. • Python Spark
•

7. •
• Python / Pandas
• Spark 2.0

8. Part 1 :

9. •
•
•
csv

10. Python
Pandas Python
Jupyter Notebook
Jenkins
Spark 2.0

11. • Spark API RDD ~1.3 DataFrame
/ DataSet 1.4~
• DataFrame API
RDD API Python Spark

12. DataFrame
• RDB /
• R Pandas Spark
Spark
R / Pandas
Spark
+

13. Part 2 :

14. CSV
zip
RDB
Parquet
Excel
CSV
Feather
Spark
Pandas / Spark

15. •
• CPU
•
• Pandas read_csv zip CSV
Pandas

16. 2
• CSV CPU
Pandas zip CSV
CPU …
• Parquet !
•

17. : Parquet
I/O
•
• Spark Parquet
• Python Parquet

18. HDFS / S3
Parquet Parquet

19. SSD
Parquet Parquet

20. Parquet
No
No
Yes
HDD

21. •
• I/O Pandas
• Spark
• DataFrame Pandas → Spark
Spark → Pandas Pandas → Spark
• Apache Arrow

22. CPU
~2010
2010~
SSD
CPU

23. Apache Spark 2.0
• 1.x
• 2.0
1.x
• DataFrame API Python
• databricks
http://go.databricks.com/mastering-apache-spark-2.0
•

24. Spark 2.0
• CPU
• CPU
• SQL DataFrame
• + SSD
• CSV zip
Pandas read_csv

25. Python + Spark
• Python serialize
• DataFrame API UDF
UDF Scala/Java
• http://www.slideshare.net/dragan10/performant-data-processing-with-pyspark-sparkr-
and-dataframe-api
Executor
JVM
DataFrame,
Cached
Python
lambda items:
items[0] == ‘abc’
transfer
DataFrame,
result
transfer
Driver