Download as PDF, PPTX
Uploaded byRyuji Tamagawa
PySparkの勘所（20170630 sapporo db analytics showcase）
This document discusses PySpark and how it relates to Spark, Hadoop, and Python for data analysis (PyData). PySpark allows users to write Spark programs using Python APIs, access Spark functionality from Python, and interface between Spark and PyData tools like pandas. It also covers Spark file formats like Parquet that can improve performance when used with PySpark and PyData tools.
More Related Content
Similar to PySparkの勘所（20170630 sapporo db analytics showcase）
![谷歌留痕技术教程[ 𝙩𝙤𝙥 𝟮𝟯𝟯. 𝙘 𝙤𝙢 ]](https://cdn.slidesharecdn.com/ss_thumbnails/top233-260130173900-2eb784f9-thumbnail.jpg?width=640&height=640&fit=bounds)
PySparkの勘所（20170630 sapporo db analytics showcase）
- 1.
- 2. ▸ facebook :Ryuji Tamagawa ▸ Twitter : tamagawa_ryuji ▸ FB ▸ Twitter
- 4.
- 5. Wes Mckinney blog ▸http://qiita.com/tamagawa-ryuji
- 6. ▸ ▸ pandas PyData ▸Spark Scala Java Spark ▸ TB
- 8. ▸ Spark Hadoop ▸PySpark ▸ PySpark ▸ Spark/Hadoop PyData PySpark
- 9.
- 10. Spark Hadoop Hadoop0.x Spark OS HDFS MapReduce OS HDFS Hivee.t.c. HBase MapReduce OS HDFS Hive e.t.c. HBaseMapReduce YARN Spark Spark Streaming, MLlib, GraphX, Spark SQL) Impala SQL YARN Spark Spark Streaming, MLlib, GraphX, Spark SQL) Mesos Spark Spark Streaming, MLlib, GraphX, Spark SQL) Spark Spark Streaming, MLlib, GraphX, Spark SQL) Windows Hadoop 0.x Hadoop 1.x Hadoop 2.x + Spark
- 11. Spark Hadoop Hadoop Spark map JVM HDFS reduce JVM map JVM reduce JVM f1 RDD ExecutorJVM HDFS f2 f3 f4 f5 f6 f7 MapReduce Spark RDD
- 12. Spark Hadoop Spark ▸ HadoopMapReduce ▸ Spark API MapReduce API ▸ Hadoop
- 13.
- 14. PySpark (Py)Spark ▸ / Spark ▸PyData ▸ Spark ▸ Spark Hadoop PyData PySpark
- 15.
- 16.
- 17.
- 18. PySpark RDD API DataFrameAPI ▸ RDD Resilient Distributed Dataset = Spark Java ▸ DataFrame RDD / R data.frame ▸ Spark 2.x DataFrame Learning PySpark ML Structured Streaming GraphFrames TensorFrame ▸ Python RDD API DataFrame API Scala / Java
- 19. Worker node PySpark Executer JVM Driver JVM Executer JVM Executer JVM Storage Python VM Worker nodeWorker node Python VM Python VM RDD API PySpark Worker node Executer JVM Driver JVM Executer JVM Executer JVM Storage Python VM Worker node Worker node Python VM Python VM DataFrame API PySpark
- 20. PySpark ▸ RDD APIExecuter JVM Python VM ▸ DataFrame API JVM ▸ UDF Python VM ▸ UDF Scala Java ▸ Spark 2.x DataFrame
- 21.
- 22. Spark PyData Spark PyData ▸Spark ▸ Python PyData ▸ ▸ Parquet ▸ Apache Arrow
- 23.
- 24. Spark PyData PyData Anaconda Python BlazeNumPy and pandas interface to Big Data'. dask Bokeh Canopy Python IPython matplotlib PyData nose numba JIT NumPy PyData Scipy PyData Statsmodels SymPy pandas NumPy SciPy scikit-image scikit-learn PyData
- 25. Spark PyData ▸ CSVJSON ▸ Spark Parquet ▸ Performance comparison of different file formats and storage engines in the Hadoop ecosystem ▸ Parquet Python ▸ fastparquet pyarrow ▸ Parquet
- 26.
- 27. Spark PyData Spark df =spark.read.csv(csvFilename, header=True, schema = theSchema).coalesce(20) df.write.save(filename, compression = 'snappy') from fastparquet import write pdf = pd.read_csv(csvFilename) write(filename, pdf, compression='UNCOMPRESSED') fastparquet import pyarrow as pa import pyarrow.parquet as pq arrow_table = pa.Table.from_pandas(pdf) pq.write_table(arrow_table, filename, compression = 'GZIP') pyarrow
- 28. Spark PyData ▸ pandasCSV Spark Spark pandas … ▸ Spark - pandas ▸ pandas → Spark … ▸ Apache Arrow
- 29. Spark PyData Apache Arrow ▸Apache Arrow ▸ PyData / OSS ▸ / https://arrow.apache.org
- 30. Spark PyData Wes blog ▸pandas Apache Arrow ▸ Blog ▸ PyData Blog Wes OK ▸ 2017 : pandas, Arrow, Feather, Parquet, Spark, Ibis http://qiita.com/tamagawa-ryuji/items/deb3f63ed4c7c8065e81
- 31.
- 32. ▸ pandas PySpark ▸PySpark DataFrame API ▸ Parquet CSV Parquet ▸ UI Jupyter Notebook Parquet PySpark DataFrame API pandas PyDataJupyter Notebook CSV