This document discusses Python and the pandas library. It provides an overview of Python's history and advantages, such as being easy to learn and having a large standard library. It also discusses the major Python data analysis packages NumPy, SciPy, matplotlib, and pandas. Pandas allows importing data from various sources, manipulating datasets, and performing operations on labeled and indexed data. The document also covers using pandas with other tools like Spark, visualization with matplotlib, and IDEs and notebooks for Python development.

1. Python / pandas
Sky

3. •
• Python 2000
(**)
• db tech showcase MongoDB
•
• FB: Ryuji Tamagawa
• Twitter : tamagawa_ryuji

5. 2015-2016

6. • Python / pandas
• Python / pandas

7. • Python
•
• Python
• NumPy, SciPy, matplotlib, pandas
• Python
• Python IPython, Jupyter notebook, Spyder, VisualStudio
• Python / pandas
• Python
• pandas
• Spark - PySpark DataFrame API
• matplotlib

8. Part 1 : Python

9. Python
•
• Google Guido
Google Google
1
•
NumPy, SciPy, matplotlib →
pandas
•
•
-2000
Linux
-2010 Web Trac
Google

10. Python
•
•
•
•
•
• ’Batteries included’

11. Python
• 2.x 3.x 32bit 64bit
64bit
• 2.x
• 3.x
3
• 2.x
3.x

12. • Ruby?
• R?
• Java?
• Scala?

13. Python
• Python ’CPython’ JIT
PyPy JVM Jython .Net IronPython
• CPython
• CPython 2
• C
• processing
PySpark

14. Python
• Python Linux Mac OS
Python Python
Mac
• Python pip 3.x 2.7.9 2.x
Python pip Linux Python
pip yum apt
• Python Anaconda Python
conda
• python 2016
http://qiita.com/y__sama/items/5b62d31cb7e6ed50f02c

15. NumPy, SciPy, matplotlib, pandas
•
• NumPy SciPy
• pandas
pandas pandas NumPy
• Anaconda Python

16. Python
•
scikit-learn http://
scikit-learn.org/stable/

17. Python
• TensorFlow
Python

18. Python
IPython
Jupyter, …
IDE
Spyder, Rodeo
Visual Studio, PyCharm,
PyDev

19. • IPython
•
•
• Anaconda

20.
• Jupyter Notebook
• Python
• IPython Notebook
Python
• Apache Zeppelin http://
zeppelin.apache.org

21. IDE
• R RStudio
• IDE
•
• 2 Spyder Rodeo
•
Spyder

22. •
• Visual Studio
• Eclipse PyDev
• PyCharm
•

23. Part 2 :
Python / pandas

24. Python / pandas
• pandas
• /
etc…
•
Spark
• pandas

25. processing
•
• 64bit Python +
GB
• Python 1
1 CPU
GIL
• processing
Jenkins
CPU/
Jenkins

26. 1 1.2 1000000
‘abc’ ’ ’
[1, 2, 3, ‘foo’, ‘bar’, ‘foo’]
(1, 2, 3, ‘foo’, ‘bar’, ‘foo’)
{‘k1’: ‘value1’, ‘k2’: ‘value2’}
set(1, 2, 3, ‘foo’, ‘bar’)

27. •
•
• split
s = ‘foo, bar, baz’
items = s.split(‘,’)
print items[0]
print items[-1]
print items[0][-2:]

28. • ,
• lambda map, reduce, filter
sList = [‘foo’, ‘bar’, ‘baz’]
lList = [len(s) for s in sList]
lList = map(lambda s:len(s), sList)
lDict = {s:len(s) for s in sList}
lList = []
for s in sList:
lList.append(len(s))
lDict = {}
for s in sList:
lDict[s] = len(s)

29. pandas
• pandas
•
matplotlib / seaborn
• NumPy
SciPy
Python
• pandas + matplotlib
OK pandas NumPy
NumPy /
SciPy
https://openbook4.me/projects/183

31. pandas
• pandas
DataFrame
• R
• RDB
2
• index Series Columns
Columns
Series Series SeriesIndex

32. IDE /
• IDE
• jupyter notebook

33. • http://sinhrks.hatenablog.com/entry/2015/01/28/073327
0 1
import pandas as pd
df[‘nValue’] = df[‘value’] / sum(df[‘value’])
id value color
sapporo 43 red
osaka 42 pink
matsumoto 40 green
id value color nValue
sapporo 43 red 0.344
osaka 42 pink 0.336
matsumoto 40 green 0.32
Python

34. pandas I/O
• CSV JSON RDB Excel
• column
• RDB
•
import pandas as pd
pd.read_csv(<filename>)
pd.read_json(<filename>)
pd.to_csv(<filename>)
pd.to_excel(<filename>)
#
pd.to_clipboard()

35. pandas.read_csv
• pandas CSV
•
•
• usecols :
• nrows :
• na_values : na
• parse_dates infer_datetime_format:
• chunksize :
• compression : zip CSV
pandas.read_csv(filepath_or_buffer, sep=', ',
delimiter=None, header='infer', names=None,
index_col=None, usecols=None, squeeze=False,
prefix=None, mangle_dupe_cols=True, dtype=None,
engine=None, converters=None, true_values=None,
false_values=None, skipinitialspace=False,
skiprows=None, skipfooter=None, nrows=None,
na_values=None, keep_default_na=True,
na_filter=True, verbose=False,
skip_blank_lines=True, parse_dates=False,
infer_datetime_format=False,
keep_date_col=False, date_parser=None,
dayfirst=False, iterator=False, chunksize=None,
compression='infer', thousands=None, decimal='.',
lineterminator=None, quotechar='"', quoting=0,
escapechar=None, comment=None,
encoding=None, dialect=None, tupleize_cols=False,
error_bad_lines=True, warn_bad_lines=True,
skip_footer=0, doublequote=True,
delim_whitespace=False, as_recarray=False,
compact_ints=False, use_unsigned=False,
low_memory=True, buffer_lines=None,
memory_map=False, float_precision=None)

36. Spark - PySpark DataFrame API
•
Python
• Spark PySpark
findSpark
Spark
• Python Spark API
DataFrame API
• Spark pandas
Spark
PySpark
Spark
node
Spark
node
Spark
node
Spark
node
driver

37. •
•
Apache Arrow
• Python / R
:
feather
• pandas 2.0, parquet for Python

38. Python / pandas

39. Questions ?