A data scientist's daily life involves collecting and storing large amounts of data from various sources, preprocessing and analyzing the data using tools like Linux, SQL databases, Python and R, and applying machine learning algorithms like clustering, classification, and regression to derive insights. The data scientist must effectively manage terabytes of data and choose the appropriate machine learning techniques and algorithms to gain knowledge from big data in an efficient and intelligent manner. Visualization tools are then used to showcase the findings and insights discovered.

1. DATA SCIENTIST’S
DAILY LIFE
BRYAN YANG 2015.09

2. A B O U T M E
• Blog
Bryan的行銷研究及資料分析筆記
http://bryannotes.blogspot.tw
• Group
Spark.TW

3. A G E N D A
• Data scientist?
• Big data and data scientist
• Data scientist’s Toolbox
• Data is the biggest

4. Derive
Knowledge
from
Big data

5. Efficiently
and
Intelligently

6. F R O M B A C K E N D T O F R O N T E N D
https://doubleclix.wordpress.com/2012/12/15/what-or-who-is-a-data-scientist/

7. WHAT IS BIG DATA?

8. WHERE DO THE DATA COME FROM
• Web Log data
• Machine data
• Transactional data
• Social media data
• …

9. https://plus.google.com/+DigitalStrategyIE

11. A WEB SERVICE RECEIVE THE LOG DATA MORE THEN 50G PER DAY
TOTAL SPACE USED LAST THREE MONTH ：4500G
TOTAL SPACE USED LAST ONE YEAR ：18,000G(17.6T)

12. • Data Storage/ Backup
• 2T/per HDD
• How to save the data MORE than 2T?
• $0.3 USD/per gigabyte
• Pay 900 USR for KEEPING data but do nothing else.
• Read/Write Speed
• Read: 131.6 MB/s / Write 131.4MB/s
• Spend 393s(6 min) reading just ONE day data.
• Large number of transactions immediately

13. HADOOP AND
MAPREDUCE

14. H A D O O P A N D H D F S
http://www.fraudtechwire.com/f-level-guide-to-hadoop-hdfs/

17. – D I S T R I B U T E D A L G O R I T H M
「The world will change，when data is distributed」

18. M A P R E D U C E
http://www.milanor.net/blog/?p=853

19. https://chamibuddhika.wordpress.com/2012/02/26/joins-with-map-reduce/

20. http://blog.agro-know.com/?p=3810

21. P E R F O R M A N C E O F H A D O O P ?
• Not good, but at least can run.
• Count 86,389,084 rows/per day in 39 sec.
(64G ram, E5 8core * 2/per node * 10)
• How about 39sec * 30days ?

22. BEFORE ANALYTIC…

23. E X T R A C T T R A S F O R M L O A D
m/e-university/data-warehouse-etl-toolkit-tutorial-201/surrounding-the-requirements-1

24. e.net/capgemini/emc-world-2014-breakout-move-to-the-business-data-lake-not-a

25. /hortonworks/modern-data-architecture-for-a-data-lake-with-informatica-and-h

26. DATA SCIENTIST’S
TOOL BOX

27. L I N U X
• The best server choice
• Free and freedom
• Easy to control system
• Easy data processing
• Hadoop is based on Linux

29. P O W E R F U L S H E L L S C R I P T

30. S Q L D A T A B A S E
• MySql, Postgresql, Hive, MongoDB(NOSQL)
• Standard SQL Language
• Store and Manage data

31. R E L A T I O N A L D A T A B A S E

32. T A B L E R E L A T I O N
https://cloudant.com/blog/foundbites-data-model-relational-db-vs-nosql-on-cloudant/

33. http://ghtorrent.org/relational.html

34. S Q L S Y N T A X

35. R & P Y T H O N
• Basic Analysis Tools
• Easy to Learn
• Many Packages

38. • Example
• http://bryannotes.blogspot.tw/2014/08/r-ptt-
wantedsocial-network-analysis.html
• http://bryannotes.blogspot.tw/2014/10/python-k-
means-script.html

39. E T C …
• Excel
• Google Analytics
• Visualisation tools (tableau)
• Web Crawler
• Version control management (git)
• ETL and job scheduling tools (jenkins)
• …

40. D A T A I S T H E B I G G E S T

41. – J O S H W I L L S
“Person who is better at statistics than any software
engineer and better at software engineering than
any statistician.”

42. S T A T I S T I C

43. W H Y D O W E N E E D M A C H I N E
L E A R N I N G ?
• Clustering
這些人可以分成幾類
• Classification
哪個人屬於哪一類？
• Regression
某個事件發生或某人屬於哪類的機率是多少？
• Dimensionality reduction
降維

44. C L U S T E R I N G
http://simplystatistics.org/2014/02/18/k-means-clustering-in-a-gif/
source http://humble-developer.blogspot.tw/2011/01/kmeans-clustering-algorithm-part-1.html

45. C L A S S I F I C A T I O N
http://letsmakerobots.com/content/tcs3200-color-sensor-with-k-nearest-neighbor-classification-algorithm

46. http://www.astroml.org/sklearn_tutorial/

47. L O G I S T I C R E G R E S S I O N
https://www.coursera.org/instructor/andrewng

48. C O S T F U N C T I O N
https://www.coursera.org/instructor/andrewng

49. O V E R F I T T I N G
https://www.coursera.org/instructor/andrewng

50. OH MY GOD!
HOW TO CHOOSE IT

51. M A C H I N E L E A R N I N G A L G O R I T H M N
http://amueller.github.io/sklearn_tutorial/

52. S T A T I S T I C V S M L
S T A T T I S T I C
M A C H I N E
L E A R N I N G
F O C U S O N
U N D E R S T A N D I N G D A T A
I N T E R M S O F M O D E L S
F O C U S O N T H E A N A L Y S I S
O F L E A R N I N G
A L G O R I T H M S
I N T E R P R E T A B I L I T Y ,
H Y P O T H E S I S T E S T I N G
G R E A T E R F O C U S O N
P R E D I C T I O N

53. S Y S T E M A T I C S A N D A U T O M A T I O N
http://www.slideshare.net/CetasAnalytics/cetas-e-baymeetupprezofinal

54. http://mlg.postech.ac.kr/projects/

55. SHOW YOUR DATA AND
FINDINGS

56. http://hortonworks.com/wp-content/uploads/2012/06/Tableau2.png

57. http://www.tableau.com

58. http://www.tableau.com

59. http://www.tableau.com

60. THE REAL CASE

61. HOW TO START?

62. • Codecademy http://www.codecademy.com/
Include kinds of programming language, i.e. python,
JavaSrtipt, even shell script and sql
• Coursera http://www.codecademy.com/
Famous self-learning MOOC website.

63. http://nirvacana.com/thoughts/becoming-a-data-scientist/