In order to move past the hype and achieve the full potential of machine learning, data scientists and software developers need to work more closely together towards their common goal of delivering well-architected, data-driven applications. Every industry is in the process of being transformed by software and data. It is in the collaboration between data scientists and software developers where the real value can be found by creating integrated data workflows that benefit from the unique knowledge and skillsets of each discipline. https://www.dncexpo.be/seminar/O105

1. THE CONVERGENCE OF DATA SCIENCE
AND SOFTWARE DEVELOPMENT
Margriet Groenendijk
Developer Advocate | IBM
14 March 2018 | Cloud & Data Expo | Brussels | Belgium

2. DATA@MargrietGr

3. DATA SCIENTISTS VS. DEVELOPERS
Data Scientists Developers
Data Static data Dynamic databases
Code Python, R JavaScript
Platform Notebooks Text editors
Design Charts, Dashboards & Models Web apps
@MargrietGr

4. @MargrietGr
Extract Data Data Engineer
Prepare
Data
Data Scientist
Build & train
models
Data Scientist
Evaluate Business Analyst
Deploy DevOps
Use models Developer
Monitor DevOps
DATA SCIENCE IS A TEAM
SPORT

5. DATA SCIENTIST
§Extract & prepare data
§Analyze & visualize data
§Build & train models
@MargrietGr
Extract Data Data Engineer
Prepare
Data
Data Scientist
Build & train
models
Data Scientist
Evaluate Business Analyst
Deploy DevOps
Use models Developer
Monitor DevOps

6. SOFTWARE DEVELOPER
§Building application
§Front-end
§Back-end
@MargrietGr
Extract Data Data Engineer
Prepare
Data
Data Scientist
Build & train
models
Data Scientist
Evaluate Business Analyst
Deploy DevOps
Use models Developer
Monitor DevOps

7. @MargrietGr
DATA SCIENCE,
MACHINE LEARNING AND
ARTIFICIAL INTELLIGENCE

8. Artificial Intelligence
(Big) Data
Data Science
Deep Learning
Machine Learning
@MargrietGr

9. Artificial Intelligence
(Big) Data
Data Science
Deep Learning
Machine Learning
@MargrietGr
Cognitive Computing

10. MACHINE LEARNING
Use historical data to train a model
and use the model to make predictions
@MargrietGr

11. @MargrietGr
Machine Learning – Loan Approval
§ Output
Loan
Approval
Model
§ Train
§ AlgorithmJohn X
§ credit_score=800
§ age=25
§ income=$900,000
§ works in Oil & Gas
§ Historical Loans
Label
Approve

12. @MargrietGr
Machine Learning – Loan Approval
Approve
§ Output
Reject
James X
§ credit_score=900
§ age=55
§ income=$1,200,000
§ works in Insurance
§ New Applicant
Loan
Approval
Model

13. @MargrietGr
Deep Learning – Imaginary celebrities
https://arxiv.org/abs/1710.10196

14. @MargrietGr
DEMO LOCALCART
https://github.com/ibm-watson-data-lab/localcart-at-think-conf

15. THE PROJECT
§Customer behaviour information, such as demographics,
shopping cart values
§A recommendation engine to encourage additional
purchases based on past buying behaviour
@MargrietGr

16. EXTRACT & PREPARE DATA
@MargrietGr

17. EXPLORE DATA
@MargrietGr

18. OPEN-SOURCE IS KEY
@MargrietGr

19. @MargrietGr
JUPYTER NOTEBOOKS

20. @MargrietGr

21. @MargrietGr

22. @MargrietGr

23. @MargrietGr

24. @MargrietGr
PIXIEDUST
https://ibm-watson-data-lab.github.io/pixiedust/

25. PixieDust - open-source Python library for
Jupyter notebook to load and visualize data
@MargrietGr

26. PixieDust - open-source Python library for
Jupyter notebook to load and visualize data
PixieDebugger – first visual debugging tool for
Jupyter notebooks
@MargrietGr

27. PixieDust - open-source Python library for
Jupyter notebook to load and visualize data
PixieDebugger – first visual debugging tool for
Jupyter notebooks
PixieApps - create dashboards in a notebook
@MargrietGr

28. PixieDust - open-source Python library for
Jupyter notebook to load and visualize data
PixieDebugger – first visual debugging tool for
Jupyter notebooks
PixieApps - create dashboards in a notebook
PixieGateway - run charts or PixieApps as
standalone web applications
@MargrietGr

29. DISPLAY DATA AS A TABLE
@MargrietGr

30. @MargrietGr
DISPLAY DATA AS A CHART

31. @MargrietGr
DISPLAY DATA AS A CHART

32. @MargrietGr
DISPLAY DATA AS A CHART

33. @MargrietGr
CUSTOMERS BY
AGE IN A
HISTOGRAM
CHART

34. @MargrietGr
CUSTOMERS
CLUSTERED
BY GENDER

35. @MargrietGr
CUSTOMERS BY
STATE IN A
MAP

36. COMBINE
CUSTOMER DATA
WITH
CENSUS OPEN
DATA OF INCOME
BY ZIP CODE
@MargrietGr

37. THE PROJECT
§Customer behaviour information, such as demographics,
shopping cart values
§A recommendation engine to encourage additional
purchases based on past buying behaviour
@MargrietGr

38. BUILD,TRAIN AND DEPLOY A MODEL
@MargrietGr

39. @MargrietGr

40. TEST MODEL
@MargrietGr

41. TEST INTERACTIVELY WITH A PIXIEAPP
@MargrietGr

42. PIXIEAPP EXAMPLE – WEATHER FORECAST
§ PixieApps
§ Dashboards within a notebook
@MargrietGr

43. CONTINUOUS
LEARNING
https://medium.com/ibm-
watson-data-lab/keeping-
your-machine-learning-
models-up-to-date-
f1ead546591b

44. THE CONVERGENCE OF DATA SCIENCE AND
SOFTWARE DEVELOPMENT
Tools
Work Flow

45. PIXIEDUST
§ Data scientists use a Python notebook to load,
enrich, analyze data, and create analytics
@MargrietGr

46. PIXIEDUST
§ Data scientists use a Python notebook to load,
enrich, analyze data, and create analytics
§ From the same notebook, developers create a
PixieApp to operationalize these analytics
@MargrietGr

47. PIXIEDUST
§ Data scientists use a Python notebook to load,
enrich, analyze data, and create analytics
§ From the same notebook, developers create a
PixieApp to operationalize these analytics
§ Developers publish the PixieApp as a web
application
@MargrietGr

48. PIXIEDUST
§ Data scientists use a Python notebook to load,
enrich, analyze data, and create analytics
§ From the same notebook, developers create a
PixieApp to operationalize these analytics
§ Developers publish the PixieApp as a web
application
§ The PixieApp can be viewed interactively by line-
of-business users with no need to access the
notebook
@MargrietGr

49. Understand Business Define Approach
Define Data
Requirements
Access DataUnderstand DataPrepare Data
Create Models Evaluate Models Deploy & Monitor
@MargrietGr

50. BRINGING
THE RIGHT
TOOLS INTO
THE DATA
SCIENCE
WORK FLOW
§ Competitive advantage
§ Discover new insights
§ Real-time decision making
§ Reduce complexity and lower cost
§ Accelerate time to market and deployment of
data science and analytics
@MargrietGr

51. BRINGING
THE RIGHT
TOOLS INTO
THE DATA
SCIENCE
WORK FLOW
ONE GOAL
Develop data driven applications
Data science is maturing
NOW is the time for integration with software
development workflow
@MargrietGr

52. THANK YOU!
Margriet Groenendijk
—
mgroenen@uk.ibm.com
@MargrietGr
https://www.linkedin.com/in/margrietgroenendijk/
Slides
https://www.slideshare.net/MargrietGroenendijk/
presentations
Blog
https://medium.com/ibm-watson-data-lab
PixieDust
https://ibm-watson-data-lab.github.io/pixiedust/
Watson Data Platform
https://dataplatform.ibm.com/
IBM Cloud
https://ibm.com/cloud