A presentation discussing how to deploy Big data solutions. The difference between structured reporting systems which feed business processes and the data science systems which do cool stuff

1. LEARN • NETWORK • COLLABORATE • INFLUENCE

2. LEARN • NETWORK • COLLABORATE • INFLUENCE
Deploying Big Data platforms
LEARN • NETWORK • COLLABORATE • INFLUENCE
Chris Kernaghan
Principal Consultant

3. LEARN • NETWORK • COLLABORATE • INFLUENCE
Cholera epidemic first use of big data

4. LEARN • NETWORK • COLLABORATE • INFLUENCE
Big Data Epidemiology by Google

5. LEARN • NETWORK • COLLABORATE • INFLUENCE
How I really got started in Big Data
John, we need
to give Chris
more grey hair
Let’s throw him
into a Big Data
demo

6. LEARN • NETWORK • COLLABORATE • INFLUENCE
My examples

7. LEARN • NETWORK • COLLABORATE • INFLUENCE

8. LEARN • NETWORK • COLLABORATE • INFLUENCE
Areas of focus
Data acquisition
and curation
Data storage Compute
infrastructure
Analysis and
Insight
Everything as Code*
* Well As much as possible

9. LEARN • NETWORK • COLLABORATE • INFLUENCE
Data Acquisition and curation
Areas of focus

10. LEARN • NETWORK • COLLABORATE • INFLUENCE
Data Lake
HANA

11. LEARN • NETWORK • COLLABORATE • INFLUENCE
How big was the Panama Papers data set

12. LEARN • NETWORK • COLLABORATE • INFLUENCE
How big was the Panama Papers data set

13. LEARN • NETWORK • COLLABORATE • INFLUENCE
Data Lake
Panama Papers Technology stack
SQL

14. LEARN • NETWORK • COLLABORATE • INFLUENCE
The tools used supported 370 journalists from
around the world
Infrastructure
was a pool of
up to 40
servers run in
AWS

15. LEARN • NETWORK • COLLABORATE • INFLUENCE
Data quality and curation are not one time activities
Remove the human element as much as possible

16. LEARN • NETWORK • COLLABORATE • INFLUENCE
Data security
• Data lake
– What data do you collect
– Do you have restrictions on what data can be combined
– How long does your data live

17. LEARN • NETWORK • COLLABORATE • INFLUENCE
Data security
• Geographical concerns
– Where does your data reside

18. LEARN • NETWORK • COLLABORATE • INFLUENCE
Data security
• Authentication
– Who is accessing your data

19. LEARN • NETWORK • COLLABORATE • INFLUENCE
Data Storage
Areas of focus

20. LEARN • NETWORK • COLLABORATE • INFLUENCE
How BIG is Big Data

21. LEARN • NETWORK • COLLABORATE • INFLUENCE

22. LEARN • NETWORK • COLLABORATE • INFLUENCE
Storage Considerations
• IOPS are still important
– Big data still uses a lot of spinning disk
• Replication and Redundancy
– Eats a lot of disk space
• Build for failure
• Sometimes you have to go in-memory

23. LEARN • NETWORK • COLLABORATE • INFLUENCE
Compute infrastructure
Areas of focus

24. LEARN • NETWORK • COLLABORATE • INFLUENCE
Structured Reporting Versus Big Data/Science
Compute requirements
2
• Structured reporting systems run business processes
– Sized and static
– Under change control
– Business centric

25. LEARN • NETWORK • COLLABORATE • INFLUENCE
Structured Reporting Versus Big Data/Science
Compute requirements
2
• Data science systems answer difficult questions irregularly
– Cloud or heavy use of virtualisation
– Developer centric
– Rapidly evolving

26. LEARN • NETWORK • COLLABORATE • INFLUENCE
What you still need to remember
2
• Compute is cheap
• Scalability is critical

27. LEARN • NETWORK • COLLABORATE • INFLUENCE
What you still need to remember
2
• Software definition for consistency
• Automate as much as possible

28. LEARN • NETWORK • COLLABORATE • INFLUENCE
2
100 Hadoop
Nodes
122GB RAM
Each = 12.2TB RAM
Build time of 3Hrs

29. LEARN • NETWORK • COLLABORATE • INFLUENCE
Use of scripted builds from VM to application
2
Disk definition
Network
defintion
Software
Install

30. LEARN • NETWORK • COLLABORATE • INFLUENCE
Use of scripted builds from VM to application
3
• Deployment was consistent for each and every node of the
cluster
– Hostnames defined the same way
– Configuration files created the same way

31. LEARN • NETWORK • COLLABORATE • INFLUENCE
Use of scripted builds from VM to application
3
• Faster deployment
– Automated build 3hrs to build and deploy 100 nodes
– Manual build 800hrs + to build and deploy 100 nodes
• Use of automated tools to detect failure and start new node
(ElasticBeanstalk)

32. LEARN • NETWORK • COLLABORATE • INFLUENCE
Use of scripted builds from VM to application
3
• Reusability of script
– Heavy use of parameters means it is adaptable
• Use of Git meant distributed development was handled easily

33. LEARN • NETWORK • COLLABORATE • INFLUENCE

34. LEARN • NETWORK • COLLABORATE • INFLUENCE
Analysis and Insight
3
Areas of focus
Presentation Tag Line

35. LEARN • NETWORK • COLLABORATE • INFLUENCE
Query the Data
• Programmatically
– Python
– R
• Application
– Lumira
– Business Objects
– Spark
– SQL
– Excel
– ElasticSearch

36. LEARN • NETWORK • COLLABORATE • INFLUENCE
Analysis and Visualisation
• Quick Analysis
– Lumira, Excel
• Graph
– Neo4J, Synerscope
• Charts
– Business Objects, Grafana, Kibana
• Dynamic
– D3
http://www.wikiviz.org/wiki/Tools

37. LEARN • NETWORK • COLLABORATE • INFLUENCE
Things to remember
• Remember the type of
platform you are using
• Storage is cheap but not
all storage is equal
• Scalability is critical
• Version control rocks
• Automate everything
you can
• Value is in the data but
not all data is valuable
• Data should not live
forever

38. LEARN • NETWORK • COLLABORATE • INFLUENCE
3
• Key Takeways

39. LEARN • NETWORK • COLLABORATE • INFLUENCE