Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURIC...
Agenda
About the project
Systems Architecture
Do’s and Don'ts in cluster operation
BigData Administration - or you're not ...
Operating MapR Haddop3 14.09.18
About the project
About the project
Operating MapR Haddop4 14.09.18
Measuring data management in automotive sector
Data can‘t be handled any...
Workflow
Operating MapR Haddop5 14.09.18
Measuring data is collected due test drives
Driver presses a button
Snapshot of a...
Why BigData (3Vs)
Operating MapR Haddop6 14.09.18
Volume
– 100 PB in 2018
– 600 PB in 2020
Variety
– Data format depending...
Target architecture
Operating MapR Haddop7 14.09.18
Tasks for us
Operating MapR Haddop8 14.09.18
Architecture of the Big Data infrastructure
Take care of the Big Data infrast...
Operating MapR Haddop9 14.09.18
Architecture
MapR Hadoop
Operating MapR Haddop10 14.09.18
Founded by former Google employees
Hadoop distribution developed by MapR Tech...
MapR Hadoop
Operating MapR Haddop11 14.09.18
Quite different to other Hadoop distributions (like Cloudera, Hortonworks,…)
...
MapR Hadoop
Operating MapR Haddop12 14.09.18
MapR-FS
– Implemented in C
– No NameNode, CLDB (container
location database) ...
Hardware
Operating MapR Haddop13 14.09.18
24 server
– 256GB to 512GB RAM
– 10TB to 80TB storage
– 16- 32 CPUs per Node
Two...
OS installation
Operating MapR Haddop14 14.09.18
Red Hat Enterprise Linux 7.2
Automated installation over Network (PXE) wi...
System architecture
Operating MapR Haddop15 14.09.18
Separation of Control and Worker Nodes
Control node’s:
– Zookeeper
– ...
System architecture
Operating MapR Haddop16 14.09.18
Worker node’s:
– MapR-FS
– Node manager
– Local NFS gateway (special
...
System architecture
Operating MapR Haddop17 14.09.18
Role allocation done according to best practices and documentation
– ...
Operating MapR Haddop18 14.09.18
Do’s and Don’ts
Do‘s and Don‘ts
Operating MapR Haddop19 14.09.18
Establish resource sharing strategy 
Use central management for clusters...
Do‘s and Don‘ts
Operating MapR Haddop20 14.09.18
Docker executed by YARN  
– Strategy necessary (Node labeling, Firewall...
Operating MapR Haddop21 14.09.18
BigData Administration
Big Data Administration
Operating MapR Haddop22 14.09.18
Administering a Big Data Cluster is more than administering the
c...
Big Data Administration
Operating MapR Haddop23 14.09.18
Yarn / Container Logs
Spark Logs OS Logs
Oozie Logs
Custom Shell ...
Big Data Administration
Operating MapR Haddop24 14.09.18
Requirements are changing/evolving quickly
– Short-term changes o...
Operating MapR Haddop25 14.09.18
Pitfalls
Pitfalls
Operating MapR Haddop26 14.09.18
Different understanding how a cluster works
– Dev Team <-> Infra Team
Suboptimal...
Pitfalls
Operating MapR Haddop27 14.09.18
Proprietary software to load data to the cluster
High effort for apparently smal...
Operating MapR Haddop28 14.09.18
Lessons learned
Lessons learned
Operating MapR Haddop29 14.09.18
Security: Plan and implement from the beginning
– Advanced security might...
Lessons learned
Operating MapR Haddop30 14.09.18
Close work with app/development team
– Big Data team
High effort for trou...
Ressource Management
Operating MapR Haddop31 14.09.18
<allocations>
<defaultQueueSchedulingPolicy>drf</defaultQueueSchedul...
Operating MapR Haddop32 14.09.18
Further Information
https://mapr.com/docs/
https://hadoop.apache.org/
http://hadoop.apach...
Markus Bente
Senior Consultant
markus.bente@trivadis.com
14.09.18 Operating MapR Haddop33
@trivadis @muehlbeyer
Michael Mü...
Upcoming SlideShare
Loading in …5
×

TechEvent Operating MapR Hadoop Cluster for a year

100 views

Published on

TechEvent Operating MapR Hadoop Cluster for a year

Published in: Technology
  • Be the first to comment

  • Be the first to like this

TechEvent Operating MapR Hadoop Cluster for a year

  1. 1. BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH Operating MapR Hadoop Cluster for a year Markus Bente Michael Mühlbeyer @trivadis @muehlbeyer
  2. 2. Agenda About the project Systems Architecture Do’s and Don'ts in cluster operation BigData Administration - or you're not an infrastructure Admin anymore ;-) Pitfalls Lessons learned 14.09.18 Operating MapR Haddop2
  3. 3. Operating MapR Haddop3 14.09.18 About the project
  4. 4. About the project Operating MapR Haddop4 14.09.18 Measuring data management in automotive sector Data can‘t be handled anymore by their legacy systems
  5. 5. Workflow Operating MapR Haddop5 14.09.18 Measuring data is collected due test drives Driver presses a button Snapshot of all tracefiles 1 min before until 1 min after is collected Snapshot is stored on disks in the car Tracefiles will be transferred to analysis systems Tracefiles have to be analyzed asap and result displayed Measuring data is collected due test drives Driver presses a button Snapshot of all tracefiles 1 min before until 1 min after is collected Snapshot is stored on disks in the car Tracefiles will be transferred to analysis systems Tracefiles have to be analyzed asap and result displayed
  6. 6. Why BigData (3Vs) Operating MapR Haddop6 14.09.18 Volume – 100 PB in 2018 – 600 PB in 2020 Variety – Data format depending on test equipment manufacturer – Sometimes even depending on test case Velocity – Test engineers need the results asap to continue with tests – Time due doubled test cases should be reduced
  7. 7. Target architecture Operating MapR Haddop7 14.09.18
  8. 8. Tasks for us Operating MapR Haddop8 14.09.18 Architecture of the Big Data infrastructure Take care of the Big Data infrastructure – OS – Hadoop Distribution – Necessary tools Implement a security solution Know How transfer
  9. 9. Operating MapR Haddop9 14.09.18 Architecture
  10. 10. MapR Hadoop Operating MapR Haddop10 14.09.18 Founded by former Google employees Hadoop distribution developed by MapR Technologies MapR-FS MapR-DB MapR-Streams Available as Enterprise and as Community Edition
  11. 11. MapR Hadoop Operating MapR Haddop11 14.09.18 Quite different to other Hadoop distributions (like Cloudera, Hortonworks,…) – Uses MapR-FS instead HDFS – No NameNode Architecture  No Single Point of Failure Architecture – Improved NFS access – Ability of one global namespace across different clusters and storage tiers
  12. 12. MapR Hadoop Operating MapR Haddop12 14.09.18 MapR-FS – Implemented in C – No NameNode, CLDB (container location database) instead – Automatic failover – 20x higher performance – Up to 1 trillion files – Different replication mechanisms FA A A E E E B B B B C D D F FC DC Namenode A B Namenode C D Namenode E F MapR-FS Classic Hadoop
  13. 13. Hardware Operating MapR Haddop13 14.09.18 24 server – 256GB to 512GB RAM – 10TB to 80TB storage – 16- 32 CPUs per Node Two clusters – Development – Pilot
  14. 14. OS installation Operating MapR Haddop14 14.09.18 Red Hat Enterprise Linux 7.2 Automated installation over Network (PXE) with Kickstart Configuration management with Puppet – Hardening – User management – MapR basic installation
  15. 15. System architecture Operating MapR Haddop15 14.09.18 Separation of Control and Worker Nodes Control node’s: – Zookeeper – CLDB – MapR-FS – External NFS access – Resource manager (YARN) – Monitoring tools – All services which are provided by the cluster
  16. 16. System architecture Operating MapR Haddop16 14.09.18 Worker node’s: – MapR-FS – Node manager – Local NFS gateway (special requirement) – Metric and log file collection daemons
  17. 17. System architecture Operating MapR Haddop17 14.09.18 Role allocation done according to best practices and documentation – Trivadis Big Data infrastructure blue prints High availability due usage of MapR Enterprise Edition features – Resource Manager – File Server Automatic failover of services possible with Zookeeper/Warden Active/Passive or Active/Active HA possible – Depends on service: Oozie (Active/Active)
  18. 18. Operating MapR Haddop18 14.09.18 Do’s and Don’ts
  19. 19. Do‘s and Don‘ts Operating MapR Haddop19 14.09.18 Establish resource sharing strategy  Use central management for clusters config files  Workshop with DEV and OPS team  Automate as much as possible 
  20. 20. Do‘s and Don‘ts Operating MapR Haddop20 14.09.18 Docker executed by YARN   – Strategy necessary (Node labeling, Firewall, …) Separate MySQL db from actual cluster  Grafana 
  21. 21. Operating MapR Haddop21 14.09.18 BigData Administration
  22. 22. Big Data Administration Operating MapR Haddop22 14.09.18 Administering a Big Data Cluster is more than administering the core infrastructure – Mindset change necessary Users of the cluster expect you to know all about the cluster ;-) – How to access services – How to run code – How to configure their client/VM Troubleshooting requires deep knowledge of all involved components
  23. 23. Big Data Administration Operating MapR Haddop23 14.09.18 Yarn / Container Logs Spark Logs OS Logs Oozie Logs Custom Shell Action Logs Lets assume we only have 3 Tasks in the Workflow 1x M/R 1x Spark 1x Shell Execute with Spark Submit Rest API Call Oozie Workflow Start Yarn / Container Logs Yarn / Container Logs First task M/R Second task spark Third task custom Shell action with Spark submit Spark Logs
  24. 24. Big Data Administration Operating MapR Haddop24 14.09.18 Requirements are changing/evolving quickly – Short-term changes on cluster environment necessary Configure and customize the cluster software with minimal documentation from MapR
  25. 25. Operating MapR Haddop25 14.09.18 Pitfalls
  26. 26. Pitfalls Operating MapR Haddop26 14.09.18 Different understanding how a cluster works – Dev Team <-> Infra Team Suboptimal Code 600 TB storage (gross) aren‘t enough ;-) Some features are , let‘s say documented, but don‘t work as described
  27. 27. Pitfalls Operating MapR Haddop27 14.09.18 Proprietary software to load data to the cluster High effort for apparently small tasks Keeping the Big Pictures is quite hard MapR components might behave different than the open source versions
  28. 28. Operating MapR Haddop28 14.09.18 Lessons learned
  29. 29. Lessons learned Operating MapR Haddop29 14.09.18 Security: Plan and implement from the beginning – Advanced security might create further issues Be aware of the Big Picture to implement security in Hadoop/MapR Performance isn‘t everything – Stability might be more important
  30. 30. Lessons learned Operating MapR Haddop30 14.09.18 Close work with app/development team – Big Data team High effort for troubleshooting End-to-End View
  31. 31. Ressource Management Operating MapR Haddop31 14.09.18 <allocations> <defaultQueueSchedulingPolicy>drf</defaultQueueSchedulingPolicy> <queue name="example"> <minResources>25600 mb,8vcores,0 disks</minResources> <maxResources>122880 mb,96vcores,80 disks</maxResources> <maxRunningApps>10</maxRunningApps> <label>docker</label> </queue>
  32. 32. Operating MapR Haddop32 14.09.18 Further Information https://mapr.com/docs/ https://hadoop.apache.org/ http://hadoop.apache.org/docs/current/
  33. 33. Markus Bente Senior Consultant markus.bente@trivadis.com 14.09.18 Operating MapR Haddop33 @trivadis @muehlbeyer Michael Mühlbeyer Senior Consultant michael.muehlbeyer@trivadis.com

×