Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Analyzing Historical Data of Applications on YARN for Fun and Profit

3,461 views

Published on

Published in: Technology
  • Be the first to comment

Analyzing Historical Data of Applications on YARN for Fun and Profit

  1. 1. Analyzing Historical Data of Applications on HadoopYARN: for Fun and Profit Mayank Bansal,Zhijie Shen
  2. 2. Agenda • Who we are ? • Why we need New History Server? • Application History Server • Timeline Server • Future Work
  3. 3. Who we are • Hadoop Architect @ ebay • Apache Hadoop Committer • Apache Oozie PMC and Committer • Current • Leading Hadoop Core Development for YARN and MapReduce @ ebay • Past • Working on Scheduler / Resource Managers • Working on Distributed Systems • Data Pipeline frameworks Mayank Bansal
  4. 4. Who we are • Software Engineer @ Hortonworks • Apache Hadoop Committer • Apache SAMZA PPMC and Committer Zhijie Shen
  5. 5. Agenda • Who we are ? • Why we need New History Server? • Application History Server • Application Timeline Server • Future Work
  6. 6. MR JobHistory Server • We already have Job History Server • It is only for Map Reduce Customized • Storage is HDFS only • Storage is very MR specific • Counters • Mappers and Reducers • If you have only Map Reduce you are good.
  7. 7. Hadoop-2 Single Use System Batch Apps Multi Purpose Platform Batch, Interactive, streaming
  8. 8. YARN
  9. 9. Issues with current Job History • What if I have other Applications • RM crashes • Hard Limit on # Apps • Upgrades / Updates
  10. 10. Agenda • Who we are ? • Why we need New History Server? • Application History Server • Timeline Server • Future Work
  11. 11. Application History Server • Separate Process • Pluggable Storage • HDFS • In-Memory • Resource Manager directly writes to Storage • Aggregated Logs • Separate UI, CLI and Rest End Point
  12. 12. Application History Server Storage: • It stores generic Data • Application level data (queue, user etc…) • List of ApplicationAttempts • Information about each ApplicationAttempt • List of containers for ApplicationAttempt • Generic information about each container.
  13. 13. Application History Server
  14. 14. Application History Server • CLI Interface $ yarn application -status <Application ID> $ yarn applicationattempt -list <Application ID> • REST APIs • http://localhost:8188/ws/v1/applicationhistory/app s/appid
  15. 15. Application History Server • Scalability for storage • One file per application • File format is protobuff • Size of HDFS files • Multiple RM threads writing to History Storage # of Containers 100 1K 10 K 100K Size of the File 19 KB 184 KB 1.8 MB 19 MB
  16. 16. Agenda • Who we are ? • Why we need New History Server? • Application History Server • Timeline Server • Future Work
  17. 17. Timeline Service - Motivation • YARN takes care of it – Relieving the application from monitoring service • Application diversity – Framework specific metadata/metrics
  18. 18. Timeline Service – Data Model • Entity Type – An abstract concept of anything • Entity – One specific instance of a entity type – Defining the relationship between entities • Event – Something happens to an entity
  19. 19. Timeline Service – Architecture • LevelDB Store • Client Library • REST Interfaces
  20. 20. Timeline Service – Store • LevelDB based store – Key-value store – Lightweight – License compatible • Implementing reader/writer interfaces • Support data retention
  21. 21. Timeline Service – Client • TimelineClient – Wrap over REST POST method – POJO objects • TimelineEntity • TimelineEvent – In Client/AM/Container
  22. 22. Timeline Service – APIs • Rest APIs, JSON as the media • Get timeline entities – http://localhost:8188/ws/v1/timeline/{entityType} • Get timeline entity – http://localhost:8188/ws/v1/timeline/{entityType}/{entityId} • Get timeline events – http://localhost:8188/ws/v1/timeline/{entityType}/events
  23. 23. Timeline Service – Security • HTTP SPNEGO • Kerberos Authentication • Delegation Token – Performance – AM/Container no Kerberos • Access Control – Admin/owner – Timeline entity-level Browser or REST clientTimeline Client Timeline Client Client AM 2. Pass DT via AppSubmissionContext 1. Get DT Put Data via Kerberos or DTs 3. Put Data w/ DT Timeline Authen ca on Filter Timeline REST APIs Timeline ACLs Manager Access Verifica on Pass authen cated user informa on Get Data via Kerberos HadoopsupportforKerberos,SPNEGO,ACLs Timeline Service security Timeline Server
  24. 24. Timeline Service – Use Case (1)
  25. 25. Timeline Service – EarlyAdopter (2)
  26. 26. Timeline Service – Early Adopter (3)
  27. 27. Agenda • Who we are ? • Why we need New History Server? • Application History Server • Timeline Server • Future Work
  28. 28. To Be Continue… • Integrating the generic history and timeline data • Rebasing MR Job history server on the timeline server • Making the timeline server rendering the timeline data
  29. 29. To Be Continue… Scale • Leveldb does not handle ebay scale • We need something which can horizontally scale • HBASE
  30. 30. Questions 30 Mayank Bansal mabansal@ebay.com mayank@apache.org Zhijie Shen zshen@hortonworks.com zjshen@apache.org

×