Downloaded 21 times
Uploaded byZhijie Shen
Hadoop Summit San Jose 2014 - Analyzing Historical Data of Applications on Hadoop YARN: for Fun and Profit
The document discusses the need for a new application history server and timeline service for Hadoop YARN, highlighting the limitations of the existing job history server, particularly for diverse applications. It outlines the architecture and features of the proposed application history server, including pluggable storage options and improved scalability, as well as the functionality of the timeline service. The presentation concludes with future work plans and the necessity for solutions that can handle larger scales for data storage.
More Related Content
Similar to Hadoop Summit San Jose 2014 - Analyzing Historical Data of Applications on Hadoop YARN: for Fun and Profit
Hadoop Summit San Jose 2014 - Analyzing Historical Data of Applications on Hadoop YARN: for Fun and Profit
- 1. Analyzing Historical Dataof Applications on HadoopYARN: for Fun and Profit Mayank Bansal，Zhijie Shen
- 2. Agenda • Who weare ? • Why we need New History Server? • Application History Server • Timeline Server • Future Work
- 3. Who we are •Hadoop Architect @ ebay • Apache Hadoop Committer • Apache Oozie PMC and Committer • Current • Leading Hadoop Core Development for YARN and MapReduce @ ebay • Past • Working on Scheduler / Resource Managers • Working on Distributed Systems • Data Pipeline frameworks Mayank Bansal
- 4. Who we are •Software Engineer @ Hortonworks • Apache Hadoop Committer • Apache SAMZA PPMC and Committer Zhijie Shen
- 5. Agenda • Who weare ? • Why we need New History Server? • Application History Server • Application Timeline Server • Future Work
- 6. MR JobHistory Server •We already have Job History Server • It is only for Map Reduce Customized • Storage is HDFS only • Storage is very MR specific • Counters • Mappers and Reducers • If you have only Map Reduce you are good.
- 7. Hadoop-2 Single Use System BatchApps Multi Purpose Platform Batch, Interactive, streaming
- 8.
- 9. Issues with currentJob History • What if I have other Applications • RM crashes • Hard Limit on # Apps • Upgrades / Updates
- 10. Agenda • Who weare ? • Why we need New History Server? • Application History Server • Timeline Server • Future Work
- 11. Application History Server •Separate Process • Pluggable Storage • HDFS • In-Memory • Resource Manager directly writes to Storage • Aggregated Logs • Separate UI, CLI and Rest End Point
- 12. Application History Server Storage: •It stores generic Data • Application level data (queue, user etc…) • List of ApplicationAttempts • Information about each ApplicationAttempt • List of containers for ApplicationAttempt • Generic information about each container.
- 13.
- 14. Application History Server •CLI Interface $ yarn application -status <Application ID> $ yarn applicationattempt -list <Application ID> • REST APIs • http://localhost:8188/ws/v1/applicationhistory/app s/appid
- 15. Application History Server •Scalability for storage • One file per application • File format is protobuff • Size of HDFS files • Multiple RM threads writing to History Storage # of Containers 100 1K 10 K 100K Size of the File 19 KB 184 KB 1.8 MB 19 MB
- 16. Agenda • Who weare ? • Why we need New History Server? • Application History Server • Timeline Server • Future Work
- 17. Timeline Service -Motivation • YARN takes care of it – Relieving the application from monitoring service • Application diversity – Framework specific metadata/metrics
- 18. Timeline Service –Data Model • Entity Type – An abstract concept of anything • Entity – One specific instance of a entity type – Defining the relationship between entities • Event – Something happens to an entity
- 19. Timeline Service –Architecture • LevelDB Store • Client Library • REST Interfaces
- 20. Timeline Service –Store • LevelDB based store – Key-value store – Lightweight – License compatible • Implementing reader/writer interfaces • Support data retention
- 21. Timeline Service –Client • TimelineClient – Wrap over REST POST method – POJO objects • TimelineEntity • TimelineEvent – In Client/AM/Container
- 22. Timeline Service –APIs • Rest APIs, JSON as the media • Get timeline entities – http://localhost:8188/ws/v1/timeline/{entityType} • Get timeline entity – http://localhost:8188/ws/v1/timeline/{entityType}/{entityId} • Get timeline events – http://localhost:8188/ws/v1/timeline/{entityType}/events
- 23. Timeline Service –Security • HTTP SPNEGO • Kerberos Authentication • Delegation Token – Performance – AM/Container no Kerberos • Access Control – Admin/owner – Timeline entity-level Browser or REST clientTimeline Client Timeline Client Client AM 2. Pass DT via AppSubmissionContext 1. Get DT Put Data via Kerberos or DTs 3. Put Data w/ DT Timeline Authen ca on Filter Timeline REST APIs Timeline ACLs Manager Access Verifica on Pass authen cated user informa on Get Data via Kerberos HadoopsupportforKerberos,SPNEGO,ACLs Timeline Service security Timeline Server
- 24. Timeline Service –Use Case (1)
- 25. Timeline Service –EarlyAdopter (2)
- 26. Timeline Service –Early Adopter (3)
- 27. Agenda • Who weare ? • Why we need New History Server? • Application History Server • Timeline Server • Future Work
- 28. To Be Continue… •Integrating the generic history and timeline data • Rebasing MR Job history server on the timeline server • Making the timeline server rendering the timeline data
- 29. To Be Continue… Scale •Leveldb does not handle ebay scale • We need something which can horizontally scale • HBASE
- 30.
Editor's Notes
- #31 This is a topic/content slide. Duplicate as many of these as are needed. Generally, there is one slide per three minutes of talk time.