Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015

800 views

Published on

Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015

Published in: Software

Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015

  1. 1. T I M E L I N E S E R V I C E N E X T G E N ( YA R N - 2 9 2 8 )
  2. 2. WHY NEXT GEN? Scalability Single global instance of writer/reader v.1 uses a local-disk-based LevelDB storage instance Usability Handle flows as first-class concept and model aggregation Elevate configuration and metrics to first-class members Existing external tooling: hRaven, Finch, Dr. Elephant, etc.
  3. 3. KEY DESIGN POINTS Distributed writer architecture Scalable storage backend (HBase) Reimagined object model API with flows built into it Separated reader instances Aggregation
  4. 4. DISTRIBUTED WRITERS & READERS !meline reader !meline reader Storage !meline reader AM !meline writer NM !meline reader pool app metrics/events container events/metrics RM !meline writer app/container events user queries
  5. 5. STATUS [DONE] timeline writers (per-app and per-node) as aux service [DONE] RM companion writer [DONE] first iteration of the object model API [DONE] file-based test writer [DONE] NM writing container events [DONE] RM writing app/container entities [DONE] AMs writing framework-specific events and metrics [DONE] first versions of Phoenix and HBase writer impls [DONE] performance benchmarking evaluation of writers
  6. 6. STATUS [WIP] timeline readers [WIP] aggregation UI enhancements Stand-alone timeline writer (per-node and per-app) Finalize implementation of supported queries Security Migration/compatibility story …
  7. 7. TEAM This is a true community collaboration! Sangjin, Vrushali and Joep (Twitter) Zhijie, Li, Junping and Vinod (Hortonworks) Naga and Varun (Huawei) Robert and Karthik (Cloudera) Input from LinkedIn, Yahoo! and Altiscale
  8. 8. QUESTIONS?

×