This document discusses Timeline Service Next Gen (YARN-2928), a scalable, distributed timeline service for YARN. It aims to improve usability, scalability, and the object model. Key points include distributed writers and readers, a scalable storage backend, and support for flows and aggregation. Status sections note work in progress on readers, aggregation, and other features. The project involves collaboration between engineers at Twitter, Hortonworks, Huawei, and Cloudera.

1. T I M E L I N E S E R V I C E N E X T G E N
( YA R N - 2 9 2 8 )

2. WHY NEXT GEN?
Scalability
Single global instance of writer/reader
v.1 uses a local-disk-based LevelDB storage instance
Usability
Handle flows as first-class concept and model aggregation
Elevate configuration and metrics to first-class members
Existing external tooling: hRaven, Finch, Dr. Elephant, etc.

3. KEY DESIGN POINTS
Distributed writer architecture
Scalable storage backend (HBase)
Reimagined object model API with flows built into it
Separated reader instances
Aggregation

4. DISTRIBUTED WRITERS & READERS
!meline
reader
!meline
reader
Storage
!meline
reader
AM
!meline
writer
NM
!meline reader pool
app metrics/events
container events/metrics
RM
!meline writer
app/container events
user queries

5. STATUS
[DONE] timeline writers (per-app and per-node) as aux service
[DONE] RM companion writer
[DONE] first iteration of the object model API
[DONE] file-based test writer
[DONE] NM writing container events
[DONE] RM writing app/container entities
[DONE] AMs writing framework-specific events and metrics
[DONE] first versions of Phoenix and HBase writer impls
[DONE] performance benchmarking evaluation of writers

6. STATUS
[WIP] timeline readers
[WIP] aggregation
UI enhancements
Stand-alone timeline writer (per-node and per-app)
Finalize implementation of supported queries
Security
Migration/compatibility story
…

7. TEAM
This is a true community collaboration!
Sangjin, Vrushali and Joep (Twitter)
Zhijie, Li, Junping and Vinod (Hortonworks)
Naga and Varun (Huawei)
Robert and Karthik (Cloudera)
Input from LinkedIn, Yahoo! and Altiscale

8. QUESTIONS?