Building applications on YARN involves considering many factors including architecture, operations, deployment, logging, metrics, configuration, security, languages, fault tolerance, isolation, and state management. The anatomy of a YARN application includes a client, application master, container code, and resource manager. The application master communicates with the resource manager and manages containers.

1. Building Applications on YARN
Chris Riccomini
10/11/2012

2. Staff Software Engineer at LinkedIn
http://riccomini.name
@criccomini

3. What I want to Talk About
Anatomy of a YARN Application
Things to consider when building your application
Architecture
Operations

4. Anatomy of a YARN App
Client
Application Master
Container Code
Resource Manager
Node Manager

5. Anatomy of a YARN App
Client
Client
Client RM
RM
Application Master
Container Code
Resource Manager
NM
NM NM
NM
Node Manager
AM
AM CC
CC
* simplified

6. A lot to consider
Deployment Logging
Metrics Fault Tolerance
Configuration Isolation
Security Dashboard
Language State

7. Deployment
HDFS
HTTP
File (NFS)
DDOS’ing your servers
What we do: Tarball over HTTP. Life is easier with HDFS,
but operational overhead is too high.

8. Metrics
Application-level metrics
YARN-level metrics
metrics2
Containers are transient
What we do: Both app-level and framework-level metrics use
same metrics framework. Pipe to in-house metrics
dashboard. We don’t use metrics2 since we don’t want a
dependency on Hadoop in our core jar.

9. Metrics

10. Configuration
YARN config (yarn-site.xml, core-site.xml, etc)
Application Configuration
Transporting Configuration
What we do: Config is fully resolved at client execution time.
No admin-override/locked config protection yet. Config is
passed from client to AM to containers via environment
variables.

11. Security
Kerberos?
Firewalls are your friend
Gateway machine
Dashboard
What we do: Firewall all YARN machines so they can only
talk to each-other. All users go through LDAP controlled
dashboard.

12. Language
Favor complexity in Application Master, and make
container-logic thin
Talk to RM via REST
Potential to talk to RM via Protobuf RPC
What we do: Application AM is Java. Tasks-side of
application has Python and Java implementations.

13. Logging
Local storage (application is running)
HDFS storage (application has stopped for a while)
Be careful with STDOUT/STDERR (rollover)
What we do: No HDFS. Logs sit for 7 days, then disappear.
Not ideal.

14. Fault Tolerance
Failure matrix
HA RM/NM
Orphaned processes
Pay attention to process trees
What we do: No HA. Manual fail over when RM dies.
Orphaned process monitor (proc start time < RM start time).

15. Fault Tolerance

16. Isolation
Memory
Disk
CPU
Network
What we do: Nothing, right now. Hoping YARN will solve
this before we need it (cgroups?).

17. Dashboard
Application-specific information
Integrate with YARN
Application Master or Standalone?
What we do: Dashboard enforces security, talks to RM/AM
via HTTP/JSON to get information about jobs.

18. Dashboard

19. State
HDFS
Deployed with Application
Remote data store
What we do: Nothing, right now.

20. Takeaways
There’s a lot more than just the YARN API
Look for examples (Spark, Storm, Map-Reduce)
Decide your level of Hadoop integration
Metrics2
HDFS
Config
Kerberos and doAs

21. Questions?