The document discusses the need for an operating system for datacenters to manage resources and provide shared services across applications and users in the datacenter. It describes how today's datacenter OS, like Hadoop MapReduce, provides some resource sharing and data sharing capabilities but lacks advanced features. The document envisions a future datacenter OS that enables more efficient resource sharing, standardized interfaces for data sharing, useful programming abstractions, and improved debugging tools to better manage and program large datacenters.

1. Datacenter Operating
System
Presented By:
Keshav Yadav
www.linkedin.com/in/keshavyadavlpu

2. What is DCOS ?
• Some have declared that “the datacenter is
the new computer”
• Claim: this new computer increasingly
needs an operating system
• Not necessarily a new host OS, but a
common software layer that manages
resources and provides shared services for
the whole datacenter, like an OS does for
one host

3. Why Datacenters Need an OS
• Growing number of applications
– Parallel processing systems: MapReduce,
Dryad, Pregel, Percolator, Dremel, MR Online
– Storage systems: GFS, BigTable, Dynamo,
SCADS
– Web apps and supporting services
• Growing number of users
– 200+ for Facebook’s Hadoop data
warehouse, running near-interactive ad hoc
queries

4. What Operating Systems Provide
• Resource sharing across applications &
users
• Data sharing between programs
• Programming abstractions (e.g. threads,
IPC)
• Debugging facilities (e.g. ptrace, gdb)
Result: OSes enable a highly interoperable
software ecosystem that we now take for
granted

5. Today’s Datacenter OS
• Hadoop MapReduce as common
execution and resource sharing platform
• Hadoop InputFormat API for data sharing
• Abstractions for productivity programmers,
but not for system builders
• Very challenging to debug across all the
layers

6. Tomorrow’s Datacenter OS
• Resource sharing:
– Lower-level interfaces for fine-grained sharing
(Mesos is a first step in this direction)
– Optimization for a variety of metrics (e.g.
energy)
– Integration with network scheduling
mechanisms (e.g. Seawall [NSDI ‘11], NOX,
Orchestra)

7. Tomorrow’s Datacenter OS
• Data sharing:
– Standard interfaces for cluster file systems,
key-value stores, etc
– In-memory data sharing (e.g. Spark, DFS
cache), and a unified system to manage this
memory
– Streaming data abstractions (analogous to
pipes)
– Lineage instead of replication for reliability
(RDDs)

8. Tomorrow’s Datacenter OS
• Programming abstractions:
– Tools that can be used to build the next
MapReduce / BigTable in a week (e.g.
BOOM)
– Efficient implementations of communication
primitives (e.g. shuffle, broadcast)
– New distributed programming models

9. Tomorrow’s Datacenter OS
• Debugging facilities:
– Tracing and debugging tools that work across
the cluster software stack (e.g. X-Trace,
Dapper)
– Replay debugging that takes advantage of
limited languages / computational models
– Unified monitoring infrastructure and APIs

10. Putting it AllTogether
• A successful datacenter OS might let
users:
– Build a Hadoop-like software stack in a week
using the OS’s abstractions, while gaining
other benefits (e.g. cross-stack replay
debugging)
– Share data efficiently between independently
developed programming models and
applications
– Understand cluster behavior without having to
log into individual nodes
– Dynamically share the cluster with other users

11. Future Of DCOS• Focus on paradigms, not only performance
– Industry is spending a lot of time on performance
• Explore clean-slate approaches
– Much datacenter software is written from scratch
– People using Erlang, Scala, functional models
(MR)
• Bring cluster computing to non-experts
– Most impactful (datacenter as the new
workstation)
– Hard to make a Google-scale stack usable
without a Google-scale ops team

12. Conclusion
• Datacenters need an OS-like software
stack for the same reasons single
computers did: manageability, efficiency &
programmability
• An OS is already emerging in an ad-hoc
way
• Researchers can help by taking a long-
term approach towards these problems