SlideShare is now on Android. 15 million presentations at your fingertips.  Get the app

×
  • Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
 

Hadoop Hardware @Twitter: Size does matter!

by on Jul 11, 2013

  • 2,544 views

At Twitter we started out with a large monolithic cluster that served most of the use-cases. As the usage expanded and the cluster grew accordingly, we realized we needed to split the cluster by ...

At Twitter we started out with a large monolithic cluster that served most of the use-cases. As the usage expanded and the cluster grew accordingly, we realized we needed to split the cluster by access pattern. This allows us to tune the access policy, SLA, and configuration for each cluster. We will explain our various use-cases, their performance requirements, and operational considerations and how those are served by the corresponding clusters. We will discuss what our baseline Hadoop node looks like. Various, sometimes competing, considerations such as storage size, disk IO, CPU throughput, fewer fast cores versus many slower cores, 1GE bonded network interfaces versus a single 10 GE card, 1T, 2T or 3T disk drives, and power draw all need to be considered in a trade-off where cost and performance are major factors. We will show how we have arrived at quite different hardware platforms at Twitter, not only saving money, but also increasing performance.

Statistics

Views

Total Views
2,544
Views on SlideShare
2,518
Embed Views
26

Actions

Likes
8
Downloads
71
Comments
1

2 Embeds 26

https://twitter.com 25
http://www.linkedin.com 1

Accessibility

Upload Details

Uploaded via SlideShare as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1 previous next

  • GregRahn GregRahn Video: http://youtu.be/rqzbLd110NY 9 months ago
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Hadoop Hardware @Twitter: Size does matter! Hadoop Hardware @Twitter: Size does matter! Presentation Transcript