Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Hadoop World 2011: Hadoop Network and Compute Architecture Considerations - Jacob Rapp, Cisco

13,723
views

Published on

Hadoop is a popular framework for web 2.0 and enterprise businesses who are challenged to store, process and analyze large amounts of data as part of their business requirements. Hadoop’s framework …

Hadoop is a popular framework for web 2.0 and enterprise businesses who are challenged to store, process and analyze large amounts of data as part of their business requirements. Hadoop’s framework brings a new set of challenges related to the compute infrastructure and underlined network architectures. This session reviews the state of Hadoop enterprise environments, discusses fundamental and advanced Hadoop concepts and reviews benchmarking analysis and projection for big data growth as related to Data Center and Cluster designs. The session also discusses network architecture tradeoffs, and the advantages of close integration between compute and networking.

Published in: Technology

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
13,723
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Cisco and Big Data – Hadoop World 2011
  • 2. Unified Fabric optimized for Big Data infrastructures with seamless integration with current data models Traditional Database RDBMS Storage SAN/NAS “ Big Data” Store And Analyze “ Big Data” Real-Time Capture, Read & Update NoSQL Application Virtualized, Bare-Metal, Cloud Sensor Data Logs Social Media Click Streams Mobility Trends Event Data Cisco Unified Fabric
  • 3.  
  • 4.
    • 128 Nodes of UCS C200 M2, 1RU Rock-Mount Servers
      • 4x2TB Drives, Dual Xeon 5670 @ 2.93GHz, 96GB RAM
    The Setup Lab environment overview
    • 16 Nodes of UCS C210 M2, 2RU Rack-Mount Servers
      • 16xSFF Drives, Dual Xeon 5670 @2.93GHz, 96GB RAM
  • 5.
    • Unified Fabric L2 &/or L3 for SAN/NAS, RDBMS, UCS, and Big Data
    • L2/L3 Top of Rack Infrastructure
    … .. … .. Note: Two topologies were tested to examine the benefits of providing an integrated solution that can support multiple technologies, such as Traditional RDBMS, SAN/NAS, Virtualization, etc. N7K N7K N7K N3k N3k N3k N3k N3k N3k N7K N7K N7K N5k N5k N5k N5k … N2k … N2k UCS
  • 6.
    • Cluster Size
      • Number of Data Nodes
    • Data Model
      • MapReduce functions
    • Input Data Size
      • Total starting dataset
    • Characteristics of Data Node
      • I/O, CPU, Memory, etc
    • Data Locality in HDFS
      • Ability to processes data where it already is located.
    • Background Activity
      • Number of Jobs running, type of jobs, Importing, exporting
    • Networking Characteristics
      • Availability, Buffering, 10GE vs 1GE, Oversubscription, Latency
  • 7. A general characteristic of an optimally configured cluster is the ability to decrease job completion times by scaling out the nodes. Test results from ETL-like Workload (Yahoo Terasort) using 1TB data set.
  • 8. The complexity of the functions used in Map and/or Reduce has a large impact on the type of job and network traffic.
    • Note:
    • Yahoo Terasort has a more balanced Map vs. Reduce functions and the same size of intermediate and final data (1TB Input, Shuffle and Output)
    • Shakespeare WordCount has most of the processing in the Map Functions, smaller intermediate and even smaller final Data. (1TB Input, 10MB Shuffle, 1MB Output)
  • 9. Network Graph of all Traffic received on an single node (80 Node run) Reducers Start Maps Finish Job Complete Maps Start These symbols represent a node sending traffic to HPC064 Note: Shortly after the Reducers start Map tasks are finishing and data is being shuffled to reducers As Maps completely finish the network is no loner used as Reducers have all the data they need to finish the job The red line is the total amount of traffic received by hpc064
  • 10. Network Graph of all Traffic received on an single node (80 Node run)
    • Output Data Replication Enabled
    • Replication of 3 enabled (1 copy stored locally, 2 stored remotely)
    • Each reduce output is replicated now, instead of just stored locally
    Note: If output replication is enabled, then the end of the terasort, must store additional copies. For a 1TB sort, 2TB will need to be replicated across the network.
  • 11. Network Graph of all Traffic received on an single node (80 node run) Reducers Start Maps Finish Job Complete Maps Start The red line is the total amount of traffic received by hpc064 These symbols represent a node sending traffic to HPC064 Note: Due the the combination of the length of the Map phase and the reduced data set being shuffled, the network is being utilized throughout the job, but by a limited amount.
  • 12. Given the same MapReduce Job, the larger the input dataset, the longer the job will take. Note: It is important to note that as dataset sizes increase completion times may not scale linearly as many jobs can hit the ceiling of I/O and/or Compute power. Test results from ETL-like Workload (Yahoo Terasort) using varying data set sizes.
  • 13. The I/O capacity, CPU and memory of the Data Node can have a direct impact on performance of a cluster. Note: A 2RU Server with 16 disks gives the node more storage, but trading off CPU per RU. On the other hand a 1RU server gives more CPU per rack.
  • 14. Data Locality – The ability to process data where it is locally stored. Note: During the Map Phase, the JobTracker attempts to use data locality to schedule map tasks where the data is locally stored. This is not perfect and is dependent on a data nodes where the data is located. This is a consideration when choosing the replication factor. More replicas tend to create higher probability for data locality. Map Tasks: Initial spike for non-local data. Sometimes a task may be scheduled on a node that does not have the data available locally.
  • 15. Hadoop clusters are generally multi-use. The effect of background use can effect any single jobs completion. Note: A given Cluster, is generally running may different types of Jobs, Importing into HDFS, Etc. Example View of 24 Hour Cluster Use Large ETL Job Overlaps with medium and small ETL Jobs and many small BI Jobs (Blue lines are ETL Jobs and purple lines are BI Jobs) Importing Data into HDFS
  • 16.
      • The relative impact of various network characteristics on Hadoop clusters.
    Impact of Network Characteristics on Job Completion times
  • 17. The failure of a networking device can affect multiple data nodes of a Hadoop cluster with a range of effects. Note: The tasks on affected nodes need to be rescheduled, schedule maintenance activities such as data rebalancing, increasing load on the cluster.
    • Important to evaluate the overall availability of the system.
      • Hadoop was designed with failure in mind, given any one node failure does not represent a huge issue, but network failures can span many nodes in the system causing rebalancing and decreased overall resources.
    • Redundancy paths and load sharing schemes.
      • General redundancy mechanisms can also increase bandwidth.
    • Ease of management and consistent Operating System
      • Main sources of outages can include human error. Ease of management and consistency are general best practices.
  • 18. Several HDFS operations and phases of MapReduce jobs are very bursty in nature Note: The extent of bursts largely depend on the type of job (ETL vs. BI). Bursty phases can include replication of data (either importing into HDFS or output replication) and the output of the mappers during the shuffle phase.
    • A network that cannot handle bursts effectively will drop packets, so optimal buffering is needed in network devices to absorb bursts.
    • Optimal Buffering
    • Given large enough incast, TCP will collapse at some point no matter how large the buffer
    • Well studied by multiple universities
    • Alternate solutions (Changing TCP behavior) proposed rather than Huge buffer switches
    • ( http://simula. stanford .edu/sedcl/files/dc tcp -final.pdf )
  • 19. Buffer being used during shuffle phase Buffer being used during output replication
    • The buffer utilization is highest during the shuffle and output replication phases.
    • Optimized buffer sizes are required to avoid packet loss leading to slower job completion times.
    Note: The Aggregation switch buffer remained flat as the bursts were absorbed at the Top of Rack layer
  • 20. Buffer being used during shuffle phase Buffer being used during output replication Note: The Fabric Extender Buffer utilization was roughly equivalent to that of the N3k, but has 32MB of buffer vs. 9MB of N3k
    • The buffer utilization is highest during the shuffle and output replication phases.
    • Optimized buffer sizes are required to avoid packet loss leading to slower job completion times.
  • 21. In a multi-use cluster described previously, multiple job types (ETL, BI, etc.) and Importing data into HDFS can be happening at the same time Note: Usage may vary depending on job scheduling options
  • 22. In the largest workloads, multi-terabytes can be transmitted across the network Note: Data taken from multi-use workload (Multi-ETL + Multi-BI + HDFS Import).
  • 23. Generally 1GE is being used largely due to the cost/performance trade-offs. Though 10GE can provide benefits depending on workload. Note: Multiple 1GE links can be bonded together to not only increase bandwidth, but increase bandwidth.
  • 24. Moving from 1GE to 10GE actually lowers the buffer requirement on the switching layer. Note: By moving to 10GE, the data node has a larger input to receive data lessening the need for buffers on the network as the total aggregate speed or amount of data does not increase substantially. This is due, in part, to limits of I/O and Compute capabilities
  • 25. Generally network latency, while consistent latency being important, does not represent a significant factor for Hadoop Clusters. Note: There is a difference in network latency vs. application latency. Optimization in the application stack can decrease application latency that can potentially have a significant benefit.
  • 26. For more information: www.cisco.com/go/bigdata
  • 27.