- Apache Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of computers. It allows for the reliable storage of petabytes of data and large-scale computations across commodity hardware. - Apache Hadoop is used widely by internet companies to analyze web server logs, power search engines, and gain insights from large amounts of social and user data. It is also used for machine learning, data mining, and processing audio, video, and text data. - The future of Apache Hadoop includes making it more accessible and easy to use for enterprises, addressing gaps like high availability and management, and enabling partners and the community to build on it through open APIs and a modular architecture.

5. Apache Hadoop Projects Programming Languages Computation Object Storage Zookeeper (Coordination) Core Apache Hadoop Related Apache Projects HDFS (Hadoop Distributed File System) MapReduce (Distributed Programing Framework) HMS (Management) Table Storage Hive (SQL) Pig (Data Flow) HBase (Columnar Storage) HCatalog (Meta Data)

6. Where Hadoop is Used

7. Everywhere! 2006 2008 2009 2010 The Datagraph Blog 2007

8. HADOOP @ YAHOO! 40K+ Servers 170 PB Storage 5M+ Monthly Jobs 1000+ Active users © Yahoo 2011

13. Traditional Enterprise Architecture Data Silos + ETL Serving Applications Unstructured Systems Traditional ETL & Message buses Traditional ETL & Message buses EDW Data Marts BI / Analytics Traditional Data Warehouses, BI & Analytics Serving Logs Social Media Sensor Data Text Systems …

14. Serving Applications Unstructured Systems Traditional ETL & Message buses Traditional ETL & Message buses Hadoop Enterprise Architecture Connecting All of Your Big Data EDW Data Marts BI / Analytics Traditional Data Warehouses, BI & Analytics Apache Hadoop EsTsL (s = Store) Custom Analytics Serving Logs Social Media Sensor Data Text Systems …

15. Serving Applications Unstructured Systems 80-90% of data produced today is unstructured Gartner predicts 800% data growth over next 5 years Traditional ETL & Message buses Traditional ETL & Message buses Hadoop Enterprise Architecture Connecting All of Your Big Data EDW Data Marts BI / Analytics Traditional Data Warehouses, BI & Analytics Serving Logs Social Media Sensor Data Text Systems … Apache Hadoop EsTsL (s = Store) Custom Analytics

17. Big Data Platforms Cost per TB, Adoption Size of bubble = cost effectiveness of solution Source:

18. Apache Hadoop Core

24. Client Read & Write Directly from Closest Server Namespace State Horizontally Scale IO and Storage Client 1 open 2 read Client 2 write 1 create End-to-end checksum Namenode Block Map b1 b5 b3 JBOD b2 b3 b1 JBOD b3 b5 b2 JBOD b1 b5 b2 JBOD

25. Actively maintain data reliability Namespace State Bad/lost block replica Periodically check block checksums Namenode Block Map b1 b5 b3 JBOD b2 b3 b4 JBOD b3 b5 b2 JBOD b1 b5 b2 JBOD 2. copy 3. blockReceived 1. replicate

27. What’s Next

30. Lines of Code Contributed to Apache Hadoop