Hadoop a Natural Choice for Data Intensive Log Processing

Apache Hadoop A Natural Choice for Data Intensive Multiform at Log Processing Date: 22 nd April’ 2011 Authored and Compiled By: Hitendra Kumar

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Hadoop Framework A Brief Background A Brief Background

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Hadoop Framework Leveraging Hadoop for High Performance over RDBMS Leveraging Hadoop over RDBMS

XML Logs CSV SQL Objects, JSONs Binary Hadoop Distributed File System (HDFS) M A P C R E A T I O N Reduce Commodity Server Cloud (Scale Out) Hadoop Environment RDBMS import Reporting Dash Boards BI Applications Enterprise High Volume Data In-Flow Map-Reduce Process Consume Results Hadoop Processing How it works? How it works?

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Hadoop Processing Map Reduce Algorithm . Map Reduce Algorithm

Hadoop Processing Map Reduce Algorithm .. Map Reduce Algorithm ..

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Hadoop Processing Map Reduce Algorithm ... Map Reduce Algorithm …

Hadoop Processing Components Map Reduce Algorithm … ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hadoop Processing HDFS File System HDFS File System ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hadoop Processing HIVE Framework and Hive QL HIVE ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hadoop Processing Hive, Why? HIVE ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hadoop Processing Pig – High Level Data Flow Language Pig – High Level Data Flow Language ,[object Object],[object Object],[object Object],[object Object]

Hadoop Processing Mahout – Extensions to Hadoop Programming Extensions to Hadoop Programming ,[object Object],[object Object],[object Object],[object Object],[object Object]

Hadoop Processing HBASE – Distrubited, Fault Tolerant and Scalable DB HBASE ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hadoop Processing Hadoop Db - Architecture Hadoop DB ,[object Object],[object Object],[object Object],[object Object]

Example System (Web Portal) Tera-Bytes of data being populated to centralized storage and processed, every week-end!

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Role Management Administration Module Monitoring Control Report Configurations Reporting Business Intelligence Module Analysis Metrics Trends Application Integration Services Application Integration Portlet Integration Rules Data Sources Business Applications Infrastructure and Business Services MyASUP Portal Application Set-Up Core Framework (Logging, Exceptions, Rule Engine, Analytics, Auditing) External Apps UI Adaptation Real Time Integration Module JMS, MQ, JDBC Channels Back End Web Portal (High Level Architecture) Web Portal - High Level Architecture Which uses Hadoop, Solr and Lucene for Backend Data Processing Web Portal – Using Hadoop/Solr/Lucene Security

DB Server J2EE Application Server HTTP HTTP DB Server J2EE Application Server Apache Web Server Tomcat mod_jk Plug-In JBOSS - J2EE Application JBOSS – Portal Web Service JBOSS – jBPM JBOSS - Portal HTTP JDBC Web Portal Servers (Apache + App Server) Web Portal Deployment Landscape Shrading Function Hadoop Processing Web Portal – Deployment Landscape Web Portal – Deployment Landscape DB LB LB DB

Example – AOL Advertising Platform http://www.cloudera.com/blog/2011/02/an-emerging-data-management-architectural-pattern-behind-interactive-web-application/

[object Object],[object Object],[object Object],[object Object],[object Object],Hadoop Processing AOL Advertising – Business Case and Solution AOL Advertising – Business Case and Solution

Hadoop a Natural Choice for Data Intensive Log Processing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Hadoop a Natural Choice for Data Intensive Log Processing

Similar to Hadoop a Natural Choice for Data Intensive Log Processing (20)

Recently uploaded

Recently uploaded (20)

Hadoop a Natural Choice for Data Intensive Log Processing