This document discusses big data analytics using Hadoop. It provides an overview of loading clickstream data from websites into Hadoop using Flume and refining the data with MapReduce. It also describes how Hive and HCatalog can be used to query and manage the data, presenting it in a SQL-like interface. Key components and processes discussed include loading data into a sandbox, Flume's architecture and data flow, using MapReduce for parallel processing, how HCatalog exposes Hive metadata, and how Hive allows querying data using SQL queries.

1. BIG DATA ANALYTICS WITH HADOOP
BY-
SWAMIL SINGH
VIPLAV MANDAL
GUIDED BY-DR S.SRIVATAVA

2. AGENDA
• Design of website clickstream data and example.
• How to load data into sandbox.
• Load data using flume and the process.
• About flume
• Flumes working process.
• Process to refine data.
• Map reduce
• Hcatalog and hcatalog work
• Hive
• Hive work & their process
• Queries

3. DESIGN OF WEBSITE CLICKSTREAM DATA
• Clickstream data is an information trail a user leaves behind while visiting a website. It is
typically captured in semi-structured website log files.
• These website log files contain data elements such as a date and time stamp, the visitor’s IP
address, the destination URLs of the pages visited, and a user ID that uniquely identifies the
website visitor.
• One of the original uses of Hadoop at Yahoo was to store and process their massive volume of
clickstream data

4. EXAMPLE OF WEBSITE CLICK STREAM DATA

5. HOW TO LOAD DATA INTO SANDBOX
• The sandbox is a fully contained Data Platform environment
• The sandbox includes the core Hadoop components (HDFS and MapReduce), as well as all
the tools needed for data ingestion and processing.
• You can access and analyze sandbox data with many Business Intelligence (BI) applications.
• By combining web logs with more traditional customer data, we can better understand our
customers, and also understand how to optimize future promotions and advertising.

6. ABOUT FLUME
• Flume’s high-level architecture is built on a streamlined codebase that is easy to use and
extend.
• . The project is highly reliable, without the risk of data loss. Flume also supports dynamic
reconfiguration without the need for a restart, which reduces downtime for its agents.
• Flume components interact in the following way
• A flow in Flume starts from the Client.
• The Client transmits the Event to a Source operating within the Agent
• The Source receiving this Event then delivers it to one or more Channels.
• One or more Sinks operating within the same Agent drains these Channels.
• Channels decouple the ingestion rate from drain rate using the familiar producer-consumer
model of data exchange.

7. LOAD DATA USING FLUME ,THE PROCESS

8. SEQUENCE DIAGRAM OF FLUME

9. • Enterprises use Flume’s powerful streaming capabilities to land data from high-throughput
streams in the HDFS. These different types of data can be landed in Hadoop for future analysis
using interactive queries in Apache Hive.
• In one specific example,
-Flume is used to log manufacturing operations. When one run of product comes off the line,
it generates a log file about that run.
• The large volume log file data can stream through Flume into a tool for same-day analysis with
Apache Storm or months or years of production runs can be stored in HDFS and analyzed by a
quality assurance engineer using Apache Hive..

10. THE PROCESS TO REFINE DATA
• Omniture logs* – website log files containing information such as URL, timestamp, IP address,
geocoded IP address, and user ID (SWID).
• Users* – CRM user data listing SWIDs (Software User IDs) along with date of birth and gender.
• Products* – CMS
• data that maps product categories to website URLs

11. MAP REDUCE

12. ABOUT THE MAP REDUCE
• A MapReduce job splits a large data set into independent chunks and organizes them into key,
value pairs for parallel processing.
• The Map function divides the input into ranges by the Input Format and creates a map task for
each range in the input
• The output of each map task is partitioned into a group of key-value pairs for each reduce.
• The Reduce function then collects the various results and combines them to answer the larger
problem that the master node needs to solve

13. HCATALOG
• Apache HCatalog is a table management layer that exposes Hive metadata to other Hadoop
applications
• HCatalog’s table abstraction presents users with a relational view of data in the Hadoop
Distributed File System (HDFS) and ensures that users need not worry about where or in what
format their data is stored
• HCatalog displays data from RCFile format, text files, or sequence files in a tabular view.

14. HOW HCATALOG WORKS
• HCatalog supports reading and writing files in any format for which a Hive SerDe (serializer-
deserializer) can be written.
• By default, HCatalog supports RCFile, CSV, JSON, and Sequence File formats. To use a
custom format, you must provide the Input Format, Output Format, and SerDe.
• HCatalog is built on top of the Hive metastore and incorporates components from the Hive
DDL.
• HCatalog provides read and write interfaces for Pig and MapReduce and uses Hive’s command
line interface for issuing data definition and metadata exploration commands.

15. HIVE
• Hive is a component of Data Platform. Hive provides a SQL-like interface to data stored in DP.
• Hive provides a database query interface to Apache Hadoop.
• Hive because of its SQL like query language is often used as the interface to an Apache
Hadoop based data warehouse.
• Pig fits in through its data flow strengths where it takes on the tasks of bringing data into
Apache Hadoop and working with it to get it into the form for querying.

16. HOW TO WORK HIVE
• The tables in Hive are similar to tables in a relational database, and data units are organized in
a taxonomy from larger to more granular units.
• Databases are comprised of tables, which are made up of partitions.
• Data can be accessed via a simple query language and Hive supports overwriting or appending
data
• Hive supports all the common primitive data formats such as BIGINT, BINARY, BOOLEAN,
CHAR, DECIMAL, DOUBLE, FLOAT, INT, SMALLINT, STRING, TIMESTAMP, and TINYINT.
• In addition, analysts can combine primitive data types to form complex data types, such as
structs, maps and arrays.

17. WORKING PROCESS OF HIVE

18. • Any queries…..
• THANK YOU