The document is a presentation by Pham Thai Hoa from 4/14/2012 about Hadoop, Hive, and how they are used at Mobion. It introduces Hadoop and Hive, explaining what they are, why they are used, and how data flows through them. It also discusses how Mobion uses Hadoop and Hive for log collection, data transformation, analysis, and reporting. The presentation concludes with Q&A and links for further information.

1. Hadoop Presentation
2012
Presenter : Pham Thai Hoa
Email : thaihoabo@gmail.com
Web : http://mobion.com/hoa
4/14/2012 Pham Thai Hoa

2. Topic
 Introduce to Hadoop
 Introduce to Hive
 Introduce to Logger
 Using Hadoop at Mobion
 Warehouse at Mobion
 Q&A
4/14/2012 Pham Thai Hoa

3. What is Hadoop
 It’s a framework for the distributed
processing
 Inspired by Google’s architecture: Map
Reduce and GFS
 A top-level Apache project
 Hadoop is the open source
 Hadoop have the two important
elements
+ Map – Reduce core
+ Hadoop Distributed File System
4/14/2012 Pham Thai Hoa

4. Why use Hadoop
 Fault-tolerant hardware is expensive
 Hadoop is designed to run on cheap
commodity hardware
 It automatically handles data
replication and node failure
 It does the hard work – you can focus
on processing data
 It has the three supported modes :
Local, Pseudo-Distributed, Fully-
Distributed Mode
4/14/2012 Pham Thai Hoa

5. Data Flow into Hadoop
4/14/2012 Pham Thai Hoa

6. Who use Hadoop
 Amazon's product search indices
using the streaming API and pre-
existing C++, Perl, and Python tools
 Yahoo : More than 100,000 CPUs in
>40,000 computers running Hadoop
 Facebook use Hadoop to store copies
of internal log and dimension data
sources and use it as a source for
reporting/analytics and machine
learning
4/14/2012 Pham Thai Hoa

7. What is Hive
 Hive is a data warehouse system for
Hadoop
 Using Map-Reduce for execution
 Using HDFS for storage
 Metadata in an RDBMS
 Scalability and performance
 Interoperability
 Using a SQL-like language called
HiveQL
4/14/2012 Pham Thai Hoa

8. Data Flow into Hive
4/14/2012 Pham Thai Hoa

9. Hive Data Model
 Tables
+ Typed columns (int, float, string,…)
+ Also, array/map/struct for JSON-like
data
 Partitions
+ e.g., to range-partition tables by
date
 Buckets
+ Hash partitions within ranges (useful
for sampling, join optimization)
4/14/2012 Pham Thai Hoa

10. Hive Metastore
 Database: namespace containing a
set of tables
 Holds Table/Partition definitions
(column types,mappings to HDFS
directories)
 Statistics
 Implemented with DataNucleus ORM.
Runs on Derby, MySQL, and many
other relational databases
4/14/2012 Pham Thai Hoa

11. Introduce to Logger
 A logging system has three broad
components
+ Client Code Interface
+ Distribution System
+ Do Something Usefullizer
 Scribe is a server for aggregating
streaming log data. It is designed to
scale to a very large number of nodes
and be robust to network and node
failures
4/14/2012 Pham Thai Hoa

12. Why use Scribe
 Scalability and performance
 Event Notification library
 Thrift framework
 Hadoop is optional
 Client using
 Distributed scribe system
 Over 1 million messages per second
for logging
 Hierarchy stores
4/14/2012 Pham Thai Hoa

13. Warehouse at Mobion
 Log Collector
 Log/Data Transformer
 Data Analyzer
 Web Reporter
 Log define
 Log integrate (into application)
 Log/Data analyze
 Report develop (API, Mobion, Music
…)
4/14/2012 Pham Thai Hoa

14. Warehouse at Mobion
 Data mining
 Music Recommendation
 Spam Detection
 Application performance
 Export data and import into MySQL for
web report
 Analytic system
4/14/2012 Pham Thai Hoa

15. Q&A
 Why use hadoop ?
 Why use Hive ?
 Why need a logging system ?
 What is the warehouse system
architecture ?
 Do we use these system for voting,
chat, message and feed ??
 How can we use them for
recommendation, suggestion ?
4/14/2012 Pham Thai Hoa

16. Following Link
 http://facebook.com
 http://highscalability.com/product-
scribe-facebooks-scalable-logging-
system
 http://hadoop.apache.org/
 http://hive.apache.org/
 http://wiki.apache.org/hadoop/Powere
dBy
 http://www.apache.org/foundation/than
ks.html 4/14/2012 Pham Thai Hoa

17. THANK YOU
4/14/2012 Pham Thai Hoa