A hands on introduction to Hadoop by using the Hortonworks Sandbox

1. Adam
Muise
–
Solu/on
Architect,
Hortonworks
HADOOP
101:
AN
INTRODUCTION
TO
HADOOP
WITH
THE
HORTONWORKS
SANDBOX

2. Who
are
we?

3. Who
is
?

4. 100%
Open
Source
–
Democra/zed
Access
to
Data
The
leaders
of
Hadoop’s
development
We
do
Hadoop
Drive
Innova/on
in
the
plaForm
–
We
lead
the
roadmap
Community
driven,
Enterprise
Focused

5. We
do
Hadoop
successfully.
Support
Training
Professional
Services

6. Enter
the
Hadoop.
………
hOp://www.fabulouslybroke.com/2011/05/ninja-­‐elephants-­‐and-­‐other-­‐awesome-­‐stories/

7. Hadoop
was
created
because
tradi/onal
technologies
never
cut
it
for
the
Internet
proper/es
like
Google,
Yahoo,
Facebook,
TwiOer,
and
LinkedIn

8. Tradi/onal
architecture
didn’t
scale
enough…
App
App
App
App
App
App
App
App
DB
DB
DB
SAN
App
App
App
App
DB
DB
DB
SAN
DB
DB
DB
SAN

9. Databases
can
become
bloated
and
useless

10. $upercompu/ng
Tradi/onal
architectures
cost
too
much
at
that
volume…
$/TB
$pecial
Hardware

11. So
what
is
the
answer?

12. If
you
could
design
a
system
that
would
handle
this,
what
would
it
look
like?

13. It
would
probably
need
a
highly
resilient,
self-­‐healing,
cost-­‐efficient,
distributed
file
system…
Storage
Storage
Storage
Storage
Storage
Storage
Storage
Storage
Storage

14. It
would
probably
need
a
completely
parallel
processing
framework
that
took
tasks
to
the
data…
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage

15. It
would
probably
run
on
commodity
hardware,
virtualized
machines,
and
common
OS
plaForms
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage

16. It
would
probably
be
open
source
so
innova/on
could
happen
as
quickly
as
possible

17. It
would
need
a
cri/cal
mass
of
users

18. Tez
Storm
YARN
Pig
HDFS
MapReduce
Apache
Hadoop
HCatalog
Hive
HBase
Ambari
Knox
Sqoop
Falcon
Flume

19. Storm
Tez
Pig
YARN
HDFS
MapReduce
Hortonworks
Data
PlaForm
HCatalog
Hive
HBase
Ambari
Knox
Sqoop
Falcon
Flume

20. We
are
going
to
learn
how
to
work
with
Hadoop
in
less
than
an
hour.

21. To
do
this,
we
need
to
install
Hadoop
right?

22. Nope.

23. Enter
the
Sandbox.

24. The
Sandbox
is
‘Hadoop
in
a
Can’.
It
contains
one
copy
of
each
of
the
Master
and
Worker
node
processes
used
in
a
cluster,
only
in
a
single
virtual
node.
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Processing
Processing
Storage
Storage
Storage
Processing
Storage
Linux
VM
Processing
Processing
Processing
Storage
Storage
Storage

25. Gefng
started
with
Sandbox
VM:
-­‐
Pick
your
flavor
of
VM
at…
hOp://www.hortonworks.com/sandbox
-­‐
Start
the
sandbox
VM
-­‐
find
the
IP
displayed
-­‐
go
to…
hOp://172.16.130.131
-­‐
Register
-­‐
Click
on
‘Start
Tutorials’
-­‐
On
the
lek
hand
nav,
click
on
‘HCatalog,
Basic
Pig
&
Hive
Commands’

26. In
this
tutorial
we
will:
-­‐
Land
files
in
HDFS
-­‐
Assign
metadata
with
HCatalog
-­‐
Use
SQL
with
Hive
-­‐
Learn
to
process
data
with
Pig

27. Try
the
other
tutorials.

28. Hadoop
is
the
new
Modern
Data
Architecture
for
the
Enterprise

29. There is NO second place
Hortonworks
…the
Bull
Elephant
of
Hadoop
InnovaGon
© Hortonworks Inc. 2012: DO NOT SHARE. CONTAINS HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION
Page
29