More than year of extremely intensive Big Data technologies development with Hadoop, HBase, MapReduce and ZooKeeper as key technologies. New company that has established infrastructure which grows pretty fast. Lot of experience in networking and distributed systems but completely new enterprise solutions world. What tasks does this bring? What issue and traps? What lessons were learned and what is considered as near future tasks? How embedded developer can enter this new world and what advantages he or she has? What challenges should you be ready to face?

1. Fall seven times
STAND UP EIGHT!
Roman Nikitchenko, 30.08.2014
JDay
2014
BIG.DATA

2. What is big data?
Big Data is like
TEENAGE SEX.
Everyone is talking
about it, nobody doing
it correctly.
www.vitech.com.ua 2

3. Agenda
WHY WRONG?
So what is wrong
with big data?
BUILD RIGHT!
Big data, our
approach.
MYTHICAL
CREATURES
Some magics
around big data.
YET NOTES
Some notes we
want to share.
www.vitech.com.ua 3

4. What is BIG DATA?
BIG DATA IS EVERYWHERE
● Really BIG DATA things: photo banks, video storage,
historical measurements.
● Intensive data transactions and high distribution: stores
(offline or online), banks, advertising networks.
● Realtime data: measurements and minitoring, gaming.
● Intensive processing: science, modelling.
● High volumes of small things: social networks,
healthcare
www.vitech.com.ua 4

5. BIG DATA in just 3 words
Indeed any real big
data is just about
DIGITAL LIFE
FOOTPRINT
www.vitech.com.ua 5

6. So...
BIG DATA is not about the
data. It is about OUR ABILITY
TO HANDLE THEM.
www.vitech.com.ua 6

7. How to do it DEFINITELY WRONG
Greedy CFO:
ONE server per
year! Controlled
costs!
Stupid architect: Looks clean
and everybody is happy!
Lazy admin:
Natural grow!
2006 2007 2008 2009 2010 … 2014 … future
Putting yearly data on new server looks nice!
www.vitech.com.ua 7

8. You are screwed!
Poor CFO:
How many...
this year?
Crazy
admin:
No way to
balance!!!
FIRED!
2006 2007 2008 2009 2010 … 2014 … future
But then we come to REALLY BAD STATE.
www.vitech.com.ua 8

9. … but why?
Analytic: Why year
average takes so long
and cluster is IDLE!
Clients: transactions
are so slow...
2006 2007 2008 2009 2010 … 2014 … future
Why? Because of cluster design.
www.vitech.com.ua 9

10. So BIG DATA our way
Linear scalability so 2
times more power
costs approximately 2
times more
No natural keys so
load balancing
No 'special' hardware
so staging is closer to
production.
www.vitech.com.ua 10

11. Why hadoop?
BIG
DATA BIG
=
+
x MAX
DATA
BIG
DATA
BIG
DATA
BIG
DATA
BIG
DATA
BIG
DATA
BIG
DATA
BIG
DATA
BIG
DATA
www.vitech.com.ua 11

12. HBase on top ot Hadoop
GROUP DANCES
ON THE ELEPHANT
BACK
● Low latency.
● Highly scalable.
● Reliability is incredible.
www.vitech.com.ua 12

13. Zookeeper
… because coordinating
distributed systems is a
Zoo
Apache
ZooKeeper
www.vitech.com.ua 13

14. Important points
POINTS
TO NOTE
www.vitech.com.ua 14

15. Hadoop design points.
BIG
DATA
● Reliable key nodes,
replacable unified
workers.
● No RAID on workers and
no any 'logical' things like
LVM.
www.vitech.com.ua 15

16. Not so commodity...
● Typecal node is expected to
include at least 64G memory
● Starting from 4 x 2T drives for
storage. 8-16 x 4T drives are not
so rare. This is for general
'workload' node.
● 12 and more CPU cores. 2 CPUs
is normal approach.
● SSD is starting to be widely used
not only for OS and caching but
for data itself.
● Main outcome — per node costs
model is changing.
HARDWARE
IS GOING
CHEAPER
AND MORE
POWERFUL
www.vitech.com.ua 16

17. Hadoop is...
… ELEPHANT
● Designed for throughput,
not for latency.
● HDFS blocks are expected
to be large. There is issue
with lot of small files.
● Write once, read many
times ideology.
www.vitech.com.ua 17

18. Most important concept about Hadoop
First ever world
DATA OS
10.000 nodes computer...
Recent technology changes are focused on
higher scale. Better resource usage and
control, lower MTTR, higher security,
redundancy, fault tolerance.
www.vitech.com.ua 18

19. Virtualization
NOT
SO
REAL
ELEPHANT
www.vitech.com.ua 19

20. Virtualization cpmcerms
CONCERNS
● Is possible for key nodes. Not for
workers unless you are really big.
● Several nodes on single physical
host: what happens if this host fail?
● Loaded services on VM: is it
meaningful? Double duties?
www.vitech.com.ua 20

21. Virtualization: practical case
● Apache ZooKeeper is
QUORUM based service.
● If host with 2 ZK fails,
Everything fail which
breaks tolerancy to 1
failure.
● Can you garantee equal
performance for ZK
service instances?
● DON'T PUT QUORUM
SERVICES IN VIRTUAL
ENVIRONMENT!
HOST
HOST
www.vitech.com.ua 21

22. ETL
LOAD
YOUR
DATA
WITH CARE
www.vitech.com.ua 22

23. ETL & BD: main stages
SQL
server
EXTRACT TRANSFORM LOAD
Table1
BIG DATA shard
Table2
JOIN Transform
Partition
Table3
BIG DATA shard
Table4 BIG DATA shard
● SQL solution are usually not so
distributed as Big Data one. How to
partition your data?
● Big data storages are mostly non-relational.
You are to map table
relations into objects. Where to put this
complexity?
www.vitech.com.ua 23

24. ETL & BD: complexity on SQL
SQL
server
Table1
BIG DATA shard
Table2
JOIN
ETL stream
Table3
BIG DATA shard
Table4 BIG DATA shard
● It's hard to transform SQL relationship
into NoSQL objects: complex joins.
● Simple stream on big data, lowered
network traffic. HUGE load on SQL.
● What if you have several SQL servers
and you need 2 times faster import?
SQL
dies on
this
www.vitech.com.ua 24

25. ETL & BD: complexity on BD side
SQL
server
Table1
ETL stream BIG DATA shard
Table2
ETL stream
JOIN
Table3
BIG DATA shard
ETL stream
Table4 ETL stream
BIG DATA shard
● Simple streaming from SQL. Things
like joins on Big Data side.
● Even if you have 100 SQL servers,
you have to scale single cluster.
● Network load is more intensive.
Much
more
scalable
www.vitech.com.ua 25

26. ETL: data partitioning
KEY DNC
KEY
KEY
2008 BIG DATA shard
BIG DATA shard
BIG DATA shard
2009
2010
A...
S...
DNC
U... DNC
● By what criteria to partition? Used storage size is to be
equally distributed. Processing load is to be shared.
Partitioning is to scale with cluster.
● Definitely no natural keys. No equal storage distribution,
no processing load share.
● Select good synthetic keys starting from input stream
(before entering big data). UUID is great example.
www.vitech.com.ua 26

27. HBase: Data and search integration
User just puts (or
deletes) data.
Replication can be
set up to column
family level.
REPLICATION
HBase
cluster
HBase regions
HDFS
Data update
Client
Search requests (HTTP)
Search responses
Translates data
changes into SOLR
index updates.
Lily HBase
NRT indexer
SOLR cloud
Apache
Zookeeper does
all coordination
Finally provides
search
Serves low level
file system.
www.vitech.com.ua 27

28. Changing Big data industry
BIG DATA INDUSTRY
face changes
www.vitech.com.ua 28

29. New concepts
DATA LAKE
Take as much data
about your business
processes as you can
take. The more data
you have the more
value you could get
from it.
www.vitech.com.ua 29

30. New concepts
ENTERPRISE DATA HUB
Don't ruine your existing data warehouse.
Just extend it with new, centralized big
data storage through data migration
solution.
www.vitech.com.ua 30

31. Networking effects
NETWORK
MATTERS
Global data storage is reality.
This makes whole picture
completely crazy.
www.vitech.com.ua 31

32. Share your knowledge!
DO NOT
HIDE YOUR
EXPERIENCE
www.vitech.com.ua 32

33. Hadoop: don't do it yourself
www.vitech.com.ua 33

34. Major Hadoop distributions
● HortonWorks are 'barely open source'. Innovative, but
'running too fast'. Most ot their key technologies are not
so mature yet. On the blade as for me.
● Cloudera is stable enough but not stale. Hadoop 2.3 with
YARN, HBase 0.96.x. Balance.
● MapR focuses on performance per node but they are
slightly outdated in term of functionality and their
distribution costs. For cases where node performance is
high priority.
www.vitech.com.ua 34

35. Questions and discussion
www.vitech.com.ua 35