Keys for Success from Streams to Queries

1. © 2014 MapR Technologies 1© 2014 MapR Technologies

2. © 2014 MapR Technologies 2
Contact Information
Ted Dunning
Chief Applications Architect at MapR Technologies
Committer & PMC for Apache’s Drill, Zookeeper & others
VP of Incubator at Apache Foundation
Email tdunning@apache.org tdunning@maprtech.com
Twitter @ted_dunning

3. © 2014 MapR Technologies 3
Goals
• Real-time or near-time
– Includes situations with deadlines
– Also includes situations where delay is simply undesirable
– Even includes situations where delay is just fine
• Micro-services
– Streaming is a convenient idiom for design
– Micro-services … you know we wanted it
– Service isolation is a key requirement

4. © 2014 MapR Technologies 4
Real-time or Near-time?
• The real point is flow versus state (catch me later today)
• One consequence of flow-based computing is real-time and
near-time become relatively easy
• Life may be a bitch, but it doesn’t happen in batches!

5. © 2014 MapR Technologies 6
Agenda
• Background / micro-services
• Global requirements
• Scale

6. © 2014 MapR Technologies 7
A microservice is
loosely coupled
with bounded context

7. © 2014 MapR Technologies 8
How to Couple Services and Break micro-ness
• Shared schemas, relational stores
• Ad hoc communication between services
• Enterprise service busses
• Brittle protocols
• Poor protocol versioning
• Single PoF schema repositories
Don’t do this!

8. © 2014 MapR Technologies 9
How to Decouple Services
• Use self-describing data
• Private database tables
• Infrastructural communication between services
• Use modern protocols
• Adopt future-proof protocol practices
• Use shared storage only where necessary due to scale

9. © 2014 MapR Technologies 11
What is the Right Structure for Flow Compute?
• Traditional message queues?
– Message queues are classic answer
– Key feature/bug is out-of-order acknowledgement
– Many implementations
– You pay a huge performance hit for persistence
• Kafka-esque Logs?
– Logs are like queues, but with ordering
– Out of order consumption is possible, acknowledgement not so much
– Canonical base implementation is Kafka
– Performance plus persistence

10. © 2014 MapR Technologies 12
Scenarios
Profile Database

11. © 2014 MapR Technologies 13
The task
?
POS 1
location, t, card #
yes/no?
POS 2
location, t, card #
yes/no?

12. © 2014 MapR Technologies 14
Traditional Solution
POS
1..n
Fraud
detector
Last card
use

13. © 2014 MapR Technologies 15
What Happens Next?
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector

14. © 2014 MapR Technologies 16
What Happens Next?
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector

15. © 2014 MapR Technologies 17
How to Get Service Isolation
POS
1..n
Fraud
detector
Last card
use
Updater
card activity

16. © 2014 MapR Technologies 18
New Uses of Data
POS
1..n
Fraud
detector
Last card
use
Updater
Card
location
history
Other
card activity

17. © 2014 MapR Technologies 19
Scaling Through Isolation
POS
1..n
Last card
use
Updater
POS
1..n
Last card
use
Updater
card activity
Fraud
detector
Fraud
detector

18. © 2014 MapR Technologies 20
Lessons
• De-coupling and isolation are key
• Private data stores/tables are important,
– But node local storage of private data is a bug
• Propagate events, not table updates
• Note that tables and streams should be as easy as files
– It should not be necessary to provision a cluster to get them

19. © 2014 MapR Technologies 21
Scenarios
IoT Data Aggregation

20. © 2014 MapR Technologies 22
Basic Situation
Each location
has many
pumps
pump data
Multiple
locations

21. © 2014 MapR Technologies 23
What Does a Pump Look Like
inlet
out let
m ot or
Temperature
Pressure
Flow
Temperature
Pressure
Flow
Winding temperature
Voltage
Current

22. © 2014 MapR Technologies 24
Basic Situation
Each location
has many
pumps
pump data
Multiple
locations

23. © 2014 MapR Technologies 25
pump data
pump data
pump data
pump data
Basic Architecture Reflects Business Structure

24. © 2014 MapR Technologies 26
Lessons
• Data architecture should reflect business structure
• Even very modest designs involve multiple data centers
• Schemas cannot be frozen in the real world
• Security must follow data ownership

25. © 2014 MapR Technologies 27
Scenarios
Global Data Recovery

26. © 2014 MapR Technologies 28
Tokyo
Corporate
HQ

27. © 2014 MapR Technologies 29
Singapore
Tokyo
Corporate
HQ

28. © 2014 MapR Technologies 30
Singapore
Tokyo
Corporate
HQ

29. © 2014 MapR Technologies 31
Singapore
Tokyo
Corporate
HQ

30. © 2014 MapR Technologies 32
Lessons
• Streams and tables are primitive building blocks
• Arbitrary number of topics important for simplicity + performance
• Updates happen in many places
• Mobility implies change in replication patterns
• Multi-master updates simplify design massively

31. © 2014 MapR Technologies 33
Converged Requirements

32. © 2014 MapR Technologies 34
What Have We Learned?
• Need persistence and performance
– Possibly for years and to 100’s of millions t/s
• Must have convergence
– Need files, tables AND streams
– Need volumes, snapshots, mirrors, permissions and …
• Must have platform security
– Cannot depend on perimeter
– Must follow business structure
• Must have global scale and scope
– Millions of topics for natural designs
– Multi-master replication and update

33. © 2014 MapR Technologies 35
The Importance of Common API’s
• Commonality and interoperability are critical
– Compare Hadoop eco-system and the noSQL world
• Table stakes
– Persistence
– Performance
– Polymorphism
• Major trend so far is to adopt Kafka API
– 0.9 API and beyond remove major abstraction leaks
– Kafka API supported by all major Hadoop vendors

34. © 2014 MapR Technologies 36
What we do at MapR

35. © 2014 MapR Technologies 37
Evolution of Data Storage
Functionality
Compatibility
Scalability
Linux
POSIX
Over decades of progress,
Unix-based systems have set the
standard for compatibility and
functionality

36. © 2014 MapR Technologies 38
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
Hadoop achieves much higher
scalability by trading away
essentially all of this compatibility
Evolution of Data Storage

37. © 2014 MapR Technologies 39
Evolution of Data Storage
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
MapR enhanced Apache Hadoop by
restoring the compatibility while
increasing scalability and performance
Functionality
Compatibility
Scalability
POSIX

38. © 2014 MapR Technologies 40
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
Evolution of Data Storage
Adding converged tables and streams
enhances the functionality of the base
file system

39. © 2014 MapR Technologies 41
http://bit.ly/fastest-big-data

40. © 2014 MapR Technologies 42
How we do this with MapR
• MapR Streams is a C++ reimplementation of Kafka API
– Advantages in predictability, performance, scale
– Common security and permissions with entire MapR converged data
platform
• Semantic extensions
– A cluster contains volumes, files, tables … and now streams
– Streams contain topics
– Can have default stream or can name stream by path name
• Core MapR capabilities preserved
– Consistent snapshots, mirrors, multi-master replication

41. © 2014 MapR Technologies 43
MapR core Innovations
• Volumes
– Distributed management
– Data placement
• Read/write random access file system
– Allows distributed meta-data
– Improved scaling
– Enables NFS access
• Application-level NIC bonding
• Transactionally correct snapshots and mirrors

42. © 2014 MapR Technologies 44
MapR's Containers
 Each container contains
 Directories & files
 Data blocks
 Replicated on servers
 No need to manage
directly
Files/directories are sharded into blocks, which
are placed into containers on disks
Containers are 16-
32 GB segments of
disk, placed on
nodes

43. © 2014 MapR Technologies 45
MapR's Containers
 Each container has a
replication chain
 Updates are transactional
 Failures are handled by
rearranging replication

44. © 2014 MapR Technologies 46
Container locations and replication
CLDB
N1, N2
N3, N2
N1, N2
N1, N3
N3, N2
N1
N2
N3Container location database
(CLDB) keeps track of nodes
hosting each container and
replication chain order

45. © 2014 MapR Technologies 47
MapR Scaling
Containers represent 16 - 32GB of data
 Each can hold up to 1 Billion files and directories
 100M containers = ~ 2 Exabytes (a very large cluster)
250 bytes DRAM to cache a container
 25GB to cache all containers for 2EB cluster
 But not necessary, can page to disk
 Typical large 10PB cluster needs 2GB
Container-reports are 100x - 1000x < HDFS block-reports
 Serve 100x more data-nodes
 Increase container size to 64G to serve 4EB cluster

46. © 2014 MapR Technologies 48
But Wait, There’s More
• Directories and files are implemented in terms of B-trees
– Key is offset, value is data blob
– Internal transactional semantics guarantees safety and consistency
– Layout algorithms give very high layout linearization
• Tables are implemented in terms of B-trees
– Twisted B-tree implementation allows virtues of log-structured merge
tree without the compaction delays
– Tablet splitting without pausing, integration with file system transactions
• Common security and permissions scheme

47. © 2014 MapR Technologies 49
And More …
• Streams are implemented in terms of B-trees as well
– Topics and consumer offsets are kept in stream, not ZK
– Similar splitting technology as MapR DB tables
– Consistent permissions, security, data replication
• Standard Kafka 0.9 API
• Plans to add OJAI for high-level structuring
• Performance is very high

48. © 2014 MapR Technologies 50
Example
Files
Table
Streams
Directories
Cluster
Volume mount point

49. © 2014 MapR Technologies 51
Cluster
Volume mount point

50. © 2014 MapR Technologies 52
Lessons
• API’s matter more than implementations
• There is plenty of room to innovate ahead of the community
• Posix, HDFS, HBASE all define useful API’s
• Kafka 0.9+ also defines a useful and broadly adopted API

51. © 2014 MapR Technologies 53
Call to action:
Require convergence

52. © 2014 MapR Technologies 54

53. © 2014 MapR Technologies 55
Short Books by Ted Dunning & Ellen Friedman
• Published by O’Reilly in 2014 - 2016
• For sale from Amazon or O’Reilly
• Free e-books currently available courtesy of MapR
http://bit.ly/ebook-real-
world-hadoop
http://bit.ly/mapr-tsdb-
ebook
http://bit.ly/ebook-
anomaly
http://bit.ly/recommend
ation-ebook

54. © 2014 MapR Technologies 56
Streaming Architecture
by Ted Dunning and Ellen Friedman © 2016 (published by O’Reilly)
Free copies at book
signing today
http://bit.ly/mapr-ebook-streams

55. © 2014 MapR Technologies 57
Thank You!

56. © 2014 MapR Technologies 58
Q&A
@mapr maprtech
tdunning@maprtech.com
Engage with us!
MapR
maprtech
mapr-technologies