Mantis - A Reactive Stream processing engine

1. Mantis in Action
Neeraj Joshi and Justin Becker
6/12/2015

2. Managing a complex
operational environment
is hard

6. Developing an understanding
of what is going on
Knowing what works

7. Developing an understanding
of what is going on
Identify what doesn’t work

8. Developing an understanding
of what is going on
Determining impact when doesn’t work

9. Developing a deep understanding
is hard, due to complexity
• Hundreds of software services
• Processing billions of requests
• For millions of users
• Operating in multiple data centers
• Across the globe

10. Help us (operators) make sense
out of complexity, need tools
specifically, we need…
Insight tools
…to help comprehend what is going on in our
operational environments

11. Make the case a relationship exists,
between complexity and comprehension

12. So, in order to manage complex
environments, need to rethink insights, shift
the curve

13. Identified three insight
‘patterns’ to help shift the curve
• Long tail analysis
• Real time tracking and trending
• Ad hoc investigation

14. Grouped three patterns
into a new effort

15. Scalable Insights
Initiative

16. Goal is to help us manage
(comprehend) our environments
given an increase in complexity

17. Mention specific insights
tools to help shift curve
• Realtime Data Explorer
• Realtime Search
• Realtime Application Monitoring

18. Mention other generic insights
jobs to help shift curve
• Short term historical anomaly detection
• Threshold-based anomaly detection
• Realtime metrics generation

19. Overview of scalable
insights in action
• 65-75 total jobs running in 3 regions
• Global access to data
• Processing 4.7 million events per second at peak
• 20 specific data sources and generic adapters
• Ability to startup jobs in ~5 seconds

20. Demo

25. Mantis, a reactive stream
processing system

26. Some Basics Concepts

27. Stream
Sequence of Events
time
1 2 3 4

28. Higher Order Functions
Transformations applied to a Stream
to create a new stream
time
1 4 9 16
map (x=>sqrt(x))
time
1 2 3 4

29. S
O
U
R
C
E
Stream IN
S
I
N
K
Result OUT
Stage 1 Stage 2 Stage n
A sequence of functions applied to a stream
map window reduce merge
Mantis Job

30. Mantis Job
S
O
U
R
C
E
Stream IN
S
I
N
K
Result OUT
Stage 1 Stage 2 Stage n

31. Named Jobs
Text
aa
Job
Name Job
Version

32. Are Parameterized
Parameter
Parameter
Parameter

33. Have SLAs
Min/Max
Instances of the
job
Min/Max
runtime
Perpetual or
Transient

34. Can be chained
Device
Logs Source
Job
TopN
Job
Server
Logs Source
Job
Anomaly
Detector
Job
Metrics
Aggregator
Job
Alert
Service

35. That is fine but...

36. How does Mantis meet
the Scalable Insights
challenge?

37. • Cost (Utilization)
Sensitive
• Optimize for low latency
• High Throughput
• Resilient
Key Requirements

38. Minimizing Costs

39. Elastic Clusters

40. Elastic Jobs

41. Job Autoscaling
Scaling Config
CPU Strategy
Network
Strategy

42. Filtering at Source
Source
Job
Consumer
Job
Data Producers

43. Low latency - High
Throughput

44. To block or not to
block?

45. RxNetty (non-blocking)
vs Tomcat (blocking)
by Brendan Gregg
@brendangregg
https://docs.google.com/presentation/d/18i-d72m7tD4wKlzm-1PCR8g62l66_9Btbg5-fuFRqf0/edit#slide=id.g761289dab_0_77

46. CPU consumption
• RxNetty
consumes less
CPU / request
• Reduced thread
migration
• Lower object
allocation rate
CPU consumed reduces
as load increases

47. Lower latency
• RxNetty has
lower latency
under high load
• fewer lock
contentions
• fewer thread
migrations
Latency knee for
Tomcat ~ 400
Latency knee for Netty
~700

48. Async Processing
Non-blocking I/O
Async Processing

49. Designed for Resilience
http://signsofpolitics.blogspot.com/2009/03/around-and-about-resilience.html

50. Server Resilience
• Servers crashes
inevitable
• Server health constantly
monitored with
heartbeats
• Crashed servers
replaced
• Lost jobs relaunched

51. Network Resilience
• Long lived connections
can fail
• Connection topology is
constantly monitored
and corrected

52. Backpressure

53. Cold Source
Amazon SQS

54. Hot Source

55. Reactive push-pull
(Cold Source)
f1 f2 f3 f4
Cold
Source
102410241024
1024
100
100 100 100

56. Push mode
f1 f2 f3 f4
Cold
Source
MaxMaxMax
Max
100
100 100 100

57. Pull mode
f1 f2 f3 f4
Cold
Source
111
1
1
1 1 1

58. Backpressure Strategies
(Hot Source)
f2 f3 f4
Hot
Source
101010
100
10 10 10
Strategy
Function
90
Drop
Buffer
Scale-up

59. Questions