Spark Summit 2016 talk by Jin Li (Microsoft) and Wesley Miao (Microsoft)

1. TEMPORAL OPERATORS FOR SPARK
STREAMING AND ITS APPLICATION FOR
OFFICE365 SERVICE MONITORING
Jin Li, Wesley Miao
Microsoft

2. Monitoring
Target
Service
Kafka
Spark Streaming
ODataAPI
Collectors
aggregate monitor
DataCenter
Management
Service
Recovery
Actions
Aggregated
Events
Alerts
Recovery
Actions
Raw Signals Topology
Data
DashboardsCassandra
Temporal
operators
Temporal
operators
normalize
Temporal
operators
RawSignals
TopologyData
Normalized
Events
Alerts
Paging Alerts
Oncall Engineer

3. Office365 Service Monitoring
• Service Monitoring Signals
– Local active monitoring
– Schema: <TopologyScopeValue, IsSuccess, …>
– TopologyScopeValue: grouping of hosts
• e.g. server, rack, site, region
• Data may arrive out of order
• Application logic defined on data time

4. Spark Streaming
aggregate monitor
Recovery
Actions
Temporal
operators
Temporal
operators
normalize
RawSignals
Topology
Data
Alerts
reorder
Temporal
operators
val rawSignals : DStream[T]
val topologyData: DStream[T1]
val normalized =
rawSignals.join(topologyData, …)

5. Spark Streaming
aggregate monitor
Recovery
Actions
Temporal
operators
Temporal
operators
normalize
RawSignals
Topology
Data
Alerts
reorder
Temporal
operators
val reordered =
normalized.reorder(Policy.ReorderAdjustAndDrop(Seconds(60), Seconds(120)))

6. Spark Streaming
aggregate monitor
Recovery
Actions
Temporal
operators
Temporal
operators
normalize
RawSignals
Topology
Data
Alerts
reorder
Temporal
operators
val aggregated = reordered.aggregate(TumblingWindow(5 minutes),
groupBy(event.rack, event.site, event.region),
Sum(event => event.isSuccess),
Count(),
Avg(event => event.latency)
)
val availabilityStats = aggregated.map{…}

7. Spark Streaming
aggregate monitor
Recovery
Actions
Temporal
operators
Temporal
operators
normalize
RawSignals
Topology
Data
Alerts
reorder
Temporal
operators
val monitoringStats = availabilityStats.join(availabilityStats,
left => JoinKey(left.topologyScopeValue),
right => JoinKey(right.topologyScopeValue),
(left, right) => left.availability < 0.99 &&
right.availability >= 0.99
TimeDiff(Seconds(-300), Seconds(0))

8. Temporal operators in depth

9. Temporal Operators – Reorder
• Reorder(policy)
– inputStream
.map { e => StreamEvent(e.timestamp, e) }
.reorder(Policy.Reorder(Seconds(60)))
10:00:0010:00:0110:00:03
10:00:0209:59:01
10:00:0010:00:0110:00:03 10:00:02… …
10:01:01
10:00:00
10:00:01
10:00:03 10:00:02… …10:01:01
HWM: 10:00:03HWM: 10:00:03HWM: 10:01:01

10. Temporal Operators – Event-time
window aggregate
• Basic Availability Metric
– SuccessEvents / TotalEvents for the every 5 minutes
• Tumbling window sum
• Event-time window aggregate
• val availabilityMetrics = input
.map { e => StreamEvent(e.timestamp, e) }
.reorder(Policy.ReorderAdjustAndDrop(Seconds(60), Seconds(120)))
.aggregate(
TumblingWindow(Seconds(300)),
event => GroupByKey((event.topologyScopeValue, “topologyScopeValue”)),
Sum(event => event.isSuccess, “sumOfSuccessEvent”),
Count(event => event, “sumOfTotalEvent”)
)
.map (…)

11. Temporal Operators – Event-time
window aggregate
• Implementation
(rack1) (SumState(150), CountState(170))
window: 10:00:00 -10:05:00
(10:03:00, rack1, 1)
(10:04:49, rack2, 0)
(rack2) (SumState(230), CountState(260))(10:05:01, rack2, 1)
… …
(10:05:00, rack1, 151, 171)
(10:05:00, rack2, 230, 261)
window: 10:00:05 -10:05:10
( … … )
(rack2) (SumState(1), CountState(1))(SumState(151), CountState(171))
(SumState(230), CountState(261))

12. Temporal Operators – Join
• Alarm
– SuccessEvents/TotalEvents > threshold for the current 5 minutes, but not the previous
5 minutes
• Temporal self join
• Temporal Join
availabilityMetrics.join(
availabilityMetrics,
left => JoinKey(x.topologyScopeValue,“leftTopologyScopeValue”),
right => JoinKey(y.topologyScopeValue,“rightTopologyScopeValue”),
(left, right) => left.availability < 0.99 && right.availability >= 0.99
TimeDiff(Seconds(-300), Seconds(0))
)

13. Temporal Operators – Join
• Temporal condition for Join
– TimeDiff(Seconds(-300), Seconds(0))
Left Right
-5 minutes
10:00:00
09:55:00
10:00:00

14. THANK YOU.
Jin Li (jliin@microsoft.com )
Wesley Miao (wemia@Microsoft.com )

15. Questions?