This document discusses temporal operators for Spark Streaming and their application for Office365 service monitoring. It introduces temporal operators like reorder, event-time window aggregate, and join that allow normalizing and aggregating streaming data that may arrive out of order. These operators are used to calculate availability metrics over tumbling windows and generate alarms when availability drops between the current and previous windows. Examples show how temporal operators reorder, aggregate, and join streaming events based on timestamps to perform time-based analysis for monitoring Office365 services.

1. TEMPORAL OPERATORS FOR SPARK
STREAMING AND ITS APPLICATION FOR
OFFICE365 SERVICE MONITORING
Jin Li, Wesley Miao
Microsoft

2. Monitoring
Target
Service
Kafka
Spark Streaming
ODataAPI
Collectors
aggregate monitor
DataCenter
Management
Service
Recovery
Actions
Aggregated
Events
Alerts
Recovery
Actions
Raw Signals Topology
Data
DashboardsCassandra
Temporal
operators
Temporal
operators
normalize
Temporal
operators
RawSignals
TopologyData
Normalized
Events
Alerts
Paging Alerts
Oncall Engineer

3. Office365 Service Monitoring
• Service Monitoring Signals
– Local active monitoring
– Schema: <TopologyScopeValue, IsSuccess, …>
– TopologyScopeValue: grouping of hosts
• e.g. server, rack, site, region
• Data may arrive out of order
• Application logic defined on data time

4. Spark Streaming
aggregate monitor
Recovery
Actions
Temporal
operators
Temporal
operators
normalize
RawSignals
Topology
Data
Alerts
reorder
Temporal
operators
val rawSignals : DStream[T]
val topologyData: DStream[T1]
val normalized =
rawSignals.join(topologyData, …)

5. Spark Streaming
aggregate monitor
Recovery
Actions
Temporal
operators
Temporal
operators
normalize
RawSignals
Topology
Data
Alerts
reorder
Temporal
operators
val reordered =
normalized.reorder(Policy.ReorderAdjustAndDrop(Seconds(60), Seconds(120)))

6. Spark Streaming
aggregate monitor
Recovery
Actions
Temporal
operators
Temporal
operators
normalize
RawSignals
Topology
Data
Alerts
reorder
Temporal
operators
val aggregated = reordered.aggregate(TumblingWindow(5 minutes),
groupBy(event.rack, event.site, event.region),
Sum(event => event.isSuccess),
Count(),
Avg(event => event.latency)
)
val availabilityStats = aggregated.map{…}

7. Spark Streaming
aggregate monitor
Recovery
Actions
Temporal
operators
Temporal
operators
normalize
RawSignals
Topology
Data
Alerts
reorder
Temporal
operators
val monitoringStats = availabilityStats.join(availabilityStats,
left => JoinKey(left.topologyScopeValue),
right => JoinKey(right.topologyScopeValue),
(left, right) => left.availability < 0.99 &&
right.availability >= 0.99
TimeDiff(Seconds(-300), Seconds(0))

8. Temporal operators in depth

9. Temporal Operators – Reorder
• Reorder(policy)
– inputStream
.map { e => StreamEvent(e.timestamp, e) }
.reorder(Policy.Reorder(Seconds(60)))
10:00:0010:00:0110:00:03
10:00:0209:59:01
10:00:0010:00:0110:00:03 10:00:02… …
10:01:01
10:00:00
10:00:01
10:00:03 10:00:02… …10:01:01
HWM: 10:00:03HWM: 10:00:03HWM: 10:01:01

10. Temporal Operators – Event-time
window aggregate
• Basic Availability Metric
– SuccessEvents / TotalEvents for the every 5 minutes
• Tumbling window sum
• Event-time window aggregate
• val availabilityMetrics = input
.map { e => StreamEvent(e.timestamp, e) }
.reorder(Policy.ReorderAdjustAndDrop(Seconds(60), Seconds(120)))
.aggregate(
TumblingWindow(Seconds(300)),
event => GroupByKey((event.topologyScopeValue, “topologyScopeValue”)),
Sum(event => event.isSuccess, “sumOfSuccessEvent”),
Count(event => event, “sumOfTotalEvent”)
)
.map (…)

11. Temporal Operators – Event-time
window aggregate
• Implementation
(rack1) (SumState(150), CountState(170))
window: 10:00:00 -10:05:00
(10:03:00, rack1, 1)
(10:04:49, rack2, 0)
(rack2) (SumState(230), CountState(260))(10:05:01, rack2, 1)
… …
(10:05:00, rack1, 151, 171)
(10:05:00, rack2, 230, 261)
window: 10:00:05 -10:05:10
( … … )
(rack2) (SumState(1), CountState(1))(SumState(151), CountState(171))
(SumState(230), CountState(261))

12. Temporal Operators – Join
• Alarm
– SuccessEvents/TotalEvents > threshold for the current 5 minutes, but not the previous
5 minutes
• Temporal self join
• Temporal Join
availabilityMetrics.join(
availabilityMetrics,
left => JoinKey(x.topologyScopeValue,“leftTopologyScopeValue”),
right => JoinKey(y.topologyScopeValue,“rightTopologyScopeValue”),
(left, right) => left.availability < 0.99 && right.availability >= 0.99
TimeDiff(Seconds(-300), Seconds(0))
)

13. Temporal Operators – Join
• Temporal condition for Join
– TimeDiff(Seconds(-300), Seconds(0))
Left Right
-5 minutes
10:00:00
09:55:00
10:00:00

14. THANK YOU.
Jin Li (jliin@microsoft.com )
Wesley Miao (wemia@Microsoft.com )

15. Questions?