The document outlines the development and architecture of a data processing system for modern data centers by Rocana, targeting high event throughput and low latency for diverse use cases such as fraud detection and user behavior analysis. It describes the system's guarantees, event modeling, custom event types, and processing mechanisms, emphasizing the importance of stream processing solutions and trade-offs involved. Additionally, it touches on data aggregation, late data handling, and system extensibility, while encouraging thorough evaluation of processing literature and technologies.

1. building a system for machine and
event-oriented data
e. sammer | @esammer | september 9, 2015
silicon valley data engineering meetup

2. © 2015 Rocana, Inc. All Rights Reserved.
context

3. © 2015 Rocana, Inc. All Rights Reserved.
me
3
• i work here: rocana – cto and cofounder
• i used to work here: cloudera (‘10 – ’14), magnetic, experian, …
• i do this: systems / distributed systems (storage, query, messaging, ...)
• i wrote this:

4. © 2015 Rocana, Inc. All Rights Reserved.
what we do
4
• we build a system for the operation of modern data centers
• triage and diagnostics, exploration, trends, advanced analytics of complex
systems
• our data: logs, metrics, human activity, anything that occurs in the data center
• “enterprise software” (i.e. we build for others.)
• today: how we built what we built

5. © 2015 Rocana, Inc. All Rights Reserved.
our typical customer use cases
5
• >100K events / sec (8.6B events / day), sub-second end to end latency, full
fidelity retention, critical use cases
• quality of service - “are credit card transactions happening fast enough?”
• fraud detection - “detect, investigate, prosecute, and learn from fraud.”
• forensic diagnostics - “what really caused the outage last friday?”
• security - “who’s doing what, where, when, why, and how, and is that ok?”
• user behavior - ”capture and correlate user behavior with system performance,
then feed it to downstream systems in realtime.”

6. © 2015 Rocana, Inc. All Rights Reserved.
depth: 3 meters

7. © 2015 Rocana, Inc. All Rights Reserved.
high level architecture
7

8. © 2015 Rocana, Inc. All Rights Reserved.
guarantees
8
• no single point of failure exists
• all components scale horizontally[1]
• data retention and latency is a function of cost, not tech[1]
• every event is delivered provided no more than N - 1 failures occur (where N is
the kafka replication level)
• all operations, including upgrade, are online[2]
• every event is (or appears to be) delivered exactly once[3]
[1] we’re positive there’s a limit, but thus far it has been cost.
[2] from the user’s perspective, at a system level.
[3] when queried via our UI. lots of details here.

9. © 2015 Rocana, Inc. All Rights Reserved.
events

10. © 2015 Rocana, Inc. All Rights Reserved.
modeling our world
10
• everything is an event
• each event contains a timestamp, type, location, host, service, body, and type-
specific attributes (k/v pairs)
• build specialized aggregates as necessary - just optimized views of the data

11. © 2015 Rocana, Inc. All Rights Reserved.
event schema
11
{
id: string,
ts: long,
event_type_id: int,
location: string,
host: string,
service: string,
body: [ null, string ],
attributes: map<string>
}

12. © 2015 Rocana, Inc. All Rights Reserved.
event types
12
• some event types are standard
– syslog, http, log4j, generic text record, …
• users define custom event types
• producers populate event type
• transformations can turn one event type into another
• event type metadata tells downstream systems how to interpret body and
attributes

13. © 2015 Rocana, Inc. All Rights Reserved.
ex: generic syslog event
13
event_type_id: 100, // rfc3164, rfc5424 (syslog)
body: … // raw syslog message bytes
attributes: { // extracted fields from body
syslog_message: “DHCPACK from 10.10.0.1 (xid=0x45b63bdc)”,
syslog_severity: “6”, // info severity
syslog_facility: “3”, // daemon facility
syslog_process: “dhclient”,
syslog_pid: “668”,
…
}

14. © 2015 Rocana, Inc. All Rights Reserved.
ex: generic http event
14
event_type_id: 102, // generic http event
body: … // raw http log message bytes
attributes: {
http_req_method: “GET”,
http_req_vhost: “w2a-demo-02”,
http_req_path: “/api/v1/search?q=service%3Asshd&p=1&s=200”,
http_req_query: “q=service%3Asshd&p=1&s=200”,
http_resp_code: “200”,
…
}

15. © 2015 Rocana, Inc. All Rights Reserved.
consumers

16. © 2015 Rocana, Inc. All Rights Reserved.
consumers
16
• …do most of the work
• parallelism
• kafka offset management
• message de-duplication
• transformation (embedded library)
• dead letter queue support
• downstream system knowledge

17. © 2015 Rocana, Inc. All Rights Reserved.
consumers
17
• …do most of the work
• parallelism
• kafka offset management
• message de-duplication
• transformation (embedded library)
• dead letter queue support
• downstream system knowledge

18. © 2015 Rocana, Inc. All Rights Reserved.
inside a consumer
18

19. © 2015 Rocana, Inc. All Rights Reserved.
metrics and time series

20. © 2015 Rocana, Inc. All Rights Reserved.
aggregation
20
• mostly for time series metrics
• two halves: on write and on query
• data model: (dimensions) => (aggregates)
• on write
– reduce(a: A, b: A) => A over window
– store “base” aggregates, all associative and commutative
• on query
– perform same aggregate or derivative aggregates
– group by the same dimensions
– we use SQL (Impala)

21. © 2015 Rocana, Inc. All Rights Reserved.
aside: late arriving data (it’s a thing)
21
• never trust a (wall) clock
• producer determines observation time, rest of the system uses this always
• data that shows up late always processed according to observation time
• aggregation consequences
– the same time window can appear multiple times
– solution: aggregate every N seconds, potentially generating multiple aggregates for
the same time bin
• this is real and you must deal with it
– do what we did or
– build a system that mutates/replaces aggregates already output (eww) or
– delay aggregate output for some slop time; drop it if late data shows up

22. © 2015 Rocana, Inc. All Rights Reserved.
ex: service event volume by host and minute
22
• dimensions: ts, window, location, host, service, metric
• on write, aggregates: count, sum, min, max, last
• epoch, 60000, us-west-2a, w2a-demo-1, sshd, event_volume =>
17, 42, 1, 10, 8
• on query:
– SELECT floor(ts / 60000) as bin, host, service, metric, sum(value_sum) FROM
events WHERE ts BETWEEN x AND y AND metric = ”event_volume” GROUP BY
bin, host, service, metric
• if late arriving data existed in events, the same dimensions would repeat with a
another set of aggregates and would be rolled up as a result of the group by
• tl;dr: normal window aggregation operations

23. © 2015 Rocana, Inc. All Rights Reserved.
extension, pain, and advice

24. © 2015 Rocana, Inc. All Rights Reserved.
extending the system
24
• custom producers
• custom consumers
• event types
• parser / transformation plugins
• custom metric definition and aggregate functions
• custom processing jobs on landed data

25. © 2015 Rocana, Inc. All Rights Reserved.
pain (aka: the struggle is real)
25
• lots of tradeoffs when picking a stream processing solution
– samza: right features, but low level programming model, not supported by vendors.
missing security features.
– storm: too rigid, too slow. not supported by all Hadoop vendors.
– spark streaming: tons of issues initially, but lots of community energy. improving.
– @digitallogic: “my heart says samza, but my head says spark streaming.”
– our (current) needs are meager; do work inside consumers.
• stack complexity, (relative im)maturity
• scaling solr cloud to billions of events per day

26. © 2015 Rocana, Inc. All Rights Reserved.
if you’re going to try this…
26
• read all the literature on stream processing[1]
• treat it like the distributed systems problem it is
• understand, make, and make good on guarantees
• find the right abstractions
• never trust the hand waving or “hello worlds”
• fully evaluate the projects/products in this space
• understand it’s not just about search
[1] wait, like all of it? yea, like all of it.

27. © 2015 Rocana, Inc. All Rights Reserved.
things I didn’t talk about
27
• reprocessing data when bad code / transformations are detected
• dealing with data quality issues (“the struggle is real” part 2)
• the user interface and all the fancy analytics
– data visualization and exploration
– event search
– anomalous trend and event detection
– metric, source, and event correlation
– motif finding
– noise reduction and dithering
• event delivery semantics (e.g. at least once, exactly once, etc.)
• alerting

28. © 2015 Rocana, Inc. All Rights Reserved.
questions?
thank you.
@esammer | esammer@rocana.com