In this session, we’ll follow the flow of data through an end-to-end system built to handle tens of terabytes an hour of event-oriented data, providing real-time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive can be stitched together to form the base platform; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. Finally, a brief demo of Rocana Ops, an application for large scale data center operations, will be given, along with an explanation about how it uses the underlying platform.

1. building a system for machine and
event-oriented data
e. sammer | @esammer | september 9, 2015
silicon valley data engineering meetup

2. © 2015 Rocana, Inc. All Rights Reserved.
context

3. © 2015 Rocana, Inc. All Rights Reserved.
me
3
• i work here: rocana – cto and cofounder
• i used to work here: cloudera (‘10 – ’14), magnetic, experian, …
• i do this: systems / distributed systems (storage, query, messaging, ...)
• i wrote this:

4. © 2015 Rocana, Inc. All Rights Reserved.
what we do
4
• we build a system for the operation of modern data centers
• triage and diagnostics, exploration, trends, advanced analytics of complex
systems
• our data: logs, metrics, human activity, anything that occurs in the data center
• “enterprise software” (i.e. we build for others.)
• today: how we built what we built

5. © 2015 Rocana, Inc. All Rights Reserved.
our typical customer use cases
5
• >100K events / sec (8.6B events / day), sub-second end to end latency, full
fidelity retention, critical use cases
• quality of service - “are credit card transactions happening fast enough?”
• fraud detection - “detect, investigate, prosecute, and learn from fraud.”
• forensic diagnostics - “what really caused the outage last friday?”
• security - “who’s doing what, where, when, why, and how, and is that ok?”
• user behavior - ”capture and correlate user behavior with system performance,
then feed it to downstream systems in realtime.”

6. © 2015 Rocana, Inc. All Rights Reserved.
depth: 3 meters

7. © 2015 Rocana, Inc. All Rights Reserved.
high level architecture
7

8. © 2015 Rocana, Inc. All Rights Reserved.
guarantees
8
• no single point of failure exists
• all components scale horizontally[1]
• data retention and latency is a function of cost, not tech[1]
• every event is delivered provided no more than N - 1 failures occur (where N is
the kafka replication level)
• all operations, including upgrade, are online[2]
• every event is (or appears to be) delivered exactly once[3]
[1] we’re positive there’s a limit, but thus far it has been cost.
[2] from the user’s perspective, at a system level.
[3] when queried via our UI. lots of details here.

9. © 2015 Rocana, Inc. All Rights Reserved.
events

10. © 2015 Rocana, Inc. All Rights Reserved.
modeling our world
10
• everything is an event
• each event contains a timestamp, type, location, host, service, body, and type-
specific attributes (k/v pairs)
• build specialized aggregates as necessary - just optimized views of the data

11. © 2015 Rocana, Inc. All Rights Reserved.
event schema
11
{
id: string,
ts: long,
event_type_id: int,
location: string,
host: string,
service: string,
body: [ null, string ],
attributes: map<string>
}

12. © 2015 Rocana, Inc. All Rights Reserved.
event types
12
• some event types are standard
– syslog, http, log4j, generic text record, …
• users define custom event types
• producers populate event type
• transformations can turn one event type into another
• event type metadata tells downstream systems how to interpret body and
attributes

13. © 2015 Rocana, Inc. All Rights Reserved.
ex: generic syslog event
13
event_type_id: 100, // rfc3164, rfc5424 (syslog)
body: … // raw syslog message bytes
attributes: { // extracted fields from body
syslog_message: “DHCPACK from 10.10.0.1 (xid=0x45b63bdc)”,
syslog_severity: “6”, // info severity
syslog_facility: “3”, // daemon facility
syslog_process: “dhclient”,
syslog_pid: “668”,
…
}

14. © 2015 Rocana, Inc. All Rights Reserved.
ex: generic http event
14
event_type_id: 102, // generic http event
body: … // raw http log message bytes
attributes: {
http_req_method: “GET”,
http_req_vhost: “w2a-demo-02”,
http_req_path: “/api/v1/search?q=service%3Asshd&p=1&s=200”,
http_req_query: “q=service%3Asshd&p=1&s=200”,
http_resp_code: “200”,
…
}

15. © 2015 Rocana, Inc. All Rights Reserved.
consumers

16. © 2015 Rocana, Inc. All Rights Reserved.
consumers
16
• …do most of the work
• parallelism
• kafka offset management
• message de-duplication
• transformation (embedded library)
• dead letter queue support
• downstream system knowledge

17. © 2015 Rocana, Inc. All Rights Reserved.
consumers
17
• …do most of the work
• parallelism
• kafka offset management
• message de-duplication
• transformation (embedded library)
• dead letter queue support
• downstream system knowledge

18. © 2015 Rocana, Inc. All Rights Reserved.
inside a consumer
18

19. © 2015 Rocana, Inc. All Rights Reserved.
metrics and time series

20. © 2015 Rocana, Inc. All Rights Reserved.
aggregation
20
• mostly for time series metrics
• two halves: on write and on query
• data model: (dimensions) => (aggregates)
• on write
– reduce(a: A, b: A) => A over window
– store “base” aggregates, all associative and commutative
• on query
– perform same aggregate or derivative aggregates
– group by the same dimensions
– we use SQL (Impala)

21. © 2015 Rocana, Inc. All Rights Reserved.
aside: late arriving data (it’s a thing)
21
• never trust a (wall) clock
• producer determines observation time, rest of the system uses this always
• data that shows up late always processed according to observation time
• aggregation consequences
– the same time window can appear multiple times
– solution: aggregate every N seconds, potentially generating multiple aggregates for
the same time bin
• this is real and you must deal with it
– do what we did or
– build a system that mutates/replaces aggregates already output (eww) or
– delay aggregate output for some slop time; drop it if late data shows up

22. © 2015 Rocana, Inc. All Rights Reserved.
ex: service event volume by host and minute
22
• dimensions: ts, window, location, host, service, metric
• on write, aggregates: count, sum, min, max, last
• epoch, 60000, us-west-2a, w2a-demo-1, sshd, event_volume =>
17, 42, 1, 10, 8
• on query:
– SELECT floor(ts / 60000) as bin, host, service, metric, sum(value_sum) FROM
events WHERE ts BETWEEN x AND y AND metric = ”event_volume” GROUP BY
bin, host, service, metric
• if late arriving data existed in events, the same dimensions would repeat with a
another set of aggregates and would be rolled up as a result of the group by
• tl;dr: normal window aggregation operations

23. © 2015 Rocana, Inc. All Rights Reserved.
extension, pain, and advice

24. © 2015 Rocana, Inc. All Rights Reserved.
extending the system
24
• custom producers
• custom consumers
• event types
• parser / transformation plugins
• custom metric definition and aggregate functions
• custom processing jobs on landed data

25. © 2015 Rocana, Inc. All Rights Reserved.
pain (aka: the struggle is real)
25
• lots of tradeoffs when picking a stream processing solution
– samza: right features, but low level programming model, not supported by vendors.
missing security features.
– storm: too rigid, too slow. not supported by all Hadoop vendors.
– spark streaming: tons of issues initially, but lots of community energy. improving.
– @digitallogic: “my heart says samza, but my head says spark streaming.”
– our (current) needs are meager; do work inside consumers.
• stack complexity, (relative im)maturity
• scaling solr cloud to billions of events per day

26. © 2015 Rocana, Inc. All Rights Reserved.
if you’re going to try this…
26
• read all the literature on stream processing[1]
• treat it like the distributed systems problem it is
• understand, make, and make good on guarantees
• find the right abstractions
• never trust the hand waving or “hello worlds”
• fully evaluate the projects/products in this space
• understand it’s not just about search
[1] wait, like all of it? yea, like all of it.

27. © 2015 Rocana, Inc. All Rights Reserved.
things I didn’t talk about
27
• reprocessing data when bad code / transformations are detected
• dealing with data quality issues (“the struggle is real” part 2)
• the user interface and all the fancy analytics
– data visualization and exploration
– event search
– anomalous trend and event detection
– metric, source, and event correlation
– motif finding
– noise reduction and dithering
• event delivery semantics (e.g. at least once, exactly once, etc.)
• alerting

28. © 2015 Rocana, Inc. All Rights Reserved.
questions?
thank you.
@esammer | esammer@rocana.com