Stratio is a big data platform that can combine stored and streaming data in real-time queries in less than 30 seconds. It uses Spark over Cassandra and MongoDB for processing. The platform provides tools like Spark Streaming, SQL (Shark), machine learning (MLLib), and graph processing (GraphX). Complex event processing (CEP) can analyze and correlate events from multiple sources to detect more complex patterns. Stratio Streaming provides a CEP engine with operators like filter, join, aggregation, and pattern/sequence processing over event streams. The platform also includes tools for easily designing extraction, transformation, and loading (ETL) workflows with Stratio Ingestion, which is based on Apache Flume.

2. Stratio is the only Big Data
platform able to combine, in
one query, stored data with
streaming data in real-time
(in less than 30 seconds).
We are polyglots as well: We
use Spark over two noSQL
databases, Cassandra &
Mongo DB.

4. •
•
•
•
•
Spark Streaming provides a high-level abstraction called discretized stream
or DStream, which represents a continuous stream of data, and in fact is
represented as a sequence of RDDs, which is Spark’s abstraction of an
immutable, distributed dataset.
Shark
(SQL)
Spark
Streaming
Mllib
(machine
learning)
GraphX
(graph)

5. • map(func), flatMap(func),
filter(func), count()
• repartition(numPartitions)
• union(otherStream)
• reduce(func),countByValue
(), reduceByKey(func,
[numTasks])
• join(otherStream,
[numTasks]), cogroup
(otherStream, [numTasks])
• transform(func)
• updateStateByKey(func)
• window(windowLength,
slideInterval)
• countByWindow(windowLength
, slideInterval)
• reduceByWindow(func,
windowLength, slideInterval)
• reduceByKeyAndWindow(func,
windowLength, slideInterval,
[numTasks])
• countByValueAndWindow(wind
owLength, slideInterval,
[numTasks])
• print()
• foreachRDD(func)
• saveAsObjectFiles(prefix,
[suffix])
• saveAsTextFiles(prefix,
[suffix])
• saveAsHadoopFiles(prefix,
[suffix])

6. •
Complex event
processing, or CEP,
is event
processing that
combines data
from multiple
sources to infer
events or patterns
that suggest more
complicated
circumstances
CEP as a
technique helps
discover complex
events by
analyzing and
correlating other
events
•

7. A CEP engine should provide operators over
streams, keeping in mind that events and streams
in a CEP are first-class citizens. In CEP, we think in
terms of event streams: event stream is a sequence
of events that arrives over time.
Users provide queries to the CEP engine whose
main mission is matching those queries against
events coming through event streams.
A CEP engine thus has notion of time and it allows
working with temporal queries that reason in terms
of temporal concepts, such as “time windows” or
“before and after” event relationships… among
others
• Filter
• Join
• Aggregation (Avg, Sum , Min, Max, Custom)
• Group by
• Having
• Conditions and Expressions (and, or, not, true/false,
==,!=, >=, >, <=, <),
• Data types (boolean, string, int, long, float, double)
• Pattern processing
• Sequence processing (zero to many, one to many, and
zero to one)

8. • You still have to integrate it in your code
• There is nothing like an interactive console
• If you want to do something with the streams, you guessed it, you have to code it!
• There is no way to remotely listen to a stream
• There are no solution patterns ready-to-use with the engine
• No statistics, no auditing
• Hard to integrate with other tools (dashboarding, log stream, batch processing)

10. With this solution you can use our API in
order to request commands to Stratio
Streaming engine in your code.
And you can also work with the interactive
shell in order to test your queries or interact
with the engine on demand.
Both tools, in fact, hide that you are sending
messages to a complex engine, built with
Zookeeper, Kafka, Spark Streaming and Siddhi
CEP Engine.

11. kafka
zookeeper
requests
events
CASSANDRA
Kafka
CASSANDRA
CASSANDRA

12. Kafka
CASSANDRA
•
•
•
•
•
•
•

13. 



14. • create --stream testStream --definition
• "name.string,data.double“
• insert --stream testStream --values
• "name.Temperature, field.testValue,data.33“
• save cassandra start --stream testStream
• alter --stream testStream --definition
"field.string"




16. CREATE --stream testStream –definition
(name.string,
data.double,
data2.int,
data3.float,
data4.double,
trueorfalse.boolean
)

17.  Filtering
 Projection
 In-built functions
 Windows (time and length)
 Join
 Event Sequences
There are a lot of CEP operators that you can use in your queries:
 Event Patterns
 Output rate limiting
 Custom windows, custom
functions
from sensor_grid #window.length(10) select name, ind,
avg(data) as data group by name insert into
sensor_grid_avg for current-events

18. 1. >, <, ==, >=, <=, !=
2. contains, instanceof
3. and, or, not
1. sum, avg, max, min, count: when aggregated (group by, having)
2. Field Type Conversion
3. Coalesce: if field null then take
another field
4. IsMatch: true or false if match regex
from orders[price >= 20 and price < 100]…
from orders select * insert into ordersB…
from orders select client, price insert into ordersB…

19. 1. Length window - a sliding window that keeps the last N events.
2. Time window - a sliding window that keeps events that have arrived within the last T time period.
3. Time and Length batch window : same concept but outputs events only at the end of the given window
4. Unique window - keeps only the latest events that are unique according to the given unique attribute.
5. First unique window - keeps the first events that are unique according to the given unique attribute.
6. External Time Window - a sliding window that processes according to timestamps defined externally
from payments[channel == ‘Paypal']#window.time( 1 min )

20. • With “on <condition>” joins only the events that matches the condition
• With “within <time>”, joins only the events that are within said time of each other
from errorStream#window.length(1) as errorStream join
allStream#window.length(1) as allStream
on errorStream.numberOfErrors > allStream.totalNumberOfEvents*0.05
select * insert into alarmByThreshold;

21. from every (a1 = infoStock[action == "buy"]
-> a2 = confirmOrder[command == "OK"] )
-> b1 = StockExchangeStream [price > infoStock.price]
within 3000
select a1.action as action, b1.price as price
insert into StockQuote
from every a1 = infoStock[action == "buy"]+,
b1 = StockExchangeStream[price > 70]?,
b2 = StockExchangeStream[price >= 75]
select a1[0].action as action, b1.price as priceA, b2.price as priceBJoin
insert into StockQuote

23. Apache Flume is a distributed,
reliable, and available system for
efficiently collecting, aggregating
and moving large amounts of log
data from many different sources
to a centralized data store.
Stratio Ingestion is an ETL for Big
Data product, based on Flume.
Design your workflows (wysiwyg)
with useful and improved sources
and sinks, transform your data on
the fly
• Create the stream if it
doesn’t exist
• It is possible to send
filtered event-flows
only to streaming
engine
• Built on the Stratio
Streaming API.

24. • Call-center Real-time monitoring
Real-time detection of client churn risk
Natural Language Processing Analysis to detect incidents in real-time
Anomaly detection in the service based on patterns
• IT services monitoring
DoS attack detection, hotlinking, etc in real-time
Warnings in monitoring of heterogeneous services
Preventive detection of downtime based on patterns
• Sensor grid monitoring
Alarms when thresholds are reached
Complex alarms involving several sensors
Real-time monitoring (landing support devices in an airport, for example)
Data Machine
Intelligence

25. SELECT sum(order.quantity), company_data.country
FROM streaming.order WITH WINDOW 15 minutes
INNER JOIN batch.company_data
ON order.company = company_data.company_name;
.
• With an powerful query planner
• Able to perform mixed queries with
streaming and batch data
SQL query example, mixing real-time data
(coming from Stratio Streaming Engine) and
batch data (stored in a noSQL database)

27. We are first going to use
the Shell to create
streams and queries.