Presenter: Antonio Alcocer, Big Data Architect at Stratio Telefonica is the incumbent telecommunications network operator in Spain and the fourth one in capitalisation in the world. Cyber security is one of our most successful businesses worldwide. We provide monitoring and protecting clients from attacks. We analyze millions of data from multiple sources including social media, DNS records, and underground internet, to generate alerts and security reports for our clients. This use case required a Big Data component capable of processing the data and extract its information in real-time; warnings and alerts are time-sensitive in order to deal efficiently with security attacks. Our original architecture was the typical one used for data fusion systems. It included several collectors, a processing layer based on legacy systems, and a data store. The initial setup included a MongoDB database and an ad-hoc application. This solution however proved to be unfit for the specific purpose of dispatching alerts. We proposed to use Cassandra and Spark instead. This approach did manage to fulfill our original specifications as intended. Our talk will explain the reasons why we migrated the architecture and how the adopted solution based on Spark and Cassandra solved our problem.

1. Spark Use Case at
Telefónica CyberSecurity (CBS)
Antonio Alcocer
antonio@stratio.com
Oscar Mendez
oscar@stratio.com
#CassandraSummit 2014 1

2. Who are we?
STRATIO
• Stratio is a Big Data Company
• Founded in 2013
• Commercially launched in 2014
• 50+ employees in Madrid
• Office in San Francisco
• Certified Spark distribution
#CassandraSummit 2014 2

3. 3

4. General info
o 1924- 2014: 317+ customer with 130.000+ employees
o 2nd European operator by revenues
o 4th global integrated operator by accesses
o 9th Telco in the Global ranking by market capitalization
o 2nd global operator for investment in R+D
#CassandraSummit 2014 4

5. Present in 24 countries
#CassandraSummit 2014 5

6. Their main brands
#CassandraSummit 2014 6

7. Other brands
#CassandraSummit 2014 7

8. Telefónica Global Solutions
Global Security Services
A global infrastructure to
safeguard your business_
#CassandraSummit 2014 8

9. Managed Security
#CassandraSummit 2014 9

10. CyberSecurity??
10

11. Why????
A picture is worth a thousand words - but a film clip, a million!
#CassandraSummit 2014 11

12. Don’t worry…
#CassandraSummit 2014 12

13. What is Cybersecurity?
What does it mean for us?
“Cybersecurity is the collection of tools, policies… capabilities to protect
the cyber environment and organization and user’s assets. Cybersecurity
strives to ensure unauthorized access to, manipulation of the integrity,
confidentiality, or availability of an information, or unauthorized
exfiltration of information.”
No rules, just guidelines.
#CassandraSummit 2014 13

14. An example of threats
Cassandra OpsCenter
World map
Wordpress
#CassandraSummit 2014 14

15. C* OpsCenter + Shodan
#CassandraSummit 2014 15

16. C* OpsCenter + Shodan
#CassandraSummit 2014 16

17. Another threats
#CassandraSummit 2014 17

18. CyberSecurity in
numbers
18

19. Numbers
Threats
• DDoS (23%)
• SQLi (19%)
• Defacement (14%)
• Account Hijacking (9%)
• Unknown (18%)
#CassandraSummit 2014 19

20. Looking for unknown threats
#CassandraSummit 2014 20

21. What did Telefonica need?
#CassandraSummit 2014 21

22. Telefonica first concept test
• Based in Mongo DB.
• Problem: the data up faster than server.
• They have their own collectors in python
to connect to Mongo DB.
• Not solve all analyst queries in batch.
• No real time processing
Collector
Collector
Collector
Collector
Mongo
#CassandraSummit 2014 22

23. Joining efforts
23

24. Required skills
#CassandraSummit 2014 24

25. in
25
Using

26. Use Case Architecture
We have three phases:
• Ingestion: based on Apache Kafka
• Data fusion: based on Apache Storm.
• Batch & Analytics: Based on Cassandra
and Spark
#CassandraSummit 2014 26

27. Data Adquisition
• Data are in several sources:
• DNS traffic
• IP
• Social media
• Underground sources
• Government sources
• …
Data sources
Sources
Sources
Sources
Sources
Sources
Sources
KAFKA
API
• There are several sources consumers pulling the info and
pushing it into a Kafka Cluster
• Sources are heterogeneous and their speed is variable.
Sources
Sources
#CassandraSummit 2014 27

28. Data fusion
• We use Storm to process and
normalize the information.
• The system must fire alerts
to the analysts.
• This use case required a Big
Data component capable of
processing the data and
extract its information in real-time.
• Warnings and alerts are time-sensitive in order to deal efficiently with security attacks.
#CassandraSummit 2014 28

29. Batch
•The data are saved in
Cassandra.
•We use Cassandra directly for
the easy queries.
•And we used Spark to extract
the information not accessible
to cassandra directly.
Data process
INTEGRATION INTEGRATION INTEGRATION
#CassandraSummit 2014 29

30. Why did we use C*?
Because we need their features:
• P2P architecture
• Read/write performance
• Fault tolerance
• Easy to deploy
• CQL
#CassandraSummit 2014 30

31. Why did we use C*?
•And we needed data modeler:
•The data in Storm is normalize by source.
• The primary key is the source key (f.e. IP) and a
time stamp to split the cluster key.
• All the data row have view tables with relationship
between entities: IP, DNS, Domain…
IP timestamp Timesplit … Domain … Table name: IP
Primary Key ((IP, timestamp)timesplit)
Domain timestamp timesplit IP1 … IPn Table name: IP_Domain
Primary Key ((Domain, timestamp)timesplit)
#CassandraSummit 2014 31

32. Why did we use C*?
IP main table
IP timestamp Timesplit … Domain … Table name: IP
Primary Key ((IP, timestamp)timesplit)
IP view for domain
Domain timestamp timesplit IP1 … IPn Table name: IP_Domain
Primary Key ((Domain, timestamp)timesplit)
Domain main table
Domain timestamp Timesplit … IP … Table name: domain
Primary Key ((domain, timestamp)timesplit)
IP view for domain
IP timestamp timesplit domain1 … Domainn Table name: Domain_IP
Primary Key ((IP, timestamp)timesplit)
#CassandraSummit 2014 32

33. 33
What have we
learned?

34. Spark + C*
Spark-Cassandra enables the implementation of any use case or
application with any number of users
Applications or dashboards with many users and much data using a predefined set of queries,
perfectly solved with Cassandra, using very few cluster resources.
BI applications or tools with few users (BI analysts or similar) executing open queries, perfectly
solved with Spark over Cassandra using the remaining power of the cluster.
https://github.com/Stratio/stratio-deep
#CassandraSummit 2014 34

35. Spark processing + NoSQL Db
https://github.com/Stratio/stratio-meta
#CassandraSummit 2014 35

36. Cross data
• Unifies batch and stream processing using a common language.
• Ability to access different data store technologies
• Combine results among datastore in a single query.
• Interactive console, a Java/Scala API, a REST API, and a ODBC
interface.
• Easy to use SQL-like language
• Distributed, scalable, fault-tolerance, P2P and extensible.
• Abstracts the users from the underlying data stores technologies:
• Users do not need to be aware of the deployment details to
store and query data
#CassandraSummit 2014 36

37. Pure Spark processing
CROSSDATA SPARK:
Stratio is able to combine, in one
query, stored data with streaming
data entering in the system
Polyglots: Spark integrated with the
main noSQL databases, starting with
Cassandra & Mongo DB.
SPARK FOR ALL :
BATCH, INTERACTIVE AND STREAMING
#CassandraSummit 2014 37

38. System administration
• Admin Application:
• Full deployment on-premise and
cloud
• Platform management and
monitoring
• System Dashboards and
Reporting
• System alerts
• Support (8*5 or 24*7)
#CassandraSummit 2014 38

39. System admin
• The Admin is the easiest way to install, manage and monitoring Stratio’s
services, and the necessary software to run them.System alerts
• Support multiple big data Databases
• Install’s only what you need
• Simple and easy to operate
• Unified management tool
• Install nodes in only 3 steps
• Simplified the scalability process
#CassandraSummit 2014 39

40. Full text search queries
• Telefonica request the ability to perform full text searches
• We have developed an integration between Lucene and
Cassandra
https://github.com/Stratio/stratio-cassandra
• With Crossdata, we simplify:
• The creation syntax
• The query syntax using the match operator
#CassandraSummit 2014 40

41. Full text search queries
C*
node
C*
node
Lucene
index
C*
node
Lucene
index
Lucene
index
C*
node
Lucene
index
C*
node
Lucene
index
SELECT * FROM logs
WHERE description
MATCH ‘*Exception’;
#CassandraSummit 2014 41

42. Upcoming challenges
42

43. Stratio Streaming
•Start using Spark Streaming for
doing some Complex Event
Processing operations.
https://github.com/Stratio/stratio-streaming
#CassandraSummit 2014 43

44. Datavis
• A solution for the creation and design of dashboards,
reports and microsites.
https://github.com/Stratio/stratio-datavis
• Connect different data sources (relational databases,
noSQL, REST services, ElasticSearch, CSV...)
• Datavis allows you to design websites with complete
freedom, where the user only has to concentrate on the
content, not in the underlying technology.
#CassandraSummit 2014 44

45. Stratio Ingestion
https://github.com/Stratio/stratio-ingestion
#CassandraSummit 2014 45

46. Thanks in
advance
#CassandraSummit 2014 46