Big Data security: Facing the challenge by Carlos Gómez at Big Data Spain 2017

Big Data Security
Facing the challenge

Experience the presentation
xlic.es/v/E98605

© Stratio 2017. Confidential, All Rights Reserved. 3
About me
• Father of a 5 year old child
• Technical leader in Architecture and Security team at Stratio
• Sailing skipper

© Stratio 2017. Confidential, All Rights Reserved.
In your opinion, how difficult is it to manage security in your
projects?
4
● Very difficult
● Difficult
● Easy
● Very Easy
● What is security?

DATA
GOVERNANCE
LOGS
CENTRALIZATION
PROJECTS FOR EVER ONGOING IN BIG COMPANIES
In a monolithic application centric it with data silos these
initiatives never get accomplished
HUNDRED OF MILLIONS OF EUROS SPENT DURING THE YEARS IN GLOBAL IT CROSS INITIATIVES
SAS
CRM
Earnix
(Pricing)
Towers Watson
ERP
Data Warehouse
Lab H0
(Plataforma Big Data
compartida por el grupo)
WebFocus
Oracle
Mainframe
MONITORING
SECURITYDATA SECURITY AUDIT

PROJECTS FOR EVER ONGOING IN BIG COMPANIES
DATA
GOVERNANCE
LOGS
CENTRALIZATION
MONITORING
DATA SECURITY AUDIT
1
2 3
4
5

GALGO CHASING ELECTRONIC RABBIT…
COMPANIES ALWAYS TRY TO GET THE RABBIT
In an application centric company with data silos you never will be able to
achieve successfully those projects
DATA
GOVERNANCE
LOGS
CENTRALIZATION MONITORING SECURITY
DATA
SECURITY AUDIT

STRUCTURAL INITIATIVES ARE SOLVED COMPLETELY WITH DATA CENTRIC
DaaS (data as a service)
Data
Data Intelligence
DATA
GOVERNANCE
LOGS
CENTRALIZATION
MONITORING
SECURITYDATA SECURITY AUDIT
Functionalities Implemented in the product

RABBIT IN A JAIL
MINIMUM EFFORT AND
COST TO GET THE RABBIT

SECURITY IN A DATA CENTRIC
Protect the data
• Perimeter security to access the cluster.
• Support identity management and authentication to prove
that a user/service is who claims to be.
• In a multi-data store platform ACLs should be centralized
to simplified the correct authorization to different data
stores.
• Audit events must be centralized to control misuse of the
cluster in real time.
• Data integrity and confidentiality in network
communications to protect data on the fly.
Protect the service
• Perimeter security to access the cluster.
• Support identity management and authentication to prove
that a user/service is who claims to be.
• A user/service should be authorized so more resources than
expected are not used.
• A user/service should not interfere with other
users/services when it is not needed.
• To control the use of resources, it should be audited.

In order to guide the security priorities in the product roadmap, we are focused on helping to comply with LOPD within the platform.
Every release of the Stratio platform, the security status is notified through:
● Results of the OWASP tests for the main components of the platform.
● Results of additional general purpose security tests defined to assure the quality expected.
● Security Risk Report that includes the known issues found.
● When Critical and High issues are found:
○ We explain how can be mitigated.
○ We plan to solve them during the next release.
19
SECURITY OVERVIEW

PERIMETER SECURITY: NETWORKING
Public
Network
Private network
Private Agents
Admin network
Admin Router
Master Nodes
Admin network
Admin Router
Public Agents
• The default network configuration allows a zone-based network
security design:
Public.
Admin.
Private.
• Using Mesos roles to identify nodes ensures that only tasks
specifically configured with this role will be executed outside
the Private zone.
• Using Marathon labels, endpoints can be registered dynamically:
Admin Router for the Admin zone.
Marathon LB for the Public zone.

The solution is integrated with LDAP and Kerberos owned by the
company where Stratio DCS is installed.
21
AUTHENTICATION, AUTHORIZATION AND AUDIT
• Authentication:
Web: OAuth2.
Services & Data Stores: Kerberos or TLS-Mutual.
• Authorization:
OAuth2
goSec Management: API Rest and website used to
manage roles, profiles and ACLs. Also it shows users,
groups and audit data.
• Audit:
authentication and authorization events are
structured and stored in a data bus (Kafka) to be
computed and collected.

Plugins are lightweight programs running within
processes of each cluster component.
They are responsible for:
• Authorization (using goSec ACLs).
• Audit of every request sent to the component.
Currently plugins have been developed for:
• Crossdata
• Sparta
• Zookeeper
• HDFS
22
AUTHENTICATION, AUTHORIZATION AND AUDIT
• Kafka
• Elasticsearch

• It is a good practice to manage secretes by key management system
instead of store them locally.
• For this purpose Stratio DCS uses HashiCorp Vault
23
KEY MANAGEMENT SYSTEM

• Can applications obtain authentication tokens in a secure way?
• Where applications save vault’s tokens?
• How are tokens protected?
• How will I know if someone steal tokens?
24
KEY MANAGEMENT SYSTEM the secret of secrets
Mesos
First secret
management
Application
MarathonAdmin

25
Mesos
First secret
management
Application
Marathon
one time secret
Run Application
Env: one time secretAdmin

26
Mesos
First secret
management
Application
Marathon
one time secret
login
Run Application
Env: one time secret
token < - > ACL
Admin

• Can applications obtain tokens in a secure way?
• How are tokens guarded?
27
Mesos
First secret
management
Application
Marathon
one time secret
Run Application

28
Mesos
First secret
management
Application
Marathon
one time secret
login
Run Application

29
Mesos
First secret
management
Application
Marathon
one time secret
login
Run Application

30
Mesos
First secret
management
Application
Marathon
one time secret
login
Run Application
Env: one time secret
Logs Alert
Admin

• Spark jobs need access to multiple data stores so that Spark
needs to support the security of Stratio DCS.
• Spark 2.x compilation has been modified by Stratio in order
to:
Access secrets that are stored in the KMS.
Allow access to Kerberized HDFS.
Allow access to PostgreSQL with TLS authentication.
Allow access to Elasticsearch TLS authentication.
Allow access to Kafka with TLS authentication.
31
DATA PROCESSING ENGINE: SPARK

PROTECT THE DATA
Admin
Perimeter security
Authentication, Authorization, Audit
Ciphered communications
- use case -

PROTECT THE DATA
Admin
Perimeter security
- use case -

• Stratio DCS cluster resources (memory, disk, cpus and port ranges) are managed by Mesos.
• Mesos, Marathon and Metronome security can be activated post-installation in order to limit the use of the available resources for each
framework.
• Once it is activated, admins will be able to:
Reserve resources for a Mesos role.
Grant permissions for each user/framework to do actions such as register frameworks, run tasks, reserve resources, create volumes, etc.
• Grant a minimum set of resources to a specific mesos role
38
MULTI-TENANCY CAPABILITIES: RESOURCES ISOLATION
Mesos Cluster
MASTER
Marathon
AGENT 1
role=slave_public
AGENT 2
role=*
AGENT 3
role=postgresql
AGENT 5
role=*
AGENT 4
role=*

MULTI-TENANCY CAPABILITIES: NETWORKS ISOLATION
• What about network isolation into containerized world?
• For this purpose Stratio DCS uses Project Calico

• Virtual networks topologies can be created dynamically.
• Virtual networks topologies can be managed by network policies.
• Virtual networks can manage all Mesos supported containerized technologies.
• Virtual networks barely impacts big data performance.
• Frameworks/apps are authorized into a network.
• Frameworks/apps can be isolated into a virtual network.
• Frameworks/apps IP addresses and ports are managed by instance.
40
MULTI-TENANCY CAPABILITIES: NETWORKS ISOLATION

Network Isolation Virtual Networks

PROTECT THE SERVICE
Admin
Framework authentication
Check resources for the role
Authorization to launch tasks
Authorization to use the network
Audit (logs and Mesos API)
- use case -

PROTECT THE SERVICE
Admin
- use case -
At least 1 core, 1GB to framework 1

PROTECT THE SERVICE
Admin
- use case -
net_2: Deny from framework 1

PROTECT THE SERVICE
User
2. Launches FRAMEWORK 1
Admin
User
- use case -

PROTECT THE SERVICE
User
Admin
User
- use case -

MULTI-DATA CENTER
49
- a use case -

Big Data security: Facing the challenge by Carlos Gómez at Big Data Spain 2017

Big Data security: Facing the challenge by Carlos Gómez at Big Data Spain 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Big Data security: Facing the challenge by Carlos Gómez at Big Data Spain 2017

Similar to Big Data security: Facing the challenge by Carlos Gómez at Big Data Spain 2017 (20)

More from Big Data Spain

More from Big Data Spain (20)

Recently uploaded

Recently uploaded (20)

Big Data security: Facing the challenge by Carlos Gómez at Big Data Spain 2017