[DO16] Mesosphere : Microservices meet Fast Data on Azure

SMACK
Microservices meet Fast Data on Azure
@joerg_schad

© 2017 Mesosphere, Inc. All Rights Reserved. 2
Jörg Schad
Distributed Systems Engineer
@joerg_schad

© 2017 Mesosphere, Inc. All Rights Reserved.
Hardware
Operating System
Application
5
Applications

noun | ˈmīkrō/ /ˈsərvəs/ :
an approach to application development in which a
large application is built as a suite of modular services.
Each module supports a specific business goal and uses
a simple, well-defined interface to communicate with
other modules.*
Microservices are designed to be flexible, resilient,
efficient, robust, and individually scalable.
*From whatis.com
OVERVIEW

Operating System Operating System Operating System
ServiceApp ServiceServiceAppApp
7
MICROSERVICES
Polyglot
Single Responsibility
Smaller Teams
Utilization
Machine types/groups
Dependency hell
Machine
Infrastructure
Machine Machine
ServiceService ServiceServiceServiceService

© Gerard Julien/AFP
Run everything in containers!

Container Internals

MapReduce is crunching
Data
Meanwhile...

But then business
demanded
FAST DATA
We need to turn faster!
Today...

Fast Data
Batch Event ProcessingMicro-Batch
Days Hours Minutes Seconds Microseconds
Solves problems using predictive and prescriptive analyticsReports what has happened using descriptive analytics
Predictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product recommendations

The SMACK Stack
EVENTS
Ubiquitous data streams
from connected devices
INGEST
Apache Kafka
STORE
Apache Spark
ANALYZE
Apache Cassandra
ACT
Akka
Ingest millions of events
per second
Distributed & highly scalable
database
Real-time and batch
process data
Visualize data and build
data driven applications
Mesos/ DC/OS
Sensors
Devices
Clients

Datacenter

NAIVE APPROACH
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Industry Average
12-15% utilization
mySQL
microservice
Cassandra
Spark/Hadoop
Kafka

MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTS
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Mesos/ DC/OS
automated schedulers, workload multiplexing onto the
same machines
mySQL
microservice
Cassandra
Spark/Hadoop
Kafka

• A top-level Apache project
• A cluster resource negotiator
• Scalable to 10,000s of nodes
• Fault-tolerant, battle-tested
• An SDK for distributed apps
• Native Docker support
19
Apache Mesos

Use: The primary resource manager and
negotiator
Why Mesos?
● 2-level scheduling
● Fault-tolerant, battle-tested
● Scalable to 10,000+ nodes
● Created by Mesosphere founder @ UC
Berkeley; used in production by 100+ web-
scale companies [1]
[1] http://mesos.apache.org/documentation/latest/powered-by-mesos/
APACHE MESOS

MESOS: FUNDAMENTAL ARCHITECTURE
Mesos
Master
Mesos
Master
Mesos
Master
Mesos AgentMesos Agent Service
Cassandra
Executor
Cassandra
Task
Cassandra
Scheduler
Container
Scheduler
Spark
Scheduler
Spark
Executor
Spark
Task
Mesos AgentMesos Agent Service
Docker
Executor
Docker
Task
Spark
Executor
Spark
Task
Two-level Scheduling
1. Agents advertise resources to Master
2. Master offers resources to Framework
3. Framework rejects / uses resources
4. Agent reports task status to Master

Datacenter Operating System (DC/OS)
Distributed Systems Kernel (Mesos)
DC/OS ENABLES MODERN DISTRIBUTED APPS
Big Data + Analytics EnginesMicroservices (in containers)
Streaming
Batch
Machine Learning
Analytics
Functions &
Logic
Search
Time Series
SQL / NoSQL
Databases
Modern App Components
Any Infrastructure (Physical, Virtual, Cloud)
23

24
THE
BASICS
DC/OS is …
● 100% open source (ASL2.0)
+ A big, diverse community
● An umbrella for ~30 OSS projects
+ Roadmap and designs
+ Docs and tutorials
● Not limited in any way
● Familiar, with more features
+ Networking, Security, CLI, UI, Service
Discovery, Load Balancing, Packages,

SMACK Stack
EVENTS
INGEST
Apache Kafka
STORE
Apache Spark
ANALYZE
Apache Cassandra
ACT
Akka
per second
database
Real-time and batch
process data
DC/OS
Sensors
Devices
Clients

Apache Kafka
ØMQ, RabbitMQ, Disque (Redis-
based), etc.
fluentd, Logstash, Flume
Akka streams
cloud-only: AWS SQS, Google Cloud
Pub/Sub
see also queues.io
MESSAGE QUEUES

High-throughput, distributed,
persistent publish-subscribe
messaging system
Originates from LinkedIn
Typically used as buffer/de-
coupling layer in online
stream processing
APACHE KAFKA

At most once—Messages may be lost
but are never redelivered.
At least once—Messages are never lost
but may be redelivered.
Exactly once—this is what people
actually want, each message is
delivered once and only once.
DELIVERY GUARANTEES
Murphy’s Law of Distributed
Systems:
Anything that can
go wrong, will go
wrong … partially!

SMACK Stack
EVENTS
INGEST
Apache Kafka
STORE
Apache Spark
ANALYZE
Apache Cassandra
ACT
Akka
per second
database
Real-time and batch
process data
DC/OS
Sensors
Devices
Clients

Apache Storm
Apache Spark
Apache Samza
Apache Flink
Apache Apex
Concord
cloud-only: AWS Kinesis,
Google Cloud Dataflow
STREAM PROCESSING

APACHE SPARK

Typical Use: distributed, large-scale data
processing; micro-batching
Why Spark Streaming?
● Micro-batching creates very low latency,
which can be faster
● Well defined role means it fits in well
with other pieces of the pipeline
APACHE SPARK (STREAMING)

Micro-Batching
EXECUTION MODEL
Native Streaming

EXAMPLE:
REAL-TIME
TRACKING

GEO-ENABLED IoT

DATA FLOW

DEMO

SMACK stack
EVENTS
INGEST
Apache Kafka
STORE
Apache Spark
ANALYZE
Apache Cassandra
ACT
Akka
per second
database
Real-time and batch
process data
DC/OS
Sensors
Devices
Clients

Keep it running!

SERVICE OPERATIONS
● Configuration Updates (ex: Scaling, re-configuration)
● Binary Upgrades
● Cluster Maintenance (ex: Backup, Restore, Restart)
● Monitor progress of operations
● Debug any runtime blockages

[DO16] Mesosphere : Microservices meet Fast Data on Azure

More Related Content

What's hot

Similar to [DO16] Mesosphere : Microservices meet Fast Data on Azure

More from de:code 2017

Recently uploaded

[DO16] Mesosphere : Microservices meet Fast Data on Azure