Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
SMACK
Microservices meet Fast Data on Azure
@joerg_schad
© 2017 Mesosphere, Inc. All Rights Reserved. 2
Jörg Schad
Distributed Systems Engineer
@joerg_schad
Video
© 2017 Mesosphere, Inc. All Rights Reserved.
Hardware
Operating System
Application
5
Applications
© 2017 Mesosphere, Inc. All Rights Reserved.
noun | ˈmīkrō/ /ˈsərvəs/ :
an approach to application development in which a
...
© 2017 Mesosphere, Inc. All Rights Reserved.
Operating System Operating System Operating System
ServiceApp ServiceServiceA...
© Gerard Julien/AFP
Run everything in containers!
© 2017 Mesosphere, Inc. All Rights Reserved.
Container Internals
© 2017 Mesosphere, Inc. All Rights Reserved.
Container Internals
© 2017 Mesosphere, Inc. All Rights Reserved. 11
MapReduce is crunching
Data
Meanwhile...
© 2016 Mesosphere, Inc. All Rights Reserved. 12
But then business
demanded
FAST DATA
We need to turn faster!
Today...
© 2016 Mesosphere, Inc. All Rights Reserved. 13
Fast Data
Batch Event ProcessingMicro-Batch
Days Hours Minutes Seconds Mic...
© 2016 Mesosphere, Inc. All Rights Reserved. 14
The SMACK Stack
EVENTS
Ubiquitous data streams
from connected devices
INGE...
© 2017 Mesosphere, Inc. All Rights Reserved. 15
Datacenter
© 2017 Mesosphere, Inc. All Rights Reserved. 16
NAIVE APPROACH
Typical Datacenter
siloed, over-provisioned servers,
low ut...
© 2017 Mesosphere, Inc. All Rights Reserved. 17
© 2017 Mesosphere, Inc. All Rights Reserved. 18
MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTS
Typical Datacenter
sil...
© 2017 Mesosphere, Inc. All Rights Reserved.
• A top-level Apache project
• A cluster resource negotiator
• Scalable to 10...
© 2016 Mesosphere, Inc. All Rights Reserved. 20
Use: The primary resource manager and
negotiator
Why Mesos?
● 2-level sche...
© 2016 Mesosphere, Inc. All Rights Reserved. 21
MESOS: FUNDAMENTAL ARCHITECTURE
Mesos
Master
Mesos
Master
Mesos
Master
Mes...
© 2017 Mesosphere, Inc. All Rights Reserved. 22
Datacenter Operating System (DC/OS)
Distributed Systems Kernel (Mesos)
DC/OS ENABLES MODERN DISTRIBUTED APPS
Big Data + An...
24
THE
BASICS
DC/OS is …
● 100% open source (ASL2.0)
+ A big, diverse community
● An umbrella for ~30 OSS projects
+ Roadm...
SMACK Stack
© 2016 Mesosphere, Inc. All Rights Reserved. 26
SMACK Stack
EVENTS
Ubiquitous data streams
from connected devices
INGEST
A...
© 2016 Mesosphere, Inc. All Rights Reserved. 27
Apache Kafka
ØMQ, RabbitMQ, Disque (Redis-
based), etc.
fluentd, Logstash,...
© 2016 Mesosphere, Inc. All Rights Reserved. 28
High-throughput, distributed,
persistent publish-subscribe
messaging syste...
© 2016 Mesosphere, Inc. All Rights Reserved. 29
At most once—Messages may be lost
but are never redelivered.
At least once...
© 2016 Mesosphere, Inc. All Rights Reserved. 30
SMACK Stack
EVENTS
Ubiquitous data streams
from connected devices
INGEST
A...
© 2016 Mesosphere, Inc. All Rights Reserved. 31
Apache Storm
Apache Spark
Apache Samza
Apache Flink
Apache Apex
Concord
cl...
© 2016 Mesosphere, Inc. All Rights Reserved. 32
APACHE SPARK
© 2016 Mesosphere, Inc. All Rights Reserved. 33
Typical Use: distributed, large-scale data
processing; micro-batching
Why ...
© 2016 Mesosphere, Inc. All Rights Reserved. 34
Micro-Batching
EXECUTION MODEL
Native Streaming
© 2016 Mesosphere, Inc. All Rights Reserved. 35
EXAMPLE:
REAL-TIME
TRACKING
© 2016 Mesosphere, Inc. All Rights Reserved. 36
GEO-ENABLED IoT
© 2016 Mesosphere, Inc. All Rights Reserved. 37
DATA FLOW
© 2016 Mesosphere, Inc. All Rights Reserved. 38
DEMO
© 2016 Mesosphere, Inc. All Rights Reserved. 39
SMACK stack
EVENTS
Ubiquitous data streams
from connected devices
INGEST
A...
© 2017 Mesosphere, Inc. All Rights Reserved. 40
Keep it running!
© 2016 Mesosphere, Inc. All Rights Reserved. 41
SERVICE OPERATIONS
● Configuration Updates (ex: Scaling, re-configuration)...
© 2017 Microsoft Corporation. All rights reserved.
本情報の内容(添付文書、リンク先などを含む)は、作成日時点でのものであり、予告なく変更される場合があります。
@dcos
users@dcos...
[DO16] Mesosphere : Microservices meet Fast Data on Azure
Upcoming SlideShare
Loading in …5
×

of

[DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 1 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 2 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 3 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 4 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 5 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 6 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 7 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 8 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 9 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 10 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 11 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 12 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 13 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 14 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 15 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 16 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 17 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 18 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 19 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 20 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 21 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 22 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 23 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 24 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 25 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 26 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 27 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 28 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 29 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 30 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 31 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 32 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 33 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 34 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 35 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 36 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 37 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 38 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 39 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 40 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 41 [DO16] Mesosphere : Microservices meet Fast Data on Azure  Slide 42
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

[DO16] Mesosphere : Microservices meet Fast Data on Azure

Download to read offline

今日、データ解析に対する需要は高まり、速度も強く求められています。例えば詐称検出やセンサデータへの対応など、多くの場合従来のバッチ処理では処理時間が遅いため要望に応えられません。このようなイベントに対しほぼリアルタイムで処理を行うために、Apache Spark Streaming, Apache Flink, Apache Storm などのストリーム処理を利用する事で大幅な改善ができます。しかし一方でこれらを単体で利用した場合は、十分ではありません。効果的でかつフォールト・トレラントなシステムにするためにはメッセージ・キューやストレージシステムとの連携も必要です。そしてこのような高速なデータ・パイプラインを実現するために SMACK スタックの利用例があります。

● Spark (Streaming) - the stream processing system
● Mesos - the cluster orchestrator
● Akka - the system for providing custom actors for reacting upon the analyses
● Cassandra - storage system
● Kafka - message queue

本セッションでは、高速なデータ・パイプラインの構築のためのベストプラクティスについてご紹介し、DC/OS 上でデモを交えながらこれらをご紹介します。

Jorg Schad
Mesosphere, Inc.
Software Engineer

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

[DO16] Mesosphere : Microservices meet Fast Data on Azure

  1. 1. SMACK Microservices meet Fast Data on Azure @joerg_schad
  2. 2. © 2017 Mesosphere, Inc. All Rights Reserved. 2 Jörg Schad Distributed Systems Engineer @joerg_schad
  3. 3. Video
  4. 4. © 2017 Mesosphere, Inc. All Rights Reserved. Hardware Operating System Application 5 Applications
  5. 5. © 2017 Mesosphere, Inc. All Rights Reserved. noun | ˈmīkrō/ /ˈsərvəs/ : an approach to application development in which a large application is built as a suite of modular services. Each module supports a specific business goal and uses a simple, well-defined interface to communicate with other modules.* Microservices are designed to be flexible, resilient, efficient, robust, and individually scalable. *From whatis.com OVERVIEW
  6. 6. © 2017 Mesosphere, Inc. All Rights Reserved. Operating System Operating System Operating System ServiceApp ServiceServiceAppApp 7 MICROSERVICES Polyglot Single Responsibility Smaller Teams Utilization Machine types/groups Dependency hell Machine Infrastructure Machine Machine ServiceService ServiceServiceServiceService
  7. 7. © Gerard Julien/AFP Run everything in containers!
  8. 8. © 2017 Mesosphere, Inc. All Rights Reserved. Container Internals
  9. 9. © 2017 Mesosphere, Inc. All Rights Reserved. Container Internals
  10. 10. © 2017 Mesosphere, Inc. All Rights Reserved. 11 MapReduce is crunching Data Meanwhile...
  11. 11. © 2016 Mesosphere, Inc. All Rights Reserved. 12 But then business demanded FAST DATA We need to turn faster! Today...
  12. 12. © 2016 Mesosphere, Inc. All Rights Reserved. 13 Fast Data Batch Event ProcessingMicro-Batch Days Hours Minutes Seconds Microseconds Solves problems using predictive and prescriptive analyticsReports what has happened using descriptive analytics Predictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product recommendations
  13. 13. © 2016 Mesosphere, Inc. All Rights Reserved. 14 The SMACK Stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Spark ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications Mesos/ DC/OS Sensors Devices Clients
  14. 14. © 2017 Mesosphere, Inc. All Rights Reserved. 15 Datacenter
  15. 15. © 2017 Mesosphere, Inc. All Rights Reserved. 16 NAIVE APPROACH Typical Datacenter siloed, over-provisioned servers, low utilization Industry Average 12-15% utilization mySQL microservice Cassandra Spark/Hadoop Kafka
  16. 16. © 2017 Mesosphere, Inc. All Rights Reserved. 17
  17. 17. © 2017 Mesosphere, Inc. All Rights Reserved. 18 MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTS Typical Datacenter siloed, over-provisioned servers, low utilization Mesos/ DC/OS automated schedulers, workload multiplexing onto the same machines mySQL microservice Cassandra Spark/Hadoop Kafka
  18. 18. © 2017 Mesosphere, Inc. All Rights Reserved. • A top-level Apache project • A cluster resource negotiator • Scalable to 10,000s of nodes • Fault-tolerant, battle-tested • An SDK for distributed apps • Native Docker support 19 Apache Mesos
  19. 19. © 2016 Mesosphere, Inc. All Rights Reserved. 20 Use: The primary resource manager and negotiator Why Mesos? ● 2-level scheduling ● Fault-tolerant, battle-tested ● Scalable to 10,000+ nodes ● Created by Mesosphere founder @ UC Berkeley; used in production by 100+ web- scale companies [1] [1] http://mesos.apache.org/documentation/latest/powered-by-mesos/ APACHE MESOS
  20. 20. © 2016 Mesosphere, Inc. All Rights Reserved. 21 MESOS: FUNDAMENTAL ARCHITECTURE Mesos Master Mesos Master Mesos Master Mesos AgentMesos Agent Service Cassandra Executor Cassandra Task Cassandra Scheduler Container Scheduler Spark Scheduler Spark Executor Spark Task Mesos AgentMesos Agent Service Docker Executor Docker Task Spark Executor Spark Task Two-level Scheduling 1. Agents advertise resources to Master 2. Master offers resources to Framework 3. Framework rejects / uses resources 4. Agent reports task status to Master
  21. 21. © 2017 Mesosphere, Inc. All Rights Reserved. 22
  22. 22. Datacenter Operating System (DC/OS) Distributed Systems Kernel (Mesos) DC/OS ENABLES MODERN DISTRIBUTED APPS Big Data + Analytics EnginesMicroservices (in containers) Streaming Batch Machine Learning Analytics Functions & Logic Search Time Series SQL / NoSQL Databases Modern App Components Any Infrastructure (Physical, Virtual, Cloud) 23
  23. 23. 24 THE BASICS DC/OS is … ● 100% open source (ASL2.0) + A big, diverse community ● An umbrella for ~30 OSS projects + Roadmap and designs + Docs and tutorials ● Not limited in any way ● Familiar, with more features + Networking, Security, CLI, UI, Service Discovery, Load Balancing, Packages,
  24. 24. SMACK Stack
  25. 25. © 2016 Mesosphere, Inc. All Rights Reserved. 26 SMACK Stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Spark ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications DC/OS Sensors Devices Clients
  26. 26. © 2016 Mesosphere, Inc. All Rights Reserved. 27 Apache Kafka ØMQ, RabbitMQ, Disque (Redis- based), etc. fluentd, Logstash, Flume Akka streams cloud-only: AWS SQS, Google Cloud Pub/Sub see also queues.io MESSAGE QUEUES
  27. 27. © 2016 Mesosphere, Inc. All Rights Reserved. 28 High-throughput, distributed, persistent publish-subscribe messaging system Originates from LinkedIn Typically used as buffer/de- coupling layer in online stream processing APACHE KAFKA
  28. 28. © 2016 Mesosphere, Inc. All Rights Reserved. 29 At most once—Messages may be lost but are never redelivered. At least once—Messages are never lost but may be redelivered. Exactly once—this is what people actually want, each message is delivered once and only once. DELIVERY GUARANTEES Murphy’s Law of Distributed Systems: Anything that can go wrong, will go wrong … partially!
  29. 29. © 2016 Mesosphere, Inc. All Rights Reserved. 30 SMACK Stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Spark ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications DC/OS Sensors Devices Clients
  30. 30. © 2016 Mesosphere, Inc. All Rights Reserved. 31 Apache Storm Apache Spark Apache Samza Apache Flink Apache Apex Concord cloud-only: AWS Kinesis, Google Cloud Dataflow STREAM PROCESSING
  31. 31. © 2016 Mesosphere, Inc. All Rights Reserved. 32 APACHE SPARK
  32. 32. © 2016 Mesosphere, Inc. All Rights Reserved. 33 Typical Use: distributed, large-scale data processing; micro-batching Why Spark Streaming? ● Micro-batching creates very low latency, which can be faster ● Well defined role means it fits in well with other pieces of the pipeline APACHE SPARK (STREAMING)
  33. 33. © 2016 Mesosphere, Inc. All Rights Reserved. 34 Micro-Batching EXECUTION MODEL Native Streaming
  34. 34. © 2016 Mesosphere, Inc. All Rights Reserved. 35 EXAMPLE: REAL-TIME TRACKING
  35. 35. © 2016 Mesosphere, Inc. All Rights Reserved. 36 GEO-ENABLED IoT
  36. 36. © 2016 Mesosphere, Inc. All Rights Reserved. 37 DATA FLOW
  37. 37. © 2016 Mesosphere, Inc. All Rights Reserved. 38 DEMO
  38. 38. © 2016 Mesosphere, Inc. All Rights Reserved. 39 SMACK stack EVENTS Ubiquitous data streams from connected devices INGEST Apache Kafka STORE Apache Spark ANALYZE Apache Cassandra ACT Akka Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications DC/OS Sensors Devices Clients
  39. 39. © 2017 Mesosphere, Inc. All Rights Reserved. 40 Keep it running!
  40. 40. © 2016 Mesosphere, Inc. All Rights Reserved. 41 SERVICE OPERATIONS ● Configuration Updates (ex: Scaling, re-configuration) ● Binary Upgrades ● Cluster Maintenance (ex: Backup, Restore, Restart) ● Monitor progress of operations ● Debug any runtime blockages
  41. 41. © 2017 Microsoft Corporation. All rights reserved. 本情報の内容(添付文書、リンク先などを含む)は、作成日時点でのものであり、予告なく変更される場合があります。 @dcos users@dcos.io /groups/8295652 /dcos /dcos/examples /dcos/demos chat.dcos.io @joerg_schad

今日、データ解析に対する需要は高まり、速度も強く求められています。例えば詐称検出やセンサデータへの対応など、多くの場合従来のバッチ処理では処理時間が遅いため要望に応えられません。このようなイベントに対しほぼリアルタイムで処理を行うために、Apache Spark Streaming, Apache Flink, Apache Storm などのストリーム処理を利用する事で大幅な改善ができます。しかし一方でこれらを単体で利用した場合は、十分ではありません。効果的でかつフォールト・トレラントなシステムにするためにはメッセージ・キューやストレージシステムとの連携も必要です。そしてこのような高速なデータ・パイプラインを実現するために SMACK スタックの利用例があります。 ● Spark (Streaming) - the stream processing system ● Mesos - the cluster orchestrator ● Akka - the system for providing custom actors for reacting upon the analyses ● Cassandra - storage system ● Kafka - message queue 本セッションでは、高速なデータ・パイプラインの構築のためのベストプラクティスについてご紹介し、DC/OS 上でデモを交えながらこれらをご紹介します。 Jorg Schad Mesosphere, Inc. Software Engineer

Views

Total views

339

On Slideshare

0

From embeds

0

Number of embeds

1

Actions

Downloads

27

Shares

0

Comments

0

Likes

0

×