SlideShare a Scribd company logo
1 of 36
Download to read offline
© 2016 MapR Technologies 1© 2016 MapR Technologies 1MapR Confidential © 2016 MapR Technologies
Architecting a hybrid cloud application using a
global publish-subscribe streaming message
system
Mathieu Dumoulin (MapR Technologies)
Strata Singapore 2016
© 2016 MapR Technologies 2© 2016 MapR Technologies 2MapR Confidential © 2016 MapR Technologies
Streaming Architecture to Connect Everything
(including Hybrid Cloud)
Mathieu Dumoulin (MapR Technologies)
Strata Singapore 2016
© 2016 MapR Technologies 3© 2016 MapR Technologies 3MapR Confidential
Mathieu Dumoulin, Data Engineer
• Master’s degree in text
classification on Hadoop at Fujitsu
Canada’s Innovation Lab and Laval
University
• In Tokyo, I’ve worked as a Data
Scientist, Search Engineer and
Data Engineer
• Working on streaming, complex
event processing and machine
learning
© 2016 MapR Technologies 4© 2016 MapR Technologies 4MapR Confidential
The new rule for the future is going
to be, “Anything that can be
connected, will be connected.”
Jacob Morgan, Forbes - May 2014
© 2016 MapR Technologies 5© 2016 MapR Technologies 5MapR Confidential
Talk Summary
• Clouds: private vs. public vs. hybrid
• It’s all about that streaming
– Streaming for IoT
– Publish-subscribe messaging systems (Kafka)
– Stream Processing (Apache Spark Streaming,
Apache Flink)
– Microservices
• Streams-based Architecture in the hybrid cloud
– Design goals
– Examples
• Recap, Q&A
© 2016 MapR Technologies 6© 2016 MapR Technologies 6MapR Confidential © 2016 MapR Technologies
Weather today for IT:
© 2016 MapR Technologies 7© 2016 MapR Technologies 7MapR Confidential
Public Cloud - Low Upfront Cost and Flexibility
The Good
• Right size instances for
application
• Grow with the business
• “Forever” extensible
• Global in a few clicks
The Bad
• New complexity, no
magic
• Costs can run away
The Ugly
• Local data is far from
processing
• Severe lock-in without
huge in-house expertise
© 2016 MapR Technologies 8© 2016 MapR Technologies 8MapR Confidential
Private Clouds - The Benefits of Ownership
The Bad
• Harder to scale vertically &
horizontally
• Cost of multiple datacenters
The Ugly
• Pay for spike, wasted
resources
• Never right size in a growing
organization
The Good
• Direct access to data
• Security, privacy and legal
compliance
• Hardware certainty
• Low running cost
© 2016 MapR Technologies 9© 2016 MapR Technologies 9MapR Confidential
Private Cloud
- Europe
Private Cloud - Tokyo
Hybrid = Public vs. + Private
Spans at least one public and
one private cloud.
• Test new ideas with low
up-front capital cost
• Cloudbursting
• High Availability and Disaster
Recovery
• Regulatory Requirements
IT infrastructure agility
© 2016 MapR Technologies 10© 2016 MapR Technologies 10MapR Confidential © 2016 MapR Technologies
It’s all about that streaming
© 2016 MapR Technologies 11© 2016 MapR Technologies 11MapR Confidential
Streaming Architecture the Norm for Data Driven
Organizations
“Stream-based computing is becoming the norm for data-driven organizations”
- Friedman & Dunning, Streaming Architecture
• Build flexible systems
– more efficient and easier to build
– Decouples dependencies between data source and processing
• Better model the way business processes take place.
• More value now… and later
– Aggregates data from many sources once
– Serves data to one or many projects immediately
– More efficient and high performance
– Run batch analytics, reprocess data
© 2016 MapR Technologies 12© 2016 MapR Technologies 12MapR Confidential
IoT is a Natural Use Case for Streaming
Connected devices produce data as real-time events that are
modelled naturally as event streams.
Event
Some actions have value only if taken immediately
– Navigation updates from traffic conditions, accident reports, disasters, …
– Slowing down or stopping a factory line in response to quality issues
– Re-routing items mid-way during shipping to increase efficiency
– Continuous engine tuning
© 2016 MapR Technologies 13© 2016 MapR Technologies 13MapR Confidential
IoT is Happening Right Now!
© 2016 MapR Technologies 14© 2016 MapR Technologies 14MapR Confidential
Streams Make the Hybrid Cloud Practical
Streams can serve for inter-cloud
communication in the exact same way they
support any other scenario.
● Abstracts the differences between
on-premise and cloud
● Standardize the expected flow of data
between modules
● Reuse data many times, break down
data silos
© 2016 MapR Technologies 15© 2016 MapR Technologies 15MapR Confidential
What Streaming Requires from a Messaging System
● The producer and consumer are fully independent
● Very high throughput 1,000+/s → 1,000,000+/s
● Persistence
○ Fault-tolerance
○ Data is kept as a replayable sequence
○ Strong ordering of events
● Naming of topics (consumers pick the data they need )
● Geo-distributed replication (for Hybrid Cloud use cases)
It’s very hard to get full isolation of producer and consumers while also keeping
very high speed, but we must have both.
© 2016 MapR Technologies 16© 2016 MapR Technologies 16MapR Confidential
What Streaming Requires from Stream Processing
Frameworks
Desirable features for real-time analytics frameworks:
• Open Source, active development and developer community
• Supports “exactly once” guarantee, stream reprocessing
• How much real-time? Microbatch vs. record-at-a-time
• Performance (latency, throughput)
• Other: Easy to use, compatibility, talent availability
To Know more: https://www.mapr.com/blog/stream-processing-everywhere-what-use
Jim Scott - Stream Processing Everywhere - What to Use? Strata San Jose 2015
Also see Data Artisan’s Blog on Stream Processing Framework Myths
© 2016 MapR Technologies 17© 2016 MapR Technologies 17MapR Confidential
Which Stream Processing Frameworks?
© 2016 MapR Technologies 18© 2016 MapR Technologies 18MapR Confidential
Summing up: Technology to support Streaming
1. Lightweight messaging system
2. Stream Processing Framework
You can get an Introduction to Flink in this Free Book published by O’Reilly
© 2016 MapR Technologies 19© 2016 MapR Technologies 19MapR Confidential
Key Ideas For Effectively Using Streams
Real-time
Analysis
Persist to
Disk
Geo-distributed
Replication
Core part of
Architecture
© 2016 MapR Technologies 20© 2016 MapR Technologies 20MapR Confidential
© 2016 MapR Technologies 21© 2016 MapR Technologies 21MapR Confidential
Streaming Architecture: Ideal Platform for Microservices
Microservices are a modern distributed architecture that realizes the
promises of SOA, Service Oriented Architecture
• Scale up from a test use case to a global deployment
• Decouples components, more modular
• Modern, agile development, testing and deployment
• More robust and responsive
See Krystal Valentine’s “The keys to an event-based microservices application”
presentation, Strata New York 2016
© 2016 MapR Technologies 22© 2016 MapR Technologies 22MapR Confidential
Monolithic to Microservices Architecture
See Fowler’s blog about microservices:
http://www.martinfowler.com/articles/microservices.html
© 2016 MapR Technologies 23© 2016 MapR Technologies 23MapR Confidential
Microservices are Truly Decoupled
© 2016 MapR Technologies 24© 2016 MapR Technologies 24MapR Confidential
When to Use Streaming Architecture
© 2016 MapR Technologies 25© 2016 MapR Technologies 25MapR Confidential © 2016 MapR Technologies
Connect Clouds with Streams:
Streams-based Architecture
© 2016 MapR Technologies 26© 2016 MapR Technologies 26MapR Confidential
Switch from thinking of computer
programs as state-oriented to thinking
of them in terms of flows”
Ted Dunning & Ellen Friedman, Streaming Architecture - O’Reilly - 2016
© 2016 MapR Technologies 27© 2016 MapR Technologies 27MapR Confidential
An End-to-End Streaming Architecture
Japan North
Data Center
Stream
GW
Global Data
Center
Stream
© 2016 MapR Technologies 28© 2016 MapR Technologies 28MapR Confidential
Example Architecture: Log Analysis
© 2016 MapR Technologies 29© 2016 MapR Technologies 29MapR Confidential
Example Architecture: Log Analysis
© 2016 MapR Technologies 30© 2016 MapR Technologies 30MapR Confidential
Example Architecture: The MapR Blueprint
Download the Finserve app from Github!
https://github.com/mapr-demos/finserv-application-blueprint
© 2016 MapR Technologies 31© 2016 MapR Technologies 31MapR Confidential
Conclusion
• The hybrid cloud matters for IT agility
• Use streams for communication between elements
• Streaming-based systems can be arbitrarily complex
– Still fast, responsive, reliable and easier to develop!
• In a streaming architecture world, a converged platform
(built-in streaming, storage and DB) makes a difference.
© 2016 MapR Technologies 32© 2016 MapR Technologies 32MapR Confidential
Suggested Reading And Video Links
Get Ted & Ellen’s book: Read it Online for Free!
New content presented by Ted Dunning:
1. Big Data in the Cloud (blog):
www.mapr.com/big-data-cloud
a. Direct video link:
https://youtu.be/90KrQAb1_Cw
2. Converged Advantages in the Cloud (blog):
www.mapr.com/converged-cloud
a. Direct video link: https://youtu.be/yjfBXNcmAHA
© 2016 MapR Technologies 33© 2016 MapR Technologies 33MapR Confidential
Q & A
@mapr
mdumoulin@mapr.com
@lordxar
Engage with us!
mapr-technologies
© 2016 MapR Technologies 34© 2016 MapR Technologies 34MapR Confidential
Key Ideas for Microservices
• Services are opaque - API only
• They communicate with only a few other services using
lightweight, flexible protocols.
– HTTP+REST - Synchronous (frontend)
– Messaging Systems (Kafka, MapR Streams) - Asynchronous (backend)
• Data formats should be future-proofed
– JSON - Human readable, easy to use, low efficiency
– Binary (Avro, Protobuf, Thrift) - Efficient but (somewhat) harder to use
{RESTful}
© 2016 MapR Technologies 35© 2016 MapR Technologies 35MapR Confidential
Spark Streaming or Flink: Case by Case
Micro-batches. Time-based
window. Latency: seconds
Continuous flow model.
Record-based window. Latency: ms
Both provide exactly once guarantee, high throughput and low
overhead of fault tolerance. Both streaming and batch supported.
© 2016 MapR Technologies 36© 2016 MapR Technologies 36MapR Confidential
The Hybrid Cloud for IoT Infrastructure
• IoT is a new use case - Need to Test
• Built-in need for baseload capacity and bursting data spikes
• Global marketplace requires geographically dispersed
datacenters
• Increasingly strict compliance requirements
• IoT Security issues need to be taken seriously
Why do IoT applications call out for the flexibility of Hybrid Cloud?

More Related Content

What's hot

How to Leverage the Cloud for Business Solutions | Strata Data Conference Lon...
How to Leverage the Cloud for Business Solutions | Strata Data Conference Lon...How to Leverage the Cloud for Business Solutions | Strata Data Conference Lon...
How to Leverage the Cloud for Business Solutions | Strata Data Conference Lon...MapR Technologies
 
Spark & Hadoop at Production at Scale
Spark & Hadoop at Production at ScaleSpark & Hadoop at Production at Scale
Spark & Hadoop at Production at ScaleMapR Technologies
 
MapR Product Update - Spring 2017
MapR Product Update - Spring 2017MapR Product Update - Spring 2017
MapR Product Update - Spring 2017MapR Technologies
 
Advanced Threat Detection on Streaming Data
Advanced Threat Detection on Streaming DataAdvanced Threat Detection on Streaming Data
Advanced Threat Detection on Streaming DataCarol McDonald
 
3 Benefits of Multi-Temperature Data Management for Data Analytics
3 Benefits of Multi-Temperature Data Management for Data Analytics3 Benefits of Multi-Temperature Data Management for Data Analytics
3 Benefits of Multi-Temperature Data Management for Data AnalyticsMapR Technologies
 
Applying Machine Learning to Live Patient Data
Applying Machine Learning to  Live Patient DataApplying Machine Learning to  Live Patient Data
Applying Machine Learning to Live Patient DataCarol McDonald
 
Streaming patterns revolutionary architectures
Streaming patterns revolutionary architectures Streaming patterns revolutionary architectures
Streaming patterns revolutionary architectures Carol McDonald
 
Applying Machine learning to IOT: End to End Distributed Distributed Pipeline...
Applying Machine learning to IOT: End to End Distributed Distributed Pipeline...Applying Machine learning to IOT: End to End Distributed Distributed Pipeline...
Applying Machine learning to IOT: End to End Distributed Distributed Pipeline...Carol McDonald
 
How Big Data is Reducing Costs and Improving Outcomes in Health Care
How Big Data is Reducing Costs and Improving Outcomes in Health CareHow Big Data is Reducing Costs and Improving Outcomes in Health Care
How Big Data is Reducing Costs and Improving Outcomes in Health CareCarol McDonald
 
Cisco & MapR bring 3 Superpowers to SAP HANA Deployments
Cisco & MapR bring 3 Superpowers to SAP HANA DeploymentsCisco & MapR bring 3 Superpowers to SAP HANA Deployments
Cisco & MapR bring 3 Superpowers to SAP HANA DeploymentsMapR Technologies
 
Demystifying AI, Machine Learning and Deep Learning
Demystifying AI, Machine Learning and Deep LearningDemystifying AI, Machine Learning and Deep Learning
Demystifying AI, Machine Learning and Deep LearningCarol McDonald
 
Machine Learning for Chickens, Autonomous Driving and a 3-year-old Who Won’t ...
Machine Learning for Chickens, Autonomous Driving and a 3-year-old Who Won’t ...Machine Learning for Chickens, Autonomous Driving and a 3-year-old Who Won’t ...
Machine Learning for Chickens, Autonomous Driving and a 3-year-old Who Won’t ...MapR Technologies
 
Streaming Patterns Revolutionary Architectures with the Kafka API
Streaming Patterns Revolutionary Architectures with the Kafka APIStreaming Patterns Revolutionary Architectures with the Kafka API
Streaming Patterns Revolutionary Architectures with the Kafka APICarol McDonald
 
Spark and MapR Streams: A Motivating Example
Spark and MapR Streams: A Motivating ExampleSpark and MapR Streams: A Motivating Example
Spark and MapR Streams: A Motivating ExampleIan Downard
 
Tugdual Grall - Real World Use Cases: Hadoop and NoSQL in Production
Tugdual Grall - Real World Use Cases: Hadoop and NoSQL in ProductionTugdual Grall - Real World Use Cases: Hadoop and NoSQL in Production
Tugdual Grall - Real World Use Cases: Hadoop and NoSQL in ProductionCodemotion
 
Predictive Maintenance Using Recurrent Neural Networks
Predictive Maintenance Using Recurrent Neural NetworksPredictive Maintenance Using Recurrent Neural Networks
Predictive Maintenance Using Recurrent Neural NetworksJustin Brandenburg
 
Fast Cars, Big Data How Streaming can help Formula 1
Fast Cars, Big Data How Streaming can help Formula 1Fast Cars, Big Data How Streaming can help Formula 1
Fast Cars, Big Data How Streaming can help Formula 1Carol McDonald
 
Spark and Hadoop at Production Scale-(Anil Gadre, MapR)
Spark and Hadoop at Production Scale-(Anil Gadre, MapR)Spark and Hadoop at Production Scale-(Anil Gadre, MapR)
Spark and Hadoop at Production Scale-(Anil Gadre, MapR)Spark Summit
 

What's hot (20)

How to Leverage the Cloud for Business Solutions | Strata Data Conference Lon...
How to Leverage the Cloud for Business Solutions | Strata Data Conference Lon...How to Leverage the Cloud for Business Solutions | Strata Data Conference Lon...
How to Leverage the Cloud for Business Solutions | Strata Data Conference Lon...
 
Spark & Hadoop at Production at Scale
Spark & Hadoop at Production at ScaleSpark & Hadoop at Production at Scale
Spark & Hadoop at Production at Scale
 
Streaming in the Extreme
Streaming in the ExtremeStreaming in the Extreme
Streaming in the Extreme
 
MapR Product Update - Spring 2017
MapR Product Update - Spring 2017MapR Product Update - Spring 2017
MapR Product Update - Spring 2017
 
Advanced Threat Detection on Streaming Data
Advanced Threat Detection on Streaming DataAdvanced Threat Detection on Streaming Data
Advanced Threat Detection on Streaming Data
 
3 Benefits of Multi-Temperature Data Management for Data Analytics
3 Benefits of Multi-Temperature Data Management for Data Analytics3 Benefits of Multi-Temperature Data Management for Data Analytics
3 Benefits of Multi-Temperature Data Management for Data Analytics
 
Applying Machine Learning to Live Patient Data
Applying Machine Learning to  Live Patient DataApplying Machine Learning to  Live Patient Data
Applying Machine Learning to Live Patient Data
 
Streaming patterns revolutionary architectures
Streaming patterns revolutionary architectures Streaming patterns revolutionary architectures
Streaming patterns revolutionary architectures
 
Applying Machine learning to IOT: End to End Distributed Distributed Pipeline...
Applying Machine learning to IOT: End to End Distributed Distributed Pipeline...Applying Machine learning to IOT: End to End Distributed Distributed Pipeline...
Applying Machine learning to IOT: End to End Distributed Distributed Pipeline...
 
How Big Data is Reducing Costs and Improving Outcomes in Health Care
How Big Data is Reducing Costs and Improving Outcomes in Health CareHow Big Data is Reducing Costs and Improving Outcomes in Health Care
How Big Data is Reducing Costs and Improving Outcomes in Health Care
 
Cisco & MapR bring 3 Superpowers to SAP HANA Deployments
Cisco & MapR bring 3 Superpowers to SAP HANA DeploymentsCisco & MapR bring 3 Superpowers to SAP HANA Deployments
Cisco & MapR bring 3 Superpowers to SAP HANA Deployments
 
MapR & Skytree:
MapR & Skytree: MapR & Skytree:
MapR & Skytree:
 
Demystifying AI, Machine Learning and Deep Learning
Demystifying AI, Machine Learning and Deep LearningDemystifying AI, Machine Learning and Deep Learning
Demystifying AI, Machine Learning and Deep Learning
 
Machine Learning for Chickens, Autonomous Driving and a 3-year-old Who Won’t ...
Machine Learning for Chickens, Autonomous Driving and a 3-year-old Who Won’t ...Machine Learning for Chickens, Autonomous Driving and a 3-year-old Who Won’t ...
Machine Learning for Chickens, Autonomous Driving and a 3-year-old Who Won’t ...
 
Streaming Patterns Revolutionary Architectures with the Kafka API
Streaming Patterns Revolutionary Architectures with the Kafka APIStreaming Patterns Revolutionary Architectures with the Kafka API
Streaming Patterns Revolutionary Architectures with the Kafka API
 
Spark and MapR Streams: A Motivating Example
Spark and MapR Streams: A Motivating ExampleSpark and MapR Streams: A Motivating Example
Spark and MapR Streams: A Motivating Example
 
Tugdual Grall - Real World Use Cases: Hadoop and NoSQL in Production
Tugdual Grall - Real World Use Cases: Hadoop and NoSQL in ProductionTugdual Grall - Real World Use Cases: Hadoop and NoSQL in Production
Tugdual Grall - Real World Use Cases: Hadoop and NoSQL in Production
 
Predictive Maintenance Using Recurrent Neural Networks
Predictive Maintenance Using Recurrent Neural NetworksPredictive Maintenance Using Recurrent Neural Networks
Predictive Maintenance Using Recurrent Neural Networks
 
Fast Cars, Big Data How Streaming can help Formula 1
Fast Cars, Big Data How Streaming can help Formula 1Fast Cars, Big Data How Streaming can help Formula 1
Fast Cars, Big Data How Streaming can help Formula 1
 
Spark and Hadoop at Production Scale-(Anil Gadre, MapR)
Spark and Hadoop at Production Scale-(Anil Gadre, MapR)Spark and Hadoop at Production Scale-(Anil Gadre, MapR)
Spark and Hadoop at Production Scale-(Anil Gadre, MapR)
 

Viewers also liked

Evolving from RDBMS to NoSQL + SQL
Evolving from RDBMS to NoSQL + SQLEvolving from RDBMS to NoSQL + SQL
Evolving from RDBMS to NoSQL + SQLMapR Technologies
 
20151128_SMeNG_態度は変えられるのか
20151128_SMeNG_態度は変えられるのか20151128_SMeNG_態度は変えられるのか
20151128_SMeNG_態度は変えられるのかTakanori Hiroe
 
20150321 医学:医療者教育研究ネットワーク@九州大学
20150321 医学:医療者教育研究ネットワーク@九州大学20150321 医学:医療者教育研究ネットワーク@九州大学
20150321 医学:医療者教育研究ネットワーク@九州大学Takanori Hiroe
 
HBase New Features
HBase New FeaturesHBase New Features
HBase New Featuresrxu
 
Apache Drill で日本語を扱ってみよう + オープンデータ解析
Apache Drill で日本語を扱ってみよう + オープンデータ解析Apache Drill で日本語を扱ってみよう + オープンデータ解析
Apache Drill で日本語を扱ってみよう + オープンデータ解析MapR Technologies Japan
 
MapR アーキテクチャ概要 - MapR CTO Meetup 2013/11/12
MapR アーキテクチャ概要 - MapR CTO Meetup 2013/11/12MapR アーキテクチャ概要 - MapR CTO Meetup 2013/11/12
MapR アーキテクチャ概要 - MapR CTO Meetup 2013/11/12MapR Technologies Japan
 
MapR Streams & MapR コンバージド・データ・プラットフォーム
MapR Streams & MapR コンバージド・データ・プラットフォームMapR Streams & MapR コンバージド・データ・プラットフォーム
MapR Streams & MapR コンバージド・データ・プラットフォームMapR Technologies Japan
 
MapR 5.2: Getting More Value from the MapR Converged Community Edition
MapR 5.2: Getting More Value from the MapR Converged Community EditionMapR 5.2: Getting More Value from the MapR Converged Community Edition
MapR 5.2: Getting More Value from the MapR Converged Community EditionMapR Technologies
 
20170225_Sample size determination
20170225_Sample size determination20170225_Sample size determination
20170225_Sample size determinationTakanori Hiroe
 
MapR Streams and MapR Converged Data Platform
MapR Streams and MapR Converged Data PlatformMapR Streams and MapR Converged Data Platform
MapR Streams and MapR Converged Data PlatformMapR Technologies
 
Apache Drill でたしなむ セルフサービスデータ探索 - 2014/11/06 Cloudera World Tokyo 2014 LTセッション
Apache Drill でたしなむ セルフサービスデータ探索 - 2014/11/06 Cloudera World Tokyo 2014 LTセッションApache Drill でたしなむ セルフサービスデータ探索 - 2014/11/06 Cloudera World Tokyo 2014 LTセッション
Apache Drill でたしなむ セルフサービスデータ探索 - 2014/11/06 Cloudera World Tokyo 2014 LTセッションMapR Technologies Japan
 
Inside MapR's M7
Inside MapR's M7Inside MapR's M7
Inside MapR's M7Ted Dunning
 
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: MapR Overview Present...
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: MapR Overview Present...Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: MapR Overview Present...
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: MapR Overview Present...ervogler
 
ストリーミングアーキテクチャ: State から Flow へ - 2016/02/08 Hadoop / Spark Conference Japan ...
ストリーミングアーキテクチャ: State から Flow へ - 2016/02/08 Hadoop / Spark Conference Japan ...ストリーミングアーキテクチャ: State から Flow へ - 2016/02/08 Hadoop / Spark Conference Japan ...
ストリーミングアーキテクチャ: State から Flow へ - 2016/02/08 Hadoop / Spark Conference Japan ...MapR Technologies Japan
 
Kappa Architecture, IoT of the cars - LibreCon 2016
Kappa Architecture, IoT of the cars - LibreCon 2016Kappa Architecture, IoT of the cars - LibreCon 2016
Kappa Architecture, IoT of the cars - LibreCon 2016LibreCon
 
Docker1.13で変わったことをわからないなりにまとめてみた
Docker1.13で変わったことをわからないなりにまとめてみたDocker1.13で変わったことをわからないなりにまとめてみた
Docker1.13で変わったことをわからないなりにまとめてみたKouta Asai
 
リクルートライフスタイルの考える ストリームデータの活かし方(Hadoop Spark Conference2016)
リクルートライフスタイルの考えるストリームデータの活かし方(Hadoop Spark Conference2016)リクルートライフスタイルの考えるストリームデータの活かし方(Hadoop Spark Conference2016)
リクルートライフスタイルの考える ストリームデータの活かし方(Hadoop Spark Conference2016)Atsushi Kurumada
 

Viewers also liked (20)

Evolving from RDBMS to NoSQL + SQL
Evolving from RDBMS to NoSQL + SQLEvolving from RDBMS to NoSQL + SQL
Evolving from RDBMS to NoSQL + SQL
 
JSME_47th_Nigata
JSME_47th_NigataJSME_47th_Nigata
JSME_47th_Nigata
 
20151128_SMeNG_態度は変えられるのか
20151128_SMeNG_態度は変えられるのか20151128_SMeNG_態度は変えられるのか
20151128_SMeNG_態度は変えられるのか
 
20150321 医学:医療者教育研究ネットワーク@九州大学
20150321 医学:医療者教育研究ネットワーク@九州大学20150321 医学:医療者教育研究ネットワーク@九州大学
20150321 医学:医療者教育研究ネットワーク@九州大学
 
20150827_simplesize
20150827_simplesize20150827_simplesize
20150827_simplesize
 
HBase New Features
HBase New FeaturesHBase New Features
HBase New Features
 
Apache Drill で日本語を扱ってみよう + オープンデータ解析
Apache Drill で日本語を扱ってみよう + オープンデータ解析Apache Drill で日本語を扱ってみよう + オープンデータ解析
Apache Drill で日本語を扱ってみよう + オープンデータ解析
 
MapR アーキテクチャ概要 - MapR CTO Meetup 2013/11/12
MapR アーキテクチャ概要 - MapR CTO Meetup 2013/11/12MapR アーキテクチャ概要 - MapR CTO Meetup 2013/11/12
MapR アーキテクチャ概要 - MapR CTO Meetup 2013/11/12
 
MapR Streams & MapR コンバージド・データ・プラットフォーム
MapR Streams & MapR コンバージド・データ・プラットフォームMapR Streams & MapR コンバージド・データ・プラットフォーム
MapR Streams & MapR コンバージド・データ・プラットフォーム
 
MapR 5.2: Getting More Value from the MapR Converged Community Edition
MapR 5.2: Getting More Value from the MapR Converged Community EditionMapR 5.2: Getting More Value from the MapR Converged Community Edition
MapR 5.2: Getting More Value from the MapR Converged Community Edition
 
20170225_Sample size determination
20170225_Sample size determination20170225_Sample size determination
20170225_Sample size determination
 
Drill超簡単チューニング
Drill超簡単チューニングDrill超簡単チューニング
Drill超簡単チューニング
 
MapR Streams and MapR Converged Data Platform
MapR Streams and MapR Converged Data PlatformMapR Streams and MapR Converged Data Platform
MapR Streams and MapR Converged Data Platform
 
Apache Drill でたしなむ セルフサービスデータ探索 - 2014/11/06 Cloudera World Tokyo 2014 LTセッション
Apache Drill でたしなむ セルフサービスデータ探索 - 2014/11/06 Cloudera World Tokyo 2014 LTセッションApache Drill でたしなむ セルフサービスデータ探索 - 2014/11/06 Cloudera World Tokyo 2014 LTセッション
Apache Drill でたしなむ セルフサービスデータ探索 - 2014/11/06 Cloudera World Tokyo 2014 LTセッション
 
Inside MapR's M7
Inside MapR's M7Inside MapR's M7
Inside MapR's M7
 
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: MapR Overview Present...
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: MapR Overview Present...Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: MapR Overview Present...
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: MapR Overview Present...
 
ストリーミングアーキテクチャ: State から Flow へ - 2016/02/08 Hadoop / Spark Conference Japan ...
ストリーミングアーキテクチャ: State から Flow へ - 2016/02/08 Hadoop / Spark Conference Japan ...ストリーミングアーキテクチャ: State から Flow へ - 2016/02/08 Hadoop / Spark Conference Japan ...
ストリーミングアーキテクチャ: State から Flow へ - 2016/02/08 Hadoop / Spark Conference Japan ...
 
Kappa Architecture, IoT of the cars - LibreCon 2016
Kappa Architecture, IoT of the cars - LibreCon 2016Kappa Architecture, IoT of the cars - LibreCon 2016
Kappa Architecture, IoT of the cars - LibreCon 2016
 
Docker1.13で変わったことをわからないなりにまとめてみた
Docker1.13で変わったことをわからないなりにまとめてみたDocker1.13で変わったことをわからないなりにまとめてみた
Docker1.13で変わったことをわからないなりにまとめてみた
 
リクルートライフスタイルの考える ストリームデータの活かし方(Hadoop Spark Conference2016)
リクルートライフスタイルの考えるストリームデータの活かし方(Hadoop Spark Conference2016)リクルートライフスタイルの考えるストリームデータの活かし方(Hadoop Spark Conference2016)
リクルートライフスタイルの考える ストリームデータの活かし方(Hadoop Spark Conference2016)
 

Similar to Streaming Architecture to Connect Everything (Including Hybrid Cloud) - Strata Singapore 2016

HUG Italy meet-up with Fabian Wilckens, MapR EMEA Solutions Architect
HUG Italy meet-up with Fabian Wilckens, MapR EMEA Solutions ArchitectHUG Italy meet-up with Fabian Wilckens, MapR EMEA Solutions Architect
HUG Italy meet-up with Fabian Wilckens, MapR EMEA Solutions ArchitectSpagoWorld
 
Where is Data Going? - RMDC Keynote
Where is Data Going? - RMDC KeynoteWhere is Data Going? - RMDC Keynote
Where is Data Going? - RMDC KeynoteTed Dunning
 
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...Mathieu Dumoulin
 
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...DataWorks Summit/Hadoop Summit
 
Map r seattle streams meetup oct 2016
Map r seattle streams meetup   oct 2016Map r seattle streams meetup   oct 2016
Map r seattle streams meetup oct 2016Nitin Kumar
 
Big Data LDN 2017: How to leverage the cloud for Business Solutions
Big Data LDN 2017: How to leverage the cloud for Business SolutionsBig Data LDN 2017: How to leverage the cloud for Business Solutions
Big Data LDN 2017: How to leverage the cloud for Business SolutionsMatt Stubbs
 
MapR and Cisco Make IT Better
MapR and Cisco Make IT BetterMapR and Cisco Make IT Better
MapR and Cisco Make IT BetterMapR Technologies
 
Rise of the Hybrid Cloud
Rise of the Hybrid CloudRise of the Hybrid Cloud
Rise of the Hybrid CloudIQBG, Inc.
 
Real World Use Cases: Hadoop and NoSQL in Production
Real World Use Cases: Hadoop and NoSQL in ProductionReal World Use Cases: Hadoop and NoSQL in Production
Real World Use Cases: Hadoop and NoSQL in ProductionCodemotion
 
Container and Kubernetes without limits
Container and Kubernetes without limitsContainer and Kubernetes without limits
Container and Kubernetes without limitsAntje Barth
 
Anomaly Detection in Telecom with Spark - Tugdual Grall - Codemotion Amsterda...
Anomaly Detection in Telecom with Spark - Tugdual Grall - Codemotion Amsterda...Anomaly Detection in Telecom with Spark - Tugdual Grall - Codemotion Amsterda...
Anomaly Detection in Telecom with Spark - Tugdual Grall - Codemotion Amsterda...Codemotion
 
Postgres Vision 2018: The Pragmatic Cloud
Postgres Vision 2018:  The Pragmatic CloudPostgres Vision 2018:  The Pragmatic Cloud
Postgres Vision 2018: The Pragmatic CloudEDB
 
How Spark is Enabling the New Wave of Converged Cloud Applications
How Spark is Enabling the New Wave of Converged Cloud Applications How Spark is Enabling the New Wave of Converged Cloud Applications
How Spark is Enabling the New Wave of Converged Cloud Applications MapR Technologies
 
DataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven OrganizationsDataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven OrganizationsEllen Friedman
 
Real-time Hadoop: The Ideal Messaging System for Hadoop
Real-time Hadoop: The Ideal Messaging System for Hadoop Real-time Hadoop: The Ideal Messaging System for Hadoop
Real-time Hadoop: The Ideal Messaging System for Hadoop DataWorks Summit/Hadoop Summit
 
Distributed Deep Learning on Spark
Distributed Deep Learning on SparkDistributed Deep Learning on Spark
Distributed Deep Learning on SparkMathieu Dumoulin
 
Geo-Distributed Big Data and Analytics
Geo-Distributed Big Data and AnalyticsGeo-Distributed Big Data and Analytics
Geo-Distributed Big Data and AnalyticsMapR Technologies
 
Choosing a Cloud Provider: Public-Private-Hybrid
Choosing a Cloud Provider: Public-Private-HybridChoosing a Cloud Provider: Public-Private-Hybrid
Choosing a Cloud Provider: Public-Private-HybridRobert Starmer
 
Machine Learning Success: The Key to Easier Model Management
Machine Learning Success: The Key to Easier Model ManagementMachine Learning Success: The Key to Easier Model Management
Machine Learning Success: The Key to Easier Model ManagementMapR Technologies
 
State of the Cloud and Data Centers 2014
State of the Cloud and Data Centers 2014State of the Cloud and Data Centers 2014
State of the Cloud and Data Centers 2014Digital Realty
 

Similar to Streaming Architecture to Connect Everything (Including Hybrid Cloud) - Strata Singapore 2016 (20)

HUG Italy meet-up with Fabian Wilckens, MapR EMEA Solutions Architect
HUG Italy meet-up with Fabian Wilckens, MapR EMEA Solutions ArchitectHUG Italy meet-up with Fabian Wilckens, MapR EMEA Solutions Architect
HUG Italy meet-up with Fabian Wilckens, MapR EMEA Solutions Architect
 
Where is Data Going? - RMDC Keynote
Where is Data Going? - RMDC KeynoteWhere is Data Going? - RMDC Keynote
Where is Data Going? - RMDC Keynote
 
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
 
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
 
Map r seattle streams meetup oct 2016
Map r seattle streams meetup   oct 2016Map r seattle streams meetup   oct 2016
Map r seattle streams meetup oct 2016
 
Big Data LDN 2017: How to leverage the cloud for Business Solutions
Big Data LDN 2017: How to leverage the cloud for Business SolutionsBig Data LDN 2017: How to leverage the cloud for Business Solutions
Big Data LDN 2017: How to leverage the cloud for Business Solutions
 
MapR and Cisco Make IT Better
MapR and Cisco Make IT BetterMapR and Cisco Make IT Better
MapR and Cisco Make IT Better
 
Rise of the Hybrid Cloud
Rise of the Hybrid CloudRise of the Hybrid Cloud
Rise of the Hybrid Cloud
 
Real World Use Cases: Hadoop and NoSQL in Production
Real World Use Cases: Hadoop and NoSQL in ProductionReal World Use Cases: Hadoop and NoSQL in Production
Real World Use Cases: Hadoop and NoSQL in Production
 
Container and Kubernetes without limits
Container and Kubernetes without limitsContainer and Kubernetes without limits
Container and Kubernetes without limits
 
Anomaly Detection in Telecom with Spark - Tugdual Grall - Codemotion Amsterda...
Anomaly Detection in Telecom with Spark - Tugdual Grall - Codemotion Amsterda...Anomaly Detection in Telecom with Spark - Tugdual Grall - Codemotion Amsterda...
Anomaly Detection in Telecom with Spark - Tugdual Grall - Codemotion Amsterda...
 
Postgres Vision 2018: The Pragmatic Cloud
Postgres Vision 2018:  The Pragmatic CloudPostgres Vision 2018:  The Pragmatic Cloud
Postgres Vision 2018: The Pragmatic Cloud
 
How Spark is Enabling the New Wave of Converged Cloud Applications
How Spark is Enabling the New Wave of Converged Cloud Applications How Spark is Enabling the New Wave of Converged Cloud Applications
How Spark is Enabling the New Wave of Converged Cloud Applications
 
DataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven OrganizationsDataOps: An Agile Method for Data-Driven Organizations
DataOps: An Agile Method for Data-Driven Organizations
 
Real-time Hadoop: The Ideal Messaging System for Hadoop
Real-time Hadoop: The Ideal Messaging System for Hadoop Real-time Hadoop: The Ideal Messaging System for Hadoop
Real-time Hadoop: The Ideal Messaging System for Hadoop
 
Distributed Deep Learning on Spark
Distributed Deep Learning on SparkDistributed Deep Learning on Spark
Distributed Deep Learning on Spark
 
Geo-Distributed Big Data and Analytics
Geo-Distributed Big Data and AnalyticsGeo-Distributed Big Data and Analytics
Geo-Distributed Big Data and Analytics
 
Choosing a Cloud Provider: Public-Private-Hybrid
Choosing a Cloud Provider: Public-Private-HybridChoosing a Cloud Provider: Public-Private-Hybrid
Choosing a Cloud Provider: Public-Private-Hybrid
 
Machine Learning Success: The Key to Easier Model Management
Machine Learning Success: The Key to Easier Model ManagementMachine Learning Success: The Key to Easier Model Management
Machine Learning Success: The Key to Easier Model Management
 
State of the Cloud and Data Centers 2014
State of the Cloud and Data Centers 2014State of the Cloud and Data Centers 2014
State of the Cloud and Data Centers 2014
 

Recently uploaded

Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jNeo4j
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptxFIDO Alliance
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGDSC PJATK
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FIDO Alliance
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxFIDO Alliance
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimaginedpanagenda
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctBrainSell Technologies
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfFIDO Alliance
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPTiSEO AI
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfSrushith Repakula
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...panagenda
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessUXDXConf
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024Stephen Perrenod
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfUK Journal
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentationyogeshlabana357357
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...FIDO Alliance
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireExakis Nelite
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...FIDO Alliance
 

Recently uploaded (20)

Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 

Streaming Architecture to Connect Everything (Including Hybrid Cloud) - Strata Singapore 2016

  • 1. © 2016 MapR Technologies 1© 2016 MapR Technologies 1MapR Confidential © 2016 MapR Technologies Architecting a hybrid cloud application using a global publish-subscribe streaming message system Mathieu Dumoulin (MapR Technologies) Strata Singapore 2016
  • 2. © 2016 MapR Technologies 2© 2016 MapR Technologies 2MapR Confidential © 2016 MapR Technologies Streaming Architecture to Connect Everything (including Hybrid Cloud) Mathieu Dumoulin (MapR Technologies) Strata Singapore 2016
  • 3. © 2016 MapR Technologies 3© 2016 MapR Technologies 3MapR Confidential Mathieu Dumoulin, Data Engineer • Master’s degree in text classification on Hadoop at Fujitsu Canada’s Innovation Lab and Laval University • In Tokyo, I’ve worked as a Data Scientist, Search Engineer and Data Engineer • Working on streaming, complex event processing and machine learning
  • 4. © 2016 MapR Technologies 4© 2016 MapR Technologies 4MapR Confidential The new rule for the future is going to be, “Anything that can be connected, will be connected.” Jacob Morgan, Forbes - May 2014
  • 5. © 2016 MapR Technologies 5© 2016 MapR Technologies 5MapR Confidential Talk Summary • Clouds: private vs. public vs. hybrid • It’s all about that streaming – Streaming for IoT – Publish-subscribe messaging systems (Kafka) – Stream Processing (Apache Spark Streaming, Apache Flink) – Microservices • Streams-based Architecture in the hybrid cloud – Design goals – Examples • Recap, Q&A
  • 6. © 2016 MapR Technologies 6© 2016 MapR Technologies 6MapR Confidential © 2016 MapR Technologies Weather today for IT:
  • 7. © 2016 MapR Technologies 7© 2016 MapR Technologies 7MapR Confidential Public Cloud - Low Upfront Cost and Flexibility The Good • Right size instances for application • Grow with the business • “Forever” extensible • Global in a few clicks The Bad • New complexity, no magic • Costs can run away The Ugly • Local data is far from processing • Severe lock-in without huge in-house expertise
  • 8. © 2016 MapR Technologies 8© 2016 MapR Technologies 8MapR Confidential Private Clouds - The Benefits of Ownership The Bad • Harder to scale vertically & horizontally • Cost of multiple datacenters The Ugly • Pay for spike, wasted resources • Never right size in a growing organization The Good • Direct access to data • Security, privacy and legal compliance • Hardware certainty • Low running cost
  • 9. © 2016 MapR Technologies 9© 2016 MapR Technologies 9MapR Confidential Private Cloud - Europe Private Cloud - Tokyo Hybrid = Public vs. + Private Spans at least one public and one private cloud. • Test new ideas with low up-front capital cost • Cloudbursting • High Availability and Disaster Recovery • Regulatory Requirements IT infrastructure agility
  • 10. © 2016 MapR Technologies 10© 2016 MapR Technologies 10MapR Confidential © 2016 MapR Technologies It’s all about that streaming
  • 11. © 2016 MapR Technologies 11© 2016 MapR Technologies 11MapR Confidential Streaming Architecture the Norm for Data Driven Organizations “Stream-based computing is becoming the norm for data-driven organizations” - Friedman & Dunning, Streaming Architecture • Build flexible systems – more efficient and easier to build – Decouples dependencies between data source and processing • Better model the way business processes take place. • More value now… and later – Aggregates data from many sources once – Serves data to one or many projects immediately – More efficient and high performance – Run batch analytics, reprocess data
  • 12. © 2016 MapR Technologies 12© 2016 MapR Technologies 12MapR Confidential IoT is a Natural Use Case for Streaming Connected devices produce data as real-time events that are modelled naturally as event streams. Event Some actions have value only if taken immediately – Navigation updates from traffic conditions, accident reports, disasters, … – Slowing down or stopping a factory line in response to quality issues – Re-routing items mid-way during shipping to increase efficiency – Continuous engine tuning
  • 13. © 2016 MapR Technologies 13© 2016 MapR Technologies 13MapR Confidential IoT is Happening Right Now!
  • 14. © 2016 MapR Technologies 14© 2016 MapR Technologies 14MapR Confidential Streams Make the Hybrid Cloud Practical Streams can serve for inter-cloud communication in the exact same way they support any other scenario. ● Abstracts the differences between on-premise and cloud ● Standardize the expected flow of data between modules ● Reuse data many times, break down data silos
  • 15. © 2016 MapR Technologies 15© 2016 MapR Technologies 15MapR Confidential What Streaming Requires from a Messaging System ● The producer and consumer are fully independent ● Very high throughput 1,000+/s → 1,000,000+/s ● Persistence ○ Fault-tolerance ○ Data is kept as a replayable sequence ○ Strong ordering of events ● Naming of topics (consumers pick the data they need ) ● Geo-distributed replication (for Hybrid Cloud use cases) It’s very hard to get full isolation of producer and consumers while also keeping very high speed, but we must have both.
  • 16. © 2016 MapR Technologies 16© 2016 MapR Technologies 16MapR Confidential What Streaming Requires from Stream Processing Frameworks Desirable features for real-time analytics frameworks: • Open Source, active development and developer community • Supports “exactly once” guarantee, stream reprocessing • How much real-time? Microbatch vs. record-at-a-time • Performance (latency, throughput) • Other: Easy to use, compatibility, talent availability To Know more: https://www.mapr.com/blog/stream-processing-everywhere-what-use Jim Scott - Stream Processing Everywhere - What to Use? Strata San Jose 2015 Also see Data Artisan’s Blog on Stream Processing Framework Myths
  • 17. © 2016 MapR Technologies 17© 2016 MapR Technologies 17MapR Confidential Which Stream Processing Frameworks?
  • 18. © 2016 MapR Technologies 18© 2016 MapR Technologies 18MapR Confidential Summing up: Technology to support Streaming 1. Lightweight messaging system 2. Stream Processing Framework You can get an Introduction to Flink in this Free Book published by O’Reilly
  • 19. © 2016 MapR Technologies 19© 2016 MapR Technologies 19MapR Confidential Key Ideas For Effectively Using Streams Real-time Analysis Persist to Disk Geo-distributed Replication Core part of Architecture
  • 20. © 2016 MapR Technologies 20© 2016 MapR Technologies 20MapR Confidential
  • 21. © 2016 MapR Technologies 21© 2016 MapR Technologies 21MapR Confidential Streaming Architecture: Ideal Platform for Microservices Microservices are a modern distributed architecture that realizes the promises of SOA, Service Oriented Architecture • Scale up from a test use case to a global deployment • Decouples components, more modular • Modern, agile development, testing and deployment • More robust and responsive See Krystal Valentine’s “The keys to an event-based microservices application” presentation, Strata New York 2016
  • 22. © 2016 MapR Technologies 22© 2016 MapR Technologies 22MapR Confidential Monolithic to Microservices Architecture See Fowler’s blog about microservices: http://www.martinfowler.com/articles/microservices.html
  • 23. © 2016 MapR Technologies 23© 2016 MapR Technologies 23MapR Confidential Microservices are Truly Decoupled
  • 24. © 2016 MapR Technologies 24© 2016 MapR Technologies 24MapR Confidential When to Use Streaming Architecture
  • 25. © 2016 MapR Technologies 25© 2016 MapR Technologies 25MapR Confidential © 2016 MapR Technologies Connect Clouds with Streams: Streams-based Architecture
  • 26. © 2016 MapR Technologies 26© 2016 MapR Technologies 26MapR Confidential Switch from thinking of computer programs as state-oriented to thinking of them in terms of flows” Ted Dunning & Ellen Friedman, Streaming Architecture - O’Reilly - 2016
  • 27. © 2016 MapR Technologies 27© 2016 MapR Technologies 27MapR Confidential An End-to-End Streaming Architecture Japan North Data Center Stream GW Global Data Center Stream
  • 28. © 2016 MapR Technologies 28© 2016 MapR Technologies 28MapR Confidential Example Architecture: Log Analysis
  • 29. © 2016 MapR Technologies 29© 2016 MapR Technologies 29MapR Confidential Example Architecture: Log Analysis
  • 30. © 2016 MapR Technologies 30© 2016 MapR Technologies 30MapR Confidential Example Architecture: The MapR Blueprint Download the Finserve app from Github! https://github.com/mapr-demos/finserv-application-blueprint
  • 31. © 2016 MapR Technologies 31© 2016 MapR Technologies 31MapR Confidential Conclusion • The hybrid cloud matters for IT agility • Use streams for communication between elements • Streaming-based systems can be arbitrarily complex – Still fast, responsive, reliable and easier to develop! • In a streaming architecture world, a converged platform (built-in streaming, storage and DB) makes a difference.
  • 32. © 2016 MapR Technologies 32© 2016 MapR Technologies 32MapR Confidential Suggested Reading And Video Links Get Ted & Ellen’s book: Read it Online for Free! New content presented by Ted Dunning: 1. Big Data in the Cloud (blog): www.mapr.com/big-data-cloud a. Direct video link: https://youtu.be/90KrQAb1_Cw 2. Converged Advantages in the Cloud (blog): www.mapr.com/converged-cloud a. Direct video link: https://youtu.be/yjfBXNcmAHA
  • 33. © 2016 MapR Technologies 33© 2016 MapR Technologies 33MapR Confidential Q & A @mapr mdumoulin@mapr.com @lordxar Engage with us! mapr-technologies
  • 34. © 2016 MapR Technologies 34© 2016 MapR Technologies 34MapR Confidential Key Ideas for Microservices • Services are opaque - API only • They communicate with only a few other services using lightweight, flexible protocols. – HTTP+REST - Synchronous (frontend) – Messaging Systems (Kafka, MapR Streams) - Asynchronous (backend) • Data formats should be future-proofed – JSON - Human readable, easy to use, low efficiency – Binary (Avro, Protobuf, Thrift) - Efficient but (somewhat) harder to use {RESTful}
  • 35. © 2016 MapR Technologies 35© 2016 MapR Technologies 35MapR Confidential Spark Streaming or Flink: Case by Case Micro-batches. Time-based window. Latency: seconds Continuous flow model. Record-based window. Latency: ms Both provide exactly once guarantee, high throughput and low overhead of fault tolerance. Both streaming and batch supported.
  • 36. © 2016 MapR Technologies 36© 2016 MapR Technologies 36MapR Confidential The Hybrid Cloud for IoT Infrastructure • IoT is a new use case - Need to Test • Built-in need for baseload capacity and bursting data spikes • Global marketplace requires geographically dispersed datacenters • Increasingly strict compliance requirements • IoT Security issues need to be taken seriously Why do IoT applications call out for the flexibility of Hybrid Cloud?