SlideShare a Scribd company logo
1 of 35
Download to read offline
© COPYRIGHT IKERLAN 2016 www.ikerlan.es
Solving the Industry 4.0. challenges on the
logistics domain using Apache Mesos
Technological Centre
Angel Conde, Cristobal Arellano, Santi
Charramendieta (IK4-Ikerlan)
Oscar Ocariz (ULMA Handling Systems)
© COPYRIGHT IKERLAN 2016 www.ikerlan.es
I. Introduction
II. Components
III. Architecture
IV. Conclusions & Future Work
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 3
ULMA Handling Systems
Provides all-round logistics systems (e.g., automatic warehouses)
Custom turnkey solutions
Design, development, assembly & maintenance
World Wide presence
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 4
An ULMA Warehouse
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 5
COMPONENTS
Conveyor
Stacker
Crane
Elevator
A system that is physically distributed
composed by independent devices
that carry out automatic tasks
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016
COMMON FAILURES
• Physical failures &Deterioration
• Logical failures
SW errors
SW updates
Logical/Physical mismatch
“Automatic elements fail sooner or later”
6
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 7
ULMA & the Industry 4.0.
Reduce down time
Integration with 3rd systems
Lower maintenance costs
Predictive maintenance
Smart Warehouse
Industry 4.0
BIG DATA
CIBERSECURITY
IIOT
CLOUD
MODELING &
SIMULATION
M2M
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 8
Supervisor
Message Bus
DDS
Adapter
DDS
Adapter
DDS
Adapter
ULMA - Supervisor
Message Bus based on DDS
Operational Data
“The Supervisor system gathers information about operational data and malfunctions”
Distributed: monitoring system nature
Scalable: different size of warehouse
Extendible: useful with different devices
Reliable: without errors
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 9
The Supervisor & Industry 4.0.
Useful for local maintenance
Local data valuable for later analytics
However….
No remote monitoring
Manual data collection
Operational data can be lost
Industry 4.0
BIG DATA
CIBERSECURITY
IIOT
CLOUD
MODELING &
SIMULATION
M2M
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 10
The Cloud Supervisor
Live Mirror of each Real Supervisor
Responsible of storing the operational data
Can define its own rules
Aggregates must be supported
Supervisor
1
Supervisor
2
Real Supervisors
Cloud
Supervisor
1
Cloud
Supervisor
1 & 2
Cloud
© COPYRIGHT IKERLAN 2016 www.ikerlan.es
I. Introduction
II. Components
III. Architecture
IV. Conclusions & Future Work
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 12
PLATFORM GOALS
Public Cloud, Private Cloud, Hybrid
Avoid Vendor Locking
Resiliency
Elasticity
Efficient resource usage
?
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 13
CLOUD AGNOSTIC PLATFORM
Why ?
From Static Resource Partitioning to Elastic Sharing
REST APICassandraSpark
Static Partitioning Elastic Sharing
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 14
MESOS OVERVIEW
HTTP
JVM
Python
Ruby
distributed resources: CPU,GPU, RAM, I/O, FS, rack locality, etc.
distributed file system
Cluster
DFS
Frameworks
Kernel
C++
Apps &
Services
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 15
MESOS FEATURES
Battle tested on Twitter
Up to 10,000 nodes
Launch any task using or cgroups
Resource isolation
Commercial support via
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 16
INFRASTRUCTURE
Launch platform on any cloud provider
Provisioning machines
Automate updates/upgrades
Configuration
“Infrastructureasacode”
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 17
BUILDING BLOCKS
Data Ingestion & Message Bus(Data Sources)
RT/Batch Data Analytics (Machine learning, failure prediction)
Data Storage (Horizontal scaling)
Advanced Visualization and User Experience
CloudInfrastructure
17
Supervisor
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 18
REVERSE PROXY&
LB
Multiple backends (Mesos, Consul, Docker…)
Dynamic Watchers for backends
Open Source
HTTPS SNI
Deployed on the “edge node”
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 19
MESSAGE BUS
Decouple the components
De Facto Standard in Big Data Architectures
Publish / Subscriber model
HA enabled
High Performance
Deployed using the Mesos Framework
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 20
STORAGE
HDFS deployed in HA (Mesos Master Nodes)
Files stored in Columnar format (Parquet)
Not deployed as Mesos framework
Horizontal scalability
“Small Files problem”
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 21
ANALYTICS
Big Data Analytics in Memory
Standard SQL Support
Deployed using the Mesos Spark Scheduler
Machine Learning
Real Time Processing
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 22
SERVICE DISCOVERY
& MONITORING
Service Discovery using DNS records
Port discovery via DNS SVR records
Service Health-monitoring
Circuit Breakers
Node/Service Monitoring
Task registration via Marathon/Consul
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 23
PLATFORM OVERVIEW
Edge Node
MESOS Agent Nodes
Consul
Client
Consul
Client
Agent
Proxy
MESOS Master Node
NameNode
Consul ServerMaster
External traffic
(HTTPS)
Secured Cloud Infrastructure
DataNode
Consul
Client
Agent
DataNode
ZooKeeper
Mesos coordinators
HDFS Namenodes
Consul for Service DiscoveryTraffic Ingestion
HTTPS SNI
Mesos Agent
Consul Client
HDFS Datanodes
Tasks (Supervisors)
© COPYRIGHT IKERLAN 2016 www.ikerlan.es
I. Introduction
II. Components
III. Architecture
IV. Conclusions & Future Work
© COPYRIGHT IKERLAN 2016 www.ikerlan.es 25
Message Bus
Ingestion/
Analytics
ULMA Cloud
Storage
Storage
Storage
Cloud
Supervisor
Cloud
Supervisor
Cloud
Supervisor
Ingestion/
Analytics
Ingestion/
Analytics
1,000 FT OVERVIEW
Supervisor
1
Supervisor
2
Supervisor
3
OperationalData
© COPYRIGHT IKERLAN 2016 www.ikerlan.es
JVM web based app
Dynamic port binding via Marathon
Cgroups used as containerizer
Entry points assigned via labels
Look for:
- Random number generation exhaustion
- JVM DNS Caching
26
PLATFORM COMPONENTS
The Cloud Supervisor
© COPYRIGHT IKERLAN 2016 www.ikerlan.es
Kafka producer module
Sends data to Kafka instead of the local system
Local queue used in case of connection problems
Each supervisor sends data to its own topic (aggregates)
27
PLATFORM COMPONENTS
The Save Interface for the (cloud) Supervisor
© COPYRIGHT IKERLAN 2016 www.ikerlan.es
Deployed using Spark Mesos Scheduler
Lambda architecture (RT / Batch)
Data Ingestion & Storage from Kafka
AVRO schemas simulating IEC61850.
Ingestion tested up to 100,000 events/sec
28
The Spark Server
PLATFORM COMPONENTS
© COPYRIGHT IKERLAN 2016 www.ikerlan.es 29
The Spark Server (II)
PLATFORM COMPONENTS
Cloud
Supervisor
1
Cloud
Supervisor
2.
Operational Data
Streaming
Retains recent data on memory
Persists operational data to HDFS
Lambda (RT/Batch) queries via SQL endpoint
REST API
SQL
Decouples the system
Each supervisor has its own topic
© COPYRIGHT IKERLAN 2016 www.ikerlan.es 30
The Spark Server (III)
PLATFORM COMPONENTS
Operational Data
Streaming
SQL
• Real-Time Data
“Recent Data” in memory table
RT algorithms
Data is saved to the “staging” HDFS directory
Uncompacted Partitioned Parquet (date/supervisor)
• Batch Data
Exposes “batch” data via “Old Data” Table
Reload compacted data from HDFS when the compaction is
done
© COPYRIGHT IKERLAN 2016 www.ikerlan.es
HDFS must: Avoid small files!!!!
How to solve it?
Scheduled task executed by Chronos
Kite project is used for the task
“Staging”  “Compacted”
31
PLATFORM COMPONENTS
The Compactor
© COPYRIGHT IKERLAN 2016 www.ikerlan.es
I. Introduction
II. Components
III. Architecture
IV. Conclusions & Future Work
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 33
CONCLUSIONS
Remote Monitoring already valuable for clients
Global data collection available 24x7
Platform easily deployable on any provider
Efficient resource usage
Analytics on recent and batch data using standard SQL
Industry 4.0
BIG DATA
CIBERSECURITY
IIOT
CLOUD
MODELING &
SIMULATION
M2M
© COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 34
FUTURE WORK
Move to
Real-time analytics
Evaluate Cassandra as backend storage
Structured streaming, dynamic allocation on Spark 2.0.+
Run more ULMA software on the platform
Overlay networks, aka “IP per container”
© COPYRIGHT IKERLAN 2016 www.ikerlan.es
www.ikerlan.es
Questions: aconde@ikerlan.es

More Related Content

Viewers also liked

logistics and the internet of things
logistics and the internet of thingslogistics and the internet of things
logistics and the internet of thingsSergey Zhdanov
 
What impact will the Internet of Things have on supply chain? A discussion - ...
What impact will the Internet of Things have on supply chain? A discussion - ...What impact will the Internet of Things have on supply chain? A discussion - ...
What impact will the Internet of Things have on supply chain? A discussion - ...Global Business Intelligence
 
Internet of Things, Connected Infrastructure & The Modern Supply Chain
Internet of Things, Connected Infrastructure & The Modern Supply ChainInternet of Things, Connected Infrastructure & The Modern Supply Chain
Internet of Things, Connected Infrastructure & The Modern Supply ChainJeff Risley
 
IOT & Autonomous Logistics
IOT & Autonomous LogisticsIOT & Autonomous Logistics
IOT & Autonomous LogisticsAkiso Yadav
 
Banking 4.0 - LibreCon 2016
Banking 4.0 - LibreCon 2016Banking 4.0 - LibreCon 2016
Banking 4.0 - LibreCon 2016LibreCon
 
How to answer the challenges of industrial IoT?
How to answer the challenges of industrial IoT?How to answer the challenges of industrial IoT?
How to answer the challenges of industrial IoT?Dieter Laevers
 
2015 06-17 - supply chaindays ibm presentation thorsten schroeer
2015 06-17 - supply chaindays ibm presentation thorsten schroeer2015 06-17 - supply chaindays ibm presentation thorsten schroeer
2015 06-17 - supply chaindays ibm presentation thorsten schroeerThorsten Schroeer
 
Imagine a Supply Chain That Can Maximize Profitability
Imagine a Supply Chain That Can Maximize ProfitabilityImagine a Supply Chain That Can Maximize Profitability
Imagine a Supply Chain That Can Maximize ProfitabilityLora Cecere
 
OT - How IoT will Impact Future B2B and Global Supply Chains - SS14
OT - How IoT will Impact Future B2B and Global Supply Chains - SS14OT - How IoT will Impact Future B2B and Global Supply Chains - SS14
OT - How IoT will Impact Future B2B and Global Supply Chains - SS14Mark Morley, MBA
 
Scm webinar digital transformation
Scm webinar   digital transformationScm webinar   digital transformation
Scm webinar digital transformationPanaya
 
Realizing Business and Supply Chain Value with Internet of Things (iot)
Realizing Business and Supply Chain Value with Internet of Things (iot) Realizing Business and Supply Chain Value with Internet of Things (iot)
Realizing Business and Supply Chain Value with Internet of Things (iot) GS1 Hong Kong
 
Winning_omnichannel_supply_chain_in_retail_-_guide
Winning_omnichannel_supply_chain_in_retail_-_guideWinning_omnichannel_supply_chain_in_retail_-_guide
Winning_omnichannel_supply_chain_in_retail_-_guideDebmalya Dutta
 
Logistics 4.0 and the Internet of Things
Logistics 4.0 and the Internet of ThingsLogistics 4.0 and the Internet of Things
Logistics 4.0 and the Internet of ThingsThorsten Huelsmann
 
Digital Supply Chain Transformation in the Networked Economy
Digital Supply Chain Transformation in the Networked EconomyDigital Supply Chain Transformation in the Networked Economy
Digital Supply Chain Transformation in the Networked EconomySAP Ariba
 
Towards a Connected World of Supply Chain - Industrie 4.0
Towards a Connected World of Supply Chain - Industrie 4.0Towards a Connected World of Supply Chain - Industrie 4.0
Towards a Connected World of Supply Chain - Industrie 4.0Sarathy Kalaichelvan
 
Rethinking Supply Chain Analytics
Rethinking Supply Chain AnalyticsRethinking Supply Chain Analytics
Rethinking Supply Chain AnalyticsCloudera, Inc.
 
Transforming big data into supply chain analytics
Transforming big data into supply chain analyticsTransforming big data into supply chain analytics
Transforming big data into supply chain analyticsTristan Wiggill
 
Smarter Supply Chain – IBM Case Study in Supply Chain Transformation and Inno...
Smarter Supply Chain – IBM Case Study in Supply Chain Transformation and Inno...Smarter Supply Chain – IBM Case Study in Supply Chain Transformation and Inno...
Smarter Supply Chain – IBM Case Study in Supply Chain Transformation and Inno...NUS-ISS
 

Viewers also liked (20)

logistics and the internet of things
logistics and the internet of thingslogistics and the internet of things
logistics and the internet of things
 
What impact will the Internet of Things have on supply chain? A discussion - ...
What impact will the Internet of Things have on supply chain? A discussion - ...What impact will the Internet of Things have on supply chain? A discussion - ...
What impact will the Internet of Things have on supply chain? A discussion - ...
 
Internet of Things, Connected Infrastructure & The Modern Supply Chain
Internet of Things, Connected Infrastructure & The Modern Supply ChainInternet of Things, Connected Infrastructure & The Modern Supply Chain
Internet of Things, Connected Infrastructure & The Modern Supply Chain
 
IOT & Autonomous Logistics
IOT & Autonomous LogisticsIOT & Autonomous Logistics
IOT & Autonomous Logistics
 
IOT & Procuement
IOT & ProcuementIOT & Procuement
IOT & Procuement
 
Banking 4.0 - LibreCon 2016
Banking 4.0 - LibreCon 2016Banking 4.0 - LibreCon 2016
Banking 4.0 - LibreCon 2016
 
How to answer the challenges of industrial IoT?
How to answer the challenges of industrial IoT?How to answer the challenges of industrial IoT?
How to answer the challenges of industrial IoT?
 
2015 06-17 - supply chaindays ibm presentation thorsten schroeer
2015 06-17 - supply chaindays ibm presentation thorsten schroeer2015 06-17 - supply chaindays ibm presentation thorsten schroeer
2015 06-17 - supply chaindays ibm presentation thorsten schroeer
 
Imagine a Supply Chain That Can Maximize Profitability
Imagine a Supply Chain That Can Maximize ProfitabilityImagine a Supply Chain That Can Maximize Profitability
Imagine a Supply Chain That Can Maximize Profitability
 
Industry 4.0. and SMEs – S3 Platform for Industrial Modernization
Industry 4.0. and SMEs – S3 Platform for Industrial ModernizationIndustry 4.0. and SMEs – S3 Platform for Industrial Modernization
Industry 4.0. and SMEs – S3 Platform for Industrial Modernization
 
OT - How IoT will Impact Future B2B and Global Supply Chains - SS14
OT - How IoT will Impact Future B2B and Global Supply Chains - SS14OT - How IoT will Impact Future B2B and Global Supply Chains - SS14
OT - How IoT will Impact Future B2B and Global Supply Chains - SS14
 
Scm webinar digital transformation
Scm webinar   digital transformationScm webinar   digital transformation
Scm webinar digital transformation
 
Realizing Business and Supply Chain Value with Internet of Things (iot)
Realizing Business and Supply Chain Value with Internet of Things (iot) Realizing Business and Supply Chain Value with Internet of Things (iot)
Realizing Business and Supply Chain Value with Internet of Things (iot)
 
Winning_omnichannel_supply_chain_in_retail_-_guide
Winning_omnichannel_supply_chain_in_retail_-_guideWinning_omnichannel_supply_chain_in_retail_-_guide
Winning_omnichannel_supply_chain_in_retail_-_guide
 
Logistics 4.0 and the Internet of Things
Logistics 4.0 and the Internet of ThingsLogistics 4.0 and the Internet of Things
Logistics 4.0 and the Internet of Things
 
Digital Supply Chain Transformation in the Networked Economy
Digital Supply Chain Transformation in the Networked EconomyDigital Supply Chain Transformation in the Networked Economy
Digital Supply Chain Transformation in the Networked Economy
 
Towards a Connected World of Supply Chain - Industrie 4.0
Towards a Connected World of Supply Chain - Industrie 4.0Towards a Connected World of Supply Chain - Industrie 4.0
Towards a Connected World of Supply Chain - Industrie 4.0
 
Rethinking Supply Chain Analytics
Rethinking Supply Chain AnalyticsRethinking Supply Chain Analytics
Rethinking Supply Chain Analytics
 
Transforming big data into supply chain analytics
Transforming big data into supply chain analyticsTransforming big data into supply chain analytics
Transforming big data into supply chain analytics
 
Smarter Supply Chain – IBM Case Study in Supply Chain Transformation and Inno...
Smarter Supply Chain – IBM Case Study in Supply Chain Transformation and Inno...Smarter Supply Chain – IBM Case Study in Supply Chain Transformation and Inno...
Smarter Supply Chain – IBM Case Study in Supply Chain Transformation and Inno...
 

Similar to Solving the Industry 4.0. challenges on the logistics domain using Apache Mesos

Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...
Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...
Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...Big Data Spain
 
Lift and shift to sparc cloud
Lift and shift to sparc cloudLift and shift to sparc cloud
Lift and shift to sparc cloudRiccardo Romani
 
C5 journey to_the_cloud_with_oracle_sparc
C5 journey to_the_cloud_with_oracle_sparcC5 journey to_the_cloud_with_oracle_sparc
C5 journey to_the_cloud_with_oracle_sparcDr. Wilfred Lin (Ph.D.)
 
MySQL in oracle public cloud
MySQL in oracle public cloudMySQL in oracle public cloud
MySQL in oracle public cloudMandy Ang
 
Copr HD OpenStack Day India
Copr HD OpenStack Day IndiaCopr HD OpenStack Day India
Copr HD OpenStack Day Indiaopenstackindia
 
MySQL in oracle_public_cloud
MySQL in oracle_public_cloudMySQL in oracle_public_cloud
MySQL in oracle_public_cloudOracleMySQL
 
AWS re:Invent 2016 - Scality's Open Source AWS S3 Server
AWS re:Invent 2016 - Scality's Open Source AWS S3 ServerAWS re:Invent 2016 - Scality's Open Source AWS S3 Server
AWS re:Invent 2016 - Scality's Open Source AWS S3 ServerScality
 
On-Premises Serverless Container-Aware Architecture (OSCAR)
On-Premises Serverless Container-Aware Architecture (OSCAR)On-Premises Serverless Container-Aware Architecture (OSCAR)
On-Premises Serverless Container-Aware Architecture (OSCAR)Germán Moltó
 
Episode 4: Operating Kubernetes at Scale with DC/OS
Episode 4: Operating Kubernetes at Scale with DC/OSEpisode 4: Operating Kubernetes at Scale with DC/OS
Episode 4: Operating Kubernetes at Scale with DC/OSMesosphere Inc.
 
OOW16 - Oracle E-Business Suite Information Discovery: Your Journey to the Cl...
OOW16 - Oracle E-Business Suite Information Discovery: Your Journey to the Cl...OOW16 - Oracle E-Business Suite Information Discovery: Your Journey to the Cl...
OOW16 - Oracle E-Business Suite Information Discovery: Your Journey to the Cl...vasuballa
 
Caching in the Cloud. Code Camp Iași April 2016.
Caching in the Cloud. Code Camp Iași April 2016. Caching in the Cloud. Code Camp Iași April 2016.
Caching in the Cloud. Code Camp Iași April 2016. Expert Network
 
Open Cloud Storage @ OpenStack Summit Paris
Open Cloud Storage @ OpenStack Summit ParisOpen Cloud Storage @ OpenStack Summit Paris
Open Cloud Storage @ OpenStack Summit Parisit-novum
 
Moving your Oracle Databases to the Oracle Cloud
Moving your Oracle Databases to the Oracle CloudMoving your Oracle Databases to the Oracle Cloud
Moving your Oracle Databases to the Oracle CloudAlex Zaballa
 
Oracle Management Cloud - HybridCloud Café - May 2016
Oracle Management Cloud - HybridCloud Café - May 2016Oracle Management Cloud - HybridCloud Café - May 2016
Oracle Management Cloud - HybridCloud Café - May 2016Bastien Leblanc
 
Oracle Cloud Café hybrid Cloud 19 mai 2016
Oracle Cloud Café hybrid Cloud 19 mai 2016Oracle Cloud Café hybrid Cloud 19 mai 2016
Oracle Cloud Café hybrid Cloud 19 mai 2016Sorathaya Sirimanotham
 
EMC World 2016 - code.13 State of the Container Ecosystem with Persistent App...
EMC World 2016 - code.13 State of the Container Ecosystem with Persistent App...EMC World 2016 - code.13 State of the Container Ecosystem with Persistent App...
EMC World 2016 - code.13 State of the Container Ecosystem with Persistent App...{code}
 
Container Orchestration Wars
Container Orchestration WarsContainer Orchestration Wars
Container Orchestration WarsKarl Isenberg
 

Similar to Solving the Industry 4.0. challenges on the logistics domain using Apache Mesos (20)

Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...
Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...
Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...
 
Lift and shift to sparc cloud
Lift and shift to sparc cloudLift and shift to sparc cloud
Lift and shift to sparc cloud
 
C5 journey to_the_cloud_with_oracle_sparc
C5 journey to_the_cloud_with_oracle_sparcC5 journey to_the_cloud_with_oracle_sparc
C5 journey to_the_cloud_with_oracle_sparc
 
MySQL in oracle public cloud
MySQL in oracle public cloudMySQL in oracle public cloud
MySQL in oracle public cloud
 
Copr HD OpenStack Day India
Copr HD OpenStack Day IndiaCopr HD OpenStack Day India
Copr HD OpenStack Day India
 
MySQL in oracle_public_cloud
MySQL in oracle_public_cloudMySQL in oracle_public_cloud
MySQL in oracle_public_cloud
 
AWS re:Invent 2016 - Scality's Open Source AWS S3 Server
AWS re:Invent 2016 - Scality's Open Source AWS S3 ServerAWS re:Invent 2016 - Scality's Open Source AWS S3 Server
AWS re:Invent 2016 - Scality's Open Source AWS S3 Server
 
Modern Software Development
Modern Software DevelopmentModern Software Development
Modern Software Development
 
On-Premises Serverless Container-Aware Architecture (OSCAR)
On-Premises Serverless Container-Aware Architecture (OSCAR)On-Premises Serverless Container-Aware Architecture (OSCAR)
On-Premises Serverless Container-Aware Architecture (OSCAR)
 
Episode 4: Operating Kubernetes at Scale with DC/OS
Episode 4: Operating Kubernetes at Scale with DC/OSEpisode 4: Operating Kubernetes at Scale with DC/OS
Episode 4: Operating Kubernetes at Scale with DC/OS
 
OOW16 - Oracle E-Business Suite Information Discovery: Your Journey to the Cl...
OOW16 - Oracle E-Business Suite Information Discovery: Your Journey to the Cl...OOW16 - Oracle E-Business Suite Information Discovery: Your Journey to the Cl...
OOW16 - Oracle E-Business Suite Information Discovery: Your Journey to the Cl...
 
Caching in the Cloud. Code Camp Iași April 2016.
Caching in the Cloud. Code Camp Iași April 2016. Caching in the Cloud. Code Camp Iași April 2016.
Caching in the Cloud. Code Camp Iași April 2016.
 
Silicon Valley Workshop: Xanadu introduction
Silicon Valley Workshop: Xanadu introduction Silicon Valley Workshop: Xanadu introduction
Silicon Valley Workshop: Xanadu introduction
 
Open Cloud Storage @ OpenStack Summit Paris
Open Cloud Storage @ OpenStack Summit ParisOpen Cloud Storage @ OpenStack Summit Paris
Open Cloud Storage @ OpenStack Summit Paris
 
Moving your Oracle Databases to the Oracle Cloud
Moving your Oracle Databases to the Oracle CloudMoving your Oracle Databases to the Oracle Cloud
Moving your Oracle Databases to the Oracle Cloud
 
OCI Overview
OCI OverviewOCI Overview
OCI Overview
 
Oracle Management Cloud - HybridCloud Café - May 2016
Oracle Management Cloud - HybridCloud Café - May 2016Oracle Management Cloud - HybridCloud Café - May 2016
Oracle Management Cloud - HybridCloud Café - May 2016
 
Oracle Cloud Café hybrid Cloud 19 mai 2016
Oracle Cloud Café hybrid Cloud 19 mai 2016Oracle Cloud Café hybrid Cloud 19 mai 2016
Oracle Cloud Café hybrid Cloud 19 mai 2016
 
EMC World 2016 - code.13 State of the Container Ecosystem with Persistent App...
EMC World 2016 - code.13 State of the Container Ecosystem with Persistent App...EMC World 2016 - code.13 State of the Container Ecosystem with Persistent App...
EMC World 2016 - code.13 State of the Container Ecosystem with Persistent App...
 
Container Orchestration Wars
Container Orchestration WarsContainer Orchestration Wars
Container Orchestration Wars
 

More from Angel Conde Manjon

Software Realibility on the Big Data Era
Software Realibility on the Big Data EraSoftware Realibility on the Big Data Era
Software Realibility on the Big Data EraAngel Conde Manjon
 
Evolución hacia las plataformas de datos modernas, el Edge-to-cloud continuum
Evolución hacia las plataformas de datos modernas, el Edge-to-cloud continuumEvolución hacia las plataformas de datos modernas, el Edge-to-cloud continuum
Evolución hacia las plataformas de datos modernas, el Edge-to-cloud continuumAngel Conde Manjon
 
Continous Delivery and Continous Integration at IKERLAN
Continous Delivery and Continous Integration at IKERLANContinous Delivery and Continous Integration at IKERLAN
Continous Delivery and Continous Integration at IKERLANAngel Conde Manjon
 
Towards an Unified API for Spark and the IIoT
Towards an Unified API for Spark and the IIoTTowards an Unified API for Spark and the IIoT
Towards an Unified API for Spark and the IIoTAngel Conde Manjon
 

More from Angel Conde Manjon (6)

Software Realibility on the Big Data Era
Software Realibility on the Big Data EraSoftware Realibility on the Big Data Era
Software Realibility on the Big Data Era
 
Evolución hacia las plataformas de datos modernas, el Edge-to-cloud continuum
Evolución hacia las plataformas de datos modernas, el Edge-to-cloud continuumEvolución hacia las plataformas de datos modernas, el Edge-to-cloud continuum
Evolución hacia las plataformas de datos modernas, el Edge-to-cloud continuum
 
Continous Delivery and Continous Integration at IKERLAN
Continous Delivery and Continous Integration at IKERLANContinous Delivery and Continous Integration at IKERLAN
Continous Delivery and Continous Integration at IKERLAN
 
Towards an Unified API for Spark and the IIoT
Towards an Unified API for Spark and the IIoTTowards an Unified API for Spark and the IIoT
Towards an Unified API for Spark and the IIoT
 
Modern Java Development
Modern Java DevelopmentModern Java Development
Modern Java Development
 
Ph.D. Defense
Ph.D. Defense Ph.D. Defense
Ph.D. Defense
 

Recently uploaded

GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jGraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jNeo4j
 
Business Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisBusiness Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisDEEPRAJ PATHAK
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxRTS corp
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsDEEPRAJ PATHAK
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdfAndrey Devyatkin
 
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...OnePlan Solutions
 
Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxAS Design & AST.
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingShane Coughlan
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdfSteve Caron
 
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive ReviewRevolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Reviewjw364beach
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slidesvaideheekore1
 
Osi security architecture in network.pptx
Osi security architecture in network.pptxOsi security architecture in network.pptx
Osi security architecture in network.pptxVinzoCenzo
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesVictoriaMetrics
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shardsChristopher Curtin
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?Alexandre Beguel
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsJean Silva
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...OnePlan Solutions
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldRoberto Pérez Alcolea
 
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfEnhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfRTS corp
 

Recently uploaded (20)

GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4jGraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
GraphSummit Madrid - Product Vision and Roadmap - Luis Salvador Neo4j
 
Business Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisBusiness Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business Analysis
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptx
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software Projects
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
 
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
 
Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptx
 
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full RecordingOpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
OpenChain AI Study Group - Europe and Asia Recap - 2024-04-11 - Full Recording
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
 
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive ReviewRevolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Review
 
Introduction to Firebase Workshop Slides
Introduction to Firebase Workshop SlidesIntroduction to Firebase Workshop Slides
Introduction to Firebase Workshop Slides
 
Osi security architecture in network.pptx
Osi security architecture in network.pptxOsi security architecture in network.pptx
Osi security architecture in network.pptx
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 Updates
 
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News UpdateVictoriaMetrics Q1 Meet Up '24 - Community & News Update
VictoriaMetrics Q1 Meet Up '24 - Community & News Update
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero results
 
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
Revolutionizing the Digital Transformation Office - Leveraging OnePlan’s AI a...
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository world
 
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdfEnhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
Enhancing Supply Chain Visibility with Cargo Cloud Solutions.pdf
 

Solving the Industry 4.0. challenges on the logistics domain using Apache Mesos

  • 1. © COPYRIGHT IKERLAN 2016 www.ikerlan.es Solving the Industry 4.0. challenges on the logistics domain using Apache Mesos Technological Centre Angel Conde, Cristobal Arellano, Santi Charramendieta (IK4-Ikerlan) Oscar Ocariz (ULMA Handling Systems)
  • 2. © COPYRIGHT IKERLAN 2016 www.ikerlan.es I. Introduction II. Components III. Architecture IV. Conclusions & Future Work
  • 3. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 3 ULMA Handling Systems Provides all-round logistics systems (e.g., automatic warehouses) Custom turnkey solutions Design, development, assembly & maintenance World Wide presence
  • 4. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 4 An ULMA Warehouse
  • 5. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 5 COMPONENTS Conveyor Stacker Crane Elevator A system that is physically distributed composed by independent devices that carry out automatic tasks
  • 6. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 COMMON FAILURES • Physical failures &Deterioration • Logical failures SW errors SW updates Logical/Physical mismatch “Automatic elements fail sooner or later” 6
  • 7. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 7 ULMA & the Industry 4.0. Reduce down time Integration with 3rd systems Lower maintenance costs Predictive maintenance Smart Warehouse Industry 4.0 BIG DATA CIBERSECURITY IIOT CLOUD MODELING & SIMULATION M2M
  • 8. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 8 Supervisor Message Bus DDS Adapter DDS Adapter DDS Adapter ULMA - Supervisor Message Bus based on DDS Operational Data “The Supervisor system gathers information about operational data and malfunctions” Distributed: monitoring system nature Scalable: different size of warehouse Extendible: useful with different devices Reliable: without errors
  • 9. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 9 The Supervisor & Industry 4.0. Useful for local maintenance Local data valuable for later analytics However…. No remote monitoring Manual data collection Operational data can be lost Industry 4.0 BIG DATA CIBERSECURITY IIOT CLOUD MODELING & SIMULATION M2M
  • 10. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 10 The Cloud Supervisor Live Mirror of each Real Supervisor Responsible of storing the operational data Can define its own rules Aggregates must be supported Supervisor 1 Supervisor 2 Real Supervisors Cloud Supervisor 1 Cloud Supervisor 1 & 2 Cloud
  • 11. © COPYRIGHT IKERLAN 2016 www.ikerlan.es I. Introduction II. Components III. Architecture IV. Conclusions & Future Work
  • 12. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 12 PLATFORM GOALS Public Cloud, Private Cloud, Hybrid Avoid Vendor Locking Resiliency Elasticity Efficient resource usage ?
  • 13. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 13 CLOUD AGNOSTIC PLATFORM Why ? From Static Resource Partitioning to Elastic Sharing REST APICassandraSpark Static Partitioning Elastic Sharing
  • 14. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 14 MESOS OVERVIEW HTTP JVM Python Ruby distributed resources: CPU,GPU, RAM, I/O, FS, rack locality, etc. distributed file system Cluster DFS Frameworks Kernel C++ Apps & Services
  • 15. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 15 MESOS FEATURES Battle tested on Twitter Up to 10,000 nodes Launch any task using or cgroups Resource isolation Commercial support via
  • 16. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 16 INFRASTRUCTURE Launch platform on any cloud provider Provisioning machines Automate updates/upgrades Configuration “Infrastructureasacode”
  • 17. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 17 BUILDING BLOCKS Data Ingestion & Message Bus(Data Sources) RT/Batch Data Analytics (Machine learning, failure prediction) Data Storage (Horizontal scaling) Advanced Visualization and User Experience CloudInfrastructure 17 Supervisor
  • 18. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 18 REVERSE PROXY& LB Multiple backends (Mesos, Consul, Docker…) Dynamic Watchers for backends Open Source HTTPS SNI Deployed on the “edge node”
  • 19. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 19 MESSAGE BUS Decouple the components De Facto Standard in Big Data Architectures Publish / Subscriber model HA enabled High Performance Deployed using the Mesos Framework
  • 20. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 20 STORAGE HDFS deployed in HA (Mesos Master Nodes) Files stored in Columnar format (Parquet) Not deployed as Mesos framework Horizontal scalability “Small Files problem”
  • 21. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 21 ANALYTICS Big Data Analytics in Memory Standard SQL Support Deployed using the Mesos Spark Scheduler Machine Learning Real Time Processing
  • 22. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 22 SERVICE DISCOVERY & MONITORING Service Discovery using DNS records Port discovery via DNS SVR records Service Health-monitoring Circuit Breakers Node/Service Monitoring Task registration via Marathon/Consul
  • 23. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 23 PLATFORM OVERVIEW Edge Node MESOS Agent Nodes Consul Client Consul Client Agent Proxy MESOS Master Node NameNode Consul ServerMaster External traffic (HTTPS) Secured Cloud Infrastructure DataNode Consul Client Agent DataNode ZooKeeper Mesos coordinators HDFS Namenodes Consul for Service DiscoveryTraffic Ingestion HTTPS SNI Mesos Agent Consul Client HDFS Datanodes Tasks (Supervisors)
  • 24. © COPYRIGHT IKERLAN 2016 www.ikerlan.es I. Introduction II. Components III. Architecture IV. Conclusions & Future Work
  • 25. © COPYRIGHT IKERLAN 2016 www.ikerlan.es 25 Message Bus Ingestion/ Analytics ULMA Cloud Storage Storage Storage Cloud Supervisor Cloud Supervisor Cloud Supervisor Ingestion/ Analytics Ingestion/ Analytics 1,000 FT OVERVIEW Supervisor 1 Supervisor 2 Supervisor 3 OperationalData
  • 26. © COPYRIGHT IKERLAN 2016 www.ikerlan.es JVM web based app Dynamic port binding via Marathon Cgroups used as containerizer Entry points assigned via labels Look for: - Random number generation exhaustion - JVM DNS Caching 26 PLATFORM COMPONENTS The Cloud Supervisor
  • 27. © COPYRIGHT IKERLAN 2016 www.ikerlan.es Kafka producer module Sends data to Kafka instead of the local system Local queue used in case of connection problems Each supervisor sends data to its own topic (aggregates) 27 PLATFORM COMPONENTS The Save Interface for the (cloud) Supervisor
  • 28. © COPYRIGHT IKERLAN 2016 www.ikerlan.es Deployed using Spark Mesos Scheduler Lambda architecture (RT / Batch) Data Ingestion & Storage from Kafka AVRO schemas simulating IEC61850. Ingestion tested up to 100,000 events/sec 28 The Spark Server PLATFORM COMPONENTS
  • 29. © COPYRIGHT IKERLAN 2016 www.ikerlan.es 29 The Spark Server (II) PLATFORM COMPONENTS Cloud Supervisor 1 Cloud Supervisor 2. Operational Data Streaming Retains recent data on memory Persists operational data to HDFS Lambda (RT/Batch) queries via SQL endpoint REST API SQL Decouples the system Each supervisor has its own topic
  • 30. © COPYRIGHT IKERLAN 2016 www.ikerlan.es 30 The Spark Server (III) PLATFORM COMPONENTS Operational Data Streaming SQL • Real-Time Data “Recent Data” in memory table RT algorithms Data is saved to the “staging” HDFS directory Uncompacted Partitioned Parquet (date/supervisor) • Batch Data Exposes “batch” data via “Old Data” Table Reload compacted data from HDFS when the compaction is done
  • 31. © COPYRIGHT IKERLAN 2016 www.ikerlan.es HDFS must: Avoid small files!!!! How to solve it? Scheduled task executed by Chronos Kite project is used for the task “Staging”  “Compacted” 31 PLATFORM COMPONENTS The Compactor
  • 32. © COPYRIGHT IKERLAN 2016 www.ikerlan.es I. Introduction II. Components III. Architecture IV. Conclusions & Future Work
  • 33. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 33 CONCLUSIONS Remote Monitoring already valuable for clients Global data collection available 24x7 Platform easily deployable on any provider Efficient resource usage Analytics on recent and batch data using standard SQL Industry 4.0 BIG DATA CIBERSECURITY IIOT CLOUD MODELING & SIMULATION M2M
  • 34. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 34 FUTURE WORK Move to Real-time analytics Evaluate Cassandra as backend storage Structured streaming, dynamic allocation on Spark 2.0.+ Run more ULMA software on the platform Overlay networks, aka “IP per container”
  • 35. © COPYRIGHT IKERLAN 2016 www.ikerlan.es www.ikerlan.es Questions: aconde@ikerlan.es

Editor's Notes

  1. Good afternoon to everybody, I´m Angel Conde DevOps & Data Engineer at Ikerlan. This is a joint Work between ULMA Handling Systems and Ikerlan about how the challenges of Industry 4.0. can be solved using Apache Mesos on the Logistics Domain.
  2. This talk is organised in four main sections. The first part is an introduction(introdaxion) to Ulma Handling and the industry 4.0. Later, we will go throughout(thruout) the different components used in the solutio. Following, the architecture(arkitecture) of the solutions will be described, And Finally, I will give some brief(brif) conclusions and future work Next, the three main parts of the system will be depicted
  3. Let´s start with an introduction (introdacxion) to ULMA Handling Systems. Ulma provides all-round logistic systems. (for example automatic warehouses) It focus on custom(castom) turnkey solutions where the desaing, development , assemble and maintenance is carried out by ULMA. Furthermore, ULMA has worldwide presence
  4. Well let’s see what is an ULMA Warehouse(werehouse),
  5. A warehouse is composed by different distributed elements For example stacker cranes, elevators and conveyors
  6. However automatiic elements fail sooner or later because phisical failures or logical failures. Software errors, incompabilities during updates, and logical pysical mismatch for example a human can move some piece to another place where it shouldn’t be for the logical program.
  7. While building the Supervisor system the world started to focus Definitely ULMA wants a SMART WAREHOUSe
  8. This lead to ULMA to build the Supervisor system. The Ulma Supervisor gathers information about operational data and malfunctions in a distributed manner.
  9. This lead to ULMA to build the Supervisor system. The Ulma Supervisor gathers information about operational data and malfunctions in a distributed manner. Valible
  10. For starting to solve those concpets Data from the warehouses should go to the cloud. Then the cloud supervisor concept born The base code must be the same That’s why the cloud The Cloud supervisor has been born. Each cloud supervisor is respónsible of storing data using custom rules. For example it can decide that an alarm generated by a real supervisor should not bestored. Therefore, this component is stateful being a “real” mirror of what’s happening on the real werehouse. Finally in some cases we would like to support aggregates, that’s it a cloud supervisor can have data from different real supervisors. This leads up to the next point
  11. Well its time to focus on the components of the plaform
  12. In order to support the cloud supervisors and store its operational data for later analytics. In this work some requisites had been defined. Goals to achiviecve our platform First of all we won like
  13. Apache Mesos is an open-source cluster manager that was developed at Berkley. It provides efficient resource isolation and sharing across distributed applications or frameworks. The software enables resource sharing in a fine-grained manner improving cluster utilization
  14. Let’s see an overview on the Meso pllatform. In the bottom part we have the psically distributed resources, next we have a distributed file system (for example the hdfs ) , next the Mesos Kernel in charge of the resources. Then frameworks and finally different apps and services. This lead me to next point, how we build this infrastructure? We start to create Amazon instances and start to installing all kind of things via the shell?
  15. Mesos has been Battle tested on Twitter first. It has support for clusters up to 10,000 nodes. It can launch and un any task using Docker contenairezation or cgroups
  16. Infrstructure as code concept To Launch ainfrastructure — from physical and virtual servers to email and DNS servers in any cloud infrastructure we have used terraform. Terraform leads as to the …. For provisioning and upgrading configuration and updates Ansible has been used
  17. Infrstructure as code concept To Launch ainfrastructure — from physical and virtual servers to email and DNS servers in any cloud infrastructure we have used terraform. Terraform leads as to the …. For provisioning and upgrading configuration and updates Ansible has been used
  18. We use domain based redirection
  19. dicaple
  20. For the platform storarge we have chosen HDFS as the tecnhonolog It’s not have been deployed using the corresponding mesos framework for avoiding possible losses
  21. We have the incoming traffic through the edge nodes where the https is terminated along the
  22. Well its time to focus on the platform architecture
  23. Here in this figure an overview of theh platform can be seen. Different Supervisots are connected to the cloud
  24. Well Spark is used as our analytical / ingestion tool. Wee use it for both batch and realtime data using the Lambda architecture. Data ingestion and storage drom the, uses the cluster resources for computing the queriies and we encode the data using
  25. Hdfs backd Recent data table Real time algorithms Data is saved to the stagin HDFS directory for later compaction Uncompacted partitioned parque (date / by supervisor)
  26. The final component of the platform is the one called the compactor. This component is respnsible for HDFS file compaction and is executed by the Chronos framework . Chronos is a Mesos framework rresponsible of batch tasks. The compactor uses Kite (kaite) project for this task where we have two partitioned folders. The stagin and the compacted one
  27. For the final part of the presentation, some conclusions and future work will be presented
  28. Let’s start with some conclusions(conclusions), Remote monitoring has already been valuable for the clients in order to provide remote maintenance. The collected data is available 24x7. The developed platform is easily deployable on any cloud provider, it has a efficient resource usage mixing big data workloads with custom services. Analytics can be done using standard SQL over recent and cold data .
  29. Finally I would like to depict some future work we would like to address: - As Mesospehere has open sourced DC/OS we would like to “move” in order to have commercial support - We would also like to explore the oversubscription resources on mesos in order to improve the usage of the resources. - Evaluate Cassandra as backend storage to avoid the HDFS compaction problem - More ULMA Handling software on the platform, hopefully stateful services or databases whether stateful apps system matures.
  30. We have come to the end of the presentation. I’d just like to thank(thenk) you for listening and would be pleased to take your comments and questions now. And if you want to get in touch just drop me an email.
  31. After going throughout the requisites let’s review the historically infrastructure archetipes. First of all we have the On-premises, later on Proprietary Hyperscale, after the cloud arisen and finally…..