Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to Solving the Industry 4.0. challenges on the logistics domain using Apache Mesos
Similar to Solving the Industry 4.0. challenges on the logistics domain using Apache Mesos (20)
More from Angel Conde Manjon
Recently uploaded
Recently uploaded (20)
Solving the Industry 4.0. challenges on the logistics domain using Apache Mesos
- 1. © COPYRIGHT IKERLAN 2016 www.ikerlan.es Solving the Industry 4.0. challenges on the logistics domain using Apache Mesos Technological Centre Angel Conde, Cristobal Arellano, Santi Charramendieta (IK4-Ikerlan) Oscar Ocariz (ULMA Handling Systems)
- 2. © COPYRIGHT IKERLAN 2016 www.ikerlan.es I. Introduction II. Components III. Architecture IV. Conclusions & Future Work
- 3. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 3 ULMA Handling Systems Provides all-round logistics systems (e.g., automatic warehouses) Custom turnkey solutions Design, development, assembly & maintenance World Wide presence
- 4. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 4 An ULMA Warehouse
- 5. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 5 COMPONENTS Conveyor Stacker Crane Elevator A system that is physically distributed composed by independent devices that carry out automatic tasks
- 6. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 COMMON FAILURES • Physical failures &Deterioration • Logical failures SW errors SW updates Logical/Physical mismatch “Automatic elements fail sooner or later” 6
- 7. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 7 ULMA & the Industry 4.0. Reduce down time Integration with 3rd systems Lower maintenance costs Predictive maintenance Smart Warehouse Industry 4.0 BIG DATA CIBERSECURITY IIOT CLOUD MODELING & SIMULATION M2M
- 8. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 8 Supervisor Message Bus DDS Adapter DDS Adapter DDS Adapter ULMA - Supervisor Message Bus based on DDS Operational Data “The Supervisor system gathers information about operational data and malfunctions” Distributed: monitoring system nature Scalable: different size of warehouse Extendible: useful with different devices Reliable: without errors
- 9. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 9 The Supervisor & Industry 4.0. Useful for local maintenance Local data valuable for later analytics However…. No remote monitoring Manual data collection Operational data can be lost Industry 4.0 BIG DATA CIBERSECURITY IIOT CLOUD MODELING & SIMULATION M2M
- 10. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 10 The Cloud Supervisor Live Mirror of each Real Supervisor Responsible of storing the operational data Can define its own rules Aggregates must be supported Supervisor 1 Supervisor 2 Real Supervisors Cloud Supervisor 1 Cloud Supervisor 1 & 2 Cloud
- 11. © COPYRIGHT IKERLAN 2016 www.ikerlan.es I. Introduction II. Components III. Architecture IV. Conclusions & Future Work
- 12. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 12 PLATFORM GOALS Public Cloud, Private Cloud, Hybrid Avoid Vendor Locking Resiliency Elasticity Efficient resource usage ?
- 13. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 13 CLOUD AGNOSTIC PLATFORM Why ? From Static Resource Partitioning to Elastic Sharing REST APICassandraSpark Static Partitioning Elastic Sharing
- 14. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 14 MESOS OVERVIEW HTTP JVM Python Ruby distributed resources: CPU,GPU, RAM, I/O, FS, rack locality, etc. distributed file system Cluster DFS Frameworks Kernel C++ Apps & Services
- 15. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 15 MESOS FEATURES Battle tested on Twitter Up to 10,000 nodes Launch any task using or cgroups Resource isolation Commercial support via
- 16. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 16 INFRASTRUCTURE Launch platform on any cloud provider Provisioning machines Automate updates/upgrades Configuration “Infrastructureasacode”
- 17. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 17 BUILDING BLOCKS Data Ingestion & Message Bus(Data Sources) RT/Batch Data Analytics (Machine learning, failure prediction) Data Storage (Horizontal scaling) Advanced Visualization and User Experience CloudInfrastructure 17 Supervisor
- 18. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 18 REVERSE PROXY& LB Multiple backends (Mesos, Consul, Docker…) Dynamic Watchers for backends Open Source HTTPS SNI Deployed on the “edge node”
- 19. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 19 MESSAGE BUS Decouple the components De Facto Standard in Big Data Architectures Publish / Subscriber model HA enabled High Performance Deployed using the Mesos Framework
- 20. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 20 STORAGE HDFS deployed in HA (Mesos Master Nodes) Files stored in Columnar format (Parquet) Not deployed as Mesos framework Horizontal scalability “Small Files problem”
- 21. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 21 ANALYTICS Big Data Analytics in Memory Standard SQL Support Deployed using the Mesos Spark Scheduler Machine Learning Real Time Processing
- 22. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 22 SERVICE DISCOVERY & MONITORING Service Discovery using DNS records Port discovery via DNS SVR records Service Health-monitoring Circuit Breakers Node/Service Monitoring Task registration via Marathon/Consul
- 23. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 23 PLATFORM OVERVIEW Edge Node MESOS Agent Nodes Consul Client Consul Client Agent Proxy MESOS Master Node NameNode Consul ServerMaster External traffic (HTTPS) Secured Cloud Infrastructure DataNode Consul Client Agent DataNode ZooKeeper Mesos coordinators HDFS Namenodes Consul for Service DiscoveryTraffic Ingestion HTTPS SNI Mesos Agent Consul Client HDFS Datanodes Tasks (Supervisors)
- 24. © COPYRIGHT IKERLAN 2016 www.ikerlan.es I. Introduction II. Components III. Architecture IV. Conclusions & Future Work
- 25. © COPYRIGHT IKERLAN 2016 www.ikerlan.es 25 Message Bus Ingestion/ Analytics ULMA Cloud Storage Storage Storage Cloud Supervisor Cloud Supervisor Cloud Supervisor Ingestion/ Analytics Ingestion/ Analytics 1,000 FT OVERVIEW Supervisor 1 Supervisor 2 Supervisor 3 OperationalData
- 26. © COPYRIGHT IKERLAN 2016 www.ikerlan.es JVM web based app Dynamic port binding via Marathon Cgroups used as containerizer Entry points assigned via labels Look for: - Random number generation exhaustion - JVM DNS Caching 26 PLATFORM COMPONENTS The Cloud Supervisor
- 27. © COPYRIGHT IKERLAN 2016 www.ikerlan.es Kafka producer module Sends data to Kafka instead of the local system Local queue used in case of connection problems Each supervisor sends data to its own topic (aggregates) 27 PLATFORM COMPONENTS The Save Interface for the (cloud) Supervisor
- 28. © COPYRIGHT IKERLAN 2016 www.ikerlan.es Deployed using Spark Mesos Scheduler Lambda architecture (RT / Batch) Data Ingestion & Storage from Kafka AVRO schemas simulating IEC61850. Ingestion tested up to 100,000 events/sec 28 The Spark Server PLATFORM COMPONENTS
- 29. © COPYRIGHT IKERLAN 2016 www.ikerlan.es 29 The Spark Server (II) PLATFORM COMPONENTS Cloud Supervisor 1 Cloud Supervisor 2. Operational Data Streaming Retains recent data on memory Persists operational data to HDFS Lambda (RT/Batch) queries via SQL endpoint REST API SQL Decouples the system Each supervisor has its own topic
- 30. © COPYRIGHT IKERLAN 2016 www.ikerlan.es 30 The Spark Server (III) PLATFORM COMPONENTS Operational Data Streaming SQL • Real-Time Data “Recent Data” in memory table RT algorithms Data is saved to the “staging” HDFS directory Uncompacted Partitioned Parquet (date/supervisor) • Batch Data Exposes “batch” data via “Old Data” Table Reload compacted data from HDFS when the compaction is done
- 31. © COPYRIGHT IKERLAN 2016 www.ikerlan.es HDFS must: Avoid small files!!!! How to solve it? Scheduled task executed by Chronos Kite project is used for the task “Staging” “Compacted” 31 PLATFORM COMPONENTS The Compactor
- 32. © COPYRIGHT IKERLAN 2016 www.ikerlan.es I. Introduction II. Components III. Architecture IV. Conclusions & Future Work
- 33. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 33 CONCLUSIONS Remote Monitoring already valuable for clients Global data collection available 24x7 Platform easily deployable on any provider Efficient resource usage Analytics on recent and batch data using standard SQL Industry 4.0 BIG DATA CIBERSECURITY IIOT CLOUD MODELING & SIMULATION M2M
- 34. © COPYRIGHT IKERLAN 2016 www.ikerlan.es© COPYRIGHT IKERLAN 2016 34 FUTURE WORK Move to Real-time analytics Evaluate Cassandra as backend storage Structured streaming, dynamic allocation on Spark 2.0.+ Run more ULMA software on the platform Overlay networks, aka “IP per container”
- 35. © COPYRIGHT IKERLAN 2016 www.ikerlan.es www.ikerlan.es Questions: aconde@ikerlan.es
Editor's Notes
- Good afternoon to everybody, I´m Angel Conde DevOps & Data Engineer at Ikerlan. This is a joint Work between ULMA Handling Systems and Ikerlan about how the challenges of Industry 4.0. can be solved using Apache Mesos on the Logistics Domain.
- This talk is organised in four main sections. The first part is an introduction(introdaxion) to Ulma Handling and the industry 4.0. Later, we will go throughout(thruout) the different components used in the solutio. Following, the architecture(arkitecture) of the solutions will be described, And Finally, I will give some brief(brif) conclusions and future work Next, the three main parts of the system will be depicted
- Let´s start with an introduction (introdacxion) to ULMA Handling Systems. Ulma provides all-round logistic systems. (for example automatic warehouses) It focus on custom(castom) turnkey solutions where the desaing, development , assemble and maintenance is carried out by ULMA. Furthermore, ULMA has worldwide presence
- Well let’s see what is an ULMA Warehouse(werehouse),
- A warehouse is composed by different distributed elements For example stacker cranes, elevators and conveyors
- However automatiic elements fail sooner or later because phisical failures or logical failures. Software errors, incompabilities during updates, and logical pysical mismatch for example a human can move some piece to another place where it shouldn’t be for the logical program.
- While building the Supervisor system the world started to focus Definitely ULMA wants a SMART WAREHOUSe
- This lead to ULMA to build the Supervisor system. The Ulma Supervisor gathers information about operational data and malfunctions in a distributed manner.
- This lead to ULMA to build the Supervisor system. The Ulma Supervisor gathers information about operational data and malfunctions in a distributed manner. Valible
- For starting to solve those concpets Data from the warehouses should go to the cloud. Then the cloud supervisor concept born The base code must be the same That’s why the cloud The Cloud supervisor has been born. Each cloud supervisor is respónsible of storing data using custom rules. For example it can decide that an alarm generated by a real supervisor should not bestored. Therefore, this component is stateful being a “real” mirror of what’s happening on the real werehouse. Finally in some cases we would like to support aggregates, that’s it a cloud supervisor can have data from different real supervisors. This leads up to the next point
- Well its time to focus on the components of the plaform
- In order to support the cloud supervisors and store its operational data for later analytics. In this work some requisites had been defined. Goals to achiviecve our platform First of all we won like
- Apache Mesos is an open-source cluster manager that was developed at Berkley. It provides efficient resource isolation and sharing across distributed applications or frameworks. The software enables resource sharing in a fine-grained manner improving cluster utilization
- Let’s see an overview on the Meso pllatform. In the bottom part we have the psically distributed resources, next we have a distributed file system (for example the hdfs ) , next the Mesos Kernel in charge of the resources. Then frameworks and finally different apps and services. This lead me to next point, how we build this infrastructure? We start to create Amazon instances and start to installing all kind of things via the shell?
- Mesos has been Battle tested on Twitter first. It has support for clusters up to 10,000 nodes. It can launch and un any task using Docker contenairezation or cgroups
- Infrstructure as code concept To Launch ainfrastructure — from physical and virtual servers to email and DNS servers in any cloud infrastructure we have used terraform. Terraform leads as to the …. For provisioning and upgrading configuration and updates Ansible has been used
- Infrstructure as code concept To Launch ainfrastructure — from physical and virtual servers to email and DNS servers in any cloud infrastructure we have used terraform. Terraform leads as to the …. For provisioning and upgrading configuration and updates Ansible has been used
- We use domain based redirection
- dicaple
- For the platform storarge we have chosen HDFS as the tecnhonolog It’s not have been deployed using the corresponding mesos framework for avoiding possible losses
- We have the incoming traffic through the edge nodes where the https is terminated along the
- Well its time to focus on the platform architecture
- Here in this figure an overview of theh platform can be seen. Different Supervisots are connected to the cloud
- Well Spark is used as our analytical / ingestion tool. Wee use it for both batch and realtime data using the Lambda architecture. Data ingestion and storage drom the, uses the cluster resources for computing the queriies and we encode the data using
- Hdfs backd Recent data table Real time algorithms Data is saved to the stagin HDFS directory for later compaction Uncompacted partitioned parque (date / by supervisor)
- The final component of the platform is the one called the compactor. This component is respnsible for HDFS file compaction and is executed by the Chronos framework . Chronos is a Mesos framework rresponsible of batch tasks. The compactor uses Kite (kaite) project for this task where we have two partitioned folders. The stagin and the compacted one
- For the final part of the presentation, some conclusions and future work will be presented
- Let’s start with some conclusions(conclusions), Remote monitoring has already been valuable for the clients in order to provide remote maintenance. The collected data is available 24x7. The developed platform is easily deployable on any cloud provider, it has a efficient resource usage mixing big data workloads with custom services. Analytics can be done using standard SQL over recent and cold data .
- Finally I would like to depict some future work we would like to address: - As Mesospehere has open sourced DC/OS we would like to “move” in order to have commercial support - We would also like to explore the oversubscription resources on mesos in order to improve the usage of the resources. - Evaluate Cassandra as backend storage to avoid the HDFS compaction problem - More ULMA Handling software on the platform, hopefully stateful services or databases whether stateful apps system matures.
- We have come to the end of the presentation. I’d just like to thank(thenk) you for listening and would be pleased to take your comments and questions now. And if you want to get in touch just drop me an email.
- After going throughout the requisites let’s review the historically infrastructure archetipes. First of all we have the On-premises, later on Proprietary Hyperscale, after the cloud arisen and finally…..