Surge openstack
•Download as PPTX, PDF•
0 likes•142 views
Komei Shimamura, Timothy Okwii, Johnu George, and Marc Solanas Tarre presented Surge, a service for deploying and managing real-time data processing pipelines on OpenStack. Surge supports Apache Kafka for distributed messaging and Apache Storm for real-time data processing. It allows 1-click deployment and scaling of pipelines and manages the processes through a web UI, making complex systems easier for users. The goal is to simplify running machine learning and analytics jobs on streaming data hosted on OpenStack.
![Komei Shimamura, Timothy Okwii, Johnu George, Marc Solanas Tarre
[kshimamu|tokwii|johnugeo|msolanas]@cisco.com
Real-Time Data Processing Systems as a Service
OpenStack Summit 2015 @ Vancouver
Surge](https://image.slidesharecdn.com/surge-openstack-150523052922-lva1-app6892/85/Surge-openstack-1-320.jpg)
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Surge openstack
Similar to Surge openstack (20)
Recently uploaded
Recently uploaded (20)
Surge openstack
- 1. Komei Shimamura, Timothy Okwii, Johnu George, Marc Solanas Tarre [kshimamu|tokwii|johnugeo|msolanas]@cisco.com Real-Time Data Processing Systems as a Service OpenStack Summit 2015 @ Vancouver Surge
- 2. OpenStack Summit 2015 @ Vancouver ‣ Deploying Data Processing Pipelines on top of OpenStack ‣ Apache Kafka and Apache Storm are currently supported. ‣ Managing the Pipeline after the Deployment ‣ It can be easily scaled up according to the server load and resources. ‣ All operation can be done from a web UI 2 The Goal of Surge Not everyone knows how to manage such systems. Let’s make it trivial.
- 3. OpenStack Summit 2015 @ Vancouver 3 Apache Kafka: Distributed Messaging Broker ‣ Handling Real time data streams ‣ High scalable distributed messaging broker ‣ Enabling the subscribing and publishing data ‣ Easily share data amongst our applications. PR BR CN thousand nodes produces the data thousand nodes subscribe the data PR: producer, BR: broker, CN: consumer handling high load streams like buffer
- 4. OpenStack Summit 2015 @ Vancouver 4 Apache Storm: Real Time Computing Service ‣ Distributed real-time computation system ‣ Enabling the real-time data processing ‣ Anomaly detection. ‣ Machine Learning Other Systems are going to be supported in the future
- 5. OpenStack Summit 2015 @ Vancouver
- 6. OpenStack Summit 2015 @ Vancouver ‣ Deploying Data Processing Pipelines on top of the OpenStack Cluster ‣ Pipeline: Apache Kafka and Apache Storm processes 6 What did we talk about? ‣ 1-click deployment and scaling. DATA OpenStack Node Apache Kafka Apache Storm Output Data Making it easy to run data processing jobs, such as Machine Learning operations with Cognitive
- 7. OpenStack Summit 2015 @ Vancouver ‣ Surge command-line version (UI not public): ‣ https://www.github.com/CiscoSystems/surge Where can you find this? Thank you! :) ‣ Cognitive Machine Learning as a Service ‣ https://www.github.com/CiscoSystems/cognitive ‣ https://github.com/stackforge/cognitive
Editor's Notes
- Helllo everyone, My name is Komei shimamura and I am a software developer in Cisco’s CTO Cloud department. I am part of the Cloud analytics team and our main goal is to analyze log and metrics data from OpenStack clusters, handle Openstack errors or optimizing and scheduling OpenStack resources based on data-driven decisions. Today I'll be talking about how we deploy a real-time data processing pipeline on OpenStack and I’ll show you an application of such a pipeline.
- Our project name is Surge, and it aims to deploy data processing pipelines on top of openstack clusters in a fully automated way. It is a stand-alone application which can be connected to any OpenStack cluster, as long as we have user credentials. We developed this app to solve a simple need: we needed to rapidly deploy, scale and destroy these kind of pipelines. Also, we needed to be able to change any configuration in runtime. The current version of Surge offers support for deploying and scaling Apache Kafka and Apache Storm as components for our pipeline, and will support for other open source systems such as spark in the future.
- Some of you might be wondering what Kafka and Storm are. Let me give you a brief overview and use cases. Kafka and Storm are two very powerful projects for managing real-time big data. They have a huge community actively developing them. Kafka is a highly-scalable distributed publisher-subscriber messaging system, which accepts data from a number of producers and immediately publishes it to thousands of consumers. This system can be used not only for sharing data but also as a buffer of high load streams. This system enables us to provide nice bus for all our systems and make it for them and any other system external to Surge to communicate in real-time.
- Apache Storm is the second system Surge currently supports. It enables us to process real-time data streams in a scalable and distributed way. Apache storm manages processes as tasks, which means if we have a couple of processes, each can be individually configured. You could use other opensource projects as a replacement for storm, for example, apache spark or apache samza. A command line version of Surge that lets you deploy to Virtualbox or OpenStack is currently available in Cisco’s github account. It is wrapper around vagrant that makes it easy to deploy multi-node systems. Please try it for yourselves and contribute!! For this demonstration, we’ll be using the UI version of Surge, which has not yet been OpenSourced, but it’s better for demos and to get an easy understanding.
- This is the Surge UI, which has the setting and pipeline pages. First, we need to connect Surge to our OpenStack Cluster so let’s fill out this form which has all the necessary information, such as user name, password, private key, and tenants, that Surge needs to connect to it. It also contains the VM specifications that Surge should use for our pipelines. Once you have filled this form, and activated it, you can launch any pipelines from this interface. So go to the Pipeline page, where you can create as many pipelines as you want. To create a pipeline, you need to write the pipeline name, that’s it. This name can be changed at any time. Next, let’s go to pipeline details. Here there are no components yet. We need to add some! let’s make make a kafka-storm pipeline as an example. To add a system to the pipeline, you need to click this button, and choose one from the list, enter the name you like, and fill out the other configurations like the number of kafka brokers. Then, let’s also add a storm cluster. you can do it in the same way as the other components as well. To create a storm cluster, we need to type the name and choose the number of storm supervisor and storm nimbus nodes. and then, we click the create button. now you can see the kafka, and storm nimbus and storm supervisors, so let’s deploy these components. To deploy the components we have defined, we just need to click the deploy button on the top of the left of this page. and waiting to deploy… yes, now all components are deployed perfectly. you can see the status changes from un-deployed to deployed. so after the deployment is completed, we can start using these services. One way is clicking the launch topology button. In Storm, a topology is basically coming from the compiling storm java file. or you can choose the cognitive dashboard to execute some machine learning algorithms or some data manipulations. In this case let’s go to cognitive as one example of the use case of such pipelines. Cognitive is the service which is providing machine learning algorithms and data manipulations via the UI. so to use Cognitive, you can simply create a user account and then you can enter the Cognitive dashboard. Cognitive has some of components in the left side, and you can choose any of them to do the data manipulation. As an example we can upload CSV file by clicking file upload component, and then you can find the new component on the cognitive dashboard. you can see the data through the UI and it also shows statistical information for given data as well. To do some manipulatinos, for example in this case, we can choose the mach formula component and it provides addition, substruction, division, and multiplication, so in this quick demonstration, we just choose the Addition and then choose the column, for example Fare, and then choose the value which will be added to the Fare column. and then you can find the math formula component on the right side of dashboard, and you can connect these component by drag and drop. Finally, you can Run the experiment, and you can get the results which represent all the operations you have chosen.
- So in this presentation, I was talking about how we quickly deploy a number of Pipelines on OpenStack. We are using apache kafka, and apache storm as a part of our systems, which provide you with a powerful real-time data processing pipeline. In addition, I showed Cognitive as one of use case of such a pipeline and now an OpenSource project part of StackForge.
- All source code is avilable under Cisco system github repository, so you can get and contribute to these 2 projects following these URLs. Thank you for attending my presentation, I hope you enjoed it. Thank you.