Download to read offline
More Related Content
![[Spark Summit 2017 NA] Apache Spark on Kubernetes](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkonkubernetespublic1-170929072840-thumbnail.jpg?width=640&height=640&fit=bounds)
![5G Explained! A High Level Overview [Introduction]](https://cdn.slidesharecdn.com/ss_thumbnails/5gexplainedahighleveloverview-260119165306-cc137a3e-thumbnail.jpg?width=640&height=640&fit=bounds)
Serverless spark
- 1. Serverless Spark Rachit Arora,Lead Software Architect, IBM Cloud
- 2. Spark Unified, open source,parallel, data processing framework for Big Data Analytics Spark Core Engine Yarn Mesos Standalon e Scheduler Kubernete s Spark SQL Interactive Queries Spark Streaming Stream processing Spark MLlib Machine Learning GraphX Graph Computation
- 3. Typical Bigdata Application Secure Catalogand Search Ingest & Store Prepare Analyze Visualize Date Engineer Date Scientist Application Developer
- 4. Let look into roleof Data Scientist • I want to run my analytics jobs • Social media analytics • Text analytics (Structure and Unstructured) • I want to run queries on demand • I want to run R scripts • I want to submit Spark jobs • I want to view History Server Logs of my application • I want to View Daemon logs • I want to write Notebooks
- 5. Evolution of SparkAnalytics On Prem Install • Acquire Hardware • Prepare Machine • Install Spark • Retry • Apply patches • security • Upgrades • Scale • High availability Virtualization • Prepare Vm Imaging Solution • Network Management • High Avilability • Patches • Scale Managed • Configure Cluster • Customize • Scale • Pay even if idle Serverless • Run analytics
- 6. IBM Watson Studio SparkEnvironments
- 8. What Kubernetes Bring in? Kubernetes isan open-source system for automating deployment, scaling, and management of containerized applications. •It Manages Containers for me •It Manages High availability •It Provides me flexibility to choose resource I WANT and Persistence I want Kubernetes – Lots of addon services: third-party logging, monitoring, and security tools Reduced operational costs Improved infrastructure utilization
- 9. Why Run Sparkon Kubernetes • Are you using data analytical pipeline which is containerized? • Resource sharing is better optimized • Leveraging Kubernetes ecosystem • Kubernetes community support
- 10. Serverless Spark Option 1:Multitenant Spark Cluster •Performance is not consistent •Wrong Library in class path impact •Secure and Complaint •ISO •HIPAA •Stability issue •Single point of failure •Maintenance and Upgrades
- 11. Serverless Spark Option 2:Function as a service • Single Node Cluster – Or No Cluster at all • Spark local mode • all in one Image • Resource Limitations • Design Limitations
- 12. Serverless Spark Option 3:Vanilla Containers • Repeatable • Application Portability • Faster Development Cycle • Reduced dev-ops load • Improved Infrastructure Utilization
- 13. Serverless Spark Option 4:-Kubernetes with Standalone cluster manager
- 14. Serverless Spark Option 5:-Kubernetes with Kubernetes cluster manager
- 15. Serverless Spark Option 6:-Kubernetes with Kubernetes cluster manager + Spark Operator
- 16. References • IBM WatsonStudio https://datascience.ibm.com • Spark Environments • IBM Watson https://www.ibm.com/analytics/us/en/watson- data-platform/tutorial/ • Analytics Engine https://www.ibm.com/cloud/analytics-engine • Apache Spark • Apache Arrow • Alluxio • Kubernetes Scheduler Design & Discussion • Kuberenetes Clusters on IBM Cloud
- 17. Thank you Rachit Arora rachitar@in.ibm.com Twitter@rachit1arora
Editor's Notes
- #3 Spark is an open source, scalable, massively parallel, in-memory execution engine for analytics applications. Think of it as an in-memory layer that sits above multiple data stores, where data can be loaded into memory and analyzed in parallel across a cluster. Spark Core: The foundation of Spark that lot of libraires for scheduling and basic I/O Spark offers over 100s of high-level operators that make it easy to build parallel apps. Spark also includes prebuilt machine-learning algorithms and graph analysis algorithms that are especially written to execute in parallel and in memory. It also supports interactive SQL processing of queries and real-time streaming analytics. As a result, you can write analytics applications in programming languages such as Java, Python, R and Scala. You can run Spark using its standalone cluster mode, on Cloud, on Hadoop YARN, on Apache Mesos, or on Kubernetes. Access data in HDFS, Cassandra, HBase, Hive, Object Store, and any Hadoop data source.
- #4 Prepare Even though you have the right data, it may not be in the right format or structure for analysis. That’s where data preparation comes in. Data engineers need to bring raw data into one interface from wherever it lives – on premises, in the cloud or on your desktop – where it can then be shaped, transformed, explored, and prepared for analysis. Data scientist: Primarily responsible for building predictive analytic models and building insights. He will analyze data that’s been cataloged and prepared by the data engineer using machine learning tools like Watson Machine Learning. He will build applications using Jupyter Notebooks, RStudio After the data scientist shares his Analytical outputs , Application developer can build APPs like a cognitive chatbot. As the chatbot engages with customers, it will continuously improve its knowledge and help uncover new insights.
- #6 As a data scientist what I was required to do On Prem to Virtuliation as demand increased in my organization for the sevrice I decided to move to virtualized VM to handle many request on demand but there still pain was more Then I decided to try services being offereed on cloud like EMR and IBM Analytics Engine or Microsoft Insights etce but there I need to order cluster sand configure them to suit my work loads Keep them running even when I do not want to use them Cover what is takes to install a hadoop/spark cluster
- #9 Kubernetes is a portable, extensible open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.
- #11 YARN is used for many production workloads and can be used to run any application. Spark treats YARN as a container management system to request with defined resource once spark acquire container it builds RPC based communication between container to run driver and executors.
- #12 YARN is used for many production workloads and can be used to run any application. Spark treats YARN as a container management system to request with defined resource once spark acquire container it builds RPC based communication between container to run driver and executors.
- #13 YARN is used for many production workloads and can be used to run any application. Spark treats YARN as a container management system to request with defined resource once spark acquire container it builds RPC based communication between container to run driver and executors.
- #14 YARN is used for many production workloads and can be used to run any application. Spark treats YARN as a container management system to request with defined resource once spark acquire container it builds RPC based communication between container to run driver and executors.
- #15 YARN is used for many production workloads and can be used to run any application. Spark treats YARN as a container management system to request with defined resource once spark acquire container it builds RPC based communication between container to run driver and executors.
- #17 IBM Watson brings together data management, data policies, data preparation, and analysis capabilities into a common framework. You can index, discover, control, and share data with Watson Knowledge Catalog, refine and prepare the data with Data Refinery, then organize resources to analyze the same data with Watson Studio. The IBM Watson apps are fully integrated to use the same user interface and framework. You can pick whichever apps and tools you need for your organization. Watson Studio (Watson Studio) provides you with the environment and tools to solve your business problems by collaboratively analyzing data What is Analytics Engine? You can use AE to Build and deploy clusters within minutes with simplified user experience, scalability, and reliability. You Custom configure the environment and Scale on demand.