How to become an big data rockstar in 15 minutes - Akmal Chaudhri
•Download as PPTX, PDF•
0 likes•96 views
How to become an big data rockstar in 15 minutes - Akmal Chaudhri
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to How to become an big data rockstar in 15 minutes - Akmal Chaudhri
Similar to How to become an big data rockstar in 15 minutes - Akmal Chaudhri (20)
More from Dataconomy Media
More from Dataconomy Media (20)
Recently uploaded
Recently uploaded (20)
How to become an big data rockstar in 15 minutes - Akmal Chaudhri
- 1. How to become a Big Data Rockstar in 15 minutes! Akmal Chaudhri GridGain Systems © 2018 GridGain Systems, Inc.
- 2. Agenda • Turbocharging SQL queries • Sharing data and state across Spark jobs • Using Ignite’s ML library for Data Science • Easing DevOps dilemmas with Kubernetes © 2018 GridGain Systems, Inc.
- 3. Apache Ignite Database and Caching Platform Memory-Centric Storage Ignite Native Persistence (Flash, SSD, Intel 3D XPoint) Third-Party Persistence (RDBMS, HDFS, NoSQL) SQL Transactions Compute Services MLStreamingKey/Value IoTFinancial Services Pharma & Healthcare E-CommerceTravel & Logistics Telco © 2018 GridGain Systems, Inc.
- 4. • Database caching use case • No “rip and replace” performance boost • Automatic read-through and write-through • ANSI-99 SQL Turbocharging SQL Queries © 2018 GridGain Systems, Inc.
- 5. Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job Spark Job Spark Worker Spark Job Spark Job Ignite Node Ignite Node Ignite Node Share state and data among Spark jobs No data movement Boost DataFrame and SQL Performance SQL on top of RDDs In-place query execution © 2018 GridGain Systems, Inc.
- 6. Machine Learning K-Means Regressions Decision Trees R C++ Python Java Server Node Server NodeServer Node Distributed Core Algebra DURABLE MEMORY DURABLE MEMORY DURABLE MEMORY Scala REST Random Forest Distributed Algorithms Dense and Sparse Algebra Large Scale Parallelization Multi-Language Support Dense and Sparse Algebra No ETL © 2018 GridGain Systems, Inc.
- 7. Top 5 by Commits 1. Hadoop 2. Ambari 3. Camel 4. Ignite 5. Beam Top 5 Developer Mailing Lists 1. Ignite 2. Kafka 3. Tomcat 4. Beam 5. James Over 1M downloads per year Top 5 User Mailing Lists 1. Lucene/Solr 2. Ignite 3. Flink 4. Kafka 5. Cassandra © 2018 GridGain Systems, Inc. Among Top 5 Apache Projects
- 8. Any Questions? Thank you for joining us. Follow the conversation. http://ignite.apache.org #apacheignite © 2018 GridGain Systems, Inc.
Editor's Notes
- The Apache Ignite Platform Apache Ignite is a memory-centric data platform that is used to build fast, scalable & resilient solutions. At the heart of the Apache Ignite platform lies a distributed memory-centric data storage platform with ACID semantics, and powerful processing APIs including SQL, Compute, Key/Value and transactions. Built with a memory-centric approach, this enables Apache Ignite to leverage memory for high throughput and low latency whilst utilizing local disk or SSD to provide durability and fast recovery. The main difference between the memory-centric approach and the traditional disk-centric approach is that the memory is treated as a fully functional storage, not just as a caching layer, like most databases do. For example, Apache Ignite can function in a pure in-memory mode, in which case it can be treated as an In-Memory Database (IMDB) and In-Memory Data Grid (IMDG) in one. On the other hand, when persistence is turned on, Ignite begins to function as a memory-centric system where most of the processing happens in memory, but the data and indexes get persisted to disk. The main difference here from the traditional disk-centric RDBMS or NoSQL system is that Ignite is strongly consistent, horizontally scalable, and supports both SQL and key-value processing APIs. Apache Ignite platform can be integrated with third-party databases and external storage mediums and can be deployed on any infrastructure. It provides linear scalability, built-in fault tolerance, comprehensive security and auditing alongside advanced monitoring & management. The Apache Ignite platform caters for a range of use cases including: Core banking services, Real-time product pricing, reconciliation and risk calculation engines, analytics and machine learning.
- Apache Ignite provides an implementation of Spark RDD abstraction which allows to easily share state in memory across Spark jobs. The main difference between native Spark RDD and IgniteRDD is that Ignite RDD provides a shared in-memory view on data across different Spark jobs, workers, or applications, while native Spark RDD cannot be seen by other Spark jobs or applications. The way IgniteRDD is implemented is as a view over a distributed Ignite cache, which may be deployed either within the Spark job executing process, or on a Spark worker, or in its own cluster. This means that depending on the chosen deployment mode the shared state may either exist only during the lifespan of a Spark application (embedded mode), or it may out-survive the Spark application (standalone mode) in which case the state can be shared across multiple Spark applications.
- DEMO: run several ML samples from the standard distribution. Main benefits: No ETL – online “in place” ML In-memory speed & scale Large scale parallelization Optimized ML/DL algorithms Last-mile GPU optimization The rationale for building ML Grid is quite simple. Many users employ Ignite as the central high-performance storage and processing systems for various data sets. If they wanted to perform ML or Deep Learning (DL) on these data sets (i.e. training sets or model inference) they had to ETL them first into some other systems like Apache Mahout or Apache Spark. The roadmap for ML Grid is to start with core algebra implementation based on Ignite co-located distributed processing. The initial version was released with Ignite 2.0. Future releases will introduce custom DSLs for Python, R and Scala, growing collection of optimized ML algorithms such as Linear and Logistic Regression, Decision Tree/Random Forest, SVM, Naive Bayes, as well support for Ignite-optimized Neural Networks and integration with TensorFlow. Current beta version of Apache Ignite Machine Learning Grid (ML Grid) supports a distributed machine learning library built on top of highly optimized and scalable Apache Ignite platform and implements local and distributed vector and matrix algebra operations as well as distributed versions of widely used algorithms.
- [1] http://globenewswire.com/news-release/2018/07/09/1534470/0/en/The-Apache-Software-Foundation-Announces-Annual-Report-for-2018-Fiscal-Year.html [2] https://blogs.apache.org/foundation/entry/apache-in-2017-by-the