Apache Ignite is an in-memory platform that can accelerate Hadoop and Spark workloads by storing data in memory. It provides a distributed in-memory file system (IGNITE) that can be used as a secondary storage layer for Hadoop. For Spark, Ignite allows sharing RDDs across jobs by storing them in an Ignite cache, avoiding the need to write to disk between jobs. The IgniteContext class provides the main entry point for integrating Spark and Ignite, allowing Spark jobs to read from and write RDD data directly to Ignite caches.

2. Akmal Chaudhri, GridGain Systems
Boost Hadoop and Spark with
in-memory technologies

3. Agenda
• Introduction to Apache Ignite
• Hadoop Acceleration
• Spark Acceleration
• Demos
• Q&A
Big Data Spain 2017

4. Apache Ignite in one slide
• Memory-centric platform
– that is strongly consistent
– and highly-available
– with powerful SQL
– key-value and processing
APIs
• Designed for
– Performance
– Scalability
Big Data Spain 2017

5. Apache Ignite
• Data source agnostic
• Fully fledged compute engine and durable storage
• OLAP and OLTP
• Fully ACID transactions across memory and disk
• In-memory SQL support
• Early ML libraries
• Growing community
Big Data Spain 2017

6. Hadoop Acceleration
• In-memory Hadoop Execution
• Alternative job tracker
– Faster MapReduce
• Built on Ignite File System (IGFS)
• Secondary File System
– Read-through and Write-through
Big Data Spain 2017

7. Ignite In-Memory File System
• Distributed in-memory
file system
• Implements HDFS
API
• Can be transparently
plugged into Hadoop
or Spark deployments
Big Data Spain 2017

8. MapReduce
Big Data Spain 2017

9. MapReduce
• Parallelize processing of data in HDFS
• Eliminate Hadoop JobTracker and TaskTracker
overhead
• Low-Latency distributed processing
• Minimal configuration change
Big Data Spain 2017

10. Spark Acceleration
• Long running applications
– Passing state between jobs
• Disk File System
– Convert RDDs to disk files and back
• Share RDDs in-memory
– Native Spark API
– Native Spark transformations
Big Data Spain 2017

11. Ignite for Spark
• Spark RDD abstraction
• Shared in-memory view
on data across different
Spark jobs, workers or
applications
• Implemented as a view
over a distributed Ignite
cache
Big Data Spain 2017

12. IgniteContext
• Main entry-point to Spark-Ignite integration
• SparkContext plus either one of
– IgniteConfiguration()
– Path to XML configuration file
• Optional Boolean client argument
– true => Shared deployment
– false => Embedded deployment
Big Data Spain 2017

13. IgniteContext examples
Big Data Spain 2017
valigniteContext= new IgniteContext(sparkContext,
()= > new IgniteConfiguration())
valigniteContext= new IgniteContext(sparkContext,
"exam ples/config/spark/exam ple-shared-rdd.xm l")

14. IgniteRDD
• Implementation of Spark RDD representing a live
view of an Ignite cache
• Mutable (unlike native RDDs)
– All changes in Ignite cache will be visible to RDD users
immediately
• Provides partitioning information to Spark executor
• Provides affinity information to Spark so that RDD
computations can use data locality
Big Data Spain 2017

15. Write to Ignite
• Ignite caches operate on key-value pairs
• Spark tuple RDD for key-value pairs and
savePairs method
– RDD partitioning, store values in parallel if possible
• Value-only RDD and saveValues method
– IgniteRDD generates a unique affinity-local key for
each value stored into the cache
Big Data Spain 2017

16. Write code example
Big Data Spain 2017
valconf= new SparkConf().setAppNam e("SparkIgniteW riter")
valsc = new SparkContext(conf)
valic = new IgniteContext(sc,
"exam ples/config/spark/exam ple-shared-rdd.xm l")
valsharedRD D :IgniteRD D [Int,Int]= ic.from Cache("sharedRD D ")
sharedRD D .savePairs(sc.parallelize(1 to 100000,10)
.m ap(i= > (i,i)))

17. Read from Ignite
• IgniteRDD is a live view of an Ignite cache
– No need to explicitly load data to Spark application
from Ignite
– All RDD methods are available to use right away after
an instance of IgniteRDD is created
Big Data Spain 2017

18. Read code example
Big Data Spain 2017
valconf= new SparkConf().setAppNam e("SparkIgniteReader")
valsc = new SparkContext(conf)
valic = new IgniteContext(sc,
"exam ples/config/spark/exam ple-shared-rdd.xm l")
valsharedRD D :IgniteRD D [Int,Int]= ic.from Cache("sharedRD D ")
valgreaterThanFiftyThousand = sharedRD D .filter(_._2 > 50000)
println("The countis "+ greaterThanFiftyThousand.count())

19. Demos
Big Data Spain 2017

20. Any Questions?
Thank you for joining us. Follow the conversation.
http://ignite.apache.org
Big Data Spain 2017