This document discusses Apache Bigtop and how it can accelerate Apache Hadoop and related projects using Apache Ignite. Bigtop provides a framework for integrating, deploying, and validating Hadoop ecosystem components on commodity hardware. It also discusses how Ignite provides an in-memory data fabric that can be used as a data exchange medium across Hadoop components without leaving memory, accelerating workloads. The document demonstrates how Ignite can accelerate MapReduce, Spark, and other workloads through its in-memory capabilities and integration with Bigtop.

1. © 2014 GridGain Systems, Inc.
KONSTANTIN BOUDNIK
VP Open Source Development, WANdisco
Member of the Apache Software Foundation
Accelerating the Hadoop data stack with
Apache Ignite™, Spark and Bigtop
http://ignite.apache.org
NIKITA IVANOV
CTO, GridGain Systems
Member of the PMC for Apache Ignite
http://bigtop.apache.org

2. Apache Bigtop primer
• A project, environment, and a philosophy to:
• Define and create software stacks (think Debian)
• Deploy and validate actual software in the real world
• Configuration management
• Guarantees of consistency and compatibility
• Empirical vs. Rational
• don't rely on someone's hearsay
• don't assume an environment: control it

3. Apache Bigdata stack
• Bigtop is the cutting edge of Apache Bigdata stack
• Delivers:
• A ready data processing stack
• Dev. env. for anyone to create their own
• Framework for easy integration/deployment/validation
• “It works on my laptop” isn't cool anymore
• 0.x release series was focused on Hadoop ecosystem

4. Let's get serious about IMC
• Bigtop boards more & more IMC(-like) components
• Provides transitional tech for legacy MR-based users
• HDFS acceleration
• MR acceleration
• Uses RAM as inter-component data media
• Crossing component boundaries w/o leaving RAM
• Advanced clustering and service models

5. Connecting the stack
• Bigtop Data Fabric Core:
• Works with HDFS/RDBMS/MR/Hive/Hbase/Spark/Storm/SQL
• Cluster memory is a natural media to exchange data
• A usecase:
• Kafka --> Data Fabric --> HBase --> Data Fabric --> SQL querying --> Spark -
-> A service Singlethon --> Data Fabric --> RDBMS/FS

6. Connecting the ...

7. Live Demo
• Deploy Apache Ignite (incubating)
• Run MR Pi on YARN
• Run same MR Pi on Apache Ignite:
• Only client config needs to be changed
• Run MR Teragen
• Run MR Terasort on YARN vs Ignite IGFS
• Run Spark queries w/ & w/o Ignite caching

8. Results review
Workload Without Ignite With Ignite
MapReduce:
CPU-bound (Pi)
MapReduce
IO-bound:
(TeraGen/Sort)
Spark SQL

9. © 2014 GridGain Systems, Inc.
In-Memory Data Fabric
Strategic Approach to IMC
• Supports Applications of
various types and
languages
• Open Source – Apache 2.0
• Simple Java APIs
• 1 JAR Dependency
• High Performance & Scale
• Automatic Fault Tolerance
• Management/Monitoring
• Runs on Commodity Hardware
• Supports existing & new data sources
• No need to rip & replace

10. © 2014 GridGain Systems, Inc.
• Plug and Play installation
• 10x to 100x Acceleration
• In-Memory Native MapReduce
• In-Process Data Colocation
• IgniteFS In-Memory File System
• Read-Through from HDFS
• Write-Through to HDFS
• Sync and Async Persistence
In-Memory Hadoop Accelerator

11. © 2014 GridGain Systems, Inc.
In-Memory Hadoop Accelerator
• Zero Code Change
• In-Memory Native Performance
• Use existing MR code
• Use existing Pig/Hive queries
• No Name Node
• Eager Push Scheduling

12. © 2014 GridGain Systems, Inc.
Spark Integration – IGFS & Shared RDD

13. © 2014 GridGain Systems, Inc.
ANY QUESTIONS?
Thank you for joining us. Follow the conversation.
http://bigtop.apache.org http://ignite.apache.org
#apacheignite#apachebigtop
KONSTANTIN BOUDNIK
VP Open Source Development, WANdisco Member
of the Apache Software Foundation
NIKITA IVANOV
CTO, GridGain Systems
Member of the PMC for Apache Ignite