SlideShare a Scribd company logo
1 of 17
Loading data into Apache Ignite
Stephen Darlington
01 May 2019
2019 © GridGain Systems
2019 © GridGain Systems GridGain Company Confidential
Apache Ignite In-Memory Computing Platform
Mainframe NoSQL HadoopIgnite Persistence
Persistent Layer
RDBMS
Machine and Deep Learning
EventsStreamingMessagingTransactionsSQLKey-Value
Service GridCompute Grid
Application Layer
Web SaaS SocialMobile IoT
In-Memory Data Store
2019 © GridGain Systems GridGain Company Confidential
How do I load data?
This Photo by
Unknown
Author is
licensed under
CC BY-SA
2019 © GridGain Systems GridGain Company Confidential
Official answer
1. Open your IDE
2. Create a project
3. Edit pom.xml to include Apache Ignite libraries
4. Create a new class
5. Code to open and parse input file
6. Boilerplate Ignite cluster code
7. IgniteDataStreamer code
8. Debug
9. Edit
10. Debug
11. Edit
12. Debug
13. Run
14. Play with resulting data
2019 © GridGain Systems GridGain Company Confidential2019 © GridGain Systems
There must be an easier way?
8
This Photo by Unknown Author is licensed under CC BY-NC-
ND
2019 © GridGain Systems GridGain Company Confidential
Apache Ignite In-Memory Computing Platform
Mainframe NoSQL HadoopIgnite Persistence
Persistent Layer
RDBMS
Machine and Deep Learning
EventsStreamingMessagingTransactionsKey-Value
Service GridCompute Grid
Application Layer
Web SaaS SocialMobile IoT
In-Memory Data Store
SQL
2019 © GridGain Systems GridGain Company Confidential
Using SQL
2019 © GridGain Systems GridGain Company Confidential
But it gets complicated…
2019 © GridGain Systems GridGain Company Confidential
SQL Streaming
Apache Ignite In-Memory Computing Platform
Mainframe NoSQL HadoopIgnite Persistence
Persistent Layer
RDBMS
Machine and Deep Learning
EventsMessagingTransactions
Service GridCompute Grid
Application Layer
Web SaaS SocialMobile IoT
In-Memory Data Store
Key-Value
2019 © GridGain Systems GridGain Company Confidential
Using Python
2019 © GridGain Systems GridGain Company Confidential
SQL
Apache Ignite In-Memory Computing Platform
Mainframe NoSQL HadoopIgnite Persistence
Persistent Layer
RDBMS
Machine and Deep Learning
EventsMessagingTransactionsKey-Value
Service GridCompute Grid
Application Layer
Web SaaS SocialMobile IoT
In-Memory Data Store
Streaming
2019 © GridGain Systems GridGain Company Confidential
Using Apache Spark
2019 © GridGain Systems GridGain Company Confidential
Using Apache Spark
2019 © GridGain Systems GridGain Company Confidential2019 © GridGain Systems
What did we learn?
17
• Many options
– Python, Spark, SQL
– Scala
– Groovy
– Node.js
• No one “best” answer
• REPLs are awesome
– …and can be used for a lot more than just loading data
2019 © GridGain Systems GridGain Company Confidential
Resources
• Apache Ignite documentation
– https://apacheignite.readme.io/docs
– https://ignite.apache.org
• Blog
– Loading Data into Ignite. https://link.medium.com/66dzsrWw4V
– Python, part 1. https://link.medium.com/CUjDnzBQcW
– Python, part 2. https://link.medium.com/3dWH1oDQcW
2019 © GridGain Systems GridGain Company Confidential
And finally…
• Get a free ticket to the In-Memory Computing Summit Europe 2019 (June
3-4) by completing this survey:
– http://bit.ly/IMCSeu2019
• More information here:
– https://www.imcsummit.org/2019/eu/
2019 © GridGain Systems GridGain Company Confidential2019 © GridGain Systems
Thank you
20
Stephen Darlington
Senior Consultant
GridGain Systems
@sdarlington

More Related Content

What's hot

Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
DataWorks Summit
 

What's hot (20)

Microservices Architectures With Apache Ignite
Microservices Architectures With Apache IgniteMicroservices Architectures With Apache Ignite
Microservices Architectures With Apache Ignite
 
In-Memory Computing Essentials for Software Engineers
In-Memory Computing Essentials for Software EngineersIn-Memory Computing Essentials for Software Engineers
In-Memory Computing Essentials for Software Engineers
 
In-Memory Computing Essentials for Architects and Engineers
In-Memory Computing Essentials for Architects and EngineersIn-Memory Computing Essentials for Architects and Engineers
In-Memory Computing Essentials for Architects and Engineers
 
In-Memory Computing Essentials
In-Memory Computing EssentialsIn-Memory Computing Essentials
In-Memory Computing Essentials
 
Big analytics meetup - Extended Jupyter Kernel Gateway
Big analytics meetup - Extended Jupyter Kernel GatewayBig analytics meetup - Extended Jupyter Kernel Gateway
Big analytics meetup - Extended Jupyter Kernel Gateway
 
Best Practices for Using Alluxio with Apache Spark with Gene Pang
Best Practices for Using Alluxio with Apache Spark with Gene PangBest Practices for Using Alluxio with Apache Spark with Gene Pang
Best Practices for Using Alluxio with Apache Spark with Gene Pang
 
Best Practices for Using Alluxio with Apache Spark with Cheng Chang and Haoyu...
Best Practices for Using Alluxio with Apache Spark with Cheng Chang and Haoyu...Best Practices for Using Alluxio with Apache Spark with Cheng Chang and Haoyu...
Best Practices for Using Alluxio with Apache Spark with Cheng Chang and Haoyu...
 
Securing Spark Applications by Kostas Sakellis and Marcelo Vanzin
Securing Spark Applications by Kostas Sakellis and Marcelo VanzinSecuring Spark Applications by Kostas Sakellis and Marcelo Vanzin
Securing Spark Applications by Kostas Sakellis and Marcelo Vanzin
 
Hadoop Everywhere & Cloudbreak
Hadoop Everywhere & CloudbreakHadoop Everywhere & Cloudbreak
Hadoop Everywhere & Cloudbreak
 
Hadoop on Docker
Hadoop on DockerHadoop on Docker
Hadoop on Docker
 
Accelerate Your Big Data Analytics Efforts with SAS and Hadoop
Accelerate Your Big Data Analytics Efforts with SAS and HadoopAccelerate Your Big Data Analytics Efforts with SAS and Hadoop
Accelerate Your Big Data Analytics Efforts with SAS and Hadoop
 
Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfac...
 Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfac... Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfac...
Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfac...
 
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
 
Camel Riders in the Cloud
Camel Riders in the CloudCamel Riders in the Cloud
Camel Riders in the Cloud
 
Big data on google cloud
Big data on google cloudBig data on google cloud
Big data on google cloud
 
Bringing Real-Time to the Enterprise with Hortonworks DataFlow
Bringing Real-Time to the Enterprise with Hortonworks DataFlowBringing Real-Time to the Enterprise with Hortonworks DataFlow
Bringing Real-Time to the Enterprise with Hortonworks DataFlow
 
Deep Learning to Big Data Analytics on Apache Spark Using BigDL with Xianyan ...
Deep Learning to Big Data Analytics on Apache Spark Using BigDL with Xianyan ...Deep Learning to Big Data Analytics on Apache Spark Using BigDL with Xianyan ...
Deep Learning to Big Data Analytics on Apache Spark Using BigDL with Xianyan ...
 
Unlock Hadoop Success with Cloudera Navigator Optimizer
Unlock Hadoop Success with Cloudera Navigator OptimizerUnlock Hadoop Success with Cloudera Navigator Optimizer
Unlock Hadoop Success with Cloudera Navigator Optimizer
 
Edge to AI: Analytics from Edge to Cloud with Efficient Movement of Machine ...
Edge to AI:  Analytics from Edge to Cloud with Efficient Movement of Machine ...Edge to AI:  Analytics from Edge to Cloud with Efficient Movement of Machine ...
Edge to AI: Analytics from Edge to Cloud with Efficient Movement of Machine ...
 
What's new with Azure Sql Database
What's new with Azure Sql DatabaseWhat's new with Azure Sql Database
What's new with Azure Sql Database
 

Similar to Loading data into Apache Ignite

Similar to Loading data into Apache Ignite (20)

Apache Spark and Apache Ignite: Where Fast Data Meets the IoT
Apache Spark and Apache Ignite: Where Fast Data Meets the IoTApache Spark and Apache Ignite: Where Fast Data Meets the IoT
Apache Spark and Apache Ignite: Where Fast Data Meets the IoT
 
Transforming Enterprise IT - Virtual Transformation Day Feb 2019
Transforming Enterprise IT - Virtual Transformation Day Feb 2019Transforming Enterprise IT - Virtual Transformation Day Feb 2019
Transforming Enterprise IT - Virtual Transformation Day Feb 2019
 
On Cloud Nine: How to be happy migrating your in-memory computing platform to...
On Cloud Nine: How to be happy migrating your in-memory computing platform to...On Cloud Nine: How to be happy migrating your in-memory computing platform to...
On Cloud Nine: How to be happy migrating your in-memory computing platform to...
 
Keynote_Disney_UnleashthePossibilitiesAvoidthePitfallswithCloud
Keynote_Disney_UnleashthePossibilitiesAvoidthePitfallswithCloudKeynote_Disney_UnleashthePossibilitiesAvoidthePitfallswithCloud
Keynote_Disney_UnleashthePossibilitiesAvoidthePitfallswithCloud
 
IT Modernization in Practice
IT Modernization in PracticeIT Modernization in Practice
IT Modernization in Practice
 
How we broke Apache Ignite by adding persistence, by Stephen Darlington (Grid...
How we broke Apache Ignite by adding persistence, by Stephen Darlington (Grid...How we broke Apache Ignite by adding persistence, by Stephen Darlington (Grid...
How we broke Apache Ignite by adding persistence, by Stephen Darlington (Grid...
 
How we broke Apache Ignite by adding persistence
How we broke Apache Ignite by adding persistenceHow we broke Apache Ignite by adding persistence
How we broke Apache Ignite by adding persistence
 
Libera la potenza del Machine Learning
Libera la potenza del Machine LearningLibera la potenza del Machine Learning
Libera la potenza del Machine Learning
 
Gimel at Dataworks Summit San Jose 2018
Gimel at Dataworks Summit San Jose 2018Gimel at Dataworks Summit San Jose 2018
Gimel at Dataworks Summit San Jose 2018
 
Dataworks | 2018-06-20 | Gimel data platform
Dataworks | 2018-06-20 | Gimel data platformDataworks | 2018-06-20 | Gimel data platform
Dataworks | 2018-06-20 | Gimel data platform
 
Introduction to pyspark new
Introduction to pyspark newIntroduction to pyspark new
Introduction to pyspark new
 
Production-Ready Environments for Kubernetes (CON307-S) - AWS re:Invent 2018
Production-Ready Environments for Kubernetes (CON307-S) - AWS re:Invent 2018Production-Ready Environments for Kubernetes (CON307-S) - AWS re:Invent 2018
Production-Ready Environments for Kubernetes (CON307-S) - AWS re:Invent 2018
 
Enabling Deep Learning in IoT Applications with Apache MXNet
Enabling Deep Learning in IoT Applications with Apache MXNetEnabling Deep Learning in IoT Applications with Apache MXNet
Enabling Deep Learning in IoT Applications with Apache MXNet
 
QCon 2018 | Gimel | PayPal's Analytic Platform
QCon 2018 | Gimel | PayPal's Analytic PlatformQCon 2018 | Gimel | PayPal's Analytic Platform
QCon 2018 | Gimel | PayPal's Analytic Platform
 
Transforming Enterprise IT - Transformation Day Montreal 2018
Transforming Enterprise IT - Transformation Day Montreal 2018Transforming Enterprise IT - Transformation Day Montreal 2018
Transforming Enterprise IT - Transformation Day Montreal 2018
 
ML Inference at the Edge
ML Inference at the EdgeML Inference at the Edge
ML Inference at the Edge
 
Big Data LDN 2018: LESSONS LEARNED FROM DEPLOYING REAL-WORLD AI SYSTEMS
Big Data LDN 2018: LESSONS LEARNED FROM DEPLOYING REAL-WORLD AI SYSTEMSBig Data LDN 2018: LESSONS LEARNED FROM DEPLOYING REAL-WORLD AI SYSTEMS
Big Data LDN 2018: LESSONS LEARNED FROM DEPLOYING REAL-WORLD AI SYSTEMS
 
Making the Most of Data in Multiple Data Sources (with Virtual Data Lakes)
Making the Most of Data in Multiple Data Sources (with Virtual Data Lakes)Making the Most of Data in Multiple Data Sources (with Virtual Data Lakes)
Making the Most of Data in Multiple Data Sources (with Virtual Data Lakes)
 
FSM integration with SAP
FSM integration with SAPFSM integration with SAP
FSM integration with SAP
 
NA Adabas & Natural User Group Meeting April 2023
NA Adabas & Natural User Group Meeting April 2023NA Adabas & Natural User Group Meeting April 2023
NA Adabas & Natural User Group Meeting April 2023
 

Recently uploaded

➥🔝 7737669865 🔝▻ Ongole Call-girls in Women Seeking Men 🔝Ongole🔝 Escorts S...
➥🔝 7737669865 🔝▻ Ongole Call-girls in Women Seeking Men  🔝Ongole🔝   Escorts S...➥🔝 7737669865 🔝▻ Ongole Call-girls in Women Seeking Men  🔝Ongole🔝   Escorts S...
➥🔝 7737669865 🔝▻ Ongole Call-girls in Women Seeking Men 🔝Ongole🔝 Escorts S...
amitlee9823
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
amitlee9823
 
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service BangaloreCall Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
amitlee9823
 
Just Call Vip call girls roorkee Escorts ☎️9352988975 Two shot with one girl ...
Just Call Vip call girls roorkee Escorts ☎️9352988975 Two shot with one girl ...Just Call Vip call girls roorkee Escorts ☎️9352988975 Two shot with one girl ...
Just Call Vip call girls roorkee Escorts ☎️9352988975 Two shot with one girl ...
gajnagarg
 
➥🔝 7737669865 🔝▻ malwa Call-girls in Women Seeking Men 🔝malwa🔝 Escorts Ser...
➥🔝 7737669865 🔝▻ malwa Call-girls in Women Seeking Men  🔝malwa🔝   Escorts Ser...➥🔝 7737669865 🔝▻ malwa Call-girls in Women Seeking Men  🔝malwa🔝   Escorts Ser...
➥🔝 7737669865 🔝▻ malwa Call-girls in Women Seeking Men 🔝malwa🔝 Escorts Ser...
amitlee9823
 
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
amitlee9823
 
Call Girls In Shivaji Nagar ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Shivaji Nagar ☎ 7737669865 🥵 Book Your One night StandCall Girls In Shivaji Nagar ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Shivaji Nagar ☎ 7737669865 🥵 Book Your One night Stand
amitlee9823
 
Just Call Vip call girls Mysore Escorts ☎️9352988975 Two shot with one girl (...
Just Call Vip call girls Mysore Escorts ☎️9352988975 Two shot with one girl (...Just Call Vip call girls Mysore Escorts ☎️9352988975 Two shot with one girl (...
Just Call Vip call girls Mysore Escorts ☎️9352988975 Two shot with one girl (...
gajnagarg
 
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
amitlee9823
 
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
amitlee9823
 
Abortion pills in Jeddah | +966572737505 | Get Cytotec
Abortion pills in Jeddah | +966572737505 | Get CytotecAbortion pills in Jeddah | +966572737505 | Get Cytotec
Abortion pills in Jeddah | +966572737505 | Get Cytotec
Abortion pills in Riyadh +966572737505 get cytotec
 
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts ServiceCall Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
Just Call Vip call girls Erode Escorts ☎️9352988975 Two shot with one girl (E...
Just Call Vip call girls Erode Escorts ☎️9352988975 Two shot with one girl (E...Just Call Vip call girls Erode Escorts ☎️9352988975 Two shot with one girl (E...
Just Call Vip call girls Erode Escorts ☎️9352988975 Two shot with one girl (E...
gajnagarg
 
Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night StandCall Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand
amitlee9823
 
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...
amitlee9823
 
Call Girls In Attibele ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Attibele ☎ 7737669865 🥵 Book Your One night StandCall Girls In Attibele ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Attibele ☎ 7737669865 🥵 Book Your One night Stand
amitlee9823
 

Recently uploaded (20)

➥🔝 7737669865 🔝▻ Ongole Call-girls in Women Seeking Men 🔝Ongole🔝 Escorts S...
➥🔝 7737669865 🔝▻ Ongole Call-girls in Women Seeking Men  🔝Ongole🔝   Escorts S...➥🔝 7737669865 🔝▻ Ongole Call-girls in Women Seeking Men  🔝Ongole🔝   Escorts S...
➥🔝 7737669865 🔝▻ Ongole Call-girls in Women Seeking Men 🔝Ongole🔝 Escorts S...
 
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Junnasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
 
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service BangaloreCall Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
 
Just Call Vip call girls roorkee Escorts ☎️9352988975 Two shot with one girl ...
Just Call Vip call girls roorkee Escorts ☎️9352988975 Two shot with one girl ...Just Call Vip call girls roorkee Escorts ☎️9352988975 Two shot with one girl ...
Just Call Vip call girls roorkee Escorts ☎️9352988975 Two shot with one girl ...
 
VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...
VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...
VIP Model Call Girls Hinjewadi ( Pune ) Call ON 8005736733 Starting From 5K t...
 
➥🔝 7737669865 🔝▻ malwa Call-girls in Women Seeking Men 🔝malwa🔝 Escorts Ser...
➥🔝 7737669865 🔝▻ malwa Call-girls in Women Seeking Men  🔝malwa🔝   Escorts Ser...➥🔝 7737669865 🔝▻ malwa Call-girls in Women Seeking Men  🔝malwa🔝   Escorts Ser...
➥🔝 7737669865 🔝▻ malwa Call-girls in Women Seeking Men 🔝malwa🔝 Escorts Ser...
 
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
Escorts Service Kumaraswamy Layout ☎ 7737669865☎ Book Your One night Stand (B...
 
Call Girls In Shivaji Nagar ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Shivaji Nagar ☎ 7737669865 🥵 Book Your One night StandCall Girls In Shivaji Nagar ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Shivaji Nagar ☎ 7737669865 🥵 Book Your One night Stand
 
Just Call Vip call girls Mysore Escorts ☎️9352988975 Two shot with one girl (...
Just Call Vip call girls Mysore Escorts ☎️9352988975 Two shot with one girl (...Just Call Vip call girls Mysore Escorts ☎️9352988975 Two shot with one girl (...
Just Call Vip call girls Mysore Escorts ☎️9352988975 Two shot with one girl (...
 
5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed
5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed
5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed
 
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
Call Girls Bommasandra Just Call 👗 7737669865 👗 Top Class Call Girl Service B...
 
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
Call Girls Bannerghatta Road Just Call 👗 7737669865 👗 Top Class Call Girl Ser...
 
Abortion pills in Jeddah | +966572737505 | Get Cytotec
Abortion pills in Jeddah | +966572737505 | Get CytotecAbortion pills in Jeddah | +966572737505 | Get Cytotec
Abortion pills in Jeddah | +966572737505 | Get Cytotec
 
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts ServiceCall Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
 
Just Call Vip call girls Erode Escorts ☎️9352988975 Two shot with one girl (E...
Just Call Vip call girls Erode Escorts ☎️9352988975 Two shot with one girl (E...Just Call Vip call girls Erode Escorts ☎️9352988975 Two shot with one girl (E...
Just Call Vip call girls Erode Escorts ☎️9352988975 Two shot with one girl (E...
 
Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night StandCall Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Hsr Layout ☎ 7737669865 🥵 Book Your One night Stand
 
SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...
SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...
SAC 25 Final National, Regional & Local Angel Group Investing Insights 2024 0...
 
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men  🔝Bangalore🔝   Esc...
➥🔝 7737669865 🔝▻ Bangalore Call-girls in Women Seeking Men 🔝Bangalore🔝 Esc...
 
Call Girls In Attibele ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Attibele ☎ 7737669865 🥵 Book Your One night StandCall Girls In Attibele ☎ 7737669865 🥵 Book Your One night Stand
Call Girls In Attibele ☎ 7737669865 🥵 Book Your One night Stand
 
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24  Building Real-Time Pipelines With FLaNKDATA SUMMIT 24  Building Real-Time Pipelines With FLaNK
DATA SUMMIT 24 Building Real-Time Pipelines With FLaNK
 

Loading data into Apache Ignite

  • 1. Loading data into Apache Ignite Stephen Darlington 01 May 2019 2019 © GridGain Systems
  • 2. 2019 © GridGain Systems GridGain Company Confidential Apache Ignite In-Memory Computing Platform Mainframe NoSQL HadoopIgnite Persistence Persistent Layer RDBMS Machine and Deep Learning EventsStreamingMessagingTransactionsSQLKey-Value Service GridCompute Grid Application Layer Web SaaS SocialMobile IoT In-Memory Data Store
  • 3. 2019 © GridGain Systems GridGain Company Confidential How do I load data? This Photo by Unknown Author is licensed under CC BY-SA
  • 4. 2019 © GridGain Systems GridGain Company Confidential Official answer 1. Open your IDE 2. Create a project 3. Edit pom.xml to include Apache Ignite libraries 4. Create a new class 5. Code to open and parse input file 6. Boilerplate Ignite cluster code 7. IgniteDataStreamer code 8. Debug 9. Edit 10. Debug 11. Edit 12. Debug 13. Run 14. Play with resulting data
  • 5. 2019 © GridGain Systems GridGain Company Confidential2019 © GridGain Systems There must be an easier way? 8 This Photo by Unknown Author is licensed under CC BY-NC- ND
  • 6. 2019 © GridGain Systems GridGain Company Confidential Apache Ignite In-Memory Computing Platform Mainframe NoSQL HadoopIgnite Persistence Persistent Layer RDBMS Machine and Deep Learning EventsStreamingMessagingTransactionsKey-Value Service GridCompute Grid Application Layer Web SaaS SocialMobile IoT In-Memory Data Store SQL
  • 7. 2019 © GridGain Systems GridGain Company Confidential Using SQL
  • 8. 2019 © GridGain Systems GridGain Company Confidential But it gets complicated…
  • 9. 2019 © GridGain Systems GridGain Company Confidential SQL Streaming Apache Ignite In-Memory Computing Platform Mainframe NoSQL HadoopIgnite Persistence Persistent Layer RDBMS Machine and Deep Learning EventsMessagingTransactions Service GridCompute Grid Application Layer Web SaaS SocialMobile IoT In-Memory Data Store Key-Value
  • 10. 2019 © GridGain Systems GridGain Company Confidential Using Python
  • 11. 2019 © GridGain Systems GridGain Company Confidential SQL Apache Ignite In-Memory Computing Platform Mainframe NoSQL HadoopIgnite Persistence Persistent Layer RDBMS Machine and Deep Learning EventsMessagingTransactionsKey-Value Service GridCompute Grid Application Layer Web SaaS SocialMobile IoT In-Memory Data Store Streaming
  • 12. 2019 © GridGain Systems GridGain Company Confidential Using Apache Spark
  • 13. 2019 © GridGain Systems GridGain Company Confidential Using Apache Spark
  • 14. 2019 © GridGain Systems GridGain Company Confidential2019 © GridGain Systems What did we learn? 17 • Many options – Python, Spark, SQL – Scala – Groovy – Node.js • No one “best” answer • REPLs are awesome – …and can be used for a lot more than just loading data
  • 15. 2019 © GridGain Systems GridGain Company Confidential Resources • Apache Ignite documentation – https://apacheignite.readme.io/docs – https://ignite.apache.org • Blog – Loading Data into Ignite. https://link.medium.com/66dzsrWw4V – Python, part 1. https://link.medium.com/CUjDnzBQcW – Python, part 2. https://link.medium.com/3dWH1oDQcW
  • 16. 2019 © GridGain Systems GridGain Company Confidential And finally… • Get a free ticket to the In-Memory Computing Summit Europe 2019 (June 3-4) by completing this survey: – http://bit.ly/IMCSeu2019 • More information here: – https://www.imcsummit.org/2019/eu/
  • 17. 2019 © GridGain Systems GridGain Company Confidential2019 © GridGain Systems Thank you 20 Stephen Darlington Senior Consultant GridGain Systems @sdarlington

Editor's Notes

  1. Inspired by trying to get up-to-speed with a new, shiny project. Anything data centric, whether machine learning or SQL, needs data. I work for GG, donated Ignite, blah
  2. Have you heard of Apache Ignite or GridGain? GridGain Systems donated the code to the Apache Ignite project. It became a top level project of the Apache Software Foundation (ASF) in 2014, the second fastest to do so. Apache Ignite is now one of the top 5 Apache Software Foundation projects, and has been for 2 years now. It’s the most active in-memory computing projects right now, used by thousands of companies worldwide. GridGain is the only commercially supported version. It adds integration, security, deployment, management and monitoring to the same core Ignite that help with business-critical applications. We also provide global support and services. We also continue to be the biggest contributor to Ignite. [1] http://globenewswire.com/news-release/2019/07/09/1534470/0/en/The-Apache-Software-Foundation-Announces-Annual-Report-for-2019-Fiscal-Year.html [2] https://blogs.apache.org/foundation/entry/apache-in-2017-by-the
  3. You are probably relying on us for some part of your personal or professional life. We have several of the top 20 banks and wealth management companies as customers. If you include FinTech, 48-50 of the world’s largest banks use us indirectly. (through Finastra) Some of the leading software companies rely on us for their speed and scale. Microsoft uses us for real-time cloud security detection. Workday used us to get the scale they needed to sell to Walmart, and then to be about to run their software on Amazon, for Amazon. There are some very large retail/e-commerce companies, including PayPal, HomeAway and Expedia. And several innovators across FinTech, adTech, IoT and other areas.
  4. Traditional databases don’t scale. Buy bigger and bigger boxes until you run out of money. Traditional compute grids have to copy data across the network, which at modern scale is just impractical. Ignite scales horizontally and sends compute to the data rather than the other way around. In memory for speed. Disk persistence for volume.
  5. You fired up a node and you want to play… how do you load data? Oracle has SQL*Loader. Most other legacy databases have something similar. Is there an Ignite equivalent?
  6. Simple 14 point process
  7. Okay, I’m being facetious. That approach is good for production. For large volumes of data. For weird and wonderful data formats. But what if you want to do something quickly, preferably without firing up an IDE?
  8. Ignite supports ANSI-99 SQL…
  9. Kind of like BULK INSERT in SQL Server. Kind of like SQL*Loader in Oracle Good news: built-in Bad news: only works for CSV Basically zero configuration sqlline -u jdbc:ignite:thin://127.0.0.1 0: jdbc:ignite:thin://127.0.0.1>COPY FROM "file.csv" INTO tablename (col1, col2) FORMAT CSV;
  10. Which means you end up using horrible command-line tricks to convert data into CSV format. Here we’re using jq to convert from JSON to CSV jq '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv' < file.json > file.csv
  11. Python
  12. Spark – kind of cheating 
  13. Start pyspark with a bunch of extra libraries so that it also understands Ignite. This is optimized for typing. You could also optimize for less code in memory. bin/pyspark --jars $IGNITE_HOME/libs/ignite-spring/*.jar,$IGNITE_HOME/libs/optional/ignite-spark/ignite-*.jar,$IGNITE_HOME/libs/*.jar,$IGNITE_HOME/libs/ignite-indexing/*.jar
  14. In one line we read a JSON file It understands the structure of the file – no further coding Filters, drop columns, etc. Functional. b = spark.read.format('json').load('filename.json’) b.filter('href is not null’) \ .drop('hash', 'meta’) \ .write.format('ignite’) \ .option('config','default-config.xml’) \ .option('table','bookmarks’) \ .option('primaryKeyFields','href’) \ .mode('overwrite’) \ .save()