It’s not enough to build a mesh of sensors or embedded devices to get more insights about the surrounding environment and optimize your production. Usually, your IoT solution needs to be capable of transferring enormous amounts of data to a storage or cloud where the data has to be processed further. Quite often, the processing of the endless streams of data has to be done almost in real-time so that you can react on the IoT subsystem’s state accordingly, and in time. During this session, see how to build a Fast Data solution that will receive endless streams from the IoT side and will be capable of processing the streams in real-time using Apache Ignite’s cluster resources. In particular, learn about data streaming to an Apache Ignite cluster from embedded devices and real-time data processing with Apache Spark.

1. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Denis Magda
GridGainProduct Manager
Apache Ignite PMC
Apache Ignite and Apache Spark
Where Fast Data Meets the IoT
http://ignite.apache.org #apacheignite #denismagda

2. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Agenda
• IoTDemands to Software
• IoTSoftware Stack
• Device OS/RTOS
• Data Collection and Enrichment
• HTAP Platform
• Application APIs
• Demo

3. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
IoT Demands to Software

4. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
IoT Demands to Software
• Real-time ingestion
• Real-time processing
• Time-series support
• Flexible Querying APIs
– SQL
– Full-text search
– Geo-spatial
• Analytics
– BI
– Machine Learning
• High-availability
• Simple scalability

5. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
IoT Software Stack

6. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
HTAP Platform
Data Collectionand Enrichment
Device OS / Real-Time OS
Application APIs
IoT Software Stack

7. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
HTAP Platform
Data Collectionand Enrichment
Device OS / Real-Time OS
Application APIs
Apache IoT Software Stack

8. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Device OS/RTOS

9. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Apache MyNewt
• Open Source RTOS
– Cortex M0-M4
– MIPS & RISC-V
• Networking
– Bluetooth Low Energy
– Wi-Fi
– TCP/IP & UPD
• Build & Package Management
• Secure bootloader and signed
images
• Remote Firmware Upgrade

10. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Data Collection and Enrichment

11. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Data Collection and Enrichment
• Spark Streaming
– Fault-Tolerant Streams
Processing
• Data Collection
– Sockets
– Kafka
– Flume
• Data Enrichment
– Spark API
• Data Storage
– Apache Ignite

12. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
HTAP Platform

13. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Use Case: Smart Metering and Utilities – delivers a comprehensive IOT platform
50+ Million
Meters
• SilverSpring Requirements
– Migrate to in-memory processing
– Add scalability & elasticity
– Use open source technologies
• SilverSpring Solution
– SaaS Architecture
• Multi-Tenancy
• Advanced Security
– Strong compute capabilities
• Co-located in-memory processing
– Demonstrated best
• On-demand elasticity & scalability
• ANSI-99 SQL Support
• Transactional consistency
GridGain
Security
SilverSpring IoT
Platform

14. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Apache Ignite HTAP Platfrom

15. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Key-Value Data Grid
• Distributed Key-Value Data Store
• Data Reliability
• High-Availability
– Active replicas, automatic
failover
• Data Consistency
– ACID distributed transactions

16. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
In-Memory Data Grid: Partitioning
Partitioned Cache Replicated Cache

17. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Streaming to Ignite
• Ignite Data Streamer
– Fastest Data Injection
– Automatic data partitioning
• Stream Receivers
– Custom logic execution
– Additional data transformation
• Stream Adapter
– Socket
– Kafka
– Flink
– RocketMQ
– etc.

18. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Application APIs

19. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Apache Ignite SQL APIs
• ANSI-99 SQL
• Geo-spatial Queries
• Full-text Search
• Always Consistent
• Fault Tolerant
• Cross-Platform
– JDBC and ODBC drivers
– DML (INSERT, UPDATE, etc.)
– DDL (CREATE, DROP, etc.)

20. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Apache Ignite SQL Grid: Queries
• Distributed Joins
• Automatic Group By,
Aggregations, Sorting
• Cross-Cache Joins, Unions, etc.
• Ad-Hoc SQL Support

21. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Apache Ignite Compute Grid
• Direct API for
MapReduce
• Direct API for ForkJoin
• Zero Deployment
• State Checkpoints
• Load Balancing
• Automatic Failover

22. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Spark Shared RDDs
• IgniteRDD
– Share RDD across jobs on
the host
– Share RDD across jobs in
the application
– Share RDD globally
• Faster SQL
– In-Memory Indexes
– SQL on top of Shared RDD

23. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Machine Learning Grid

24. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Demo

25. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
Resources
• Documentation:
– Apache Ignite: https://apacheignite.readme.io/docs
– Apache Spark Streaming: http://spark.apache.org/streaming/
– Ignite and Spark Integration: https://apacheignite-
fs.readme.io/docs/ignite-for-spark
– Apache MyNewt: http://mynewt.apache.org
• Demo Source Code:
– https://github.com/dmagda/IgniteSparkIoT

26. Apache®, Apache Ignite, Ignite®, and the Apache Ignite logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
ANY QUESTIONS?
Thank you for joining us. Follow the conversation.
http://ignite.apache.org
#apacheignite #denismagda