Downloaded 49 times
Uploaded byApache Geode
Build your first Internet of Things app today with Open Source
This document provides an overview of Apache Geode, an in-memory data management platform. It discusses using Geode for high-performance and scalable applications that require fast access to critical datasets. Key concepts explained include regions, caching of data, and the use of functions to enable distributed processing across a Geode cluster. The document also mentions integrations with Spark and Cloud Foundry that allow persisting RDDs in Geode and exposing regions as RDDs.
More Related Content
![DevOps, CD and [Data] Microservices](https://cdn.slidesharecdn.com/ss_thumbnails/qconsp-devopsfordata-160405174944-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Build your first Internet of Things app today with Open Source
Build your first Internet of Things app today with Open Source
- 1. 1 William Markito @william_markito Fred Melo @fredmelo_br Buildyour first Internet of Things app today with open source
- 2. Trilateration and itslimitations 2
- 3. Particle Filters - Calculatingthe optimum solution 3
- 4. The solution 4 1. Capturesignal strength 2. Calculate distance from antenna 3. Trilaterate different sensors to predict location in real-time 4. Show on a map with live updates
- 5. Architecture Overview 5 Ingest SpringXD Groovy JSON HTTP + Distance TransformSink Calculate Device Distance Predict Location Spring Boot Application Platform GUI
- 6.
- 7. A distributed, memory-baseddata management platform for data oriented apps that need: • high performance, scalability, resiliency and continuous availability • fast access to critical data set • location aware distributed data processing • event driven data architecture 7 Introduction
- 8. • Cache • In-memorystorage and management for your data • Configurable through XML, Spring, Java API or CLI • Collection of Region 8 Concepts Region Region Region Cache JVM
- 9. • Region • Distributedjava.util.Map on steroids (Key/Value) • Consistent API regardless of where or how data is stored • Observable (reactive) • Highly available, redundant on cache Member (s). 9 Concepts Region Cache java.util.Map JVM Key Value K01 May K02 Tim
- 10. • Region • Local,Replicated or Partitioned • In-memory or persistent • Redundant • LRU • Overflow 10 Concepts Region Cache java.util.Map JVM Key Value K01 May K02 Tim Region Cache java.util.Map JVM Key Value K01 May K02 Tim LOCAL LOCAL_HEAP_LRU LOCAL_OVERFLOW LOCAL_PERSISTENT LOCAL_PERSISTENT_OVERFLOW PARTITION PARTITION_HEAP_LRU PARTITION_OVERFLOW PARTITION_PERSISTENT PARTITION_PERSISTENT_OVERFLOW PARTITION_PROXY PARTITION_PROXY_REDUNDANT PARTITION_REDUNDANT PARTITION_REDUNDANT_HEAP_LRU PARTITION_REDUNDANT_OVERFLOW PARTITION_REDUNDANT_PERSISTENT PARTITION_REDUNDANT_PERSISTENT_OVERFLOW REPLICATE REPLICATE_HEAP_LRU REPLICATE_OVERFLOW REPLICATE_PERSISTENT REPLICATE_PERSISTENT_OVERFLOW REPLICATE_PROXY
- 11. • Persistent Regions •Durability • WAL for efficient writing • Consistent recovery • Compaction 11 Concepts Modify k1->v5 Create k6->v6 Create k2->v2 Create k4->v4 Oplog2.crf Member 1 Modify k4->v7Oplog3.crf Put k4->v7 Region Cache java.util.Map JVM Key Value K01 May K02 Tim Region Cache java.util.Map JVM Key Value K01 May K02 Tim Server 1 Server N
- 12. • Member • Aprocess that has a connection to the system • A process that has created a cache • Embeddable within your application 12 Concepts Client Locator Server
- 13. • Client cache •A process connected to the Geode server(s) • Can have a local copy of the data • Can be notified about events on the servers 13 Concepts Application GemFire Server Region Region RegionClient Cache
- 14. • Functions • Usedfor distributed concurrent processing (Map/Reduce, stored procedure) • Highly available • Data oriented • Member oriented 14 Concepts Submit (f1) f1 , f2 , … fn Execute Functions
- 15. 15 Concepts Server Server FunctionService.onRegion.withFilter.execute ResultCollector.getResult Server DistributedSystem execute Server Server 6 1 result execute execute result result 2 5 3 4 3 4 Server Partitioned Region Data Store - X Partitioned Region Data Store - Y Partitioned Region Data Store - Z Partitioned Region Data Accessor Partitioned Region Data Accessor filter = Keys X, Y Client Region • Functions
- 16. • Listeners • CacheWriter/ CacheListener • AsyncEventListener (queue / batch) • Parallel or Serial • Conflation 16 Concepts
- 17.
- 18. Introduction to SpringXD 18 Runsas a distributed application or as a single node
- 19. Spring XD 19 A streamis composed from modules. Each module is deployed to a container and its channels are bound to the transport.
- 20.
- 21. • HDFS Persistence •Off-heap memory storage • Lucene Search • Spark Integration • Cloud Foundry service 21 Apache Geode Roadmap
- 22. 22 Spark Connector "An RDDin Spark is simply an immutable distributed collection of objects. Each RDD is split into multiple partitions, which may be computed on different nodes of the cluster. RDDs can contain any type of Python, Java, or Scala objects, including user-defined classes." Learning Spark By Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia Publisher: O'Reilly Media • Geode Spark Connector allows: • Regions exposed as RDD • RDDs persisted in Geode • Execute queries on Geode using OQL
- 23. 23 Spark Connector • Webbased REPL • Multiple Interpreters • Apache Spark • Markdown • Flink • Python… • Iterative & Exploratory • Data store • In-memory & Persistent • Highly Consistent • Transaction processing • Thousands of concurrent clients • Data Processing • Columnar queries • RDDs • Machine Learning • Analytics • Streaming
- 24.
- 25. 25 https://github.com/Pivotal-Open-Source-Hub/WifiAnalyticsIoT Source code anddetailed instructions available at: