Downloaded 16 times
More Related Content
Similar to ApexMeetup Geode - Talk1 2016-03-17
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
ApexMeetup Geode - Talk1 2016-03-17
- 1. Apache Geode, and Pivotal'sleadership role in open sourcing (Gemfire) Nitin Lamba (incubating)
- 2. Pivotal’s Open Sourcestrategy What is Apache Geode? History Differentiators Basic Concepts Resources Q & A Agenda 2
- 3.
- 4. 4 In 2015, Pivotalgranted the components of its Big Data Suite to open source 6 Million Lines of Code 4 new open source communities
- 5. 5 May 2015 Sept2015 Sept 2015Oct 2015
- 6. From GEMFIRE toGEODE… 6
- 7. A distributed, memory-baseddata management platform for data oriented apps that need: • high performance, scalability, resiliency and continuous availability • fast access to critical data sets • location-aware distributed data processing • event-driven data architecture What is GEODE? 7
- 8. • 1000+ systemsin production (real customers) • Cutting edge use cases Incubating but ROCK solid… 8 <2000 2004 2008 2012 2016 Early drivers • Data Volumes • Margins/ transactions • IT maintenance costs • Elasticity needs Real-time needs • Real-time response • Time to market needs • Flexible Data Models • Persistent+In-memory Global Data • Visibility across DC • Fast Ingest • Device to enterprise • Uptime (always on) Open Source! • Apache Incubation • Gemfire > Geode • Geode M1 release • 1st Geode Summit Financial Services US DoD Trade Clearing Travel Portal Online Gambling Telcos Manufacturing Auto Insurance Payroll processing Rail systems
- 9. …with both SCALEand SPEED, … 9 40K Transactions per second 3TB Data in-memory 17B Records in-memory 120K Concurrent users
- 10. … and impactinga LOT of people! 10 China Railway Corporation Indian Railways 17% 19% 36% of the world population
- 11. High-level Architecture 11 Powerful appdevelopment kit • APIs: Java & REST • Adapters: Redis, Lucene*, Spark*, … Multiple persistence options • Filesystem, RDBMS or HDFS* • Sync: read-through, write-through • Async: write-behind Durable <K,V> cache/ store • Data replicated or partitioned • Redundant storage in-memory/ disk • Flexible data retention policies Î Locator Server Server Server Server + A Peer-2-Peer in-memory Distributed System REST * Experimental and waiting community feedback
- 12. • Minimize copying •Minimize contention points • Run user code in-process • Partitioning & parallelism • Avoid disk seeks • Automated benchmarks What makes it go FAST? 12
- 13. • Cache • Region •Member • Client Cache • Persistence • Functions Let’s talk about a few BASIC CONCEPTS… 13
- 14. • In-memory storageand management for your data • Configurable through XML, Java API or CLI • Collection of Region What is a CACHE? 14
- 15. • Distributed java.util.Mapon steroids (Key/Value) • Consistent API regardless of where or how data is stored • Observable (reactive) • Highly available, redundant on cache Member (s). What is a REGION? 15
- 16. • Local, Replicatedor Partitioned • In-memory or persistent • Redundant • LRU • Overflow Region: Types & Options 16 LOCAL LOCAL_HEAP_LRU LOCAL_OVERFLOW LOCAL_PERSISTENT LOCAL_PERSISTENT_OVERFLOW PARTITION PARTITION_HEAP_LRU PARTITION_OVERFLOW PARTITION_PERSISTENT PARTITION_PERSISTENT_OVERFLOW PARTITION_PROXY PARTITION_PROXY_REDUNDANT PARTITION_REDUNDANT PARTITION_REDUNDANT_HEAP_LRU PARTITION_REDUNDANT_OVERFLOW PARTITION_REDUNDANT_PERSISTENT PARTITION_REDUNDANT_PERSISTENT_OVERFLOW REPLICATE REPLICATE_HEAP_LRU REPLICATE_OVERFLOW REPLICATE_PERSISTENT REPLICATE_PERSISTENT_OVERFLOW REPLICATE_PROXY
- 17. • Durability • WALfor efficient writing • Consistent recovery • Compaction Persistent Regions 17 Server 1 Server N
- 18. • A processthat has a connection to the system • A process that has created a cache • Embeddable within your application What is a MEMBER? 18 Client Locator Server
- 19. • A processconnected to the Geode server(s) • Can have a local copy of the data • Run OQL queries on local data • Can be notified about events on the servers What is a CLIENT CACHE? 19
- 20. Persistence - SharedNothing 20 Server 3Server 2Server 1
- 21. Persistence - SharedNothing 21 Server 3Server 2Server 1 B1 B3 B2 B1 B3 B2 Primary Secondary
- 22. Persistence - SharedNothing 22 Server 3Server 2Server 1 B1 B3 B2 B1 B3 B2 Primary Secondary
- 23. Persistence - SharedNothing 23 Server 3Server 2Server 1 B1 B3 B2 B1 B3 B2 Primary Secondary
- 24. Persistence - SharedNothing 24 Server 3Server 2Server 1 B1 B3 B2 B1 B3 B2 Primary Secondary B3 B2 Server 1 waits for others when it starts
- 25. Persistence - SharedNothing 25 Server 3Server 2Server 1 B1 B3 B2 B1 B3 B2 Primary Secondary Fetches missed operations on restart
- 26. Persistence - OperationalLogs 26 Create k1->v1 Create k2->v2 Modify k1->v3 Create k4->v4 Modify k1->v5 Create k6->v6 Member 1 Put k6->v6 Oplog2.crf Oplog1.crf Append to operation log
- 27. Persistence - OperationalLogs: Compaction 27 Create k1->v1 Create k2->v2 Modify k1->v3 Create k4->v4 Modify k1->v5 Create k6->v6 Member 1 Put k6->v6 Oplog2.crf Oplog1.crf Append to operation log Copy live data forward
- 28. • Used fordistributed concurrent processing (Map/Reduce, stored procedure) • Highly available • Data oriented • Member oriented Functions 28
- 29.
- 30. 30 • Check out:http://geode.incubator.apache.org • Subscribe: user-subscribe@geode.incubator.apache.org • Download: http://geode.incubator.apache.org/releases/ Join the Community!
- 31.
- 32.
- 33.
- 34. …and horizontal, consistentSCALABILITY! 34 Horizontal scaling for reads, consistent latency and CPU 0. 4.5 9. 13.5 18. 0. 1.25 2.5 3.75 5. 6.25 2 4 6 8 10 Speedup Server Hosts speedup latency (ms) CPU % • Scaled from 256 clients and 2 servers to 1280 clients and 10 servers • Partitioned region with redundancy and 1K data size
- 35.
Editor's Notes
- #4 Pivotal employs committers, project leads and contributors to a variety of open source projects and community initiatives Examples: The popular Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Redis is an open source, networked, in-memory, key-value data store with optional durability. Written in ANSI C, it is often referred to as a data structure server, because keys can contain strings, hashes, lists, sets and sorted sets. Redis is open source software released under the terms of the three clause BSD license. Redis is the most popular key-value store in the cloud. RabbitMQ is a complete and highly-reliable enterprise messaging system, based on the AMQP standard. RabbitMQ is easy to use, runs on all major operating systems and supports a huge number of developer platforms. It is both open source and commercially supported. Released under version 1.1 of the Mozilla Public License, RabbitMQ is the most popular AMQP implementation on the planet. MADlib is an open source library for scalable in-database analytics. It provides data-parallel implementations of mathematical, statistical and machine-learning methods for structured and unstructured data. Apache Tomcat is a popular open source application server that implements the Java Servlet, JavaServer Pages, Java Unified Expression Language and Java WebSocket technologies. Apache Tomcat powers numerous large-scale, mission-critical web applications (including those written in Spring) across a diverse range of industries and organizations. It is released under the Apache License version 2. Apache HTTP Server is the most popular web server on the Internet. Powering over 100 million websites, it is one of the oldest open source projects in use today. Apache HTTP Server provides functionality around authentication, security and content delivery. It is released under the Apache License version 2. The Eclipse Foundation is a not-for-profit, member-supported corporation that hosts the Eclipse projects. Full details of Eclipse and the Eclipse Foundation are available athttp://www.eclipse.org. Pivotal is an Eclipse solutions member of the Eclipse Foundation and participates in several projects at Eclipse, including JDT Core, AJDT, AspectJ, Orion and Flux.