Gemfire

4,051 views

Published on

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,051
On SlideShare
0
From Embeds
0
Number of Embeds
65
Actions
Shares
0
Downloads
137
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Gemfire

    1. 1. GemFire Enterprise Data Fabric GemStone Systems Inc. Michael I Lazar, Federal Technology Director [email_address] 301 325 8405 Steve Rixse, Director Public Sector [email_address] 301 664 8494
    2. 2. GemStone at a Glance <ul><li>Founded 1982, with over 200 installed customers, in the Global 2000. Includes Bear Sterns, JP Morgan, Merrill Lynch, Northrop Grumman, NCI </li></ul><ul><li>Unique and patented expertise in object management, high performance caching and data distribution technologies. </li></ul><ul><li>Profitable, well funded and experienced management team. </li></ul><ul><li>Office Locations - Beaverton OR, Santa Clara CA, New York City, Washington DC; 60% in R&D, 20 % in Sales and Marketing </li></ul><ul><li>Dedicated Federal and Financial teams </li></ul><ul><li>24x7x365 global support. </li></ul>
    3. 3. What do we do ? <ul><li>Distributed Operational Data Infrastructure </li></ul><ul><ul><li>Not just a distributed cache </li></ul></ul><ul><ul><li>Key semantics of a database – ACID </li></ul></ul><ul><ul><li>Key semantics of a Message bus </li></ul></ul><ul><li>Enable data sharing and event notifications </li></ul><ul><ul><li>At memory speeds </li></ul></ul><ul><li>Enable apps to continuously analyze and react to very fast moving data </li></ul>A Data Fabric for the Enterprise
    4. 4. What do we do - GemFire Value proposition <ul><li>Enable notification of data changes to users and other applications </li></ul><ul><li>Create framework for high performance data access </li></ul><ul><li>Scale applications to meet unpredictable demands for information </li></ul><ul><li>Boost performance across applications without increasing other hardware/software requirements </li></ul><ul><li>Reduces network load and can work over low bandwidth networks </li></ul><ul><li>Enhance the performance, scalability, and network characteristics of other software . </li></ul><ul><li>Single product with multiple uses, easy to implement and little to no management requirements </li></ul><ul><li>Standards compliant interfaces </li></ul>
    5. 5. GemFire Enterprise GemFire Enterprise (GFE Java and GFE C++) <ul><li>A scalable and high-performance software data infrastructure that </li></ul><ul><li>Caches and distributes data in multiple formats (Java, C++, XML) across distributed applications </li></ul><ul><li>Connects to backend systems like databases, messaging systems, etc. </li></ul><ul><li>Provides high data availability mechanisms </li></ul><ul><li>Supports distributed transactions and standards-based querying </li></ul><ul><li>Highly instrumented </li></ul>GemFire Real-Time Events <ul><li>In memory complex event processing (CEP) solution that facilitates agile, event-driven architectures by analyzing streams of data in conjunction with other static/historical data sources </li></ul><ul><li>JDBC and SQL based, with Continuous Queries </li></ul><ul><li>Client side materialized view </li></ul><ul><li>Messaging paradigm with connected and disconnect clients </li></ul><ul><li>Highly instrumented </li></ul>
    6. 6. EDF - Where Does it Fit? Mainframes Historical Data Customer Data Enterprise Data Fabric Distributed Caching High Availability Custom Application Data Sources Event Processing Data distribution Data Modeling Data Access Application Access High Availability Data Management Data Distribution Analytics Portals Execution Reporting SQL XML Java C++/ C# Grid Computing SOA EDA
    7. 7. What is GemFire EDF - Key Features? Virtualize data and events from many sources Manage and partition data in distributed main-memory for blazing performance Provide data access for applications in multiple formats to support heterogeneity High-speed data distribution (TCP/IP, Multicast) across multiple nodes - Move data on demand Replicate data to mirror nodes synchronously for high availability Overflow or persist to disk to scale to large data volumes and for data recovery Standards-based Querying and Continuous Querying on structured data and streaming data Distributed transactions for reliable operations Event Streams ENTERPRISE DATA FABRIC Analytics Portals Execution Reporting Query SQL XML Java C++/ C# Disk Grid Computing SOA Event-Processing
    8. 8. SOA – Composite Applications GEMFIRE ENTERPRISE DATA FABRIC Java, C/C++/C#, XML:DB, SOAP Interfaces Transactions, Logging, Authentication, Discovery Session Management, Entitlements ESB Business Process Composite Applications: e.g,, Web Portals File systems Data Warehouses Databases Custom Application Data Sources Process State <ul><li>Authentication </li></ul><ul><li>Session/Process </li></ul><ul><li>State Management </li></ul><ul><li>Reference data access </li></ul><ul><li>XML Data Transformation </li></ul>High Availability Data Management Data Distribution
    9. 9. SOA - Composite Processes Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N <ul><li>Authentication </li></ul><ul><li>Session State Management </li></ul><ul><li>Reference data access </li></ul><ul><li>XML Data Transformation </li></ul>Web Service 1 Web Service 2 Web Service 3 Web Service N GemFire Enterprise
    10. 10. SOA - Authentication Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N Web Service 1 Web Service 2 Web Service 3 Web Service N Identify and Authenticate - Get Authorizations (IA&A) Travel over network Calls Policy Server/LDAP repository Calls RDBMS Repeat for each step in application flow Often more time than actual service step GemFire eliminates redundant lookups This is also used for Authentication Management for Enterprise Portals GemFire Enterprise GFE
    11. 11. SOA - Session/Process State Management Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N Web Service 1 Web Service 2 Web Service 3 Web Service N In SOA process state is either Returned as XML between steps Often stored by workflow manager in RDBMS Stored in RDBMS by state aware services GemFire removes latency introduced by network and RDBMS This is also used for Session State Management for Enterprise Portals GemFire Enterprise GemFire
    12. 12. SOA – Reference Data Service calls back-end data source to look up same information over and over GemFire removes need to make redundant calls to back end Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N GemFire Enterprise Web Service 1 Web Service 2 Web Service 3 Web Service N
    13. 13. SOA – XML data transformation Service returns same information over and over (for a period of time) If it is called with the same request GemFire removes need to retrieve data from back end, and transform it into XML. Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N GemFire Enterprise Web Service 1 Web Service 2 Web Service 3 Web Service N
    14. 14. SOA – XML data transformation Service returns same information over and over (for a period of time) If it is called with the same request GemFire removes need to retrieve data from back end, and transform it into XML. <ul><li>GemFire also provides </li></ul><ul><li>Optimized XML data representation </li></ul><ul><ul><li>Less Memory </li></ul></ul><ul><ul><li>Less Java GC/overhead </li></ul></ul><ul><li>Optimized XML data access </li></ul>Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N GemFire Enterprise Web Service 1 Web Service 2 Web Service 3 Web Service N
    15. 15. GemFire Design Patterns Application Survivability and Network Usage Reduction App Server Web Services Applications App Server or SOA applications Wide Area Network Wireless App Server GemFire Cache Web Browsers PDAs And Wireless Devices GemFire Cache
    16. 16. GF Real Time Events Functional Overview GemFire Real-Time Events Real-Time Data Streams Continuously analyze (execute continuous queries on event streams) Identify patterns of interest through correlation with other data Selectively notify systems and personnel of relevant changes and updates in real-time Financial Markets Data Battlefield Information RFID Data Wireless Network Data Register Continuous Queries
    17. 17. Summary Highlights <ul><li>Solve numerous challenges in SOA and distributed systems environments </li></ul><ul><li>Enhance Scalability, Performance, Reliability of systems </li></ul><ul><li>Potentially reduce hardware and software costs </li></ul><ul><li>Easy to insert into new and existing systems </li></ul><ul><li>Highly Available, Highly instrumented, Highly configurable </li></ul><ul><li>NSA Acquisition Security approved </li></ul>
    18. 18. EDF - Key Technical Concepts GemStone Systems Inc. This presentation contains information that is CONFIDENTIAL to GemStone Systems Inc. Do not distribute or copy this presentation without express written permission.
    19. 19. GemFire Enterprise GemFire Enterprise <ul><li>A scalable and high-performance data infrastructure that </li></ul><ul><li>Caches and distributes data in multiple formats, across distributed </li></ul><ul><li>applications </li></ul><ul><li>Connects to backend systems like databases, messaging systems, etc. </li></ul><ul><li>Provides high data availability mechanisms </li></ul><ul><li>Supports distributed transactions and standards-based querying </li></ul><ul><li>Offers interfaces for C/C++/C# applications as well </li></ul><ul><li>Target Environments </li></ul><ul><li>Java/J2EE based platforms </li></ul><ul><li>XML/Web Services </li></ul>
    20. 20. GemFire Enterprise -- C++ GemFire Enterprise -- C++ <ul><li>An extremely high throughput and performance data infrastructure that </li></ul><ul><li>Native C++ bindings to avoid inefficiencies </li></ul><ul><li>Highly parallel distribution with optimized transport </li></ul><ul><li>Avoids unnecessary user-kernel crossings with no contention or </li></ul><ul><li>context-switching </li></ul><ul><li>Connects to backend systems like databases, messaging systems, etc. </li></ul><ul><li>Provides high data availability mechanisms </li></ul><ul><li>Offers interfaces for Java applications as well (future) </li></ul><ul><li>Target Environments </li></ul><ul><li>C++ -based application environments </li></ul><ul><li>Performance and throughput are primary concerns </li></ul>
    21. 21. GemFire Real-Time Events GemFire Real-Time Events <ul><li>GemFire Real-Time Events is a complex event processing (CEP) solution that enables an enterprise to facilitate agile, event-driven architectures by analyzing streams of data in conjunction with other static/historical data sources </li></ul><ul><li>Define relevant business events through simple SQL queries </li></ul><ul><li>Discern patterns through continuous querying and event </li></ul><ul><li>correlation with other data sources </li></ul><ul><li>Distribute appropriate information to selective client applications </li></ul><ul><li>Target Environments </li></ul><ul><li>Fast-moving data streams </li></ul><ul><li>Real-time pattern detection and analytics </li></ul>
    22. 22. Enterprise Data Fabric - What is it ? <ul><li>Middle tier Distributed Data Infrastructure </li></ul><ul><ul><li>Operational Data </li></ul></ul><ul><ul><li>Combines Distributed Caching (main-memory), DB semantics, Reliable Messaging/ Intelligent Routing and Continuous Analytics </li></ul></ul>Distributed Data Space Data warehouses Applications Rational databases
    23. 23. Where is the data in the Fabric? <ul><li>All across the Network </li></ul><ul><li>NETWORK IS THE DATABASE </li></ul>Cache Cache Cache In Local process memory Copies in multiple nodes
    24. 24. Enterprise Data Fabric - Physical View <ul><li>All across the Network </li></ul><ul><li>NETWORK IS THE DATABASE </li></ul>Cache Cache Cache Rational databases Clustered Application Spread across many nodes On Disk Or lazily fetched from external source
    25. 25. 5 Key Features <ul><li>1. Data Storage </li></ul><ul><ul><li>Optimized concurrent main-memory structures </li></ul></ul><ul><ul><li>Topology </li></ul></ul><ul><ul><ul><li>Embedded in application process </li></ul></ul></ul><ul><ul><ul><li>Client caches and Cache servers </li></ul></ul></ul><ul><ul><ul><li>Partitioned across many servers </li></ul></ul></ul><ul><ul><ul><li>n-way in-memory replication for HA </li></ul></ul></ul><ul><ul><ul><li>On Disk </li></ul></ul></ul><ul><ul><ul><ul><li>Overflow to disk </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Recoverable disk regions – Asynchronous or Synchronous </li></ul></ul></ul></ul>
    26. 26. 5 Key Features <ul><li>2. Access Model </li></ul><ul><ul><li>Object Caching API (get()/put()). </li></ul></ul><ul><ul><li>Querying through Xpath, OQL and SQL </li></ul></ul><ul><ul><li>XML /Web Services (HTTP/SOAP) </li></ul></ul><ul><ul><li>APIs in Java, C++, and C# </li></ul></ul><ul><ul><li>JTA Transactions </li></ul></ul><ul><ul><li>Management APIs – JMX </li></ul></ul>
    27. 27. 5 Key Features <ul><li>3. Data Distribution Model </li></ul><ul><ul><li>Consistency Model </li></ul></ul><ul><ul><ul><li>Distribution without ACKs </li></ul></ul></ul><ul><ul><ul><li>With ACKs </li></ul></ul></ul><ul><ul><ul><li>With global locking </li></ul></ul></ul><ul><ul><li>Reliable Pub-Sub semantics </li></ul></ul><ul><ul><ul><li>Novel declarative “role” based model to ensure message delivery </li></ul></ul></ul><ul><ul><ul><li>Dealing with Slow subscribers, net/node failures, etc </li></ul></ul></ul><ul><ul><li>Multiple transports </li></ul></ul><ul><ul><ul><li>TCP, Reliable UDP MultiCast </li></ul></ul></ul>
    28. 28. 5 Key Features <ul><li>4. Integration with External Data Sources </li></ul><ul><ul><li>A Framework for read-through, write-through, and write-behind </li></ul></ul><ul><ul><li>Custom Connectors link to DBMS, Middleware Messaging, EII solutions </li></ul></ul><ul><ul><li>Components - Loaders, Writers and Listeners </li></ul></ul>
    29. 29. 5 Key Features <ul><li>Dealing with high event rate </li></ul><ul><ul><li>RFID streaming data, etc </li></ul></ul><ul><li>Many clients can subscribe to portions of the Data Fabric </li></ul><ul><ul><li>Express interest using complex query </li></ul></ul><ul><li>CQ engine continuously analyses data events </li></ul><ul><ul><li>Continuously calculates how the query view has been impacted </li></ul></ul><ul><li>Ships “delta” and merges into client result set. </li></ul><ul><ul><li>Notifications delivered with guaranteed latency threshold </li></ul></ul>5. Continuous Analytics

    ×