GemFire Enterprise Data Fabric GemStone Systems Inc. Michael I Lazar, Federal Technology Director [email_address]   301 325 8405 Steve Rixse, Director Public Sector [email_address]   301 664 8494
GemStone at a Glance Founded 1982, with over 200 installed customers, in the Global 2000. Includes Bear Sterns, JP Morgan, Merrill Lynch, Northrop Grumman, NCI Unique and patented expertise in object management, high performance caching and data distribution technologies. Profitable, well funded and experienced management team.  Office Locations - Beaverton OR,  Santa Clara CA, New York City, Washington DC;  60% in R&D, 20 % in Sales and Marketing Dedicated Federal and Financial teams  24x7x365 global support.
What do we do ? Distributed Operational Data Infrastructure Not just a distributed cache Key semantics of a database – ACID Key semantics of a Message bus Enable data sharing and event notifications At memory speeds Enable apps to continuously analyze and react to very fast moving data A Data Fabric for the Enterprise
What do we do - GemFire Value proposition Enable notification  of data changes to users and other applications Create framework  for high performance data access Scale applications  to meet unpredictable demands for information Boost performance  across applications without increasing other hardware/software requirements Reduces network load  and can work over low bandwidth networks  Enhance  the performance, scalability, and network characteristics of  other software . Single product with multiple uses,  easy to implement  and little to  no management  requirements Standards compliant  interfaces
GemFire Enterprise GemFire Enterprise  (GFE Java and GFE C++) A scalable and high-performance software data infrastructure that Caches and distributes data in multiple formats (Java, C++, XML) across distributed applications  Connects to backend systems like databases, messaging systems, etc. Provides high data availability mechanisms Supports distributed transactions and standards-based querying Highly instrumented GemFire Real-Time Events In memory complex event processing (CEP) solution that facilitates agile, event-driven architectures by analyzing streams of data in conjunction with other static/historical data sources JDBC and SQL based, with Continuous Queries Client side materialized view Messaging paradigm with connected and disconnect clients Highly instrumented
EDF - Where Does it Fit? Mainframes Historical Data Customer  Data Enterprise Data Fabric Distributed Caching  High Availability  Custom Application  Data Sources Event Processing Data distribution  Data Modeling  Data Access  Application Access High Availability Data Management Data Distribution Analytics Portals Execution Reporting SQL XML Java C++/ C# Grid Computing  SOA  EDA
What is GemFire EDF - Key Features? Virtualize data and events  from many sources Manage and partition data in  distributed  main-memory  for  blazing performance Provide data access for applications in multiple formats to support heterogeneity High-speed data  distribution  (TCP/IP, Multicast)  across multiple nodes - Move data on demand Replicate data to mirror nodes synchronously for high availability Overflow or persist to  disk to scale  to large data volumes and for data recovery Standards-based  Querying and Continuous  Querying on structured  data and streaming data Distributed transactions for reliable operations Event Streams ENTERPRISE DATA FABRIC Analytics Portals Execution Reporting Query SQL XML Java C++/ C# Disk Grid Computing  SOA  Event-Processing
SOA  – Composite Applications GEMFIRE ENTERPRISE DATA FABRIC Java, C/C++/C#, XML:DB, SOAP Interfaces Transactions,  Logging, Authentication, Discovery Session Management,  Entitlements ESB Business Process Composite Applications:  e.g,, Web Portals File systems Data Warehouses Databases Custom Application  Data Sources Process State Authentication Session/Process State Management Reference data access XML Data Transformation High Availability Data Management Data Distribution
SOA  - Composite Processes Web Service 1  Composite Application Flow User/Process Web Service 2  Web Service 3 Web Service N  Authentication Session State Management Reference data access XML Data Transformation Web Service 1  Web Service 2  Web Service 3 Web Service N  GemFire Enterprise
SOA  - Authentication Web Service 1  Composite Application Flow User/Process Web Service 2  Web Service 3 Web Service N  Web Service 1  Web Service 2  Web Service 3 Web Service N  Identify and Authenticate  - Get Authorizations (IA&A) Travel over network Calls Policy Server/LDAP repository Calls RDBMS Repeat for each step in application flow Often more time than actual service step GemFire eliminates redundant lookups This is also used for  Authentication Management for Enterprise Portals GemFire Enterprise GFE
SOA  - Session/Process State Management Web Service 1  Composite Application Flow User/Process Web Service 2  Web Service 3 Web Service N  Web Service 1  Web Service 2  Web Service 3 Web Service N  In SOA process state is either Returned as XML between steps  Often stored by workflow manager in RDBMS Stored in RDBMS by state aware services GemFire removes latency introduced by network and RDBMS This is also used for  Session State Management for Enterprise Portals GemFire Enterprise GemFire
SOA  – Reference Data  Service calls back-end data source to  look up same information over and over GemFire removes need to make redundant calls to back end  Web Service 1  Composite Application Flow User/Process Web Service 2  Web Service 3 Web Service N  GemFire Enterprise Web Service 1  Web Service 2  Web Service 3 Web Service N
SOA  – XML data transformation Service returns same information over and over  (for a period of time) If it is called with the same request GemFire removes need to retrieve data from back end, and transform it into XML. Web Service 1  Composite Application Flow User/Process Web Service 2  Web Service 3 Web Service N  GemFire Enterprise Web Service 1  Web Service 2  Web Service 3 Web Service N
SOA  – XML data transformation Service returns same information over and over (for a period of time)  If it is called with the same request GemFire removes need to retrieve data from back end, and transform it into XML. GemFire also provides Optimized XML data representation Less Memory Less Java GC/overhead Optimized XML data access Web Service 1  Composite Application Flow User/Process Web Service 2  Web Service 3 Web Service N  GemFire Enterprise Web Service 1  Web Service 2  Web Service 3 Web Service N
GemFire Design Patterns Application Survivability and Network Usage Reduction App Server Web Services Applications App Server or SOA applications Wide Area Network Wireless App Server GemFire  Cache Web Browsers PDAs And Wireless Devices GemFire  Cache
GF Real Time Events Functional Overview GemFire Real-Time Events Real-Time Data Streams Continuously analyze (execute continuous queries on event streams) Identify patterns of  interest through  correlation with other data Selectively notify systems and personnel of relevant changes and updates in real-time Financial Markets Data Battlefield Information RFID Data Wireless Network Data Register Continuous Queries
Summary Highlights Solve numerous challenges in SOA and distributed systems environments Enhance Scalability, Performance, Reliability of systems Potentially reduce hardware and software costs Easy to insert into new and existing systems Highly Available, Highly instrumented, Highly configurable NSA Acquisition Security approved
EDF - Key Technical Concepts GemStone Systems Inc. This presentation contains information that is CONFIDENTIAL to GemStone Systems Inc. Do not distribute or copy this presentation without express written permission.
GemFire Enterprise GemFire Enterprise A scalable and high-performance data infrastructure that Caches and distributes data in multiple formats, across distributed applications  Connects to backend systems like databases, messaging systems, etc. Provides high data availability mechanisms Supports distributed transactions and standards-based querying Offers interfaces for C/C++/C# applications as well Target Environments Java/J2EE based platforms XML/Web Services
GemFire Enterprise -- C++ GemFire Enterprise -- C++ An extremely high throughput and performance data infrastructure that Native C++ bindings to avoid inefficiencies Highly parallel distribution with optimized transport Avoids unnecessary user-kernel crossings with no contention or  context-switching  Connects to backend systems like databases, messaging systems, etc. Provides high data availability mechanisms Offers interfaces for Java applications as well (future) Target Environments C++ -based application environments Performance and throughput are primary concerns
GemFire Real-Time Events GemFire Real-Time Events GemFire Real-Time Events is a complex event processing (CEP) solution that enables an enterprise to facilitate agile, event-driven architectures by analyzing streams of data in conjunction with other static/historical data sources Define relevant business events through simple SQL queries Discern patterns through continuous querying and event  correlation with other data sources  Distribute appropriate information to selective client applications Target Environments Fast-moving data streams Real-time pattern detection and analytics
Enterprise Data Fabric - What is it ?  Middle tier Distributed Data Infrastructure Operational Data Combines Distributed Caching (main-memory), DB semantics, Reliable Messaging/ Intelligent Routing and Continuous Analytics Distributed Data Space Data warehouses Applications Rational databases
Where is the data in the Fabric?  All across the Network NETWORK IS THE DATABASE Cache Cache Cache In Local process memory Copies in multiple nodes
Enterprise Data Fabric -  Physical View  All across the Network NETWORK IS THE DATABASE Cache Cache Cache Rational databases Clustered Application Spread across many nodes On Disk Or lazily fetched from  external source
5 Key Features 1. Data Storage Optimized concurrent main-memory structures Topology Embedded in application process Client caches and Cache servers Partitioned across many servers n-way in-memory replication for HA On Disk Overflow to disk Recoverable disk regions – Asynchronous or Synchronous
5 Key Features 2. Access Model Object Caching API (get()/put()). Querying through Xpath, OQL and SQL XML /Web Services (HTTP/SOAP) APIs in Java, C++, and C# JTA Transactions Management APIs – JMX
5 Key Features 3. Data Distribution Model Consistency Model Distribution without ACKs With ACKs With global locking Reliable Pub-Sub semantics Novel declarative “role” based model to ensure message delivery  Dealing with Slow subscribers, net/node failures, etc Multiple transports TCP, Reliable UDP MultiCast
5 Key Features 4. Integration with External Data Sources A Framework for read-through, write-through, and write-behind Custom Connectors link to DBMS, Middleware Messaging, EII solutions Components - Loaders, Writers and Listeners
5 Key Features Dealing with high event rate RFID streaming data, etc Many clients can subscribe to portions of the Data Fabric Express interest using complex query CQ engine continuously analyses data events  Continuously calculates how the query view has been impacted Ships “delta” and merges into client result set. Notifications delivered with guaranteed latency threshold 5. Continuous Analytics

Gemfire

  • 1.
    GemFire Enterprise DataFabric GemStone Systems Inc. Michael I Lazar, Federal Technology Director [email_address] 301 325 8405 Steve Rixse, Director Public Sector [email_address] 301 664 8494
  • 2.
    GemStone at aGlance Founded 1982, with over 200 installed customers, in the Global 2000. Includes Bear Sterns, JP Morgan, Merrill Lynch, Northrop Grumman, NCI Unique and patented expertise in object management, high performance caching and data distribution technologies. Profitable, well funded and experienced management team. Office Locations - Beaverton OR, Santa Clara CA, New York City, Washington DC; 60% in R&D, 20 % in Sales and Marketing Dedicated Federal and Financial teams 24x7x365 global support.
  • 3.
    What do wedo ? Distributed Operational Data Infrastructure Not just a distributed cache Key semantics of a database – ACID Key semantics of a Message bus Enable data sharing and event notifications At memory speeds Enable apps to continuously analyze and react to very fast moving data A Data Fabric for the Enterprise
  • 4.
    What do wedo - GemFire Value proposition Enable notification of data changes to users and other applications Create framework for high performance data access Scale applications to meet unpredictable demands for information Boost performance across applications without increasing other hardware/software requirements Reduces network load and can work over low bandwidth networks Enhance the performance, scalability, and network characteristics of other software . Single product with multiple uses, easy to implement and little to no management requirements Standards compliant interfaces
  • 5.
    GemFire Enterprise GemFireEnterprise (GFE Java and GFE C++) A scalable and high-performance software data infrastructure that Caches and distributes data in multiple formats (Java, C++, XML) across distributed applications Connects to backend systems like databases, messaging systems, etc. Provides high data availability mechanisms Supports distributed transactions and standards-based querying Highly instrumented GemFire Real-Time Events In memory complex event processing (CEP) solution that facilitates agile, event-driven architectures by analyzing streams of data in conjunction with other static/historical data sources JDBC and SQL based, with Continuous Queries Client side materialized view Messaging paradigm with connected and disconnect clients Highly instrumented
  • 6.
    EDF - WhereDoes it Fit? Mainframes Historical Data Customer Data Enterprise Data Fabric Distributed Caching High Availability Custom Application Data Sources Event Processing Data distribution Data Modeling Data Access Application Access High Availability Data Management Data Distribution Analytics Portals Execution Reporting SQL XML Java C++/ C# Grid Computing SOA EDA
  • 7.
    What is GemFireEDF - Key Features? Virtualize data and events from many sources Manage and partition data in distributed main-memory for blazing performance Provide data access for applications in multiple formats to support heterogeneity High-speed data distribution (TCP/IP, Multicast) across multiple nodes - Move data on demand Replicate data to mirror nodes synchronously for high availability Overflow or persist to disk to scale to large data volumes and for data recovery Standards-based Querying and Continuous Querying on structured data and streaming data Distributed transactions for reliable operations Event Streams ENTERPRISE DATA FABRIC Analytics Portals Execution Reporting Query SQL XML Java C++/ C# Disk Grid Computing SOA Event-Processing
  • 8.
    SOA –Composite Applications GEMFIRE ENTERPRISE DATA FABRIC Java, C/C++/C#, XML:DB, SOAP Interfaces Transactions, Logging, Authentication, Discovery Session Management, Entitlements ESB Business Process Composite Applications: e.g,, Web Portals File systems Data Warehouses Databases Custom Application Data Sources Process State Authentication Session/Process State Management Reference data access XML Data Transformation High Availability Data Management Data Distribution
  • 9.
    SOA -Composite Processes Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N Authentication Session State Management Reference data access XML Data Transformation Web Service 1 Web Service 2 Web Service 3 Web Service N GemFire Enterprise
  • 10.
    SOA -Authentication Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N Web Service 1 Web Service 2 Web Service 3 Web Service N Identify and Authenticate - Get Authorizations (IA&A) Travel over network Calls Policy Server/LDAP repository Calls RDBMS Repeat for each step in application flow Often more time than actual service step GemFire eliminates redundant lookups This is also used for Authentication Management for Enterprise Portals GemFire Enterprise GFE
  • 11.
    SOA -Session/Process State Management Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N Web Service 1 Web Service 2 Web Service 3 Web Service N In SOA process state is either Returned as XML between steps Often stored by workflow manager in RDBMS Stored in RDBMS by state aware services GemFire removes latency introduced by network and RDBMS This is also used for Session State Management for Enterprise Portals GemFire Enterprise GemFire
  • 12.
    SOA –Reference Data Service calls back-end data source to look up same information over and over GemFire removes need to make redundant calls to back end Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N GemFire Enterprise Web Service 1 Web Service 2 Web Service 3 Web Service N
  • 13.
    SOA –XML data transformation Service returns same information over and over (for a period of time) If it is called with the same request GemFire removes need to retrieve data from back end, and transform it into XML. Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N GemFire Enterprise Web Service 1 Web Service 2 Web Service 3 Web Service N
  • 14.
    SOA –XML data transformation Service returns same information over and over (for a period of time) If it is called with the same request GemFire removes need to retrieve data from back end, and transform it into XML. GemFire also provides Optimized XML data representation Less Memory Less Java GC/overhead Optimized XML data access Web Service 1 Composite Application Flow User/Process Web Service 2 Web Service 3 Web Service N GemFire Enterprise Web Service 1 Web Service 2 Web Service 3 Web Service N
  • 15.
    GemFire Design PatternsApplication Survivability and Network Usage Reduction App Server Web Services Applications App Server or SOA applications Wide Area Network Wireless App Server GemFire Cache Web Browsers PDAs And Wireless Devices GemFire Cache
  • 16.
    GF Real TimeEvents Functional Overview GemFire Real-Time Events Real-Time Data Streams Continuously analyze (execute continuous queries on event streams) Identify patterns of interest through correlation with other data Selectively notify systems and personnel of relevant changes and updates in real-time Financial Markets Data Battlefield Information RFID Data Wireless Network Data Register Continuous Queries
  • 17.
    Summary Highlights Solvenumerous challenges in SOA and distributed systems environments Enhance Scalability, Performance, Reliability of systems Potentially reduce hardware and software costs Easy to insert into new and existing systems Highly Available, Highly instrumented, Highly configurable NSA Acquisition Security approved
  • 18.
    EDF - KeyTechnical Concepts GemStone Systems Inc. This presentation contains information that is CONFIDENTIAL to GemStone Systems Inc. Do not distribute or copy this presentation without express written permission.
  • 19.
    GemFire Enterprise GemFireEnterprise A scalable and high-performance data infrastructure that Caches and distributes data in multiple formats, across distributed applications Connects to backend systems like databases, messaging systems, etc. Provides high data availability mechanisms Supports distributed transactions and standards-based querying Offers interfaces for C/C++/C# applications as well Target Environments Java/J2EE based platforms XML/Web Services
  • 20.
    GemFire Enterprise --C++ GemFire Enterprise -- C++ An extremely high throughput and performance data infrastructure that Native C++ bindings to avoid inefficiencies Highly parallel distribution with optimized transport Avoids unnecessary user-kernel crossings with no contention or context-switching Connects to backend systems like databases, messaging systems, etc. Provides high data availability mechanisms Offers interfaces for Java applications as well (future) Target Environments C++ -based application environments Performance and throughput are primary concerns
  • 21.
    GemFire Real-Time EventsGemFire Real-Time Events GemFire Real-Time Events is a complex event processing (CEP) solution that enables an enterprise to facilitate agile, event-driven architectures by analyzing streams of data in conjunction with other static/historical data sources Define relevant business events through simple SQL queries Discern patterns through continuous querying and event correlation with other data sources Distribute appropriate information to selective client applications Target Environments Fast-moving data streams Real-time pattern detection and analytics
  • 22.
    Enterprise Data Fabric- What is it ? Middle tier Distributed Data Infrastructure Operational Data Combines Distributed Caching (main-memory), DB semantics, Reliable Messaging/ Intelligent Routing and Continuous Analytics Distributed Data Space Data warehouses Applications Rational databases
  • 23.
    Where is thedata in the Fabric? All across the Network NETWORK IS THE DATABASE Cache Cache Cache In Local process memory Copies in multiple nodes
  • 24.
    Enterprise Data Fabric- Physical View All across the Network NETWORK IS THE DATABASE Cache Cache Cache Rational databases Clustered Application Spread across many nodes On Disk Or lazily fetched from external source
  • 25.
    5 Key Features1. Data Storage Optimized concurrent main-memory structures Topology Embedded in application process Client caches and Cache servers Partitioned across many servers n-way in-memory replication for HA On Disk Overflow to disk Recoverable disk regions – Asynchronous or Synchronous
  • 26.
    5 Key Features2. Access Model Object Caching API (get()/put()). Querying through Xpath, OQL and SQL XML /Web Services (HTTP/SOAP) APIs in Java, C++, and C# JTA Transactions Management APIs – JMX
  • 27.
    5 Key Features3. Data Distribution Model Consistency Model Distribution without ACKs With ACKs With global locking Reliable Pub-Sub semantics Novel declarative “role” based model to ensure message delivery Dealing with Slow subscribers, net/node failures, etc Multiple transports TCP, Reliable UDP MultiCast
  • 28.
    5 Key Features4. Integration with External Data Sources A Framework for read-through, write-through, and write-behind Custom Connectors link to DBMS, Middleware Messaging, EII solutions Components - Loaders, Writers and Listeners
  • 29.
    5 Key FeaturesDealing with high event rate RFID streaming data, etc Many clients can subscribe to portions of the Data Fabric Express interest using complex query CQ engine continuously analyses data events Continuously calculates how the query view has been impacted Ships “delta” and merges into client result set. Notifications delivered with guaranteed latency threshold 5. Continuous Analytics