In Memory Data Grids in Action with Oracle Coherence presented to No SQL users.
The "transactions" chapter is missing as it has been rescheduled to another session.
Creating an RAD Authoratative Data Environmentanicewick
Sharing data in agencies can be a burden, with users placing data on numerous desktop packages, the idea of sharing becomes impossible. However, new RAD tools allow quick web applications to be developed to replace the Excel, MSAcces, and Filemaker data stores, with real , controlled authoritative database integration.
This presentation defines both the problem space, and the proposed solution.
See www.data4USA.com for more information
Engineered Systems: Oracle’s Vision for the FutureBob Rhubart
Oracle’s Exadata and Exalogic are impressive products in their own right. But working in combination they deliver unparalleled transaction processing performance with up to a 30x increase over existing legacy systems, with the lowest cost of ownership over a 3 or 5 year basis than any other hardware. In this session you’ll learn how to leverage Oracle’s Engineered Systems within your enterprise to deliver record-breaking performance at the lowest TCO.
(As presented by Kelly Goetsch at
Creating an RAD Authoratative Data Environmentanicewick
Sharing data in agencies can be a burden, with users placing data on numerous desktop packages, the idea of sharing becomes impossible. However, new RAD tools allow quick web applications to be developed to replace the Excel, MSAcces, and Filemaker data stores, with real , controlled authoritative database integration.
This presentation defines both the problem space, and the proposed solution.
See www.data4USA.com for more information
Engineered Systems: Oracle’s Vision for the FutureBob Rhubart
Oracle’s Exadata and Exalogic are impressive products in their own right. But working in combination they deliver unparalleled transaction processing performance with up to a 30x increase over existing legacy systems, with the lowest cost of ownership over a 3 or 5 year basis than any other hardware. In this session you’ll learn how to leverage Oracle’s Engineered Systems within your enterprise to deliver record-breaking performance at the lowest TCO.
(As presented by Kelly Goetsch at
PHP Development In The Cloud (php|tek edition)Ivo Jansch
This talk is the companion talk to the 'PHP Development in the Cloud' book I wrote with Vito Chin. It's an introduction into cloud computing for PHP developers. Delivered on may 25 at php|tek in Chicago.
Which Problems Does a Multi-Language Virtual Machine Need to Solve in the Mul...Stefan Marr
While parallel programming for very regular problems has been used in the scientific community by non-computer-scientists successfully for a few decades now, concurrent programming and solving irregular problems remains hard. Furthermore, we shift from few expert system programmers mastering concurrency for a constrained set of problems to mainstream application developers being required to master concurrency for a wide variety of problems.
Consequently, high-level language virtual machine (VM) research faces interesting questions. What are processor design changes that have an impact on the abstractions provided by VMs to provide platform independence? How can application programmers’ diverse needs be facilitated to solve concurrent programming problems?
We argue that VMs will need to be ready for a wide range of different concurrency models that allow solving concurrency problems with appropriate abstractions. Furthermore, they need to abstract from heterogeneous processor architectures, varying performance characteristics, need to account for memory access cost and inter-core communication mechanisms but should only expose the minimal useful set of notions like locality, explicit communication, and adaptable scheduling to maintain their abstracting nature.
Eventually, language designers need to be enabled to guarantee properties like encapsulation, scheduling guarantees, and immutability also when an interaction between different problem-specific concurrency abstractions is required.
Engineered Systems: Oracle's Vision for the FutureBob Rhubart
Oracle's Exadata and Exalogic are impressive products in their own right. But working in combination they deliver unparalleled transaction processing performance with up to a 30x increase over existing legacy systems, with the lowest cost of ownership over a 3 or 5 year basis than any other hardware. In this session you'll learn how to leverage Oracle's Engineered Systems within your enterprise to deliver record-breaking performance at the lowest TCO.
Open a window, see the clouds - php|tek 2011Rafael Dohms
Is your application ready to take off into the cloud? What does this mean? What do you need to worry about? What frameworks and tools do you need to look at and what new concepts do you need to put in your PHP toolbox to prepare? Get acquainted with Microsoft Azure's platform and features, and discover what tools and frameworks are ready to benefit from this new paradigm.
Netapp Evento Virtual Business Breakfast 20110616Bruno Banha
Apresentação efectuada pela Netapp no evento Virtual Business Breakfast, realizado no dia 16 de Junho de 2011, no Porto.
O evento realizado em conjunto com a VMware e a NextiraOne Portugal.
Join-fu is the art of performance-tuning your application's SQL. Join Jay in a fun, irreverent look at the common ways application developers misuse and abuse their database.
Embracing Observability in CI/CD with OpenTelemetryCyrille Le Clerc
Discover how observability and OpenTelemetry offer unprecedented solutions for both CI/CD administrators and dev teams to troubleshoot CI platforms and solve much more problems thanks to a vibrant community and a growing ecosystem. We will see with real life CI/CD pipelines using Jenkins, Maven, and Ansible how OpenTelemetry offers unprecedented solutions to troubleshoot software delivery pipelines. How the open source and standard nature of OpenTelemetry enables the emergence of a vibrant ecosystem of OpenTelemetry aware CI/CD tools to observe the entire software supply chain and help DevOps teams solve problems that go way beyond the observability use cases we have in mind.
https://community.cncf.io/events/details/cncf-cloud-native-canada-presents-november-2021-eastern-canadian-cncf-meetup-kubernetes-123-release-update-and-cicd-observability/
Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)Cyrille Le Clerc
Fast feedback from monitoring is a key of Continuous Delivery. JMX is the right Java API to do so but it unfortunately stayed underused and underappreciated as it was difficult to connect to monitoring and graphing systems.
Throw in the sin bin the poor solutions based on log files and weakly secured web interfaces! A new generation of Open Source tooling makes it easy to graph java application metrics and integrate them to traditional monitoring systems like Nagios.
Following the logic of DevOps, we will look together how best to integrate the monitoring dimension in a project: from design to development, to QA and finally to production on both traditional deployment and in the Cloud.
Come and discover how the JmxTrans-Graphite ticket can make your life easier.
More Related Content
Similar to Paris NoSQL User Group - In Memory Data Grids in Action (without transactions chapter)
PHP Development In The Cloud (php|tek edition)Ivo Jansch
This talk is the companion talk to the 'PHP Development in the Cloud' book I wrote with Vito Chin. It's an introduction into cloud computing for PHP developers. Delivered on may 25 at php|tek in Chicago.
Which Problems Does a Multi-Language Virtual Machine Need to Solve in the Mul...Stefan Marr
While parallel programming for very regular problems has been used in the scientific community by non-computer-scientists successfully for a few decades now, concurrent programming and solving irregular problems remains hard. Furthermore, we shift from few expert system programmers mastering concurrency for a constrained set of problems to mainstream application developers being required to master concurrency for a wide variety of problems.
Consequently, high-level language virtual machine (VM) research faces interesting questions. What are processor design changes that have an impact on the abstractions provided by VMs to provide platform independence? How can application programmers’ diverse needs be facilitated to solve concurrent programming problems?
We argue that VMs will need to be ready for a wide range of different concurrency models that allow solving concurrency problems with appropriate abstractions. Furthermore, they need to abstract from heterogeneous processor architectures, varying performance characteristics, need to account for memory access cost and inter-core communication mechanisms but should only expose the minimal useful set of notions like locality, explicit communication, and adaptable scheduling to maintain their abstracting nature.
Eventually, language designers need to be enabled to guarantee properties like encapsulation, scheduling guarantees, and immutability also when an interaction between different problem-specific concurrency abstractions is required.
Engineered Systems: Oracle's Vision for the FutureBob Rhubart
Oracle's Exadata and Exalogic are impressive products in their own right. But working in combination they deliver unparalleled transaction processing performance with up to a 30x increase over existing legacy systems, with the lowest cost of ownership over a 3 or 5 year basis than any other hardware. In this session you'll learn how to leverage Oracle's Engineered Systems within your enterprise to deliver record-breaking performance at the lowest TCO.
Open a window, see the clouds - php|tek 2011Rafael Dohms
Is your application ready to take off into the cloud? What does this mean? What do you need to worry about? What frameworks and tools do you need to look at and what new concepts do you need to put in your PHP toolbox to prepare? Get acquainted with Microsoft Azure's platform and features, and discover what tools and frameworks are ready to benefit from this new paradigm.
Netapp Evento Virtual Business Breakfast 20110616Bruno Banha
Apresentação efectuada pela Netapp no evento Virtual Business Breakfast, realizado no dia 16 de Junho de 2011, no Porto.
O evento realizado em conjunto com a VMware e a NextiraOne Portugal.
Join-fu is the art of performance-tuning your application's SQL. Join Jay in a fun, irreverent look at the common ways application developers misuse and abuse their database.
Embracing Observability in CI/CD with OpenTelemetryCyrille Le Clerc
Discover how observability and OpenTelemetry offer unprecedented solutions for both CI/CD administrators and dev teams to troubleshoot CI platforms and solve much more problems thanks to a vibrant community and a growing ecosystem. We will see with real life CI/CD pipelines using Jenkins, Maven, and Ansible how OpenTelemetry offers unprecedented solutions to troubleshoot software delivery pipelines. How the open source and standard nature of OpenTelemetry enables the emergence of a vibrant ecosystem of OpenTelemetry aware CI/CD tools to observe the entire software supply chain and help DevOps teams solve problems that go way beyond the observability use cases we have in mind.
https://community.cncf.io/events/details/cncf-cloud-native-canada-presents-november-2021-eastern-canadian-cncf-meetup-kubernetes-123-release-update-and-cicd-observability/
Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)Cyrille Le Clerc
Fast feedback from monitoring is a key of Continuous Delivery. JMX is the right Java API to do so but it unfortunately stayed underused and underappreciated as it was difficult to connect to monitoring and graphing systems.
Throw in the sin bin the poor solutions based on log files and weakly secured web interfaces! A new generation of Open Source tooling makes it easy to graph java application metrics and integrate them to traditional monitoring systems like Nagios.
Following the logic of DevOps, we will look together how best to integrate the monitoring dimension in a project: from design to development, to QA and finally to production on both traditional deployment and in the Cloud.
Come and discover how the JmxTrans-Graphite ticket can make your life easier.
Monitoring Open Source pour Java avec JmxTrans, Graphite et Nagios - DevoxxFR...Cyrille Le Clerc
Le feedback rapide offert par le monitoring est un element essentiel des bonnes pratiques de Continuous Delivery. Java dispose dans son ecosysteme d'un composant robuste dedie a cela : JMX.
Cependant, la difficulte de raccordement de JMX a des outils de supervision et de graphe a longtemps ete un frein a son adoption.
Jetez aux orties les solutions bancales a base de logs applicatifs ou d'interface web mal protegees, et venez decouvrir une voie ouverte. Une nouvelle generation d'outils Open Source permet de grapher simplement les metriques de vos applications et de les fournir a un systeme de supervision et d'alerte.
Dans une logique DevOps, nous verrons ensemble comment integrer la dimension Monitoring dans un projet : de la conception des metriques par les developpeurs, a l'integration des besoins des equipes Ops et Q&A, en deploiement traditionnel ou dans le Cloud. JmxTrans, Graphite et Nagios, ce tryptique peut vous faciliter la vie, venez decouvrir comment.
L'application demo : http://demo-cocktail.jmxtrans.cloudbees.net
Le code source de l'application demo : https://github.com/jmxtrans/embedded-jmxtrans-samples/tree/master/embedded-jmxtrans-webapp-coktail
Embedded JmxTrans : https://github.com/jmxtrans/embedded-jmxtrans
Bonnes pratiques des applications java prêtes pour la productionCyrille Le Clerc
Les bonnes pratiques des applications Java prêtes pour la production.
Les enjeux :
* Améliorer la disponibilité des applications
* Réduire le cycle de vie des projets
* Améliorer les plateformes
* Diminuer le coût d’exploitation
Les axes clefs :
* Le déploiement
* La supervision et le monitoring
* La gestion des logs
* La robustesse
* L’organisation
Cyrille Le Clerc (Xebia), Erwan Alliaume (Xebia), Jean Michel Bea (Fast Connect) ont présenté au Paris Java User Group les principes du Data Grid.
Cache distribué, Network Attached Memory, Data Grid ou Cloud Computing sont des termes très à la mode qui s’inscrivent dans la même tendance.
Nous présenterons pendant cette soirée le chemin qui nous à conduit d’un simple EH Cache à des grilles de centaines de giga octects de données qui s’étalent sur des data center.
CACHES DISTRIBUES
Les Cache Distribués se sont banalisés avec les frameworks Open Source Jboss Cache et EH Cache distribué. Où en sommes nous aujourd’hui ?
- Quels sont les cas d’utilisation d’un cache distribué ? Quels gains en attendre ?
- Comment migrer d’un cache local à un cache distribué ? Nos frameworks sont-ils adaptés à ces caches distribués ?
- Comment fonctionne un cache distribué ?
NETWORK ATTACHED MEMORY
Le concept de Network Attached Memory a décollé dans l’univers Java avec Terracotta et offre à nos applications un espace mémoire encore inimaginable il y a peu. Que se cache-t-il derrière ?
- Quels sont les cas d’utilisation des technologies de Network Attached Memory ?
- Cette mémoire virtuellement infinie n’introduit-elle pas des contraintes ?
- Si la mémoire est partagée, qu’en est-il des traitements ?
- Quelles sont les perspectives des technologies de Network Attached Memory ?
DATA GRID
Le concept de data grid s’est popularisé avec les services Google Big Table ou Amazon S3 mais aussi avec des sites comme eBay qui annoncent des data center gigantesques. Cela va-t-il arriver dans l’informatique classique ?
- Qu’est-ce qu’une grille de donnée ? Comment ça marche ?
- Qui a besoin de Data Grid ? Est-ce réservé aux hyper scalable comme eBay ou Facebook ? Comment faisait-on avant ? En ai-je besoin ?
- Comment structurer une application pour utiliser une grille de données ? Cela la change-t-il la façon de programmer ?
- Map Reduce est-il un pattern utilisable avec une grille de données ? Est-ce le seul ?
- Les grilles de données vont-elles remplacer les bases de données traditionnelles ? Comment peuvent-elles cohabiter ?
DATA GRID, CLOUD ET LES AUTRES
Data Grid, Grid Computing, Cloud Computing et eXtreme Transaction Processing (XTP) sont fréquement associés.
Comment positionner Data Grid par rapport à ces technologies ?
Quels positionnements ont les acteurs de cet univers ? Amazon S3&EC2 ? Coherence ? Gigaspace ? Google App engine & Big Table ? Grid Gain ? Terracotta ? Websphere eXtreme Scale ?
Et les mainframes dans tout ça ?
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Key Trends Shaping the Future of Infrastructure.pdf
Paris NoSQL User Group - In Memory Data Grids in Action (without transactions chapter)
1. Transactions chapter will be presented
during another session
In Memory Data Grid in Action
with Oracle Coherence
for Paris NoSQL User Group
Cyrille Le Clerc
Wednesday, May 25, 2011
2. Speaker
@cyrilleleclerc
blog.xebia.fr
Cyrille Le Clerc
Large Scale
In Memory Data Grid
Open Source
(Apache CXF, ...)
“you build it, you run it”
2
Wednesday, May 25, 2011
4. On the Financial side
- Released Coherence in 2001
Needs within financial market :
- Started as a distributed cache
• Very low latency
• Rich queries & transactions
• Scalability
- Released Gigaspaces XAP in 2001
• Data consistency - Started as a data grid
4
Wednesday, May 25, 2011
5. Let’s define an In Memory Data Grid ...
5
Wednesday, May 25, 2011
6. Let’s define an In Memory Data Grid
eXtreme Scale
This is an In Memory Data Grid
6
Wednesday, May 25, 2011
7. Let’s define an In Memory Data Grid
This is Network Attached Memory
7
Wednesday, May 25, 2011
8. Let’s define an In Memory Data Grid
Similarities with NoSQL document oriented
Partitioned, distributed Hastable, schema-less, value is not opaque,
scale-out scalability
Very fast
In memory (persistence coming), business logic inside the data
Consistent and Available
Transactional, redundant
Written in Java, data are POJOs
Not necessary
Clients in Java, Microsoft, etc
8
Wednesday, May 25, 2011
9. Use cases for this presentation
9
Wednesday, May 25, 2011
10. Train Booking System
trains, stations,
seats, booking and
passengers
10
Wednesday, May 25, 2011
18. Data Access Patterns
This is not traditional Java EE coding style !
Can apply very complex business logic inside the
data
Stored Procedures Style
Change management challenge !
18
Wednesday, May 25, 2011
25. Data Access Patterns
This is not traditional Java EE coding style
Change management
Don’t forget “Map Reduce” = “Distributed Table
Scan”
Use Indexes
25
Wednesday, May 25, 2011
26. CAP Theorem & In Memory Data Grids
26
Wednesday, May 25, 2011
27. CAP Theorem and In Memory Data Grid
Only 2 of these 3
properties can be
Consistency achieved at any given
moment in time
Brewer’s Conjecture
Availability
Partition
Tolerance
http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf
27
Wednesday, May 25, 2011
28. CAP Theorem and In Memory Data Grid
Data Grids Only 2 of these 3
properties can be
Consistency achieved at any given
moment in time
Brewer’s Conjecture
Availability
Partition
Tolerance
http://lpd.epfl.ch/sgilbert/pubs/BrewersConjecture-SigAct.pdf
28
Wednesday, May 25, 2011
29. Cross Data Center Data Consistency
London
New York
Tokyo
World wide replication
for financial market
29
Wednesday, May 25, 2011
30. Cross Data Center Data Consistency
{
"name": "Barbie Computer",
"stock": 147,
"weigth" : 200
}
{
"name": "Barbie Computer",
West Coast "stock": 147,
"weigth" : 200
}
East Coast
Warehouse stocks
30
Wednesday, May 25, 2011
31. Cross Data Center Data Consistency
set stock to 146
{
"name": "Barbie Computer",
"stock": 147,
"weigth" : 200
}
{
"name": "Barbie Computer",
West Coast "stock": 147,
"weigth" : 200
}
East Coast
propagation delay !
31
Wednesday, May 25, 2011
32. Cross Data Center Data Consistency
set stock to 146
{
"name": "Barbie Computer",
"stock": 147,
"weigth" : 200
}
{
"name": "Barbie Computer",
West Coast "stock": 147,
"weigth" : 200
}
East Coast
set weight 175
reconciliation API needed !
32
Wednesday, May 25, 2011
33. Cross Data Center Data Consistency
set stock to 146
{
"name": "Barbie Computer",
"stock": 147,
"weigth" : 200
}
{
"name": "Barbie Computer",
West Coast "stock": 147,
"weigth" : 200
}
East Coast
set weight 175
Network partitioning
33
Wednesday, May 25, 2011
35. Data Modeling
Dominant Question Driven Design
Opposite to Relational which is Domain Driven Design
Constrained Tree Schema
Because RPC matters
Denormalized
Due to dominant questions and CTS
35
Wednesday, May 25, 2011
36. Data Modeling
Seat
Booking Passenger
number
reduction name
price
Train
code
type
TrainStation
TrainStop
code
date
name
Typical relational data model
36
Wednesday, May 25, 2011
37. Data Modeling
Partitioning ready
entities tree
e ntity
Root Seat
Booking Passenger
number
reduction name
price
Train
code
type Du R
pli
ca efer
ted en
TrainStation in ce d
ea
TrainStop ch ata
code gri
date dn
name od
e
Find the root entity and denormalize
37
Wednesday, May 25, 2011
38. Data Modeling
Remove unused data
Seat
Booking Passenger
number
reduction name
price
booked
Train
code
type
TrainStation
TrainStop
code
date
name
Partitioned
Replicated
38
Wednesday, May 25, 2011
39. Data Modeling
Seat
number
price
booked
Train
code
type
TrainStation
TrainStop
code
date
name
Partitioned
Replicated
Data Grid Ready data structure
39
Wednesday, May 25, 2011
41. Data Modeling is Hard !
Account Account
number number
from to
CashWitdrawal MoneyTransfer CashWitdrawal
date id date
amount date amount
amount
Two root entities for the
same MoneyTransfer !
41
Wednesday, May 25, 2011
42. Data Modeling is Hard !
Account Account
number number
CashWitdrawal MoneyTransferIn MoneyTransferOut CashWitdrawal
date id id date
amount date date amount
amount amount
Split MoneyTransfer
42
Wednesday, May 25, 2011
43. Data Modeling is Hard !
Account Account
number number
CashWitdrawal MoneyTransferIn MoneyTransferOut CashWitdrawal
date id id date
amount date date amount
amount amount
Split MoneyTransfer
43
Wednesday, May 25, 2011
44. Data Modeling is Hard !
Account
number
CashWitdrawal MoneyTransferOut MoneyTransferIn
date id id
amount date date
amount amount
Data Grid Ready data structure
44
Wednesday, May 25, 2011
46. Data Serialization
Used for data transfer and byte oriented storage
Must support evolvable data structure
Hot topic like Apache Thrift, Apache Avro, Google
Protocol Buffer
46
Wednesday, May 25, 2011
47. Data Storage
Store Java Beans in the grid
No need to unmarshall for inprocess operations
Beware of garbage collector !
Store byte arrays in the grid
Pay unmarshalling at each read and write
Low-level / byte-oriented APIs to read data
Slightly more garbage collector friendly
47
Wednesday, May 25, 2011
48. Communication Protocols
UDP Multi Cast (Coherence, Gigaspaces)
TCP/IP (Websphere eXtreme Scale)
48
Wednesday, May 25, 2011
49. Topology
Partitions made of shards : 1 primary + 0..*
backups)
Dynamic shards location (changes at runtime and
at restart)
Can use dedicated “directory servers” or embed it
in the “data nodes”
49
Wednesday, May 25, 2011
50. JVM and Memory
Many editors recommend tiny 1.4 Go JVM !
Garbage collector hell
More than ten JVM per server
Management hell
More and more IMDG support large heaps
50
Wednesday, May 25, 2011
52. Raw Java Mapping with Oracle Coherence
public class Train extends AbstractEvolvable implements PortableObject {
enum Type {
HIGH_SPEED, NORMAL
}
/** Key of the Cache */
String code;
/** Indexed */ Seat
String name;
number
Type type;
price
booked
List<Seat> seats = new ArrayList<Seat>(); Train
code
int version; type
List<TrainStop> trainStops = new ArrayList<TrainStop>();
TrainStop
@Override date
public int getImplVersion() {
return 1;
}
@Override
public void readExternal(PofReader pofReader) throws IOException {
this.code = pofReader.readString(0);
this.name = pofReader.readString(1);
this.type = (Type) pofReader.readObject(2);
pofReader.readCollection(3, this.seats);
pofReader.readCollection(4, this.trainStops);
this.version = pofReader.readInt(5);
hand-coded serialization
}
@Override
JUnit is your friend !
public void writeExternal(PofWriter pofWriter) throws IOException {
pofWriter.writeString(0, this.code);
pofWriter.writeString(1, this.name);
pofWriter.writeObject(2, this.type);
pofWriter.writeCollection(3, this.seats, Seat.class);
pofWriter.writeCollection(4, this.trainStops, TrainStop.class);
pofWriter.writeInt(5, this.version);
}
}
52
Wednesday, May 25, 2011
53. JPA Style Mapping with Websphere eXtreme
Scale
@Entity(schemaRoot=true)
public class Train { Seat
number
price
@Id
booked
String code; Train
code
@Index type
@Basic
TrainStop
String name;
date
@OneToMany(cascade=CascadeType.ALL)
List<Seat> seats = new ArrayList<Seat>();
@Version
int version;
...
}
sub entities can have
cross relations
53
Wednesday, May 25, 2011
54. Map API with Oracle Coherence
NamedCache trainCache = CacheFactory.getCache("train-cache");
/** Save */
void persist(Train train) {
trainCache.put(train.getCode(), train);
}
/** Find by key */
Train findByCode(String code) {
return (Train) trainCache.get(code);
}
/** Find by Query Language */
Train findByTrainName(String name) {
Filter filter = QueryHelper.createFilter("name = :name" ,
Collections.singletonMap("name", name));
Set<Map.Entry<String, Train>> trainEntrySet = trainCache.entrySet(filter);
if (trainEntrySet.isEmpty()) {
return null;
} else {
return trainEntrySet.iterator().next().getValue();
}
}
Map API
54
Wednesday, May 25, 2011
55. JPA Style with Websphere eXtreme Scale
/** Save */
void persist(Train train) {
entityManager.persist(train);
}
/** Find by key */
Train findByCode(String code) {
return (Train) entityManager.find(Train.class, code);
}
/** Query Language */
Train findByTrainName(String name) {
Query q = entityManager.createQuery("select t from Train t where t.name=:name");
q.setParameter("name", name);
return (Train) q.getSingleResult();
}
JPA Style Entity Manager
55
Wednesday, May 25, 2011
58. Indexes with Websphere eXtreme Scale
@Entity(schemaRoot=true)
class Train {
@Index
@Basic eXtreme Scale
String name;
@Index
Collection<String> getTrainStationsCodes() {
return Collections2.transform(trainStops, ...);
}
...
}
Query query = em.createQuery("select t from Train t where t.name=:name");
query.getPlan();
This is an execution plan
for q2 in Train ObjectMap using INDEX on name = ( ?name)
filter ( q2.c[0] = ?name )
returning new Tuple( q2 )
58
Wednesday, May 25, 2011
59. More APIs
Another Java EE versus Spring battle ?
JSR 347 Data Grids vs. Spring Data
Serialization / Object to Tuple Mapping API ?
Unified API ontop of NoSQL stores ?
59
Wednesday, May 25, 2011
60. Data Grid <-> Relational Database Interactions
60
Wednesday, May 25, 2011
61. Data Grid <-> Relational Database
Data Grids are “In Memory” -> we need to persist data on disk !
61
Wednesday, May 25, 2011
62. Data Grid <-> Relational Database
update / insert / delete
“select directly modified in DB”
62
Wednesday, May 25, 2011
63. Data Grid <-> Relational Database
Data Grid -> Relational Database
backend DB
Highly available write behind queues
+ SQL batched statements
63
Wednesday, May 25, 2011
64. Data Grid <-> Relational Database
Data Grid -> Relational Database
Seat
number
price
booked
Train
code
type
TrainStation
TrainStop
code
date
name
Constrained Tree Schema <-> Relational
Impedance Mismatch
64
Wednesday, May 25, 2011
65. Data Grid <-> Relational Database
DB writes MUST succeed !
Prefer raw SQL rather than reused business logic
Denormalize the database
Remove the foreign keys, use same PKs in DB and data grid
Support unordered SQL statements
Align the database on the Data Grid model !
65
Wednesday, May 25, 2011
66. Data Grid <-> Relational Database
Relational Database -> Data Grid
select * from train
where last_modif > ?
backend DB
Data Grid Originated Scheduled Refresh
(Oracle System Change Number, etc)
66
Wednesday, May 25, 2011
67. Data Grid <-> Relational Database
Relational Database -> Data Grid
backend DB
Database Originated Push
JMS = durable subscription
(Oracle Database Change Notification, etc)
67
Wednesday, May 25, 2011
68. Data Grid <-> Relational Database
In Memory -> prepare for reloading after
maintenance operations !
Need for “graceful shutdown with disk persistence”
Prepare consistency checkers
68
Wednesday, May 25, 2011
72. Data Grids and Operations
Standard packaging?
Do It Yourself (layout, scripts, etc)
Limited Management
Do It Yourself (stop/start, detecting data loss, etc)
Limited debugging tools
Do It Yourself (debugging consoles, troubleshooting agents)
JVM pandemia
Dozens of JVM to manage !
72
Wednesday, May 25, 2011
73. Data Grids and Operations
Dev / Ops collaboration is required
Experts only !
73
Wednesday, May 25, 2011
74. The right tool for the right job
74
Wednesday, May 25, 2011
75. The right tool for the right job
Incredibly fast
Scalable
If you solve the data loading issue
Good at data replication (when it implements it)
Reconciliation api, etc
Very geeky on both dev and ops side
Not an enterprise grade data store
Requires very skilled people + change management
“Quite” expensive
75
Wednesday, May 25, 2011