Clustered Architecture Patterns: Delivering Scalability and Availability Qcon San Francisco, 2007 Ari Zilka

This Session "Clustered Architecture Patterns: Delivering Scalability and Availability" Solution Track: Architecting for Performance & Scalability Wednesday 14:30 - 15:30

"Clustered Architecture Patterns: Delivering Scalability and Availability"

Solution Track: Architecting for Performance & Scalability

Wednesday 14:30 - 15:30

Agenda The Clustering Patterns Network Attached Memory / JVM-level clustering Database and Developer Abuse Use Case #1: Hibernate disconnected mode Use Case #2: Event-based architecture Lessons Learned

The Clustering Patterns

Network Attached Memory / JVM-level clustering

Database and Developer Abuse

Use Case #1: Hibernate disconnected mode

Use Case #2: Event-based architecture

Lessons Learned

The Clustering Patterns Types of clustering: Load-balanced Scale Out Partitioned Scale Out Both Trade-off Scalability or availability (usually by hand) in different ways Simplicity tends to get lost because frameworks rely on replication Copy-on-read Copy-on-write Examples: serialization, Hibernate, JMS

Types of clustering:

Load-balanced Scale Out

Partitioned Scale Out

Both Trade-off Scalability or availability (usually by hand) in different ways

Simplicity tends to get lost because frameworks rely on replication

Copy-on-read

Copy-on-write

Examples: serialization, Hibernate, JMS

Changing the Assumptions: JVM-level Clustering

JVM-level Clustering Addresses the Patterns SIMPLE Honors Language Spec across JVM boundaries Transparent to source code Thread Coordination too SCALABLE Virtual Heap Read from Heap Write deltas, only where needed AVAILABLE Persist to disk at wire speed Active / Passive and Active / Active strategies Models Load balanced stays naïve Partitioned stays POJO (abstractions are easy) Models Load balanced scales through implicit locality Partitioned scales by avoiding data sharing Models Load balanced apps made available by writing heap to disk Partitioned made available by using the cluster to store workflow “ILITIES” Scale Out Model

SIMPLE

Honors Language Spec across JVM boundaries

Transparent to source code

Thread Coordination too

SCALABLE

Virtual Heap

Read from Heap

Write deltas, only where needed

AVAILABLE

Persist to disk at wire speed

Active / Passive and Active / Active strategies

Models

Load balanced stays naïve

Partitioned stays POJO (abstractions are easy)

Models

Load balanced scales through implicit locality

Partitioned scales by avoiding data sharing

Models

Load balanced apps made available by writing heap to disk

Partitioned made available by using the cluster to store workflow

The foundation for all this thinking… At Walmart.com we started like everyone else: stateless + load-balanced + Oracle (24 cpus, 24GB) Grew up through distributed caching + partitioning + write behind We realized that “ilities” conflict Scalability: avoid bottlenecks Availability: write to disk ( and I/O bottleneck ) Simplicity: No copy-on-read / copy-on-write semantics (relentless tuning, bug fixing) And yet we needed a stateless runtime for safe operation Start / stop any node regardless of workload Cluster-wide reboot needed to be quick; could not wait for caches to warm The “ilities” clearly get affected by architecture direction and the stateless model leads us down a precarious path

At Walmart.com we started like everyone else: stateless + load-balanced + Oracle (24 cpus, 24GB)

Grew up through distributed caching + partitioning + write behind

We realized that “ilities” conflict

Scalability: avoid bottlenecks

Availability: write to disk ( and I/O bottleneck )

Simplicity: No copy-on-read / copy-on-write semantics (relentless tuning, bug fixing)

And yet we needed a stateless runtime for safe operation

Start / stop any node regardless of workload

Cluster-wide reboot needed to be quick; could not wait for caches to warm

The “ilities” clearly get affected by architecture direction and the stateless model leads us down a precarious path

The precarious path: These are common-place tools Stateless load-balanced architecture Sticky Session with replicate-only-on-change Clustered DB cache Separate caching server JMS for Async cache update

Stateless load-balanced architecture

Sticky Session with replicate-only-on-change

Clustered DB cache

Separate caching server

JMS for Async cache update

Large publisher gets caught down the path with Oracle Scaling Out or Up?

Breaking the pattern without leaving “Load-Balanced” world $1.5 Million DB & HW savings Doubled business More than halved database load

$1.5 Million DB & HW savings

Doubled business

More than halved database load

Counter Examples Load Balanced Application Publishing Company Happy with availability and simplicity using Hibernate + Oracle Not happy with scalability SOLUTION: Hibernate disconnected mode Partitioned Application Travel Company Happy with MQ-based availability, 4 dependent apps mean no API changes allowed System of Record too expensive to keep scaling SOLUTION: Proxy the System or Record; Partition for scale

Load Balanced Application

Publishing Company

Happy with availability and simplicity using Hibernate + Oracle

Not happy with scalability

SOLUTION: Hibernate disconnected mode

Partitioned Application

Travel Company

Happy with MQ-based availability, 4 dependent apps mean no API changes allowed

System of Record too expensive to keep scaling

SOLUTION: Proxy the System or Record; Partition for scale

Stateless Architecture

Stateless By Hand is Cumbersome and Inefficient Baseline Application 3 User Requests during one Conversation 2 POJO Updates per Request Total DB Load: 9 User Conversation

Baseline Application

3 User Requests during one Conversation

2 POJO Updates per Request

Total DB Load: 9

So We Add Hibernate Add Hibernate Eliminate Direct Connection to the DB via JDBC Eliminate Hand-Coded SQL Eliminate Intermediate POJO Updates Total DB Load: 6 User Conversation

Add Hibernate

Eliminate Direct Connection to the DB via JDBC

Eliminate Hand-Coded SQL

Eliminate Intermediate POJO Updates

Total DB Load: 6

Then We Turn on Caching Serialization is required BLOB Replication requirements are heavy User Conversation Enable 2nd Level cache Eliminates Intermediate Loads Total DB Load: 4

Enable 2nd Level cache

Eliminates Intermediate Loads

Total DB Load: 4

So We Disconnect But Lose Availability Detached POJOs Eliminates Intermediate Commits Total DB Load: 2 Can lose state in case of failure! Replication is expensive Hibernate says to keep graphs small User Conversation

Detached POJOs

Eliminates Intermediate Commits

Total DB Load: 2

JVM-Level Clustering + Hibernate Together Cluster 2nd Level Cache - Hibernate Performance Curve Level 2 EHCache Support Built in to the product Advantages Coherent Cache Across the cluster Easy to integrate with existing applications Performs very well Eliminate the artificial cache misses in clustered environment Disadvantages Objects are represented as BLOBs by Hibernate Doesn’t take direct advantage of Terracotta Scale-Out Features Cluster Detached POJOs - Hibernate Performance Curve Level 3 Cluster Pure POJOs Re-attach Session in the same JVM or a different JVM Advantages Scales the best Take Advantage of POJOs - Fine-grained changes, replicate only where resident Disadvantages Some code changes required to refactor Hibernate’s beginTransaction(), commit()

Cluster 2nd Level Cache - Hibernate Performance Curve Level 2

EHCache Support Built in to the product

Advantages

Coherent Cache Across the cluster

Easy to integrate with existing applications

Performs very well

Eliminate the artificial cache misses in clustered environment

Disadvantages

Objects are represented as BLOBs by Hibernate

Doesn’t take direct advantage of Terracotta Scale-Out Features

Cluster Detached POJOs - Hibernate Performance Curve Level 3

Cluster Pure POJOs

Re-attach Session in the same JVM or a different JVM

Advantages

Scales the best

Take Advantage of POJOs - Fine-grained changes, replicate only where resident

Disadvantages

Some code changes required to refactor Hibernate’s beginTransaction(), commit()

Demonstration Application Simple CRUD application Based on Hibernate Tutorial (Person, Event) Already Refactored for Detached POJOs Simple Session Management in Terracotta Environment - POJO wrapper Detached Strategy requires a flush operation CREATE OPERATION Creates a new Person UPDATE OPERATION UpdateAge -> updates the age UpdateEvent -> creates a new event and adds to Person READ OPERATION Sets the current object to a random Person DELETE OPERATION Not implemented FLUSH OPERATION Re-attaches Session and writes modified POJO to DB

Simple CRUD application

Based on Hibernate Tutorial (Person, Event)

Already Refactored for Detached POJOs

Simple Session Management in Terracotta Environment - POJO wrapper

Detached Strategy requires a flush operation

CREATE OPERATION

Creates a new Person

UPDATE OPERATION

UpdateAge -> updates the age

UpdateEvent -> creates a new event and adds to Person

READ OPERATION

Sets the current object to a random Person

DELETE OPERATION

Not implemented

FLUSH OPERATION

Re-attaches Session and writes modified POJO to DB

Source Code

Performance Tests ReadAgeHibernate 25k iterations Reads a Person object, reads the age, commits Run with and without 2nd level cache UpdateAgeHibernate 25k iterations Reads a Person object, updates the age, commits Run with and without 2nd level cache ReadAgeTC Reads a Person object Sets person object into Terracotta clustered graph 25k iterations Reads the age UpdateAgeTC Reads a Person object Sets person object into Terracotta clustered graph 25k iterations Updates the age Commits

ReadAgeHibernate

25k iterations

Reads a Person object, reads the age, commits

Run with and without 2nd level cache

UpdateAgeHibernate

25k iterations

Reads a Person object, updates the age, commits

Run with and without 2nd level cache

ReadAgeTC

Reads a Person object

Sets person object into Terracotta clustered graph

25k iterations

Reads the age

UpdateAgeTC

Reads a Person object

Sets person object into Terracotta clustered graph

25k iterations

Updates the age

Commits

Results: Hibernate vs. Detached POJOs ~ 500,000 ops / sec Terracotta Read ~ 1800 ops / sec Hibernate + 2nd Level Cache Read ~ 1000 ops / sec Hibernate Read Results Type Operation Results Type Operation ~ 7000 ops / sec Terracotta Update ~ 1800 ops / sec Hibernate + 2nd Level Cache Update ~ 1000 ops / sec Hibernate Update

User Was Happy Database was still the SoR which kept reporting and backup simple Scalability was increased by over 10X Availability was not compromised since test data was still on disk, but in memory-resident format instead of relational

Database was still the SoR which kept reporting and backup simple

Scalability was increased by over 10X

Availability was not compromised since test data was still on disk, but in memory-resident format instead of relational

Partitioned Architecture

Example Caching Service Reduce utilization of System of Record Support 4 BUs 10K queries / second today Headroom for 40K queries / second (Note: all performance goals)

Reduce utilization of System of Record

Support 4 BUs

10K queries / second today

Headroom for 40K queries / second

(Note: all performance goals)

BU #1 BU #2 BU #3 BU #4 Data center Websphere MQ Series - MOM Messaging Infrastructure (JMS Topics) Caching Layer SoR MQ API MQ API Existing MQ Cache Node 1 Cache Node 27 Terracotta Server Terracotta Server . . . SoR API single pair

BU #1 BU #2 BU #3 BU #4 Data center Websphere MQ Series - MOM Messaging Infrastructure (JMS Topics) Caching Layer SoR MQ API MQ API Existing MQ Cache Node 1 Cache Node 27 Cache Node 14 Terracotta Server Terracotta Server Terracotta Server Terracotta Server Cache Node 15 . . . SoR API . . .

BU #1 BU #2 BU #3 BU #4 Data center Websphere MQ Series - MOM Messaging Infrastructure (JMS Topics) Caching Layer SoR MQ API MQ API Existing MQ Cache Node 1 Cache Node 27 Cache Node 14 Terracotta Server Terracotta Server Terracotta Server Terracotta Server Cache Node 15 . . . . . . SoR API Stateless Software Load Balancer

Demo 2 Shared Queue

User Was Unhappy Simplicity was lost. The Partitioning leaked up the application stack Availability was no longer easy (failure had to partition as well) Scalability was the only “Scale Out Dimension” delivered

Simplicity was lost. The Partitioning leaked up the application stack

Availability was no longer easy (failure had to partition as well)

Scalability was the only “Scale Out Dimension” delivered

Lessons Learned: Scalability + Availability + Simplicity Stop the Madness Stop the hacking! Stop the clustering! Start naïve and get more sophisticated on demand Balancing the 3 requires scalable, durable memory across JVM boundaries (spread out to scale out) Simplicity  Require no specific coding model and no hard-coded replication / persistence points Scalability  Read from Local Memory; write only the deltas in batches Availability  Write to external process + write to disk

Stop the Madness

Stop the hacking! Stop the clustering!

Start naïve and get more sophisticated on demand

Balancing the 3 requires scalable, durable memory across JVM boundaries (spread out to scale out)

Simplicity  Require no specific coding model and no hard-coded replication / persistence points

Scalability  Read from Local Memory; write only the deltas in batches

Availability  Write to external process + write to disk

Load-balanced Scale Out Simplicity  Ignore the impedance mismatch. Don’t be afraid of the DB. Scalability  Just cache it! (EHCache, JBossCache, custom) Disconnect from the DB as often as you can Availability  Distributed caches can be made durable / reliable and shared with JVM-level clustering

Simplicity  Ignore the impedance mismatch. Don’t be afraid of the DB.

Scalability  Just cache it! (EHCache, JBossCache, custom) Disconnect from the DB as often as you can

Availability  Distributed caches can be made durable / reliable and shared with JVM-level clustering

Partitioned Scale Out Simplicity  Share concurrency events, not data. Abstract as necessary (the caching service or SEDA or master / worker) Scalability  Once in the correct JVM, don’t move data. push only what changes Availability  Guarantee both the events and the data cannot be lost Honestly. Punt on partitioning if you can. Most people who need it will know, based on the use case outrunning disk, network, CPU, etc. Example: Pushing more than 1GBit on a single node where multiple nodes could each push 1GBit

Simplicity  Share concurrency events, not data. Abstract as necessary (the caching service or SEDA or master / worker)

Scalability  Once in the correct JVM, don’t move data. push only what changes

Availability  Guarantee both the events and the data cannot be lost

Honestly. Punt on partitioning if you can. Most people who need it will know, based on the use case outrunning disk, network, CPU, etc.

Example: Pushing more than 1GBit on a single node where multiple nodes could each push 1GBit

Thank You Learn more at http://www.terracotta.org/

Learn more at http://www.terracotta.org/