Scalability

Architecting systems that
support millions of TPS
Petter Graff
Founder of SciSpike
petter.graff@scispike.com
twitter: nor1nor
blog: pettergraff.blogspot.com

www.scispike.com Copyright © SciSpike 2015
Outline
More is better than bigger
Don't do now what you can do later
Sh*t happens
The one with a hammer...
Conversation Engine
2

© 2014 PayPal Inc. All rights reserved. Confidential and proprietary. Global Technology Education
Impact of Scalability on Business
3
• Google
• 500 ms reduces traffic to sites by 20%
• Yahoo
• 400 ms reduces traffic y 5-9%
• Amazon
• 100 ms reduces revenue by 1%
• Compuware
• Study of 33 major retailers
• 10 million home page views
• 1 sec delay reduces conversion by 7%

Moore’s Law
5

Scale Up vs. Scale Out
6
Capability
Cost
Scale Up
Capability
Cost Scale Out

Amdahl’s Law
7

Clients As Computation Extensions
 The beauty of JavaScript and Node.js:
– The same code can run on the client as on the server
 Every time someone connects, you can direct some of
the computation to your client
 Provides a natural scale
– ...as long as you can avoid bottlenecks predicted by
Amdahl's law
8

Stateless Computation Nodes
 Avoid dependencies on specific nodes
 If possible
– Use mementos for state
 Avoid
– Sticky sessions
– Expensive database state recovery
9

Actor Based Model
 A computation model that is
known to promote parallelism
is Actor Model
– Published by Carl Hewitt in
1973
 Proven by implementations
such as Ericson's ERLANG
 Known to scale near linear with
computational the number of
computational nodes
10

Node and Actor Model
 Arguably on the right track already with its asynchronous
nature
 At the time we started our quest on Node there were no
good actor models available
– Created our own
– Part of what we're about to open-source
– A bit different than the regular actor model
• We call it agent-model to make sure that they are not
confused
 Now, there are some:
– nactor (inspired by drama). Not much activity
– seneca.js (http://senecajs.org/) looks promising
11

Never put off till tomorrow what may be done day after
tomorrow just as well
-- Mark Twain

Piers Steel's Equation
 A 10-year study of procrastination provides insights into -
- and a formula for -- motivation
– U  Desire to complete a task
– E  The expectation of success
– V  The value of completion
– I  The immediacy of the task
– D  The personal sensitivity to delay
13

What is Important!
 Batch processing over a stream sources
– Highly scalable
– Very efficient
 Real-time processing of events trying to achieve a
consistent view over a distributed computing cluster
– Difficult to scale
– Eventually will suffer from Amdahl's law
 Therefore
– Minimize real-time processing and maximize the batch
computing
14

Real-Time Decision Components
 Minimize the work that has to be performed in real-time
 For your real-time decision module
– Expect significant optimization efforts
– Strive for stateless-ness
– Strive for atomic updates
– Build asynchrony into the interfaces where possible
 Absolute performance of the processing of individual
events are not as important as achieving near-linear
scale
– Typical throughput 100-100,000 requests per
second/CPU
15

CAP Theorem
 We will explore where some popular stores fit into this
model
16
C
A P

Consistency + Availability
 Consistency
 Availability
 Partition tolerance
17
C
A P
RDBMS (Oracle, DB2,
MySQL, SQLServer)
Vertica

Consistency + Partition Tolerance
 Consistency
 Availability
18
C
A P
BerkeleyDB
Big Table
HBase
Hypertable
MongoDB
Redis

Availability + Partition Tolerance
 Consistency
 Availability
19
C
A P
Dynamo
Voldemort
Cassandra
SimpleDB
CouchDB
Riak

Eventual Consistency
 In a decentralized database, a value needs to be written
at multiple location to provide for fault tolerance
 Design choice:
– Lock the system until all nodes confirm that the data is
written  poor performance
– Allow reads to reflect values that may not be the latest
 “Eventually” typically means milliseconds
20
Procrastination is the art of keeping up with yesterday
-- Don Marquis

Procrastination Is a Great Virtue
 Upon a request
– Execute only what's required to respond to the request
– Everything else can wait!
21
Time-Series
Database
Immediate Processing
Delayed Processing
Batch
Processors
Request
Handler
Read
Only DB
Read
Only DB
Read
Only DB
Real-time
View
Real-Time
Processor

Merge
Results
Lambda Architecture
22
Event (Speed) Layer
Real Time
Data
Batch Layer Serving Layer
Master
Dataset
Batch
View
Incoming
Data
Real Time Update
Batch Update
Queries
Rolling Values

Node Specific Issues
 Easy to introduce middleware that ensures storage of
incoming events
– We integrated with:
• Cassandra
• Hadoop
 Have to determine what events from within the business
logic deserves to be stored
 Writing has to be VERY fast
– Most cases we store the data in "raw" format
23

Murphy's Law
 It is found that anything that can go wrong at sea
generally does go wrong sooner or later, so it is not to be
wondered that owners prefer the safe to the scientific ....
Sufficient stress can hardly be laid on the advantages of
simplicity. The human factor cannot be safely neglected
in planning machinery. If attention is to be obtained, the
engine must be such that the engineer will be disposed
to attend to it.
 In short:
– Anything that can possibly go wrong, does
 Edward Aloysius Murphy, Jr. (January 11, 1918 – July
17, 1990)
25

When Scaling Out, Failure is Inevitable!
 The more computers/nodes, the more frequent we'll fail
 Upon failure
– Recover without loss
– At worst, slow
– At best, seamless
 This means:
– You have to design for failure
– You have to test failures
– You have to inject failure
26

Don't Be Afraid of Multiple Tools
 Polyglot persistence
– RDBMS are good for some things but not for
everything
– We looked at the CAP theorem earlier...
 Some tools are not written in Node.js and that's OK...
– LMAX
– C++
– etc.
 I love node.js, but...
– ... I'll use any other technology if that makes sense

NoSQL Impact
29
Disks
Processors
x1000 x1000 x1000
Cost/Performance
1M 1B 1T 1Q …HUGE!!!x1000
Relational
Database
NoSQL
Tomorrow - Volume is
out of reach
Today - Doable, but
expensive and slow
Stabilize Cost &
Increase Performance
Enable Unlimited
Volume Growth

NoSQL Stores and Their Categories
 Choose a store that is a best match for your application
 It is fine to have several different stores used
– "Polyglot persistence"
30
k v
Key-ValueColumn-
Family
Document-
Oriented
Graph DB

NoSQL Stores: Scale vs. Complexity
31
k v
Key-Value
Column-
Family
Document-
Oriented
complexity
scalability
Graph DB
needs of most applications

Polyglot Persistence FTW!
32
Columnar
Price
updates
Logs
Document
Product
info
Graph
Customer
Agent
relation-
ships
RDB
XA data
Hadoop
Oper.
analytics
Price
analytics
Key/Value
Session
data
Applications

Other Issues
 Keep computation nodes stateless
– Any node can process a request
 Data structures and algorithms matters
– Learn to profile
– Profile clusters (transactions per node, scale graph...)
 HTTP(S) is expensive and cause latency
– Web-sockets (with proper reconnect logic)
– MQTT
 Atomic updates instead of transactions
 Business CRUD must not block real-time processing
33

Observation
 Finding developers that can strive all these technologies
is nearly impossible
 Where developers struggle:
– Sound architectural decisions
– Asynchronous programming
– Profiling and debugging
– Concurrency
– Basic data structures and algorithms
 Programming sometimes reminds me of:
– "Parking by Sound Effect"
35

How We 'Solved' It...
 Find the best developers you can find and have them
– ... develop the underlying platform
– ... develop intent languages from which we can
generate the difficult code
 OUCH!!!
– THAT SOUNDS LIKE A FRAMEWORK!!!
36

What Is It?
 A set of tools focused on:
– Rapid requirement gathering
 A set of generators that generate
– Documentation
– Visualization
– Implementations
 The node.js generator
– Generates a scalable 'agent (actor) – based'
implementation
38

The SciSpike Node.js Architecture
41
Time-Series
Database
Immediate Processing
Delayed Processing
Batch
Processors
Request
Handler
Read
Only DB
Read
Only DB
Read
Only DB
Real-time
View
Real-Time
Processor

Conversation Engine in the Cloud
42
MessagingBUS
...
 Near linear scale
 Tooling for
deployment
 Tooling for failover
 Tooling for dynamic
scale

Technologies
43
HTTP, but
prefer MQTT
WebSockets
Node.js-based
middleware
Node.js Actor-
Based Model
MongoDB
(Redis/MemCac
he)
Any Database
(RDBMS,
Cassandra,
OrientDB, etc.)
PIG, Hive,
Scalding,...
Cassandra or
Hadoop

IDE and Command Line Tools
44

Graphical Visualization
45

Simulate Business Requirements
46

One Source, Multiple Outputs
47
Generated
Application
Documentation
Compliance
Suite
Graphical
Models
Textual
Intent
Languag
e

Documentation Generation
4848
Models
Generator
Documentation (in ASCIIDoctor)
Generator
HTML
Documentation
Site
ePubDocbookPDF

Hand Written
Code
Generated
Application
User Agent
Application Generation
49
UI Elements
Services
Business Logic
Interactions
Basic Data Functionality
Data Access
DB
UI Refinement &
Interactions
Business Logic
Generator
Business
Language
Specs
Generated
Cloud Deployment
Generation
Rules &
Templates

What Is Generated (back-end)?
50
CL Generator DM Generator Hand Coded
Lambda RT
Processor
Generated Domain and cache
updates
Lambda Batch
Processor
Business Logic
Query Processor REST API
connected to
domain model.
Basic OR mapping
with the main
queries supported.
Nonsalable views
(e.g., security based
projections)
Canonical Domain
Model
Generates schema
and partial API’s
Triggers and
complex non-map
able queries
Time Series
Database
Automatic. Production of time-
series events
possible
Distributed Cache State of
conversations

Technology Stack
51
ServerSide
CE Runtime
Generated Code
Dev. written logic
Messaging
DB
REST
Dev. written UI
UserInterface
RDB
Systems Integration & Persistence
Generated UI

DEMO

Required Before Going Open Source
 Better documentation of the language and the
generators
– The documentation is based on last years tools
– Minor effort
 Divorce generators from Eclipse
– Some of our generators are still Eclipse based
• Using the M2T project in Eclipse
– We're in the process of divorcing them from Eclipse
– The U/I generators already are separated
 Customization of generators
 Self-serve support forum and website
54

Call to Action
 We want to accelerate our path to open source
– Developers
– Tech writer
– etc.
 If you're interested:
– Contact me:
petter.graff@scispike.com
55

Scalability

More Related Content

Similar to Scalability

Recently uploaded

Scalability