Successful Scalability
Principles - Part 1
Ronald Bradford
http://ronaldbradford.com
2012.08
EffectiveMySQL.com - Performa...
OUTLINE
Necessary Principles
System Architecture
Data Availability
Best Practices
Being proactive

EffectiveMySQL.com - Pe...
1

EffectiveMySQL.com - Performance, Scalability & Business Continuity
ACTION

Integrated
monitoring and
instrumentation

EffectiveMySQL.com - Performance, Scalability & Business Continuity
EXAMPLE
Question:

✘

How do you know when your server is
down or not accessible?
Answer:
The users will let us know.

Eff...
HOW

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Monitoring/Alerting
Graphical
Historical
Necessary
Generally missing/incomplete
Useless for real-time analysis
Effecti...
HOW

E
V
I
T
C
A
E
R
Monitoring/Alerting

Graphical
Historical
Necessary
Generally missing/incomplete
Useless for real-tim...
HOW

http://rb42.com/monitoring-needs
EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Dashboard
The state of NOW
Near real-time statistics
Sampling at 1s/3s/5s
e.g. 0.1% of throughput
http://rb42.com/moni...
HOW
Dashboard

E
V
I
T

C
A
O
R
P
The state of NOW

Near real-time statistics
Sampling at 1s/3s/5s

e.g. 0.1% of throughpu...
HOW

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Instrumentation
Important to business viability
e.g. orders per minute (KPI)
page load time

Seamless implementation
i...
HOW
Instrumentation

E
V
I
T

C
A
O
R
P

Important to business viability
e.g. orders per minute (KPI)
page load time

Seam...
Why

Monitoring +
Instrumentation =
Preemptive Analysis

EffectiveMySQL.com - Performance, Scalability & Business Continui...
How NOT TO

✘

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW TO

✔
< 1 second

EffectiveMySQL.com - Performance, Scalability & Business Continuity
WHY
Preemptive Analysis
Fail early
Faster user response
Less physical resource impact

EffectiveMySQL.com - Performance, S...
TIP

EffectiveMySQL.com - Performance, Scalability & Business Continuity
TIP
Have a status website
allow for comments (e.g. blog)

EffectiveMySQL.com - Performance, Scalability & Business Continu...
TIP
Have a status website
allow for comments (e.g. blog)

Have a public dashboard website

EffectiveMySQL.com - Performanc...
TIP
Have a status website
allow for comments (e.g. blog)

Have a public dashboard website
Host them somewhere else!
Effect...
EffectiveMySQL.com - Performance, Scalability & Business Continuity
EffectiveMySQL.com - Performance, Scalability & Business Continuity
2

EffectiveMySQL.com - Performance, Scalability & Business Continuity
ACTION

Seamless
automated
server
deployment
EffectiveMySQL.com - Performance, Scalability & Business Continuity
EXAMPLE
Actual Client
100+ production database servers

✘

Full-time DBAs
No automated software installation
No single doc...
HOW
Basics
Version Control
Automated Build & Release
Integrated Monitoring

EffectiveMySQL.com - Performance, Scalability ...
HOW
Intermediate
Health Checks
Runtime config file management
Benchmarking

EffectiveMySQL.com - Performance, Scalability & ...
HOW
Advanced
Server request queue
Automated discovery
Proactive scaling (up & down)

EffectiveMySQL.com - Performance, Sca...
GOAL
Consistent and timely provisioning
Unattended installation with correct
and reproducible configuration
Automated disco...
3

EffectiveMySQL.com - Performance, Scalability & Business Continuity
ACTION

Disaster
is
inevitable
EffectiveMySQL.com - Performance, Scalability & Business Continuity
EXAMPLE
Question:

✘

Have you ever performed a database recovery?

Answer:
No, why?

EffectiveMySQL.com - Performance, Sc...
EXAMPLE
Consultant:

✘

Do you know that your daily backups only recover
the data up to that time (e.g. 1 am).You know you...
HOW

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Have a DR plan

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Have a DR plan
Documented

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Have a DR plan
Documented
Tested

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Have a DR plan
Documented
Tested
Timed

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Have a DR plan
Documented
Tested
Timed
Verified
EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Test under production conditions

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Test under production conditions
System Load

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Test under production conditions
System Load
Database Size

EffectiveMySQL.com - Performance, Scalability & Business C...
HOW
Test under production conditions
System Load
Database Size
End to End

EffectiveMySQL.com - Performance, Scalability &...
GOAL
Know and practice for disaster
Provide confidence to
management

✔

Be as confident about your
recovery as your scalabi...
SIDEBAR!

What does your website
look like when it’s down?

EffectiveMySQL.com - Performance, Scalability & Business Conti...
SIDEBAR

✘

Google Picasa

EffectiveMySQL.com - Performance, Scalability & Business Continuity
SIDEBAR

✘

EffectiveMySQL.com - Performance, Scalability & Business Continuity
SIDEBAR

✘

Mozilla

EffectiveMySQL.com - Performance, Scalability & Business Continuity
SIDEBAR

✔

EffectiveMySQL.com - Performance, Scalability & Business Continuity
SIDEBAR

✔

EffectiveMySQL.com - Performance, Scalability & Business Continuity
4

EffectiveMySQL.com - Performance, Scalability & Business Continuity
ACTION

Application
Programming
Interface
EffectiveMySQL.com - Performance, Scalability & Business Continuity
EXAMPLE

✘

3 access paths to same data
with different business rules
Public API
Application web site
Batch processes

Eff...
WHY an API?
One code path for business
functionality
Implied business documentation
Enforced data exchange standard
Testab...
GOAL
Technology independence
Business specification
dependence (API)

✔

Stress testable

EffectiveMySQL.com - Performance,...
REQUIREMENTS
Input Specification
Output Specification
Standard for data communication
Throughput needs
Latency requirements
...
SIDEBAR!

Testing is not about
what works, it’s about
breaking your software

EffectiveMySQL.com - Performance, Scalabilit...
5

EffectiveMySQL.com - Performance, Scalability & Business Continuity
ACTION

Support different
levels of data
availability

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Data can be:

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Data can be:
Read & Write

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Data can be:
Read & Write
Read

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Data can be:
Read & Write
Read
Cached

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Data can be:
Read & Write
Read
Cached
No Access
EffectiveMySQL.com - Performance, Scalability & Business Continuity
EXAMPLE
Development Team:
We need a maintenance
window for software
upgrades and new
releases.
EffectiveMySQL.com - Perfor...
EXAMPLE
Management Team:
No Downtime

EffectiveMySQL.com - Performance, Scalability & Business Continuity
EXAMPLE
Development Team:
But we need this to fix
problems and improve
performance.

EffectiveMySQL.com - Performance, Scal...
EXAMPLE
Management Team:
No Downtime

EffectiveMySQL.com - Performance, Scalability & Business Continuity
EXAMPLE

What is your
definition of no
downtime?

EffectiveMySQL.com - Performance, Scalability & Business Continuity
WHY
Support downtime
Software upgrades

Support disasters
Reduce high load
Support partial functionality
EffectiveMySQL.co...
GOAL
Meet the business needs of
availability
There are ways to give greater
perception of availability

✔

EffectiveMySQL....
6

EffectiveMySQL.com - Performance, Scalability & Business Continuity
ACTION

Support different
scalability
principles

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Read Scalability
Write Scalability
Caching

EffectiveMySQL.com - Performance, Scalability & Business Continuity
HOW
Depends on your R/W ratio
Depends on your business needs
Depends on rate of data change
(caching)

EffectiveMySQL.com ...
ISSUES
Consistency
Data Interoperability

EffectiveMySQL.com - Performance, Scalability & Business Continuity
GOAL

✔

Support large scale growth with
appropriate software architecture
Minimize human interaction

Ongoing review and ...
SIDEBAR!

Availability options &
scalability principles
= scalable datastore
EffectiveMySQL.com - Performance, Scalability...
+

EffectiveMySQL.com - Performance, Scalability & Business Continuity
WHAT's NEXT

I only had
one hour!

EffectiveMySQL.com - Performance, Scalability & Business Continuity
WHAT's NEXT
Part 2

EffectiveMySQL.com - Performance, Scalability & Business Continuity
WHAT's NEXT
Part 2
Reduce deployment time
The right business metrics
Team dynamics
Managing risk
Continual improvement
Dat...
WHAT's NEXT
Part 2
Reduce deployment time
The right business metrics
Team dynamics
Managing risk
Continual improvement
Dat...
C

EffectiveMySQL.com - Performance, Scalability & Business Continuity
CONCLUsION
Monitoring. Before, during and after NOW.
You may not be able to predict the future but
you can preempt the fut...
CONCLUSION
Does your business exist without your data?
How long can your site be unavailable before
customers go elsewhere...
CONCLUSION
Support 3 levels of real time data access
Read/Write, Read and no access
Support 3 different aspects of scalabi...
REFERENCE
Presentation
The most common MySQL scalability
mistakes, and how to avoid them.

EffectiveMySQL.com - Performanc...
REFERENCE
1.

My website is slow?

2.

I want to add new H/W. How do I change my application to support this?

3.

The dat...
Ronald Bradford
http://effectiveMySQL.com
EffectiveMySQL.com - Performance, Scalability & Business Continuity
Upcoming SlideShare
Loading in...5
×

Successful Scalability Principles - Part 1

2,338

Published on

Every website wants to become successful. Few websites however undertake the basic and fundamental steps to build a rock solid foundation to ensure a scalable

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,338
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
20
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Successful Scalability Principles - Part 1

  1. 1. Successful Scalability Principles - Part 1 Ronald Bradford http://ronaldbradford.com 2012.08 EffectiveMySQL.com - Performance, Scalability & Business Continuity
  2. 2. OUTLINE Necessary Principles System Architecture Data Availability Best Practices Being proactive EffectiveMySQL.com - Performance, Scalability & Business Continuity
  3. 3. 1 EffectiveMySQL.com - Performance, Scalability & Business Continuity
  4. 4. ACTION Integrated monitoring and instrumentation EffectiveMySQL.com - Performance, Scalability & Business Continuity
  5. 5. EXAMPLE Question: ✘ How do you know when your server is down or not accessible? Answer: The users will let us know. EffectiveMySQL.com - Performance, Scalability & Business Continuity
  6. 6. HOW EffectiveMySQL.com - Performance, Scalability & Business Continuity
  7. 7. HOW Monitoring/Alerting Graphical Historical Necessary Generally missing/incomplete Useless for real-time analysis EffectiveMySQL.com - Performance, Scalability & Business Continuity
  8. 8. HOW E V I T C A E R Monitoring/Alerting Graphical Historical Necessary Generally missing/incomplete Useless for real-time analysis EffectiveMySQL.com - Performance, Scalability & Business Continuity
  9. 9. HOW http://rb42.com/monitoring-needs EffectiveMySQL.com - Performance, Scalability & Business Continuity
  10. 10. HOW Dashboard The state of NOW Near real-time statistics Sampling at 1s/3s/5s e.g. 0.1% of throughput http://rb42.com/monitoring-needs EffectiveMySQL.com - Performance, Scalability & Business Continuity
  11. 11. HOW Dashboard E V I T C A O R P The state of NOW Near real-time statistics Sampling at 1s/3s/5s e.g. 0.1% of throughput http://rb42.com/monitoring-needs EffectiveMySQL.com - Performance, Scalability & Business Continuity
  12. 12. HOW EffectiveMySQL.com - Performance, Scalability & Business Continuity
  13. 13. HOW Instrumentation Important to business viability e.g. orders per minute (KPI) page load time Seamless implementation i.e. no code changes to view real-time extensible EffectiveMySQL.com - Performance, Scalability & Business Continuity
  14. 14. HOW Instrumentation E V I T C A O R P Important to business viability e.g. orders per minute (KPI) page load time Seamless implementation i.e. no code changes to view real-time extensible EffectiveMySQL.com - Performance, Scalability & Business Continuity
  15. 15. Why Monitoring + Instrumentation = Preemptive Analysis EffectiveMySQL.com - Performance, Scalability & Business Continuity
  16. 16. How NOT TO ✘ EffectiveMySQL.com - Performance, Scalability & Business Continuity
  17. 17. HOW TO ✔ < 1 second EffectiveMySQL.com - Performance, Scalability & Business Continuity
  18. 18. WHY Preemptive Analysis Fail early Faster user response Less physical resource impact EffectiveMySQL.com - Performance, Scalability & Business Continuity
  19. 19. TIP EffectiveMySQL.com - Performance, Scalability & Business Continuity
  20. 20. TIP Have a status website allow for comments (e.g. blog) EffectiveMySQL.com - Performance, Scalability & Business Continuity
  21. 21. TIP Have a status website allow for comments (e.g. blog) Have a public dashboard website EffectiveMySQL.com - Performance, Scalability & Business Continuity
  22. 22. TIP Have a status website allow for comments (e.g. blog) Have a public dashboard website Host them somewhere else! EffectiveMySQL.com - Performance, Scalability & Business Continuity
  23. 23. EffectiveMySQL.com - Performance, Scalability & Business Continuity
  24. 24. EffectiveMySQL.com - Performance, Scalability & Business Continuity
  25. 25. 2 EffectiveMySQL.com - Performance, Scalability & Business Continuity
  26. 26. ACTION Seamless automated server deployment EffectiveMySQL.com - Performance, Scalability & Business Continuity
  27. 27. EXAMPLE Actual Client 100+ production database servers ✘ Full-time DBAs No automated software installation No single documented installation process No version control No auditing/verification EffectiveMySQL.com - Performance, Scalability & Business Continuity
  28. 28. HOW Basics Version Control Automated Build & Release Integrated Monitoring EffectiveMySQL.com - Performance, Scalability & Business Continuity
  29. 29. HOW Intermediate Health Checks Runtime config file management Benchmarking EffectiveMySQL.com - Performance, Scalability & Business Continuity
  30. 30. HOW Advanced Server request queue Automated discovery Proactive scaling (up & down) EffectiveMySQL.com - Performance, Scalability & Business Continuity
  31. 31. GOAL Consistent and timely provisioning Unattended installation with correct and reproducible configuration Automated discovery ✔ EffectiveMySQL.com - Performance, Scalability & Business Continuity
  32. 32. 3 EffectiveMySQL.com - Performance, Scalability & Business Continuity
  33. 33. ACTION Disaster is inevitable EffectiveMySQL.com - Performance, Scalability & Business Continuity
  34. 34. EXAMPLE Question: ✘ Have you ever performed a database recovery? Answer: No, why? EffectiveMySQL.com - Performance, Scalability & Business Continuity
  35. 35. EXAMPLE Consultant: ✘ Do you know that your daily backups only recover the data up to that time (e.g. 1 am).You know you have lost all your sales(aka $2 million sales today) and data changes since then. Customer: No, I didn’t know that. EffectiveMySQL.com - Performance, Scalability & Business Continuity
  36. 36. HOW EffectiveMySQL.com - Performance, Scalability & Business Continuity
  37. 37. HOW Have a DR plan EffectiveMySQL.com - Performance, Scalability & Business Continuity
  38. 38. HOW Have a DR plan Documented EffectiveMySQL.com - Performance, Scalability & Business Continuity
  39. 39. HOW Have a DR plan Documented Tested EffectiveMySQL.com - Performance, Scalability & Business Continuity
  40. 40. HOW Have a DR plan Documented Tested Timed EffectiveMySQL.com - Performance, Scalability & Business Continuity
  41. 41. HOW Have a DR plan Documented Tested Timed Verified EffectiveMySQL.com - Performance, Scalability & Business Continuity
  42. 42. HOW EffectiveMySQL.com - Performance, Scalability & Business Continuity
  43. 43. HOW Test under production conditions EffectiveMySQL.com - Performance, Scalability & Business Continuity
  44. 44. HOW Test under production conditions System Load EffectiveMySQL.com - Performance, Scalability & Business Continuity
  45. 45. HOW Test under production conditions System Load Database Size EffectiveMySQL.com - Performance, Scalability & Business Continuity
  46. 46. HOW Test under production conditions System Load Database Size End to End EffectiveMySQL.com - Performance, Scalability & Business Continuity
  47. 47. GOAL Know and practice for disaster Provide confidence to management ✔ Be as confident about your recovery as your scalability EffectiveMySQL.com - Performance, Scalability & Business Continuity
  48. 48. SIDEBAR! What does your website look like when it’s down? EffectiveMySQL.com - Performance, Scalability & Business Continuity
  49. 49. SIDEBAR ✘ Google Picasa EffectiveMySQL.com - Performance, Scalability & Business Continuity
  50. 50. SIDEBAR ✘ EffectiveMySQL.com - Performance, Scalability & Business Continuity
  51. 51. SIDEBAR ✘ Mozilla EffectiveMySQL.com - Performance, Scalability & Business Continuity
  52. 52. SIDEBAR ✔ EffectiveMySQL.com - Performance, Scalability & Business Continuity
  53. 53. SIDEBAR ✔ EffectiveMySQL.com - Performance, Scalability & Business Continuity
  54. 54. 4 EffectiveMySQL.com - Performance, Scalability & Business Continuity
  55. 55. ACTION Application Programming Interface EffectiveMySQL.com - Performance, Scalability & Business Continuity
  56. 56. EXAMPLE ✘ 3 access paths to same data with different business rules Public API Application web site Batch processes EffectiveMySQL.com - Performance, Scalability & Business Continuity
  57. 57. WHY an API? One code path for business functionality Implied business documentation Enforced data exchange standard Testability EffectiveMySQL.com - Performance, Scalability & Business Continuity
  58. 58. GOAL Technology independence Business specification dependence (API) ✔ Stress testable EffectiveMySQL.com - Performance, Scalability & Business Continuity
  59. 59. REQUIREMENTS Input Specification Output Specification Standard for data communication Throughput needs Latency requirements EffectiveMySQL.com - Performance, Scalability & Business Continuity
  60. 60. SIDEBAR! Testing is not about what works, it’s about breaking your software EffectiveMySQL.com - Performance, Scalability & Business Continuity
  61. 61. 5 EffectiveMySQL.com - Performance, Scalability & Business Continuity
  62. 62. ACTION Support different levels of data availability EffectiveMySQL.com - Performance, Scalability & Business Continuity
  63. 63. HOW EffectiveMySQL.com - Performance, Scalability & Business Continuity
  64. 64. HOW Data can be: EffectiveMySQL.com - Performance, Scalability & Business Continuity
  65. 65. HOW Data can be: Read & Write EffectiveMySQL.com - Performance, Scalability & Business Continuity
  66. 66. HOW Data can be: Read & Write Read EffectiveMySQL.com - Performance, Scalability & Business Continuity
  67. 67. HOW Data can be: Read & Write Read Cached EffectiveMySQL.com - Performance, Scalability & Business Continuity
  68. 68. HOW Data can be: Read & Write Read Cached No Access EffectiveMySQL.com - Performance, Scalability & Business Continuity
  69. 69. EXAMPLE Development Team: We need a maintenance window for software upgrades and new releases. EffectiveMySQL.com - Performance, Scalability & Business Continuity
  70. 70. EXAMPLE Management Team: No Downtime EffectiveMySQL.com - Performance, Scalability & Business Continuity
  71. 71. EXAMPLE Development Team: But we need this to fix problems and improve performance. EffectiveMySQL.com - Performance, Scalability & Business Continuity
  72. 72. EXAMPLE Management Team: No Downtime EffectiveMySQL.com - Performance, Scalability & Business Continuity
  73. 73. EXAMPLE What is your definition of no downtime? EffectiveMySQL.com - Performance, Scalability & Business Continuity
  74. 74. WHY Support downtime Software upgrades Support disasters Reduce high load Support partial functionality EffectiveMySQL.com - Performance, Scalability & Business Continuity
  75. 75. GOAL Meet the business needs of availability There are ways to give greater perception of availability ✔ EffectiveMySQL.com - Performance, Scalability & Business Continuity
  76. 76. 6 EffectiveMySQL.com - Performance, Scalability & Business Continuity
  77. 77. ACTION Support different scalability principles EffectiveMySQL.com - Performance, Scalability & Business Continuity
  78. 78. HOW Read Scalability Write Scalability Caching EffectiveMySQL.com - Performance, Scalability & Business Continuity
  79. 79. HOW Depends on your R/W ratio Depends on your business needs Depends on rate of data change (caching) EffectiveMySQL.com - Performance, Scalability & Business Continuity
  80. 80. ISSUES Consistency Data Interoperability EffectiveMySQL.com - Performance, Scalability & Business Continuity
  81. 81. GOAL ✔ Support large scale growth with appropriate software architecture Minimize human interaction Ongoing review and improvement EffectiveMySQL.com - Performance, Scalability & Business Continuity
  82. 82. SIDEBAR! Availability options & scalability principles = scalable datastore EffectiveMySQL.com - Performance, Scalability & Business Continuity
  83. 83. + EffectiveMySQL.com - Performance, Scalability & Business Continuity
  84. 84. WHAT's NEXT I only had one hour! EffectiveMySQL.com - Performance, Scalability & Business Continuity
  85. 85. WHAT's NEXT Part 2 EffectiveMySQL.com - Performance, Scalability & Business Continuity
  86. 86. WHAT's NEXT Part 2 Reduce deployment time The right business metrics Team dynamics Managing risk Continual improvement Data interoperability ... EffectiveMySQL.com - Performance, Scalability & Business Continuity
  87. 87. WHAT's NEXT Part 2 Reduce deployment time The right business metrics Team dynamics Managing risk Continual improvement Data interoperability ... Asynchronous Write once data Consistency EffectiveMySQL.com - Performance, Scalability & Business Continuity
  88. 88. C EffectiveMySQL.com - Performance, Scalability & Business Continuity
  89. 89. CONCLUsION Monitoring. Before, during and after NOW. You may not be able to predict the future but you can preempt the future. Operate below 90%. That 10% is your insurance policy. Invest in insurance. EffectiveMySQL.com - Performance, Scalability & Business Continuity
  90. 90. CONCLUSION Does your business exist without your data? How long can your site be unavailable before customers go elsewhere? EffectiveMySQL.com - Performance, Scalability & Business Continuity
  91. 91. CONCLUSION Support 3 levels of real time data access Read/Write, Read and no access Support 3 different aspects of scalability Read, Write and Caching EffectiveMySQL.com - Performance, Scalability & Business Continuity
  92. 92. REFERENCE Presentation The most common MySQL scalability mistakes, and how to avoid them. EffectiveMySQL.com - Performance, Scalability & Business Continuity
  93. 93. REFERENCE 1. My website is slow? 2. I want to add new H/W. How do I change my application to support this? 3. The database is slow. My webpage takes five seconds to load. 4. Why is my database executing so many qps? 5. My server has crashed with a hard drive failure 6. My replication slave can't keep up? 7. I can't access my website? 8. 8. Why is my database so large? 9. My website seems to freeze or responds randomly? EffectiveMySQL.com - Performance, Scalability & Business Continuity
  94. 94. Ronald Bradford http://effectiveMySQL.com EffectiveMySQL.com - Performance, Scalability & Business Continuity
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×