Lightning talks percona live mysql_2012

Lightning talks
Giuseppe Maxia
Continuent, Inc

April 11, 2012
Wednesday, April 11, 12

Lineup
Stewart Smith The 5 minutes tour of Drizzle
Jeff Mace What can possibly go wrong?
Nicolai Plum MAKE.DATABASES.FAST
Bill Karwin Schemadoc
Robert Hodges It's the application, stupid!
Robert Buck NuoDB overview
David Holoboff Diagnosis and possible ﬁxes
Lawrence Schwartz OLTP and OLAP
Baron Schwartz Building a scalable system
Sheeri Cabral How to become a rock star DBA
Giuseppe Maxia Cruelty on database clusters

The 5 minutes tour of
Drizzle
Stewart Smith


2008...

http://www.flickr.com/photos/shaylor/53819053/


2011...

Drizzle7

2012...

Drizzle7.1

What is
Drizzle?

Modular
architecture

Pluggable

Infrastructure
Aware

Community
Developed

Multicore


Concurrency


Web focus


UTF-8


EBCDIC
!=

Enable
others

Modern
codebase

Boost


No
linked-list
innovation

Collaboration


Participation


Build
businesses

Support & Consulting


Testing


Transactional


Logging


Replication


Multi-source
Replication

In database
tables

No bad data


strict_mode


No GOTCHAs


No implicit
COMMIT

BSD Client
Libraries

New in 7.1


XtraBackup


IPv6 data
type

ZeroMQ


Syslog is
default

query_log
plugin

Auth from
tables

Server-side
Javascript

HTTP JSON
API

Percona
InnoDB
Patches

What could possibly go
wrong?
Jeff Mace


What Could Possibly
Go Wrong?
Je! Mace

©Continuent 2012.


http://mingle2.com/blog/view/web-developer-mind

©Continuent 2012 59


http://www.ﬂickr.com/photos/brhefele/4026340146/



http://www.ﬂickr.com/photos/themoly/4336260658/



What can I possibly do?

• Relax

• Have a plan

• Keep it simple



http://www.ﬂickr.com/photos/devenlaney/5767271943



MAKE.DATABASES.FAST
Nicolai Plum


MAKE.DATABASES.FAST

Nicolai Plum

Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices


The business
• Booking.com
– Leading online hotel reservation agency
– Part of the Priceline Group of Companies
– Sustained rapid business growth
– All powered by MySQL



The challenge



The problem…
• Hundreds of database servers
• Many replication slaves
• Few DBAs, some more sysadmins
• Need to make tens of servers each day



If you do this manually
You are doing it wrong



Among our weapons are…
• Network OS installation
• Puppet configuration management
• Database cloning
• and a fanatical devotion to automation!



Server installation
• Server Database
• PXE netboot
• CentOS kickstart



System configuration
• Now we have a server that can run Puppet
• Puppet does
– OS setup
– Software installation
– LVM setup and filesystem creation (tailored by role)
• Prefer ongoing configuration management tools to
single perfect installation tools



Database cloning
• Now we have a server ready to run a database
• Copying databases based on LVM snapshots
• create_clone program
– Copy via SSH and rsync
– LVM snapshot
– Fix my.cnf (server id!) automatically



Things we learned
• Waiting for actions to complete takes a lot of person
time – automate batches of work as much as
possible
• You really need a Server Database, and drive
everything from it
• Most hardware can’t support being a clone source
and in service, you need spares
• Online snapshots break in subtle ways; best stop the
database.



Things we learned

DBAs are toolsmiths,
not database installers



Thank you
• … if you like the way we work, we are hiring
• http://www.booking.com/jobs



Schemadoc
Bill Karwin


Schemadoc
Bill Karwin, Percona Inc.


Schema Can Be Unclear
CREATE TABLE AccountActivity (
Id INT PRIMARY KEY,
Name VARCHAR(100),
Status VARCHAR(20),
EntityNum INT,
Order INT
);

www.percona.com

Id INT PRIMARY KEY,
whose name?
Name VARCHAR(100),
Status VARCHAR(20),
EntityNum INT,
Order INT
);

www.percona.com

Id INT PRIMARY KEY,
whose name?
Name VARCHAR(100),
Status VARCHAR(20), what are the values?

EntityNum INT,
Order INT
);

www.percona.com

Id INT PRIMARY KEY,
whose name?
Name VARCHAR(100),

EntityNum INT, what’s an entity?
Order INT what’s a number?
);

www.percona.com

Id INT PRIMARY KEY,
whose name?
Name VARCHAR(100),

EntityNum INT, what’s an entity?
Order INT what’s a number?
); commerce, or ordinal?

www.percona.com

Comments!
Id INT PRIMARY KEY,
Name VARCHAR(100)
COMMENT ‘person who placed the order’,
Status VARCHAR(20)
COMMENT ‘new, open, or closed’,
EntityNum INT
COMMENT ‘how many items in the order’,
Order INT
COMMENT ‘reference to the Orders table’
) COMMENT ‘any change to e-commerce
orders’;
www.percona.com

Length
• Limits increased in MySQL 5.5.3:
• Per column: 1024 characters
• Per index: 1024 characters
• Per table: 2048 characters
• Per partition: 80 characters
(MySQL 5.6 increases this to 1024)

www.percona.com

“Is there a tool for MySQL
like javadoc for Java code?”

www.percona.com

development
dba

documentation

www.percona.com

schemadoc

www.percona.com

Usage
$ schemadoc -v h=dbserver,D=EcommerceDB,u=root,p=xxxx
Loading metadata...
Running online analysis...
done.
Outputting to html.
Rendering pages for tables. done.
Rendering pages for views done.
Rendering pages for procedures done.
Rendering pages for triggers done.
Rendering page for schema summary... done.
Rendering page for schema link... done.
Rendering page for deprecated-list... done.
Rendering page for all-index... done.
Rendering page for help-doc... done.
Rendering page for allschema-frame... done.
Rendering page for index... done.

www.percona.com

hooray.

www.percona.com

But Wait, There’s More
• Parses SQL dump file
• Reads from a live MySQL instance
• Document many schemas
• Update one schema at a time

www.percona.com

Schema Analysis
• Number of indexes and columns per data type
• Columns with same name but different type
• ENUM columns that aren’t NOT NULL
• Use of FLOAT or DOUBLE
• Tables with one index per column
• Tables with one index over all columns
• Tables with no indexes
• INT(N) other than default N
• Lack of INT UNSIGNED columns
www.percona.com

Schema Analysis
• Tables that look like Entity-Attribute-Value
• Tables that look like Polymorphic Associations
• Superfluous primary key
• Report table with least / most indexes
• IP addresses stored in VARCHAR
• Excessive use of VARCHAR(255),
report actual max string length
• ...And supports plugins for other checks!

www.percona.com

yes, we get it.

www.percona.com

Future of Schemadoc
• Integrate with Percona Toolkit library
• Read extra metadata (foreign keys)
• Show table sizes and growth rates
• Show SQL privileges (who has access?)
• CLI for adding / updating comments
• Batch-mode output
• Redesign HTML output with modern look & feel

www.percona.com

Future of Schemadoc

www.percona.com

launchpad.net/
schemadoc

www.percona.com

It's the application,
stupid!
Robert Hodges


It’s the Application, Stupid!
(Or: Multi-Master Enlightenment
in 5 Minutes or Less)

Robert Hodges, Continuent CEO

©Continuent 2012.


Multi-Master Replication Is This Easy

tungsten-sandbox-2.0.4 -m 5.1.54
-i $RELDIR/tungsten-replicator-2.0.5
-n 3 --topology=star --hub=2

Spoke Hub Spoke


Multi-Master Apps Not So Much

Airline Airline Airline
Reservations Reservations Reservations

Spoke Hub Spoke


Who Is in Which Seat on Flight 3523?

Book Bob Bump Bob from Book Sue
in Seat 11E Flight 3523 in Seat 11E
Airline Airline Airline
Reservations Reservations Reservations

Spoke Hub Spoke


Rules for Multi-Master Apps
1. Isolate destructive changes to a single
master to avoid harmful con!icts
➡ “Change gate for !ight 3523 to C28”
2. Use non-destructive INSERTs for
everything else
➡ “Please assign Sue to seat 11A”



Rules for Multi-Master Apps
3. Coordinate across masters to aggregate
results
➡ “Generate "nal passenger manifest for !ight 3523”
4. Be prepared to apologize
➡ “Add 5000 frequent !yer miles to Bob’s account”



Multi-Master Enlightenment
Scalable multi-master applications are
eventually consistent
1. Think through the whole problem:
Business processes are often eventually
consistent already
2. Follow design rules for eventually
consistent applications
3. Use appropriate replication technology for
MySQL*

* As in Tungsten :)



NuoDB overview
Robert Buck


The Elastically Scalable Database™

Copyright © 2012 NuoDB


NuoDB Summary
‣ SQL/ACID: Emergent Architecture
‣ Elastically Scale: Demand-driven, auto-balancing
‣ Single Logical Database: No sharding, no partitioning
‣ Run Anywhere: On-Premises / Private, Public, or Hybrid Cloud
‣ Store Anywhere: KV-Store backend (flexible, e.g. Files, S3, HDFS)
‣ Multi-Tenant: DBs share machines dynamically
‣ Geo-Distributed: Active/active across several data centers
‣ Extreme Availability: Massively redundant, permitting live upgrade
‣ Minimal Admin: No knobs, auto everything

Copyright © 2012 NuoDB 107


Emergent Database Architecture

“An emergent
behavior can
appear when a
number of simple
agents operate in
an environment,
forming more
complex behaviors
as a collective.”

- Wikipedia



Instantly Elastic
‣ Second machine more
than doubles TPS due to
caching effects

‣ Second machine is
added to live database
while it is running at
1,000’s of TPS

‣ Performance increase is
immediate

‣ BTW - you can take Second Machine
either machine away Instant Performance Increase
and the database keeps
running without data
loss



Linearly Scalable

30000"
Nodes TPS

MySQL 1 3,000 25000"

NuoDB 1 4,500 20000"
NuoDB 9 27,000
15000"

Technical Details: TPS 10000"

‣ 2-9 Tx engines
5000"
‣ 1 storage manager
‣ Best sustained TPS and #
0"
clients combination
1" 2" 3" 4" 5" 6" 7" 8" 9"
‣ 50% updates

Number of Nodes

NuoDB running on 9 nodes was approx. 9x faster than MySQL running
on 1 node.


... even on Amazon EC2
‣ NuoDB scales linearly
12000"

on EC2
10000"

‣ EC2 performance
bound by AMZ TCP/ 8000"

IP traffic shaping
‣ Linearly scales at 50%
6000"
TPS
of rate for physical
4000"

commodity hardware
‣ Whereas, Amazon 2000"

RDS runs on 1 node,
and gets overloaded 0"
1" 2" 3" 4" 5" 6" 7" 8" 9"

at 10+ connections
Number of EC2 Nodes



We Work with Your SQL Tools

MS Excel (and other MS tools)

SQuirreLSQL

DbVisualizer


NuoDB System Architecture

TM1
TM1 SM
TM1
TM1
TM1
TM1
B1
B1 TM1
B1 TM1
TM1 SM
B TM1
TM1
TM1
TM1
TM1
NuoDB Brokers TM1
TE SM
Application
(JDBC, ODBC, SQL) NuoDB Transaction NuoDB Storage
Engines Manager

Tx Scalability Add as many of B, TE, or SM as you like
I/O Scalability Distributed IO, e.g. Hadoop HDFS
100% Uptime No single point of failure, geo-distributed
Optimal Utilization Multiple Databases on ﬁnite resources
Low People Costs Single Console Management



NuoDB Summary
‣ 100% SQL
‣ 100% ACID
‣ 100% Elastic Scaling
‣ If you have more questions...
‣ Prove it to yourself. Try it out!
‣ Download it at: www.nuodb.com
‣ Chat with us afterward



NuoDB
The Elastically Scalable Database™

Copyright © 2012 NuoDB


Diagnosis and possible
ﬁxes
David Holoboff


DIAGNOSIS AND
POSSIBLE FIXES:

A MULTIPLE CHOICE PRIMER
FOR BEGINNING DBAS


This talk is for entertainment value only.
Any perceived similarity between what you
will hear in the next ﬁve minutes and true
life scenarios are merely coincidental, with
no intent of harm or liability.


Are you being asked the right question?


What do you do with a slow database server?


What do you do if swap memory is always being used up?


OLTP and OLAP
Lawrence Schwartz


OLTP and OLAP
Have Your Cake and Eat it Too!

Lawrence Schwartz


Avoiding the B-tree Valley of Death

“The real end-game for Big Data is to have transactional a
analytic data on the same database.”

However, Today’s database categories - OLAP, OLTP are
compromises to get around dated, rigid indexing technology

B-tree
Valley of
Usage

Death
OLAP OLTP
Columnar DBs Hadoop
MongoDB

Read-intensive Write-intensive


B-trees Perform Poorly on Disk

MySQL + InnoDB MySQL + TokuDB
Based on B-Tree Indexes Based on Fractal Tree Indexes

• Random Workloads Lead To Frequent • Continuous Rebalancing and
Drive Head Movement Aggregation at Nodes

• MySQL Can’t Take Advantage of Full • Eliminates Fragmentation,
Drive I/O Potential Maximizes Disk I/O


B-trees Can’t Handle Big Data


• Performance Drops to a Trickle above • 20x Insertion Rates at One
50-100M Rows Billion Rows

• Delayed Checkpoints Diminish • Stand Up More Indexes for
Performance Better Query Performance


B-trees Aren’t Agile


• Big Database Schema Changes Take • Changes Complete in Seconds
Hours To Complete
• Allows Business' Data Models
• Changes Have to Wait for Scheduled to Immediately Adapt
Downtime


B-trees Don’t Compress Well


• Compression ~2x to ~2.5x Typical • High Compression

• Compression Impacts Performance • No Impact to Performance


TokuDB Eliminates Death Valley

“While TokuDB® is effectively an operational database technology, it
does blur the lines between operations and analytics”

eCommerce
Personalization
Social
Cloud Networks
Assets
Logfile analysis

Fractal Tree® Indexes
are the Solution
Usage

Real Time
OLAP Analytic
OLTP
OLAP OLTP

Read-intensive Write-intensive


Building a scalable
system
Baron Schwartz


How to become a rock
star DBA
Sheeri Cabral


How to Become a Rock Star DBA

Sheeri Cabral, Mozilla
Oracle ACE Director
MySQL Community Award Winner^2


Do what you love....


What you really love


Have a Twin Brother


Code


Write

Tutorials / HOW-TOs

MariaDB/Percona Documentation

forums.mysql.com
#irc


Make MisteaksMistakes


Teach others


Teach others

You know more than you think you do!


Keep Learning


Share


Organize

User Group Magazine

Ignite Event
Unconference


Connect People


Avoid burnout


Have a hobby


Have Support


Be tall...or not


Find Ways Others
Can Help You


Some Help You Can Get

MySQL Community Team
Volunteers – UG video: Richard Laskey
“What can I help you do?”
Technocation for grants
- meetup.com fees
- Open SQL/DB Camp
- Postgres Open
- Open Source Bridge
- Drizzle Day

(donate to www.technocation.org!)


It Takes a Community

Permission to use pictures granted by:

Ronald Bradford
Francesca Simon-Carboni Photography
Colin Charles
Jeremy Cole
James Duncan Davidson
Keith Larson
Gerry Narvaja


Cruelty on database
clusters
Giuseppe Maxia


Cruel things that you can
inflict on your replicated
cluster ... and get away with it!
Giuseppe Maxia
Continuent, Inc

©Continuent 2012.


I must confess



I must confess

I am a serial killer



I kill



I kill

• database servers



I kill

• client connections



I kill

• transactions



I kill

• transactions
• "lesystems



I am armed and dangerous

+
= "class 2" weapon


And of course I carry ...

(GASP) Perl!



I have a criminal mindset

• Like a bank robber

• or an airplane hijacker

• I look around, and watch for weak spots to
exploit



Whenever I get a new database
cluster
• I open "re

:)
@ @

-
$_

#!
#!
#!
#!
#!
#!



when a cluster is working
just "ne ...



just "ne ...

• I pull the rug from under
its feet



just "ne ...

its feet
• I switch o# a master's
database server



just "ne ...

its feet
• I switch o# a master's
database server
• or I delete its "les


I am a Harry Potter fan



So I get carried away, and I use the
unforgivable curses

$ kill -9



We have di#erent
standpoints



We have di#erent
standpoints

• My colleagues say that I
am cruel



We have di#erent
standpoints

• My colleagues say that I
am cruel
• I call it QA



Have you ever reported a bug ...
• about a failure that happens after
9000 switches?
• Or a master that slows down
when handling 512 slaves?



Have you ever reported a bug ...
• about a failure that happens after
9000 switches?
• Or a master that slows down
when handling 512 slaves?

I DID


Things I do that make developers say
"it is not fair"



"it is not fair"
• using the installer with wrong options



"it is not fair"

• running the installer without a mandatory
option



"it is not fair"

option

• Using the wrong version of Java, or Ruby, or
MySQL



"it is not fair"

option

MySQL

• running the installer in a system with only
200 MB of free storage



"it is not fair"

option

MySQL

• running the installer in a system with only
200 MB of free storage

• Counting the records after a performance
test (and "nding that some are missing)



Why do I do cruel things to replication
clusters?



clusters?
• Because, most of the time:
• Developers are not users
• Users do what developers can't
imagine



clusters?
• Because, most of the time:
• Developers are not users
• Users do what developers can't
imagine

• To make a fool-proof
system you need to act
like a fool


in the end, I get away with cruelty

I am not bad. I am
just drawn that way
(Jessica Rabbit)



Lightning talks percona live mysql_2012

Recommended

Recommended

More Related Content

Similar to Lightning talks percona live mysql_2012

Similar to Lightning talks percona live mysql_2012 (20)

More from Giuseppe Maxia

More from Giuseppe Maxia (20)

Recently uploaded

Recently uploaded (20)

Lightning talks percona live mysql_2012