0
1
Database Scalability
Sweep the Query Log & Keep it Clean
2
What Is Scalability?
Webserver tier - Apache, Nginx
Object caches - Memcache
Search servers - Sphinx
Queueing servers - ...
3
Why is DB hard to
Scale?
Relational Databases
Single authoritative master
Scaling a write db is hard
Durability
SQL code...
4
ORM Advantages
Middleware, multi-database support
Reusable code, encapsulation & abstraction
Time saving - auto generate...
5
ORMs + Scalability
Negates standard optimization techniques
Difficult to divorce after you are wedded
Limit query tuning...
6
ORM ExampleEx. from http://mattiasgeniar.be
<?php
// First, get all the companies in your database
$companies = $this->g...
7
ORM Example<?php
// First, get all the companies in your database
SELECT * FROM "company";
$totalValue = 0;
foreach (......
8
ORM Example as SQL
SELECT SUM(o.VALUE) AS TotalValue, c.name AS CompanyName
FROM company AS c
LEFT JOIN "order" AS o ON ...
9
ORM Articles
ORM is an anti-pattern - Laurie Voss
Bad ORM is infinitely worse than bad SQL - Mattias
Geniar
Case against...
10
Database Tuning
Process
Identify heavy queries in slow log
Find queries in code
Rewrite, run explain & profile
Benchmar...
11
Being Reactive?
Sweep out the slow query log
Rewrite queries
12
Being Proactive
Keep slow query log quiet
With new code deploys, new slow queries will pop up
Tune early, tune often
13
Use Profiling
set profiling=1;
<run query>
show profile for query 1;
14
Enable Session
Profiling
(sean@localhost:mysql.sock) [sakila]> set profiling = 1;
Query OK, 0 rows affected (0.00 sec)
...
15
Anatomy of a Profile
(sean@localhost:mysql.sock) [sakila]> show profiles;
+----------+------------+--------------------...
16
Non Cached Query
Profile
| Status | Duration |
+--------------------------------+----------+
| starting | 0.000052 |
| ...
17
Single Column Index
create index d_idx on a (d);
select email from a where d=99;
18
Multi-column Index
create index def_idx on a (d, e, f);
SELECT email FROM a WHERE d=1 and e=2 and
f=3;
19
Left-most Prefix Index
create index abc_idx on a (d, e, f);
SELECT name FROM a where d = 1 and e = 2;
SELECT name FROM ...
20
Covering Index
create index cover_id on (id, name);
SELECT name FROM a WHERE id = 5;
21
Use EXPLAIN
Illustrates database engine’s path to the data
Shows logical & physical I/Os
Shows sorting, temp tables, in...
22
EXPLAIN - no index
(sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = 'ELIZABET...
23
EXPLAIN - basic index
(sean@localhost:mysql.sock) [sakila]> create index cust_email on customer(email);
(sean@localhost...
24
EXPLAIN - covering
index
(sean@localhost:mysql.sock) [sakila]> create index cust_email_ln on customer(email, last_name)...
25
Test With Large
Datasets
As data grows, indexing more crucial
Test with million row tables
Slow desktop or test boxes?
26
Can Devops Help?
Teamwork - way of working
Less siloing of departments
DBAs helping identify queries
DBAs helping rewri...
27
About Sean Hull
shull@iheavy.com
Join 8000 & please follow me: @hullsean
www.iheavy.com/blog
www.iheavy.com/signup-scal...
Upcoming SlideShare
Loading in...5
×

Appnexus scalability

8,309

Published on

Achieving scalability - eliminating ORM usage, and sweeping slow query log

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
8,309
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Appnexus scalability "

  1. 1. 1 Database Scalability Sweep the Query Log & Keep it Clean
  2. 2. 2 What Is Scalability? Webserver tier - Apache, Nginx Object caches - Memcache Search servers - Sphinx Queueing servers - SQS, RabbitMQ
  3. 3. 3 Why is DB hard to Scale? Relational Databases Single authoritative master Scaling a write db is hard Durability SQL code weight non-obvious
  4. 4. 4 ORM Advantages Middleware, multi-database support Reusable code, encapsulation & abstraction Time saving - auto generated model objects Built for wide audience, reduce need for SQL Sacrifice a bit of efficiency for code agility
  5. 5. 5 ORMs + Scalability Negates standard optimization techniques Difficult to divorce after you are wedded Limit query tuning at the bare metal Wide audience not high-scale applications Bad at deducing JOINs Fetch 20 cols when you need only 2? “death by a thousand queries” - Laurie Voss
  6. 6. 6 ORM ExampleEx. from http://mattiasgeniar.be <?php // First, get all the companies in your database $companies = $this->getAllCompanies(); $totalValue = 0; foreach ($companies as $company) { // For each company there is, retrieve the total value of all their orders $orders = $company->getOrders(); foreach ($orders as $order) { $totalValue += (float) $order->getValue(); } echo "This company made us ". $totalValue ." euro already.";
  7. 7. 7 ORM Example<?php // First, get all the companies in your database SELECT * FROM "company"; $totalValue = 0; foreach (...) { // For each company there is, retrieve the total value of all their orders SELECT * FROM "order" WHERE companyid = $companyid; foreach (...) { $totalValue += $row->value; } echo "This company made us ". $totalValue ." euro already."; }
  8. 8. 8 ORM Example as SQL SELECT SUM(o.VALUE) AS TotalValue, c.name AS CompanyName FROM company AS c LEFT JOIN "order" AS o ON o.companyid = c.companyid GROUP BY o.companyid;
  9. 9. 9 ORM Articles ORM is an anti-pattern - Laurie Voss Bad ORM is infinitely worse than bad SQL - Mattias Geniar Case against ORM frameworks - Todd Hoff 5 things toxic to scalability - Sean Hull
  10. 10. 10 Database Tuning Process Identify heavy queries in slow log Find queries in code Rewrite, run explain & profile Benchmark with large dataset Deploy fixed query
  11. 11. 11 Being Reactive? Sweep out the slow query log Rewrite queries
  12. 12. 12 Being Proactive Keep slow query log quiet With new code deploys, new slow queries will pop up Tune early, tune often
  13. 13. 13 Use Profiling set profiling=1; <run query> show profile for query 1;
  14. 14. 14 Enable Session Profiling (sean@localhost:mysql.sock) [sakila]> set profiling = 1; Query OK, 0 rows affected (0.00 sec) (sean@localhost:mysql.sock) [sakila]> select customer_id, last_name from customer where customer_id = 5; +-------------+-----------+ | customer_id | last_name | +-------------+-----------+ | 5 | BROWN | +-------------+-----------+ 1 row in set (0.00 sec) (sean@localhost:mysql.sock) [sakila]> select SQL_NO_CACHE customer_id, last_name from customer where customer_id = 5; +-------------+-----------+ | customer_id | last_name | +-------------+-----------+
  15. 15. 15 Anatomy of a Profile (sean@localhost:mysql.sock) [sakila]> show profiles; +----------+------------+--------------------------------------------------------------------------------+ | Query_ID | Duration | Query | +----------+------------+--------------------------------------------------------------------------------+ | 1 | 0.00015500 | select customer_id, last_name from customer where customer_id = 5 | | 2 | 0.00054700 | select SQL_NO_CACHE customer_id, last_name from customer where customer_id = 5 | +----------+------------+--------------------------------------------------------------------------------+ 2 rows in set (0.00 sec) (sean@localhost:mysql.sock) [sakila]> show profile for query 1; +--------------------------------+----------+ | Status | Duration | +--------------------------------+----------+ | starting | 0.000062 | | checking query cache for query | 0.000020 | | checking privileges on cached | 0.000019 | | sending cached result to clien | 0.000033 |
  16. 16. 16 Non Cached Query Profile | Status | Duration | +--------------------------------+----------+ | starting | 0.000052 | | checking query cache for query | 0.000103 | | Opening tables | 0.000025 | | System lock | 0.000014 | | Table lock | 0.000022 | | init | 0.000043 | | optimizing | 0.000022 | | statistics | 0.000095 | | preparing | 0.000030 | | executing | 0.000011 | | Sending data | 0.000045 | | end | 0.000013 | | end | 0.000010 | | query end | 0.000012 |
  17. 17. 17 Single Column Index create index d_idx on a (d); select email from a where d=99;
  18. 18. 18 Multi-column Index create index def_idx on a (d, e, f); SELECT email FROM a WHERE d=1 and e=2 and f=3;
  19. 19. 19 Left-most Prefix Index create index abc_idx on a (d, e, f); SELECT name FROM a where d = 1 and e = 2; SELECT name FROM a WHERE d=1 and e=2 and f=3; SELECT name FROM a WHERE e=2;
  20. 20. 20 Covering Index create index cover_id on (id, name); SELECT name FROM a WHERE id = 5;
  21. 21. 21 Use EXPLAIN Illustrates database engine’s path to the data Shows logical & physical I/Os Shows sorting, temp tables, index usage
  22. 22. 22 EXPLAIN - no index (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = 'ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | customer | ALL | NULL | NULL | NULL | NULL | 541 | Using where | +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ 1 row in set (0.00 sec)
  23. 23. 23 EXPLAIN - basic index (sean@localhost:mysql.sock) [sakila]> create index cust_email on customer(email); (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = ' ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ | 1 | SIMPLE | customer | ref | cust_email | cust_email | 153 | const | 1 | Using where | +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ 1 row in set (0.02 sec)
  24. 24. 24 EXPLAIN - covering index (sean@localhost:mysql.sock) [sakila]> create index cust_email_ln on customer(email, last_name); (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = ' ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ | 1 | SIMPLE | customer | ref | cust_email_ln | cust_email_ln | 153 | const | 1 | Using where; Using index | +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ 1 row in set (0.00 sec)
  25. 25. 25 Test With Large Datasets As data grows, indexing more crucial Test with million row tables Slow desktop or test boxes?
  26. 26. 26 Can Devops Help? Teamwork - way of working Less siloing of departments DBAs helping identify queries DBAs helping rewrite queries Devs on top of new code deploys Clean slow-log as new queries popup
  27. 27. 27 About Sean Hull shull@iheavy.com Join 8000 & please follow me: @hullsean www.iheavy.com/blog www.iheavy.com/signup-scalable-startups-newsletter mobile: +1-917-442-3939
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×