Your SlideShare is downloading. ×
0
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Appnexus scalability
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Appnexus scalability

8,288

Published on

Achieving scalability - eliminating ORM usage, and sweeping slow query log

Achieving scalability - eliminating ORM usage, and sweeping slow query log

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
8,288
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 1 Database Scalability Sweep the Query Log & Keep it Clean
  • 2. 2 What Is Scalability? Webserver tier - Apache, Nginx Object caches - Memcache Search servers - Sphinx Queueing servers - SQS, RabbitMQ
  • 3. 3 Why is DB hard to Scale? Relational Databases Single authoritative master Scaling a write db is hard Durability SQL code weight non-obvious
  • 4. 4 ORM Advantages Middleware, multi-database support Reusable code, encapsulation & abstraction Time saving - auto generated model objects Built for wide audience, reduce need for SQL Sacrifice a bit of efficiency for code agility
  • 5. 5 ORMs + Scalability Negates standard optimization techniques Difficult to divorce after you are wedded Limit query tuning at the bare metal Wide audience not high-scale applications Bad at deducing JOINs Fetch 20 cols when you need only 2? “death by a thousand queries” - Laurie Voss
  • 6. 6 ORM ExampleEx. from http://mattiasgeniar.be <?php // First, get all the companies in your database $companies = $this->getAllCompanies(); $totalValue = 0; foreach ($companies as $company) { // For each company there is, retrieve the total value of all their orders $orders = $company->getOrders(); foreach ($orders as $order) { $totalValue += (float) $order->getValue(); } echo "This company made us ". $totalValue ." euro already.";
  • 7. 7 ORM Example<?php // First, get all the companies in your database SELECT * FROM "company"; $totalValue = 0; foreach (...) { // For each company there is, retrieve the total value of all their orders SELECT * FROM "order" WHERE companyid = $companyid; foreach (...) { $totalValue += $row->value; } echo "This company made us ". $totalValue ." euro already."; }
  • 8. 8 ORM Example as SQL SELECT SUM(o.VALUE) AS TotalValue, c.name AS CompanyName FROM company AS c LEFT JOIN "order" AS o ON o.companyid = c.companyid GROUP BY o.companyid;
  • 9. 9 ORM Articles ORM is an anti-pattern - Laurie Voss Bad ORM is infinitely worse than bad SQL - Mattias Geniar Case against ORM frameworks - Todd Hoff 5 things toxic to scalability - Sean Hull
  • 10. 10 Database Tuning Process Identify heavy queries in slow log Find queries in code Rewrite, run explain & profile Benchmark with large dataset Deploy fixed query
  • 11. 11 Being Reactive? Sweep out the slow query log Rewrite queries
  • 12. 12 Being Proactive Keep slow query log quiet With new code deploys, new slow queries will pop up Tune early, tune often
  • 13. 13 Use Profiling set profiling=1; <run query> show profile for query 1;
  • 14. 14 Enable Session Profiling (sean@localhost:mysql.sock) [sakila]> set profiling = 1; Query OK, 0 rows affected (0.00 sec) (sean@localhost:mysql.sock) [sakila]> select customer_id, last_name from customer where customer_id = 5; +-------------+-----------+ | customer_id | last_name | +-------------+-----------+ | 5 | BROWN | +-------------+-----------+ 1 row in set (0.00 sec) (sean@localhost:mysql.sock) [sakila]> select SQL_NO_CACHE customer_id, last_name from customer where customer_id = 5; +-------------+-----------+ | customer_id | last_name | +-------------+-----------+
  • 15. 15 Anatomy of a Profile (sean@localhost:mysql.sock) [sakila]> show profiles; +----------+------------+--------------------------------------------------------------------------------+ | Query_ID | Duration | Query | +----------+------------+--------------------------------------------------------------------------------+ | 1 | 0.00015500 | select customer_id, last_name from customer where customer_id = 5 | | 2 | 0.00054700 | select SQL_NO_CACHE customer_id, last_name from customer where customer_id = 5 | +----------+------------+--------------------------------------------------------------------------------+ 2 rows in set (0.00 sec) (sean@localhost:mysql.sock) [sakila]> show profile for query 1; +--------------------------------+----------+ | Status | Duration | +--------------------------------+----------+ | starting | 0.000062 | | checking query cache for query | 0.000020 | | checking privileges on cached | 0.000019 | | sending cached result to clien | 0.000033 |
  • 16. 16 Non Cached Query Profile | Status | Duration | +--------------------------------+----------+ | starting | 0.000052 | | checking query cache for query | 0.000103 | | Opening tables | 0.000025 | | System lock | 0.000014 | | Table lock | 0.000022 | | init | 0.000043 | | optimizing | 0.000022 | | statistics | 0.000095 | | preparing | 0.000030 | | executing | 0.000011 | | Sending data | 0.000045 | | end | 0.000013 | | end | 0.000010 | | query end | 0.000012 |
  • 17. 17 Single Column Index create index d_idx on a (d); select email from a where d=99;
  • 18. 18 Multi-column Index create index def_idx on a (d, e, f); SELECT email FROM a WHERE d=1 and e=2 and f=3;
  • 19. 19 Left-most Prefix Index create index abc_idx on a (d, e, f); SELECT name FROM a where d = 1 and e = 2; SELECT name FROM a WHERE d=1 and e=2 and f=3; SELECT name FROM a WHERE e=2;
  • 20. 20 Covering Index create index cover_id on (id, name); SELECT name FROM a WHERE id = 5;
  • 21. 21 Use EXPLAIN Illustrates database engine’s path to the data Shows logical & physical I/Os Shows sorting, temp tables, index usage
  • 22. 22 EXPLAIN - no index (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = 'ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | customer | ALL | NULL | NULL | NULL | NULL | 541 | Using where | +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ 1 row in set (0.00 sec)
  • 23. 23 EXPLAIN - basic index (sean@localhost:mysql.sock) [sakila]> create index cust_email on customer(email); (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = ' ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ | 1 | SIMPLE | customer | ref | cust_email | cust_email | 153 | const | 1 | Using where | +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ 1 row in set (0.02 sec)
  • 24. 24 EXPLAIN - covering index (sean@localhost:mysql.sock) [sakila]> create index cust_email_ln on customer(email, last_name); (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = ' ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ | 1 | SIMPLE | customer | ref | cust_email_ln | cust_email_ln | 153 | const | 1 | Using where; Using index | +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ 1 row in set (0.00 sec)
  • 25. 25 Test With Large Datasets As data grows, indexing more crucial Test with million row tables Slow desktop or test boxes?
  • 26. 26 Can Devops Help? Teamwork - way of working Less siloing of departments DBAs helping identify queries DBAs helping rewrite queries Devs on top of new code deploys Clean slow-log as new queries popup
  • 27. 27 About Sean Hull shull@iheavy.com Join 8000 & please follow me: @hullsean www.iheavy.com/blog www.iheavy.com/signup-scalable-startups-newsletter mobile: +1-917-442-3939

×