Your SlideShare is downloading. ×
Appnexus scalability
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Appnexus scalability

8,276
views

Published on

Achieving scalability - eliminating ORM usage, and sweeping slow query log

Achieving scalability - eliminating ORM usage, and sweeping slow query log

Published in: Technology

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
8,276
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 1 Database Scalability Sweep the Query Log & Keep it Clean
  • 2. 2 What Is Scalability? Webserver tier - Apache, Nginx Object caches - Memcache Search servers - Sphinx Queueing servers - SQS, RabbitMQ
  • 3. 3 Why is DB hard to Scale? Relational Databases Single authoritative master Scaling a write db is hard Durability SQL code weight non-obvious
  • 4. 4 ORM Advantages Middleware, multi-database support Reusable code, encapsulation & abstraction Time saving - auto generated model objects Built for wide audience, reduce need for SQL Sacrifice a bit of efficiency for code agility
  • 5. 5 ORMs + Scalability Negates standard optimization techniques Difficult to divorce after you are wedded Limit query tuning at the bare metal Wide audience not high-scale applications Bad at deducing JOINs Fetch 20 cols when you need only 2? “death by a thousand queries” - Laurie Voss
  • 6. 6 ORM ExampleEx. from http://mattiasgeniar.be <?php // First, get all the companies in your database $companies = $this->getAllCompanies(); $totalValue = 0; foreach ($companies as $company) { // For each company there is, retrieve the total value of all their orders $orders = $company->getOrders(); foreach ($orders as $order) { $totalValue += (float) $order->getValue(); } echo "This company made us ". $totalValue ." euro already.";
  • 7. 7 ORM Example<?php // First, get all the companies in your database SELECT * FROM "company"; $totalValue = 0; foreach (...) { // For each company there is, retrieve the total value of all their orders SELECT * FROM "order" WHERE companyid = $companyid; foreach (...) { $totalValue += $row->value; } echo "This company made us ". $totalValue ." euro already."; }
  • 8. 8 ORM Example as SQL SELECT SUM(o.VALUE) AS TotalValue, c.name AS CompanyName FROM company AS c LEFT JOIN "order" AS o ON o.companyid = c.companyid GROUP BY o.companyid;
  • 9. 9 ORM Articles ORM is an anti-pattern - Laurie Voss Bad ORM is infinitely worse than bad SQL - Mattias Geniar Case against ORM frameworks - Todd Hoff 5 things toxic to scalability - Sean Hull
  • 10. 10 Database Tuning Process Identify heavy queries in slow log Find queries in code Rewrite, run explain & profile Benchmark with large dataset Deploy fixed query
  • 11. 11 Being Reactive? Sweep out the slow query log Rewrite queries
  • 12. 12 Being Proactive Keep slow query log quiet With new code deploys, new slow queries will pop up Tune early, tune often
  • 13. 13 Use Profiling set profiling=1; <run query> show profile for query 1;
  • 14. 14 Enable Session Profiling (sean@localhost:mysql.sock) [sakila]> set profiling = 1; Query OK, 0 rows affected (0.00 sec) (sean@localhost:mysql.sock) [sakila]> select customer_id, last_name from customer where customer_id = 5; +-------------+-----------+ | customer_id | last_name | +-------------+-----------+ | 5 | BROWN | +-------------+-----------+ 1 row in set (0.00 sec) (sean@localhost:mysql.sock) [sakila]> select SQL_NO_CACHE customer_id, last_name from customer where customer_id = 5; +-------------+-----------+ | customer_id | last_name | +-------------+-----------+
  • 15. 15 Anatomy of a Profile (sean@localhost:mysql.sock) [sakila]> show profiles; +----------+------------+--------------------------------------------------------------------------------+ | Query_ID | Duration | Query | +----------+------------+--------------------------------------------------------------------------------+ | 1 | 0.00015500 | select customer_id, last_name from customer where customer_id = 5 | | 2 | 0.00054700 | select SQL_NO_CACHE customer_id, last_name from customer where customer_id = 5 | +----------+------------+--------------------------------------------------------------------------------+ 2 rows in set (0.00 sec) (sean@localhost:mysql.sock) [sakila]> show profile for query 1; +--------------------------------+----------+ | Status | Duration | +--------------------------------+----------+ | starting | 0.000062 | | checking query cache for query | 0.000020 | | checking privileges on cached | 0.000019 | | sending cached result to clien | 0.000033 |
  • 16. 16 Non Cached Query Profile | Status | Duration | +--------------------------------+----------+ | starting | 0.000052 | | checking query cache for query | 0.000103 | | Opening tables | 0.000025 | | System lock | 0.000014 | | Table lock | 0.000022 | | init | 0.000043 | | optimizing | 0.000022 | | statistics | 0.000095 | | preparing | 0.000030 | | executing | 0.000011 | | Sending data | 0.000045 | | end | 0.000013 | | end | 0.000010 | | query end | 0.000012 |
  • 17. 17 Single Column Index create index d_idx on a (d); select email from a where d=99;
  • 18. 18 Multi-column Index create index def_idx on a (d, e, f); SELECT email FROM a WHERE d=1 and e=2 and f=3;
  • 19. 19 Left-most Prefix Index create index abc_idx on a (d, e, f); SELECT name FROM a where d = 1 and e = 2; SELECT name FROM a WHERE d=1 and e=2 and f=3; SELECT name FROM a WHERE e=2;
  • 20. 20 Covering Index create index cover_id on (id, name); SELECT name FROM a WHERE id = 5;
  • 21. 21 Use EXPLAIN Illustrates database engine’s path to the data Shows logical & physical I/Os Shows sorting, temp tables, index usage
  • 22. 22 EXPLAIN - no index (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = 'ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | customer | ALL | NULL | NULL | NULL | NULL | 541 | Using where | +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ 1 row in set (0.00 sec)
  • 23. 23 EXPLAIN - basic index (sean@localhost:mysql.sock) [sakila]> create index cust_email on customer(email); (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = ' ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ | 1 | SIMPLE | customer | ref | cust_email | cust_email | 153 | const | 1 | Using where | +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ 1 row in set (0.02 sec)
  • 24. 24 EXPLAIN - covering index (sean@localhost:mysql.sock) [sakila]> create index cust_email_ln on customer(email, last_name); (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = ' ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ | 1 | SIMPLE | customer | ref | cust_email_ln | cust_email_ln | 153 | const | 1 | Using where; Using index | +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ 1 row in set (0.00 sec)
  • 25. 25 Test With Large Datasets As data grows, indexing more crucial Test with million row tables Slow desktop or test boxes?
  • 26. 26 Can Devops Help? Teamwork - way of working Less siloing of departments DBAs helping identify queries DBAs helping rewrite queries Devs on top of new code deploys Clean slow-log as new queries popup
  • 27. 27 About Sean Hull shull@iheavy.com Join 8000 & please follow me: @hullsean www.iheavy.com/blog www.iheavy.com/signup-scalable-startups-newsletter mobile: +1-917-442-3939

×