Appnexus scalability
Upcoming SlideShare
Loading in...5
×
 

Appnexus scalability

on

  • 1,554 views

Achieving scalability - eliminating ORM usage, and sweeping slow query log

Achieving scalability - eliminating ORM usage, and sweeping slow query log

Statistics

Views

Total Views
1,554
Views on SlideShare
1,550
Embed Views
4

Actions

Likes
0
Downloads
4
Comments
0

1 Embed 4

https://twitter.com 4

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Appnexus scalability Appnexus scalability Presentation Transcript

  • 1 Database Scalability Sweep the Query Log & Keep it Clean
  • 2 What Is Scalability? Webserver tier - Apache, Nginx Object caches - Memcache Search servers - Sphinx Queueing servers - SQS, RabbitMQ
  • 3 Why is DB hard to Scale? Relational Databases Single authoritative master Scaling a write db is hard Durability SQL code weight non-obvious
  • 4 ORM Advantages Middleware, multi-database support Reusable code, encapsulation & abstraction Time saving - auto generated model objects Built for wide audience, reduce need for SQL Sacrifice a bit of efficiency for code agility
  • 5 ORMs + Scalability Negates standard optimization techniques Difficult to divorce after you are wedded Limit query tuning at the bare metal Wide audience not high-scale applications Bad at deducing JOINs Fetch 20 cols when you need only 2? “death by a thousand queries” - Laurie Voss
  • 6 ORM ExampleEx. from http://mattiasgeniar.be <?php // First, get all the companies in your database $companies = $this->getAllCompanies(); $totalValue = 0; foreach ($companies as $company) { // For each company there is, retrieve the total value of all their orders $orders = $company->getOrders(); foreach ($orders as $order) { $totalValue += (float) $order->getValue(); } echo "This company made us ". $totalValue ." euro already.";
  • 7 ORM Example<?php // First, get all the companies in your database SELECT * FROM "company"; $totalValue = 0; foreach (...) { // For each company there is, retrieve the total value of all their orders SELECT * FROM "order" WHERE companyid = $companyid; foreach (...) { $totalValue += $row->value; } echo "This company made us ". $totalValue ." euro already."; }
  • 8 ORM Example as SQL SELECT SUM(o.VALUE) AS TotalValue, c.name AS CompanyName FROM company AS c LEFT JOIN "order" AS o ON o.companyid = c.companyid GROUP BY o.companyid;
  • 9 ORM Articles ORM is an anti-pattern - Laurie Voss Bad ORM is infinitely worse than bad SQL - Mattias Geniar Case against ORM frameworks - Todd Hoff 5 things toxic to scalability - Sean Hull
  • 10 Database Tuning Process Identify heavy queries in slow log Find queries in code Rewrite, run explain & profile Benchmark with large dataset Deploy fixed query
  • 11 Being Reactive? Sweep out the slow query log Rewrite queries
  • 12 Being Proactive Keep slow query log quiet With new code deploys, new slow queries will pop up Tune early, tune often
  • 13 Use Profiling set profiling=1; <run query> show profile for query 1;
  • 14 Enable Session Profiling (sean@localhost:mysql.sock) [sakila]> set profiling = 1; Query OK, 0 rows affected (0.00 sec) (sean@localhost:mysql.sock) [sakila]> select customer_id, last_name from customer where customer_id = 5; +-------------+-----------+ | customer_id | last_name | +-------------+-----------+ | 5 | BROWN | +-------------+-----------+ 1 row in set (0.00 sec) (sean@localhost:mysql.sock) [sakila]> select SQL_NO_CACHE customer_id, last_name from customer where customer_id = 5; +-------------+-----------+ | customer_id | last_name | +-------------+-----------+
  • 15 Anatomy of a Profile (sean@localhost:mysql.sock) [sakila]> show profiles; +----------+------------+--------------------------------------------------------------------------------+ | Query_ID | Duration | Query | +----------+------------+--------------------------------------------------------------------------------+ | 1 | 0.00015500 | select customer_id, last_name from customer where customer_id = 5 | | 2 | 0.00054700 | select SQL_NO_CACHE customer_id, last_name from customer where customer_id = 5 | +----------+------------+--------------------------------------------------------------------------------+ 2 rows in set (0.00 sec) (sean@localhost:mysql.sock) [sakila]> show profile for query 1; +--------------------------------+----------+ | Status | Duration | +--------------------------------+----------+ | starting | 0.000062 | | checking query cache for query | 0.000020 | | checking privileges on cached | 0.000019 | | sending cached result to clien | 0.000033 |
  • 16 Non Cached Query Profile | Status | Duration | +--------------------------------+----------+ | starting | 0.000052 | | checking query cache for query | 0.000103 | | Opening tables | 0.000025 | | System lock | 0.000014 | | Table lock | 0.000022 | | init | 0.000043 | | optimizing | 0.000022 | | statistics | 0.000095 | | preparing | 0.000030 | | executing | 0.000011 | | Sending data | 0.000045 | | end | 0.000013 | | end | 0.000010 | | query end | 0.000012 |
  • 17 Single Column Index create index d_idx on a (d); select email from a where d=99;
  • 18 Multi-column Index create index def_idx on a (d, e, f); SELECT email FROM a WHERE d=1 and e=2 and f=3;
  • 19 Left-most Prefix Index create index abc_idx on a (d, e, f); SELECT name FROM a where d = 1 and e = 2; SELECT name FROM a WHERE d=1 and e=2 and f=3; SELECT name FROM a WHERE e=2;
  • 20 Covering Index create index cover_id on (id, name); SELECT name FROM a WHERE id = 5;
  • 21 Use EXPLAIN Illustrates database engine’s path to the data Shows logical & physical I/Os Shows sorting, temp tables, index usage
  • 22 EXPLAIN - no index (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = 'ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | customer | ALL | NULL | NULL | NULL | NULL | 541 | Using where | +----+-------------+----------+------+---------------+------+---------+------+------+-------------+ 1 row in set (0.00 sec)
  • 23 EXPLAIN - basic index (sean@localhost:mysql.sock) [sakila]> create index cust_email on customer(email); (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = ' ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ | 1 | SIMPLE | customer | ref | cust_email | cust_email | 153 | const | 1 | Using where | +----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+ 1 row in set (0.02 sec)
  • 24 EXPLAIN - covering index (sean@localhost:mysql.sock) [sakila]> create index cust_email_ln on customer(email, last_name); (sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = ' ELIZABETH.BROWN@sakilacustomer.org'; +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ | 1 | SIMPLE | customer | ref | cust_email_ln | cust_email_ln | 153 | const | 1 | Using where; Using index | +----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+ 1 row in set (0.00 sec)
  • 25 Test With Large Datasets As data grows, indexing more crucial Test with million row tables Slow desktop or test boxes?
  • 26 Can Devops Help? Teamwork - way of working Less siloing of departments DBAs helping identify queries DBAs helping rewrite queries Devs on top of new code deploys Clean slow-log as new queries popup
  • 27 About Sean Hull shull@iheavy.com Join 8000 & please follow me: @hullsean www.iheavy.com/blog www.iheavy.com/signup-scalable-startups-newsletter mobile: +1-917-442-3939