• Like
  • Save
What every developer should know about Performance
Upcoming SlideShare
Loading in...5
×
 

What every developer should know about Performance

on

  • 1,279 views

 

Statistics

Views

Total Views
1,279
Views on SlideShare
1,255
Embed Views
24

Actions

Likes
0
Downloads
15
Comments
0

2 Embeds 24

http://www.scoop.it 20
http://www.linkedin.com 4

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    What every developer should know about Performance What every developer should know about Performance Presentation Transcript

    • What every developer should know about Performance. Morgan Tocker <morgan@percona.com> 1Friday, April 22, 2011
    • About Percona ★ Professional Services company offering training, consulting, support, development around MySQL. ★ Best known for “the blog and the red book”. ✦ mysqlperformanceblog.com ✦ High Performance MySQL 2nd Ed. 2Friday, April 22, 2011
    • Introduction ★ I normally give talks on “Optimizing MySQL” or something else related. ✦ I’m very happy not to be doing this again, this year. ★ My view of performance is skewed with bitterness. ✦ I dislike the word “scalability”. ✦ Scalability is misapplied to every axis (data growth, concurrent requests, etc.) 3Friday, April 22, 2011
    • Agenda ★ Short Stories ★ Where and why we measure. ★ The suite of definitions that are not response. ★ Practical example of optimization. ★ Fight bad measurement. 4Friday, April 22, 2011
    • Short StoriesFriday, April 22, 2011
    • The average temperature of the patient in the hospital is not important. Lesson: Customers feel variance.Friday, April 22, 2011
    • Would you like cockroaches with your fries? Lesson: Customers feel variance.Friday, April 22, 2011
    • Fred & Ted go to a conference. Item Unit Cost Sub Total Item Unit Cost Sub Total Hotel $99 $396 Hotel $130 $520 Car Rental $30 $150 Taxis $30 $60 Parking $20 $80 $580 Petrol $20 $20 $646 Lesson: You need a profile to optimize.Friday, April 22, 2011
    • Looking at a CPU for ‘load’ is like looking at a bank machine over a 24 hr period. Lesson: You need a profile to optimize.Friday, April 22, 2011
    • “Random Arrivals” ★ Not all tasks arrive on time. Take the following example of a manufacturing process: Each widget is exactly one second apart. Mechanical arm can pick up 1 widget/second, stamp it, and place it on the second belt. 10Friday, April 22, 2011
    • “Random Arrivals” ★ Not all tasks arrive on time. Take the following example of a manufacturing process: Each widget is exactly one second apart. Mechanical arm can pick up 1 widget/second, stamp it, and place it on the second belt. 10Friday, April 22, 2011
    • “Random Arrivals” ★ Not all tasks arrive on time. Take the following example of a manufacturing process: M Each widget is exactly one second apart. Mechanical arm can pick up 1 widget/second, stamp it, and place it on the second belt. 10Friday, April 22, 2011
    • “Random Arrivals” ★ Not all tasks arrive on time. Take the following example of a manufacturing process: M M Each widget is exactly one second apart. Mechanical arm can pick up 1 widget/second, stamp it, and place it on the second belt. 10Friday, April 22, 2011
    • Throughput Question ★ There is only one mechanical arm - no parallelism is possible. ✦ Service time of the mechanical arm is 1 second. ✦ Maximum capacity is 60 boxes/minute. ★ Can we have a throughput of 60 boxes/minute and a response time of 1 second? In this example we can. But only because the arrival rate of the widgets is controlled. 11Friday, April 22, 2011
    • Important Real-Life Difference ★ The arrival rate of requests is not evenly distributed: M A lot of queuing Timeslice is not 0.5 seconds apart. applies to this last used - and ‘lost’ Some queuing has request. forever. to apply. 12Friday, April 22, 2011
    • Where and Why (a.k.a. “I already know where the problem is”)Friday, April 22, 2011
    • Do you really? WWW WWW WWW WWW 15 second login? MySQL WWW WWW WWW WWW 14Friday, April 22, 2011
    • If you said that... ★ The Database ✦ You’ll be right most of the time - but you’re not being 100% honest with yourself. ✦ The database has more scalability challenges than the other components. For the most part we can just add web servers. 15Friday, April 22, 2011
    • However; ★ We can lead ourselves into a real trap by guessing based on previous experience. ★ Proving is probably a lot more important than knowing. 16Friday, April 22, 2011
    • What’s interesting... ★ What’s more interesting than drawing the stack is drawing the flow of information between each component of the stack. ★ It’s important to be able to do this while users execute tasks. 17Friday, April 22, 2011
    • Following the flow ★ For a given task, measure the breakdown in time: Browser Web Server Database Server Submit Login Form Check if user exists Return Results Update Last Login Date Confirm Render page Shown as a UML sequence diagram. A profile contains 18 the same data, but aggregated.Friday, April 22, 2011
    • Wait, what.!? ★ Updating the last_login_date takes a sizeable amount of time? ★ For the value that it provides, why are we spending so long on that sub-task? 19Friday, April 22, 2011
    • My analysis: ★ Query is: ✦ UPDATE users SET last_login_date=NOW() WHERE id = N; ★ Schema is: ✦ CREATE TABLE users ( id INT NOT NULL PRIMARY KEY auto_increment, username CHAR(32) NOT NULL, .. last_login_date DATETIME, UNIQUE (username) ) ENGINE=MyISAM; 20Friday, April 22, 2011
    • mysql> show processlist; +------+------+-----------+------------------+---------+------+----------+-----------------------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +------+------+-----------+------------------+---------+------+----------+-----------------------------------------------------------+ | 1 | root | localhost | myapp_production | Query | 0 | NULL | show processlist | | 9688 | root | localhost | myapp_production | Query | 2 | Updating | UPDATE users SET last_login_date=NOW() WHERE id = 1657903 | | 9689 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 986755 | | 9690 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 607334 | | 9691 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1802251 | | 9692 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1076084 | | 9693 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 141037 | | 9694 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1418038 | | 9695 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1156819 | | 9696 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 165878 | | 9697 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1345988 | | 9698 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1783549 | | 9699 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 665358 | | 9700 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 168566 | | 9701 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1531867 | | 9702 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 931161 | | 9703 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 342250 | | 9704 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 437672 | | 9705 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 976963 | | 9706 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 615735 | | 9707 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1152889 | | 9708 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1748237 | | 9709 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 652162 | | 9710 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1067106 | | 9711 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1920992 | | 9712 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1698141 | | 9713 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1649822 | | 9714 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 94358 | | 9715 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 983337 | | 9716 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1091145 | | 9717 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 255341 | | 9718 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 25397 | | 9719 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1223432 | | 9720 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1001712 | | 9721 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1995106 | | 9722 | root | localhost | myapp_production | Query | 2 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 508775 | | 9723 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1121464 | | 9724 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 946613 | | 9725 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1153740 | | 9726 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1656344 | | 9727 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 102982 | | 9728 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1672517 | | 9729 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 350907 | | 9730 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1826733 | | 9731 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 648288 | | 9732 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1690209 | | 9733 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 330725 | 21 | 9734 | root | localhost | myapp_production | Query | 9735 | root | localhost | myapp_production | Query | | 1 | Locked 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1664562 | | UPDATE users SET last_login_date=NOW() WHERE id = 1786465 | | 9736 | root | localhost | myapp_production | Query | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1957560 |Friday, April | root | localhost | myapp_production | Query | 9737 22, 2011 | 1 | Locked | UPDATE users SET last_login_date=NOW() WHERE id = 1738744 |
    • $ uptime 15:00 up 11 days, 16:58, 5 users, load averages: 0.88 0.61 0.44 22Friday, April 22, 2011
    • $ uptime 15:00 up 11 days, 16:58, 5 users, load averages: 0.88 0.61 0.44 mysql> show global status like slow_queries%; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | Slow_queries | 3 | +---------------+-------+ 1 row in set (0.00 sec) 23Friday, April 22, 2011
    • Guessing is misleading ★ Some problems come without “load”. ✦ You can’t accurately look at utilization rates to be able to tell where the problem is. 24Friday, April 22, 2011
    • What do we measure? ★ The response time for every request ✦ To the end user, response time is everything. ✦ Using good logging we can analyze the data effectively. ★ When possible measure the response time of all sub requests ✦ Database ✦ Cache ✦ Expensive operations 25Friday, April 22, 2011
    • Where do we measure it? ★ The Application. ✦ We’re concerned most about whether or not users can complete the tasks they are trying to complete. ★ Looking at metrics one level removed may hide problems: ✦ i.e. 10,000 fast queries for one web page. 26Friday, April 22, 2011
    • Set a Response Goal ★ It will be different depending on what content you are generating: ✦ 50ms is advertising networks? ✦ 300ms-1s an a non-ajax website? ✦ 50ms on AJAX requests? ✦ 1-2s on search results? 27Friday, April 22, 2011
    • Why is response so important? http://www.flickr.com/photos/sundazed/1450388845/sizes/o/ 28Friday, April 22, 2011
    • Why is response so important? Think of a call center. They optimize for throughput (every operator to be busy). What is your experience like? http://www.flickr.com/photos/sundazed/1450388845/sizes/o/ 28Friday, April 22, 2011
    • The suite of definitions that are not response.Friday, April 22, 2011
    • Related Concepts to Response Load Utilization Scalability Throughput Concurrency Capacity 30Friday, April 22, 2011
    • Related Concepts (cont.) Load: how much work is Utilization Scalability incoming? or, how big is the backlog? Throughput Concurrency Capacity 31Friday, April 22, 2011
    • Related Concepts (cont.) Load: Utilization: how much work is how much of a Scalability incoming? or, how systems resources big is the backlog? are used? Throughput Concurrency Capacity 32Friday, April 22, 2011
    • Related Concepts (cont.) Scalability: Load: Utilization: what is the how much work is how much of a relationship incoming? or, how systems resources between utilization big is the backlog? are used? and R? Throughput Concurrency Capacity 33Friday, April 22, 2011
    • Related Concepts (cont.) Scalability: Load: Utilization: what is the how much work is how much of a relationship incoming? or, how systems resources between utilization big is the backlog? are used? and R? Throughput: X - how many Concurrency Capacity tasks can be done per unit of time? 34Friday, April 22, 2011
    • Related Concepts (cont.) Scalability: Load: Utilization: what is the how much work is how much of a relationship incoming? or, how systems resources between utilization big is the backlog? are used? and R? Throughput: Concurrency: X - how many how many tasks Capacity tasks can be done can we do at per unit of time? once? 35Friday, April 22, 2011
    • Related Concepts (cont.) Scalability: Load: Utilization: what is the how much work is how much of a relationship incoming? or, how systems resources between utilization big is the backlog? are used? and R? Capacity: Throughput: Concurrency: X - how many how big can X go how many tasks tasks can be done without making can we do at per unit of time? other things once? unacceptable? 36Friday, April 22, 2011
    • R = Time / Task X = Task / TimeFriday, April 22, 2011
    • Throughput != Performance (If you have complete 100 tasks in 100 seconds, you can not infer each took 1 second due to queuing, parallelism, etc.)Friday, April 22, 2011
    • Practical example of optimization.Friday, April 22, 2011
    • Tools to use Tool Language New Relic PHP, Java, Ruby, .NET http://www.newrelic.com XHProf PHP http://pecl.php.net/package/xhprof Instrumentation for PHP PHP http://code.google.com/p/instrumentation-for-php/ Most other languages have a profiling tool. The difficult is that you want to find one that is designed to work in production. 40Friday, April 22, 2011
    • Boardreader.com example: ★ CREATE TABLE `performance_log_090721` ( `ip` varchar(15) NOT NULL, `server_ip` varchar(25) NOT NULL, `page` varchar(3000) NOT NULL, `utime` float NOT NULL, `stime` float NOT NULL, `wtime` float NOT NULL, `mysql_time` float NOT NULL, `sphinx_time` float NOT NULL, `mysql_count_queries` int(11) NOT NULL, `mysql_queries` text NOT NULL, `sphinx_count_queries` int(11) NOT NULL, `sphinx_real_count_queries` int(11) NOT NULL, `sphinx_queries` text NOT NULL, `logged` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, `user_agent` varchar(255) NOT NULL, `referer` varchar(255) NOT NULL, `bot` enum(,google,yahoo,msn,lycos,other) NOT NULL, `js_cookie` tinyint(1) unsigned NOT NULL default 0, `page_type` enum(, search, ajax, forumprofile,siteprofile, threadprofile, topicprofile, domainprofile,other) NOT NULL, `id` char(32) NOT NULL default ) ENGINE=MyISAM DEFAULT CHARSET=latin1; 41Friday, April 22, 2011
    • Boardreader.com (cont.) ★ *************************** 5. row *************************** ip: 91.148.82.211 server_ip: web08.boardreader.com page: boardreader.com/s/nba29k.html?f=47977&extended_search=1 utime: 0.129981 wtime: 0.242401 mysql_time: 0.004417 sphinx_time: 0.083193 sphinx_results_time: 0.078 mysql_count_queries: 15 mysql_queries: sphinx_count_queries: 3 sphinx_real_count_queries: 3 sphinx_queries: stime: 0.008998 logged: 2009-07-20 20:55:48 user_agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; .NET CLR 2.0.50727; InfoPath.2) referer: http://boardreader.com/fp/FileForums_14910/ PC_Games_CD_2_DVD_Conversion_47977.html bot: js_cookie: 1 page_type: search id: 5ab03bc440ffa0c62610a62db988cb81 42Friday, April 22, 2011
    • Example Query ★ Average time for search queries: select avg(wtime) request, avg(stime+utime)/avg(wtime) cpu ,avg (mysql_time)/avg(wtime) mysql, avg(sphinx_time)/avg(wtime) sphinx, avg(wtime-stime-utime-sphinx_time-mysql_time)/avg(wtime) rest from performance_log_090721 where page_type=search G *************************** 1. row *************************** request: 1.2175869055517 cpu: 0.16983144536072 mysql: 0.1544487152423 sphinx: 0.61537297006254 rest: 0.060346869334443 1 row in set (4.16 sec) 43Friday, April 22, 2011
    • Hourly distribution ★ mysql> select date_format(logged,"%H") h ,round(avg(wtime),3) r, round(avg(stime+utime)/avg (wtime),2) cpup ,round(avg(mysql_time)/avg(wtime),2) mp, round(avg(sphinx_time)/avg(wtime),2) sp, round(avg(wtime-stime-utime-sphinx_time-mysql_time)/avg(wtime),2) rst from performance_log_090721 where page_type=search group by 1; +------+-------+------+------+------+------+ | h | r | cpup | mp | sp | rst | +------+-------+------+------+------+------+ | 00 | 1.816 | 0.11 | 0.14 | 0.70 | 0.05 | | 01 | 1.480 | 0.17 | 0.18 | 0.59 | 0.06 | | 02 | 1.394 | 0.16 | 0.22 | 0.53 | 0.09 | .... | 08 | 1.384 | 0.13 | 0.09 | 0.74 | 0.04 | | 09 | 1.315 | 0.17 | 0.11 | 0.67 | 0.04 | | 10 | 0.950 | 0.20 | 0.15 | 0.60 | 0.05 | | 11 | 0.874 | 0.21 | 0.16 | 0.57 | 0.06 | | 12 | 1.139 | 0.17 | 0.13 | 0.65 | 0.05 | | 13 | 1.191 | 0.16 | 0.14 | 0.65 | 0.05 | | 14 | 1.349 | 0.16 | 0.19 | 0.58 | 0.06 | | 15 | 1.076 | 0.20 | 0.21 | 0.53 | 0.06 | | 16 | 1.526 | 0.14 | 0.14 | 0.58 | 0.13 | | 17 | 0.853 | 0.24 | 0.19 | 0.50 | 0.07 | | 18 | 0.978 | 0.25 | 0.23 | 0.43 | 0.09 | | 19 | 0.924 | 0.23 | 0.17 | 0.54 | 0.06 | | 20 | 1.310 | 0.18 | 0.26 | 0.47 | 0.09 | | 21 | 1.211 | 0.17 | 0.24 | 0.51 | 0.08 | | 22 | 1.538 | 0.14 | 0.19 | 0.59 | 0.08 | | 23 | 1.450 | 0.15 | 0.18 | 0.60 | 0.06 | +------+-------+------+------+------+------+ 24 rows in set (4.33 sec) 44Friday, April 22, 2011
    • Interpreting Results ★ Do you care about the average time it takes? ✦ The maximum time? 45Friday, April 22, 2011
    • Interpreting Results (cont.) ★ If you look at the average or mean, we are looking at a lot of requests - ✦ There could still be a lot of people suffering, and it won’t show the full picture. 46Friday, April 22, 2011
    • Interpreting Results (cont.) ★ If you are looking at many requests, there are always going to be some statistical outliers. ✦ Maximum and minimum are not a good measurement either. 47Friday, April 22, 2011
    • Interpreting Results (cont.) ★ Ideally you care about setting a goal for a percentile of requests. i.e. ✦ You want 95% of requests served within 300ms. ✦ 99% of requests in 1200ms. ★ You want this to happen uniformly across the day. ★ You want this to happen uniformly for all users. http://www.mysqlperformanceblog.com/2010/06/07/performance-optimization- 48 and-six-sigma/Friday, April 22, 2011
    • Fight for better measurement.Friday, April 22, 2011
    • The weather model. ★ You have a 50% chance of guessing the weather tomorrow just by looking at today! ✦ For some people this model is good enough. ✦ For others with more at stake, they want something they can rely on more. 50Friday, April 22, 2011
    • “You have the data” ★ Why would you use a service which sends requests from some arbitrary DC to your webserver to tell you how long page generation takes?* ✦ It’s better to measure it from all the requests you send! ✦ This helps you find any performance holes / variance. * Understanding a “more complete” profile is interesting however. 51 See Boomerang / Philip Tellis at this conference ;)Friday, April 22, 2011
    • Don’t fear measurement ★ You can intelligently sample profiles, but in general don’t fear the cost of gathering data. ✦ If you need to know the cost, it’s -10%. ✦ You will not be able to make optimizations if you don’t have data. 52Friday, April 22, 2011
    • Fear Benchmarks ★ They typically measure throughput. ✦ Omit Response time. ✦ Omit 95 percentile. ✦ Very easy to cheat. 53Friday, April 22, 2011
    • Think when you optimize (1) ★ Is this an optimization for throughput (capacity) or response? ✦ “Rails doesn’t scale” is garbage. ✦ If by scale we mean that X can become bigger, it absolutely can by adding more webservers. ✦ You just need to make sure you don’t add stupid affinity needs in the app layer. • And you economize the database access. 54Friday, April 22, 2011
    • Think when you optimize (2) ★ If response time is bad, maybe you can find a way to cheat. ✦ Background tasks with message queues. ✦ You can extract far more throughput, if you’re willing to let each task (response) take longer. 55Friday, April 22, 2011
    • The End ★ Questions? ★ Photo Credits: ✦ http://www.flickr.com/photos/roboppy/439226022/ ✦ http://www.flickr.com/photos/padorange/2581008342 ✦ http://www.flickr.com/photos/goulao/2109155110/ ✦ http://www.flickr.com/photos/mac-ash/3861215906 56Friday, April 22, 2011