Porting Flickr to YUI3
- 1. Porting Flickr toYUI 3 Ross Harmes
- 2. Porting Flickr toYUI 3 Lessons in Performance Ross Harmes
- 3. A few monthsago we launched a new photo page…
- 4. We decided tocompletely demolish the old page and rebuild with YUI3
- 5. Using YUI3 wasa delight…
- 6. …but there werea few small problems
- 7.
- 8.
- 9. One of thefirst perf. recommendations is to move all the JS to the bottom of the page
- 10. This means UIelements that need JS to work will fallback to the non-JS case until the scripts load
- 11. Our solution: ActionQueueing
- 12. This isn't makingthe JS load faster
- 13. Getting creative withthe loading indicators helps
- 14. Second problem: longcombo-handler URLs
- 15. We use alot of small JS modules on the page. This made our combo URLs very long:http://l.yimg.com/g/combo.gne?event/event-min.js&j/query-string- args.js.v85201.14&j/flickr_location_search.js.v85793.14&j/flickr_nav.js.v92497.14&base/base-min.js&anim/anim-min.js&dump/dump- min.js&datatype/datatype-xml-min.js&substitute/substitute-min.js&queue-promote/queue-promote-min.js&io/io-min.js&json/json- min.js&j/flickr_api.js.v93039.14&j/history-manager.js.v90829.14&j/photo-data.js.v92868.14&j/context-data.js.v92557.14&j/context- manager.js.v91220.14&j/sprintf.js.v90343.14&j/transjax-base.js.v85036.14&j/focus-tracker.js.v93044.14&event-simulate/event-simulate-min.js&j/photo- button-bar-transjax-en-us.js.v92588.14&j/image-fader.js.v85225.14&j/number-transjax-en-us.js.v90582.14&j/number.js.v87306.14&j/photo-filmstrip- transjax-en-us.js.v90793.14&j/photo-filmstrip.js.v92881.14&event/event-synthetic-min.js&j/event-annotations.js.v91160.14&j/event- mousedrag.js.v90789.14&j/math.js.v87441.14&j/fave-star.js.v91965.14&j/global-dialog-transjax-en-us.js.v85507.14&j/global-dialog- zeus.js.v92830.14&j/keyboard-shortcut-manager.js.v92698.14&node/node-event-simulate-min.js&j/photo- permalink.js.v91170.14&j/yahoo/autocomplete_2.5.1-zeus.js.v92829.14&j/bo-selecta-transjax-en-us.js.v90792.14&j/bo-selecta- zeus.js.v91866.14&cookie/cookie-min.js&j/dejaview-zeus.js.v90642.14&j/photo-people-transjax-en-us.js.v90822.14&j/photo-people- controller.js.v88235.14&j/input-hint.js.v86479.14&j/photo-comments-transjax-en-us.js.v92483.14&j/swfobject.js.v85491.14&j/photo- comments.js.v92853.14&j/photo-keyboard-shortcuts.js.v92892.14&j/box-host.js.v89305.14&j/photo-notes-transjax-en-us.js.v93010.14&j/string- filters.js.v91087.14&j/photo-notes-zeus.js.v93044.14&j/excanvas.js.v39120.14&j/bitmap-text-zeus.js.v87486.14&j/bitmap-type- silkscreen.js.v87486.14&j/photo-sidebar-transjax-en-us.js.v90794.14&stylesheet/stylesheet-min.js&j/photo-sidebar.js.v92813.14&j/photo-context- menu-transjax-en-us.js.v90793.14&j/photo-lightbox-transjax-en-us.js.v92868.14&j/ywa.js.v89879.14&j/photo-ywa- tracking.js.v92723.14&j/occult.js.v90963.14&j/yahoo-ult.js.v92052.14&j/photo-zeus.js.v93054.14&j/photo-people-list.js.v92992.14&j/photo-button- bar.js.v92891.14&j/photo-context-menu.js.v92706.14&j/photo-lightbox.js.v93054.14&j/insitu-transjax-en-us.js.v90792.14&j/insitu- zeus.js.v91793.14&j/photo-insitu.js.v91169.14&j/photo-group-invites-transjax-en-us.js.v90793.14&j/photo-group-invites.js.v91020.14&j/tagrs_zeus- transjax-en-us.js.v93081.14&j/tagrs_zeus.js.v93081.14&j/photo-sidebar-owner-transjax-en-us.js.v91626.14&j/photo-sidebar- owner.js.v92860.14&j/photo-sidebar-admin.js.v92656.14&j/photo-geolocation-transjax-en-us.js.v92191.14&j/photo- geolocation.js.v92894.14&j/personmenu-transjax-en-us.js.v90792.14&j/personmenu-zeus.js.v92796.14&j/share-menu-zeus-transjax-en- us.js.v92581.14&j/share-menu-zeus.js.v92971.14 2,792 characters
- 16. Turns out thata small but vocal minority of users sit behind firewalls that restrict URL length
- 17. The algorithm wesettled on was string substitution:http://l.yimg.com/g/combo.gne?event/event-min.js&j/.H-.K.A.vNKE8&j/.CP-.U-.DE.A.vKEJx&j/.J_.BR_.CA.A.vKYke&j/.J_.DB.A.vPpBR&base/base- min.js&anim/anim-min.js&dump/dump-min.js&datatype/datatype-xml-min.js&substitute/substitute-min.js&queue-promote/queue-promote-min.js&io/io- min.js&json/json-min.js&j/.J_.DS.A.vPFJk&j/.CE-.K.A.vNy2Z&j/.B-.BY.A.vPADv&j/.H-.BY.A.vPrpi&j/.CC.A.vNiA4&j/.C-.BL.A.vPL3k&j/.CV- .CH.A.vPFSX&event-simulate/event-simulate-min.js&j/.B-.T-.CI-.C-.F.A.vPJPD&j/.CW-.CU.A.vKFrV&j/.Y-.C-.F.A.vNqG8&j/.Y.A.vLKiR&j/.B-.M-.C- .F.A.vPKTH&j/.B-.M.A.vPKTH&event/event-synthetic-min.js&j/.G-.BD.A.vNHSF&j/.G-.BO.A.vNwR2&j/.DL.A.vLPjB&j/.CX-.CY.A.vP8NB&j/.X-.W-.C- .F.A.vKPQ8&j/.X-.W-.D.A.vPzvZ&j/.Q-.BX-.K.A.vPvAp&node/node-event-simulate-min.js&j/.B-.BP.A.vNJaV&j/.CM/.BA_2.5.1-.D.A.vPzug&j/bo-.S-.C- .F.A.vNwWc&j/bo-.S-.D.A.vP5RV&cookie/cookie-min.js&j/.BZ-.D.A.vNstz&j/.B-.L-.C-.F.A.vNxPV&j/.B-.L-.BH.A.vMdVz&j/.CN-.DD.A.vLjHZ&j/.B-.O-.C- .F.A.vPpcH&j/.BM.A.vKPmx&j/.B-.O.A.vPHa6&j/.B-.Q-.BQ.A.vPBmT&j/.DR-.DG.A.vMLJp&j/.B-.BE-.C-.F.A.vPHP2&j/.U-.CG.A.vNFGP&j/.B-.BE- .D.A.vPFSX&j/.BV.A.vm3Ux&j/.Z-.DK-.D.A.vLQEc&j/.Z-.DJ-.BJ.A.vLQEc&j/.B-.I-.C-.F.A.vPKTH&stylesheet/stylesheet-min.js&j/.B-.I.A.vPLW4&j/.B-.H- .BB-.C-.F.A.vNwXV&j/.B-.N-.C-.F.A.vPADv&j/.CL.A.vN4N4&j/.B-.CL-.BW.A.vPwkv&j/.CF.A.vNC22&j/.CM-.DO.A.vPboB&j/.B-.D.A.vPGbc&j/.B-.L- .CZ.A.vPJpv&j/.B-.T-.CI.A.vPKDV&j/.B-.H-.BB.A.vPvQc&j/.B-.N.A.vPGbc&j/.B-.DM-.CO-.C-.F.A.vNwXV&j/.B-.DM-.CO.A.vNDHi&j/.BF_.D-.C- .F.A.vPGYK&j/.BF_.D.A.vPGYK&j/.B-.I-.CQ-.BK-.C-.F.A.vNwZD&j/.B-.I-.CQ-.BK.A.vLWQP&j/.B-.I-ad.E.A.vPukZ&j/.B-.R-.C-.F.A.vPfwg&j/.B- .R.A.vPBqk&j/.CB-.C-.F.A.vNwWc&j/.CB-.D.A.vPyvn&j/.DN-.BB-.D-.C-.F.A.vPs7F&j/.DN-.BB-.D.A.vPM5F 1,702 characters (40% smaller)
- 18. This fixes theproblem for almost all users… but Sonicwall turns out to have a limit below 1600 characters
- 19. And then therewas the XXX problem
- 20. Third problem: conveniencemethods aren't free
- 21. We noticed thaton IE8/7, scrolling was very jerky
- 22. The culprit: pollersin Y.delegate() and Y.on()
- 23. We replaced bothwith Y.all().on(), and more customized solutions
- 24. Convenience methods havea cost
- 25. Forth problem: oldmetrics are meaningless
- 26. What do wemeasure?
- 27. We take timingdata from 1% of all users
- 28. rrdtool is fantasticfor displaying timing data
- 29. Browsers behave sodifferently that lumping them in together hides successes
- 30.
- 31.
- 32. If you can'thave real performance, fake it
- 33. Dig deeply intothe JS library you use
- 34. Measure the momentsimportant to you
- 35. And NEVER includeXXX in a URL
- 36. Thank you! ross@flickr.com Slides at:lanyrd.com/spdm Photo credits http://www.flickr.com/photos/markscott/1117392453/ http://www.flickr.com/photos/cybertoad/2102752062/ http://www.flickr.com/photos/whiteoakart/471538245/ http://www.flickr.com/photos/wwarby/3016567069/ http://www.flickr.com/photos/richoz/3791167457/ http://www.flickr.com/photos/lin/372711782/ http://www.flickr.com/photos/cdhc/274211112/ http://www.flickr.com/photos/ennor/353250218/ http://www.flickr.com/photos/jensaar/386863409/ http://www.flickr.com/photos/sterlic/4299631538/http://www.flic kr.com/photos/candyflossblackmarket/1139767634/ http://www.flickr.com/photos/bobcatrock/2653120251/ http://www.flickr.com/photos/jm2c/3677835375/ http://www.flickr.com/photos/kevinkyen/4721020630/ http://www.flickr.com/photos/sindykids/2666402195/ http://www.flickr.com/photos/simonhua/4696240744/ http://www.flickr.com/photos/inkiboo/203350186/ http://www.flickr.com/photos/sahlgoode/5012048467/ http://www.flickr.com/photos/sshb/3981130921/ http://www.flickr.com/photos/blueskin808/1422588776/ http://www.flickr.com/photos/roadsidepictures/1389358202/ http://www.flickr.com/photos/boliston/3958674786/ http://www.flickr.com/photos/httpwwwactionpixsmarukocom/4812 699768/
Editor's Notes
- #3 I'm Ross, a frontend engineer at Flickr.
- #4 This is a story about problems we found after the launch of that photo page, and a few of the unanticipated consequences of building a page from scratch using YUI3 and the Y! Performance Guidelines The primary goal of the project, from the engineering side, was to improve performance: The old page took 4-6 seconds to load The frontend code was 5 years old To put some perspective on it, the JS contained hand-coded special cases for Opera 7 There were tons of global variables, and a dependency tree that was so complicated that we were terrified to pull out pieces of JS It was ready to be put to rest.
- #5 We investigated doing a retrofit with YUI3 We determined that it would take longer to do that than it would to throw everything out and start from scratch We also determined that retrofitting wouldn't really improve performance: The only way we could meet our goal was to throw everything away We started from scratch using YUI3 and the Y! Performance Guidelines And I mean, really started from scratch. We used empty CSS and JS files and built the page again
- #6 Page load times are down significantly (900ms for Safari, 1.5 for Chrome) We essentially cut the load time in half for the worst case (IE), and reduced it by 80% for the best case (Safari) Code is modular and maintainable The architecture will last us for the next 5 years, even if the individual modules that make it up don't Modules and dependencies mean that 5 engineers can work on the page at the same time Development time was extremely fast Existing gallery modules mean you don’t have to reinvent the wheel Convenience methods mean you can do extremely complex things with very little code (more on this later) Good documentation means that even though we weren't initially familiar with YUI3's syntax, we soon picked it up Porting code from YUI2 to 3 is an easy process – analogous functions exist, and it's a simple matter of changing the syntax
- #7 99% of the things we used YUI3 for went off without a hitch But there were a few things that caught us by surprise Some where caused by using YUI3, and some were caused by implementing the Performance Guidelines The guidelines are taken as gospel at this point; everyone know them and tried to implement them We found that, at the edges of normal use, some of the guidelines start to break down I'm going to spend the rest of this talk going over the 4 problems that we found, and how we fixed each one. These all happened after launch, and most of them were reported by users
- #9 Most users were having a great experience… but a vocal minority of users were having huge problems, especially on IE These users were persistent in telling us that the page was significantly slower for them now; we have pages and pages of forum posts outlining problems.
- #10 This makes sense in most cases, since script tags block and slow down the page from being rendered. But if your page depends heavily on JS, it can actually increase perceived loading time
- #11 It means that the bulk of your JS won't be downloaded and executed until a second or two after the rest of the UI Having no-JS fallbacks also makes it worse Instead of a button that does nothing for a short time, it actually takes you off the page The problem is exasperated by using YUI3's deferred module loading
- #12 Include a small amount of JS in the head, enough to attach click events and to show a busy indicator Queue up clicks When a module registers itself, have it check the queue and perform the necessary actions The action queue code handles cleanup of the busy indicators
- #13 It tricks people into thinking the UI is responsive Action queueing doesn't actually solve the problem, it just improves the perceived load time
- #14 Page load times are down significantly (900ms for Safari, 1.5 for Chrome) Code is modular and maintainable The architecture will last us for the next 5 years, even if the individual modules that make it up don't
- #17 These users get pages that sort of work, but don't have all the JS modules It's their problem to fix, but most can't do anything about it We had to fix it on our end
- #18 We replaced the names of modules with single or double character strings. This algorithm wasn't reversible, but it was easy to implement and doesn't require a database to store shortened names. We have to run the script that builds the substitution dictionary every once in a while, but that's easy
- #19 They are a small minority at the moment, but eventually we'll have to deal with this problem. 1600 characters or lower is the limit we found covers pretty much everyone.
- #20 It seems that some corporate firewall refuse to load any URL that has the string 'xxx' in it. We had to implement a check that would use the longer version of the string if xxx was detected
- #22 Not just scrolling, but almost all UI interactions requiring a click or a drag
- #23 Both of these need to poll, with setInterval() in case the elements don't exist yet. We found that we were able to create our own delegate method, using Y.on and Y.Node.test that was twice as fast, when run through a massive selenium test. This isn't to say that ours is better (it isn't), but that Y.delegate() has to handle a lot of cases, and has baggage for most uses cases.
- #24 This probably goes without saying, but customizing the solutions for each situation is obviously going to be faster. What was surprising to me was just how slow delegate and on were before we customized them.
- #25 Don't assume that, just because you're using an established library, you don't need to know what's going on under the hood. We removed all instances of Y.delegate and Y.on, and massively improved the click performance of IE. YMMV.
- #27 DOMReady and window.load are meaningless The important moments for us are: When the photo is visible, and When the JS is done loading and the UI is fully functional
- #28 This gives us enough data to create smooth graphs, without burdening too many page loads with the extra code needed to gather and report the timings.
- #29 RRD tool is a round robin database that stores data at decreasing granularity as time goes on: http://www.mrtg.org/rrdtool/ Track everything, but only actively monitor a few important metrics Also serves as quality assurance, it lets us know when site performance is down
- #30 Performance is excellent for Safari, Chrome and Firefox. IE8 is mediocre and IE7 is terrible. If we were to average all of these numbers together, it would mask the 900ms page load times we're getting with Safari. Lumping them in together also hides browser-specific problems. After a recent deploy, we noticed that the IE8 load times were slightly higher than normal; having separate graphs allowed us to find and fix the problem. The real solution is to have reference systems: a fixed computer, OS, connection, and browser configuration, that we can use to compare the performance of the page over time.
- #31 YUI3 is amazing If you can't have real performance, fake it Dig deeply into the JS library you use Measure the moments important to you NEVER include XXX in a URL
- #32 YUI3 is amazing If you can't have real performance, fake it Dig deeply into the JS library you use Measure the moments important to you NEVER include XXX in a URL
- #33 YUI3 is amazing If you can't have real performance, fake it Dig deeply into the JS library you use Measure the moments important to you NEVER include XXX in a URL
- #34 YUI3 is amazing If you can't have real performance, fake it Dig deeply into the JS library you use Measure the moments important to you NEVER include XXX in a URL
- #35 YUI3 is amazing If you can't have real performance, fake it Dig deeply into the JS library you use Measure the moments important to you NEVER include XXX in a URL
- #36 YUI3 is amazing If you can't have real performance, fake it Dig deeply into the JS library you use Measure the moments important to you NEVER include XXX in a URL