0
Building configurable applications for the web Tom Melendez, Yahoo!
Building configurable applications for the web <ul><li>About Me </li></ul><ul><li>Using PHP since 1998 (3.0.5) </li></ul><...
Building configurable applications for the web <ul><li>What does “configurable” mean? </li></ul><ul><li>What are we really...
Building configurable applications for the web <ul><li>Intended audience </li></ul><ul><li>Developers   (who don’t want to...
Building configurable applications for the web <ul><li>Anyone who doesn’t want to get paged in the middle of the night. </...
Building configurable applications for the web <ul><li>We’re talking about application and system configuration </li></ul>...
Building configurable applications for the web <ul><li>Why highly configurable systems? </li></ul><ul><li>The Need for: </...
Building configurable applications for the web <ul><li>Why now? </li></ul><ul><li>A bug in production costs 200 times more...
Building configurable applications for the web <ul><li>I’m a one-man band: do I really need all this stuff? </li></ul><ul>...
Building configurable applications for the web <ul><li>YES. (Besides, the alternative is that we sit here in silence.) </l...
Building configurable applications for the web <ul><li>These are goals, not implementation requirements! </li></ul><ul><li...
Building configurable applications for the web <ul><li>Our ultimate goal is allow the system to be as configurable as poss...
Building configurable applications for the web <ul><li>I do have some requirements though. </li></ul><ul><li>photo courtes...
Building configurable applications for the web <ul><li>Automated deployment: Host, Application and Configuration </li></ul...
Building configurable applications for the web <ul><li>Other nice to haves would be: </li></ul><ul><li>Automated Builds </...
Building configurable applications for the web <ul><li>So today, things that we’ll touch on: </li></ul><ul><li>The Cloud  ...
Building configurable applications for the web <ul><li>Now lets make our app configurable.  </li></ul><ul><li>Question: Wh...
Building configurable applications for the web <ul><li>Seriously. </li></ul><ul><li>Look at your app and  prioritize  how ...
Building configurable applications for the web <ul><li>Lets look at some examples: </li></ul><ul><li>Twitter </li></ul><ul...
Building configurable applications for the web <ul><li>I would argue that the previous exercise will probably make you com...
Building configurable applications for the web <ul><li>Now that we have our priorities for app features, we begin to make ...
Building configurable applications for the web <ul><li>You will also use the priority list you created as the basis for yo...
Building configurable applications for the web <ul><li>What about a given feature should be configurable?  It depends, but...
Building configurable applications for the web <ul><li>Is there a strategy to figure this out?  Well, ask yourself: </li><...
Building configurable applications for the web <ul><li>Further strategy (and a little detour).  Please write debuggable co...
Building configurable applications for the web <ul><li>Writing debuggable code </li></ul><ul><li>We need extended informat...
Building configurable applications for the web <ul><li>It is 3am .  </li></ul><ul><li>I don’t know why something isn’t wor...
Building configurable applications for the web <ul><li>Now that your code is debuggable, strive for Runbooks </li></ul><ul...
Building configurable applications for the web <ul><li>Writing defensive code </li></ul><ul><li>Expected objects need to b...
Building configurable applications for the web <ul><li>Degrading Service </li></ul><ul><li>A mechanism to request data wit...
Building configurable applications for the web <ul><li>Degrading Service </li></ul><ul><li>Assuming that you shouldn’t fai...
Building configurable applications for the web <ul><li>Managing your cache </li></ul><ul><li>File cache, local cache, dist...
Building configurable applications for the web <ul><li>Storing and reading your configuration </li></ul><ul><li>Single PHP...
Building configurable applications for the web <ul><li>Configuration via Proxy: Introducing Apache Traffic Server </li></u...
Building configurable applications for the web <ul><li>Configuration via Proxy: configured with YAML </li></ul>Tom Melende...
Building configurable applications for the web <ul><li>Wishlist: If I could have it my way </li></ul>Tom Melendez, Yahoo! ...
Building configurable applications for the web <ul><li>Summary </li></ul><ul><li>Give others the options, tools and info n...
Building configurable applications for the web <ul><li>Questions? </li></ul><ul><li>[email_address] </li></ul>Tom Melendez...
Upcoming SlideShare
Loading in...5
×

Building configurable applications for the web

3,466

Published on

Today's web application needs to be many things: fast, intuitive, always available. In addition, today's web app needs to be something else: highly configurable. In this session, we'll discuss strategies in building your application so that it can be tuned in real time, minimizing downtime. Also discussed is configuration management, a necessity in deploying apps to multiple environments.

Published in: Technology, Business
2 Comments
2 Likes
Statistics
Notes
No Downloads
Views
Total Views
3,466
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
50
Comments
2
Likes
2
Embeds 0
No embeds

No notes for slide
  • As infrastructure, I’m responsible for building backend components as well as telling everyone else why their components don’t work. We’ll talk more about that during the presentation
  • It means a lot to me. I’m using the term configurable “loosely”
  • No big ideologies or frameworks here. If you want to go on vacation and/or work on cool stuff, give others the power to control your app so they don’t need to come to you.
  • Unique states used for debugging or bucket testing for example. We’ll definitely be sloped toward site availability in this talk but these are all important
  • Or maybe you’re a small company or startup or a manager who has engineers that are overworked and deadlines that are too tight.
  • Build info into your app and your process to let others manage it effectively. This is not a selfish endeavor; we all want to spend more time doing the fun things in programming like coding and design. Not triage.
  • You won’t get the bang for the buck if you don’t have this. You want have the power to quickly and reliably deploy fixes, configuration changes and even roll back. In the same vain as what Sebastian said yesterday, “You want to make deployments a non-event”. This is hard, as in order for them to truly be routine, you need extensive automated testing and such in place. But, the first thing you need is the mechanism to actually build the host.
  • We want to have a baseline as to how much you can handle. something like siege, ab, http_load
  • Twitter – don’t forget the API
  • Getting this on the table early is important. You’re doing this with wireframes, right? Once everyone knows at what points these things happen, discussions can begin as to what to do at those points and how to mitigate them
  • High value low cost: if something is a priority 3 but really expensive resource-wise to produce you might want to take that early on, especially if you are expecting increased traffic
  • Go over Defcon levels. Defcon levels also have business procedures as well (Notify XYZ of situation, etc.). Your priority list serves as the basis as to what levels and knobs to engage. Example Level 4 for News: Emmy awards, MJ tribute. Level 3 is where we plan on degradation of service based on expections, Level 2 is where we implement degradation of service
  • Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
  • Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
  • Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
  • Lets also talk about defensive code a bit
  • Lets also talk about defensive code a bit
  • Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
  • Flesh this out, talk about a db or web service that isn’t there. Yes, you can fetch it from the cache but you shouldn’t try to connect to it if it likely won’t be there
  • Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
  • Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
  • Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
  • Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
  • Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
  • Application defaults, you shouldn’t hardcore ‘5’ in the SQL query to determine how many blog posts to fetch. Display settings you want those sorted by date, or maybe even what fields are displayed?
  • As infrastructure, I’m responsible for building backend components as well as telling everyone else why their components don’t work. We’ll talk more about that during the presentation
  • Transcript of "Building configurable applications for the web"

    1. 1. Building configurable applications for the web Tom Melendez, Yahoo!
    2. 2. Building configurable applications for the web <ul><li>About Me </li></ul><ul><li>Using PHP since 1998 (3.0.5) </li></ul><ul><li>Lead Infrastructure Engineer, Yahoo! News </li></ul><ul><li>Native New Yorker (so, I’m skeptical of everything) </li></ul>Tom Melendez, Yahoo!
    3. 3. Building configurable applications for the web <ul><li>What does “configurable” mean? </li></ul><ul><li>What are we really talking about? </li></ul><ul><li>Am I in the right place? </li></ul>Tom Melendez, Yahoo!
    4. 4. Building configurable applications for the web <ul><li>Intended audience </li></ul><ul><li>Developers (who don’t want to annoy their Operations folks) </li></ul><ul><li>Ops folks (who don’t want to constantly harass their Developers) </li></ul><ul><li>QA folks (who want to make sure Dev and Ops have their sh*t together) </li></ul><ul><li>Managers and Execs (who want to tell their bosses, “Yeah, we can do that”) </li></ul><ul><li>and… </li></ul>Tom Melendez, Yahoo!
    5. 5. Building configurable applications for the web <ul><li>Anyone who doesn’t want to get paged in the middle of the night. </li></ul>Tom Melendez, Yahoo!
    6. 6. Building configurable applications for the web <ul><li>We’re talking about application and system configuration </li></ul><ul><li>What's on the box? </li></ul><ul><li>What state is the box in? </li></ul><ul><li>What is it allowed to do? </li></ul><ul><li>Not personalized user configuration of your web app. </li></ul>Tom Melendez, Yahoo!
    7. 7. Building configurable applications for the web <ul><li>Why highly configurable systems? </li></ul><ul><li>The Need for: </li></ul><ul><li>Multiple Deployment Platforms </li></ul><ul><li>Multiple Experiences </li></ul><ul><li>Different Architectures </li></ul><ul><li>Multiple installations for different purposes </li></ul><ul><li>Different data and/or presentation of that data </li></ul><ul><li>Unique states of some instances of the application </li></ul><ul><li>Real-time tuning </li></ul><ul><li>Backwards compatibility; Deployment and Rollback Situations </li></ul>Tom Melendez, Yahoo!
    8. 8. Building configurable applications for the web <ul><li>Why now? </li></ul><ul><li>A bug in production costs 200 times more to fix than what it would have if caught prior. * </li></ul><ul><li>Something similar could be said about adding in more flexibility and configuration, because really, it’s a bug. </li></ul><ul><li>Plus, when your app gets Oprahified or Slashdotted you’ll have plenty of levers to pull and knobs to turn. </li></ul><ul><li>* Sebastian and Stefan's talk on PHP and CI on Monday at IPC 2009 </li></ul>Tom Melendez, Yahoo!
    9. 9. Building configurable applications for the web <ul><li>I’m a one-man band: do I really need all this stuff? </li></ul><ul><li>photo courtesy of origamidon@ on Flickr </li></ul>Tom Melendez, Yahoo!
    10. 10. Building configurable applications for the web <ul><li>YES. (Besides, the alternative is that we sit here in silence.) </li></ul>Tom Melendez, Yahoo!
    11. 11. Building configurable applications for the web <ul><li>These are goals, not implementation requirements! </li></ul><ul><li>The Need for: </li></ul><ul><li>Multiple Deployment Platforms </li></ul><ul><li>Multiple Experiences </li></ul><ul><li>Different Architectures </li></ul><ul><li>Multiple installations for different purposes </li></ul><ul><li>Different data and/or presentation of that data </li></ul><ul><li>Unique states of some instances of the application </li></ul><ul><li>Real-time tuning </li></ul><ul><li>Backwards compatibility/Deployment and Rollback Procedures </li></ul>Tom Melendez, Yahoo!
    12. 12. Building configurable applications for the web <ul><li>Our ultimate goal is allow the system to be as configurable as possible without changing code and without doing too much work. </li></ul>Tom Melendez, Yahoo!
    13. 13. Building configurable applications for the web <ul><li>I do have some requirements though. </li></ul><ul><li>photo courtesy of duncan@ on Flickr </li></ul>Tom Melendez, Yahoo!
    14. 14. Building configurable applications for the web <ul><li>Automated deployment: Host, Application and Configuration </li></ul><ul><li>All of this needs to be in SCM </li></ul><ul><li>Going forward without the above is bad </li></ul>Tom Melendez, Yahoo!
    15. 15. Building configurable applications for the web <ul><li>Other nice to haves would be: </li></ul><ul><li>Automated Builds </li></ul><ul><li>Monitoring </li></ul><ul><li>Profiling </li></ul>Tom Melendez, Yahoo! Despite the logos above, even a poor man’s shell script will do. We want to build our software without effort, know that the app is up, and make sure performance didn’t drop dramatically.
    16. 16. Building configurable applications for the web <ul><li>So today, things that we’ll touch on: </li></ul><ul><li>The Cloud (well, not really, but you’ll be cloud-deployable) </li></ul><ul><li>CI </li></ul><ul><li>Caching </li></ul><ul><li>Performance </li></ul><ul><li>Proxies </li></ul><ul><li>Defensive coding/Debuggable code </li></ul>Tom Melendez, Yahoo!
    17. 17. Building configurable applications for the web <ul><li>Now lets make our app configurable. </li></ul><ul><li>Question: What should be made configurable? </li></ul><ul><li>Answer: Everything. </li></ul>Tom Melendez, Yahoo!
    18. 18. Building configurable applications for the web <ul><li>Seriously. </li></ul><ul><li>Look at your app and prioritize how important it is for the business to have that feature/widget right where it is showing the data it is currently showing. </li></ul><ul><li>Examples that come to mind: </li></ul>Tom Melendez, Yahoo! <ul><li>Form Fields </li></ul><ul><li>Payment screens </li></ul><ul><li>AJAX updates </li></ul><ul><li>Account Information </li></ul><ul><li>Summary Data </li></ul><ul><li>Data about your contacts </li></ul><ul><li>Previous transactions </li></ul><ul><li>Recommendations </li></ul><ul><li>Photos/Multimedia </li></ul><ul><li>Ads </li></ul>
    19. 19. Building configurable applications for the web <ul><li>Lets look at some examples: </li></ul><ul><li>Twitter </li></ul><ul><li>Facebook </li></ul><ul><li>Y! News </li></ul>Tom Melendez, Yahoo!
    20. 20. Building configurable applications for the web <ul><li>I would argue that the previous exercise will probably make you come up with a better solution. </li></ul><ul><li>It also sets expectations with your stakeholders as to what points you would degrade your service and at which points you would fail. </li></ul>Tom Melendez, Yahoo!
    21. 21. Building configurable applications for the web <ul><li>Now that we have our priorities for app features, we begin to make them configurable. </li></ul><ul><li>Take the lowest priority items first </li></ul><ul><li>If increased load occurs, it is a no-brainer to turn these off </li></ul><ul><li>Since they are low priority, they are likely to be changed for something else that could be a higher priority </li></ul><ul><li>You’ll want to apply some high value/low cost logic to the above. </li></ul>Tom Melendez, Yahoo!
    22. 22. Building configurable applications for the web <ul><li>You will also use the priority list you created as the basis for your DEFCON levels. Example: </li></ul><ul><li>Level 5 – normal operation </li></ul><ul><li>Level 4 – planned high traffic event </li></ul><ul><li>Level 3 – unplanned high traffic </li></ul><ul><li>Level 2 – component failure </li></ul><ul><li>Level 1 – site failure </li></ul>Tom Melendez, Yahoo! http://en.wikipedia.org/wiki/Defcon
    23. 23. Building configurable applications for the web <ul><li>What about a given feature should be configurable? It depends, but generally: </li></ul><ul><li>Hostnames/Ports </li></ul><ul><li>Database names </li></ul><ul><li>Usernames/Passwords </li></ul><ul><li>Cache times </li></ul><ul><li>Base file locations </li></ul><ul><li>Timeouts </li></ul><ul><li>Application defaults/Flags </li></ul><ul><li>Display settings </li></ul><ul><li>Minimums and Maximums </li></ul>Tom Melendez, Yahoo!
    24. 24. Building configurable applications for the web <ul><li>Is there a strategy to figure this out? Well, ask yourself: </li></ul><ul><li>Could it ever change, and be changed, without affecting anything else? (u/p, db names) </li></ul><ul><li>Would someone else ever need to change it? (hostnames, different platforms, diff envs) </li></ul><ul><li>Do I really want to get out of bed, edit some code, build and deploy just for this change? </li></ul>Tom Melendez, Yahoo!
    25. 25. Building configurable applications for the web <ul><li>Further strategy (and a little detour). Please write debuggable code </li></ul><ul><li>Or else </li></ul>Tom Melendez, Yahoo!
    26. 26. Building configurable applications for the web <ul><li>Writing debuggable code </li></ul><ul><li>We need extended information available to us for troubleshooting/diagnostics </li></ul><ul><li>Any service calls need terse logging available to show: </li></ul><ul><ul><li>the request made to me </li></ul></ul><ul><ul><li>The URL that I requested </li></ul></ul><ul><ul><li>What parameters/headers/cookies I sent </li></ul></ul><ul><ul><li>the HTTP code I got back </li></ul></ul><ul><ul><li>The response I got back </li></ul></ul><ul><li>Ideally need to do this while box is in use, but definitely without changing code </li></ul>Tom Melendez, Yahoo!
    27. 27. Building configurable applications for the web <ul><li>It is 3am . </li></ul><ul><li>I don’t know why something isn’t working. </li></ul><ul><li>I don’t want to spend an hour trying to figure it out. </li></ul><ul><li>I should be able to flip a switch and get this information in the log. </li></ul><ul><li>Better yet, someone else should be able to do it for me. </li></ul><ul><li> </li></ul>Tom Melendez, Yahoo!
    28. 28. Building configurable applications for the web <ul><li>Now that your code is debuggable, strive for Runbooks </li></ul><ul><li>Step by step decision tree on how to handle a situation and how to escalate if necessary </li></ul><ul><li>When you’ve reached this stage you now have given someone else control over the maintenance of your application. </li></ul><ul><li>Congrats! </li></ul><ul><li>http://en.wikipedia.org/wiki/Runbook </li></ul>Tom Melendez, Yahoo!
    29. 29. Building configurable applications for the web <ul><li>Writing defensive code </li></ul><ul><li>Expected objects need to be validated </li></ul><ul><ul><li>is_object($foo) is not enough </li></ul></ul><ul><ul><li>instanceof FooClass may not be enough </li></ul></ul><ul><ul><li>You should ensure that what you have is a valid object that you can use </li></ul></ul>Tom Melendez, Yahoo! if (is_object($xml_resp) ) { //site blows up $xml_resp->xpath(“/something/that/does_not/exist”); } if (is_object($xml_resp) && $xml_resp instanceof SimpleXMLElement) { //better, but you still don’t know if it is valid as you see it. Could be an error message in XML format $xml_resp->xpath(“/something/that/does_not/exist”); }
    30. 30. Building configurable applications for the web <ul><li>Degrading Service </li></ul><ul><li>A mechanism to request data without killing your connections </li></ul><ul><li>Your code depends on something which might not be available </li></ul><ul><li>Or, you’re just a smart cookie and know that at any time, anything might not be available </li></ul><ul><li>Can you still serve traffic? Are you eating up connections by connecting to something that probably won’t be there? </li></ul><ul><li>Should you be serving traffic? </li></ul><ul><li>Review your priority list as to when you should fail </li></ul>Tom Melendez, Yahoo!
    31. 31. Building configurable applications for the web <ul><li>Degrading Service </li></ul><ul><li>Assuming that you shouldn’t fail, the service should be degraded </li></ul><ul><li>At some point, you should stop requesting that service, you’re just eating up connections. </li></ul><ul><li>How it works </li></ul><ul><ul><li>How many times should I try to fetch and have it fail? </li></ul></ul><ul><ul><li>How long should I wait until I try again to fetch? </li></ul></ul>Tom Melendez, Yahoo!
    32. 32. Building configurable applications for the web <ul><li>Managing your cache </li></ul><ul><li>File cache, local cache, distributed cache, proxy cache, CDN </li></ul><ul><li>Do you have/use tools to manage your caches (CRUD operations) ? </li></ul><ul><ul><li>Can’t tell you how many issues I’ve seen to bad caching, whether it be expiration or poision </li></ul></ul><ul><ul><li>All Devs and Ops need to know how to use these tools (Runbooks, remember?) </li></ul></ul>Tom Melendez, Yahoo!
    33. 33. Building configurable applications for the web <ul><li>Storing and reading your configuration </li></ul><ul><li>Single PHP file, auto prepended? </li></ul><ul><li>Apache SetEnv (requires restart)? </li></ul><ul><li>Database? </li></ul><ul><li>APC? </li></ul><ul><ul><li>Beware of apc cache slam </li></ul></ul>Tom Melendez, Yahoo!
    34. 34. Building configurable applications for the web <ul><li>Configuration via Proxy: Introducing Apache Traffic Server </li></ul><ul><li>“ Traffic Server is a high-performance web proxy cache that improves network efficiency and performance by caching frequently-accessed information at the edge of the network.” </li></ul><ul><li>http://incubator.apache.org/trafficserver/docs/admin/ </li></ul><ul><li>How do we know which INTL to show? We use an Apache TS remap rule to proxy the user to the front end passing in the INTL in the URL </li></ul><ul><li>Super-high performing: Unofficial “test” had it handling 90k rps (!) on 12 boxes in 3 colos </li></ul><ul><li>Enabling caching on TS is our DEFCON 1. We’ve never needed that. </li></ul><ul><li>Allows for other cool things like bucket testing and URL rewriting. </li></ul>Tom Melendez, Yahoo!
    35. 35. Building configurable applications for the web <ul><li>Configuration via Proxy: configured with YAML </li></ul>Tom Melendez, Yahoo! vars: story_fe_ep: <internal dns name> templates: news.template paths: ... /s/: endpoint: %{story_fe_ep}/news/story/v3/en-US/s ... hosts: news.yahoo.com template: news.template
    36. 36. Building configurable applications for the web <ul><li>Wishlist: If I could have it my way </li></ul>Tom Melendez, Yahoo! <ul><li>Dev writes tests and code, commits </li></ul><ul><li>Commit build runs on build server, passes OK </li></ul><ul><li>Packages are deployed to Integration periodically and BAT and Smoke Tests are run </li></ul><ul><li>Packages are deployed to a QA env, Selenium and regression tests are run </li></ul><ul><li>In parallel, packages are deployed to another environment and perf tests are run </li></ul><ul><li>All looks good, we automatically deploy to a bucket </li></ul><ul><li>See good performance, high clicks, user satisfaction in the metrics, we initiate the process to go live. </li></ul>
    37. 37. Building configurable applications for the web <ul><li>Summary </li></ul><ul><li>Give others the options, tools and info necessary to administer the app </li></ul><ul><ul><li>An ounce of prevention is worth a pound of cure </li></ul></ul><ul><li>Don’t get paged! </li></ul>Tom Melendez, Yahoo! photo courtesy of red5standingby@ on Flickr
    38. 38. Building configurable applications for the web <ul><li>Questions? </li></ul><ul><li>[email_address] </li></ul>Tom Melendez, Yahoo!
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×