How soa makes_moving_to_the_cloud_easier


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • A real world case study reather than academic 2-3 minute background about myself
  • History will be a few slides of overview about why, when, and how we started to move into the cloud Power of SOA discusses the flexibility and freedom of moving away from single point of data access Complications range from data ownership to data freshness to scalability Example application will cover our Skimmer and Best Sellers applications and their move
  • We first started working with services by exposing our structured data to external developers for our TimesOpen event in 2009.  Very little of that was being used internally. When the community platform moved to the cloud, some services were created so internal pages could talk to the cloud servers We had some very structured venue data for our travel section. In the previous version, if an editor updated a venue, it took up to an hour to make it to the SQLite DB and onto the page.  A change was published to XML. XML was picked up by the loader once an hour and update the SQLite.  We created services that allowed the front-end to call them in real time and then allow the editor to update the MySQL DB in real time via the same services.
  • Definition not as important as culture. It's the concept that data is flexible and self contained. Doesn't rely on specific servers, environments, or configurations  A lot of our applications were being built around how we access the data we use and what was available on what system. Instead, we should be thinking about what do we want the application to actually do and have the data component be an accent to that, not a hinderence SOA and cloud are related, but not mutual required Ability to output results in multiple data formats.  In our case, offering skimmer feeds in json greatly sped up the app because no need to parse RSS or XML
  • SOA means that you can get the data sliced up many different ways. It is not rigid, but allows much flexibility. Because of this, it allows the same services to be used by many different applications One of the items that gets lost a lot when people talk about SOA is that it has to be a two way street. There can be restrictions, but the ability to update the data via the same services is key to keeping these services truly reusable
  • As you can see from the previous slide, using RSS feeds to build applications isn't SOA. But if it starts to allow the applications to not think about where they get their data from, then it's a great start RSS is very rigid, doesn't offer the ability to sort or filter and doesn't allow two way data transfer On our first front-end application, TimesSkimmer, we used RSS to allow a mainly javascript application to free itself from a shared mount point for data. But the new incarnation of Skimmer shows the power of having services versus just RSS.
  • One of the biggest hurdles we ran into was determining data ownership. The core data lived in a database, but was presented and updated via services. So was the group in charge of the interface in charge of cleansing input or the service owner?  If there were new fields added to the data set, how could we decide if it would break the applications currently using it. Versioning allowed us to solve some of this, but then left older applications orphaned. Say application1 is using version2 and new application2 wants 3 new fields. We would create version3 of the services. But then if application1 wants a new field, what version would get it? By asking utilizing applications to be tolerant to new functions and fields, as long as the ones they need stay consistent, then updating a current version won't break other application. When new formatting is truly required, then change version and ask utilizing applications to upgrade. Similar issue to OS updates. By adding keys for each application using the services is useful, it isn't pure security
  • After we got past the core data side of things, we realized that there were other complications that came up. Other related, but not data items existed on the file system.  Some shared navigation modules were reading directly off the shared file system.  Pulling information for related content that existed on the file system like inlines One of the key ones was that the way we dealt with ad calls were purely local. So we had to create a service layer to abstract this. But had to be as responsive as local call
  • Service had existed for a long time, so much of the heavy lifted was done Addition of E-Books meant there needed to be the ability to add new lists to the data set Static articles meant that users couldn't easily navigate between weeks
  • Right now, the applications poll the services for every request, or on some interval, even if there haven't been any changes. Should change it to where there services push out changes when they happen The services are fairly flexible, but still require a good deal of maintenance when new data is needed. Trying to abstract that out to be more tolerant
  • We've opened sourced many of the frameworks that allowed us to make this transition
  • How soa makes_moving_to_the_cloud_easier

    1. 1. How SOA Makes Moving to the Cloud Easier Paul Robbins Sr. Product Manager, @robbinspaul
    2. 2. Agenda <ul><ul><li>Beginnings of moving to the cloud </li></ul></ul><ul><ul><li>How We Started with SOA </li></ul></ul><ul><ul><li>Power of SOA </li></ul></ul><ul><ul><li>Complications of SOA and the Cloud </li></ul></ul><ul><ul><li>Example Applications </li></ul></ul><ul><ul><li>What's Next </li></ul></ul>
    3. 3. Metrics   <ul><ul><li>33 million uniques </li></ul></ul><ul><ul><li>611 million pageviews </li></ul></ul><ul><li> April 2011 (comScore) </li></ul>
    4. 4. Beginnings of on the Cloud <ul><ul><li>Community Infrastructure  </li></ul></ul><ul><ul><ul><li>Fairly new set of standardized user interactions </li></ul></ul></ul><ul><ul><ul><li>Unknown scale and usage </li></ul></ul></ul><ul><ul><li>TimesSkimmer </li></ul></ul><ul><ul><ul><li>Javascript web app </li></ul></ul></ul><ul><ul><ul><li>Prototypical in nature, difficult to position for new servers  </li></ul></ul></ul>
    5. 5. How We Started with SOA <ul><ul><li>Services for external developers (TimesOpen) </li></ul></ul><ul><ul><li>Community services  </li></ul></ul><ul><ul><li>Travel Guides application </li></ul></ul><ul><ul><ul><li>Previous version was reading SQLite on the fly </li></ul></ul></ul><ul><ul><ul><li>If a change was made, took 1 hour to replicate to SQLite </li></ul></ul></ul><ul><ul><ul><li>New version didn't live in the cloud, but built with SOA </li></ul></ul></ul>
    6. 6. Power of SOA <ul><ul><li>Service Oriented Architecture </li></ul></ul><ul><ul><li>SOA is Freedom. SOA is Flexibility. </li></ul></ul><ul><ul><li>Think about what applications should do, not how they get data </li></ul></ul><ul><ul><li>Multiple data formats </li></ul></ul>
    7. 7. SOA Key Concepts <ul><ul><li>Ability to access data your way </li></ul></ul><ul><ul><ul><li>Sorting, filtering, and organizing is abstracted </li></ul></ul></ul><ul><ul><ul><li>Means services can be reused, no need to rewrite data accessors for each new application </li></ul></ul></ul><ul><ul><li>Ability for internal users to update data with no direct access to the database </li></ul></ul><ul><ul><ul><li>This can achieved via a REST interface to the service </li></ul></ul></ul>
    8. 8. RSS ≠ SOA But it's a good start
    9. 9. Complications of SOA <ul><ul><li>Data ownership and maintenance </li></ul></ul><ul><ul><ul><li>Data lives in a DB, but is updated by a service </li></ul></ul></ul><ul><ul><li>Backwards compatibility versus new features and fields </li></ul></ul><ul><ul><ul><li>Versioning solves some issues, but causes new ones </li></ul></ul></ul><ul><ul><ul><li>Best to keep client applications robust to accommodate changes </li></ul></ul></ul><ul><ul><li>Security for updates </li></ul></ul><ul><ul><ul><li>Adding API keys helps, but isn't fail-proof </li></ul></ul></ul><ul><ul><li>Caching </li></ul></ul><ul><ul><ul><li>Clearing data cache and front-end cache on updates </li></ul></ul></ul>
    10. 10. Complications of the Cloud <ul><ul><li>Many complications of mounted file system  </li></ul></ul><ul><ul><ul><li>If we replicated filesystem in the cloud, latency and bandwidth issues </li></ul></ul></ul><ul><ul><li>Some shared modules were hard-coded local </li></ul></ul><ul><ul><ul><li>Initial solution was to pull these modules over http </li></ul></ul></ul><ul><ul><ul><li>Long term solution is to make these modules be accessed via service </li></ul></ul></ul><ul><ul><li>Ad calls were locally initiated </li></ul></ul><ul><ul><ul><li>Had to build a service layer </li></ul></ul></ul>
    11. 12. Example Applications
    12. 13. Skimmer and Chrome App <ul><ul><li>Skimmer [ ] </li></ul></ul><ul><ul><ul><li>Newest version uses SOA and lives in the cloud </li></ul></ul></ul><ul><ul><ul><li>Utilizing JSON formatting and richer data than previous RSS version </li></ul></ul></ul><ul><ul><li>App for Chrome Web Store [ ] </li></ul></ul><ul><ul><ul><li>Similar to Skimmer, optimized for the Chrome browser </li></ul></ul></ul>
    13. 15. Best Sellers <ul><ul><li>One of the best known brands for the Times </li></ul></ul><ul><ul><ul><li>[ ] </li></ul></ul></ul><ul><ul><li>Online version needed refresh, was adding E-Books </li></ul></ul><ul><ul><ul><li>Service was created in 2009 for TimesOpen event </li></ul></ul></ul><ul><ul><ul><li>Front-end was simply a static article, no ability to navigate between weeks </li></ul></ul></ul><ul><ul><li>Redesign meant more than aesthetic, we redesigned architecture and interactions between platforms </li></ul></ul>
    14. 17. Restaurants Search <ul><ul><li>Restaurant listings around NYC </li></ul></ul><ul><ul><ul><li>[ ] </li></ul></ul></ul><ul><ul><li>Not only pulling data from front-end, but heavily curated set of data by producers </li></ul></ul><ul><ul><ul><li>Built an interface into content management system to update via services </li></ul></ul></ul><ul><ul><li>Refine search filters handled purely by passing query parameters to the service </li></ul></ul><ul><ul><ul><li>Lightens the load on the front-end application </li></ul></ul></ul>
    15. 19. Demos!
    16. 20. What's Next <ul><ul><li>Push-based services </li></ul></ul><ul><ul><li>Migrate more applications to SOA </li></ul></ul><ul><ul><li>Expand flexibility of services, still somewhat restrictive </li></ul></ul><ul><ul><li>Complications with full text search via services, need to refine that process </li></ul></ul>
    17. 21. Items We've Open Sourced <ul><ul><li>NYT_Transformer </li></ul></ul><ul><ul><ul><li>Framework for converting structured data to and from different formats, from flat files to DB </li></ul></ul></ul><ul><ul><ul><li>[ ] </li></ul></ul></ul><ul><ul><li>DBSlayer </li></ul></ul><ul><ul><ul><li>Database abstraction layer </li></ul></ul></ul><ul><ul><ul><li>[ ] </li></ul></ul></ul><ul><ul><li>Nimbul  </li></ul></ul><ul><ul><ul><li>Interface for managing Amazon EC2 servers </li></ul></ul></ul><ul><ul><ul><li>[ ] </li></ul></ul></ul><ul><ul><li>Cloudsource  </li></ul></ul><ul><ul><ul><li>Code deployment tool for SVN cloud deploys </li></ul></ul></ul><ul><ul><ul><li>[ ] </li></ul></ul></ul>
    18. 22. Questions?? <ul><ul><li>Also, be sure to check out our developer Open blog </li></ul></ul><ul><ul><ul><li>[ ] </li></ul></ul></ul><ul><ul><li>And our developer network </li></ul></ul><ul><ul><ul><li>[ ] </li></ul></ul></ul><ul><ul><li>Find me on twitter </li></ul></ul><ul><ul><ul><li>[ ] </li></ul></ul></ul>