Successfully reported this slideshow.

Evolving the nuget.org Architecture

2

Share

Loading in …3
×
1 of 46
1 of 46

Evolving the nuget.org Architecture

2

Share

From Oredev 2014

http://oredev.org/2014/sessions/evolving-the-nuget-org-architecture

After 3 years of usage growth though, NuGet.org needs to be more reliable, scalable, and maintainable, with the goal of 99.999% availability of package downloads. To achieve this, we’re evolving the architecture away from a monolithic website/service into several independent services, connected purely through HTTP. Come learn how we’re starting to use Command Query Responsibility Segregation (CQRS), JSON-LD/Linked Data, and event sourcing.

From Oredev 2014

http://oredev.org/2014/sessions/evolving-the-nuget-org-architecture

After 3 years of usage growth though, NuGet.org needs to be more reliable, scalable, and maintainable, with the goal of 99.999% availability of package downloads. To achieve this, we’re evolving the architecture away from a monolithic website/service into several independent services, connected purely through HTTP. Come learn how we’re starting to use Command Query Responsibility Segregation (CQRS), JSON-LD/Linked Data, and event sourcing.

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Evolving the nuget.org Architecture

  1. 1. Evolving the nuget.org Architecture Jeff Handley @JeffHandley
  2. 2. 275 000 000 Package downloads
  3. 3. 265 000 Total packages
  4. 4. 28 000 Unique package IDs
  5. 5. 7 500 000 Requests per day
  6. 6. 1 500 000 Searches per day
  7. 7. 700 000 Package downloads per day
  8. 8. 550 Package uploads per day
  9. 9. 60 User registrations per day
  10. 10. 85 Requests per second
  11. 11. Uptime (October 2013 – September 2014) 99.86% 99.88% 99.91% 100.00% 99.95% 99.90% 99.85% 99.80% package installation and restore feed / search registration, login, and package uploads
  12. 12. http://lapulpadigital.files.wordpress.com/2013/12/simo n-cowell-people-sound-bad-when-recorded.jpg
  13. 13. Line of Business application Using familiar technology
  14. 14. API v2 architecture SQL Azure database Azure Blob Storage Gallery / API v2 Azure web role (3 instances) EntityFramework code-first WCF Data Services and ASP.NET MVC and Razor OData Search and feed requests Website requests Content Delivery Network (CDN) Package downloads
  15. 15. http://www.thecoralgablesstory.com/2010/08/17/sellers-regret- is-the-new-buyers-remorse-in-real-estate/
  16. 16. Basing architecture on usage Instead of familiar technology
  17. 17. all other reads 71% user registrations 0.001% package downloads 9% package uploads 0.01% searches 20%
  18. 18. Avoiding failures • Writes should not impact read or query performance • Command-Query Responsibility Segregation • Append-only system for incremental processing • Reads should not require server compute • Presentation models (views) are regenerated (and persisted) asynchronously • The materialized views can then be treated as static documents on the CDN • Queries should not require external resources • Lucene search index can be held entirely in memory
  19. 19. Catalog and materialized views Resolver metadata Another catalog HTML Web site catalog LUCENE Search
  20. 20. Building the catalog page0.json A B C page1.json D E F page2.json G H I index.json
  21. 21. Reading the catalog index.json page0.json A B C page1.json D E F page2.json G H I
  22. 22. Commit timestamps 1 | 1 | 2 1 1 2 3 | 3 | 3 3 3 3 4 | 5 | 5 4 5 5 2 | 3 | 5
  23. 23. Incremental updates 1 | 1 | 2 1 1 2 3 | 3 | 3 3 3 3 4 | 5 | 5 4 5 5 2 | 3 | 5 Cursor: 45
  24. 24. Materializing views as blobs Read new catalog entries Apply view Identify blobs to be written Read any existing blobs Merge data using RDF Write the new blob
  25. 25. http://jerryfahrni.com/2009/12/biometric-identification-and-facial-recognition/
  26. 26. Database Server http://jerryfahrni.com/2009/12/biometric-identification-and-facial-recognition/
  27. 27. What we planned •Add API v3 support to nuget.org • Update the NuGet client to use API v3 • Factor the server into NuGet packages • Factor the client into NuGet packages
  28. 28. Design resource formats • Search • Dependency resolution •Website rendering • Other package metadata reads • Gallery mirroring • Custom metadata consumption scenarios
  29. 29. Representing a graph in JSON • Collection+JSON and Collection.Next+JSON • Very similar to ATOM • Includes query templates • JSON-API and most others • Emphasis on RESTful CRUD • Client specification of data shape in some • We didn’t want to expose our raw data model • Scenario-specific views instead
  30. 30. http://www.flickr.com/photos/foxypar4/1004464889/
  31. 31. JSON-LD HTTP://JSON-LD.ORG/ { "@context": "http://json-ld.org/contexts/person.jsonld", "@id": "http://dbpedia.org/resource/John_Lennon", "name": "John Lennon", "born": "1940-10-09", "spouse": "http://dbpedia.org/resource/Cynthia_Lennon" }
  32. 32. Linked Data • @context allows the JSON document to be treated as an RDF data-set • RDF = Resource Definition Framework • Fully-qualified names and relationships (URI triples) • RDF data-sets benefits • Easily merged and queried • Idempotent • Everything is namespaced • Based on W3C standards • RDF 1.0: 1999 • RDF 1.1: February 2014 - http://www.w3.org/RDF/ • JSON-LD: January 2014 - http://www.w3.org/TR/json-ld/ (W3C Recommendation)
  33. 33. Azure Blob Storage API v3 search (feed) Search Service Azure worker role (3+ instances) Lucene.NET Full index loaded into memory Backend jobs Azure VM (1 instance) SQL Azure warehouse Gallery / API v2 Azure web role (3+ instances) Content Delivery Network (CDN) Package Downloads SQL Azure database Website requests API v3 Package Metadata Metrics Service Azure website (3+ instances)
  34. 34. API v3 clients • Get service index from CDN • Links are followed to resources: • Search service • Dependency resolver views • Other metadata views • Packages • Metrics service
  35. 35. http://preview.nuget.org/ver3-preview/index.json Humanizer RavenDB.Client
  36. 36. http://readingafterbedtime.wordpress.com/2012/11/01/october-2012-in-review/
  37. 37. http://freethoughtblogs.com/biodork/2012/03/19/cross-country-connections-tricky/
  38. 38. API v2 client behavior locked • Lots of existing clients in the wild • All requests are made through a single Gallery pipeline • Download requests • Record statistics to database • Redirect to the CDN to get the nupkg
  39. 39. Azure Blob Storage API v3 search (feed) Search Service Azure worker role (3+ instances) Lucene.NET Full Index Loaded into Memory Backend jobs Azure VM (1 instance) SQL Azure warehouse Gallery / API v2 Azure web role (3+ instances) Content Delivery Network (CDN) Package Downloads SQL Azure database Website requests API v3 Package Metadata Metrics Service Azure website (3+ instances) API v2 OData
  40. 40. Search service integration Gallery receives OData request Search queries identified and stopped API v3 request to search service JSON-LD response Transformed into OData XML OData XML returned
  41. 41. Metrics service integration Gallery receives download request Call to database is removed API v3 request to metrics service JSON-LD POST Fire and forget Redirect to package on CDN
  42. 42. Iterative success • Usage-based priorities for availability • JSON-LD as our API v3 resource format • Scenario-focused API v3 subsystems • API v2 gallery became the first API v3 client • API v3 responses are transformed into API v2 responses • Getting API v3 benefits for existing API v2 users
  43. 43. NUGET 3.0 – TRANSITIONING FROM ODATA TO JSON-LD Friday at 14:20 in KITT
  44. 44. Thanks Come get NuGet stickers  Gallery and Backend Services http://github.com/NuGet Catalog and Collectors http://github.com/NuGet/NuGet.Services.Metadata JSON-LD Processor http://github.com/NuGet/json-ld.net http://www.nuget.org/packages/json-ld.net @jeffhandley | jeffhandley.com | jeff.handley@microsoft.com | blog.nuget.org

Editor's Notes

  • Visual Studio 2012 launch day
    3GB memory grant inside SQL
  • We thought JSON would save us!
  • ×