Uploaded byJeff Handley
Evolving the nuget.org Architecture
From Oredev 2014 http://oredev.org/2014/sessions/evolving-the-nuget-org-architecture After 3 years of usage growth though, NuGet.org needs to be more reliable, scalable, and maintainable, with the goal of 99.999% availability of package downloads. To achieve this, we’re evolving the architecture away from a monolithic website/service into several independent services, connected purely through HTTP. Come learn how we’re starting to use Command Query Responsibility Segregation (CQRS), JSON-LD/Linked Data, and event sourcing.
Evolving the nuget.org Architecture
- 1. Evolving the nuget.org Architecture Jeff Handley @JeffHandley
- 4. 275 000 000 Package downloads
- 5. 265 000 Totalpackages
- 6. 28 000 Uniquepackage IDs
- 7. 7 500 000 Requests per day
- 8. 1 500 000 Searches per day
- 9. 700 000 Packagedownloads per day
- 10. 550 Package uploadsper day
- 11. 60 User registrationsper day
- 12. 85 Requests persecond
- 13. Uptime (October 2013– September 2014) 99.86% 99.88% 99.91% 100.00% 99.95% 99.90% 99.85% 99.80% package installation and restore feed / search registration, login, and package uploads
- 14.
- 15. Line of Businessapplication Using familiar technology
- 16. API v2 architecture SQL Azure database Azure Blob Storage Gallery / API v2 Azure web role (3 instances) EntityFramework code-first WCF Data Services and ASP.NET MVC and Razor OData Search and feed requests Website requests Content Delivery Network (CDN) Package downloads
- 17.
- 18. Basing architecture onusage Instead of familiar technology
- 19. all other reads 71% user registrations 0.001% package downloads 9% package uploads 0.01% searches 20%
- 20. Avoiding failures •Writes should not impact read or query performance • Command-Query Responsibility Segregation • Append-only system for incremental processing • Reads should not require server compute • Presentation models (views) are regenerated (and persisted) asynchronously • The materialized views can then be treated as static documents on the CDN • Queries should not require external resources • Lucene search index can be held entirely in memory
- 21. Catalog and materializedviews Resolver metadata Another catalog HTML Web site catalog LUCENE Search
- 22. Building the catalog page0.json A B C page1.json D E F page2.json G H I index.json
- 23. Reading the catalog index.json page0.json A B C page1.json D E F page2.json G H I
- 24. Commit timestamps 1| 1 | 2 1 1 2 3 | 3 | 3 3 3 3 4 | 5 | 5 4 5 5 2 | 3 | 5
- 25. Incremental updates 1| 1 | 2 1 1 2 3 | 3 | 3 3 3 3 4 | 5 | 5 4 5 5 2 | 3 | 5 Cursor: 45
- 26. Materializing views asblobs Read new catalog entries Apply view Identify blobs to be written Read any existing blobs Merge data using RDF Write the new blob
- 27.
- 28.
- 29. What we planned •Add API v3 support to nuget.org • Update the NuGet client to use API v3 • Factor the server into NuGet packages • Factor the client into NuGet packages
- 30. Design resource formats • Search • Dependency resolution •Website rendering • Other package metadata reads • Gallery mirroring • Custom metadata consumption scenarios
- 31. Representing a graphin JSON • Collection+JSON and Collection.Next+JSON • Very similar to ATOM • Includes query templates • JSON-API and most others • Emphasis on RESTful CRUD • Client specification of data shape in some • We didn’t want to expose our raw data model • Scenario-specific views instead
- 32.
- 33. JSON-LD HTTP://JSON-LD.ORG/ { "@context": "http://json-ld.org/contexts/person.jsonld", "@id": "http://dbpedia.org/resource/John_Lennon", "name": "John Lennon", "born": "1940-10-09", "spouse": "http://dbpedia.org/resource/Cynthia_Lennon" }
- 34. Linked Data •@context allows the JSON document to be treated as an RDF data-set • RDF = Resource Definition Framework • Fully-qualified names and relationships (URI triples) • RDF data-sets benefits • Easily merged and queried • Idempotent • Everything is namespaced • Based on W3C standards • RDF 1.0: 1999 • RDF 1.1: February 2014 - http://www.w3.org/RDF/ • JSON-LD: January 2014 - http://www.w3.org/TR/json-ld/ (W3C Recommendation)
- 35. Azure Blob Storage API v3 search (feed) Search Service Azure worker role (3+ instances) Lucene.NET Full index loaded into memory Backend jobs Azure VM (1 instance) SQL Azure warehouse Gallery / API v2 Azure web role (3+ instances) Content Delivery Network (CDN) Package Downloads SQL Azure database Website requests API v3 Package Metadata Metrics Service Azure website (3+ instances)
- 36. API v3 clients • Get service index from CDN • Links are followed to resources: • Search service • Dependency resolver views • Other metadata views • Packages • Metrics service
- 37.
- 38.
- 39.
- 40. API v2 clientbehavior locked • Lots of existing clients in the wild • All requests are made through a single Gallery pipeline • Download requests • Record statistics to database • Redirect to the CDN to get the nupkg
- 41. Azure Blob Storage API v3 search (feed) Search Service Azure worker role (3+ instances) Lucene.NET Full Index Loaded into Memory Backend jobs Azure VM (1 instance) SQL Azure warehouse Gallery / API v2 Azure web role (3+ instances) Content Delivery Network (CDN) Package Downloads SQL Azure database Website requests API v3 Package Metadata Metrics Service Azure website (3+ instances) API v2 OData
- 42. Search service integration Gallery receives OData request Search queries identified and stopped API v3 request to search service JSON-LD response Transformed into OData XML OData XML returned
- 43. Metrics service integration Gallery receives download request Call to database is removed API v3 request to metrics service JSON-LD POST Fire and forget Redirect to package on CDN
- 44. Iterative success •Usage-based priorities for availability • JSON-LD as our API v3 resource format • Scenario-focused API v3 subsystems • API v2 gallery became the first API v3 client • API v3 responses are transformed into API v2 responses • Getting API v3 benefits for existing API v2 users
- 45. NUGET 3.0 –TRANSITIONING FROM ODATA TO JSON-LD Friday at 14:20 in KITT
- 46. Thanks Come getNuGet stickers Gallery and Backend Services http://github.com/NuGet Catalog and Collectors http://github.com/NuGet/NuGet.Services.Metadata JSON-LD Processor http://github.com/NuGet/json-ld.net http://www.nuget.org/packages/json-ld.net @jeffhandley | jeffhandley.com | jeff.handley@microsoft.com | blog.nuget.org