Everybody is consuming or producing NuGet packages these days. It’s easy, right? We’ll look beyond what everyone is doing. How can we use the NuGet client API to fetch data from NuGet? Can we build an application plugin system based on NuGet? What hidden gems are there in the NuGet server API? Can we create a full copy of NuGet.org?
4. 5
NuGet?
Project started in 2011 to make dependency management easier & better
Several things
Repositories
Public repository at www.nuget.org
Private repositories everywhere (www.myget.org)
Clients & tools
NuGet.exe, Visual Studio, Rider, Xamarin Studio, Paket, …
dotnet CLI tools
Conventions
SemVer, metadata and symbols
5. 6
Maturity model
No package
manager in
use
Consuming
publicly
available
packages
Building and
consuming
own packages
Proper
SemVer,
promotion
flow, license
and
vulnerability
management
7. 8
NuGet
More than just a package manager
Extensibility (dotnet new, dotnet CLI)
NuGet is… a protocol!
API for transferring metadata and binaries
Server API
Client-side libraries
Versioning system built-in
9. 10
Project/item templates as a package
New MVC project:
dotnet new mvc --auth Individual
New Web.config file:
dotnet new webconfig
10. 11
Project/item templates as a package
Roll your own templates!
https://docs.microsoft.com/en-us/dotnet/core/tutorials/create-custom-template
https://github.com/dotnet/templating/wiki/Available-templates-for-dotnet-new
TL;DR:
Add /.template.config/template.json file in package
Add template files, e.g. /ItemTemplate.cs
Create a NuGet package containing the above
14. 15
Extending dotnet CLI
Why?
Existing examples
dotnet nuget – the NuGet command line, integrated
dotnet ef – Entity Framework command line tools
dotnet watch – Automatically load updates to an ASP.NET Core application
Create tools your team/customers can use (please go build these for me )
dotnet protobuf – Companion tool for creating protobuf models
dotnet optimize-images – Optimize images in a project
dotnet outdated – Scan and show outdated package references
dotnet mirror – Mirror referenced packages to a local folder
15. 16
Extending dotnet CLI
How?
Create a console application
Name it dotnet-<something>
Package it with <PackageType>DotnetCliTool</PackageType>
Install as a CLI tool (or in %PATH%)
(per project in .NET Core <= 2.0, global in .NET Core >= 2.1)
https://github.com/dotnet/docs/blob/master/docs/core/tools/extensibility.md
18. 19
Consuming NuGet programatically
Why?
Find package metadata
Download and extract package
Custom build task, custom tool used in CI, …
Distribute applications / plugins as NuGet packages
Transport and versioning come free!
dotnet new, NuGet Package Explorer, R# extension manager, Chocolatey, Octopus Deploy,
Seq, CMS like EPIserver, SiteCore, Orchard, …
19. 20
Consuming NuGet programatically
How?
Old world: NuGet.Core
New world: NuGet v3 client
Series of packages, NuGet.PackageManagement being the main package
Open-source on GitHub
Dave Glick has a good series (part 1, part 2, part 3)
20. 21
Packages to install
NuGet.PackageManagement
Overall management of packages, sources, …
NuGet.Protocol.Core.v2
V2 protocol client (OData)
NuGet.Protocol.Core.v3
V3 protocol client (JSON and randomness)
NuGet.ProjectManagement
Project system (inside IDE and inside file system)
NuGet.Packaging
Read and write .nuspec and .nupkg
NuGet.Versioning
Implements working with versions and ranges
NuGet.Commands (optional)
Command-line commens (spec, pack, publish, ...) implemented as static classes mostly
22. 23
A few things to know…
All operations performed via resources (e.g. PackageSearchResource,
ListCommandResource, …)
All operations performed against a project
Folder (note: only supports installing as there is no tracking)
PackagesConfig
MSBuild
Your own implementation
Package manager orchestrates all operations
Package repository holds packages (can be remote or local)
25. 26
NuGet talks to a repository
Can be on disk/network share
Or remote over HTTP(S)
2* API’s
V2 – OData based (used by pretty much all NuGet servers out there)
V3 – JSON based (available on NuGet.org and MyGet.org)
* This is a lie…
26. 27
V2 Protocol
OData, started as “OData-to-LINQ-to-Entities” (V1 protocol)
Optimizations added to reduce # of random DB queries (VS2013+ & NuGet 2.x)
Search – Package manager list/search
FindPackagesById – Package restore (Does it exist? Where to download?)
GetUpdates – Package manager updates
https://www.nuget.org/api/v2 (code in https://github.com/NuGet/NuGetGallery)
27. 28
V3 Protocol
JSON based
More of a “resource provider” (see client SDK) of various endpoints per purpose
Catalog (NuGet.org only) – append-only event log
Registrations – latest incarnation of a package
Flat container - .NET Core package restore (and VS autocompletion)
Report abuse URL template
Statistics
…
https://api.nuget.org/v3/index.json (code in https://github.com/NuGet/NuGet.Services.Metadata)
28. 29
V3 Protocol is… interesting
Service index points to resources
@id, @type (versioned), comment
Client should load-balance on @type occurrences
NuGet.org does this to perform search etc.: RetryingHttpClientWrapper
(also has a request queue)
Many views on the same data
Specific to a scenario in the client
Some compute based, some storage based
Search, autocomplete -> compute
Registrations, flatcontainer -> storage
Catalog -> NuGet.org internal use
30. 31
How does it fit together?
User uploads to NuGet.org
Data added to database
Data added to catalog (append-only data stream)
Various jobs run over catalog using a cursor
Registrations (last state of a package/version), reference catalog entry
Flatcontainer (fast restores)
Search index (search, autocomplete, NuGet Gallery search)
31. 32
How does it fit together?
User uploads to NuGet.org
Data added to database
Data added to catalog (append-only data stream)
Various jobs run over catalog using a cursor
Registrations (last state of a package/version), reference catalog entry
Flatcontainer (fast restores)
Search index (search, autocomplete, NuGet Gallery search)
33. 34
Catalog data
Updates (add/update)
Deletes
Chronological
Can continue where left off
Can restore NuGet.org to a given point in time
Why use it?
Mirror locally
Send notifications when certain packages changed
Build own index, build own datastore based on data
E.g. https://packagesearch.azurewebsites.net/
…
Open package in NPE
Show people around the files
Add as template to Rider
Create new project
Show the tool in action!
Open Rider, create new class library
Edit project file, add <ItemGroup><DotNetCliToolReference Include="DotNetInit" Version="*" /></ItemGroup>
Run dotnet restore & dotnet init (in correct folder)
Profit
How to create such tool?
Not going to go through code completely as it’s on GitHub, but in simple terms:
Open https://github.com/maartenba/dotnetcli-init
See it’s just Program.cs that takes args
Can be run standalone
Edit project file, show:
To be part of dotnet-cli -> name it dotnet-<something>
Package type (case sensitive)
Create new Console Application in Rider
Explain we will build 2 scenarios: a search which displays package metadata, and an install/extract
Add NuGet package: NuGet.PackageManagement (should bring down all dependencies we need) AND NuGet.Protocol.Core.v2
Let’s start with search first.
Program.cs -> SearchPackageAndDumpDetails("json");
Code – make sure to explain:
ILogger
Everything is based around resource providers (like protocol support, search, …)
private static async Task SearchPackageAndDumpDetails(string searchTerm){ var logger = new NullLogger(); var providers = new List<Lazy<INuGetResourceProvider>>(); providers.AddRange(Repository.Provider.GetCoreV2()); providers.AddRange(Repository.Provider.GetCoreV3()); var packageSource = new PackageSource("https://api.nuget.org/v3/index.json"); var repository = new SourceRepository(packageSource, providers); var searchResource = await repository.GetResourceAsync<PackageSearchResource>(); var searchResults = await searchResource.SearchAsync(searchTerm, new SearchFilter(includePrerelease: true), 0, 10, logger, CancellationToken.None); foreach (var searchResult in searchResults) { Console.WriteLine(searchResult.Identity.Id + " " + searchResult.Identity.Version); } Console.ReadLine();}
Now let’s also install a package. For this, we need a project context.
Mention context provider for logging or use EmptyNuGetProjectContext
Code:
private static async Task InstallPackage(string packageId, string packageVersion, bool includeDependencies){ // Package we ant to install var packageIdentity = new PackageIdentity(packageId, NuGetVersion.Parse(packageVersion)); // Remote var providers = new List<Lazy<INuGetResourceProvider>>(); providers.AddRange(Repository.Provider.GetCoreV2()); providers.AddRange(Repository.Provider.GetCoreV3()); var packageSource = new PackageSource("https://api.nuget.org/v3/index.json"); var repository = new SourceRepository(packageSource, providers); // Local var projectPath = "c:\\users\\maart\\desktop\\temp"; var packagesPath = "c:\\users\\maart\\desktop\\temp\\packages"; var project = new FolderNuGetProject(projectPath); var settings = Settings.LoadDefaultSettings(projectPath, null, new XPlatMachineWideSetting()); // Create package manager var repositoryProvider = new SourceRepositoryProvider(settings, providers); var packageManager = new NuGetPackageManager(repositoryProvider, settings, packagesPath) { PackagesFolderNuGetProject = project }; // Create dependency resolution context (no prerelease/unlisted) var dependencyBehaviour = includeDependencies ? DependencyBehavior.Lowest : DependencyBehavior.Ignore; var resolutionContext = new ResolutionContext(dependencyBehaviour, false, false, VersionConstraints.None); // Install the package already var projectContext = new LoggingNuGetProjectContext(); await packageManager.InstallPackageAsync(packageManager.PackagesFolderNuGetProject, packageIdentity, resolutionContext, projectContext, repository, new SourceRepository[] {}, CancellationToken.None);}
Open SearchPortal application, demo things (install plugin, see it shows up in portal)
In code, walk through PluginManager class
Plugins stored in plugins folder
ListPlugins -> searches remote and combines with local which is an issue with FolderNuGetProject! (no tracking)
SearchPortalNuGetProject -> wraps a packages config project as well to keep track of what is actually installed
Install -> just like in previous demo
Uninstall -> again using override as FolderNuGetProject has no tracking
(will not always work as appdomain needs to be unloaded to remove binaries, could do shadow copy to overcome this but meh, demo purposes)
Open Fiddler files
Go through V2 one
Go through V3 one and look at various resources, mention @id, @type, comment
Mention versions as well (e.g. registrations with/without semver2 support)
Open a few resources and try some requests
Autocomplete: https://api-v2v3search-0.nuget.org/autocomplete?q=json&supportedFramework=any&prerelease=true
Autocomplete versions: https://api-v2v3search-0.nuget.org/autocomplete?id=newtonsoft.json
Search: https://api-v2v3search-0.nuget.org/query?q=json&supportedFramework=any&prerelease=true
Note only latest version + list of version numbers
Note link to catalog and registrations
Registration: https://api.nuget.org/v3/registration0/newtonsoft.json/index.json
Note metadata
Note paging, ranges, ...
Registration version: https://api.nuget.org/v3/registration0/newtonsoft.json/10.0.1.json
Flat container (templated): https://api.nuget.org/v3/flatcontainer/newtonsoft.json/index.json
Flat container download (templated): https://api.nuget.org/v3/flatcontainer/newtonsoft.json/10.0.1/newtonsoft.json.10.0.1.nupkg
Catalog: list of commits https://api.nuget.org/v3/catalog0/index.json
Catalog: specific page https://api.nuget.org/v3/catalog0/page2318.json
Open collector demo project
Show packges installed, most important: NuGet.Services.Metadata.Catalog
Contains “collectors” that traverse over the catalog and to which we can subscribe
Quite an internal implementation, but created a SimpleCommitCollector
Handles OnProcessBatch and does some data transformation
Fires either ProcessPackageUpdateAsync or ProcessPackageDeleteAsync
View hierarchy (Ctrl+E, Ctrl+H)
Mention NuGet implementations galore in NuGet.Services.Metadata, even unused ones (but also the ones that create search index, flatcontainer, ...)
Example: EchoCollector
Run it
Breakpoint, show properties available
Example: MirrorCollector
Run it
See it mirrors packages
Dirty way of building download URL, should use service index to get template, but it works anyway