Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 51

NuGet beyond Hello World - DotNext Piter 2017

2

Share

Download to read offline

Everybody is consuming NuGet packages these days. It’s easy, right? But how can we create and share our own packages? What is .NET Standard? How should we version, create, publish and share our package?
Once we have those things covered, we’ll look beyond what everyone is doing. How can we use the NuGet client API to fetch data from NuGet? Can we build an application plugin system based on NuGet? What hidden gems are there in the NuGet server API? Can we create a full copy of NuGet.org?
Good questions! In this talk, we will get them answered.

More Related Content

You Might Also Like

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

NuGet beyond Hello World - DotNext Piter 2017

  1. 1. 1 NuGet beyond Hello World Maarten Balliauw @maartenballiauw
  2. 2. 3 Who am I? Maarten Balliauw Antwerp, Belgium Developer Advocate, JetBrains Founder, MyGet AZUG Focus on web ASP.NET MVC, Azure, SignalR, ... Former MVP Azure & ASPInsider Big passion: Azure http://blog.maartenballiauw.be @maartenballiauw
  3. 3. 4 Agenda Introduction Maturing NuGet (conventions) Beyond package dependencies Extending tools Using client SDK Server API
  4. 4. 5 Introduction
  5. 5. 6 NuGet? Project started in 2011 to make dependency management easier & better Several things Repositories Public repository at www.nuget.org Private repositories everywhere (www.myget.org ) Clients & tools NuGet.exe, Visual Studio, Rider, Xamarin Studio, Paket, … dotnet CLI tools Conventions
  6. 6. 7 NuGet evolution Others have been doing package management for decades Perl (CPAN.org) Linux (RPM/YUM/APT-GET/...) PHP (PEAR, Composer) Node (npm) Ruby (Gems) We can learn a great deal from these! (and we did, with .NET Core / new .csproj)
  7. 7. 8 Learnings Every project is a package Clearly identifyable Proper Semantic Versioning Tooling makes it easy to package Every project becomes discoverable Nice description, release notes, ... Add it to a (private) feed so developers can find it Dependencies stay out of source control Dependencies are versioned
  8. 8. 9 Maturing NuGet
  9. 9. 10 Package producers are responsible for consumers. - Maarten Balliauw
  10. 10. 11 It’s easy to break the world… NPM and left-pad http://www.haneycodes.net/npm-left-pad-have-we-forgotten-how-to-program/ Package removed… Switch to .NET 4.6.1, suddenly can’t start application https://github.com/dotnet/announcements/issues/6 Caused by System.Net.Http changes Change in behaviour (new Exception type thrown) Some public method signatures changed Need for conventions!
  11. 11. 12 Don’t break the world! Never delete a package (unless it’s absolutely unused) https://docs.microsoft.com/en-us/nuget/policies/deleting-packages Versioning = specify intent Detect public API changes Use semantic versioning Targeting = specify what is supported Metadata and symbols = troubleshooting help
  12. 12. 13 Versioning
  13. 13. 14 Use semantic versioning to state intent <major>.<minor>.<patch>-<prerelease>+<metadata> Increment major on breaking change Increment minor on new feature Increment patch for bugfixes Prerelease means it’s not stable yet Metadata can annotate build info (not supported on NuGet.org yet) http://semver.org/
  14. 14. 15 State intent & acceptable changes Semantic versioning states intent Did we break something? Do you need this update? (feature/bugfix) Are we on scary territory? (prerelease) Helps consumers specify dependency ranges 1.0 = 1.0 ≤ x (,1.0] = x ≤ 1.0 (,1.0) = x < 1.0 [1.0] = x == 1.0 (1.0,) = 1.0 < x (1.0,2.0) = 1.0 < x < 2.0 [1.0,2.0] = 1.0 ≤ x ≤ 2.0 1.0-* = give me all prereleases in 1.0 ≤ x ≤ 1.1
  15. 15. 16 Use the tools API Comparer Compare public API surface and detect breaking API’s https://github.com/ParticularLabs/APIComparer Example: http://apicomparer.particular.net/compare/rabbitmq.client/3.4.3...3.5.0 GitVersion Auto-generate package version number based on Git info https://github.com/GitTools/GitVersion
  16. 16. 17 Targeting
  17. 17. 18 Frameworks and versions .NET has so many frameworks! .NET framework, .NET Core, Xamarin, Tizen, Micro Framework Windows Phone Nice list in NuGet repo: FrameworkConstants.cs Our NuGet package may not (want to) support them all Be specific about targeting – state platform support Target one or multiple
  18. 18. 19 Multi-targeting demo
  19. 19. 20 .NET Standard Multi-targeting can be exhausting (add and test all TFM configurations) Just select one TFM: .NET Standard https://github.com/dotnet/standard Defines API’s a platform should implement as a minimum Specification describing supported features (not platforms) Higher version: more fx features exposed to your library Lower version: less features but more platforms supported https://docs.microsoft.com/en-us/dotnet/articles/standard/library
  20. 20. 21 Metadata & symbols
  21. 21. 22 Metadata Make it easy for people to find: Description Release notes Project site License info ... (also shown in IDE)
  22. 22. 23 Symbols Help people troubleshoot! Their own code by understanding yours Your code (better issues logged / PR) Publish package symbols Symbols packages – IncludeSource & IncludeSymbols Needs symbol server like SymbolSource.org / MyGet.org / … Portable PDB (symbols + sources embedded) – tools like GitLink / SourceLink Just works™ - but some effort needed as a package author
  23. 23. 24 Metadata and symbols demo
  24. 24. 25 Beyond project dependencies
  25. 25. 26 NuGet More than just a package manager Extensibility (dotnet CLI) NuGet is… a protocol! API for transferring metadata and binaries Server API Client-side libraries Versioning system built-in
  26. 26. 27 Extending dotnet CLI
  27. 27. 28 Extending dotnet CLI Why? Examples dotnet nuget – the NuGet command line, integrated dotnet ef – Entity Framework command line tools Create tools your team/customers can use dotnet protobuf – Companion tool for creating protobuf models After this talk, go and build for me  dotnet outdated – Scan and show outdated package references dotnet mirror – Mirror referenced packages to a local folder dotnet nuke – Clear all NuGet cache locations
  28. 28. 29 Extending dotnet CLI How? Create a console application Name it dotnet-<something> Package it with <PackageType>DotnetCliTool</PackageType> Install as a CLI tool (or in %PATH%) https://github.com/dotnet/docs/blob/master/docs/core/tools/extensibility.md
  29. 29. 30 Extending dotnet CLI demo https://github.com/maartenba/dotnetcli-init
  30. 30. 31 Client SDK
  31. 31. 32 Consuming NuGet programatically Why? Find package metadata Download and extract package Custom build task, custom tool used in CI, … Distribute applications / plugins as NuGet packages Transport and versioning come free! NuGet Package Explorer, R# extension manager, Chocolatey, Octopus Deploy, Seq, CMS like EPIserver, SiteCore, Orchard, …
  32. 32. 33 Consuming NuGet programatically How? Old world: NuGet.Core New world: NuGet v3 client Series of packages, NuGet.PackageManagement being the main package Open-source on GitHub Dave Glick has a good series (part 1, part 2, part 3)
  33. 33. 34 Packages to install NuGet.PackageManagement Overall management of packages, sources, … NuGet.Protocol.Core.v2 V2 protocol client (OData) NuGet.Protocol.Core.v3 V3 protocol client (JSON and randomness) NuGet.ProjectManagement Project system (inside IDE and inside file system) NuGet.Packaging Read and write .nuspec and .nupkg NuGet.Versioning Implements working with versions and ranges NuGet.Commands (optional) Command-line commens (spec, pack, publish, ...) implemented as static classes mostly
  34. 34. 35 NuGet, via C# demo
  35. 35. 36 A few things to know… All operations performed via resources (e.g. PackageSearchResource, ListCommandResource, …) All operations performed against a project Folder (note: only supports installing as there is no tracking) PackagesConfig MSBuild Your own implementation Package manager orchestrates all operations Package repository holds packages (can be remote or local)
  36. 36. 37 SearchPortal demo
  37. 37. 38 Server API
  38. 38. 39 NuGet talks to a repository Can be on disk/network share Or remote over HTTP(S) 2* API’s V2 – OData based (used by pretty much all NuGet servers out there) V3 – JSON based (available on NuGet.org and MyGet.org) * This is a lie…
  39. 39. 40 V2 Protocol OData, started as “OData-to-LINQ-to-Entities” (V1 protocol) Optimizations added to reduce # of random DB queries (VS2013+ & NuGet 2.x) Search – Package manager list/search FindPackagesById – Package restore (Does it exist? Where to download?) GetUpdates – Package manager updates https://www.nuget.org/api/v2 (code in https://github.com/NuGet/NuGetGallery)
  40. 40. 41 V3 Protocol JSON based More of a “resource provider” (see client SDK) of various endpoints per purpose Catalog (NuGet.org only) – append-only event log Registrations – latest incarnation of a package Flat container - .NET Core package restore (and VS autocompletion) Report abuse URL template Statistics … https://api.nuget.org/v3/index.json (code in https://github.com/NuGet/NuGet.Services.Metadata)
  41. 41. 42 V3 Protocol is… interesting Service index points to resources @id, @type (versioned), comment Client should load-balance on @type occurrences NuGet.org does this to perform search etc.: RetryingHttpClientWrapper (also has a request queue) Many views on the same data Specific to a scenario in the client Some compute based, some storage based Search, autocomplete -> compute Registrations, flatcontainer -> storage Catalog -> NuGet.org internal use
  42. 42. 43 Server API demo
  43. 43. 44 How does it fit together? User uploads to NuGet.org Data added to database Data added to catalog (append-only data stream) Various jobs run over catalog using a cursor Registrations (last state of a package/version), reference catalog entry Flatcontainer (fast restores) Search index (search, autocomplete, NuGet Gallery search)
  44. 44. 45 How does it fit together? User uploads to NuGet.org Data added to database Data added to catalog (append-only data stream) Various jobs run over catalog using a cursor Registrations (last state of a package/version), reference catalog entry Flatcontainer (fast restores) Search index (search, autocomplete, NuGet Gallery search)
  45. 45. 46 NuGet.org catalog demo
  46. 46. 47 Catalog data Updates (add/update) Deletes Chronological Can continue where left off Can restore NuGet.org to a given point in time Why use it? Mirror locally Send notifications when certain packages changed Build own index, build own datastore based on data E.g. https://packagesearch.azurewebsites.net/ …
  47. 47. 48 Other extensibility points
  48. 48. 49 Other extensibility points NuGet Credential Provider https://docs.microsoft.com/en-us/nuget/api/nuget-exe-credential-providers NuGet package download https://github.com/NuGet/Home/wiki/NuGet-Package-Download-Plugin
  49. 49. 50 Conclusion
  50. 50. 51 Conclusion Go out and create packages! Follow conventions, help your consumers, help mature NuGet Versioning Targeting Symbols Consider NuGet not only as package dependencies but as a protocol Tooling extensibility Protocol for metadata and versioning
  51. 51. 52 Thank you! http://blog.maartenballiauw.be @maartenballiauw

Editor's Notes

  • https://pixabay.com
  • https://pixabay.com/en/tires-used-tires-pfu-garbage-1846674/
  • Talk about how we learned from the others to consider everythign we make a package. In .NET Core this is made easier by having tooling to quickly set metadata and publish the package.
  • https://pixabay.com/en/tires-used-tires-pfu-garbage-1846674/
  • Multi-targeting used to be a pain, different compilation steps, manually compose NuGet package
    Now: easy!
    Open VS and create new project
    View project properties, Packaging, tick the box, provide some metadata
    Open the .csproj file to show this data is in the .csproj
    Build, see package is created
    Increment major version as we are changing TFM’s which is probably breaking, be good citizen!
    Edit project file and add more TFM’s <TargetFrameworks>netstandard16;net45;net462</TargetFrameworks> (NOTE THE PLURAL S)
    Explain we can use directives now, too, e.g. #if NET462
    Compile and build, open package in package explorer
    See we now have a package that supports multiple, and NuGet client will use correct on in consumer projects
  • Open VS project from previous demo
    Show metadata options that can be provided (license, release notes, ...)
    How to enable symbols publishing?
    Symbols package – add <IncludeSource>True</IncludeSource> and add <IncludeSymbols>True</IncludeSymbols>
    Show resulting packages and what is in them
    Explain both need to be uploaded in order to provide debugger step through etc
    Good to keep code private / only share symbols with known users
    Portable/embedded PDB where symbols and sources are embedded using GitLink/SourceLink
    Check website on how to set it up
    Result: it just works
    Create console app


  • https://pixabay.com/en/tires-used-tires-pfu-garbage-1846674/
  • Show the tool in action!
    Open Rider, create new class library
    Edit project file, add <ItemGroup><DotNetCliToolReference Include="DotNetInit" Version="*" /></ItemGroup>
    Run dotnet restore & dotnet init (in correct folder)
    Profit
    How to create such tool?
    Not going to go through code completely as it’s on GitHub, but in simple terms:
    Open https://github.com/maartenba/dotnetcli-init
    See it’s just Program.cs that takes args
    Can be run standalone
    Edit project file, show:
    To be part of dotnet-cli -> name it dotnet-<something>
    Package type (case sensitive)
  • Create new Console Application in Rider
    Explain we will build 2 scenarios: a search which displays package metadata, and an install/extract
    Add NuGet package: NuGet.PackageManagement (should bring down all dependencies we need) AND NuGet.Protocol.Core.v2
    Let’s start with search first.
    Program.cs -> SearchPackageAndDumpDetails("json");
    Code – make sure to explain:
    ILogger
    Everything is based around resource providers (like protocol support, search, …)
    private static async Task SearchPackageAndDumpDetails(string searchTerm) { var logger = new NullLogger(); var providers = new List<Lazy<INuGetResourceProvider>>(); providers.AddRange(Repository.Provider.GetCoreV2()); providers.AddRange(Repository.Provider.GetCoreV3()); var packageSource = new PackageSource("https://api.nuget.org/v3/index.json"); var repository = new SourceRepository(packageSource, providers); var searchResource = await repository.GetResourceAsync<PackageSearchResource>(); var searchResults = await searchResource.SearchAsync(searchTerm, new SearchFilter(includePrerelease: true), 0, 10, logger, CancellationToken.None); foreach (var searchResult in searchResults) { Console.WriteLine(searchResult.Identity.Id + " " + searchResult.Identity.Version); } Console.ReadLine(); }

    Now let’s also install a package. For this, we need a project context.
    Mention context provider for logging or use EmptyNuGetProjectContext
    Code:

    private static async Task InstallPackage(string packageId, string packageVersion, bool includeDependencies) { // Package we ant to install var packageIdentity = new PackageIdentity(packageId, NuGetVersion.Parse(packageVersion)); // Remote var providers = new List<Lazy<INuGetResourceProvider>>(); providers.AddRange(Repository.Provider.GetCoreV2()); providers.AddRange(Repository.Provider.GetCoreV3()); var packageSource = new PackageSource("https://api.nuget.org/v3/index.json"); var repository = new SourceRepository(packageSource, providers); // Local var projectPath = "c:\\users\\maart\\desktop\\temp"; var packagesPath = "c:\\users\\maart\\desktop\\temp\\packages"; var project = new FolderNuGetProject(projectPath); var settings = Settings.LoadDefaultSettings(projectPath, null, new XPlatMachineWideSetting()); // Create package manager var repositoryProvider = new SourceRepositoryProvider(settings, providers); var packageManager = new NuGetPackageManager(repositoryProvider, settings, packagesPath) { PackagesFolderNuGetProject = project }; // Create dependency resolution context (no prerelease/unlisted) var dependencyBehaviour = includeDependencies ? DependencyBehavior.Lowest : DependencyBehavior.Ignore; var resolutionContext = new ResolutionContext(dependencyBehaviour, false, false, VersionConstraints.None); // Install the package already var projectContext = new LoggingNuGetProjectContext(); await packageManager.InstallPackageAsync(packageManager.PackagesFolderNuGetProject, packageIdentity, resolutionContext, projectContext, repository, new SourceRepository[] {}, CancellationToken.None); }
  • Open SearchPortal application, demo things (install plugin, see it shows up in portal)
    In code, walk through PluginManager class
    Plugins stored in plugins folder
    ListPlugins -> searches remote and combines with local which is an issue with FolderNuGetProject! (no tracking)
    SearchPortalNuGetProject -> wraps a packages config project as well to keep track of what is actually installed
    Install -> just like in previous demo
    Uninstall -> again using override as FolderNuGetProject has no tracking
    (will not always work as appdomain needs to be unloaded to remove binaries, could do shadow copy to overcome this but meh, demo purposes)
  • Open Fiddler files
    Go through V2 one
    Go through V3 one and look at various resources, mention @id, @type, comment
    Mention versions as well (e.g. registrations with/without semver2 support)
    Open a few resources and try some requests
    Autocomplete: https://api-v2v3search-0.nuget.org/autocomplete?q=json&supportedFramework=any&prerelease=true
    Autocomplete versions: https://api-v2v3search-0.nuget.org/autocomplete?id=newtonsoft.json
    Search: https://api-v2v3search-0.nuget.org/query?q=json&supportedFramework=any&prerelease=true
    Note only latest version + list of version numbers
    Note link to catalog and registrations
    Registration: https://api.nuget.org/v3/registration0/newtonsoft.json/index.json
    Note metadata
    Note paging, ranges, ...
    Registration version: https://api.nuget.org/v3/registration0/newtonsoft.json/10.0.1.json
    Flat container (templated): https://api.nuget.org/v3/flatcontainer/newtonsoft.json/index.json
    Flat container download (templated): https://api.nuget.org/v3/flatcontainer/newtonsoft.json/10.0.1/newtonsoft.json.10.0.1.nupkg
    Catalog: list of commits https://api.nuget.org/v3/catalog0/index.json
    Catalog: specific page https://api.nuget.org/v3/catalog0/page2318.json
  • Open collector demo project
    Show packges installed, most important: NuGet.Services.Metadata.Catalog
    Contains “collectors” that traverse over the catalog and to which we can subscribe
    Quite an internal implementation, but created a SimpleCommitCollector
    Handles OnProcessBatch and does some data transformation
    Fires either ProcessPackageUpdateAsync or ProcessPackageDeleteAsync
    View hierarchy (Ctrl+E, Ctrl+H)
    Mention NuGet implementations galore in NuGet.Services.Metadata, even unused ones (but also the ones that create search index, flatcontainer, ...)
    Example: EchoCollector
    Run it
    Breakpoint, show properties available
    Example: MirrorCollector
    Run it
    See it mirrors packages
    Dirty way of building download URL, should use service index to get template, but it works anyway
  • ×