0
Developing NuGet
JEFF HANDLEY | DEVELOPMENT LEAD | NUGET | MICROSOFT
@JEFFHANDLEY
JEFFHANDLEY.COM
1
Airborne (1993)
2SOURCE: WWW.IMDB.COM
3
4
5
NuGet Landscape
WHAT MAKES UP THE NUGET PROJECT
6
NuGet Project
Open Source (owned by Outercurve Foundation)
◦ Releases shipped by Microsoft
Included in all Visual Studio i...
Outercurve and Microsoft
NUGET CLIENT
Outercurve’s
NuGet Project
Microsoft’s
NuGet-Based
Package Manager
In the Box with
V...
NuGet’s Core Team
NUGET CLIENT
Visual Studio Extension
WebMatrix Extension
NuGet.exe Command-Line
NuGet Packages
3 Develop...
NuGet’s Core Team
NUGET CLIENT
Visual Studio Extension
WebMatrix Extension
NuGet.exe Command-Line
NuGet Packages
3 Develop...
Release Management
NUGET CLIENT
11
NuGet Releases
New release every 10 weeks
since NuGet 1.5
12
2/ 12/ 2011
1.1
3/ 30/ 2011
1.2
4/ 25/ 2011
1.3
6/ 17/ 2011
1...
NuGet in Visual Studio Releases
13
2/ 12/ 2011
1.1
3/ 30/ 2011
1.2
4/ 25/ 2011
1.3
6/ 17/ 2011
1.4 8/ 30/ 2011
1.5
12/ 13/...
Release Tiers
CI build from Outercurve Foundation always available at http://build.nuget.org
◦ Usually 100 or fewer users
...
Backlog
CodePlex: 858 open issues as of 10/4/2013
Microsoft Internal Issues
◦ TFS: Fluctuates from 0 to 25
◦ Dr. Watson re...
Strategic Product Features
Package Manager for the Microsoft Development Platform
◦ Not just ASP.NET
◦ Not just .NET
◦ Inc...
Triaging Issues
1. Is it a high-priority issue (crash, hang, regression)?
◦ Include it in the current release
2. Is the bu...
Priorities
1. Strategic roadmap that integrates with the Microsoft development platform
◦ Features planned for 2-3 release...
Philosophy
Small and easy bug fixes are low priority
◦ Otherwise, we’d spend 100% of our time making small changes that af...
Conflicting Requirements
Microsoft’s Customers vs. NuGet’s Ecosystem
◦ Group Policy control over package sources
Bold New ...
Source Control
NUGET CLIENT
21
master
merge
Feature
A
Feature
B
Branch: 2.7
Branch: 2.7.1
mergemerge
Branch: 2.8
Tag: 2.7
Tag: 2.7.1
Git
Branch per featu...
Continuous Integration
Outercurve Builds
◦ TeamCity – running on Virtual Machines in Windows Azure
◦ 1 VM build agent with...
Going Dark
Microsoft-confidential work cannot go to CodePlex
◦ Integration with new versions of Visual Studio
◦ Support fo...
Testing
NUGET CLIENT
25
Release Timing
Release sign-off takes about 4 weeks after code-complete
◦ 40% of our 10-week release cadence is spent test...
Test Matrix Dimensions
Visual Studio SKUS
◦ VS 2010
◦ Express for Web
◦ Express for Phone
◦ Pro+
◦ VS 2012+ - All SKUs
Pro...
Achieving Release Sign-Off
Unit Tests managed and run by developers
End-to-end Functional tests managed and run by develop...
Throughput
109 Weeks
• Total time working
on those 10 releases
10 Releases
• 816 work items have
been completed
3 Develope...
Success
Never defined by throughput
No regressions
No minor updates necessary
Users delighted by new features
Took steps t...
Success Rate
NuGet 1.6
◦ Required a 1.6.1 update
◦ Regressions
◦ Missed bugs in new features
NuGet 1.7
◦ Success
NuGet 1.8...
Tiered Releases are Key
Our tiered release approach worked for 2.7
2.7.1 update in place in time for integrating
into Visu...
Q & A
Get a NuGet Sticker!
Twitter: @jeffhandley
Website: jeffhandley.com
33
Upcoming SlideShare
Loading in...5
×

Developing NuGet

606

Published on

Learn about how the core NuGet team the NuGet Client project. The project has challenges related to:

- Open Source
- Hard Deadlines
- Very Large Backlog
- Many Conflicting Requirements
- Git

We'll talk about our strategies for:

- Issue tracking - Branching
- Continuous integration
- Testing
- Release sign-off

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
606
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Developing NuGet"

  1. 1. Developing NuGet JEFF HANDLEY | DEVELOPMENT LEAD | NUGET | MICROSOFT @JEFFHANDLEY JEFFHANDLEY.COM 1
  2. 2. Airborne (1993) 2SOURCE: WWW.IMDB.COM
  3. 3. 3
  4. 4. 4
  5. 5. 5
  6. 6. NuGet Landscape WHAT MAKES UP THE NUGET PROJECT 6
  7. 7. NuGet Project Open Source (owned by Outercurve Foundation) ◦ Releases shipped by Microsoft Included in all Visual Studio installations Millions of users (some knowingly, some not) around the world Dozens of high-profile dependent projects at Microsoft International contributors ◦ Only a few external contributors are in the US Very large backlog of issues Conflicting requirements Backward compatibility Mixture of hard and soft deadlines Extremely costly test passes 7
  8. 8. Outercurve and Microsoft NUGET CLIENT Outercurve’s NuGet Project Microsoft’s NuGet-Based Package Manager In the Box with Visual Studio NUGET GALLERY Outercurve’s NuGet Gallery Project www.nuget.org Azure Hosting provided by Microsoft Operated by Microsoft on behalf of Outercurve 8
  9. 9. NuGet’s Core Team NUGET CLIENT Visual Studio Extension WebMatrix Extension NuGet.exe Command-Line NuGet Packages 3 Developers 2 Testers NUGET GALLERY www.nuget.org staging.nuget.org docs.nuget.org build.nuget.org 3 Developers 2 Testers Always 1 member on ops/support rotation 9
  10. 10. NuGet’s Core Team NUGET CLIENT Visual Studio Extension WebMatrix Extension NuGet.exe Command-Line NuGet Packages 3 Developers 2 Testers NUGET GALLERY www.nuget.org staging.nuget.org docs.nuget.org build.nuget.org 3 Developers 2 Testers Always 1 member on ops/support rotation 10
  11. 11. Release Management NUGET CLIENT 11
  12. 12. NuGet Releases New release every 10 weeks since NuGet 1.5 12 2/ 12/ 2011 1.1 3/ 30/ 2011 1.2 4/ 25/ 2011 1.3 6/ 17/ 2011 1.4 8/ 30/ 2011 1.5 12/ 13/ 2011 1.6 4/ 4/ 2012 1.7 5/ 23/ 2012 1.8 6/ 19/ 2012 2.0 10/ 4/ 2012 2.1 12/ 12/ 2012 2.2 2/ 15/ 2013 2.2.1 4/ 25/ 2013 2.5 6/ 26/ 2013 2.6 8/ 22/ 2013 2.7 10/ 7/ 2013 2.7.1 1/ 14/ 2011 1.0
  13. 13. NuGet in Visual Studio Releases 13 2/ 12/ 2011 1.1 3/ 30/ 2011 1.2 4/ 25/ 2011 1.3 6/ 17/ 2011 1.4 8/ 30/ 2011 1.5 12/ 13/ 2011 1.6 Introduced with ASP.NET MVC3 and WebPages v1 4/ 4/ 2012 1.7 5/ 23/ 2012 1.8 6/ 19/ 2012 2.0 10/ 4/ 2012 2.1 12/ 12/ 2012 2.2 2/ 15/ 2013 2.2.1 4/ 25/ 2013 2.5 6/ 26/ 2013 2.6 8/ 22/ 2012 Visual Studio 2012 RTM 4/ 4/ 2013 Visual Studio 2012 Update 2 6/ 26/ 2013 Visual Studio 2013 Preview 9/ 9/ 2013 Visual Studio 2013 RC 1/ 14/ 2011 1.0 5/ 31/ 2012 Visual Studio 2012 RC 2/ 28/ 2012 Visual Studio 2012 Beta 8/ 22/ 2013 2.7 10/ 7/ 2013 2.7.1 Visual Studio 2013 RTM
  14. 14. Release Tiers CI build from Outercurve Foundation always available at http://build.nuget.org ◦ Usually 100 or fewer users ◦ Build created from every commit Beta/RC builds from Microsoft occasionally available at http://nuget.codeplex.com ◦ 1,000 – 2,500 users ◦ Published after we hit code-complete ◦ Real-world testing of newly introduced features or significant design changes Final builds from Microsoft published to the Visual Studio Extension Gallery ◦ 500,000 – 750,000 users ◦ Released after our test team signs off, a few weeks before Visual Studio locks down for its next release ◦ Last stage real-world testing from users that are knowingly using NuGet and are a bit more forgiving In the box with Visual Studio ◦ Millions of users ◦ Many of these users don’t know what NuGet is and will file Visual Studio bugs if we break them 14
  15. 15. Backlog CodePlex: 858 open issues as of 10/4/2013 Microsoft Internal Issues ◦ TFS: Fluctuates from 0 to 25 ◦ Dr. Watson reports ◦ Microsoft Connect ◦ Microsoft shipping requirements ◦ Future Visual Studio release issues Strategic Product Features ◦ Schedule alignment for release within Visual Studio and Visual Studio Update 15
  16. 16. Strategic Product Features Package Manager for the Microsoft Development Platform ◦ Not just ASP.NET ◦ Not just .NET ◦ Includes C++, Windows Store, and Windows Phone ◦ Cross-platform support with Command-Line, Xamarin Studio, MonoDevelop, and SharpDevelop Improving Package Discovery (through collaboration with the NuGet Gallery crew) Avoiding Package Version Hell Platform Multi-Targeting and Re-Targeting New Usage Workflows for Improved Application Maintenance and Ecosystem Growth Helping Improve Gallery Reliability 16
  17. 17. Triaging Issues 1. Is it a high-priority issue (crash, hang, regression)? ◦ Include it in the current release 2. Is the bug fix or feature needed for the current release? ◦ Include it in the current release 3. Is it a high value item that fits in the next release? ◦ Include it in the next release 4. Do we agree with the concept? ◦ Put it in the “Up For Grabs” release ◦ Ideally picked up by the community ◦ High-voted items are reviewed when planning releases and some items are pulled in 5. Do we disagree with the concept? ◦ Close it 17
  18. 18. Priorities 1. Strategic roadmap that integrates with the Microsoft development platform ◦ Features planned for 2-3 releases at a time ◦ Align with Visual Studio releases: Preview, Beta, RC, RTM, Visual Studio Update 2. Community-reported high-value bug fixes or features ◦ Including those that come with external pull requests 3. Highly-voted community feature requests 4. Community-reported high-value bug fixes 5. Microsoft partner team bug reports 6. Microsoft partner team feature requests 7. Community-reported low-value bug fixes 18
  19. 19. Philosophy Small and easy bug fixes are low priority ◦ Otherwise, we’d spend 100% of our time making small changes that affect only a handful of users Don’t prematurely act upon cool ideas ◦ Does the idea spark new strategic concepts? ◦ Do other new ideas relate? ◦ Can the feature be implemented with backwards compatibility? ◦ Could this feature become obsolete based on other potential strategic directions? Favor community-requested features over partner team features ◦ Avoids scenario-specific features that conflict with broad usage ◦ Encourages partner teams to find workarounds with existing features ◦ Those workarounds often surface pain points others might also be experiencing We’d rather delight 1,000,000 external users than gain 1 more internal partner team 19
  20. 20. Conflicting Requirements Microsoft’s Customers vs. NuGet’s Ecosystem ◦ Group Policy control over package sources Bold New Features vs. Backwards Compatibility ◦ Package Restore ◦ Build-time reference resolution NuGet User Experience vs. Microsoft’s shipping requirements ◦ Online consent ◦ Loading only Microsoft-signed assemblies ◦ Modified content files Favor user desires as much as possible Work within the constraints of Microsoft’s shipping requirements Negotiate and compromise with privacy, security, and legal representatives to arrive at acceptable implementations 20
  21. 21. Source Control NUGET CLIENT 21
  22. 22. master merge Feature A Feature B Branch: 2.7 Branch: 2.7.1 mergemerge Branch: 2.8 Tag: 2.7 Tag: 2.7.1 Git Branch per feature Branch per version Tag for release 22
  23. 23. Continuous Integration Outercurve Builds ◦ TeamCity – running on Virtual Machines in Windows Azure ◦ 1 VM build agent with .NET 4.0 for building the VS Extension for VS 2010 and VS 2012 ◦ 1 VM build agent with .NET 4.5 for building the VS Extension for VS 2013 ◦ Hosted at http://build.nuget.org ◦ Produces non-localized, unsigned builds with Outercurve branding and licensing ◦ “Developer Branches” build configuration builds all “dev-” prefixed branches Microsoft Builds ◦ TeamCity – running on Virtual Machines in a group lab ◦ 1 VM build agent with .NET 4.0 for building the VS Extension for VS 2010 and VS 2012 ◦ 1 VM build agent with .NET 4.5 for building the VS Extension for VS 2013 ◦ Only accessible on the corporate network ◦ Produces localized, signed builds with Microsoft branding and licensing 23
  24. 24. Going Dark Microsoft-confidential work cannot go to CodePlex ◦ Integration with new versions of Visual Studio ◦ Support for new platforms (e.g. Windows Phone 8) Team Foundation Service private project ◦ Fork the NuGet Git repo into that project ◦ Move confidential development into that fork ◦ Keep all other development on CodePlex Merge commits into TFS daily Configure our Internal Builds to use the TFS fork instead of CodePlex 24
  25. 25. Testing NUGET CLIENT 25
  26. 26. Release Timing Release sign-off takes about 4 weeks after code-complete ◦ 40% of our 10-week release cadence is spent testing and fixing bugs Snap releases to the Visual Studio schedule Release to the gallery first, and then integrate into Visual Studio’s build Between Visual Studio Beta and RTM, we have 2 or 3 tight quality-driven releases ◦ Meet Microsoft’s release requirements ◦ Address any partner team blocking issues ◦ Focus on extremely broad use of the product After Visual Studio hits RTM, we go big with features and aim for a 3-month release 26
  27. 27. Test Matrix Dimensions Visual Studio SKUS ◦ VS 2010 ◦ Express for Web ◦ Express for Phone ◦ Pro+ ◦ VS 2012+ - All SKUs Project Types ◦ Virtually all Microsoft Project Types ◦ Lightswitch ◦ Windows Store ◦ Windows Phone ◦ Portable Class Libraries Languages: C#, VB, F#, JavaScript Operating Systems ◦ Windows XP ◦ Windows Server 2003+ ◦ Windows Vista and Windows 7 ◦ Windows 8 ◦ Windows 8.1 Different types of NuGet Packages Different types of NuGet Package Sources ◦ Gallery ◦ NuGet.Server ◦ Local Disk ◦ UNC Share Source Control Integration ◦ TFS ◦ Git, Mercurial ◦ None  NuGet Clients ◦ VS Extension ◦ WebMatrix Extension ◦ NuGet.exe Command-Line Plus, all of the combinations of actual features 27
  28. 28. Achieving Release Sign-Off Unit Tests managed and run by developers End-to-end Functional tests managed and run by developers and through automation ◦ Based on a PowerShell test system included within our project ◦ Runs within Visual Studio, automating VS through the DTE QA Tests for interactive testing ◦ Mostly automated, but exploratory testing always finds more bugs ◦ Run across a farm of machines in our group’s test lab, covering the machine configuration matrix ◦ A representative set of combinations give us confidence ◦ Uses an internal VS automation library to simulate user actions in VS Group-wide bug bashes Company-wide partner team bug bashes Public Beta testing from CodePlex builds Reach 0 bugs in CodePlex for the current release 28
  29. 29. Throughput 109 Weeks • Total time working on those 10 releases 10 Releases • 816 work items have been completed 3 Developers • Averaging 27 work items per release 2 Testers • Each testing 3.7 bug fixes or features per week 29
  30. 30. Success Never defined by throughput No regressions No minor updates necessary Users delighted by new features Took steps toward our strategic roadmap Addressed highly-voted community bugs and features Accepted a level of failure in smaller user base releases 30
  31. 31. Success Rate NuGet 1.6 ◦ Required a 1.6.1 update ◦ Regressions ◦ Missed bugs in new features NuGet 1.7 ◦ Success NuGet 1.8 ◦ Success NuGet 2.0 ◦ Success NuGet 2.1 ◦ Success NuGet 2.2 ◦ Success NuGet 2.5 ◦ Success NuGet 2.6 ◦ Success NuGet 2.7 ◦ Required a 2.7.1 update ◦ Regression ◦ Missed bugs in new features 31
  32. 32. Tiered Releases are Key Our tiered release approach worked for 2.7 2.7.1 update in place in time for integrating into Visual Studio Thousands of users affected instead of millions As Visual Studio’s broad user base jumps from NuGet 2.2 to NuGet 2.7, they’ll have no idea we ever introduced regressions You wouldn’t have noticed the camera in Airborne if I hadn’t pointed it out to you 32
  33. 33. Q & A Get a NuGet Sticker! Twitter: @jeffhandley Website: jeffhandley.com 33
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×