Tools used for different purposes—working with requirements, specifying architecture, developing code, testing code, and project management—should all be able to work with a common set of interconnected artifacts stored in a common place. As Figure 2 shows, those artifacts can include requirements, design documents, various versions of source code, test cases, statistics about this development project, and more. This kind of integration allows all sorts of useful things: automatically recording code check-ins, associating tests with source code versions, generating historical reports of bug counts, and more. More important, it allows optimizing the entire process. The flow of work is now apparent, and so making that flow as smooth as possible gets easier.When our industry converges on an architecture, it means there’s broad consensus about the best way to do something.For the most part, the first generation of team development tools didn’t take this approach. The vendors have learned from their experience, however, and team-based tools today can provide this broad integration. Microsoft’s Visual Studio Team System, for example, includes specialized tools for architecture, development, testing, and more, all of which rely on Team Foundation Server (TFS) to store the artifacts they work with.
Step 1 – clean up the development process, introduce as many agile concepts as possible. Increases team communications, team capability, project repeatability and lowers costs.
Step 2 - Improve quality at the beginning, not during or after. Build in quality. You may have success now, and it might work on smaller projects, but as projects grow and become more complex, or change and have additions over time – then tests become crucial to code quality.
3 – Handle requirements in a clever way - at the same time reduce scope creep, and create living documentation. BDD tests in “Cucumber” language (Given / When / Then) format mean that you reduce the chances of miscommunicating a requirement – Cucumber is a DSL – a Domain Specific Language – one that works between users and developers. We use StoryQ internally (see http://storyq.codeplex.com ) though there are many others.
3. As we primarily produce software to run under windows, our developers use GitExtensions http://code.google.com/p/gitextensions/We simply use shared folders and ordinary file access to repositories, rather than implement MSysGit or other more complete full git:// implementations. This works well for us, and means we don’t need to fiddle with keys, Cygwin or Linux.
Image source: http://bit.ly/9bUvdC
Real life: do things manually and they fall off, automate as much as possible … leads into the next slide
Many options, we’ve settled for TeamCity from Jetbrains (makers of Resharper). See: http://www.jetbrains.com/teamcity/Alternatives CruiseControl, TFS, Hudson/Jenkins etc. For a good list to start from, go to: http://en.wikipedia.org/wiki/Continuous_Integration
For more info see: http://www.jetbrains.com/teamcity/features/index.html
XP guideline of a 10 minute build is reasonable and achievable for most projects. It s worth the effort to streamline and speed up the build – affects developers every time they commit.Provide continuous visibility and feedback: Targeted feedback. Send the right information to the right people at the right time in the right way.
Frequent broken builds suggest developers are not running private builds. Development culture and “people issues” are often more difficult to solve than technical issues. But you need to keep the build (relatively) fast so that developers are willing to run private builds. May require trade-off: allow for occasional broken builds in order to avoid expecting developers to run time-consuming or cumbersome build processes.
Reduced risk; you eliminate the blind spot of deferred integration Makes bugs dramatically easier to find and remove; If you introduce a bug and detect it quickly it's far easier to fix. Plus, bugs are also cumulative. The more bugs you have, the harder it is to remove each one.Generate deployable software at any time – Removes one of the biggest barriers to frequent deployment
Staged build(or build pipeline) – multiple builds done in sequence. First build is triggered by a commit – should be quick (smoke tests/unit tests). Second build runs slower tests/inspections – integration tests, etc.; takes much longer.Parallel builds – can also work well with staged builds. For instance, can split up the secondary build across multiple build processes or machines.Continuous database integration – Automate the database integration. Rebuild your database and test data every time you run a CI build. Database integration artifacts should be stored in the version control system: DDL, DML, config files, etc. Generate the database from build scripts. Automated scripts for database creation, configuration, population Apply continuous testing and continuous inspection against your database(s)
Martin Fowler: When I've described this practice to people, I commonly find two reactions: "it can't work (here)" and "doing it won't make much difference".
Que nos espera a los ALM Dudes para el 2013?
¿ALM en el 2013?Bruno Capuano Innovation Manager @elbruno
Bruno Capuano Innovation Manager @elbrunohttp://www.elbruno.co m
Typical ALM challenges “We don’t have good visibility into project status” “Our teams are not communicating effectively” “Requirements are not sufficiently defined or tracked” “We need lightweight, agile development processes” “Software is not adequately tested” “Cost of maintaining and operating the solution exceeds the business benefit”
THE WAY WE WORK Must change … ;) And also our tolos … ;)
ALM Tools … must have DevelopmentArchitecture Tool Test Tool Tool Design Documents Source Test Cases Code Requirements Versions Project Stats ProjectRequirements Management Tool Tool Shared Server
Agile, Scrum, Kanban, … etc• If you aren’t using some form of agile development, see what you can use from Scrum, Kanban or XP• Projects become more predictable and repeatable• Project costs will be lowered• Teams will grow in capability• Teams become more responsive to change• Scope is managed• Customers benefit, being delivered a product that is closer to their needs
TFD: Test First DevelopmentTDD:Test Driven Development• Proven to create better solution designs, with less bad coupling = more maintainable code• Provides safeguards when refactoring• 10% more effort now (with TFD/TDD) or 50% more effort later (without TDD)• A suite of tests built up during the life of the project provide valuable feedback when performance testing• Tests may replace most traditional documentation, and provide “living” documentation beside the code
ATDD and BDD• Acceptance Test Driven Development (ATDD) or Behaviour Driven Development (BDD) will sift out the poorly spec’d out user stories, allowing less scope shift and faster development• BDD tests can be automated providing fast response to the team and to users, faster turnaround of work• BDD tests create a common language that everyone can “speak” – users, developers, testers, managers• BDD tests provide living project documentation• BDD can be applied to smaller or larger units of work
Source Control/Version Control• Must be fast and reliable• Must be simple to use• Handy if it is also inexpensive• Wehave adopted GIT – using local repositories as well as a central repository• Git has offline capabilities and works well with peer-to-peer push/pull• eg., desktop local repo <-> USB stick repo <-> laptop
GITCan your source control do this?If not, get Git.Better yet, use GIT with TFS2012.
The Friction:• If done manually: - manual testing gets boring and quality drops off - manual builds can become too much work - source control: varying quality code, 5pm code oops’s - synchronising code libraries: everyone for themselves• Developers cannot easily see how to change if they are too busy fighting fires• Sometimes too close to issues, cannot see that change is needed, or that change is even possible• Managers push for the quick result, usually less tests
The Fix: Continuous Integration• Automate everything you can: - testing, builds, and deployment where possible• Manyways to do this from DIY, open source, to commercial packages.• Any improvement will help• Find the easiest or cheapest, and start there• We have adopted JetBrains TeamCity (free version)• TeamCity has 3 models: free, paid, enterprise• Supports many different build, test and deploy configurations for Java, and .Net
TeamCity integration• Integration with just about everything:• CVS,Subversion, ClearCase, GIT, Perforce, TFS, Mercurial, S ourceSafe, StarTeam …• Ant, Maven 2, IntelliJ IDEA, Nant, MSBuild …• Email, RSS, Jabber, IDE, Windows Tray …• Eclipse, IntelliJ IDEA, Visual Studio …• Junit, testNG, EMMA, Cobertura …• Rake, NuGet, Powershell, Command Line, FxCop, Gradle …
Broken code doesn’t make it intosource control any more
Local Version Control • Old-school version control like RCSImage stolen from http://progit.org/book/ which is really good, go read it.
Centralized Version Control • This is how Subversion worksImage stolen from http://progit.org/book/ which is really good, go read it.
Distributed Version Control • This is how Git worksImage stolen from http://progit.org/book/ which is really good, go read it.
Why Do I Care?• Because it has been mandated that you care about version control by Prof. Laurendeau• Because you will use version control at every single coding job you get in the real world • If your company doesn’t have version control in place, that’s a huge warning sign• Because it will change the way you think about programming, for the better• Because it stops you from completely screwing yourselves the day before an iteration is due
Installing Git• Let’s install Git and try some things out.• Linux: use your favourite package manager • Ubuntu: aptitude install git• Use the MSysGit installer • http://code.google.com/p/msysgit/
Learning Git • Read Pro Git, it’s excellent and free • http://progit.org/book/ • Git is extraordinarily powerful and you’ll be a better programmer if you take the time to understand it • If in doubt, always use the built in manual • Every git command has a --help flag • e.g. git cherry-pick --helpSeriously go read http://progit.org/book/.
Getting Started• Let’s tell Git our name and email address • It attaches these to any commits we make so people know who to kill for breaking the build an hour before an iteration is due• git config --global user.name “Your Name”• git config --global user.email “email@example.com” • --global sets the variable for your OS user • --system sets the variable for all users on your machine • Omit the --global and --system flags to set a git config variable for just this repository
Creating a Repository• git init • Creates a local, empty git repository in the folder where the command is run • Creates a .git folder for the guts of the repository • Only one .git folder, at the root of your repository • This is way nicer than creating a new .svn folder for every single subfolder in your repository• We’ll go over how to work with other people on a remote repository soon
Anatomy of a RepositoryImage stolen from http://progit.org/book/ which is really good, go read it.
Anatomy of a Repository• Crazy important! Memorize this! • The working directory consists of the actual files and folders on your machine • The staging area lets you build commits out of snapshots taken of the files and folders in your working directory • The repository maintains a collection and complete hierarchical history of all commits
Oh God, What’s Happening? • git status • Shows you the state of your staging area and of your working directory • git log • Shows a commit history of your repository • git diff • Shows changes between commits, your working directory, the staging area, etc. • git diff --help to learn moreOr you could just go read http://progit.org/book/ and you’ll be an expert.
Adding & Removing Files• git add file • Adds a snapshot of file to your staging area • You can change file and the snapshot will remain as it is• git rm file • Removes the file snapshot from your staging area and deletes file from your working directory• git rm --cached file • Removes the file snapshot from your staging area while keeping file in your working directory intact
.gitignore• Tells Git which files to ignore outright• Uses glob syntax for pattern matching • There’s a decent summary of glob syntax at linux.about.com/library/cmd/blcmdl7_glob.htm• Git adds re-inclusion rules with !pattern • “Include any files matched by pattern, even if they’ve been ignored by a previous pattern.”• There’s a sample .gitignore on the website • Stack Overflow is a good source of .gitignore files
Making Commits • git commit -m “your message” • Creates a commit from the contents of your staging area and adds it to the repository • -m “your message” sets the commit message • Ifwe keep adding commits we get a linked list that represents the history of our repository • gitk --all gives a graphical history of all branches • Leaving out the --all shows just this branchAnd the book at http://progit.org/book/ has a whole bunch more information. Read the damn book!
Branching & Why It Is Awesome• Branching allows us to create a tree of commits instead of a linked list • Merging will let us turn this into a DAG• Easy,painless branching is the most important and powerful feature of Git • Give each new feature its own branch, which can be merged back into the main (master) branch after it’s been completed and is stable
Creating Branches• Gitmaintains a pointer to the current checked out branch, called HEAD• git branch newbranch • Creates a new branch starting at the current commit, but does not move the HEAD pointer• git checkout newbranch • Changes HEAD to point to newbranch• Rungit branch then git checkout to create and start working from a new branch • Shortcut: git checkout -b newbranch
Branching Example $ git branch testing $ git checkout testingImages again stolen from http://progit.org/book/ which you should read because it’s good.
Branching Example $ git commit $ git checkout masterImages stolen from http://progit.org/book/
Branching Example $ git commit • At this point, the branch history has diverged • We want to branch for new features and merge them back into the master branch • This makes your life infinitely easierImages once again stolen from http://progit.org/book/
Merge Tools• git config --global merge.tool toolname • Sets the merge tool for your OS user• A mergetool allows you to fix any conflicts that arise from merging two branches• Google will give you a list of merge tools • I use p4merge, it’s hard to go wrong just choosing a random merge tool and using it• git mergetool • Run this command if anything goes wrong with a merge, it’ll allow you to fix things
Merging• git merge branchname • Allows you to take another branch and merge its changes into the currently checked out branch• Git has hyper-intelligent algorithms that track your content, not your files • If you move or rename a file and make changes to it, it will still be detected as the same content
Merging ExampleImage stolen from http://progit.org/book/
Merging ExampleImage stolen from http://progit.org/book/
Branching & Merging in SVN• It’s relevant to see how the approach taken by Git differs from that of terrible legacy systems like SVN• Subversion doesn’t have real branches • A branch is just a copy of your repository in a named folder • SVN has no concept of branch history and therefore cannot determine common ancestry to help with merging• Subversion doesn’t have real merging • The svn merge command should be called svn diff-and-apply-patch because that’s all it does • “If a merge fails, run svn revert and do it by hand.” • There is no way to tell whether a given set of changes were the result of a merge or were just straight edits
Remote Repositories• git remote add reponame url • Adds an external Git repository called name• git fetch reponame • Fetches updated branches from reponame including all updated data • A remote branch shows up as reponame/branchname• Your local information about remote repositories isn’t updated automatically • You need to run git fetch periodically on your remotes to get new branch/commit information
Remote RepositoriesImage stolen from http://progit.org/book/
Pushing• git push reponame branchname • Adds a local branch to a remote Git repository • You need to have write access to the remote repository • Alternatively, merges your local branch into a remote branch of the same name• git push reponame localbranch:remotebranch • Explicit syntax for merging your changes from localbranch into remotebranch• Always fetch and merge before you push • Save yourself grief and error messages
Pulling• git pull reponame branchname • Fetches from reponame then merges the remote branchname into your local branchname• git pull reponame localbranch:remotebranch • Fetches from reponame then merges reponame/remotebranch into localbranch• Syntactic sugar for fetching and merging • Your workflow should be “git pull; git push”
Tracking Branches• git checkout -b branch remote/otherbranch • Gives you a tracking branch that you can work on, that starts where remote/otherbranch is• Trackingbranches are local branches that have a direct relationship to a remote branch • You can just call git push/pull with no arguments and it know which remote branch to change
Cloning a Repository• git clone url • Creates a new local git repository, creates new tracking branches for each branch in the cloned repository, creates and checks out an initial branch that is forked from the remote’s active branch• Watch out if the remote repository doesn’t have any commits yet (really common thing) • Cloning will fail, you need to do git init, git remote add origin url, create an initial commit with some content, then run git push origin master
CI – Project Practices Run tests and inspections as part of build • • All tests and inspections must pass • Build automatically upon code commit • Keep the build fast • Want fast feedback; minutes, not hours • Design builds to fail fast • Run tests/inspections that are most likely to fail first • Want to find out about failures as soon as possible • Provide continuous visibility and feedback • Project status dashboard • Continuous feedback devices
CI - Developer Practices• Commit code frequently (at least daily) • Reduces magnitude of changes to be integrated with each build • Reduces likelihood of stale (out-of-sync) code changes • Reduces effort of troubleshooting failures – less has changed• Run private builds before check-in • Execute same set of tests and inspections that CI will • IDE may help here (e.g. PMD or Checkstyle plugin)• Fix broken builds immediately• Don’t commit broken code • Broken code should fail the private build
CI - Benefits• Reduced risk• Reduced manual, repetitive processes• Easier debugging / troubleshooting• Generate deployable software at any time• Better project visibility• Greater confidence in the software
Continuous Database Integration• Rebuild your database and test data for every build.• Store database artifacts in version control • DDL, DML, config files, etc.• Generate the database from build scripts • Scripts for database creation, configuration, population• Apply continuous testing / inspection against your database(s)• LiquiBase, Rails Migrations
Continuous Integration vs Build Management“Build Management, essentially, aims to produce a set of artifacts that correspond to a specified snapshot of source code, and to be able to hand off those artifacts to third parties outside of development”• UrbanCode - http://www.cmcrossroads.com/pdf/DrawingtheLine.pdf
CI – BM Comparisons Continuous Integration Build ManagementWhat is the purpose? To integrate the latest To provide a definitive set changes and determine of usable artifacts based on current code quality a given code slice in timeWho is the audience? Internal External (non-developer)What are the products? Pass/Fail, Metrics, Code Deployable artifact qualityWhat is the code base? The latest changes Fixed time referenceWho benefits the most? Developers Infrastructure / Support
Questions to Ponder• Who should own the governance for CI? How about for build management?• Shouldsimilar, different, or exactly the same build processes be used for CI and BM?• Whatis needed to verify the quality of the artifact generated by the BM process?
CI Developer Anti-Patterns• Infrequent check-ins• Premature check-ins (failing code)• Ignoring broken builds• Turning off tests to correct builds• Waiting until end of day to commit changes• Inconsistent developer build environments (works on my machine)
CI Tooling Anti-Patterns• Insufficient feedback• Too much (spam) feedback• Underpowered (slow) build machine• Bloated build• Weak builds that never fail (Continuous Ignorance)• UsingScheduled Builds versus building software with every code change• Failing to clean out old artifacts before each buildhttp://www.ibm.com/developerworks/java/library/j-ap11297/index.html
CI - Challenges“CI is not just a technical implementation; it is also an organizational and cultural implementation” – Paul Duvall, Continuous Integration
Challenges – Cultural • Development culture • Not familiar with continuous integration • Ignore failing tests or broken builds • Not checking in code frequently (daily) • Not running tests before check-in • Lack of encouragement for CI practices
Challenges –Personnel/Experience • Lack of expertise • Continuous Integration • CruiseControl / Hudson • Deployment (e.g., WebSphere) • No dedicated resources for CI (anymore) • Made great progress when we did have that.
Challenges – Infrastructure• Obtaining system accounts, disk space, etc. to run CI• Insufficient and constrained environments • Not enough, not always available • Test environments, app servers, databases, etc.
Challenges – TechnicalImplementation• Stuck on outdated technologies • Java 1.4, Tru64, PVCS, VB, VisualC++• Technology stacks • Grails, Java, Struts, etc – all with differing folder structures• Database changes in separate source repo • Database changes in PVCS; main code in CVS• Disparate systems for CI integration • CVS, PVCS, PICCT, QualityCenter, VersionOne, Fortify• Cruise Control / Hudson – learning curve
Challenges – Running Builds• Brittle tests (e.g., data-dependency)• Slow tests or lots of time-consuming tests• Build “noise” • CI build fails for reasons outside developer control (e.g., bug in Hudson or file access issues)• Interactions between builds • “Access denied” and “Cannot delete …” errors on Hudson • Shortcomings of Windows build server
NUGETGestionando paquetes como unos campeones en Visual Studio
What is NuGet?• .NET Package Manager• Original OSS Project Nu now NuGet• Ships with ASP.NET MVC3• OS Managed by Outercurve Foundation hosted on nuget.codeplex.com• 32,000+ Packages• Current version 1.6• Extendable and is a protocol
What is Package Management?• Tools to simplify discovering, obtaining, installing and updating project references.• Eliminate dependency hell.• Similar to Ruby Gems and OpenWrap – http://openwrap.org• Not like a system package manager like, Linux apt-get.• Windows System Package Managers • Chocolatey – http://chocolatey.org • CoApp – http://coapp.org
What is the goal of NuGet?• Promotion of projects and libraries to ensure great software is readily available and usable.• Eliminate discovering, obtaining, and complex configuration as a barrier to entry for leveraging powerful software.• Make Open Source Software ( OSS ) easy as “Add Reference”.• Eliminate dependency hell.• Improve developer experience and productivity.• Leverage Conventions over configuration• Tight integration with Visual Studio IDE and leverage power of PowerShell.
How do I get started?• Recommend installing NuGet Visual Studio Extension, NuGet Command Line and NuGet Explorer http://nuget.codeplex.com• Visit http://nuget.org• Do a one over the NuGet Documentation - http://docs.nuget.org/• Create a new VS 2010 solution or open an existing solution then hack away on the Open Package Manager Console consuming packages• Start small, replacing a few references with NuGet Package References.• Watch Scott Hansleman’s screen casts on NuGet for in depth knowledge ( references section )
NuGet Package Gallery• Official NuGet Package Gallery - http://nuget.org• Provides official community package library.• Publish Packages with a FREE account• Main NuGet Feed for consuming packages• Package Stats – http://stats.nuget.org• NuGet Team Blog - http://blog.nuget.org/• 32,000+ packages
What makes up a NuGetPackage?• Meta Data• Dependencies• Content • Transforms • Templates• Libs • dlls• Tools • Init.ps1 • Install.ps1 • Uninstall.ps1
Web Start-Up Code• Simple way to enable start-up of your library in Web Apps• WebActivator NuGet library• Replacements of the .NET 4 attributes • [PreApplicationStartMethod] • [PostApplicationStartMethod]• Content • App_Start• Examples • IoC libraries like Ninject and StructureMap • Commmon • $rootnamespace$
Creating NuGet Packages?• Create Manifest • Manual • Command line • Assembly • Visual Studio Project• Create Packages • Manually using command line • NuGet Explorer • Visual Studio Project • Package Manager Console using New-Package • TeamCity
Package Source• NuGet Visual Studio Extension supports multiple Package Sources in hierarchy order.• NuGet Source Options • Nuget.org • Own Local or Network Directories • Local Cache • Own Read Only Server • Own NuGet Gallery • Own NuGet Orchard Gallery• .NET Community Source Options • MyGet.org • NuGetFeed.org • TeamCity
Why host internal NuGetGallery?• Open Source - https://github.com/NuGet/NuGetGallery• Provides full ownership • Publish proprietary internal packages • Publish backups of external packages • Browse able gallery • Manage internal/external packages • Extend or integrate with own requirements• Extending NuGet Gallery • Baseline Feed • Client/Project Feeds • Community Feeds • Enterprise Feeds • Personal Feeds
What about ContinuousIntegration?• No Commit Strategy • Tracking Package Metadata • Package Restoration• TeamCity Pull, Package and Publish Support• Automated Deployment with Octopus • Convention Based • Shipping Application in NuGet Package • Deploying Applications to servers
What is the value to the team,.NET COP, enterprise, client?• Reduce project dependency management. Simplify setup and onboarding. Improving version management.• Promote reuse, standardization, discovery, collaboration regarding packages across all levels of the organization.• Removal of dependencies from VCS saving valuable disk space.• Improves continuous integration process and opens the opportunity or automated deployments with products like Octopus.• Offers extension points with Chocolatey to improve developer experience regarding machine setup and overall maintenance.• Integration with commercial vendor NuGet feeds like DevExpress
Referencias• Download and tutorials: • Official GIT site http://msysgit.github.com/ • Git for Windows http://code.google.com/p/msysgit/downloads/list?q=full+installer+official+git • GitHub for Windows http://windows.github.com/ • JetBrains - Team City http://www.jetbrains.com/teamcity/ • Download TeamCity http://www.jetbrains.com/teamcity/download/index.html• Presentations: • Git from Erick Lawless http://bit.ly/VOJgXo • NuGet – FrozenBlocks http://bit.ly/ZfZBfk • Scaling up and bottoming up http://bit.ly/V2sFOz
Bruno Capuano Innovation Manager @elbrunohttp://www.elbruno.co m