Productive parallel teamwork: Decentralized Version Control Systems

images/logo
Developing, maintaining, and sharing software tools for research
Productive parallel teamwork: Decentralized Version Control Systems
Danilo Pianini
danilo.pianini@unibo.it
Alma Mater Studiorum—Universit`a di Bologna
Ph.D. course in Data Science and Computation
May 21, 2018 - Bologna (Italy)
D. Pianini (UniBo) 02 - Teamwork May 21, 2018 1 / 62

images/logo
Outline
1 Versioning
Motivation
git as versioning tool
Terminology and basic concepts
Managing a repository
2 Sharing
Managing multiple instances of the same repository
Public hosting for git repositories
A transparent development process
3 A successful branching model
with a shared repository
with forks and pull requests

images/logo
Versioning Motivation
Outline
1 Versioning
Motivation
2 Sharing

images/logo
Keep track of changes
Have you ever had the need to:
roll back some project to a working version
roll back some subpart of a project to an older version
compare with the previous state and see the changes
Many had

images/logo
Do it yourself
Basic strategy everybody applied at least once (be honest...)
1 work on project
2 if status is kinda ok, copy everything in a folder
3 optionally, put a version or a date to the folder name
4 optionally, compress the folder
5 go to 1
Why is bad
Expensive in time
Snapshotting costs a lot in space (tons of duplication)
Difficult to compare with previous
Very hard to roll back different parts at different versions
Remembering differences between versions is impossible over time

images/logo
Let a tool do it
Desiderata
Diﬀerential tracking: only store diﬀerences, not copies
History navigation: ability to roll back in time
Partial roll backs: ability to roll back only sub-part of the project
Annotation of versions with messages
Annotation of versions with numbers or names
Automatic generation of dates
Author tracking
Possibility to select the parts of a project to save
You are only interested in non-regenerable resources
They often are only a part of a project

images/logo
Versioning git as versioning tool
Outline
1 Versioning
Motivation
2 Sharing

images/logo
Bits of history
In April 2005, BitKeeper, the SCM Linux was developed with,
withdrawn the free (as in beer) use
No other SCM met the requirements of Torvalds
Performance was the real issue with such a code base
Torvalds decided to write his own
The project was successful, and Torvalds appointed maintenance to
Hamano
Why the name
I’m an egotistical bastard, and I name all my projects after myself.
First ’Linux’, now ’git’. a
— Linus Torvalds
a
From the project Wiki. “git” is slang for “pig headed, think they are always correct, argumentative”

images/logo
The git README.md ﬁle
GIT - the stupid content tracker
"git" can mean anything, depending on your mood.
- random three-letter combination that is pronounceable, and not
actually used by any common UNIX command. The fact that it is a
mispronounciation of "get" may or may not be relevant.
- stupid. contemptible and despicable. simple. Take your pick from the
dictionary of slang.
- "global information tracker": you're in a good mood, and it actually
works for you. Angels sing, and a light suddenly fills the room.
- "goddamn idiotic truckload of sh*t": when it breaks

images/logo
Features
UNIX-oriented: tracks stuff like UNIX file permits.
Distributed development (the whole development history is copied
locally)
Diff-based history tracking
Implicit file naming (history preserved on renames)
Very strong support to non-linear development
Written in C
Approximately 10 times faster than Mercurial, 100 times faster than
other DVCS (e.g. Bazaar)
Uses existing protocols (ssh, http, ftp, rsync...)
Pluggable merge strategies (defaults to recursive three-ways merge or
octopus for 3+ heads)

images/logo
Popularity (data from: http://archive.is/bBNkZ) I

images/logo
Popularity (data from: http://archive.is/bBNkZ) II

images/logo
Popularity (data from: http://archive.is/bBNkZ) III

images/logo
Popularity (data from: http://archive.is/bBNkZ) IV

images/logo
Popularity (data from: http://archive.is/bBNkZ) V

images/logo
Versioning Terminology and basic concepts
Outline
1 Versioning
Motivation
2 Sharing

images/logo
Repository
Folder containing all the files tracked by git. Includes:
Set of files that are or have been tracked
Metadata related to each file
permits
owner
date changed
Information required to retrieve any previous version of any file ever
tracked
The whole history of save points
All the metadata are stored in a .git hidden subfolder

images/logo
Differential tracking
git tracks differences, not snapshots
the official manual says the opposite
Every save point can be seen as the set of differences between the
previous save and the current state
Every state of the project is conceptually a sequence of differences
that, applied in order, transform an initially empty repository to the
desired state
Efficient for text files
Less efficient for binaries and compressed files
if a binary resource is changed, is usually necessary to duplicate it,
falling back to tacking snapshots

images/logo
Commit
Every “save point” is named “commit”. A commit includes:
A reference to a parent (previous) commit
If it’s the first commit, the parent is an empty repository
The set of differences that transform the status of the world at the
parent commit into the current state
The author of the commit
The email of the author
The date of the commit
A unique hash code
A message describing differences
Optionally, a symbolic name and a second message (tag)

images/logo
Staging area
Changes in queue for the next commit
We don’t always want to save all the changes
Often, we work on multiple ﬁles, but only want to commit a subset
The creation of a commit counts two phases:
1 Staging – Selection of ﬁles whose changes will be saved
2 Commit – Actual creation of the save point
Files can be added and removed from the stage before committing
At commit time, only the staged changes will be saved

images/logo
Branch
Diverging development line
0
1 2 3
4 5 6
Going to an older commit and progressing from there creates a new
development line
Such development line is called branch
Branches can be worked on in parallel

images/logo
Merge
Converging development lines
0
1 2 3
4 5 6
7
Branches can be reunited with a merge operation
Changes from both the development lines are merged together
Delicate operation, merge conﬂicts can arise
Historically a horrible mess
git and Mercurial made branching and merging easy enough to let
them into the usual operations performed during development

images/logo
Head
A reference to the commit we are currently working on
It is tipically at the end of a branch
When we are working on an intermediate commit (e.g. because we
might want to start a branch) the status is called “detached head”

images/logo
Versioning Managing a repository
Outline
1 Versioning
Motivation
2 Sharing

images/logo
Global config I
User and email configuration
The author and email can be configured globally, so git won’t ask for them
at every commit
git config --global user.name "YOUR NAME"
git config --global user.email "your.email@provider"
Newlines
Newline behavior is critical if your team uses different operating systems
git tries to be smart and auto-convert newlines, making things worse
disable auto CRLF and configure your IDE appropriately
git config --global core.autocrlf false

images/logo
Global conﬁg II
Preferred editor
Git may ask for input for some operations, set your favorite editor
Defaults on vim
if you don’t know vim, you will probably have a hard time just for
quitting it
git config --global user.editor editorcommand
The editor must be installed and reachable with editorcommand
e.g. for nano
git config --global user.editor nano

images/logo
Repository initialization
git init
Marks the current folder as a new git repository
The folder can be non empty
A .git hidden folder is created, storing the metadata
Better not to ﬁddle with it
No recursive parent of the folder can be a git repository
Repositories can’t be nested
Bad mistakes
Be sure to be in the right folder before issuing the command!
Issuing the command in the wrong folder (e.g. your home) will mark
it as git repository, creating mayhem
To ﬁx the mistake, delete the .git folder and start over

images/logo
Repository status inspection I
git status
Prints information on the current status of the working directory and
staging area
Lists changes queued for the next commit
Lists ﬁles changed, added, removed but not yet added to the stage
Use it often!

images/logo
Repository status inspection II
Example
$ git status
On branch master
Your branch is up to date with 'origin/master'.
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file: 02-Teamwork.tex
Untracked files:
(use "git add <file>..." to include in what will be committed)
02-Teamwork.pdf
_minted-02-Teamwork/
img/

images/logo
Adding changes to the staging area I
git add
Adds the current changes of the files specified as argument to the staging
area
Files must be added explicitly, or they won’t get tracked
Wildcards (*) are allowed
Adding a folder adds all the contents recursively
Under Unix, also **
If a file is changed between an add and a commit, it must be
re-added to register changes

images/logo
Adding changes to the staging area II
What to track
It is paramount to select carefully what to add to the tracker
Track sources and non-regenerable resources
Don’t track regenerable ﬁles!
Compiled binaries
Generable documentation (javadoc, scaladoc, dokka...)
Generated reports (coverage, style...)
Generated charts (e.g. produced with Matlab or matplotlib)
Best practices
Don’t use wildcards if you are not super sure
Don’t add folders if you are not super sure
Always run git status before adding and committing

images/logo
Ignoring files I
The .gitignore file
It’s possible to specify a list of files to ignore
Plain text file, one entry per line
Each entry is a path relative to the .gitignore folder, in Unix format
Wildcards supported
Folders can be entered by terminating with /
Extremely useful: makes it possible to use add with wildcards or
folders reasonably safely
Keep using git status anyway
Multiple .gitignore files can be created
Each one affects the folder where it is and its contents, recursively

images/logo
Ignoring files II
Best practices
Immediately create your .gitignore file upon creating a repository
Even if initially empty
.gitignore must be tracked
One way to populate it is to proceed incrementally:
1 Execute git status
2 if there are files that are not tracked, decide, and either:
track them, or
add them to your .gitignore
Don’t let them “float” into your repo, ready to screw your work as
soon as you do something bad with wildcards or folders
There are some ready to use files online
I always had problems with them: populate your own manually,
customizing to your needs for the specific project

images/logo
Ignoring ﬁles III
Example:
../.gitignore
*.aux
*.log
*.nav
*.out
*.snm
*.gz
*.vrb
*.toc
**/_minted*/

images/logo
Removing changes from the staging area
git reset
Removes changes to the files specified as argument from the staging area
Does not change files on disk
Wildcards are allowed
Resetting a folder resets all the contents recursively

images/logo
Committing I
git commit
Creates a new commit containing the changes currently in the staging area
Date, author, email, and hash are automatically attached
If git have been appropriately conﬁgured
The text editor pops up and requires a commit message
alternatively, the -m "message" option allows for including the
message directly in the command
empty messages are not allowed
The -a option is an equivalent of git add * && git commit
Just don’t, please

images/logo
Committing II
Best practices
Commit often
It’s totally ok (actually recommended) to commit minimal changes
Just make sure there is consistency (e.g. the project compiles correctly)
Leave a clear, short commit message
If you want to add more detail, leave a short message and on a new
line describe better
Avoid generic messages such as “Fix” or “Commit”
The reader must get an immediate feel of what changed
Rather than committing widespread changes, try to break up your
current status in multiple, sequential commits

images/logo
Adding further metadata to commits
git tag tagname
Creates a symbolic name tagname for the current commit
Pops up the editor for including a message
alternatively, the -m "message" option allows for including the
message directly in the command
empty messages are not allowed
The commit can then be referenced directly as tagname
Best practices
Tipically used to assign a version to a certain state of the project
e.g. git tag 1.0 -m "First release"

images/logo
Visualizing history
git log --all --graph
Opens an interactive terminal window listing the project history
One entry per commit
Includes date author, email, message, hash, tag
--all visualizes all branches, --graph pictures development lines
We are going to start branching soon

images/logo
Visualizing diﬀerences
git diff
Visualizes diﬀerences line-by-line between the status of the staging area
and the last commit, or between the provided commits
Commits can be provided by their hash
a short part of the hash is ok if it’s enough to identify the commit
univocally
Commits can be provided by their tag name (if any)
Commits can be provided as relative to their current position of HEAD
git diff HEAD HEAD~2 compares the HEAD commit with the parent
of its parent

images/logo
Time travelling along branches
git checkout commitref
Moves HEAD to commitref, updating also the repository ﬁles
commitref can be
A (possibly short) commit hash
A valid tag name
Relative to previous HEAD position (e.g. HEAD~3
A valid branch name (moves to the last commit of the branch)
If the commit is not the last of its branch, the system enters
“detached head” mode
When in “detached head” committing is not allowed
(actually it’s just necessary to explicitly name a new branch)

images/logo
Branching, inspecting, and deleting branches
git checkout -b branchname
Creates a new branch branchname, originating from the current HEAD
Enables committing when in “detached head” mode
Because the HEAD is now attached to the new branch :)
The branch names are valid commit references
They point to the last commit on the branch
git branch
Lists the branches of the current repository
git branch -d branchname
Deletes branchname from current repository
If the branch contains changes (commits) that are not found in any
other branch, a warning is issued

images/logo
Merging
git merge branchname
Merges branchname into the current branch
All changes (commits) of branchname are included in the current
branch
Conflicts may arise, for instance:
if two developers changed the same file at the same line with two
different changes
if the same binary file is changed differently
if there are colliding renames

images/logo
Tackling merge conflicts
When a conflict is detected, a conflict file gets created
conflict.txt
<<<<<<< HEAD
Changes made on the branch that is being merged into. In most cases,
this is the branch that I have currently checked out (i.e. HEAD).
=======
Changes made on the branch that is being merged in.
>>>>>>> branchname
1 Remove the conflicting lines
2 Keep the lines you want to have in the merged version of the file
3 Add to the staging area, then commit
4 It’s ok to keep the suggested commit name (Merge X with Y)
In case of binary files, you must choose one version or the other.
Merge conflicts need manual intervention: the tool does not understand
the semantics of the files it’s managing!

images/logo
Again on the importance of tracking the right files
Putting the wrong files in tracking paves the way to a hell:
If you have binary regenerable resources (e.g. compiled files), they
will cause a merge conflict every time you have a merge
If you got many of them, you will have hundreds of conflicts
Conflicts on binaries get usually “solved” by deleting, regenerating,
and committing...
...leading to an explosion in the repository size
Takeaway
Only track non-regenerable resources
Think carefully before adding to the tracker
Configure your .gitignore soon and correctly
Spend more time looking at git status, as it will save you the time you
are gonna waste for solving merge conflicts that should not be there

images/logo
Sharing Managing multiple instances of the same repository
Outline
1 Versioning
Motivation
2 Sharing

images/logo
Decentralized Version Control System
We now have tools to manage a repository and its history. Next step:
managing multiple copies.
Why decentralized
In git, there is no concept of reference repository where everybody
works
Though the concept can be artiﬁcially rebuilt
Parallel work happen on clones of a repository
Every copy (clone) of a repository has the entire history
There are tools to share changes (commits) with other clones

images/logo
Cloning existing repositories
git clone targeturi destination
Clones the git repository at targeturi inside destination
targeturi and destination are URIs in a supported protocol, e.g.
git clone ~/myrepo ~/myclone
git clone git@myrepo.com:myproject.git ~/myproject
git clone https://myrepo.mysite.com ~/myproject
An error is thrown if targeturi is not a valid git repository
An error is thrown if you have no writing permission on destination
Omitting destination will make git create a new folder inside the
current one, with the name of the last segment of the targeturi
URI.

images/logo
Conﬁguring remote repositories
git remote
Git subcommand which enables assigning names to remote sources
git remote -v
Lists the conﬁgured remotes (name and URI)
git remote add name uri
Assignes the symbolic name name to uri
The uri can be local (so yes, a local remote is ok in git)
git remote rm name
Deletes the remote name
If a repository is cloned rather than initialized, a remote named origin is
automatically created and associated to the upstream

images/logo
Conﬁguring upstream branches
In git, branches can be conﬁgured to track a remote branch. In case
sharing commands are issued without an explicit target, the current branch
upstream will be used.
git branch -u remoteName/remoteBranch
Sets the branch remoteBranch of the remote remoteName as upstream
for the current branch
By default, cloned repositories have branches set up to track their
correspondents on origin

images/logo
Importing remote branches
In git, by default, only the main branch (usually called master) is cloned.
Other remote branches must be checked out manually.
git checkout -b localname remoteName/remoteBranch
Creates a new branch localname with the history of remoteBranch on
remoteName
remoteName/remoteBranch is automatically set as upstream
Tipically, the local name is the same of the remote name, e.g.:
git checkout -b develop origin/develop

images/logo
Fetching new commits from remote branches
git fetch remote branch
Downloads from remote all the updates to branch
It does not merge them in the current branch
remote can be an URI or a configured remote name
if remote and branch are unspecified, the upstream branch is used (if
appropriately configured)
If you want the changes to be included in your current branch, you must
merge after fetch.

images/logo
Pulling changes into the local branches
The operation of fetching changes and merging them is so common that
it’s got a dedicated subcommand
git pull remote branch
Equivalent to:
git fetch remote branch && git merge remote/branch
Can be source of surprises if you branch a lot and rewrite history
Some people advocate not to use ita
In fact, fetch + merge is ﬁner-grained and more idiomatic
De facto very used
a
http://archive.is/jua7s

images/logo
Pushing changes to remote branches
The opposite of pulling is pushing: adding new commits to remote
branches
git push remote branch
Pushes the current branch to remote/branch, adding all the missing
commits
You must have writing permits on remote
There must be the same root for the repository
You cannot push if it’d create a new remote head
pull (and solve merge conﬂicts locally) before pushing
In case remote and branch are omitted, the upstream branch is used

images/logo
Sharing Public hosting for git repositories
Outline
1 Versioning
Motivation
2 Sharing

images/logo
Hosting repositories online
Fetch and push operations require direct network access between
peers
The (current) Internet stucture is not P2P, though
A third, shared repository accessible to both could be the answer
Current status
There are a number of competing services
Github
Bitbucket
Gitlab
They all have free and paid plans
The idea is: attract the user, then make it pay when it can’t live
without the features

images/logo
Desiderata
Repository hosting
Access management
Privacy
Support for several network protocols
Support for teams
Web services
Inspection of the code base
Inspection of the history
Issue tracking
Documentation handling
Release hosting
Forking (and related features)
...

images/logo
In this course I
GitHub
De facto standard for hosting open source projects
Rich in services
Rich ecosystem of third-part services
Free for open source projects
As such, not always suitable
(unless you got money, of course)
Great subsystem for documentation (Github pages)
One fork per project, no self-forking

images/logo
In this course II
Bitbucket
Similar in features to Bitbucket
Poorer ecosystem of third-part services
Integrated continuous integrator (pipelines)
Similar to Travis CI, but way less rich in features
Free for open and private projects
With a limit on number of collaborators in private projects
Academic accounts for UniBo members!
Great for scientiﬁc papers!
Self-forking allowed

images/logo
Sharing A transparent development process
Outline
1 Versioning
Motivation
2 Sharing

images/logo
Sharing A transparent development process
Public issue tracker
Both Github and Bitbucket provide a public issue tracker
It can be used to track bugs, sure
It is also useful to track and discuss ideas
Opening the decision process fosters contribution
Big projects have a dedicated repository for discussions1
Interesting proposals may generate very interesting discussions2
1
e.g. http://archive.is/WAhNe
2
e.g. http://archive.is/5eiW8

images/logo
A successful branching model with a shared repository
Outline
1 Versioning
Motivation
2 Sharing

images/logo
Git Flow as a development model
Mainline branch: master
Always alive
Development branch: develop
Branches from master
Never merges
New feature: feature-*
Branches from and merges to
develop
New release: release-*
Branches from develop
Merges to master and develop
Creates a new version
Fix a bug on mainline: hotfix-*
Branches from master
Merges to master and develop
Creates a new version

images/logo
Developing new features
When a new feature must be added to the software, the team responsible
for it should branch a feature-myfeature branch from develop.
git checkout -b feature-myfeature develop
Once all the work has been done, the branch should get merged in
develop and deleted. Even if fast-forward, the merge should be visible, to
prevent history loss.
git checkout develop
git merge --no-ff feature-myfeature
git branch -d feature-myfeature
In order to minimize the merging eﬀort, it is a good practice to incorporate
changes from develop from time to time (e.g. when another team
completed another feature).

images/logo
Releasing a new version
When the status of the develop branch is (besides the very last bits) the
status of the next production release, a release-version should branch
from develop.
git checkout -b release-version develop
In this branch, only things like version number changes or last minute bug
ﬁxes should get incorporated. Once done, we merge the
release-version branch back into develop...
git merge --no-ff release-version
...and master. Plus, we tag master, so that we keep a reference to the
exact repository status at release time. Then, we delete the
release-version branch.
git checkout master
git merge --no-ff release-version
git tag -a version
git branch -d release-version

images/logo
Fix severe bugs affecting the mainline
You spot a bug in your current production branch. The fix must be
delivered immediately.
Start a new hotfix-version branch:
git checkout -b hotfix-version master
Change the version number, fix the bug (also add a regression test). Once
done, repeat the procedure already seen for a normal release.
git merge --no-ff hotfix-version
git checkout master
git merge --no-ff hotfix-version
git tag -a version
git branch -d hotfix-version

images/logo
git flow
This workflow, first suggested by Vincent Driessen, got very popular.
The command sequence is repetitive, and got automated
A git extension (not part of the git distribution) is available:
Introduces the git flow subcommand
Starts and finishes feature, release, hotfix (and support) branches
Under the hood, it calls exactly the commands listed previously
My suggestion
learn git flow as a development model
Get acquainted with it using standard git
When you are very confident that you know what the tool is doing with
your repository, use git flow
This is a good approach in general to new tools:
understand the idea
learn the basics
understand what goes on under the hood
use the advanced features productively

images/logo
A successful branching model with forks and pull requests
Outline
1 Versioning
Motivation
2 Sharing

images/logo
A higher degree of control
It’s not always a good idea to give writing permissions on your repository
A student may push bad commits
A wrong push on the wrong branch could have bad eﬀects
Your boss made a change but got tired of the push being refused and
just uses --force
You want ﬁne grained control on the code that is being pushed
e.g. perform code review
How to enforce a higher degree of control, retaining the possibility to
contribute?

images/logo
Forking
fork
A hosted clone of a repository associated to a diﬀerent user / team
Both Bitbucket and GitHub allow it
Bitbucket allows multiple forks of the same repo by the same user
Every developer (or team) has his own fork where to work
Makes changes freely
Once complete, asks the maintainer of the original fork to pull from
its repository
The maintainer of the original repository can choose whether or not to
pull
Possibly, provide a list of changes required for the pull to happen

images/logo
Working with pull requests
pull request
An official request to incorporate changes from a branch of a fork to a
branch of another fork.
Both Bitbucket and GitHub support the process
A list of changes (commits) is pictured
The merge is tested, and in case of conflict the pull request is
suspended and can’t be merged
It is up to the requestor to provide mergeable code and fix conflicts
Code review can be performed by asking a number of changes and
tracking the evolution
Comments can be provided line by line
Once the pull request satisfies the requirements, it can be merged and
becomes part of the original repository

Productive parallel teamwork: Decentralized Version Control Systems

More Related Content

Similar to Productive parallel teamwork: Decentralized Version Control Systems

More from Danilo Pianini

Recently uploaded

Productive parallel teamwork: Decentralized Version Control Systems