Advanced GitHub Enterprise Administration

Advanced GitHub Enterprise Administration
Lars Schneider | Technical Lead for GitHub, Autodesk Inc.
Todd O’Connor | Senior SCM Engineer, Adobe Systems Incorporated

© 2017 Adobe Systems Incorporated. All Rights Reserved.
© 2017 Autodesk Inc. All Rights Reserved.
Company Introduction
Adobe Systems Incorporated
• Founded December 1982
• 17,000+ employees in 37 countries
• Leading provider for Digital Media and Digital Marketing
Autodesk Inc.
• Founded January 1982
• 9,000+ employees worldwide
• Leader in 3D design, architecture, engineering, construction, manufacturing,
media, and entertainment industries software
2

Outline
• General Setup and Configuration
• Helping Administrators
• Helping Users
• Diagnosing and Addressing Issues
3

One More Thing...
4
ACME-Org
Org1
Org2
Org3

One More Thing...
5
3 ACME members contributed to
SharedComponents.

General Setup and Configuration

General Setup and Configuration
• Server Considerations
• Admin Center
• Server HA / DR Considerations
• Other Techniques
7

Server Considerations
• Dedicated VM host
• Allocate a lot of RAM
• Sustained memory usage should stay below 50% of total RAM under normal use
• Excess RAM used as disk cache - IO caching improves performance
significantly
• Increase number and/or speed of CPUs
• Large root partition
• Why? Logs can grow rapidly by users running commands that generate a lot of
log activity
• Can only (re)size when building a new server
• Keep fewer logs
8

Server Considerations continued …
• Use a staging server
• Covered by the GitHub license
• Use GitHub Backup utilities on a dedicated backup server (just a simple VM)
• Run backups often
• Dedicated backup server storage should be sized as large, if not larger than
your primary, in order to retain X number of backups
• Make sure to backup your backup server!
• Long term log retention
• Forward to an external server like Splunk
• Set this in the Management Console
9

Admin Center
Lots of configuration items to consider
• Force pushes
• Repository upload limit
• Don’t use unlimited!
• Large File Support (LFS)
• Keep large files out of your repos
• Cloned over and over …
• Reactivate suspended users
10

Server HA / DR Considerations
• Use a replica server
• Single server setup there is only a 1:1 pairing – one primary and one replica
• Cluster acts as its’ own HA so there is no separate replica – only a DR copy
 
HA : replica in same data center, very low (to no) latency, very quick recovery 
Used when primary system fails  
(i.e. software crash, hardware failure, VM host failure, network outage (if HA is on separate network)) 
Only 1:1 mapping
 
DR : system at remote site, (re)built from backup/restore 
Used in the event of prolonged outage or catastrophic event at the primary site
11

Other Techniques
• Overarching Org Owner
• Owner of every Org, useful for data mining
Ex. Web page of all Org owners to contact for access
• Aggressively suspend users that
• Have left the company
• Are not active in certain time period
• Consider selective Pre-receive hooks
• Block certain (restricted) filetypes (secret files, certificate files, …)
• Check commit messages for bug IDs or other business required information
12

New Features
• Rate Limits
• API, Abuse, Git
• Be careful, could hit limits quickly so be able to react quickly (Plan B)
• Dedicated alambic storage
• LFS, releases, file uploads, avatars, …
• GEO Replication (early access feature)
• Read access to replicas based on user’s location
13

Helping Administrators
• Leverage ghe-announce
• Self-service Portal
• Scotty: Helper Scripts
• How to write a support ticket
15

Leverage ghe-announce
• Sets a message that is visible to everyone - on every page
• Can use HTML
16
$ ghe-announce -s "<b>Useful links:</b> <a href='URL'>Self-service portal</a> | <a href='URL'>Wiki</a> |
<a href='URL'>Activity</a> | <a href='URL'>Stage</a> <i>(new release preview and testing only)</i>”
$ ghe-announce -h
Usage: ghe-announce [-s <message>|-u|-g]
Get, set or clear a global announcement banner, to be displayed to all users.
OPTIONS:
-h Show this message
-s MESSAGE Set a global announcement banner
-g Get the global announcement banner
-u Unset the global announcement banner

Self-service Portal
Useful for:
• Accounts that cannot login via a browser
• Create, manage SSH keys
• Setting up integrations where admin accounts/secret keys are involved
17

Self-service Portal …
18

19

20

Tools: Scotty
21
https://github.com/larsxschneider/scotty

Tools: Scotty
22
SAFE: Read-only access!
DANGER: Modifies your instance!

Tools: Scotty - Help
23
$ ./reports/git-versions.sh -h
Prints the recently used Git client versions per IP address or per user.
Usage:
git-versions.sh
Options:
-u, --user Print results per user account (default: IP address)
-a, --all Process all available logs (rolled logs)
-n Dry-run; only show what would be done.
-h, --help Display this message.
Example:
git-versions.sh

Tools: Scotty Usage
24
$ ./reports/git-versions.sh
2.14.1 510
2.14.0 12
2.13.5 6
2.13.4 2
...
Data from the last 24h hours
$ ./reports/git-versions.sh -a
2.14.1 2011
2.14.0 31
2.13.5 97
2.13.4 3
...
Data from all available logs
(usually 7 days)

How to Write a Support Ticket?
25
• Search your previous tickets
• Explain the problem kindly, short, and precise
• Avoid company specific slang/abbreviations/code names
• Add details that might be relevant to the investigation:
• Steps to reproduce the problem
• Time of the incident in UTC
• IP address, Git/Git LFS/browser/OS version, ...
• Usernames, repository names, Pull Request/Issue IDs, ....
• Screenshots
• Set the appropriate severity (not everything is urgent!)

26
• Send a support bundle from the instance
$ ghe-support-bundle -t <ticket-id>< 24h
$ ghe-support-bundle -x -t <ticket-id>> 24h
$ ghe-cluster-support-bundle -t <ticket-id>primary+replica
$ ghe-support-bundle -t <ticket-id> -f <path-to-file>upload arbitrary file
Run in "screen"!

27
• Upload a file from a user client
• Send quick diagnostics
$ ./admin/support-bundle-local-attach.sh <ticket-id> <path-to-file>
$ ./admin/send-ghe-diagnostics.sh <ticket-id>

Helping Users
• Unassociated Commits
• Recover Branches
• Completely Delete Git Repositories
29

Unassociated Commits
30
$ git push
[master c417679] Commit with invalid email
...
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
remote: WARNING:
remote: WARNING: The author of at least one commit on 'master' does not have an Autodesk email address.
remote: WARNING: commit: c417679e0a184062f887066137b08aa13161c3cb
remote: WARNING: author email: lars@mydomain.com
remote: WARNING:
remote: WARNING: Fix the issue by configuring the Autodesk Git environment: https://autodesk.com/github
remote: WARNING:
To https://autodesk.com/demo/repo
df9beef..c417679 master -> master
[user]
name = Lars Schneider
email = lars@mydomain.com
~/.gitconfig

• Pre-Receive Hooks 
https://github.com/github/platform-samples
• Autodesk Git Environment 
https://github.com/Autodesk/enterprise-config-for-git
Unassociated Commits
31

Recover Branches
32

!
Recover Branches
32

!
$ ./reports/repo-audit.sh org/repo | grep thebranch
refs/heads/fix/doc-build-scripts
9e3f1805a4b6d94982896eeed466729c74f3e1d7
0000000000000000000000000000000000000000
Lars Schneider <lars.schneider@autodesk.com> 1505314979 +0200
{
"real_ip": "10.111.222.33",
"repo_name": "myorg/myrepo",
"repo_public": true,
"user_login": "schneil",
"user_agent": "Mozilla/5.0 ...",
"from": "branches#destroy",
"via": "branches page delete button",
"user_name": "Lars Schneider",
"user_email": "lars.schneider@autodesk.com",
"server": "git-autodesk-com-primary",
"user_id": 210
}
refs/heads/thebranch
9300c21e0da295d304ca1cb2e1dd72d62657f8b3
0000000000000000000000000000000000000000
{
"real_ip": "10.111.222.33",
"repo_name": "org/repo",
"user_login": "lars",
"server": "git-company-com-primary",
"user_id": 123
}
refs/heads/thebranch
9300c21e0da295d304ca1cb2e1dd72d62657f8b3
0000000000000000000000000000000000000000
{
"real_ip": "10.111.222.33",
"repo_name": "org/repo",
"user_login": "lars",
"server": "git-company-com-primary",
"user_id": 123
}
Recover Branches
32
"
https://git.company.com/org/repo/tree/9300c

Recover Branches
https://help.github.com/articles/about-protected-branches/
33

Completely Delete Git Repositories
34
$ ./admin/repo-purge-archived.sh org repo
GitHub Enterprise appliance does not instantly purge repositories
after deletion. They are held on disk for 90 days before disk deletion.
Does not take care of forks, clones, and backups!⚠

Debug: Local Client - What is Git doing?
$ git clone https://gitserver.com/org/repo.git
36

$ git clone https://gitserver.com/org/repo.git
Cloning into 'repo'...
remote: Counting objects: 31, done.
remote: Total 31 (delta 0), reused 0 (delta 0), pack-reused 31
Unpacking objects: 100% (31/31), done.
Downloading test.bin (6 B)
Downloading test2.bin (6 B)
37

$ GIT_TRACE=1 git clone https://gitserver.com/org/repo.git
12:27:45.113068 git.c:328 trace: built-in: git 'clone' 'https://gitserver.com/org/repo.git' 'test'
Cloning into 'test'...
12:27:45.119455 run-command.c:626 trace: run_command: 'git-remote-https' 'origin' 'https://server/org/repo.git'
12:27:46.029989 run-command.c:626 trace: run_command: 'git credential-osxkeychain get'
12:27:46.036845 git.c:560 trace: exec: 'git-credential-osxkeychain' 'get'
12:27:46.037392 run-command.c:626 trace: run_command: 'git-credential-osxkeychain' 'get'
12:27:46.310512 run-command.c:626 trace: run_command: 'git credential-osxkeychain store'
12:27:46.317397 git.c:560 trace: exec: 'git-credential-osxkeychain' 'store'
12:27:46.317963 run-command.c:626 trace: run_command: 'git-credential-osxkeychain' 'store'
12:27:46.348240 run-command.c:626 trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' ...
12:27:46.351145 git.c:328 trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' ...
12:27:46.616268 run-command.c:626 trace: run_command: 'unpack-objects' '--pack_header=2,31'
12:27:46.620899 git.c:328 trace: built-in: git 'unpack-objects' '--pack_header=2,31'
...
38

$ GIT_TRACE_PERFORMANCE=1 git fetch
13:35:10.004634 trace.c:435 performance: 0.018234000 s: git command: 'git' 'credential-osxkeychain' 'get'
13:35:10.533870 trace.c:435 performance: 0.028022000 s: git command: 'git' 'credential-osxkeychain' 'store'
remote: Compressing objects: 100% (16/16), done.
13:35:11.472630 trace.c:435 performance: 0.009882000 s: git command: '/usr/git-core/git' 'unpack-objects' ...
13:35:11.473454 trace.c:435 performance: 0.913069000 s: git command: '/usr/git-core/git' 'fetch-pack' ...
13:35:11.757573 trace.c:435 performance: 0.272907000 s: git command: '/usr/git-core/git' 'rev-list' ...
From https://server/org/repo
* [new branch] foo -> origin/foo
13:35:11.791641 trace.c:435 performance: 0.001260000 s: git command: '/usr/git-core/git' 'gc' '--auto'
13:35:11.791864 trace.c:435 performance: 2.897341000 s: git command: 'git' 'fetch'
39

$ GIT_CURL_VERBOSE=1 git fetch
* Couldn't find host server in the .netrc file; using defaults
* Trying 10.35.74.87...
* Connected to server (10.22.33.44) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: gitserver.com
* Server certificate: Symantec Class 3 Extended Validation SHA256 SSL CA
* Server certificate: VeriSign Universal Root Certification Authority
> GET /org/repo.git/info/refs?service=git-upload-pack HTTP/1.1
Host: gitserver.com
User-Agent: git/2.14.1
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache
< HTTP/1.1 401 Authorization Required
< Server: GitHub Babel 2.0
< Content-Type: text/plain
< Content-Length: 26
< WWW-Authenticate: Basic realm="GitHub"
...
HTTPS
40

$ GIT_SSH_COMMAND="ssh -vvv" git fetch
OpenSSH_6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /Users/bob/.ssh/config
debug1: /Users/bob/.ssh/config line 6: Applying options for github.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to github.com [192.30.253.113] port 22.
debug1: Connection established.
debug1: identity file /Users/bob/.ssh/private_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/bob/.ssh/private_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: Remote protocol version 2.0, remote software version libssh_0.7.0
debug1: no match: libssh_0.7.0
debug2: fd 5 setting O_NONBLOCK
debug1: Authenticating to github.com:22 as 'git'
debug3: hostkeys_foreach: reading file "/Users/bob/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/bob/.ssh/known_hosts:1
...
SSHSSH
41

Debug: Local Client - Git Environment Variables
https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables
42

Debug: GitHub Enterprise Components
Client HAProxy babeld gitauth
nginx alambic
unicorn
lfs-server
43

Client HAProxy
./reports/haproxy-requests.sh -d
132556 Sep 14 alambic
264 Sep 14 babeld_git
320429 Sep 14 babeld_http
94388 Sep 14 babeld_ssh
34 Sep 14 babeld_svn
1977 Sep 14 codeload
276 Sep 14 dumbhttp
1154 Sep 14 ernicorn
661 Sep 14 http_protocol
14905 Sep 14 https_protocol
15405 Sep 14 lfs_server
10152 Sep 14 longpoll
227937 Sep 14 web_unicorns
/var/log/haproxy.log
• Handles all GitHub Enterprise traffic
• What kind of requests are made?
• How many requests are made?
• Where the requests come from?
44

$ ./reports/babeld-errors.sh -d
Sep 14 | 1154 Permission
Sep 14 | 3279 failed to verify pubkey
Sep 14 | 4408 Invalid username or password
Sep 14 | 4927 account is suspended
Sep 14 | 4991 Repository not found
Sep 14 | 9905 not a valid username or token
...
$ ./reports/babeld-errors-sources.sh
297 10.133.144.225
524 10.155.177.213
573 10.177.97.130
669 10.111.381.104
3166 10.22.136.197
4106 10.33.96.220
...
Client HAProxy babeld
/var/log/babeld/babeld.log
• Handles all Git traffic
• What kind of Git errors happen?
• What clients generate Git errors?
45

/var/log/github/gitauth.log
• Authenticates users for Git access
• What are the HTTPS/SSH authentication issues?
• Who authenticates with username/password?
$ ./reports/auth-http-without-token.sh
456 alice path=revit/server.git
668 bot path=autocad/autocad.git
46

Web content proxy
nginx
/var/log/nginx/*.log
47

nginx alambic
• Avatars
• Releases
• Uploads
48

nginx alambic
GitHub Enterprise website
/var/log/github/unicorn.log
unicorn$ /reports/unicorn-errors.sh -d
10887 2017-09-14 HTTP 404
8 2017-09-14 HTTP 406
1545 2017-09-14 HTTP 422
493 ...
76741 not verifying SSL hostname of LDA...
49

nginx alambic
unicorn
lfs-server
/var/log/lfs-server/lfs-server.log
50

=== HUBBLE ENTERPRISE ===
> all systems at 100 %
> interactive demo online:
> autodesk.github.io/hubble
53

https://github.com/autodesk/hubble
Hubble – Collaboration Visualization
5454
ACME-Org

5555
ACME-Org
Org1
Org2
Org3

5656
ACME-Client
ACME-Server

13 ACME members contributed to
SharedComponents.
57
ProductA
ProductC
ProductB

Hubble – Usage Visualization
58

59

60

61

62
Repository size: ~200 MB

=== HUBBLE ENTERPRISE ===
> all systems at 100 %
> interactive demo online:
> autodesk.github.io/hubble
63

• Hubble Enterprise - GitHub Enterprise Dashboard 
https://github.com/Autodesk/hubble
• Scotty - GitHub Enterprise Helper Scripts 
• Enterprise Config for Git 
https://github.com/Autodesk/enterprise-config-for-git
• GitHub Enterprise Pre-Receive Hooks 
https://github.com/github/platform-samples
Summary @kit3bus
Follow for updates!
toddocon@adobe.com
lars.schneider@autodesk.com
Ask questions!
65

Advanced GitHub Enterprise Administration

Advanced GitHub Enterprise Administration

In this document

More Related Content

What's hot

Similar to Advanced GitHub Enterprise Administration

Recently uploaded

Advanced GitHub Enterprise Administration