Greg Anderson's slides for DrupalCon Vienna Session: Automate Your Automation

1. Practical DevOps
Greg Anderson
Automate Your Automation

2. 2
Greg Anderson
Platform + Open Source Engineer
Contributor to several popular OSS Projects:
● Drush
● Robo
● Consolidation Tool
● CGR
greg_1_anderson

3. Automation
Planning
● Identify something to Automate
● Make it good
● Survey available tools
● Automate!
● Repeat
●
3

4. What Can Be
Automated?
● Development tasks
● Testing
● Deployment
● Maintenance (updates)
4

5. Investment and
Benefit
● More reliable and repeatable
● Easier to bring on new people
● Improvements compound
● It becomes possible to do more
5
https://xkcd.com/1205/

6. Cost of Not
Automating
● Prone to errors and mistake
● Risk of lost knowledge
● Deferred maintenance piles up
● Discouragement
6

7. Extra Art
7
Tools of the Trade

8. Service APIs
● Create pull requests
● Post comments
● Configure test credentials
● Web hooks
8

9. Create Pull
Requests
1. Push branch to GitHub
2. Create the PR
○ Use hub tool
○ Use REST API
●
9

10. 10
Push Branch to GitHub
Pushing to a remote, and authenticated via ssh (e.g. from command line):
$ git push -u origin my-pr-branch
From a script with no remote set, and using an OAuth token:
$ push https://$GITHUB_TOKEN:x-oauth-basic@github.com/repo.git master

11. Install and Use hub CLI Tool
11
Create a pull request:
$ hub pull-request -m "This is my awesome pull request."
MacOS
brew install hub choco install hub sudo apt-get install git-hub
https://github.com/github/hub

12. Call GitHub REST API with Guzzle
12
function gitHubAPI($uri, $token, $data = [], $method = 'GET')
{
$url = "https://api.github.com/$uri";
$headers = [
'Content-Type' => 'application/json',
'User-Agent' => 'my-org/my-app',
$headers['Authorization'] = "token " . $token();;
}
$guzzleParams = [ 'headers' => $headers, ];
if (!empty($data)) {
$method = 'POST';
$guzzleParams['json'] = $data;
}
$client = new GuzzleHttpClient();
$res = $client->request($method, $url, $guzzleParams);
$resultData = json_decode($res->getBody(), true);
$httpCode = $res->getStatusCode();
}

13. Create Pull Request via REST API
13
https://developer.github.com/v3/pulls/#create-a-pull-request
gitHubAPI(
"/repos/$org/$repo/pulls",
$token,
[
"Title" => "Amazing new feature",
"Body" => "Please pull this in!",
"Head" => "$forkedRepo:new-feature",
"Base" => "master"
],
'POST'
);

14. Post Comments
on Pull Requests
The hub tool does not support
comments*
, but the REST API is
still available.
14
* Help improve hub! https://github.com/github/hub/pull/1465
Are you
sure this
will fly? Make sure the
mock tests are
passing before
the demo.
Has someone
volunteered to
do the cliff
jump demo?

15. Add a Comment via REST API
15
https://developer.github.com/v3/issues/comments/#create-a-comment
gitHubAPI(
"/repos/$org/$repo/commits/$sha/comments",
$token,
[
"Body" => "Me too",
],
'POST'
);

16. Configure Test
Credentials
● Circle CI
○ REST API
● Travis CI
○ CLI Tool
○ REST API
16

17. 17
Circle CI REST API
curl
-X POST
--header "Content-Type: application/json"
-d '{"name":"foo", "value":"bar"}'
https://circleci.com/api/v1.1/project/github/$user/$proj/envvar?circle-token=$t
Obtain an OAuth token for circle-token at https://circleci.com/account/api.
Important note: Circle uses github in its API URLs, even though it shortens this
to gh in all of the URLs in its web application.
https://circleci.com/docs/2.0/env-vars/#injecting-environment-variables-with-the-api

18. Travis CI CLI Tool
18
Set environment variables via env command:
$ travis env set GITHUB_TOKEN $GITHUB_TOKEN
For web API, see https://docs.travis-ci.com/api/#settings:-environment-variables
https://github.com/travis-ci/travis.rb#env
MacOS
brew install travis Use the RubyInstaller sudo apt-get install ruby1.9.1-dev

19. Spinning Up New
Projects
● Composer create-project
● Post-create project scripts
19

20. 20
Composer create-project
● Copies and renames a repository and orphans / detatches the result.
● Updates from the source may still be pulled by adding a second remote:
○ git remote add upstream git@github.com :parent-org/parent-project.git
○ git pull upstream master
● Usually, parent is treated as a template project, and updates are not pulled.
● Things that will evolve over time are placed in dependencies.
● Dependencies are updated via composer update, as usual.
https://github.com/drupal-composer/rupal-project

21. Post create-project scripts
21
● Composer will run user-specified scripts after commands (e.g. install, update)
● This is used in drupal-composer/drupal-scaffold to download addition files.
● This allows files that cannot be stored in a dependency to be managed
independently from the template repository.
https://getcomposer.org/doc/articles/scripts.md

22. Extra Art
22
Testing Practices

23. Write Testable
Code
● Pass values to functions
● Return values from functions
23

24. Functional Testing
● Uses a working system to test
● Usually easy to write tests
● Sometimes hard to maintain
(false failures creep in)
● Takes longer to run
24

25. Mocks
● Replaces an actual system
● Useful for providing values
● Can also collect results
● Sometimes hard to maintain
(false passes can creep in)
● Best to avoid testing
implementation, e.g.
confirming order of API calls
25

26. Leveraging
Docker
● Circle CI 2.0 or Travis
● Maintain custom images
● Store system scripts in image
● Automatically rebuild your
docker image whenever its
source repository changes.
26

27. Parallelism in Circle 2.0
27
https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs
defaults: &defaults
docker:
- image: quay.io/pantheon-public/build-tools-ci:1.x
working_directory: ~/example_drops_8_composer
environment:
ADMIN_USERNAME: admin
version: 2
jobs:
build:
<<: *defaults
steps:
- checkout
- run:
name: environment
command: /build-tools-ci/scripts/set-environment

28. Create Custom Docker Image
28
DockerFile URL
# Use an official Python runtime as a parent image
FROM drupaldocker/php:7.1-cli
# Set the working directory to /build-tools-ci
WORKDIR /build-tools-ci
# Copy the current directory into the container at /build-tools-ci
ADD . /build-tools-ci
# Collect the components we need for this image
RUN apt-get update
RUN composer -n global require -n "hirak/prestissimo:^0.3"
RUN mkdir -p /usr/local/share/drush
RUN /usr/bin/env COMPOSER_BIN_DIR=/usr/local/bin composer -n
--working-dir=/usr/local/share/drush require drush/drush "^8"

29. Automate with Quay and GitHub
29

30. Generating Tests
using an IDE
30

31. Example: PHPStorm
31
https://confluence.jetbrains.com/display/PhpStorm/Creating+PHPUnit+Tests+in+PhpStorm

32. Extra Art
32
Deployment

33. Automating
Releases
● Travis release steps
● Self Updates
33

34. Automate setting up releases
34
Use the travis tool to automatically edit .travis.yml to include a release section:
$ travis setup releases --token=$GITHUB_TOKEN
Detected repository as org/my-tool, is this correct? |yes|
Username: greg-1-anderson
Password for greg-1-anderson: *************
Two-factor authentication code for greg-1-anderson:[REDACTED]
File to Upload: my-tool.phar
Deploy only from org/my-tool? |yes|
Encrypt API key? |yes|
If a token is provided, then the credentials prompt will be omitted.

35. 35
Self-Updating Phars
Robo PHP Task
Runner
Robo as a framework: https://robo.li/framework
Instructions on using Robo to create a standalone phar; adds
self:update command automatically.
$commandClasses = [ MyProjectCommandsRoboFile::class ];
$statusCode = RoboRobo::run(
$_SERVER['argv'],
$commandClasses,
'MyAppName',
'0.0.0-alpha0',
$output,
'my-org/my-project'
);
exit($statusCode);

36. Extra Art
36
Maintenance Tasks

37. Automating
Composer Update
● Schedule via Travis cron
● Composer Lock Updater
● Auto-merge after tests pass
37

38. 38
Automated Dependency Update Process
Travis
Cron
Composer
Lock Update
GitHub
Pull Request
Travis
Test Run
Merge
Pull Request
Exec hub REST REST
[1] [2]
1. Pull request is only created if there are updates available.
2. Pull request is only merged if the test passes.

39. 39
Set Up Travis CI Cron
Once configured, Travis schedules job to start right away:

40. 40
Add GitHub Token to Travis Settings
From https://github.com/settings/tokens:
Circle CI REST API URL

41. 41
Specify Scopes
Circle CI REST API URL

42. 42
Add GitHub Token to Travis Settings
$ travis env set GITHUB_TOKEN "[PASTE VALUE FROM GITHUB]"
Circle CI REST API URL

43. 43
Prevent Redundant Execution
If doing highest / lowest testing with a matrix configuration entry, define an
environment variable to control which Travis run will do the post-build actions:
matrix:
include:
- php: 7.1
env: dependencies=highest
env: DO_POST_BUILD_ACTIONS=1
- php: 7.0.11
- php: 5.6
- php: 5.5
env: dependencies=lowest

44. 44
Only Update Dependencies in Cron
after_success:
- travis_retry php vendor/bin/coveralls -v
- |
if [ -z "$DO_POST_BUILD_ACTIONS" ] ; then
return
fi
if [ "$TRAVIS_EVENT_TYPE" != "cron" ] ; then
echo "Not a cron job; exiting."
return
fi
if [ -z "$GITHUB_TOKEN" ]; then
echo "No GITHUB_TOKEN defined; exiting."
return
fi
# … execution continues below

45. 45
Install and Run Composer Lock Updater
# … execution continues here from "after_success:" example
export PATH="$HOME/.composer/vendor/bin:$PATH"
composer global require danielbachhuber/composer-lock-updater
mkdir -p $HOME/bin
wget -O $HOME/bin/security-checker.phar [URL]
chmod +x $HOME/bin/security-checker.phar
export PATH="$HOME/bin:$PATH"
wget -O hub.tgz [URL]
tar -zxvf hub.tgz
export PATH=$PATH:$PWD/hub-linux-amd64-2.2.9/bin/
clu
For [URL], see full instructions in: https://github.com/danielbachhuber/composer-lock-updater

46. 46
Maintain Compatibility in composer.lock
The versions of dependencies returned for a given run of composer update
may vary depending on what version of php you are running. For example, a lock
file built with php 7.1 may include dependencies that do not work with php 5.6.
Fix this with a "platform": "php" entry:
"config": {
"optimize-autoloader": true,
"preferred-install": "dist",
"sort-packages": true,
"platform": {
"php": "5.6"
}
},

47. 47
Update Details in Commit Comment
PR is attributed to user who provided GitHub token.
Comment lists dependencies that were updated.
Security report included to bring attention to vulnerabilities.
Commit itself is authored by composer-lock-updateuser.

48. 48
Automatic Merge after Tests Pass
remote=$(git config --get remote.origin.url)
git remote set-url origin "${remote/github.com/$GITHUB_TOKEN:x-oauth-basic@github.com}"
if [ -n "$TRAVIS_COMMIT_RANGE" ] && [ "master" == "$TRAVIS_BRANCH" ]; then
changed_files=$(git diff --name-status "$TRAVIS_COMMIT_RANGE" | tr 't' ' ')
if [[ "$changed_files" == "M composer.lock" ]] ; then
(
echo "Only composer.lock was modified: auto-merging to $TRAVIS_BRANCH in $remote."
git reset "$TRAVIS_BRANCH"
git stash
git checkout "$TRAVIS_BRANCH"
git stash pop
git add composer.lock
git commit -m "Auto-update dependencies."
git push origin "$TRAVIS_BRANCH"
) 2>&1 | sed -e "s/$GITHUB_TOKEN/REDACTED/g"
else
echo "Not auto-merging, because multiple files were changed"
fi
fi

49. Automate the
automation of your
automation
Run a command to
automatically configure Travis to
automate the updating of your
project’s composer.lock file.
49

50. 50
The ci travis:clu Command
● Install ci.phar.
● Change your working directory to your PHP project.
● Define $GITHUB_TOKEN
● Run ci travis:clu
● Inspect the changes. Push them up to GitHub.
● Turn on cron in the Travis admin interface
● Be amazed.

51. Extra Art
51
Finished! Not Finished!

52. JOIN US FOR
CONTRIBUTION SPRINT
Friday, 29 September, 2017
First time
Sprinter Workshop
Mentored
Core Sprint General Sprint
9:00-12:00
Room: Lehar 1 - Lehar 2
9:00-18:00
Room: Stolz 2
9:00-18:00
Room: Mall
#drupalsprints

53. WHAT DID YOU THINK?
Locate this session at the DrupalCon Vienna website:
http://vienna2017.drupal.org/schedule
Take the survey!
https://www.surveymonkey.com/r/drupalconvienna