In this talk, you’ll be introduced to an innovative, powerful, flexible, and easy-to-use tool developed by the folks at NerdWallet that simplifies some of the complexities associated with making horizontal changes in service oriented systems via automation.

1. Need to make a horizontal change across 100+
microservices? No worries, Shepherd's got you
covered
Austin + NYC NodeJS Meetup
June 17th, 2020

2. whoami
speaker: Aori Nevo, PhD
handle: @aorinevo
- Work experience:
- SDM @ AWS (currently)
- Sr. Eng Mng @ Pager (~1.5
years)
- Eng Mng @ Bloomingdale’s (~3
years)
- OSS contributor/maintainer
(~10+ years)
- Shepherd, Yargs, Hapi,
Handlebars, New Relic,
Foundation, …
- Army (4 years)
- Education:
- BA, MA, MS, PhD in Mathematics
(12 years)
- Hobbies:
- Coding :P

3. Abstract
In this talk, you’ll be introduced to an innovative,
powerful, flexible, and easy-to-use tool developed by the
folks at NerdWallet that simplifies some of the complexities
associated with making horizontal changes in service
oriented systems via automation.

4. Problem Statement
In a highly distributed system comprised of multiple repos,
make a change to a non-empty subset programmatically.

5. Why?
(by examples)
Also, why?

6. Adding a New File - Current Workflow
1. Suppose you have n repos and want to make a change to a
non-empty subset s (i.e. change the engines property in
package.json, etc...)
2. For each repo in s
a. fork and clone the repo to your local
b. add the file
c. stage the changes
d. commit the changes
e. push the commit to your remote
f. open a PR to upstream
g. solicit reviews and approvals
h. land commits on upstream
i. release changes to lower-level environments, test, and promote
through to production

7. Making a Code Change - Current Workflow
1. Suppose you have n repos and want to make a change to a
non-empty subset s (i.e. at renovate.json, eslintrc,
etc…)
2. For each repo in s
a. fork and clone the repo to your local
b. find and update the relevant code
c. stage the changes
d. commit the changes
e. push the commit to your remote
f. open a PR to upstream
g. solicit reviews and approvals
h. land commits on upstream
i. release changes to lower-level environments, test, and promote
through to production

8. Obvservations
Note that in both examples, steps 2a - 2f are the
same given we choose the appropriate abstraction
layer.
The time needed to make these changes is directly
proportional to the size of s.
Also, provided there is a programmatic way of
identifying which repos should be in set s as well as
a programmatic way to apply the changes to the repos,
then we can effectively abstract out these steps into
a script. You can think of Shepherd as that
abstraction.

9. Shepherd
Shepherd is an open-source CLI tool developed at NerdWallet
that coordinates the application of code changes across
multiple repos, from checking them out, to making the
changes, to submitting pull requests.
Think of Shepherd in this way:
Shepherd is to code as what migrations are to databases.

10. Install Steps
just kidding

11. Install Steps
Pretty short:
- npm install -g @nerdwallet/shepherd
- That’s it!
Verify the install via cli by running the following command:
- shepherd --help
Notes: If installing off of clone, run npm run build
post-install

12. Demo(s)!

13. Demo 1 - Add manifest.yml file
# shepherd.yml
id: 2020.05.20-add-manifest-file
title: |
feat: add manifest file
adapter:
type: github
search_query: repo:aorinevo/shepherd-test-repo-1 repo:aorinevo/shepherd-test-repo-2 path:/
hooks:
should_migrate:
- test ! -f manifest.yml
apply: $SHEPHERD_MIGRATION_DIR/main.sh
pr_message: |
echo "Add manifest file"

14. Demo 1 cont’d
- Shepherd commands
- shepherd checkout .
- shepherd apply .
- shepherd commit .
- shepherd push .
- shepherd pr-preview .
- shepherd pr .
Note: All commands can be verified by inspecting Shepherd’s
working directory (defaults to ~/.shepherd).

15. Demo 2: Exercise 1: Add engines property to package.json
Q: What are the shell commands to apply this change?
Q: What would the shepherd.yml file look like?
Q: Can you do this only using shepherd.yml and no other
files?

16. ROI
No, it’s not wool!

17. ROI
- Reduce time to delivery
- Increase productivity
- Increase velocity
- Increase efficiency
- Self documenting
- Repeatable
- Identify opportunities to simplify (i.e. horizontal
changes like these are somewhat of an indicator we should
consider different approaches)

18. How to get involved

19. How to get involved
- Slack
- https://bit.ly/node-shepherd
- GitHub
- https://github.com/NerdWalletOSS/shepherd
- Contribute
- Good first issue
- https://github.com/NerdWalletOSS/shepherd/labels/good%20first%20i
ssue
- Example
- https://github.com/NerdWalletOSS/shepherd/issues/190

20. Future Work

21. Future Work
1. Upgrade to latest Octokit
○ https://github.com/NerdWalletOSS/shepherd/issues/109
2. Branding
○ https://github.com/NerdWalletOSS/shepherd/issues/145
3. Improved configuration system
○ https://github.com/NerdWalletOSS/shepherd/issues/91
4. Use GitHub's GraphQL api
○ https://github.com/NerdWalletOSS/shepherd/issues/15
5. Common migration script utilities
○ https://github.com/NerdWalletOSS/shepherd/issues/62
6. Workflow improvements
○ https://github.com/NerdWalletOSS/shepherd/issues/89

22. References
- https://www.nerdwallet.com/blog/engineering/shepherd-auto
mating-code-changes/
- https://www.npmjs.com/package/@nerdwallet/shepherd
- https://github.com/NerdWalletOSS/shepherd

23. Thank you!

24. Q&A