Presentation given at Apache: Big Data and ApacheCon North America 2016. "In this time of rapidly growing software projects and software capabilities, where it is expected for “software to eat the world,” there is still a huge challenge going from source code to a tested, fully functional release. This is the “last mile problem,” ensuring that vision and coding become real, deployable software. To help address this problem, members of the extended Apache Hadoop/”big data” ecosystem have joined forces to create tools that reduce the burden of pre-commit testing, release note compilation and interface documentation. In this talk, Allen Wittenauer, a PMC member of the Apache Yetus project, will discuss the various components that make up the Yetus toolset, as well as how Apache Hadoop and other projects are using Apache Yetus to improve release quality. "

1. Apache Yetus: Helping Solve the Last Mile Problem
Allen Wittenauer
Effective Machines
Apache: Big Data North America 2016-05
ApacheCon North America 2016-05

2. ! email: aw @ effectivemachines.com, aw @ apache.org
! github: aw-was-here
! gitlab: _a__w_
! slideshare: allenwittenauer
! twitter: @_a__w_ ( 1 a 2 w 1 )
22016-05-12

3. 3

4. 4

5. https://www.flickr.com/photos/83633410@N07/7658225516/ 52016-05-12

6. ! “How do these changes actually impact me?”
! “The method my code uses changed!”
! “The quality of this software is awful. Is it actually tested?”
62016-05-12

7. Apache Yetus is a collection of libraries and tools
that enable contribution and release processes
for software projects.

8. Release Doc Maker

9. 92016-05-12

10. 10

11. releasedocmaker
! Input
! One or more JIRA projects
! Version, multiple versions, or a range of versions
! Optional
! Title
! Project Name
! Linting
! Output
! Changes and Release Notes
! Per version
! Grouped by Type and Flags (Important, Incompatible)
! Markdown
112016-05-12

12. 12

13. 13

14. 14

15. 15

16. 16

17. Audience Annotations for Java

18. What’s “safe” to use?

19. Compiler vs. Human

20. 20

21. Audience
! Private
! Limited-Private
! Public
212016-05-12

22. Stability
! Stable
! Evolving
! Unstable
! Deprecated
222016-05-12

23. @InterfaceAudience.Public
@InterfaceStability.Evolving
public class GlobFilter implements PathFilter
{...}
@InterfaceAudience.LimitedPrivate({"MapReduce"})
@InterfaceStability.Unstable
public class Progress {...}
import
org.apache.yetus.audience.InterfaceAudience
import
org.apache.yetus.audience.InterfaceStability
232016-05-12

24. JavaDoc/... Support
! ...audience.tools.ExcludePrivateAnnotationsStandardDoclet
! ...audience.tools.ExcludePrivateAnnotationsJDiffDoclet
! ...audience.tools.IncludePublicAnnotationsStandardDoclet
242016-05-12

25. Shelldocs

26. HADOOP-9902

27. ## @description Print a message to stderr
## @audience public
## @stability stable
## @replaceable no
## @param string
function yetus_error
{
echo "$*" 1>&2
}
272016-05-12

28. 28

29. smart-apply-patch

30. 30
smart-apply-patch --plugins=github
--github-user=apache --github-repo=orc 55
smart-apply-patch --plugins=jira HADOOP-1
smart-apply-patch https://my/cool/server/patch

31. Committer Mode (--committer)
! Only works with git format-patch created patches
! Always --commit
! Always --signoff
! Always --whitespace=fix
! TODO: Add more support for other patch formats (YETUS-158)
! TODO: Add --sign (YETUS-315)
312016-05-12

32. precommit (aka test-patch)

33. 33

34. 34

35. 35

36. 36

37. 37

38. 38

39. 39

40. Driver
Personality Bug Systems Build Tools Tests
Language
Support
Static Analysis
Unit Tests
Test Log
Processing
40

41. ! Compiled Language Support
– C/C++
– Java
– Scala
! Interpreted Language Support
– bash/ksh/sh: shellcheck
– perl: Perl::Critic
– Python: pylint
– Ruby: rubocop, ruby-lint
Tool Support (part 1)
! Build Tools
– ant
– autoconf
– cmake
– gradle
– maven
– make
– “nobuild”
! Bug Systems
– Bugzilla (RO)
– GitHub
– JIRA
412016-05-12

42. ! Unit Test Formats
–ctest
–Junit
–TAP
! Runtime Support
–Automation (e.g., Jenkins)
–Docker
–Multiple JVMs
–Patch branches
–Per-instance maven repos
Tool Support (part 2)
! Specialized Tests
– Apache RAT
–@author
–Java checkstyle
–Javadoc
–Java findbugs
–maven site
–Scaladoc
–whitespace
–XML
422016-05-12

43. 43

44. 44

45. 452016-05-12
test-patch --plugins=all
--github-user=apache
--github-repo=tajo
--jira-re='^TAJO-[0-9]+$'
--branch-default=master
--buildtool=cmake
--basedir=tajo
TAJO-1

46. Personality
462016-05-12
#!/usr/bin/env bash
personality_plugins "all"
function personality_globals
{
BUILDTOOL="maven"
PATCH_BRANCH_DEFAULT=master
JIRA_ISSUE_RE='^TAJO-[0-9]+$'
GITHUB_REPO="apache/tajo"
}

47. Running
test-patch --personality=file --basedir=tajo TAJO-1
test-patch --personality=file --basedir=tajo
https://example.com/file-branch.patch
test-patch --personality=file --basedir=tajo /tmp/patchfile
smart-apply-patch --personality=file --basedir=tajo TAJO-1
472016-05-12

48. Bundled Sample Personalities
482016-05-12

49. 49

50. qbt (quality build tool)

51. 51

52. https://www.flickr.com/photos/83633410@N07/7658230838/ 522016-05-12

53. Administrivia
! Website: https://yetus.apache.org
! Mailing list: dev@yetus.apache.org
! Twitter: @ApacheYetus
! Current Release: 0.2.1 (2016-04-14)
! Yetus is an archaic synonym of the Cymbium genus of gastropods
! First ASF TLP with a ‘Y’
532016-05-12