Presentation given by Jack Humphrey (Engineering Director, Indeed) at Keep Austin Agile 2014. http://keepaustinagile2014.sched.org/event/7faaa9add53a23a59185a595bca50230#.UzCTx61dVt0 For more presentations from Keep Austin Agile 2014, see: http://2014conf.agileaustin.org/?q=presentations For more presentations from Indeed, see http://engineering.indeed.com/talks.

1. forget methodology
focus on what matters
Jack Humphrey
Keep Austin Agile
March 21, 2014

2. JACK
HUMPHREY

3. About Me
● I’ve been…
● a software developer for 19 years,
● a manager for 10 of those years,
● at companies including:
○ Trilogy (3 years)
○ Coremetrics (3 years)
○ Google (5 years)
○ Indeed (5 years and counting)

4. About Me
● I’ve worked on products like:
○ a web content management system,
○ enterprise software development frameworks,
○ a large online data collection platform,
○ features for online advertisers, and
○ a few large-scale consumer web applications

5. I help
people
get jobs.

6. About Indeed
More job searches worldwide than any other employment website
● Over 100 million unique users
● Over 3 billion searches/month
● Over 24 million jobs
● 50+ countries, 28 languages
Jun-07 Jun-08 Jun-09 Jun-10 Jun-11 Jun-12 Jun-13
Unique Visitors / Month

7. focus on what matters…

8. focus on what matters…
SUCCESS

9. success
for software products

10. build a solution

11. build a solution
to a real problem

12. build a solution
to a real problem
that people want to use

13. build a solution
to a real problem
that (enough) people want to use

14. ● Planning (what to do)
● Process (how you do it)
Build a solution to a real problem…

15. ● Timing
● Pricing
● Competition
● Promotion
● Quality
● Random chance
● etc.
… that enough people want to use.

16. We identified the problem, now where do we start?
Build a solution to a real problem…

17. STEP 1
define success

18. STEP 1
define success

19. STEP 1
build a great team

20. talent
+
passion

21. talent
+
passion
+
opportunity

22. STEP 2
define success

23. What is the overall mission?

24. Does everyone understand it and agree?
What is the overall mission?

25. Does everyone understand it and agree?
Does project success line up with the mission?
If not, then a) it’s not success, or b) wrong mission!
What is the overall mission?

26. ● Not a subjective determination
Success should be quantifiable

27. ● Not a subjective determination
● Shipping is not success
Success should be quantifiable

28. ● Not a subjective determination
● Shipping is not success
● Prove success with data
Success should be quantifiable

29. I help
people
get jobs.

30. Job Search helps people get jobs
...by helping them find relevant job postings.
Success for Indeed Job Search

33. Success: Unique Visitors?

34. Success: Job Searches?

35. success?

36. ● Demonstrate success with data
● Evaluate data objectively (unique visitors?)
Avoid Subjectivity

37. ● You may need short-term approximations.
When Success is Hard To Measure

38. ● Best metric for “help people get jobs”: hires
Success for Job Search, take 2

39. ● Best metric for “help people get jobs”: hires
● What’s better than unique visitors & searches?
repeat visitors, bounce rate, click-through rate
Success for Job Search, take 2

40. ● Best metric for “help people get jobs”: hires
● What’s better than unique visitors & searches?
repeat visitors, bounce rate, click-through rate
● What’s even better?
applies (IndeedApply)
Success for Job Search, take 2

41. ● Measurement should be repeatable
● Analysis should be:
○ accurate: based on correct/complete data
○ consistent: metrics don’t change meaning
○ fast: seconds/minutes, not hours/days/weeks
Measure and Learn

42. Google Analytics

43. In-House Analysis Tools

44. ● We use Google Analytics
● We build our own frameworks and tools
○ logrepo: capturing what happens
○ ramses, imhotep: detailed analysis in seconds
○ Demo of these tools @IndeedEng on April 30th
http://engineering.indeed.com/talks
Measurement & Analysis Tools

45. Use whatever tools work for you, but make sure:
● measurement is repeatable
● analysis is accurate, consistent, fast
Measurement & Analysis Tools

46. STEP 1
build a great team

47. STEP 2
define success

48. STEP 3
turn the crank

49. process can never replace
talent and passion

50. process won’t deliver a
successful product

51. but the right process
can help

52. Repeatable and predictable mechanisms for:
How process can help

53. Repeatable and predictable mechanisms for:
● Communication: what, who, how, when
How process can help

54. Repeatable and predictable mechanisms for:
● Communication: what, who, how, when
● Priority: do things in the right order
How process can help

55. Repeatable and predictable mechanisms for:
● Communication: what, who, how, when
● Priority: do things in the right order
● Memory: what happened, what next
How process can help

56. Repeatable and predictable mechanisms for:
● Communication: what, who, how, when
● Priority: do things in the right order
● Memory: what happened, what next
● Delivery: get it done as efficiently as possible
How process can help

57. Repeatable and predictable mechanisms for:
● Communication: what, who, how, when
● Priority: do things in the right order
● Memory: what happened, what next
● Delivery: get it done as efficiently as possible
● Quality: it should work intended
How process can help

58. Repeatable and predictable mechanisms for:
● Communication: what, who, how, when
● Priority: do things in the right order
● Memory: what happened, what next
● Delivery: get it done as efficiently as possible
● Quality: it should work intended
● Risk: minimize the chance of bad things
How process can help

59. “forget about methodology” is an exaggeration
How methodologies can help

60. Can be templates for improving:
● communication
● priority
● delivery
How methodologies can help

61. Can be templates for improving:
● communication
● priority
● delivery
...helping your team iterate on process
How methodologies can help

62. Can help you think about
● What is working well for us?
● What could be better?
How methodologies can help

63. Can help you think about
● What is working well for us?
● What could be better?
… and offer prescribed process changes
How methodologies can help

64. Recently, an experienced engineering manager
described a process change that he loved:
An interesting anecdote

65. Recently, an experienced engineering manager
described a process change that he loved:
“we stopped tracking bugs”
An interesting anecdote

66. Ask yourself, what techniques can help your team:
Focus on what matters.

67. Ask yourself, what techniques can help your team:
● communicate
Focus on what matters.

68. Ask yourself, what techniques can help your team:
● communicate
● prioritize
Focus on what matters.

69. Ask yourself, what techniques can help your team:
● communicate
● prioritize
● remember what has been done
Focus on what matters.

70. Ask yourself, what techniques can help your team:
● communicate
● prioritize
● remember what has been done
● know what needs to be done
Focus on what matters.

71. Ask yourself, what techniques can help your team:
● communicate
● prioritize
● remember what has been done
● know what needs to be done
● build solutions that work, efficiently
Focus on what matters.

72. Ask yourself, what techniques can help your team:
● communicate
● prioritize
● remember what has been done
● know what needs to be done
● build solutions that work, efficiently
● manage risk associated with changes
Focus on what matters.

73. Does your process (or lack thereof)
enable success or just get in the way?
Focus on what matters.

74. Start small and iteratively build process.
...or start big and iteratively reduce/modify process.
Focus on what matters.

75. the evolution of process
at Indeed

76. Process can be summarized simply:
GET STUFF DONE
Started Lean

77. Started Lean
Deliver
MeasureLearn

78. Started Lean
Quickly!
Deliver
MeasureLearn

79. Idea to implementation, as quickly as possible!
● Software Architecture (Service-Oriented)
● Tools, like automated translation
● Risk Management with Rapid Iteration
Engineering Velocity
@IndeedEng Talk:
http://go.indeed.com/velocity

80. Rapid Iteration… to production (multiple times a week)
Velocity and Delivery

81. Rapid Iteration… to production (multiple times a week)
● Don’t wait to fail: means no big failures
Velocity and Delivery

82. Rapid Iteration… to production (multiple times a week)
● Don’t wait to fail: means no big failures
● Work that’s ready doesn’t have to wait
Velocity and Delivery

83. Rapid Iteration… to production (multiple times a week)
● Don’t wait to fail: means no big failures
● Work that’s ready doesn’t have to wait
● More satisfying for developers
Velocity and Delivery

84. Rapid Iteration… to production (multiple times a week)
● Don’t wait to fail: means no big failures
● Work that’s ready doesn’t have to wait
● More satisfying for developers
● Less deadline pressure
Velocity and Delivery

85. Rapid Iteration… to production (multiple times a week)
● Don’t wait to fail: means no big failures
● Work that’s ready doesn’t have to wait
● More satisfying for developers
● Less deadline pressure
● Developers own quality
Velocity and Delivery

86. Velocity and Quality

87. ● Encourages modular thinking
Velocity and Quality

88. ● Encourages modular thinking
● Fewer integration issues
Velocity and Quality

89. ● Encourages modular thinking
● Fewer integration issues
● Fewer changes → faster, comprehensive QA
○ Release multiple times a week for 12 weeks, vs.
○ Release once in 12 weeks
Velocity and Quality

90. ● Encourages modular thinking
● Fewer integration issues
● Fewer changes → faster, comprehensive QA
○ Release multiple times a week for 12 weeks, vs.
○ Release once in 12 weeks
● Encourages automated testing
Velocity and Quality

91. ● Team was small, talented, passionate, focused.
● They iterated quickly.
● Job search was successful.
Success...

92. Process was extremely ad-hoc
● not very consistent (e.g. bug tracking)
● not very repeatable or predictable
● wasn’t creating a memory for the future
...and Mess

93. Process was extremely ad-hoc
● not very consistent (e.g. bug tracking)
● not very repeatable or predictable
● wasn’t creating a memory for the future
… and that was fine. For a while.
...and Mess

94. ...and develop predictable, repeatable ways to:
● Coordinate effort
● Improve communication and transparency
● Create a memory for the future
● Manage risk
● Ensure quality
Time to reflect on process...

95. Infrastructure for Process

96. Infrastructure for Process

97. Infrastructure for Process
...for everything!

98. Type: PRODUCT
Product Management in JIRA
Issue Type:
Initiative
requirements

99. Project Management in JIRA
Type: PROJECT
Issue Type:
New Feature
Issue Type:
Bug
Issue Type:
Improvement
Issue Type:
Task
developer work
deployable
component

100. “Incorporates”
Service A
Service B
Feature
Improvement
Product
Initiative
Webapp X
Feature
incorporates

101. “Depends Upon”
Service A
Service B
Feature
Improvement
Product
Initiative
Webapp X
Feature
depends
upon

102. Links are bidirectional
Service B
Improvement
Product
Initiative
Webapp X
Feature
is depended
on by
is incorporated by

103. When will the feature be ready?
Webapp X
Feature
Fix Version: 2014-03-20
target release

104. When will the initiative complete?
Product
Initiative
Fix Version: March 2014
completion timeframe

105. Self-Documenting Workflow
● Workflow states and actions should suggest a
clear meaning
● Custom workflows for different types of issues

106. Example: Project Issue Workflow
Pending Triage
Pending Acceptance
Accepted
Pending Verification
Pending Closure
Closed
Create
Triage
Accept
Resolve
Verify
Close

107. ● Fix Version deployed to production
→ issues are closed
→ future code changes require new issues
● Every code commit references issue key
Closing the Memory Loop

108. Closing the Memory Loop
JASX-4643
Type: New Feature
Fix Version: 2011-04-21
is incorporated by
UPDOG-76
Type: Initiative
Fix Version: June 2011

109. iterate on process.
it’s never done.

110. translation process
production deploys
a/b testing

111. ● Planning rapid international growth
Iteration: Translation Process

112. ● Planning rapid international growth
● Need a localization strategy that wouldn’t slow
us down
Iteration: Translation Process

113. ● Planning rapid international growth
● Need a localization strategy that wouldn’t slow
us down
● Must coordinate many translators
Iteration: Translation Process

114. ● Planning rapid international growth
● Need a localization strategy that wouldn’t slow
us down
● Must coordinate many translators
● We modeled it in JIRA.
Iteration: Translation Process

115. Custom Translation Issues
Project
Explosion
Translation
(French)
Translation
(Spanish)
Translation
(Japanese)
incorporates
Translation
(*)

116. “Explosion” Issue Workflow
Pending Triage
Pending
QA Instructions
Pending Review
Pending
Issue Explosion
Pending Translations
Pending Verification
Create
Triage
Submit
Reviewed
Explode
Complete
Verify
…

117. “Explosion” Issue Workflow
Pending Triage
Pending
QA Instructions
Pending Review
Pending
Issue Explosion
Pending Translations
Pending Verification
Create
Triage
Submit
Reviewed
Explode
Complete
Verify
…
?

118. Translation Issue Workflow
Pending Acceptance
Accepted
Pending Verification
Pending Closure
Closed
Accept
Resolve
Verify
Close
1 issue per language

119. ● Custom issue types and workflows
○ clear communication
○ smooth transitions
How JIRA helped Translation

120. ● Custom issue types and workflows
● Automation → 1 issue per language
How JIRA helped Translation

121. ● Custom issue types and workflows
● Automation → 1 issue per language
● Subsequent iterations:
○ machine translation
○ translation service integration
How JIRA helped Translation

122. Iteration: Production Deploys

123. Iteration: Production Deploys
50/week

124. ● Require a lot of communication and
coordination between roles:
○ Create the release candidate deployable
○ Get it into a testing environment
○ Find the bugs
○ Fix the bugs
○ Declare it ready for production deploy
○ Deploy it to production
Iteration: Production Deploys

125. ● Require a lot of communication and
coordination between roles
● Many people care about current deploy status
Iteration: Production Deploys

126. ● Require a lot of communication and
coordination between roles
● Many people care about current deploy status
● We modeled it in JIRA.
Iteration: Production Deploys

127. Deploy Issue Workflow
Pre-Deploy
QA Ready
In QA
QA Complete
Deploy Ready
In Production
Ready
In QA
QA Complete
Ready
Deploy
Create

128. ● Clear communication, smooth hand-offs
● Transparency for all stakeholders
● Ability to look at trends in deploys
How JIRA helped Deploys

129. ● Proctor: data-driven framework for A/B testing
○ No full code deploy required for test updates
○ Lighter-weight, but still requires a process for
verification and deployment
Iteration: A/B Testing

130. ● Proctor: data-driven framework for A/B testing
○ No full code deploy required for test updates
○ Lighter-weight, but still requires a process for
verification and deployment
● We modeled it in JIRA.
Iteration: A/B Testing

131. Managing Data-Driven A/B Tests

132. Experiments and “Pro(ctor)Tests”
Service A
ProTest
Product
Experiment
Webapp X
ProTest
incorporates

133. ProTest Issue Workflow
Pending
Requirements
Pending Triage
Pending Acceptance
Accepted
Submit
Triage
Accept
Resolve
Create
…
QA Ready

134. Managing Data-Driven A/B Tests

135. Managing Data-Driven A/B Tests

136. Managing Data-Driven A/B Tests
QA Ready
In QA
In QA

137. ProTest Issue Workflow (cont.)
QA Ready
In QA
Production Ready
In Production
Pending Closure
Closed
Deploy
Verify
Deploy
Verify
Close
Resolve
…

138. ● Clear communication
● Tool integration saves time
● Manages risk through verification process
How JIRA helped A/B Testing

139. Indeed’s approach: Iterate. Iterate. Iterate.
Summary

140. Find the best process for your team’s success.
Iterate on process as necessary.
Stay focused on success.
Focus on Success

141. good luck,
and thanks!

142. Q & A
for more about Indeed Engineering:
engineering.indeed.com
for more about me:
indeed.com/me/JackHumphrey