Christian Legnitto manages mobile release engineering at Facebook. He oversees releases of Facebook's iOS and Android apps, ensuring they are deployed every 4 weeks through an automated continuous integration and deployment process. Previously he held release management roles at Mozilla for Firefox and at Apple for OS X updates.

1. Mobile @ Facebook
Christian Legnitto (christianl@fb.com)
Mobile release engineering

2. Who is this guy?

3. Who is this guy?
Facebook
• Manage Mobile Release Engineering
• I’m hiring
• Run the iOS and Android releases
• Ran Facebook.com web pushes for ~3 months

4. Who is this guy?
Facebook
• Manage Mobile Release Engineering
• I’m hiring
• Run the iOS and Android releases
• Ran Facebook.com web pushes for ~3 months
Mozilla
• Firefox release manager
• Helped design & implement 6 week releases
• Desktop ( 3 platforms ) & mobile ( Android )
• Still a community member

5. Who is this guy?
Facebook
• Manage Mobile Release Engineering
• I’m hiring
• Run the iOS and Android releases
• Ran Facebook.com web pushes for ~3 months
Mozilla
• Firefox release manager
• Helped design & implement 6 week releases
• Desktop ( 3 platforms ) & mobile ( Android )
• Still a community member
Apple
• Jaguar → Lion updates
• OS X pre-release updates
• Safari, Xcode, Java, iApps
• Build system, scrubber, PM tools

7. 874,000,000 MAUs
507,000,000 DAUs

9. iPads &
iPhones
native
& web
supports
iOS6+

11. native
& web
+1M beta
testers
supports
Froyo+
15 apks
+20K
alpha
testers

14. How we USED to develop

15. Mobile core team owned mobile
Desktop Web
iOS
Group
messages
Group
messages
Group
messages
Chat
Messages team
Android
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
...
...
...
Events team
Photos team
Android core team
iOS core team

16. Doesn’t SCALE

17. Mobile core team owned mobile
Desktop Web
iOS
Group
messages
Group
messages
Group
messages
Chat
Messages team
Android
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
Events team
Photos team
Android core team
iOS core team

18. Mobile core team owned mobile
Desktop Web
Events team
Photos team
Android core team
iOS core team
Group
messages
Group
messages
Group
messages
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
20
iOS
Chat
Messages team
Android
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
10
30
20
20

19. Mobile core team owned mobile
Desktop Web
Photos team
Android core team
iOS core team
(numbers made up)
Group
messages
Group
messages
Group
messages
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Photo albums
Photo albums
Photo picker
Events team
20
iOS
Chat
Messages team
Android
Photo picker
Photo picker
10
30
20
20

20. Mobile core team owned mobile
Desktop Web
Events team
Photos team
Android core team
iOS core team
Group
messages
Group
messages
Group
messages
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
20
iOS
Chat
Messages team
Android
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
10
30
20
20

21. Mobile core team owned mobile
Desktop Web
Photos team
Android core team
iOS core team
Group
messages
Group
messages
Chat
Events team
iOS
Group
messages
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Messages team
Android
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
10
30
20
20
20

22. Mobile core team owned mobile
Desktop Web
iOS
Group
messages
Group
messages
Group
messages
Chat
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Messages team
Android
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
Events team
Photos team
Android core team
iOS core team
30
20
20
20
10

23. Mobile core team owned mobile
Desktop Web
iOS
Group
messages
Group
messages
Group
messages
Chat
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Messages team
Android
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
Events team
Photos team
Android core team
iOS core team
20
20
20
60
30
10

24. Mobile core team owned mobile
Desktop Web
iOS
Group
messages
Group
messages
Group
messages
Chat
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Messages team
Android
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
20
60
30
10
20
Events team
Photos team
Android core team
iOS core team
20

25. Mobile core team owned mobile
Desktop Web
iOS
Group
messages
Group
messages
Group
messages
Chat
Messages team
Android
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
20
60
30
10
20
20
Events team
Photos team
Android core team
iOS core team

26. Doesn’t produce the BEST product

27. Mobile core team owned mobile
Desktop Web
iOS
Group
messages
Group
messages
Group
messages
Chat
Messages team
Android
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
Events team
Photos team
Android core team
iOS core team

28. Mobile core team owned mobile
Desktop Web
iOS
Group
messages
Group
messages
Group
messages
Chat
Messages team
Android
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
Events team
Photos team
Android core team
iOS core team
product experts

29. Mobile core team owned mobile
Desktop Web
iOS
Group
messages
Group
messages
Group
messages
Chat
Messages team
Android
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
Events team
Photos team
Android core team
iOS core team
product experts
platform experts

30. How we USED to ship

31. Feature-based releases

32. Feature-based releases
Decide on features

33. Feature-based releases
Decide on features
Furiously work on them

34. Feature-based releases
Decide on features
Furiously work on them
Test

35. Feature-based releases
Decide on features
Furiously work on them
Test
Ship when all features are done

36. All features are PRIORITY #1

37. Decide on features
Desktop Web
Group
messages
Chat
Upcoming
events
Birthdays
Photo albums
Photo picker
...

38. Decide on features
Desktop Web
1. Emoji
2. Green dot
3. Naming convos
Group
messages
Chat
Upcoming
events
Birthdays
Photo albums
Photo picker
...

39. Decide on features
Desktop Web
1. Emoji
2. Green dot
3. Naming convos
Group
messages
1. Reply inline
2. Birthday reminder
3. Gifts integration
Upcoming
events
Chat
Birthdays
Photo albums
Photo picker
...

40. Decide on features
Desktop Web
1. Emoji
2. Green dot
3. Naming convos
Group
messages
1. Reply inline
2. Birthday reminder
3. Gifts integration
Upcoming
events
1. Automatic upload
2. Tag suggestions
3. Change cover photo
Chat
Birthdays
Photo albums
Photo picker
...

41. Decide on features
Desktop Web
Android
1. Emoji
2. Green dot
3. Naming convos
Group
messages
Group
messages
Chat
Chat
1. Reply inline
2. Birthday reminder
3. Gifts integration
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Photo albums
Photo albums
Photo picker
Photo picker
...
...
1. Automatic upload
2. Tag suggestions
3. Change cover photo

42. Decide on features
Desktop Web
Android
1. Emoji
2. Green dot
3. Naming convos
Group
messages
Group
messages
Chat
Chat
1. Reply inline
2. Birthday reminder
3. Gifts integration
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Photo albums
Photo albums
Photo picker
Photo picker
...
...
1. Automatic upload
2. Tag suggestions
3. Change cover photo
1. Emoji
1. Reply inline
1. Automatic upload

43. Decide on features
Desktop Web
Android
1. Emoji
2. Green dot
3. Naming convos
Group
messages
Group
messages
Chat
Chat
1. Reply inline
2. Birthday reminder
3. Gifts integration
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Photo albums
Photo albums
Photo picker
Photo picker
...
...
1. Automatic upload
2. Tag suggestions
3. Change cover photo
1. Emoji
1. Reply inline
1. Automatic upload
2. Green dot
2. Birthday reminder
2. Tag suggestions

44. Decide on features
Desktop Web
Android
1. Emoji
2. Green dot
3. Naming convos
Group
messages
Group
messages
Chat
Chat
1. Reply inline
2. Birthday reminder
3. Gifts integration
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Photo albums
Photo albums
Photo picker
Photo picker
...
...
1. Automatic upload
2. Tag suggestions
3. Change cover photo
1. Emoji
1. Reply inline
1. Automatic upload
2. Green dot
2. Birthday reminder
2. Tag suggestions
3. Naming convos
3. Gifts integration
3. Change cover photo

45. The market and company
CHANGES RAPIDLY

46. Decide on features
Android
Group
messages
Chat
Upcoming
events
Birthdays
Photo albums
Photo picker
...
1. Emoji
2. Reply inline
3. Automatic upload
4. Green dot
5. Birthday reminder
6. Tag suggestions
7. Naming convos
8. Gifts integration
9. Change cover photo

47. Decide on features
Android
Group
messages
Chat
Upcoming
events
Birthdays
Photo albums
Photo picker
...
1. Emoji
2. Reply inline
3. Automatic upload
4. Green dot
5. Birthday reminder
6. Tag suggestions
7. Naming convos
8. Gifts integration
9. Change cover photo

48. Decide on features
Android
Group
messages
Chat
Upcoming
events
Birthdays
Photo albums
Photo picker
...
1. Emoji
2. Reply inline
3. Automatic upload
4. Green dot
5. Birthday reminder
6. Tag suggestions
7. Naming convos
8. Gifts integration
9. Change cover photo

49. Decide on features
Android
Group
messages
Chat
Upcoming
events
Birthdays
Photo albums
Photo picker
...
1. Emoji
2. Reply inline
3. Automatic upload
4. Green dot
5. Birthday reminder
6. Tag suggestions
7. Naming convos
8. Gifts integration
9. Change cover photo

50. Decide on features
Android
Group
messages
Chat
Upcoming
events
Birthdays
Photo albums
Photo picker
...
1. Emoji
2. Reply inline
3. Automatic upload
4. Green dot
5. Birthday reminder
6. Tag suggestions
7. Naming convos
8. Gifts integration
9. Change cover photo

51. Decide on features
Android
Group
messages
Chat
Upcoming
events
Birthdays
Photo albums
Photo picker
...
1. Emoji
2. Reply inline
3. Automatic upload
4. Green dot
5. Birthday reminder
6. Tag suggestions
7. Naming convos
8. Gifts integration
9. Change cover photo

52. Decide on features
Android
Group
messages
Chat
Upcoming
events
Birthdays
Photo albums
Photo picker
...
1. Emoji
2. Reply inline
3. Automatic upload
4. Green dot
5. Birthday reminder
6. Tag suggestions
7. Naming convos
8. Gifts integration
9. Change cover photo

53. Decide on features
Android
Group
messages
Chat
Upcoming
events
Birthdays
Photo albums
Photo picker
...
1. Emoji
2. Reply inline
3. Automatic upload
4. Green dot
5. Birthday reminder
6. Tag suggestions
7. Naming convos
8. Gifts integration
9. Change cover photo

54. Decide on features
Android
Group
messages
Chat
Upcoming
events
Birthdays
Photo albums
Photo picker
...
1. Emoji
2. Reply inline
3. Automatic upload
4. Green dot
5. Birthday reminder
6. Tag suggestions
7. Naming convos
8. Gifts integration
9. Change cover photo

55. Don’t know when code SHIPS

56. Feature-based releases
Can’t judge risk vs reward

57. Feature-based releases
Can’t judge risk vs reward
80% done?

58. Feature-based releases
Can’t judge risk vs reward
80% done?
Good enough

59. Confuses DEV and RELEASE

60. Development process
Create the best product possible
→ Focus on improving

61. Development process
Create the best product possible
→ Focus on improving
Release process
Ship finished work in a timely manner
→ Focus on not regressing

62. Took lessons learned from
DESKTOP WEB and applied
them to MOBILE

63. How we develop NOW

64. Old and busted
Desktop Web
iOS
Group
messages
Group
messages
Group
messages
Chat
Messages team
Android
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
...
...
...
Events team
Photos team
Android core team
iOS core team

65. New hotness
Desktop Web
iOS
Group
messages
Group
messages
Group
messages
Chat
Messages team
Android
Chat
Chat
Upcoming
events
Upcoming
events
Upcoming
events
Birthdays
Birthdays
Birthdays
Photo albums
Photo albums
Photo albums
Photo picker
Photo picker
Photo picker
...
...
...
Events team
Photos team
Android core team
iOS core team

66. Engineers committing mobile code
iOS and Android

67. How we ship NOW

68. features
quality
schedule

69. features
quality
schedule
pick 2

70. features
quality
schedule
pick 2

71. Date-based releases
Rule #1
We ship on time

72. Date-based releases
Rule #2
Make users
no worse off

73. Date-based releases
Rule #3
There’s always
the next one

74. Date-based releases
Rule #4
Retreat to safety

75. tiv
na
Automated cut every 4 weeks
e
m
ile
ob
4 weeks of development
4 weeks of development
master
3.5 weeks of stabilization
release branch
RC

76. p
to
sk
de
Automated cut every 1 week
w
eb
1 week of development
1 week of development
trunk
Stabilize
Release branch
release branch
Weekly
Twice Daily

77. tiv
na
Automated cut every 4 weeks
e
m
ile
ob
4 weeks of development
4 weeks of development
master
3.5 weeks of stabilization
release branch
RC

78. tiv
na
Dogfooding
e
m
ile
ob
master
release branch

79. How we write CODE

80. Source control
Git
>100,000 commits
>150,000 files
>300 engineers

81. Code review
Phabricator

83. Continuous integration
Buildbot
Builds
Lint / static analysis
Tests
Facebook
i386
Facebook
arm
Messenger
i386
Messenger
arm
...
regex
clang
android
findbugs
...
OCUnit
snapshot
JUnit
robolectric
...
https://github.com/facebook/ios-snapshot-test-case

84. Continuous integration
Buildbot
Before diff
Create diff
(optional)
(mandatory)
Diff updated Landing queue
(mandatory)
(optional)
Git commit
(mandatory)

85. Continuous integration
Mac minis

86. Android

87. Building & running tests
Buck - build tool
•
•
•
•
Fast
•
Distributed Cassandra build cache
•
buckd for background builds
Scalable and extensible
“buck clean is a bug”
Android and generic Java support

88. Building & running tests
Buck - build tool
•
•
•
•
Fast
•
Distributed Cassandra build cache
•
buckd for background builds
Scalable and extensible
“buck clean is a bug”
Android and generic Java support

89. Building & running tests
Buck - build tool
“Gerrit Code Review recently switched its build process from
Maven to Buck, reducing build time by 67-98% while improving
build reproducibility, maintainability and developer happiness.
The Gerrit project migrated itself out of the Maven tarpit into a
”
better build.
- Shawn Pearce (Google)

90. Building & running tests
Buck - join us!
Start using Buck:
http://facebook.github.io/buck/
Get help in the discussion group:
https://groups.google.com/forum/#!forum/buck-build
File bugs and submit patches on GitHub:
https://github.com/facebook/buck

91. iOS

92. Building & running tests
xctool - build tool
Makes iOS & Mac continuous integration easier.
• CLI tool, runs tests just as Xcode.app would
• Can parallelize test runs
• Can emit JSON objects for every build/test event
• Reporters for Phabricator, JUnit XML
• Attractive, concise output

93. Building & running tests
xctool - “pretty” reporter

94. Building & running tests
xctool - “json-stream” reporter

95. Building & running tests
xctool - how it works
xctool
xcodebuild
xcodebuild-shim.dylib
Injected
otest
(SenTestingKit)
xctest
(XCTest)
test-shim.dylib
Injected
test-shim.dylib
Injected

96. Building & running tests
xctool - join us!
On GitHub:
github.com/facebook/xctool
On Homebrew:
brew install xctool
Also available on Travis-CI

97. Crash reports
atos

98. Crash reports
atosl

99. Crash reports
atosl
github.com/facebook/atosl

100. Changed how we DEVELOP
Changed how we SHIP
Changed how we WRITE CODE
Kept our CULTURE

101. Facebook is mobile
Learn from our experience and use our tools

102. Links to mentioned awesomeness
Contact me
Buck
christianl@fb.com, https://fb.me/legnitto, @LegNeato
http://github.com/facebook/buck
XCTool
http://github.com/facebook/xctool
atosl
http://github.com/facebook/atosl
Snapshot Test http://github.com/facebook/ios-snapshot-test-case

103. Mobile @ Facebook
Christian Legnitto (christianl@fb.com)
Mobile release engineering