We've learned a lot through doing DevOps: Every commit is automatically integrated, tested, and deployed to a staging environment. And then it only takes one push of a button and the release goes live... Unfortunately, it's not as simple anymore when operating mobile applications: How can you quickly update your mobile software when the app store provider wants to test your software first for a few days? How can you update your configuration when your app can run offline? And how do you track down errors when the data is distributed to millions of mobile clients? Those were just some of the challenges we encountered during the operation of mobile games with millions of daily users. In this talk we will talk about the solutions we have found to address them.

1. DevOps goes
Mobile
Jesper Richter-Reichhelm
@jrirei

2. A story about a
nice little game …
… that was very
successful.

3. November 2013
!? … …
…
!
But the new
version had a
serious bug.
Apple just
needed 4 days of
testing - quite ok!

4. November 2013
!?
…
… …
…
Thanksgiving
day in US while we
were waiting…

5. “In recognition of your
incredible efforts and
achievements, I’m happy to
announce that we’re
extending the Thanksgiving
holiday this year.”
- Tim Cook
Quite cool move
by Apple’s CEO…

6. … but it did add
our tension as we
were waiting.

7. November 2013
!?
… … …
… …
…
Apple took five
days to approve our
patch.

8. Crashing
15%
Not affected
50%
Old version
35%
>200,000 users
affected
A bug like that
could kill a game -
or a company.
We were lucky
that only ‘few’ users
were affected.

9. 8000 crash
reports within the
first hour…

10. November 2013 (on web)
5 hours crashing, not 5 days!
Reaction
times on mobile
are much worse
than on web.

12. 270 employees
100 engineers with
only 3 managers

13. We started 2009
on Facebook with
Flash games
Since 2013 we
focus on mobile
first.

14. Independent teams
FE Dev
BE Dev
Art
Product
One team per
game with lots of
autonomy

15. “You build it, you run it.
- Werner Vogels
We are not the
only ones working
like that

16. Classic way
!
!
Ops
!
Dev
Devs want to
change, Ops want
to avoid risk.
This is a classic
conflict.

17. DevOps way
!
Dev
!
!
Ops
DevOps is about
bringing Dev and
Ops together.
Silos don’t help
in making the
product.

18. Wooga way
!
Dev
Ops
!
Dev
Ops
In small
teams dev and ops
can even be in one
person.

19. • Agile admins
• Faster releases
• Virtualization
• Automation tools
Some say DevOps is about
All true, all
good, but a bit
missing the point.

20. I say it’s about
Feedback loops
back from ops to
dev.

21. Cooperation
Collaboration
Feedback loops
I say it’s about

22. Mobile is
different
especially for apps

23. DevOps is
different
for Mobile

24. Challenges

25. “There are three challenges:
mobile, devices, and not
having control.”
- Me

26. Mobile
Network
Network
conditions change
while the device
moves.

27. Latency
Bandwidth
Offline mode

28. Runs on a
Device

29. Operating system
Form factor
Device specs
So many devices
to make it work
on…

30. But even more
important …

31. … the devices
are not within our
reach.

32. Local storage
Logfile access
Offline handling
You have a
distributed
network of millions
of computers.
Now keep up the
data integrity! :-)
And it’s hard to
see what did go
wrong…

33. Deploy w/o
Control
So many
practices like CD to
be in control…
… and most does
not work on
mobile.

34. Even harder on
iOS than on
Android

35. Mandatory tests
Strict rules
Only full deploys

36. We WANT canary
testing!

37. Staged rollout
Reduce risk
Fast feedback

38. Staged rollout
Reduce risk
Fast feedback
Google provides
staged rollouts.
Great! :-)

40. Pull updates
User can interfere
But we still have
no full control over
updates.

41. Pull updates
User can interfere

42. 0,0%
20,0%
40,0%
60,0%
80,0%
4.4.2 4.4.1 4.4 4.3.x 4.2.x 4.1.x 4.0.x 3.x 2.x 1.x
5% 3%
Version
distribution of our
game Diamond
Dash (iOS)

43. 0,0%
20,0%
40,0%
60,0%
80,0%
2014 ... ... ... ... ... ... 2013 2012 2011
5% 3%
~10% play
on versions that
are older than 18
months.

44. WORD
Jelly Splash iOS
has a similar
problem.
Even forced
updates can be
partially avoided.

45. Wooga
Solutions

46. Cross
Platform

47. HTML5 not viable
Unity
Apportable
Others? We have
some solutions
but we are not
happy yet.
So sad that
HTML5 is not yet
good enough…

48. Devices

49. Restrict devices
Test pool
Internal ‘app store’
How to live with
the pain…

50. WORD
For Apple you
add ‘fake’
requirements.
Google provides
a way to restrict to
devices.

51. Still we need
manual test on
physical devices…

52. Just kidding. :-)
… and lot’s of
alcohol afterwards.

53. Jenkins
CI helps (of course)

54. Internal app store
helps, too.
Makes it easier
to distribute and
test internally.

56. No manual builds
Keep dSYM file
Jailbreaked iPhone
Copy Live 2 Test
Lessons
learned from the
Thanksgiving bug:

57. WORD
Simple Backend Services
Following the EC2
example …
… Wooga
now provides
internal services to
mobile games.

58. SBS way
!
Game
!
Game
!
SBS
!
SBS
Game debs write
90% of the SDKs.
SBS provides a
simple REST API to
its services.

59. SDK

60. Async. calls
Offline handling
Local storage
Conflict assist
Encryption
Most
network problems
are handled by the
SDK.

61. Key Value
Store

62. REST API
Authorisation
ETags
ETags are the
key to conflict
resolution.

63. Clients are
responsible for
solving conflicts.
Updates without
a valid Etag are
invalid.
On every update
a new ETag is
created.

64. Config
Service

65. Variable config
Offline mode
Environments
AB testing
Online
configuration
allows AB testing.
And that enables
feature switching!

66. Simple
configuration with
basic version
control…
… and easy
support of
environments.

67. AB tests can be
tested and run
individually.

68. • It’s hell - live with it
• form factor, OS version
• Restrict devices (camera, min. OS)
• cross platform dev.
• many local devices
• Apple Enterprise acc.
• complex tool chain Simple definition
how big the groups
are.

69. • It’s hell - live with it
• form factor, OS version
• Restrict devices (camera, min. OS)
• cross platform dev.
• many local devices
• Apple Enterprise acc.
• complex tool chain Groups define
variations of the
basic config.

70. The BI team
provides tools for
analysis.

71. But analysts
within game teams
are responsible.

72. With all this
feature switching
becomes easy.

73. Error
Analytics

74. Real time
Crash reports
Segmentation
Normalisation
Staged rollout are
only useful …
… if you have the
right tooling.

75. 0
25.000
50.000
75.000
100.000
Users Errors
Version 1 Version 2
Which version is
better?

76. 0
1.250
2.500
3.750
5.000
Users Errors
Version 1 Version 2
Version 1 has
more errors …
… but it also has
more users.

77. 0%
2,5%
5%
7,5%
10%
Users Errors Affected Users
Version 1 Version 2
Normalisation
makes that easy.
The tradeoff is
having a lot of more
traffic.

78. Wrap up

79. Mobile is
different
especially for apps

80. DevOps is
different
for Mobile

81. Continuous
Delivery
helps

82. November 2013
5 hours crashing, not 5 days!
This can kill you.

83. “MTTR > MTBF”
- John Allspaw
The Mean Time
To React is more
important …
… than the
Mean Time
Between Failures.

84. “MTTR is more important than
MTBF (for most types of F)!”
- John Allspaw
If you are dead,
you cannot recover
(of course).

86. Dev + Ops
is needed
You need both -
Dev and Ops - to
succeed.

87. Be able to
react fast
even on Mobile
The one thing
that matters most
in operation.

88. Questions?
@jrirei
http://wooga.com/jobs