DevOps hat uns viel gelehrt: Jedes Commit wird automatisch integriert, getestet und in eine Stagingumgebung installiert. Und einen Knopfdruck später geht das Release dann live. Aber leider funktioniert das und vieles mehr, was uns in Zeiten von DevOps als Normal erscheint, nicht bei mobilen Applikationen. Wie kann ich also schnell meine mobile Software anpassen, wenn der Betreiber des App Stores erst tagelang testen will? Wie kann ich Konfiguration anpassen, wenn die App auch offline laufen soll? Und wie funktioniert Fehlersuche, wenn die Daten auf Millionen von mobilen Clients verteilt sind? Beim Betrieb von mobilen Spielen mit Millionen täglicher Nutzer standen wir genau diesen Fragen gegenüber. Der Vortrag wird darlegen, welche Antworten wir darauf gefunden haben.

1. Jesper Richter-Reichhelm (@jrirei) | Wooga
DevOps goes Mobile

3. November 2013
!? … …
…
!

5. November 2013
…
!?
…
… …
…

6. “In recognition of your
incredible efforts and
achievements, I’m happy to
announce that we’re
extending the Thanksgiving
holiday this year.”
- Tim Cook

7. November 2013
!?
… … …
… …
…

8. Crashing
15%
Not affected
50%
Old version
35%
>200,000 users
affected

10. November 2013 (on web)
• 5 hours crashing, not 5 days!
• 9 days live earlier

13. • 2009 focus on Facebook
• 2011 first iOS game
• 2013 mobile first strategy
Wooga

14. “You build it, you run it.
- Werner Vogels

15. • Agile admins
• Faster releases
• Virtualization
• Automation tools
Some say DevOps is about

16. • Big Picture Mindset
• Learning together
• Reality Feedback
• => Collaboration
I say it’s about

17. Mobile is
different
especially for apps

18. DevOps is
different
for Mobile

19. Problems &
Solutions

20. Mobile
Network

21. • use async. communication
• use compression
• batch of requests
• monitor network differences
Unreliable network

22. • queue monitoring calls locally
• queue server updates locally
• store state local
Offline mode

23. Runs on a
Device

25. WORD

29. • consistency problems
• merge/conflict handling
• versioning of data
• esp. tricky when allowing usage of
multiple devices
Local storage

30. • Different mindsets of mobile devs
• Reproduce error vs. log lookup
• Using an error analytics service
• not only crashes
• must work offline!
• no OS related crashes
• just memory warning
Log file access

31. Deploy w/o
Control

32. Jenkins

34. • Live with it
• Use online config
• AB testing
• balancing changes
• careful of what is allowed
• use local storage to cache
Apple Test Cycle

35. • Only pulls
• Users can deactivate
No push updates

37. 0,0%
20,0%
40,0%
60,0%
80,0%
4.4.2 4.4.1 4.4 4.3.x 4.2.x 4.1.x 4.0.x 3.x 2.x 1.x
5% 3%

38. 0,0%
20,0%
40,0%
60,0%
80,0%
2014 ... ... ... ... ... ... 2013 2012 2011
5% 3%

39. • Only pulls
• Users can deactivate
• Or force users to update in app
• check server for newer version
• Online config helps a lot
No push updates

40. WORD

41. Canary
Testing

44. Wooga
Solutions

45. Cross
Platform

49. Test Pool

52. • No manual builds
• Keep dSYM file
• Jailbreaked iPhone
• Copy live to staging
Lessons learned

53. WORD
Simple Backend Services

54. Key Value
Store

57. Config
Service

59. AB Test
Service

61. • It’s hell - live with it
• form factor, OS version
• Restrict devices (camera, min. OS)
• cross platform dev.
• many local devices
• Apple Enterprise acc.
• complex tool chain

62. • It’s hell - live with it
• form factor, OS version
• Restrict devices (camera, min. OS)
• cross platform dev.
• many local devices
• Apple Enterprise acc.
• complex tool chain

66. Error
Analytics

67. 0
25.000
50.000
75.000
100.000
Users Errors
Version 1 Version 2

68. 0
1.250
2.500
3.750
5.000
Users Errors
Version 1 Version 2

69. 0%
2,5%
5%
7,5%
10%
Users Errors Affected Users
Version 1 Version 2

70. SDK

72. Wrap up

73. Mobile is
different
especially for apps

74. DevOps is
different
for Mobile

75. Dev + Ops
is needed

76. Continuous
Delivery
is good

77. Be able to
react
always

78. Questions?
@jrirei
http://wooga.com/jobs