Speaker: Dmitry Unkovsky, Software Engineer at Grammarly Summary: We will tell the story of DevOps at Grammarly since 2013. We’ll talk about how we managed infrastructure growth while keeping up with the rapid pace of product development; what worked for us and what did not, and why; and what it’s like to make technical choices as an engineer at our company. We will share our current vision and future plans.

2. Table of Contents
● Story of DevOps at Grammarly 2013-2018
○ skim over 2013-2016
○ pause at present
● What’s learned
● Future plans

3. 2013 2014 2015 2016 2017 2018
2013: <Teamwork>

4. 2013 2014 2015 2016 2017 2018

5. 2013 2014 2015 2016 2017 2018

6. 2013 2014 2015 2016 2017 2018

7. 2013 2014 2015 2016 2017 2018

8. 2013 2014 2015 2016 2017 2018

9. 2013 2014 2015 2016 2017 2018

10. 2013 2014 2015 2016 2017 2018

11. 2013 2014 2015 2016 2017 2018

12. 2013 2014 2015 2016 2017 2018

13. 2013 2014 2015 2016 2017 2018

14. 2013: Results
● + Product is experimenting and moving forward
○ - QA is a bottleneck
● - Too many outages
○ +- Shared responsibility
● - This won’t scale
2013 2014 2015 2016 2017 2018

15. Wisdom Buffer
● Teamwork saves when nothing else does!
● Shared responsibility is good, but also bad
2013 2014 2015 2016 2017 2018

16. 2014: TODO
● Company initiatives
○ Improve stability
○ Release more often
○ Start testing Freemium
● What we are going to do
○ Monitoring for developers
○ Further integrate Chef
○ Autoscaling
2013 2014 2015 2016 2017 2018

17. 2014: <Monitoring>
2013 2014 2015 2016 2017 2018

18. 2014: What we are doing
● zabbix -> graphite+sensu
● widely use ELK
● datadog
● monitoring for devs
● update graphite and sensu; remove datadog
● rundeck and ansible – improve deployments
● widely use microservices
● QAs go wild towards CD
2013 2014 2015 2016 2017 2018

19. 2013 2014 2015 2016 2017 2018

20. 2013 2014 2015 2016 2017 2018

21. 2013 2014 2015 2016 2017 2018

22. 2014: Results
● + Better service introspection
● + Improved stability
● + Improved releases frequency somewhat
● - Ops are a bottleneck
○ Many things cannot be done without Ops
○ Devs can’t control production runtime env
○ No autoscaling
● +- Chef helped Ops, but hasn’t enable DevOps
2013 2014 2015 2016 2017 2018

23. Wisdom Buffer
● Teamwork saves when nothing else does!
● Shared responsibility is good, but also bad
● Ownership really works
2013 2014 2015 2016 2017 2018

24. 2015: TODO
● Company initiatives
○ Freemium launch, prepare to scale
○ Full-stack product teams (ownership)
● What we are going to do
○ Moving Ops responsibilities to Dev teams. But how?
■ Provision instances, deploy services
■ Configure Balancers
○ Autoscaling
2013 2014 2015 2016 2017 2018

25. 2015: Platform Team
● Ops team –> Platform team
● Service team –> Software Dev team
● Platform is a product
● We make tools
○ and best practices
○ Netflix and Etsy – inspiration
● Hire: +Dev skills/background
● Still lots of Ops stuff to do
2013 2014 2015 2016 2017 2018

26. 2015: <Containers>
2013 2014 2015 2016 2017 2018

27. 2015: What we are doing
● Testing Docker
● Make Rocker and Rocker-compose
○ features for nice integrations with various workflows and CI
○ support soft-restarting etc
○ get rid of bash scripts on CI
○ speed up builds
● After we dockerized 3-4 services, experiment went out of control
○ Teamwork again :)
● Create instances through Rundeck
2013 2014 2015 2016 2017 2018

28. 2013 2014 2015 2016 2017 2018

29. 2013 2014 2015 2016 2017 2018

30. 2013 2014 2015 2016 2017 2018

31. 2013 2014 2015 2016 2017 2018

32. 2015: Results
● QA team distributed between Dev teams
● Developers have control
○ critical path of Ops responsibility shifted to Dev teams
● Platform focused on tools
● Devs still depend on Platform team
○ Provision non-standard instances
○ Load balancer configuration
○ Other AWS resources configuration such as Security Groups, S3 buckets, etc.
2013 2014 2015 2016 2017 2018

33. Wisdom Buffer
● Teamwork saves when nothing else does!
● Shared responsibility is good, but also bad
● Ownership really works
● Containers are a good abstraction
2013 2014 2015 2016 2017 2018

34. 2016: TODO
● Company initiatives
○ Scaling
○ Moving faster, again :)
○ Full copy of Grammarly production env
● What we are going to do
○ Make Dev teams self-sufficient (finally)
○ Autoscaling
2013 2014 2015 2016 2017 2018

35. 2016: <Terraform>
2013 2014 2015 2016 2017 2018

36. 2016: What we are doing
2013 2014 2015 2016 2017 2018
● ELK and Graphite rehack
● Tried Rundeck + Terraform
● CLI tool that wraps terraform
● Facilitated set up of a completely separate
environment
● nginx –> ALB(ELB)

37. 2013 2014 2015 2016 2017 2018

38. 2013 2014 2015 2016 2017 2018

39. 2013 2014 2015 2016 2017 2018

40. 2013 2014 2015 2016 2017 2018

41. 2016: Results
● + Terraform is cool
○ - collaborative work with it is not
○ - version migrations
● - IAM access challenges – “all or none”
● - Developers overloaded with infrastructure work
○ Mess around the system (plus many more micro-services)
● - Stability issues with on-prem ELK and Graphite
○ As a result of focus on tools approach
● Hard to hire for our team
○ Ops+Dev profile is very rare
○ You hire Devs and then they go to other teams :)
2013 2014 2015 2016 2017 2018

42. Wisdom Buffer
● Teamwork saves when nothing else does!
● Shared responsibility is good, but also bad
● Ownership really works
● Containers are a good abstraction
● Developers are not Ops x2
● Tools are hard to maintain
2013 2014 2015 2016 2017 2018

43. 2017: TODO
● Company initiatives
○ Scale more
○ Stabilize monitoring infrastructure (ELK, Graphite)
○ Control production access
● What we are going to do
○ Separate AWS accounts to solve IAM access problems
○ Rehack ELK and later Graphite
○ Autoscaling
2013 2014 2015 2016 2017 2018

44. 2017: What we are doing
2013 2014 2015 2016 2017 2018
● Separate AWS accounts
○ support in tools, rehack infra, redeploy everything
● Rehack ELK
○ realized the cost of appropriate implementation
● Research SaaS log solutions and choose one
● Use everything AWS – don’t support instances
○ move to Auroradb
○ tried to use AWS ELK
○ EMR instead of Spark
● Tools around system integrity (AWS Config, etc.)

45. 2017: Results
● AWS separation
● ELK –> Sumo Logic
● Suspend Rocker development
● Approach (framework) for vendor evaluation
● Software Dev team –> Cloud Infra team
2013 2014 2015 2016 2017 2018

46. Wisdom Buffer
● Teamwork saves when nothing else does!
● Shared responsibility is good, but also bad
● Ownership really works
● Containers are a good abstraction
● Developers are not Ops
● Tools are hard to maintain
● Use everything AWS
● Vendors are hard to choose from
● Platform architects solutions, not develops tools
2013 2014 2015 2016 2017 2018

47. Wrap-up Wisdom Buffer
● Basics first; “Autoscale” last
● Challenge your requirements harder
● As we grow up, tools and approaches mature
● Doing things properly requires team capacity
● We are not Netflix to maintain our tools
● Do unique things and not generic that exist

48. TODO
● Company initiatives
○ Scaling more
○ Maturity everywhere
● What we are going to do
○ New deployments, including autoscaling
○ - tools
○ Move Infra to cloud solutions, SaaS vendors
○ Mature processes and Infra

49. Нам нужны Девопсы
Почитайте это:
goo.gl/pmpvQJ