Алексей Ященко, Engineering Manager Grammarly

1. Grammarly
engineering

2. Who am I?
Aleksey Yashchenko
Software Engineer at Grammarly
@tuxslayer

3. The Company
Ukrainian startup
60 engineers
Offices in Kyiv and San-Francisco

4. The Product
Grammar
Spell
Style
Plagiarism
… checker

5. 4.5M weekly active users on Chrome and growing
100k words/sec
400 instances on AWS
10M+ WebSocket connections/day
The Product

6. Dozens of services in
Java, Lisp, Scala and Python
6 UI applications in
JS, TypeScript and F#
The Product

7. Our Apps
Funnels on www.grammarly.com
Browser Plugins: Chrome, Safari, FF
Web App
M$ Office Add-in
Desktop Editor

8. Extension =>

10. <= Synonym cards
Mistake cards =>

11. Low development pace
Buggy releases =>
regular outages =>
high downtime
The Problem
…but why?

12. Horizontal and service teams
&
Too much manual work
The Problem

13. The Problem
Backend
QAFrontEnd
Ops
silos, tasks ping-pong, no DRIs

14. Horizontal
teams
The Problem
Buggy releases,
regular outages,
high downtime
Slow
development
Manual work

15. Our Solution
Organization Technical

16. Our Solution
Engineers at
Cross-functional Teams
Fully own services and releases, while
Tools solve all the hassle

17. Cross-func Teams
QA => SE in Test
QA Engineers =>
Feature Teams
Solution /1
Continuos Delivery
Monitoring
ORG: crossfit TECH: CI

18. Solution /1
Feature Cross-functional Teams

19. no-QA: From QA to SET
Catch bugs
Build for quality
Write code
… well… catch bugs

20. CI/CD is easy…
Feature branches
CI Server (TeamCity)
Unit / Integration / Load / … tests
Continuous Deployment

21. CI/CD is easy…
Feature branches
CI Server (TeamCity)
Unit / Integration / Load …tests
Continuous Deployment
Doesn’t work

22. Monitor all the things!

23. Monitor all the things!
Make it an engineering practice
Measure everything, choose what to
monitor then
Validate monitoring data!

24. Monitor all the things!
Logs (servers, FE, access logs) =>
Kafka (queuing) =>
Logstash (processing) =>
ElasticSearch (storage/querying) =>
Kibana (UI)

25. Monitor all the things!
Metrics (timers, counters, gauges ) =>
StatsD (aggregation) =>
Graphite (storage, querying) =>
Graphana (UI)

26. Monitor all the things!
Logs & Metrics (storage) =>
Sensu Client (monitoring, alerting) =>
OpsGeniue (night calls)
More: www.slideshare.net/tuxSlayer/frontend-monitoring-grammarly

27. Results /1
Zero ping-pong time between teams
Developers can release “at will”
QA-people are involved from day 1
Everybody writes tests
Everybody* can fix a bug
* almost

28. Results /1
Time to release from 1-2h to 1-10 min
Rollback instead of fixing =>
Short outages
Whole team does not get distracted by
accidents

29. Release Calendar

30. Questions /1
?

31. Platform Team
Embedded Ops
in Teams
Solution /2
docker
rocker
rocker-compose
ORG: no-Ops TECH: containers

32. Platform Team
Not a “service” team
PaaS
Strategic rather than tactical
Software Engineers + DevOps
Embedded Ops for Feature Teams

33. Solution /2

34. The Funnel
Huge project
Multiple releases per day
Static site generator
1k lines of nginx configs
Static content
Nginx configs
Nginx
+
+

35. The Funnel
Static content
Nginx configs
Nginx
+
Dev Team Ops Kitchen
blah-blah
Chef

36. Container Platform
The Funnel
Static content
Nginx configs
Nginx
Responsible Dev Team Happy Ops Kitchen
ship it!
Container
Container
Container
…

37. Docker Rocks… almost
Custom tools:
rocker (building)
rocker-compose (deployment)
More: https://goo.gl/wXUbHa

38. Results /2
Wait no more for DevOps tasks
Reproducible environments
Safe deployments
Configuration as a code (versioning, automation,
templating etc)
Multi-service deployments (e.g. DB+Redis+App+Nginx)
… and…

39. A Victory
Engineers at
Cross-functional Teams
Fully own services and releases, while
Tools solve all the hassle

40. Take aways
Tech should support org
Org should enforce [hi-]tech
Responsibility matters
Automate everything

41. Questions?

42. More Info
Rocker: Breaking the limits of Dockerfile
How We Deploy Containers at Grammarly
Frontend Monitoring at Grammarly

43. Thank you!
Aleksey Yashchenko
Software Engineer at Grammarly
@tuxslayer