The document discusses how a video user profile service originally built in Ruby on Rails was scaled using Elixir. It initially increased resources and used containers, improving performance by 32% and 12% respectively. It was found one endpoint handled 80% of traffic. Rewriting this in Elixir improved performance by 95% and reduced containers by 35%. The results showed Elixir was able to significantly improve scalability and metrics for the growing user profile API.

1. Using Elixir to scale
Video User Profile
Service
Emerson Macedo
@emerleite

3. CONTEXT
User Profile API was developed in 2012.
It’s a well written Ruby on Rails
application , responsible for track
logged user actions

4. CONTEXT
User Profile API was developed in 2012.
It’s a well written Ruby on Rails
application , responsible for track
logged user actions

5. CONTEXT
Our Video Player POST Watched
Percentage to our Endpoint so we can
provide Keep Watching Percentage
to our Logged Users. It does it
every 10 seconds

6. CONTEXT
Our Video Player POST Watched
Percentage to our Endpoint so we can
provide Keep Watching Percentage
to our Logged Users. It does it
every 10 secondsx
x

7. CONTEXT
Globo is making
Binge Watching
experiments combined
with Online First for
new TV Shows
TV Shows

8. CONTEXT
Prepare all Video
Applications for the
Olympics

9. Online First
Binge Watching

10. THE PROBLEM
2013
10k
The throughput increased
between 2013 and 2016

11. 2013 2016
10k
60k
THE PROBLEM
The throughput increased
between 2013 and 2016

12. 2013 2016
10k
60k
THE PROBLEM
The throughput increased
between 2013 and 2016 ~600%

13. 2013 2016
10k
60kTHE PROBLEM
It was hard to predict the new
Appication Throughput
2016 / 2
?

14. INFRASTRUCTURE
2 bare metals, each one
with 24 CPUs and 64GB
of RAM

15. THE PROBLEM
The average response time
was good, but percentiles
were hurting the application
Avg
70ms

16. 70ms
THE PROBLEM
The average response time
was good, but percentiles
were hurting the application
99
2.5s
95
1s
Avg

17. FIRST CHANGE
We increased the
computational resources
from 2 to 4 bare metals

18. FIRST CHANGE
We increased the
computational resources
from 2 to 4 bare metals

19. FIRST CHANGE
We increased the
computational resources
from 2 to 4 bare metals Each one with
24 CPUs and
64GB of RAM

20. 70ms
99
2.5s
95
1s
Avg
FIRST CHANGE
This change improved
metrics by ~32%

21. FIRST CHANGE
This change improved
metrics by ~32%
9995
1.7s
0.7s
Avg
47ms

22. FIRST CHANGE
This change improved
metrics by ~32%
9995
1.7s
0.7s
Avg
47ms
~32%
better

23. SECOND CHANGE
We decided to try a deployment with
Tsuru containers with auto scaling, each
one with 1-4 vCPU and 2GB of RAM

24. SECOND CHANGE
We decided to try a deployment with
Tsuru containers with auto scaling, each
one with 1-4 vCPU and 2GB of RAM
93 containers

25. SECOND CHANGE
9995
1.7s
0.7s
Avg
47ms
Migrate to containers improved
metrics by ~12%

26. SECOND CHANGE
9995
1.5s
0.5s
Avg
41ms
Migrate to containers improved
metrics by ~12%

27. SECOND CHANGE
9995
1.5s
0.5s
Avg
41ms
Migrate to containers improved
metrics by ~12%
~12%
better

28. CONTEXT
Our Video Player POST Watched
Percentage to our Endpoint so we can
provide Keep Watching Percentage
to our Logged Users. It does it
every 10 seconds

29. CONTEXT
Our Video Player POST Watched
Percentage to our Endpoint so we can
provide Keep Watching Percentage
to our Logged Users. It does it
every 10 secondsx
x

30. Online First
Binge Watching

31. ARCHITECTURAL OVERVIEW
User Profile API is a
classical Ruby on Rails
application, which also
uses Resque for
background jobs

32. ARCHITECTURAL OVERVIEW
User Profile API is a
classical Ruby on Rails
application, which also
uses Resque for
background jobs

33. ARCHITECTURAL OVERVIEW
User Profile API is a
classical Ruby on Rails
application, which also
uses Resque for
background jobs
BLOCK
BLOCK
BLOCK

34. ARCHITECTURAL OVERVIEW
User Profile API is a
classical Ruby on Rails
application, which also
uses Resque for
background jobs

35. CONTAINER DISTRIBUTION
Our containers was distributed for the
Rails App, Resque Workers and
Resque Scheduler
93 containers

36. 30 containers63 containers

37. 30 containers63 containers
Resque
Rails

38. THIRD CHANGE
We saw that just ONE
endpoint was responsible for
80%, of application
throughput

39. THIRD CHANGE
We saw that just ONE
endpoint was responsible for
80%, of application
throughput

40. THIRD CHANGE
x
x
We saw that just ONE
endpoint was responsible for
80%, of application
throughput

41. CONTEXT
Our Video Player POST Watched
Percentage to our Endpoint so we can
provide Keep Watching Percentage
to our Logged Users. It does it
every 10 seconds

42. CONTEXT
Our Video Player POST Watched
Percentage to our Endpoint so we can
provide Keep Watching Percentage
to our Logged Users. It does it
every 10 secondsx
x

43. THIRD CHANGE
We saw that just ONE endpoint was
responsible for 80%, of application
throughput

44. THIRD CHANGE
We saw that just ONE endpoint was
responsible for 80%, of application
throughput
Pareto

45. THIRD CHANGE
We saw that just ONE endpoint was
responsible for 80%, of application
throughput
Pareto
20% of effort will solve
80% of the problem

46. THIRD CHANGE
We rewrote the Ruby POST
endpoint from scratch to an
Elixir version

47. THIRD CHANGE
We rewrote the Ruby POST
endpoint from scratch to an
Elixir version

48. THIRD CHANGE
We rewrote the Ruby POST
endpoint from scratch to an
Elixir version

49. WHY ELIXIR
Elixir is being the
cutting-edge for the
Ruby community

50. WHY ELIXIR
Elixir is being the
cutting-edge for the
Ruby community

51. WHY ELIXIR
Elixir is being the
cutting-edge for the
Ruby community
x

52. WHY ELIXIR
Elixir is being the
cutting-edge for the
Ruby community

53. WHY ELIXIR
Elixir is being the
cutting-edge for the
Ruby community
x

54. WHY ELIXIR
Elixir is being the
cutting-edge for the
Ruby community

55. WHY ELIXIR
Elixir is being the
cutting-edge for the
Ruby community
x

56. WHY ELIXIR
Elixir is the language
Java programmers
were looking when
they choose Ruby

57. WHY ELIXIR
Elixir is the language
Java programmers
were looking when
they choose Ruby
x
x
x

58. WHY ELIXIR
Elixir generates Erlang
byte code and runs
on BEAM VM which
has 30 years of
development

59. WHY ELIXIR
Elixir generates Erlang
byte code and runs
on BEAM VM which
has 30 years of
development
x

60. THIRD CHANGE
We rewrote the Ruby POST
endpoint from scratch to an
Elixir version

61. THIRD CHANGE
We rewrote the Ruby POST
endpoint from scratch to an
Elixir version

62. THIRD CHANGE
We rewrote the Ruby POST
endpoint from scratch to an
Elixir version

63. THIRD CHANGE
We rewrote the Ruby POST
endpoint from scratch to an
Elixir version
SAME APPLICATION

64. THIRD CHANGE
We rewrote the Ruby POST
endpoint from scratch to an
Elixir version
SAME APPLICATION
ERL PROCESS
ERL PROCESS

65. THIRD CHANGE
The result was a starting
point to change the app to
CQRS Architecture

66. THIRD CHANGE
The result was a starting
point to change the app to
CQRS Architecture
COMMAND
QUERY

67. THIRD CHANGE
9995
1.5s
0.5s
Avg
41ms
Migrate to Elixir improved
metrics by ~95%

68. THIRD CHANGE
9995
30ms
15ms
Avg
4ms
Migrate to Elixir improved
metrics by ~95%

69. THIRD CHANGE
9995
30ms
15ms
Avg
4ms
Migrate to Elixir improved
metrics by ~95%
~95%
better

70. 30 containers63 containers
Resque
Rails

71. THIRD CHANGE
Migrate to Elixir reduced
containers by ~35%
3 containers30 containers
ElixirRuby

72. THIRD CHANGE
Migrate to Elixir reduced
containers by ~35%
33 containers

73. TOOLS
We chosse phoenix
framework to create
our Elixir API and
we’re using many other
community libs

74. TOOLS
We chosse phoenix
framework to create
our Elixir API and
we’re using many other
community libs
Ecto HTTPoison
Exrm CacheX
GenRetry FakeServer
Corsica

75. PROBLEMS
MongoDB Driver did not has
Replica Sets support. We had
to implement it

76. PROBLEMS
MongoDB Driver did not has
Replica Sets support. We had
to implement it
http//github.com/emerleite/mongox
http//github.com/emerleite/mongox_ecto

77. PROBLEMS
Elixir did not has NewRelic support.
We need to create an ad-hoc
implementation using Exometer

78. PROBLEMS
Elixir did not has NewRelic support.
We need to create an ad-hoc
implementation using Exometer
https://github.com/Feuerlabs/exometer_core

79. 2013 2016
10k
60kFINAL RESULT
After the Olympics and with
Binge Watching, the
throughput increased ~50%

80. 2013 2016
10k
60kFINAL RESULT
After the Olympics and with
Binge Watching, the
throughput increased ~50%
2016 / 2
90k

81. 70ms
99
2.5s
95
1s
Avg
FINAL RESULT
Migrate to Elixir improved
metrics by ~95%

82. FINAL RESULT
9995
30ms
15ms
Avg
4ms
Migrate to Elixir improved
metrics by ~95%

83. FINAL RESULT
9995
30ms
15ms
Avg
4ms
Migrate to Elixir improved
metrics by ~95%
~95%
better

84. Perguntas?
Emerson Macedo
@emerleite
https://blog.emerleite.com