Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
VideoAuthorization
Fromchaosto
25msresponsetime
EmersonMacedo
@emerleite
https://emerleite.com
Contentauthorization
isalways
achallenge
VideoAuthorizationprocess
VideoAuthorizationprocess
Videoauthorization
mustbereally
veryfast
AuthorizationProvider
Rules
1-Userhasavalid
session
2-Userhasthechannelof
thisvideoinhisPayTV
subscription
3-Userparentalcontrol
matchesvideocontent
rating
OR...
1-Userhasavalid
session
2-Videochannelhasa
trialhappening
3-Userisabletojointhe
currenttrial
VideoAuthorizationprocess
responsetime>5seconds
CircuitBreakeropens
Nobodycanplay
anysubscriberonly
video
Weneedto
dosomething
aboutthis
1-splitthecircuitbreaker
Thecircuitbreakerwaslikethis
Thenewcircuitbreakerisperprovider
Acircuitopenforone
provider
shouldnotaffectany
otherproviders
2-OptimizeAuthProvider
GlobosatPlay
AuthorizationProvider
Usually
~500msresponsetime
Sometimes
5sto10sresponsetime
Videoauthorization
mustbereally
veryfast
NewRelicTransactionTrace
NewRelicTransactionTrace
2.1-Snapshotsome
externalservicesdata
GlobosatPlay
AuthorizationProvider
Aftereveryusersignin
BeforeUserInfo
Snapshotdata
AfterUserInfo
Snapshotdata
Onuservideoauthorization
BeforeProfileConfig
Snapshotdata
AfterProfileConfig
Snapshotdata
Onanyvideoauthorization
BeforeVideoinfo
Snapshotdata
AfterVideInfo
Snapshotdata
Intheworstscenario
WehaveTHREEexternalcalls
Inthebestscenario
WehaveNOexternalcalls
2.2-Besafewithall
externalservicescalls
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
ContentGateway1
1
https://goo.gl/fg5Nzf
2.3-Cacheloggedrequests
whenit'ssafe
Loggedrequest-Success
Loggedrequest-Failure
NginxCacheConfig
WithNginxcache
Wedid
averyintesivejob
andreducedthe
responsetime
Usually
~25msresponsetime
Usually
~25msresponsetime
BUT...
Sometimes
5sto10sresponsetime
Sometimes
5sto10sresponsetime
3-Takealookatour
infrastructure
3.1-MongoDB
MongoDBhasabad
reputationinsoftware
developercommunity
EverybodyBlamesMongoeverytime2
2
http://rhaas.blogspot.com.br/2014/04/why-clock-is-ticking-for-mongodb.html
EverybodyBlamesMongoeverytime3
3
http://cryto.net/~joepie91/blog/2015/07/19/why-you-should-never-ever-ever-use-mongodb/
Databasechangeis
anarchitecturaldecision
architecturaldecisions
arehardandvery
expensivetochange
Sometimes
5sto10sresponsetime
Strangequeryconvultions
Tracesconfirmsconvultions
Logsconfirmsconvultions
Wecheckedallindexes
andfoundnothing
hurtingoursystem
3.2-Servers
CPUWaitconfirmsconvultions
Loadaverageconfirmsconvultions
MongoDBdocsconfirmsthat
Somethingwaswrong4
4
https://docs.mongodb.org/manual/administration/production-notes/#remote-files...
MongoDBusersgroupconfirmsthat
Somethingwaswrong5
5
https://groups.google.com/forum/#!msg/mongodb-user/Kd85b2HHVn8/7SnwTyeQ...
WeremovedNFS
fromourMongodb
serverssetup
convultionsstopped
Usually
~25msresponsetime
Usually
~25msresponsetime
AND...
Sometimes
100msto500msresponsetime
LessonsLearned
1-Godeepintoyourproblem
2-Don'tpanic
3-Mostofthetime
thesolutionisnot
torewritefromscratch
Questions?
QCon SP 2016 - Video Authorization: from chaos to 25ms response time
QCon SP 2016 - Video Authorization: from chaos to 25ms response time
QCon SP 2016 - Video Authorization: from chaos to 25ms response time
QCon SP 2016 - Video Authorization: from chaos to 25ms response time
QCon SP 2016 - Video Authorization: from chaos to 25ms response time
QCon SP 2016 - Video Authorization: from chaos to 25ms response time
QCon SP 2016 - Video Authorization: from chaos to 25ms response time
QCon SP 2016 - Video Authorization: from chaos to 25ms response time
QCon SP 2016 - Video Authorization: from chaos to 25ms response time
Upcoming SlideShare
Loading in …5
×

QCon SP 2016 - Video Authorization: from chaos to 25ms response time

1,461 views

Published on

This talk was about our Video Playback Authorization for Globosat Play PayTV Subscribers.

Published in: Technology
  • Be the first to comment

QCon SP 2016 - Video Authorization: from chaos to 25ms response time

  1. 1. VideoAuthorization Fromchaosto 25msresponsetime
  2. 2. EmersonMacedo @emerleite https://emerleite.com
  3. 3. Contentauthorization isalways achallenge
  4. 4. VideoAuthorizationprocess
  5. 5. VideoAuthorizationprocess
  6. 6. Videoauthorization mustbereally veryfast
  7. 7. AuthorizationProvider Rules
  8. 8. 1-Userhasavalid session
  9. 9. 2-Userhasthechannelof thisvideoinhisPayTV subscription
  10. 10. 3-Userparentalcontrol matchesvideocontent rating
  11. 11. OR...
  12. 12. 1-Userhasavalid session
  13. 13. 2-Videochannelhasa trialhappening
  14. 14. 3-Userisabletojointhe currenttrial
  15. 15. VideoAuthorizationprocess
  16. 16. responsetime>5seconds CircuitBreakeropens
  17. 17. Nobodycanplay anysubscriberonly video
  18. 18. Weneedto dosomething aboutthis
  19. 19. 1-splitthecircuitbreaker
  20. 20. Thecircuitbreakerwaslikethis
  21. 21. Thenewcircuitbreakerisperprovider
  22. 22. Acircuitopenforone provider shouldnotaffectany otherproviders
  23. 23. 2-OptimizeAuthProvider
  24. 24. GlobosatPlay AuthorizationProvider
  25. 25. Usually ~500msresponsetime
  26. 26. Sometimes 5sto10sresponsetime
  27. 27. Videoauthorization mustbereally veryfast
  28. 28. NewRelicTransactionTrace
  29. 29. NewRelicTransactionTrace
  30. 30. 2.1-Snapshotsome externalservicesdata
  31. 31. GlobosatPlay AuthorizationProvider
  32. 32. Aftereveryusersignin
  33. 33. BeforeUserInfo Snapshotdata
  34. 34. AfterUserInfo Snapshotdata
  35. 35. Onuservideoauthorization
  36. 36. BeforeProfileConfig Snapshotdata
  37. 37. AfterProfileConfig Snapshotdata
  38. 38. Onanyvideoauthorization
  39. 39. BeforeVideoinfo Snapshotdata
  40. 40. AfterVideInfo Snapshotdata
  41. 41. Intheworstscenario WehaveTHREEexternalcalls
  42. 42. Inthebestscenario WehaveNOexternalcalls
  43. 43. 2.2-Besafewithall externalservicescalls
  44. 44. ContentGateway1 1 https://goo.gl/fg5Nzf
  45. 45. ContentGateway1 1 https://goo.gl/fg5Nzf
  46. 46. ContentGateway1 1 https://goo.gl/fg5Nzf
  47. 47. ContentGateway1 1 https://goo.gl/fg5Nzf
  48. 48. ContentGateway1 1 https://goo.gl/fg5Nzf
  49. 49. ContentGateway1 1 https://goo.gl/fg5Nzf
  50. 50. ContentGateway1 1 https://goo.gl/fg5Nzf
  51. 51. ContentGateway1 1 https://goo.gl/fg5Nzf
  52. 52. ContentGateway1 1 https://goo.gl/fg5Nzf
  53. 53. ContentGateway1 1 https://goo.gl/fg5Nzf
  54. 54. ContentGateway1 1 https://goo.gl/fg5Nzf
  55. 55. ContentGateway1 1 https://goo.gl/fg5Nzf
  56. 56. ContentGateway1 1 https://goo.gl/fg5Nzf
  57. 57. ContentGateway1 1 https://goo.gl/fg5Nzf
  58. 58. ContentGateway1 1 https://goo.gl/fg5Nzf
  59. 59. ContentGateway1 1 https://goo.gl/fg5Nzf
  60. 60. ContentGateway1 1 https://goo.gl/fg5Nzf
  61. 61. ContentGateway1 1 https://goo.gl/fg5Nzf
  62. 62. 2.3-Cacheloggedrequests whenit'ssafe
  63. 63. Loggedrequest-Success
  64. 64. Loggedrequest-Failure
  65. 65. NginxCacheConfig
  66. 66. WithNginxcache
  67. 67. Wedid averyintesivejob andreducedthe responsetime
  68. 68. Usually ~25msresponsetime
  69. 69. Usually ~25msresponsetime
  70. 70. BUT...
  71. 71. Sometimes 5sto10sresponsetime
  72. 72. Sometimes 5sto10sresponsetime
  73. 73. 3-Takealookatour infrastructure
  74. 74. 3.1-MongoDB
  75. 75. MongoDBhasabad reputationinsoftware developercommunity
  76. 76. EverybodyBlamesMongoeverytime2 2 http://rhaas.blogspot.com.br/2014/04/why-clock-is-ticking-for-mongodb.html
  77. 77. EverybodyBlamesMongoeverytime3 3 http://cryto.net/~joepie91/blog/2015/07/19/why-you-should-never-ever-ever-use-mongodb/
  78. 78. Databasechangeis anarchitecturaldecision
  79. 79. architecturaldecisions arehardandvery expensivetochange
  80. 80. Sometimes 5sto10sresponsetime
  81. 81. Strangequeryconvultions
  82. 82. Tracesconfirmsconvultions
  83. 83. Logsconfirmsconvultions
  84. 84. Wecheckedallindexes andfoundnothing hurtingoursystem
  85. 85. 3.2-Servers
  86. 86. CPUWaitconfirmsconvultions
  87. 87. Loadaverageconfirmsconvultions
  88. 88. MongoDBdocsconfirmsthat Somethingwaswrong4 4 https://docs.mongodb.org/manual/administration/production-notes/#remote-filesystems
  89. 89. MongoDBusersgroupconfirmsthat Somethingwaswrong5 5 https://groups.google.com/forum/#!msg/mongodb-user/Kd85b2HHVn8/7SnwTyeQKsEJ
  90. 90. WeremovedNFS fromourMongodb serverssetup
  91. 91. convultionsstopped
  92. 92. Usually ~25msresponsetime
  93. 93. Usually ~25msresponsetime
  94. 94. AND...
  95. 95. Sometimes 100msto500msresponsetime
  96. 96. LessonsLearned
  97. 97. 1-Godeepintoyourproblem
  98. 98. 2-Don'tpanic
  99. 99. 3-Mostofthetime thesolutionisnot torewritefromscratch
  100. 100. Questions?

×