Building the Real-Time Performance Panel
•Download as PPTX, PDF•
0 likes•487 views
Speaker: Jonathan Balsano, Software Engineer, Cloud, MongoDB Level: 200 (Intermediate) Track: How We Build MongoDB If you’ve ever had to deploy a tool within an internal network that you didn’t (and even if you did) set up you’re probably aware of the many pitfalls that can be caused by firewalls, load balancers, and latency. When the MongoDB Cloud Team implemented the Real Time Performance Panel, a tool that lets you monitor operations and performance in your deployment in real time, we leveraged our existing tools and architecture along with some new optimizations to provide a live monitoring experience with minimal latency that works within a variety of network architectures. In this talk, you will learn about the overall architecture of Cloud and Ops Manager, along with the specific optimizations you can use to provide a real time experience for clients when building your own monitoring solutions.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Building the Real-Time Performance Panel
Similar to Building the Real-Time Performance Panel (20)
More from MongoDB
More from MongoDB (20)
Recently uploaded
Recently uploaded (20)
Building the Real-Time Performance Panel
- 1. #MDBW17 An Engineering Journey through Constraints and Portability BUILDING THE REAL TIME PERFORMANCE PANEL
- 2. #MDBW17
- 3. #MDBW17
- 4. #MDBW17
- 5. #MDBW17
- 6. #MDBW17
- 7. #MDBW17
- 8. #MDBW17
- 9. #MDBW17
- 10. #MDBW17
- 11. #MDBW17
- 12. #MDBW17
- 13. #MDBW17 Highest Granularity: 10 seconds
- 14. #MDBW17 Refresh Rate: 1 minute
- 15. #MDBW17 A BETTER EXPERIENCE • 10x Higher Granularity • Live Experience
- 18. #MDBW17 GETTING METRIC DATA FROM MONGODB TO THE CLOUD Understanding Our Architecture
- 21. #MDBW17 UNDERSTANDING OUR ARCHITECTURE DB DB DB Client Customer Network Agent Application Server
- 22. #MDBW17 UNDERSTANDING OUR ARCHITECTURE L O A D B A L A N C E R Application Server Application Server DB DB DB Client Customer Network Agent
- 23. #MDBW17 GETTING METRIC DATA FROM MONGODB TO THE CLOUD Understanding Our Architecture Ideal Real Time Setup
- 24. #MDBW17 IDEAL REAL TIME SETUP L O A D B A L A N C E R Application Server Application Server DB DB DB Client Customer Network Agent I have data! Pub Sub I have data!
- 25. #MDBW17 HTTP VS. WEBSOCKETS HTTP 1.1 Websockets Request/Response Request from Client Client can push, Server cannot Stateless Two way connection Initial Request from Client Server and Client can push Stateful
- 26. #MDBW17 Pub Sub WEBSOCKETS WITH LOAD BALANCER L O A D B A L A N C E R Application Server Application Server DB DB Client Customer Network Agent
- 27. #MDBW17 GETTING METRIC DATA FROM MONGODB TO THE CLOUD Understanding Our Architecture Ideal Real Time Setup The Reality
- 28. #MDBW17 Pub Sub ATLAS PROCESS LOCATION L O A D B A L A N C E R Application Server Application Server DB DB Client Customer Network Agent
- 29. #MDBW17 Pub Sub CLOUD MANAGER PROCESS LOCATION L O A D B A L A N C E R Application Server Application Server DB DB Client Customer Network Agent
- 30. #MDBW17 Pub Sub OPS MANAGER PROCESS LOCATION L O A D B A L A N C E R Application Server Application Server DB DB Client Customer Network Agent L O A D B A L A N C E R
- 31. #MDBW17 GETTING METRIC DATA FROM MONGODB TO THE CLOUD Understanding Our Architecture Ideal Real Time Setup The Reality Our Solution
- 33. #MDBW17 TRADITIONAL POLLING Server ClientAgent Got my data?Hm, nope How about now? Nope, nada Hey, here’s the data! And now? Oh, yeah! Here it is!
- 34. #MDBW17 LONG POLLING Server ClientAgent Got my data? Hey, here’s the data! It’s not here. Nap time! Still not here. Nap time! Oh, yeah! Here it is!
- 35. #MDBW17 SOLUTION: LONG POLLING L O A D B A L A N C E R Application Server Application Server DB DB DB Client Customer Network Agent
- 37. #MDBW17 HOW DO WE MAKE THIS... L O A D B A L A N C E R Application Server Application Server DB DB DB Client Customer Network Agent
- 38. #MDBW17 Pub Sub ...BEHAVE LIKE THIS? L O A D B A L A N C E R Application Server Application Server DB DB Client Customer Network Agent
- 39. #MDBW17 HOW TO MAKE "REAL TIME" MEAN "REAL TIME" Time to First Point
- 40. #MDBW17 PREVIOUS POLLING Server ClientAgent t = 0 t = 30
- 41. #MDBW17 NEW POLLING Server ClientAgent t = 0 t = 30
- 42. #MDBW17 AGENT INACTIVE/ACTIVE Inactive Active Agent Inactive 30s Polling Active 1s Polling
- 43. #MDBW17 TIME TO FIRST POINT 30s Agent Polling + 100ms Network Latency +1s Second Point 100ms Network Latency +
- 44. #MDBW17 THREE CONNECTION SOLUTION • Agent Polling • Client Long Polling • Client Heartbeat
- 45. #MDBW17 AGENT AND CLIENT STATES Inactive ActiveDormant Agent Data Heartbeat Inactive 30s Polling Not Polling Not Polling Active 1s Polling Long Polling 30s Polling Dormant 1s Polling Not Polling 30s Polling
- 46. #MDBW17 HOW TO MAKE "REAL TIME" MEAN "REAL TIME" Time to First Point Clock Skew
- 47. #MDBW17 NAIVE APPROACH Server ClientAgent t = 0 t = 3000 t = 2000 t = 1000 sampleTime = 1500 sampleTime = 2500 Got anything with sampleTime > 1200? Got anything with sampleTime > 2200?
- 48. #MDBW17 CLOCK SKEW Server ClientAgent t = 0 t = 3000 t = 2000 t = 1000
- 49. #MDBW17 CLOCK SKEW Server ClientAgent t = 0 t = 3000 t = 2000 t = 1000 sampleTime = 800 sampleTime = 1800 Got anything with sampleTime > 1200?
- 50. #MDBW17 SOLUTION Server ClientAgent t = 0 t = 3000 t = 2000 t = 1000 sampleTime = 800 sampleTime = 1800 Got anything? sampleTime = 800 Got anything with sampleTime > 800?
- 51. #MDBW17 HOW TO MAKE "REAL TIME" MEAN "REAL TIME" Time to First Point Clock Skew Respond ASAP
- 52. #MDBW17 RESPOND ASAP Server ClientAgent t = 0 t = 2 t = 1
- 53. #MDBW17 RESPOND ASAP Server ClientAgent t = 0 t = 2 t = 1
- 54. #MDBW17 Last Point was at t = 1020 Add 1000ms for agent Eh, 100ms for db time sounds good RESPOND ASAP Server ClientAgent t = 0 t = 3000 t = 2000 t = 1000 tReceived = 1020 1020 1000 + 100 2120 What time is it? Oh, 1600 so… 2120 -1600 520 Sleep for 520msNailed it! tReceived = 1020
- 55. #MDBW17 HOW TO MAKE "REAL TIME" MEAN "REAL TIME" Time to First Point Clock Skew Respond ASAP Keep it Smooth
- 56. #MDBW17
- 57. #MDBW17 TIMING OUR "SWING" • Ball = Data • Bat = Animation
- 58. #MDBW17 TOO FAST
- 59. #MDBW17 TOO SLOW
- 60. #MDBW17 SOLUTION? • Rolling Average • Assume 1000ms • Track time • Average actual time
- 61. #MDBW17
- 64. #MDBW17 HOW WE DID IT • Process • Work inside constraints • Optimize, optimize, optimize! • Smooth it over on the client
- 65. #MDBW17 Cloud Software Engineer, MongoDB JONATHAN BALSANO @jrbalsano