Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Creating a MongoDB Based Logging System in a Webservice Heavy Environment

3,183 views

Published on

A talk about how to create a spohisticated logging system to help you track data from several webservices in one central place. How to make the logged data searchable, so it's possible to run analytics to spot bugs in the software. It's also a story about MongoDB adoption on Youwin.com - an online bookmaker, where we integrate 30+ webservices (payment providers, game providers, etc.).

  • Be the first to comment

Creating a MongoDB Based Logging System in a Webservice Heavy Environment

  1. 1. Creating MongoDB Based Logging System in Webservice Heavy Environment Arkadiusz Goralski @MongoDB London 2013piątek, 29 marca 13
  2. 2. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server npiątek, 29 marca 13
  3. 3. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server n Gaming Providers Sportsbook Slots Live Mini betting games Casino Poker Live Account dealer Managementpiątek, 29 marca 13
  4. 4. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server n Gaming Providers Payment Providers Sportsbook Slots Skrill Ukash Ecocard Live Mini Neteller Paysafe Instadebit betting games Bank Casino Poker Visa Ecocard transfers Live Account Mastercard Envoy Webmoney dealer Managementpiątek, 29 marca 13
  5. 5. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server n Gaming Providers Payment Providers Other Subsystems Bonus Common Sportsbook Slots Skrill Ukash Ecocard engine Wallet Live Mini Neteller Paysafe Instadebit Price feeds Mobile API betting games Bank Casino Poker Visa Ecocard Anti fraud Geo IP transfers Live Account Mastercard Envoy Webmoney dealer Managementpiątek, 29 marca 13
  6. 6. Different dialects SOAP JSON RPC Web REST JSON XML RPC Application REST XML HTTP Callspiątek, 29 marca 13
  7. 7. How to keep sanity? • why the deposit failed for user ... ? • user ... claims he didn’t place this bet? • can you give me a number of users that ... ? • can you tell me what user ... was doing on ... ? • why user ... couldn’t log in? • why customer ... couldn’t open an account?piątek, 29 marca 13
  8. 8. Why MongoDB is a good fit • schemaless • fire and forget • scalable and easy to replicate • capped collections & tailable cursor • centralized • easy remote accesspiątek, 29 marca 13
  9. 9. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server npiątek, 29 marca 13
  10. 10. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server n Replica Set Payment Providers MongoDB MongoDB MongoDB MongoDB (host 1) (host 2) (host 3) (hidden)piątek, 29 marca 13
  11. 11. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server n Replica Set Payment Providers MongoDB MongoDB MongoDB MongoDB (host 1) (host 2) (host 3) (hidden) Log Viewer RealPayment Providers time Simple Simple Real time log viewer queries reports alerts Secondary preferredpiątek, 29 marca 13
  12. 12. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server n Replica Set Payment Providers MongoDB MongoDB MongoDB MongoDB (host 1) (host 2) (host 3) (hidden) Log Viewer Data mining RealPayment Providers time Simple Simple Real time Payment Providers Heavy Analytics log viewer queries reports alerts queries Secondary preferred Hidden node onlypiątek, 29 marca 13
  13. 13. Log entry schemapiątek, 29 marca 13
  14. 14. Log entry schema Prepare web service requestpiątek, 29 marca 13
  15. 15. Log entry schema Prepare web service request JSON-ize Send itpiątek, 29 marca 13
  16. 16. Log entry schema Prepare web service request JSON-ize Send it JSON-ize Get the resultpiątek, 29 marca 13
  17. 17. Log entry schema Prepare web service request JSON-ize Send it JSON-ize Get the result Was is successfulpiątek, 29 marca 13
  18. 18. Log entry schema Prepare web service request JSON-ize Send it JSON-ize Get the result Was is successful Proceed with Save it executionpiątek, 29 marca 13
  19. 19. So youre logging data to MongoDB, now what? • all data is searchable (including web service specific data) • you can use the aggregation framework • trace user sessions across different web services • just sit and watch the logspiątek, 29 marca 13
  20. 20. Detect anomalies in the data • realtime issue analysis • subsystem failures • fraud • traffic anomaliespiątek, 29 marca 13
  21. 21. Data mining • create a replica for analytics • analyze customer behaviour • spot fraudulent activity • create reports • proactive support • deployment analytics • police enquiriespiątek, 29 marca 13
  22. 22. Share the logs • don’t be a human interface to MongoDB • the more you share the more time is saved, your time • just a simple query interface is enough • give separate node to marketing peoplepiątek, 29 marca 13
  23. 23. Push notifications When something goes really bad Alert Scenarios Email Tailable cursor Node.js Severity Text MongoDB app rating message Call people, its serious!piątek, 29 marca 13
  24. 24. Open Source Alternatives • GrayLog (uses MongoDB) • LogStash (MongoDB output plugins) • Fluent (MongoDB storage)piątek, 29 marca 13
  25. 25. Thank you Have fun with MongoDBpiątek, 29 marca 13

×