• Like

Creating a MongoDB Based Logging System in a Webservice Heavy Environment

  • 1,532 views
Uploaded on

A talk about how to create a spohisticated logging system to help you track data from several webservices in one central place. How to make the logged data searchable, so it's possible to run …

A talk about how to create a spohisticated logging system to help you track data from several webservices in one central place. How to make the logged data searchable, so it's possible to run analytics to spot bugs in the software. It's also a story about MongoDB adoption on Youwin.com - an online bookmaker, where we integrate 30+ webservices (payment providers, game providers, etc.).

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,532
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
25
Comments
0
Likes
6

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Creating MongoDB Based Logging System in Webservice Heavy Environment Arkadiusz Goralski @MongoDB London 2013piątek, 29 marca 13
  • 2. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server npiątek, 29 marca 13
  • 3. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server n Gaming Providers Sportsbook Slots Live Mini betting games Casino Poker Live Account dealer Managementpiątek, 29 marca 13
  • 4. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server n Gaming Providers Payment Providers Sportsbook Slots Skrill Ukash Ecocard Live Mini Neteller Paysafe Instadebit betting games Bank Casino Poker Visa Ecocard transfers Live Account Mastercard Envoy Webmoney dealer Managementpiątek, 29 marca 13
  • 5. Web service Heavy Environment Web Application Web Server 1 Web Server 2 ... Web Server n Gaming Providers Payment Providers Other Subsystems Bonus Common Sportsbook Slots Skrill Ukash Ecocard engine Wallet Live Mini Neteller Paysafe Instadebit Price feeds Mobile API betting games Bank Casino Poker Visa Ecocard Anti fraud Geo IP transfers Live Account Mastercard Envoy Webmoney dealer Managementpiątek, 29 marca 13
  • 6. Different dialects SOAP JSON RPC Web REST JSON XML RPC Application REST XML HTTP Callspiątek, 29 marca 13
  • 7. How to keep sanity? • why the deposit failed for user ... ? • user ... claims he didn’t place this bet? • can you give me a number of users that ... ? • can you tell me what user ... was doing on ... ? • why user ... couldn’t log in? • why customer ... couldn’t open an account?piątek, 29 marca 13
  • 8. Why MongoDB is a good fit • schemaless • fire and forget • scalable and easy to replicate • capped collections & tailable cursor • centralized • easy remote accesspiątek, 29 marca 13
  • 9. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server npiątek, 29 marca 13
  • 10. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server n Replica Set Payment Providers MongoDB MongoDB MongoDB MongoDB (host 1) (host 2) (host 3) (hidden)piątek, 29 marca 13
  • 11. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server n Replica Set Payment Providers MongoDB MongoDB MongoDB MongoDB (host 1) (host 2) (host 3) (hidden) Log Viewer RealPayment Providers time Simple Simple Real time log viewer queries reports alerts Secondary preferredpiątek, 29 marca 13
  • 12. MongoDB deployment Web Application Web Server 1 Web Server 2 ... Web Server n Replica Set Payment Providers MongoDB MongoDB MongoDB MongoDB (host 1) (host 2) (host 3) (hidden) Log Viewer Data mining RealPayment Providers time Simple Simple Real time Payment Providers Heavy Analytics log viewer queries reports alerts queries Secondary preferred Hidden node onlypiątek, 29 marca 13
  • 13. Log entry schemapiątek, 29 marca 13
  • 14. Log entry schema Prepare web service requestpiątek, 29 marca 13
  • 15. Log entry schema Prepare web service request JSON-ize Send itpiątek, 29 marca 13
  • 16. Log entry schema Prepare web service request JSON-ize Send it JSON-ize Get the resultpiątek, 29 marca 13
  • 17. Log entry schema Prepare web service request JSON-ize Send it JSON-ize Get the result Was is successfulpiątek, 29 marca 13
  • 18. Log entry schema Prepare web service request JSON-ize Send it JSON-ize Get the result Was is successful Proceed with Save it executionpiątek, 29 marca 13
  • 19. So youre logging data to MongoDB, now what? • all data is searchable (including web service specific data) • you can use the aggregation framework • trace user sessions across different web services • just sit and watch the logspiątek, 29 marca 13
  • 20. Detect anomalies in the data • realtime issue analysis • subsystem failures • fraud • traffic anomaliespiątek, 29 marca 13
  • 21. Data mining • create a replica for analytics • analyze customer behaviour • spot fraudulent activity • create reports • proactive support • deployment analytics • police enquiriespiątek, 29 marca 13
  • 22. Share the logs • don’t be a human interface to MongoDB • the more you share the more time is saved, your time • just a simple query interface is enough • give separate node to marketing peoplepiątek, 29 marca 13
  • 23. Push notifications When something goes really bad Alert Scenarios Email Tailable cursor Node.js Severity Text MongoDB app rating message Call people, its serious!piątek, 29 marca 13
  • 24. Open Source Alternatives • GrayLog (uses MongoDB) • LogStash (MongoDB output plugins) • Fluent (MongoDB storage)piątek, 29 marca 13
  • 25. Thank you Have fun with MongoDBpiątek, 29 marca 13