Trading up: Adding Flexibility and Scalability to Bouygues Telecom with MongoDB

1,823 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,823
On SlideShare
0
From Embeds
0
Number of Embeds
926
Actions
Shares
0
Downloads
22
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Team like to try new things, why not used node ?
    Go quick
  • Scalable database, Scalable front-end but provisioning part will not scale for the customer intergation part if we keep doing our traditional parallelized batch
  • GO QUICK
    Storm well know now for real time processing but you can also use this for a batch
    It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate.
    Suits for real-time data acquisition treatment but also for batch

  • This is the new design of our system.
    POC tests where what we expected. We fell confident enougth to start the building part
  • Focus on three key parts of our project project
  • Trading up: Adding Flexibility and Scalability to Bouygues Telecom with MongoDB

    1. 1. Adding Flexibility and Scalability to Bouygues Telecom with MongoDB Pierre-Alban DEWITTE @__pad__ Bouygues Telecom MongoDB World – 06/24/2014
    2. 2. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Bouygues Telecom 13 M Customers
    3. 3. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Who is speaking ? @__pad__
    4. 4. Thanks !
    5. 5. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Agenda Adding Flexibility and Scalability to Bouygues Telecom with MongoDB 1.Existing problems 2.Design choices 3.Build 4.A few steps back
    6. 6. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Agenda 1.Existing problems
    7. 7. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Existing problems Consumer Customers Existing Sytem Compagny Customers LDAP access Micropayment system Identity and Oauth management system Network Connexion system ETL
    8. 8. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Existing problems Two years delay Schema customization very hard to design and implement Performance issues
    9. 9. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Existing problems
    10. 10. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Agenda 1.Existing problems 2.Design choices A.MongoDB B.Tomcat C.Storm
    11. 11. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Design choices - MongoDB Schema less data integration High availability Sharding Less expensive data storage allowing duplication
    12. 12. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Design choices - MongoDB Team with development and architecture skills Schema design experience Capability to train new actors
    13. 13. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld POC between Tomcat and NodeJS We choose our battle, MongoDB ! Design choices - Tomcat
    14. 14. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
    15. 15. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Design choices - Storm Distributed real-time computation system
    16. 16. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Design choices - Storm Consumer Customer MongoDB database Compagny Customers REST Micropayment system Identity and Oauth management system Network Connexion system API STORM
    17. 17. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Agenda 1.Existing problems 2.Design choices 3.Build A.Storm topology B.Schema design and refactor C.DevOps
    18. 18. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Storm topology Customer files are pushed every night Read file line by line 1 … DetectFile Spout ProcessLine Bolt 2 line
    19. 19. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Storm topology
    20. 20. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Storm topology Emitting tuple for each set of functional collections ProcessLine Bolt 4 Oauth UpdaterBolt Identity UpdaterBoltTransform a line into a document & check delta 3
    21. 21. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Storm topology
    22. 22. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Storm topology
    23. 23. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Design choices - Storm Message failure handle ack(Object msgId) failed(Object msgId) Take care of system stability
    24. 24. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Design choices - Storm We scaled and easily !
    25. 25. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld
    26. 26. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - Schema design and refactor Conception User -Name -Compagny -Billcycle -Payment Mode Acces Point -Name -Compagny -Billcycle -Payment Mode 1 1..n Conception
    27. 27. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - Schema design and refactor Development
    28. 28. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - Schema design and refactor Qualification
    29. 29. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - Schema design and refactor Development
    30. 30. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - Schema design and refactor « That works » Qualification Conception
    31. 31. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - Schema design and refactor StressTest
    32. 32. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - Schema design and refactor ConceptionConception
    33. 33. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - Schema design and refactor ConceptionConception
    34. 34. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - Schema design and refactor Development
    35. 35. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - Schema design and refactor Development Qualification Conception StressTest
    36. 36. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - Schema design and refactor PERFECT ! Development Conception StressTes
    37. 37. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - DevOps Tooling first
    38. 38. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - DevOps Moving the line between Dev and Ops
    39. 39. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Build - DevOps Internal training by development team Redaction of a play-book dedicated to ops Pre production tests played by both team Early write of stress tests, played during qualification tests
    40. 40. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld 6 months later
    41. 41. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Agenda 1.Existing problems 2.Design choices 3.Iterations during build 4.A few steps back
    42. 42. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld A few steps back Apache Tomcat Apache STORM Flexibility and Scale with a 100% Open Source Stack !
    43. 43. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld A few steps back 1.Choose your battle 2.Use the right tool for the right job 3.Be prepare to fail and to rework 4.To move organization, training is a key
    44. 44. Thank you for listening Questions ? Further talks @__pad__
    45. 45. Pierre-Alban DEWITTE @__pad__ #MongoDBWorld Photo credits 100 m start : http://fr.wikipedia.org/wiki/100_m%C3%A8tres_%28athl%C3%A9tisme%29#mediaviewer/Fichier :Mens_100m_finals_British_Champs_and_Olympic_Trials.jpg by Paul Foot from Birmingham, UK Question mark block : https://www.flickr.com/photos/jarbo/9379813470 by Jared Cherup Pelleteuse orange : http://fr.123rf.com/photo_3669950_un-grand-pelleteuse-orange-stationne-a-un-chantier-de- construction.html by Stephen Mcsweeny Punaise : https://www.flickr.com/photos/24362608@N05/3501112978/sizes/l/in/photolist- f5R8g4-6kiVfR-6koahu-6ko7US-6ko8Qh-d9NUKU-nDi2eb-ffg13f-8h56wx-cgAcib-hchCtD-decZ4p- 6kocrL-6kj2J6-5doxaw-gbTPmK-nCFBBq-672Snf-2sCzvw-e1zAjM-8voRbc-c9Rh1m-amFFGt- 2iZNBq-cFSnFU-81Ckk4-2XHZAy-9Ggpfw-m8FfKz-8hW6r5-4xSuxC-awXsp4-5aVn7g-fbXVHZ- 9GAPLX-9qr7uU-fcddgA-cP1EN7-np2q92-6V8oBT-cenSsb-cyzaEd-8h5a2F-6V2b3F-ch1RME-fJ52s3- 9XthVN-kBs5mW-6qSPaF-5cA9Qb/ par dractrain94 Une longue vue qui louche : https://www.flickr.com/photos/la_bretagne_a_paris/3847733265/ par Yann Caradec NASA-Apollo8-Dec24-Earthrise : http://www.hq.nasa.gov/office/pao/History/alsj/a410/AS8-14- 2383HR.jpg par NASA / Bill Anders

    ×