From Big GORM-centered into a cloud of fast redis nodes

1,178 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,178
On SlideShare
0
From Embeds
0
Number of Embeds
447
Actions
Shares
0
Downloads
12
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

From Big GORM-centered into a cloud of fast redis nodes

  1. 1. From big GORM-centered into a cloud of fast nodes Jorge Uriarte Aretxaga @jorgeuriartehttp://www.slideshare.com/gailen/
  2. 2. About me
  3. 3. @jorgeuriartejorge.uriarte@gailen.es About me
  4. 4. Origins
  5. 5. In the beginning...
  6. 6. In the beginning...
  7. 7. Well-known territory
  8. 8. Well-known territoryJVM based!
  9. 9. Well-known territoryJVM based!Old good SQL & RDBMs
  10. 10. Well-known territoryJVM based!Old good SQL & RDBMsGORM magic!
  11. 11. Productive development environment
  12. 12. Productive development environment Affordable learning curve
  13. 13. Productive development environment Affordable learning curve Good-enough performance
  14. 14. Evolution
  15. 15. “Mmmhhh... I wouldn’t build it that way nowadays...” Enrique Amodeo
  16. 16. “Mmmhhh... I wouldn’t build it that way nowadays...” Enrique Amodeo
  17. 17. “Mmmhhh... I wouldn’t build it that way nowadays...” Enrique Amodeo “Mmmhhh... nowadays... neither would I!” Me
  18. 18. New brands& countries
  19. 19. Reality Check
  20. 20. Reality bites
  21. 21. Reality bites
  22. 22. crementalMode -XX: +UseTLAB -XX:+CMSIn seConcMarkSweepGC ingThreshold=0 -XX: +UseParNewGC -XX:+U tyCy cle=10 -XX:MaxTenur X:MaxPermSize=128m X:CMSIncrementalDu-s erver -Xmx1024m -X ment alDutyCycleMin=0 -X itGC acing -XX:CMSIncre -XX:+DisableExplic-X X:+CMSIncrementalP gOcc upancyFraction=60 6 -XX:CMSInitiatin-X X:SurvivorRatio=25
  23. 23. C P arNewG e -X X:+Use ntalMod e =128m SIncreme MaxP ermSiz -XX:+CM 0 l-XX:in= crementalMode eM +CMSIn -X se: rNewGC -XX:+U AB rkSwee alDut e=10 c MaxTen 0 y 4g X:+UX PaX:+UseTLseConcMammentpGC tyCyclyC -XX:old=uringThreshold=0 x -X -XX:+UseTLAB 3g X:-Xm 8m mentC DutyCycleMin=0M-XX:ncrere entalDu GCgThresh-server -Xmx r -Xms -X CMSIncre pG al - 1024m -XX:MaxPermS ize=12 X CMSInc SI +Disableenurit n 60 X:+CserveentalP MSIncrem acing kSweegOccupng FractionC 0 -XX: MaxT Explic i cMar atin ancy -XX: =6 a ction= ancyFr-X - X:SurvivorRatio=25on s eC 6 -XX:CMSInitialPaci - XX: p -XX:+U SIncrement utyCycle=10 tiatingOccu-X i - XX:+CM ncrementalD 6 -XX:CMSIn I 5 - XX:CMS ivorRatio=2 GC -X X:Surv bleExplicit sa - XX:+Di
  24. 24. Productivity slows down...
  25. 25. Fragility...
  26. 26. Evolution stopped...
  27. 27. Slow development
  28. 28. Slow developmentFragile system... prone to errors
  29. 29. Slow developmentFragile system... prone to errors Tuning hell with JVM heap
  30. 30. Slow developmentFragile system... prone to errors Tuning hell with JVM heap Terracota cluster issues
  31. 31. Slow developmentFragile system... prone to errors Tuning hell with JVM heap Terracota cluster issues Overall excessive complexity
  32. 32. Change? Surgery? Refactor?keep tuning?
  33. 33. Meanwhile... in another mental thread...
  34. 34. A solution looking for a problem?
  35. 35. But get back to the previous point...
  36. 36. But get back to the previous point... Will you dare jump off it?
  37. 37. • New side-project• Potentially deadly traffic• Still to be tighly connected withour core system
  38. 38. • New side-project• Potentially deadly traffic• Still to be tighly connected withour •core system Solutions? •Dedicated system? •More machines & tuning? •Catch that train!
  39. 39. Proof of concept and... JUMP!
  40. 40. Some details?
  41. 41. Fast importer from Quick & dirty (1w) existing XMLs Elasticsearch Prototype - show me the code - Redis Groovy!!! Fast Controllers (XML parsing, JSONAjax (fewer requests) rendering, grails controllers, ...)
  42. 42. • Simply groovy.util.XmlSlurper
  43. 43. • Simply groovy.util.XmlSlurper• Two importers, really
  44. 44. • Simply groovy.util.XmlSlurper• Two importers, really • Bulk
  45. 45. • Simply groovy.util.XmlSlurper• Two importers, really • Bulk • Incremental (continuous)
  46. 46. • Simply groovy.util.XmlSlurper• Two importers, really • Bulk • Incremental (continuous)• Born as a temporary hack...
  47. 47. • Simply groovy.util.XmlSlurper• Two importers, really • Bulk • Incremental (continuous)• Born as a temporary hack... soon to give way to a better system
  48. 48. • Redis master/slaves replication
  49. 49. • Redis master/slaves replication• Fast
  50. 50. • Redis master/slaves replication• Fast• Trivial
  51. 51. • Redis master/slaves replication• Fast• Trivial• Fast
  52. 52. • Redis master/slaves replication• Fast• Trivial• Fast• Trivial
  53. 53. • Redis master/slaves replication• Fast• Trivial• Fast• Trivial• Did I mention fast and trivial?
  54. 54. Deadly simple Grails app
  55. 55. Deadly simple Grails app
  56. 56. Deadly simple Grails app
  57. 57. Deadly simple Grails app“Look mum, no GORM!”
  58. 58. From ‘searchable’ to ElasticSearch
  59. 59. • Fat-trimmed new fronts• Fast, *light* application• Information still safe in the coresystem• Much more capacity with lessresources• Simple integration
  60. 60. Old system New system Wait... where did the red arrows go?
  61. 61. Subdomain based write-through
  62. 62. Subdomain based write-through• Login, session, tracking...
  63. 63. Subdomain based write-through• Login, session, tracking...• Purchases, payments, subscriptions
  64. 64. Subdomain based write-through• Login, session, tracking...• Purchases, payments, subscriptions• Everything goes to our *good old coresystem*
  65. 65. Subdomain based write-through• Login, session, tracking...• Purchases, payments, subscriptions• Everything goes to our *good old coresystem*• ACID, consistency, GORM, SQL,transactions...
  66. 66. Subdomain based write-through• Login, session, tracking...• Purchases, payments, subscriptions• Everything goes to our *good old coresystem*• ACID, consistency, GORM, SQL,transactions...• ...but now with a small fraction of theload
  67. 67. Evaluation
  68. 68. Terracota/Clusteringapproach vs “new thing” Evaluation
  69. 69. Terracota/Clusteringapproach vs “new thing” EvaluationMemory, speed, size, codebase
  70. 70. Terracota/Clusteringapproach vs “new thing” EvaluationMemory, speed, size, codebase Decoupled search system
  71. 71. Terracota/Clusteringapproach vs “new thing” EvaluationMemory, speed, size, codebase New problems, synchronization Decoupled search system
  72. 72. Productivity increasing again...
  73. 73. New landscape
  74. 74. New landscapeGet rid of XML importer(event driven pub/subs...?)
  75. 75. New landscapeGet rid of XML importer(event driven pub/subs...?) Delete old code (yeah!)
  76. 76. Keep pushing for speed New landscapeGet rid of XML importer(event driven pub/subs...?) Delete old code (yeah!)
  77. 77. Keep pushing for speed Keep it simple (this time) New landscapeGet rid of XML importer(event driven pub/subs...?) Delete old code (yeah!)
  78. 78. Just ask me... in two years
  79. 79. @jorgeuriarte
  80. 80. "From big GORM-centered into a cloud of fast nodes" por Jorge Uriarte se encuentra bajo una Licencia Creative Commons Atribución- CompartirIgual 3.0 Unported. External sources PITAhttp://www.quicksales.com.au/ad/p-i-t-a-pain-in-the-ass-car-number-plate/2606513 Hamster http://www.flickr.com/photos/sualk61/3117477410/ Cliff view http://www.flickr.com/photos/epsos/4376727123/sizes/l/ Knight in the white horse http://www.flickr.com/photos/photoplod/7982134342/ Snail http://www.flickr.com/photos/raphaelquinet/693813637/ Fast snail http://www.flickr.com/photos/zorro13/4086173313 Timanfaya devil http://www.flickr.com/photos/yelacis/6941891989/ Light patterns http://www.flickr.com/photos/aerosolhalos/6676192013/ Kick in the top of the cliff http://www.flickr.com/photos/4elevenpix/5056089861/ Homer evolution http://www.flickr.com/photos/photonquantique/1858685882/ Lego machine http://www.flickr.com/photos/djimison/3058818283/ Landscape http://www.flickr.com/photos/blmiers2/6112610781/ Exam http://www.flickr.com/photos/albertogp123/5843577306/ Future is past http://www.flickr.com/photos/tind/7533681980/

×