NoSQL learnings from the world of Telco

603
-1

Published on

Being a decent-sized Telecommunications provider, we process a lot of calls (hundreds/second), and need to keep track of all the events on each call. Technically speaking, this is "A Lot" of data - data that our clients (and our own people!) want real-time access to in a myriad of ways. We've ended up going through quite a few NoSQL stores in our quest to satisfy everyone - and the way we do things now has very little to do with where we started out. Join me as I describe our experience and what we've learned, focusing on the Big 4, viz

-The "solution-oriented" nature of NoSQL repeatedly changed our understanding of our problem-space - sometimes drastically.
- The system behavior , particularly the failure modes, were significantly different at scale
-The software model kept getting overhauled - regardless of how much we planned ahead
-We came to value agility - the ability to change direction - above all (yes, even at a Telco!)

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
603
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Data model is(more) variable
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • And this is an issue. If the problem space changes…
  • Images here
  • And once the problem has changed, you are left w/ a solution that no longer applies
  • Reporting
  • The two most dangerous words in Product Development
  • I person cares about biling. Another about reporting. No one actually cares about calls over a day old. This stuff is – mostly – irrelevant.
  • A solution designed to process massive numbers of calls is useless if no-one wants to process massive numbers of calls
  • A solution designed to process massive numbers of calls is useless if no-one wants to process massive numbers of calls
  • Still one user, but uses it all the timeOh s**t, people actually use this.
  • Which leads, inexporably, to the next point, which is that
  • Efficient markets
  • Friction can be very useful in avoiding issues
  • The two most dangerous words in Product Development
  • With callrooms, people want to do voicemail in real time. Worldwide. Ack!
  • But I designed for scaleAdd servers, rebalance, etc
  • But I designed for scaleAdd servers, rebalance, etc
  • You can’t restart servers, ‘cos your clients are “always on”
  • You can’t restart servers, ‘cos your clients are “always on”
  • You can’t predict crazy spikes (Obama)
  • Order of magnitude changes are, well, a PITA
  • Server crash – clients reloaded from data store – overload - argh
  • Server crash – clients reloaded from data store – overload - argh
  • Spike, data being stored, overload, argh
  • The bigger they are, the harder they fall
  • I thought we build products?
  • Anniversary billing. Monthly billing. Billing!!!
  • Multiple accounts w/ one single relationship
  • Customer service? Really?
  • Sure, if this floats your boat, but not the point
  • You can’t predict crazy spikes (Obama)
  • S**t is going to happenPlanning is goodThe ability to react (well) is gooder
  • Why not Postgres?“Just Because”Also, Erlang/JSON
  • Provides AutomaticScalingFault Tolerance
  • Very few people cared about callsCall info is 99.999…. % of the dataMonster CRUD improvement
  • Anniversary billing. Monthly billing. Billing!!!
  • Multiple accounts w/ one single relationship
  • Don’t need Map/Reduce for multi-client queriesPhone informationBilling information
  • I person cares about biling. Another about reporting. No one actually cares about calls over a day old. This stuff is – mostly – irrelevant.
  • Message / VM / Fax countsBut need to do it by date-rangeNo-one cares about stuff more than a week oldBut some doLearn to say “No”
  • Expiring calls? Argh!
  • Break call records into individual databases
  • Server crash – clients reloaded from data store – overload - argh
  • Specifically for phonesBrief outage causes huge spikesMnesia as K-V store
  • You can’t predict crazy spikes (Obama)
  • Order of magnitude changes are, well, a PITA
  • Customer service? Really?
  • Billing flexibilityMove billing its own systemStream data from other stores to itCan rebuild offlineExcel is your friend!
  • I thought we build products?
  • Moved back-office ops to its own storeSingle Source of TruthTightly controlled
  • NoSQL learnings from the world of Telco

    1. 1. NoSQL the Telco way { @dieswaytoofast (V.P. Ubiquiti Networks)
    2. 2. The Business
    3. 3.  Phone services for SMBsThe Business
    4. 4.  Phone services for SMBs  Hosted Phone services for SMBsThe Business
    5. 5.  Phone services for SMBs  Hosted Phone services for SMBs  Hosted Cloud Communications service for SMBsThe Business
    6. 6.  Phone services for SMBs T?  Hosted Phone services for SMBs HA  Hosted Cloud Communications service for SMBs W H HUThe Business
    7. 7. The Metrics
    8. 8.  Phone Calls per SecondThe Metrics
    9. 9.  Phone Calls per Second x 1000The Metrics
    10. 10.  Simultaneous Phone CallsThe Metrics
    11. 11.  Simultaneous Phone Calls x 10,000The Metrics
    12. 12.  HTTP API requestsThe Metrics
    13. 13.  HTTP API requests ∞The Metrics
    14. 14.  Self-hosted (kinda)The Infrastructure
    15. 15.  Self-hosted (kinda)  Big IP pipesThe Infrastructure
    16. 16.  Self-hosted (kinda)  Big IP pipes  ErlangThe Infrastructure
    17. 17.  Self-hosted (kinda)  Big IP pipes  Erlang  Polyglot PersistenceThe Infrastructure
    18. 18.  Self-hosted (kinda)  Big IP pipes  Erlang  Polyglot Persistence H? HUThe Infrastructure
    19. 19.  Domain-specific data storesPolyglot Persistence
    20. 20.  Domain-specific data stores SQLPolyglot Persistence
    21. 21. NoSQL  Domain-specific data stores SQLPolyglot Persistence
    22. 22. NoSQL  Domain-specific data stores SQLPolyglot PersistenceFiles
    23. 23. NoSQL  Domain-specific data storesText SQLPolyglot PersistenceFiles
    24. 24. NoSQLExcel  Domain-specific data storesText SQLPolyglot PersistenceFiles
    25. 25. NoSQLExcel  Domain-specific data stores Post-ItText SQLPolyglot PersistenceFiles
    26. 26. NoSQLExcel  Domain-specific data stores Post-ItText SQLPolyglot PersistenceFiles
    27. 27.  Not (necessarily) structured dataNoSQL
    28. 28.  Not (necessarily) structured data  Solution OrientedNoSQL
    29. 29.  Not (necessarily) structured data  Solution Oriented H? HUNoSQL
    30. 30. SQ L
    31. 31. No SQ L
    32. 32. 75 bh p
    33. 33. What d’you want the data to look like when you fetch it from the database? - Casey RosenthalSolution Oriented
    34. 34. Key-ValueSolution Oriented
    35. 35. Key-ValueSolution OrientedObject
    36. 36. Key-Value ColumnSolution OrientedObject
    37. 37. Key-ValueDocument Column Solution Oriented Object
    38. 38. Key-Value GraphDocument Column Solution Oriented Object
    39. 39. Key-Value Graph is llyDocument ns tua nt Column te en Ev Solution Oriented Object Co
    40. 40. Key-Value d re Graph deOr is llyDocument ns tua nt Column te en Ev Solution Oriented Object Co
    41. 41. Key-Value d re Graph de Or is llyDocument ns tua nt y Column teor en em Solution Oriented Ev ObjectM Co
    42. 42. lu le Key-Value d Va tip re ns tua e Graph de ul M Or is llyDocument nt y Column teor en em Solution Oriented Ev ObjectM Co
    43. 43. lu le Key-Value d Va tip re ns tua e Graph de ul M Or is llyDocument nt y Column teor en em Solution Oriented Ev ObjectM Co
    44. 44. http://techcrunch.com/2012/10/27/big-data-right-now-five-trendy-open-source-technologies/
    45. 45. Example please?
    46. 46. "Everybody Knows"
    47. 47. "E VE RY B OD YK NO W S
    48. 48. EN O GI V E NE R ER IN G!
    49. 49. Anything else?
    50. 50.  I lied about reportsAnything else?
    51. 51.  I lied about reports (kind-of)Anything else?
    52. 52. Do tell…
    53. 53. If its easy, people might actually use it - <name withheld>Sad but true…
    54. 54. Friction - Bad...
    55. 55. Friction - Good...
    56. 56. Friction - Good...Really?
    57. 57. Friction - Good...Example please?
    58. 58. Anything else?
    59. 59. Scaling Matters
    60. 60. S W NO YK BOD RYScaling Matters VE "E
    61. 61. Scaling is easy
    62. 62. S W NO YK BOD RYScaling is easy VE "E
    63. 63. Automatic Scaling is hard
    64. 64. Automatic Scaling is hard
    65. 65. Automatic Scaling is hard
    66. 66. Automatic Scaling is hard
    67. 67. Automatic Scaling is hard
    68. 68. Automatic Scaling is hard
    69. 69. And the Failure modes!
    70. 70. And the Failure modes!
    71. 71. And the Failure modes!
    72. 72. And the Failure modes!
    73. 73. And the Failure modes!
    74. 74. Back Office Systems
    75. 75. New CFO
    76. 76. New CEO
    77. 77. AGILITYThe Bottom Line
    78. 78. a·gil·i·ty /əˈdʒɪlɪti/ noun the power of moving quickly and easily; nimblenessAgility
    79. 79.  Loose CouplingAgility
    80. 80.  Loose Coupling  Hot UpgradesAgility
    81. 81.  Loose Coupling  Hot Upgrades  Polyglot PersistenceAgility
    82. 82.  Move call information into one (per-client) databaseRedesign
    83. 83. New CFO
    84. 84.  Preprocess Call information  Separate out billing informationRedesign
    85. 85.  Preprocess Call information  Separate out billing information What d’you want the data to look like when you fetch it from the database? - Casey RosenthalRedesign
    86. 86. New CEO
    87. 87.  Move all client info into one DatabaseRedesign
    88. 88. "E VE RY B OD YK NO W S
    89. 89.  More pre-computations (Date ranges! Argh!)Redesign
    90. 90.  Expiring calls? Argh!Redesign
    91. 91. And the Failure modes!
    92. 92.  Decouple authentication MnesiaRedesign
    93. 93. Automatic Scaling is hard
    94. 94.  Caches Caches Everywhere… MnesiaRedesign
    95. 95. Automatic Scaling is hard
    96. 96.  Mnesia Caches Caches Everywhere… MnesiaRedesign
    97. 97.  Mnesia Caches Caches Everywhere… Mnesia MnesiaRedesign
    98. 98. MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign
    99. 99. Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign
    100. 100. Mnesia Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign
    101. 101. Mnesia Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign Mnesia
    102. 102. Mnesia Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign Mnesia Mnesia
    103. 103. Redesign
    104. 104. Back Office Systems
    105. 105. Redesign

    ×