Your SlideShare is downloading. ×
NoSQL learnings from the world of Telco
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

NoSQL learnings from the world of Telco

479

Published on

Being a decent-sized Telecommunications provider, we process a lot of calls (hundreds/second), and need to keep track of all the events on each call. Technically speaking, this is "A Lot" of data - …

Being a decent-sized Telecommunications provider, we process a lot of calls (hundreds/second), and need to keep track of all the events on each call. Technically speaking, this is "A Lot" of data - data that our clients (and our own people!) want real-time access to in a myriad of ways. We've ended up going through quite a few NoSQL stores in our quest to satisfy everyone - and the way we do things now has very little to do with where we started out. Join me as I describe our experience and what we've learned, focusing on the Big 4, viz

-The "solution-oriented" nature of NoSQL repeatedly changed our understanding of our problem-space - sometimes drastically.
- The system behavior , particularly the failure modes, were significantly different at scale
-The software model kept getting overhauled - regardless of how much we planned ahead
-We came to value agility - the ability to change direction - above all (yes, even at a Telco!)

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
479
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Data model is(more) variable
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • And this is an issue. If the problem space changes…
  • Images here
  • And once the problem has changed, you are left w/ a solution that no longer applies
  • Reporting
  • The two most dangerous words in Product Development
  • I person cares about biling. Another about reporting. No one actually cares about calls over a day old. This stuff is – mostly – irrelevant.
  • A solution designed to process massive numbers of calls is useless if no-one wants to process massive numbers of calls
  • A solution designed to process massive numbers of calls is useless if no-one wants to process massive numbers of calls
  • Still one user, but uses it all the timeOh s**t, people actually use this.
  • Which leads, inexporably, to the next point, which is that
  • Efficient markets
  • Friction can be very useful in avoiding issues
  • The two most dangerous words in Product Development
  • With callrooms, people want to do voicemail in real time. Worldwide. Ack!
  • But I designed for scaleAdd servers, rebalance, etc
  • But I designed for scaleAdd servers, rebalance, etc
  • You can’t restart servers, ‘cos your clients are “always on”
  • You can’t restart servers, ‘cos your clients are “always on”
  • You can’t predict crazy spikes (Obama)
  • Order of magnitude changes are, well, a PITA
  • Server crash – clients reloaded from data store – overload - argh
  • Server crash – clients reloaded from data store – overload - argh
  • Spike, data being stored, overload, argh
  • The bigger they are, the harder they fall
  • I thought we build products?
  • Anniversary billing. Monthly billing. Billing!!!
  • Multiple accounts w/ one single relationship
  • Customer service? Really?
  • Sure, if this floats your boat, but not the point
  • You can’t predict crazy spikes (Obama)
  • S**t is going to happenPlanning is goodThe ability to react (well) is gooder
  • Why not Postgres?“Just Because”Also, Erlang/JSON
  • Provides AutomaticScalingFault Tolerance
  • Very few people cared about callsCall info is 99.999…. % of the dataMonster CRUD improvement
  • Anniversary billing. Monthly billing. Billing!!!
  • Multiple accounts w/ one single relationship
  • Don’t need Map/Reduce for multi-client queriesPhone informationBilling information
  • I person cares about biling. Another about reporting. No one actually cares about calls over a day old. This stuff is – mostly – irrelevant.
  • Message / VM / Fax countsBut need to do it by date-rangeNo-one cares about stuff more than a week oldBut some doLearn to say “No”
  • Expiring calls? Argh!
  • Break call records into individual databases
  • Server crash – clients reloaded from data store – overload - argh
  • Specifically for phonesBrief outage causes huge spikesMnesia as K-V store
  • You can’t predict crazy spikes (Obama)
  • Order of magnitude changes are, well, a PITA
  • Customer service? Really?
  • Billing flexibilityMove billing its own systemStream data from other stores to itCan rebuild offlineExcel is your friend!
  • I thought we build products?
  • Moved back-office ops to its own storeSingle Source of TruthTightly controlled
  • Transcript

    • 1. NoSQL the Telco way { @dieswaytoofast (V.P. Ubiquiti Networks)
    • 2. The Business
    • 3.  Phone services for SMBsThe Business
    • 4.  Phone services for SMBs  Hosted Phone services for SMBsThe Business
    • 5.  Phone services for SMBs  Hosted Phone services for SMBs  Hosted Cloud Communications service for SMBsThe Business
    • 6.  Phone services for SMBs T?  Hosted Phone services for SMBs HA  Hosted Cloud Communications service for SMBs W H HUThe Business
    • 7. The Metrics
    • 8.  Phone Calls per SecondThe Metrics
    • 9.  Phone Calls per Second x 1000The Metrics
    • 10.  Simultaneous Phone CallsThe Metrics
    • 11.  Simultaneous Phone Calls x 10,000The Metrics
    • 12.  HTTP API requestsThe Metrics
    • 13.  HTTP API requests ∞The Metrics
    • 14.  Self-hosted (kinda)The Infrastructure
    • 15.  Self-hosted (kinda)  Big IP pipesThe Infrastructure
    • 16.  Self-hosted (kinda)  Big IP pipes  ErlangThe Infrastructure
    • 17.  Self-hosted (kinda)  Big IP pipes  Erlang  Polyglot PersistenceThe Infrastructure
    • 18.  Self-hosted (kinda)  Big IP pipes  Erlang  Polyglot Persistence H? HUThe Infrastructure
    • 19.  Domain-specific data storesPolyglot Persistence
    • 20.  Domain-specific data stores SQLPolyglot Persistence
    • 21. NoSQL  Domain-specific data stores SQLPolyglot Persistence
    • 22. NoSQL  Domain-specific data stores SQLPolyglot PersistenceFiles
    • 23. NoSQL  Domain-specific data storesText SQLPolyglot PersistenceFiles
    • 24. NoSQLExcel  Domain-specific data storesText SQLPolyglot PersistenceFiles
    • 25. NoSQLExcel  Domain-specific data stores Post-ItText SQLPolyglot PersistenceFiles
    • 26. NoSQLExcel  Domain-specific data stores Post-ItText SQLPolyglot PersistenceFiles
    • 27.  Not (necessarily) structured dataNoSQL
    • 28.  Not (necessarily) structured data  Solution OrientedNoSQL
    • 29.  Not (necessarily) structured data  Solution Oriented H? HUNoSQL
    • 30. SQ L
    • 31. No SQ L
    • 32. 75 bh p
    • 33. What d’you want the data to look like when you fetch it from the database? - Casey RosenthalSolution Oriented
    • 34. Key-ValueSolution Oriented
    • 35. Key-ValueSolution OrientedObject
    • 36. Key-Value ColumnSolution OrientedObject
    • 37. Key-ValueDocument Column Solution Oriented Object
    • 38. Key-Value GraphDocument Column Solution Oriented Object
    • 39. Key-Value Graph is llyDocument ns tua nt Column te en Ev Solution Oriented Object Co
    • 40. Key-Value d re Graph deOr is llyDocument ns tua nt Column te en Ev Solution Oriented Object Co
    • 41. Key-Value d re Graph de Or is llyDocument ns tua nt y Column teor en em Solution Oriented Ev ObjectM Co
    • 42. lu le Key-Value d Va tip re ns tua e Graph de ul M Or is llyDocument nt y Column teor en em Solution Oriented Ev ObjectM Co
    • 43. lu le Key-Value d Va tip re ns tua e Graph de ul M Or is llyDocument nt y Column teor en em Solution Oriented Ev ObjectM Co
    • 44. http://techcrunch.com/2012/10/27/big-data-right-now-five-trendy-open-source-technologies/
    • 45. Example please?
    • 46. "Everybody Knows"
    • 47. "E VE RY B OD YK NO W S
    • 48. EN O GI V E NE R ER IN G!
    • 49. Anything else?
    • 50.  I lied about reportsAnything else?
    • 51.  I lied about reports (kind-of)Anything else?
    • 52. Do tell…
    • 53. If its easy, people might actually use it - <name withheld>Sad but true…
    • 54. Friction - Bad...
    • 55. Friction - Good...
    • 56. Friction - Good...Really?
    • 57. Friction - Good...Example please?
    • 58. Anything else?
    • 59. Scaling Matters
    • 60. S W NO YK BOD RYScaling Matters VE "E
    • 61. Scaling is easy
    • 62. S W NO YK BOD RYScaling is easy VE "E
    • 63. Automatic Scaling is hard
    • 64. Automatic Scaling is hard
    • 65. Automatic Scaling is hard
    • 66. Automatic Scaling is hard
    • 67. Automatic Scaling is hard
    • 68. Automatic Scaling is hard
    • 69. And the Failure modes!
    • 70. And the Failure modes!
    • 71. And the Failure modes!
    • 72. And the Failure modes!
    • 73. And the Failure modes!
    • 74. Back Office Systems
    • 75. New CFO
    • 76. New CEO
    • 77. AGILITYThe Bottom Line
    • 78. a·gil·i·ty /əˈdʒɪlɪti/ noun the power of moving quickly and easily; nimblenessAgility
    • 79.  Loose CouplingAgility
    • 80.  Loose Coupling  Hot UpgradesAgility
    • 81.  Loose Coupling  Hot Upgrades  Polyglot PersistenceAgility
    • 82.  Move call information into one (per-client) databaseRedesign
    • 83. New CFO
    • 84.  Preprocess Call information  Separate out billing informationRedesign
    • 85.  Preprocess Call information  Separate out billing information What d’you want the data to look like when you fetch it from the database? - Casey RosenthalRedesign
    • 86. New CEO
    • 87.  Move all client info into one DatabaseRedesign
    • 88. "E VE RY B OD YK NO W S
    • 89.  More pre-computations (Date ranges! Argh!)Redesign
    • 90.  Expiring calls? Argh!Redesign
    • 91. And the Failure modes!
    • 92.  Decouple authentication MnesiaRedesign
    • 93. Automatic Scaling is hard
    • 94.  Caches Caches Everywhere… MnesiaRedesign
    • 95. Automatic Scaling is hard
    • 96.  Mnesia Caches Caches Everywhere… MnesiaRedesign
    • 97.  Mnesia Caches Caches Everywhere… Mnesia MnesiaRedesign
    • 98. MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign
    • 99. Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign
    • 100. Mnesia Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign
    • 101. Mnesia Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign Mnesia
    • 102. Mnesia Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign Mnesia Mnesia
    • 103. Redesign
    • 104. Back Office Systems
    • 105. Redesign

    ×