NoSQL learnings from the world of Telco
Upcoming SlideShare
Loading in...5
×
 

NoSQL learnings from the world of Telco

on

  • 636 views

Being a decent-sized Telecommunications provider, we process a lot of calls (hundreds/second), and need to keep track of all the events on each call. Technically speaking, this is "A Lot" of data - ...

Being a decent-sized Telecommunications provider, we process a lot of calls (hundreds/second), and need to keep track of all the events on each call. Technically speaking, this is "A Lot" of data - data that our clients (and our own people!) want real-time access to in a myriad of ways. We've ended up going through quite a few NoSQL stores in our quest to satisfy everyone - and the way we do things now has very little to do with where we started out. Join me as I describe our experience and what we've learned, focusing on the Big 4, viz

-The "solution-oriented" nature of NoSQL repeatedly changed our understanding of our problem-space - sometimes drastically.
- The system behavior , particularly the failure modes, were significantly different at scale
-The software model kept getting overhauled - regardless of how much we planned ahead
-We came to value agility - the ability to change direction - above all (yes, even at a Telco!)

Statistics

Views

Total Views
636
Slideshare-icon Views on SlideShare
616
Embed Views
20

Actions

Likes
0
Downloads
7
Comments
0

3 Embeds 20

http://www.linkedin.com 11
https://www.linkedin.com 8
https://si0.twimg.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Images here!!!
  • Data model is(more) variable
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • Images here
  • And this is an issue. If the problem space changes…
  • Images here
  • And once the problem has changed, you are left w/ a solution that no longer applies
  • Reporting
  • The two most dangerous words in Product Development
  • I person cares about biling. Another about reporting. No one actually cares about calls over a day old. This stuff is – mostly – irrelevant.
  • A solution designed to process massive numbers of calls is useless if no-one wants to process massive numbers of calls
  • A solution designed to process massive numbers of calls is useless if no-one wants to process massive numbers of calls
  • Still one user, but uses it all the timeOh s**t, people actually use this.
  • Which leads, inexporably, to the next point, which is that
  • Efficient markets
  • Friction can be very useful in avoiding issues
  • The two most dangerous words in Product Development
  • With callrooms, people want to do voicemail in real time. Worldwide. Ack!
  • But I designed for scaleAdd servers, rebalance, etc
  • But I designed for scaleAdd servers, rebalance, etc
  • You can’t restart servers, ‘cos your clients are “always on”
  • You can’t restart servers, ‘cos your clients are “always on”
  • You can’t predict crazy spikes (Obama)
  • Order of magnitude changes are, well, a PITA
  • Server crash – clients reloaded from data store – overload - argh
  • Server crash – clients reloaded from data store – overload - argh
  • Spike, data being stored, overload, argh
  • The bigger they are, the harder they fall
  • I thought we build products?
  • Anniversary billing. Monthly billing. Billing!!!
  • Multiple accounts w/ one single relationship
  • Customer service? Really?
  • Sure, if this floats your boat, but not the point
  • You can’t predict crazy spikes (Obama)
  • S**t is going to happenPlanning is goodThe ability to react (well) is gooder
  • Why not Postgres?“Just Because”Also, Erlang/JSON
  • Provides AutomaticScalingFault Tolerance
  • Very few people cared about callsCall info is 99.999…. % of the dataMonster CRUD improvement
  • Anniversary billing. Monthly billing. Billing!!!
  • Multiple accounts w/ one single relationship
  • Don’t need Map/Reduce for multi-client queriesPhone informationBilling information
  • I person cares about biling. Another about reporting. No one actually cares about calls over a day old. This stuff is – mostly – irrelevant.
  • Message / VM / Fax countsBut need to do it by date-rangeNo-one cares about stuff more than a week oldBut some doLearn to say “No”
  • Expiring calls? Argh!
  • Break call records into individual databases
  • Server crash – clients reloaded from data store – overload - argh
  • Specifically for phonesBrief outage causes huge spikesMnesia as K-V store
  • You can’t predict crazy spikes (Obama)
  • Order of magnitude changes are, well, a PITA
  • Customer service? Really?
  • Billing flexibilityMove billing its own systemStream data from other stores to itCan rebuild offlineExcel is your friend!
  • I thought we build products?
  • Moved back-office ops to its own storeSingle Source of TruthTightly controlled

NoSQL learnings from the world of Telco NoSQL learnings from the world of Telco Presentation Transcript

  • NoSQL the Telco way { @dieswaytoofast (V.P. Ubiquiti Networks)
  • The Business
  •  Phone services for SMBsThe Business
  •  Phone services for SMBs  Hosted Phone services for SMBsThe Business
  •  Phone services for SMBs  Hosted Phone services for SMBs  Hosted Cloud Communications service for SMBsThe Business
  •  Phone services for SMBs T?  Hosted Phone services for SMBs HA  Hosted Cloud Communications service for SMBs W H HUThe Business
  • The Metrics
  •  Phone Calls per SecondThe Metrics
  •  Phone Calls per Second x 1000The Metrics
  •  Simultaneous Phone CallsThe Metrics
  •  Simultaneous Phone Calls x 10,000The Metrics
  •  HTTP API requestsThe Metrics
  •  HTTP API requests ∞The Metrics
  •  Self-hosted (kinda)The Infrastructure
  •  Self-hosted (kinda)  Big IP pipesThe Infrastructure
  •  Self-hosted (kinda)  Big IP pipes  ErlangThe Infrastructure
  •  Self-hosted (kinda)  Big IP pipes  Erlang  Polyglot PersistenceThe Infrastructure
  •  Self-hosted (kinda)  Big IP pipes  Erlang  Polyglot Persistence H? HUThe Infrastructure
  •  Domain-specific data storesPolyglot Persistence
  •  Domain-specific data stores SQLPolyglot Persistence
  • NoSQL  Domain-specific data stores SQLPolyglot Persistence
  • NoSQL  Domain-specific data stores SQLPolyglot PersistenceFiles
  • NoSQL  Domain-specific data storesText SQLPolyglot PersistenceFiles
  • NoSQLExcel  Domain-specific data storesText SQLPolyglot PersistenceFiles
  • NoSQLExcel  Domain-specific data stores Post-ItText SQLPolyglot PersistenceFiles
  • NoSQLExcel  Domain-specific data stores Post-ItText SQLPolyglot PersistenceFiles
  •  Not (necessarily) structured dataNoSQL
  •  Not (necessarily) structured data  Solution OrientedNoSQL
  •  Not (necessarily) structured data  Solution Oriented H? HUNoSQL
  • SQ L
  • No SQ L
  • 75 bh p
  • What d’you want the data to look like when you fetch it from the database? - Casey RosenthalSolution Oriented
  • Key-ValueSolution Oriented
  • Key-ValueSolution OrientedObject
  • Key-Value ColumnSolution OrientedObject
  • Key-ValueDocument Column Solution Oriented Object
  • Key-Value GraphDocument Column Solution Oriented Object
  • Key-Value Graph is llyDocument ns tua nt Column te en Ev Solution Oriented Object Co
  • Key-Value d re Graph deOr is llyDocument ns tua nt Column te en Ev Solution Oriented Object Co
  • Key-Value d re Graph de Or is llyDocument ns tua nt y Column teor en em Solution Oriented Ev ObjectM Co
  • lu le Key-Value d Va tip re ns tua e Graph de ul M Or is llyDocument nt y Column teor en em Solution Oriented Ev ObjectM Co
  • lu le Key-Value d Va tip re ns tua e Graph de ul M Or is llyDocument nt y Column teor en em Solution Oriented Ev ObjectM Co
  • http://techcrunch.com/2012/10/27/big-data-right-now-five-trendy-open-source-technologies/
  • Example please?
  • "Everybody Knows"
  • "E VE RY B OD YK NO W S
  • EN O GI V E NE R ER IN G!
  • Anything else?
  •  I lied about reportsAnything else?
  •  I lied about reports (kind-of)Anything else?
  • Do tell…
  • If its easy, people might actually use it - <name withheld>Sad but true…
  • Friction - Bad...
  • Friction - Good...
  • Friction - Good...Really?
  • Friction - Good...Example please?
  • Anything else?
  • Scaling Matters
  • S W NO YK BOD RYScaling Matters VE "E
  • Scaling is easy
  • S W NO YK BOD RYScaling is easy VE "E
  • Automatic Scaling is hard
  • Automatic Scaling is hard
  • Automatic Scaling is hard
  • Automatic Scaling is hard
  • Automatic Scaling is hard
  • Automatic Scaling is hard
  • And the Failure modes!
  • And the Failure modes!
  • And the Failure modes!
  • And the Failure modes!
  • And the Failure modes!
  • Back Office Systems
  • New CFO
  • New CEO
  • AGILITYThe Bottom Line
  • a·gil·i·ty /əˈdʒɪlɪti/ noun the power of moving quickly and easily; nimblenessAgility
  •  Loose CouplingAgility
  •  Loose Coupling  Hot UpgradesAgility
  •  Loose Coupling  Hot Upgrades  Polyglot PersistenceAgility
  •  Move call information into one (per-client) databaseRedesign
  • New CFO
  •  Preprocess Call information  Separate out billing informationRedesign
  •  Preprocess Call information  Separate out billing information What d’you want the data to look like when you fetch it from the database? - Casey RosenthalRedesign
  • New CEO
  •  Move all client info into one DatabaseRedesign
  • "E VE RY B OD YK NO W S
  •  More pre-computations (Date ranges! Argh!)Redesign
  •  Expiring calls? Argh!Redesign
  • And the Failure modes!
  •  Decouple authentication MnesiaRedesign
  • Automatic Scaling is hard
  •  Caches Caches Everywhere… MnesiaRedesign
  • Automatic Scaling is hard
  •  Mnesia Caches Caches Everywhere… MnesiaRedesign
  •  Mnesia Caches Caches Everywhere… Mnesia MnesiaRedesign
  • MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign
  • Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign
  • Mnesia Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign
  • Mnesia Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign Mnesia
  • Mnesia Mnesia MnesiaMnesia  Caches Caches Everywhere… Mnesia Mnesia Redesign Mnesia Mnesia
  • Redesign
  • Back Office Systems
  • Redesign