Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
UNLEASH THE POWER OF
COUCHBASE THROUGH
N1QL (NICKEL)
ARVIND JADE (ARVIND.JADE@NIELSEN.COM)
GOVINDARAJAN RAGHUNATHAPURAM
GO...
AGENDA
 About Nielsen
 Answers on Demand, Big Data Platform
 Business Challenges
 Why NoSQL with Couchbase?
 Couchbas...
ABOUT NIELSEN
Nielsen is a leading global information and measurement
company that enables companies to understand consume...
ECOSYSTEM
REPORTING
Avg 175,000 Scan reports per
month (>25% in cross cat)
Avg 7,000 Panel reports per
month
MONTHLY USAGE
Avg 5,500...
AOD PLATFORM
Disaggregated
Data Warehouse
USER EXPERIENCEONDEMAND ENGINESOURCE DATA
Flexible Adapters to
accommodate multi...
AOD APPLICATION
AOD DATA TIER
REPORT BUILDER
REPORT PLAYER
DATA SELECTOR
BUSINESS CHALLENGES
WHERE WE WERE?
BIG Problems
o Growth - Expensive scaling
o Relational Fatigue
o Fragmented Caching
o No Unified Analytics
...
Unified
Big Data
Platform
Caching
Layer
Netezza
Reporting
Application
Log
Oracle
Report
Metadata
SQL Server
Audit Data
Acc...
WHY NOSQL
Schema-less – Schema updates and cost of
change were very high
Horizontal scaling – Sharding and replication
wit...
COUCHBASE JOURNEY
Couchbase 2.0
Mobilization,
Prototyping
Couchbase 2.0, 4
node cluster live
in 1 data center,
for documen...
COUCHBASE USAGE MODEL
 As Reverse index store using map/reduce for faster look up
 For Unified analytics combining Index...
USAGE MODEL 1
DOCUMENT STORE
REPORT SELECTION MODEL
1
9
REPORT DEFINITION MODEL
2
0
USAGE MODEL 2
INCREMENTAL MAP/REDUCE
MAP REDUCE SAMPLE
2
2
USAGE MODEL 3
UNIFIED ANALYTICS
26
Reporting
Data
Report Audit
Data
Application
Log Data
Custom Connector
Map/Reduce Log StashJDBC Connector
Metrics
1.4 B...
27
UNIFIED JSON MODEL
28
UNIFIED ANALYTICS
USAGE MODEL 4
AD HOC QUERYING
N1QL BUSINESS USE CASES
3
0
 Derive metrics – Get stats on user selections, usage patterns
 Quantify Impacts - During a ...
N1QL DEMO
3
1
BUSINESS GAINS
3
2
Faster Performance - Over all processing time of fixing client
reports is now reduced to 1/5th
Smart se...
33
HEADING TOWARDS
 Upgrade cluster to Couchbase 4.0 to leverage Multi
Dimensional Scaling
 Prototype Couchbase Lite for...
34
Q & A
Upcoming SlideShare
Loading in …5
×

Interactive Data Analytics with Couchbase N1QL at Nielsen: Couchbase Connect 2015

2,187 views

Published on

In this session we’ll discuss how Couchbase’s query language, N1QL provided Nielsen with an interactive querying capability that significantly increased our ability to gather meaningful insights into stored client data. In this session, you will learn how we gather those insights and interact with data analytics while leveraging SQL for JSON, N1QL. For context, Nielsen’s Answers on Demand (AOD) services deliver ratings data and other information for businesses in more than 100 countries. With the inflow of massive volumes of data and the requirement to deliver highly targeted results for clients, the ability to sift through datasets quickly and effectively is critical. The AOD services need to provide powerful analytics and reporting capabilities – essentially aggregations on the fly – through an on-demand big data platform. We at Nielsen turned to Couchbase to persist client report definitions, selections, and cache enabling us to sidestep many of the limitations of relational databases operating in a multitenant environment. The Couchbase solution delivered a 50 percent boost in response time by pre-indexing metadata and gave us the ability to query against the index or target specific documents with N1QL.

Published in: Software
  • Be the first to comment

Interactive Data Analytics with Couchbase N1QL at Nielsen: Couchbase Connect 2015

  1. 1. UNLEASH THE POWER OF COUCHBASE THROUGH N1QL (NICKEL) ARVIND JADE (ARVIND.JADE@NIELSEN.COM) GOVINDARAJAN RAGHUNATHAPURAM GOVINDARAJAN.RAGHUNATHAPURAM@NIELSEN.COM
  2. 2. AGENDA  About Nielsen  Answers on Demand, Big Data Platform  Business Challenges  Why NoSQL with Couchbase?  Couchbase Usage Models  Next Steps  Q & A
  3. 3. ABOUT NIELSEN Nielsen is a leading global information and measurement company that enables companies to understand consumers and consumer behavior Nielsen measures and monitors what consumers watch (programming, advertising) and what consumers buy (categories, brands, products) on a global and local basis Nielsen has a presence in approximately 100 countries spread across Africa, Asia, Australia, Europe, Middle East, North America, South America and Russia
  4. 4. ECOSYSTEM
  5. 5. REPORTING Avg 175,000 Scan reports per month (>25% in cross cat) Avg 7,000 Panel reports per month MONTHLY USAGE Avg 5,500 unique users AND 11,000 named users on the system DATABASES Scan • Core – 159 • Custom - 141 Panel • Core – 34 • Custom - 65 PROCESSING 175B records processed per week on scan, 10X that amount on the monthly 430MM purchase txns across 100000 Nielsen panelists per week DATA VOLUMES 1.5PB of scan data - 130K stores, 4.3M UPCs, - 5 years SLAS Guaranteed system availability of 14 hours per day M-F Weekly refreshed scan data updates avail +9 6AM ET Weekly refreshed panel data updates avail +16 6AM ET METRICS
  6. 6. AOD PLATFORM Disaggregated Data Warehouse USER EXPERIENCEONDEMAND ENGINESOURCE DATA Flexible Adapters to accommodate multiple input source files. Disaggregated data warehouse supports ultimate flexibility. Messaging architecture supports seamless orchestration. Powerful BI and Rendering engine to provide fast, rich insights. Trips T-log POS Prod Ref Stores House holds Loyalty Cards ETLPOWEREDBYIBM MESSAGING AND ORCHESTRATION POWERED BY TIBCO FACT PROCESSING DIMENSION BUILDS PROCESSING POWERED BY IBM’S NETEZZA BI ENGINE POWERED BY JAVA RENDERING ENGINE BY EXTJS Panel ODS Scan ODS Loyalty ODS Fact Dim Dim Dim Dim Dim On-the-fly Virtual Aggregations Answer s
  7. 7. AOD APPLICATION
  8. 8. AOD DATA TIER
  9. 9. REPORT BUILDER
  10. 10. REPORT PLAYER
  11. 11. DATA SELECTOR
  12. 12. BUSINESS CHALLENGES
  13. 13. WHERE WE WERE? BIG Problems o Growth - Expensive scaling o Relational Fatigue o Fragmented Caching o No Unified Analytics o Ad Hoc querying capability
  14. 14. Unified Big Data Platform Caching Layer Netezza Reporting Application Log Oracle Report Metadata SQL Server Audit Data Access Log OUR WANTS & NEEDS Better Scalability – Be elastic to accommodate new data growth with ease. Faster Performance Cheaper – Can we get utopia for cheap? Insights – Ability to run analytics Faster feature release.
  15. 15. WHY NOSQL Schema-less – Schema updates and cost of change were very high Horizontal scaling – Sharding and replication with no single point of failure Deep Analytics – Incremental map/reduce, aggregated searching Cost – Commodity hardware High Performance – Low latency and high throughput
  16. 16. COUCHBASE JOURNEY Couchbase 2.0 Mobilization, Prototyping Couchbase 2.0, 4 node cluster live in 1 data center, for document and cache storage Upgraded Couchbase 2.5.1, 16 node clusters, in 2 data centers, advanced views, Unified Analytics w/i ElasticSearch Upgrade to Couchbase 4.0, adopt Nickel for ad hoc querying, Couchbase Lite for mobile prototyping 2012 2013 2014 2015
  17. 17. COUCHBASE USAGE MODEL  As Reverse index store using map/reduce for faster look up  For Unified analytics combining Indexes from Couchbase and Elastic  Needed a solution that keeps client reports agnostic of back end changes by updating reports of magnitude (Millions)  Provide holistic insight into report metrics and system health  As Document and Cache persistence store  Real time application uses Couchbase for responsive UI  For Ad Hoc Querying – Instant Analytics (NEW)  Ability to query key spaces and indexes using SQL-like interfaces N1QL – SQL-Like Query Language
  18. 18. USAGE MODEL 1 DOCUMENT STORE
  19. 19. REPORT SELECTION MODEL 1 9
  20. 20. REPORT DEFINITION MODEL 2 0
  21. 21. USAGE MODEL 2 INCREMENTAL MAP/REDUCE
  22. 22. MAP REDUCE SAMPLE 2 2
  23. 23. USAGE MODEL 3 UNIFIED ANALYTICS
  24. 24. 26 Reporting Data Report Audit Data Application Log Data Custom Connector Map/Reduce Log StashJDBC Connector Metrics 1.4 B Reporting Data Points 3 TB of Index Size 20 M Audit Records 5 TB of Application Log UNIFIED ANALYTICS
  25. 25. 27 UNIFIED JSON MODEL
  26. 26. 28 UNIFIED ANALYTICS
  27. 27. USAGE MODEL 4 AD HOC QUERYING
  28. 28. N1QL BUSINESS USE CASES 3 0  Derive metrics – Get stats on user selections, usage patterns  Quantify Impacts - During a data refresh, identify the set of impacted reports to predict cost of change and impact  Identify and update JSON documents - Operational Need
  29. 29. N1QL DEMO 3 1
  30. 30. BUSINESS GAINS 3 2 Faster Performance - Over all processing time of fixing client reports is now reduced to 1/5th Smart search - With creation of reverse index, able to perform targeted search and convert only affected documents. Real time insights – Combining Couchbase and Elasticsearch, able to derive instant analytics, near real time. Scalable - Able to onboard new clients rapidly Adhoc Querying – Able to empower Analysts to run adhoc analytics.
  31. 31. 33 HEADING TOWARDS  Upgrade cluster to Couchbase 4.0 to leverage Multi Dimensional Scaling  Prototype Couchbase Lite for mobile certification for AOD Application  Leverage the power of N1QL for Instant Analytics
  32. 32. 34 Q & A

×