Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
INTERACTIVE DATA ANALYTICS WITH N1QL
ARVIND JADE
GOVINDARAJAN RAGHUNATHAPURAM
AGENDA
 About Nielsen
 Answers on Demand, Big Data Platform
 Business Challenges
 Why Couchbase?
 Couchbase Usage Mod...
ABOUT NIELSEN
Nielsen is a leading global information and measurement
company that enables companies to understand consume...
ECOSYSTEM
AOD PLATFORM
Disaggregated
Data Warehouse
USER EXPERIENCEONDEMAND ENGINESOURCE DATA
Flexible Adapters to
accommodate multi...
REPORTING
Avg 175,000 Scan reports per
month (>25% in cross cat)
Avg 7,000 Panel reports per
month
MONTHLY USAGE
Avg 5,500...
AOD APPLICATION
AOD REPORT BUILDER
Behind the Smart UI
• Negligible Transformations – Translating to Responsive User Interface
• Highly Ef...
AOD REPORT PLAYER
Behind the Smart UI
• Fast Intelligent Caching – Faster render times, selective cache clearing
• Can sca...
AOD DATA SELECTOR
Behind the Smart UI
• Deep Analytics – Derive Business insights with user meta data (Views)
• Interactiv...
AOD FRONT END
AOD DATA TIER
Couchbase
• Document storage (Json)
• Unified Caching layer
• Incremental Map/Reduce for search
• Sharding a...
BUSINESS CHALLENGES
WHERE WE WERE?
BIG Problems
o Growth –Expensive scaling
o Relational Fatigue
o Fragmented Caching
o No Analytics or Smart ...
WHY NOSQL
Schema-less – Cost of change is low
Horizontal scaling – Sharding and
replication with no single point
of failur...
NOSQL JOURNEY
Couchbase 2.0
Mobilization,
Prototyping.
Elasticsearch
0.9.0 prototyped
search
Couchbase 2.0, 4
node cluster...
COUCHBASE USAGE MODEL
 As Reverse index store using map/reduce for faster look up
 For Unified analytics combining Index...
USAGE MODEL 1
DOCUMENT STORE
REPORT SELECTION MODEL
1
9
REPORT DEFINITION MODEL
2
0
USAGE MODEL 2
INCREMENTAL MAP/REDUCE
MAP REDUCE SAMPLE
2
2
MAP REDUCE SAMPLE
2
3
MAP REDUCE CODE
USAGE MODEL 3
UNIFIED ANALYTICS
26
Reporting
Data
Report Audit
Data
Application
Log Data
Custom Connector
Map/Reduce Log StashJDBC Connector
Metrics
1.4 B...
27
UNIFIED JSON MODEL
28
UNIFIED ANALYTICS
USAGE MODEL 4
INTERACTIVE ANALYTICS
N1QL DEMO
Storage
• 16 node CB/ES cluster in
CBTS and Oldmar
• Sharded data with no
single point of failure
• Very fast <=300ms
resp...
31
HEADING TOWARDS
Upgrading to Couchbase 4.0
and ElasticSearch 1.6
Continue engagement with
external tech groups on NOSQL...
32
Q & A
Upcoming SlideShare
Loading in …5
×

Nielsen's Interactive Data Analytics with N1QL: SQL for JSON

1,401 views

Published on

In this session we’ll discuss how Couchbase’s query language, N1QL provided Nielsen with an interactive querying capability that significantly increased our ability to gather meaningful insights into stored client data. In this session, you will learn how we gather those insights and interact with data analytics while leveraging SQL for JSON, N1QL. For context, Nielsen’s Answers on Demand (AOD) services deliver ratings data and other information for businesses in more than 100 countries. With the inflow of massive volumes of data and the requirement to deliver highly targeted results for clients, the ability to sift through datasets quickly and effectively is critical. The AOD services need to provide powerful analytics and reporting capabilities – essentially aggregations on the fly – through an on-demand big data platform. We at Nielsen turned to Couchbase to persist client report definitions, selections, and cache enabling us to sidestep many of the limitations of relational databases operating in a multitenant environment. The Couchbase solution delivered a 50 percent boost in response time by pre-indexing metadata and gave us the ability to query against the index or target specific documents with N1QL

Published in: Software
  • Be the first to comment

Nielsen's Interactive Data Analytics with N1QL: SQL for JSON

  1. 1. INTERACTIVE DATA ANALYTICS WITH N1QL ARVIND JADE GOVINDARAJAN RAGHUNATHAPURAM
  2. 2. AGENDA  About Nielsen  Answers on Demand, Big Data Platform  Business Challenges  Why Couchbase?  Couchbase Usage Models  Demo  Q & A
  3. 3. ABOUT NIELSEN Nielsen is a leading global information and measurement company that enables companies to understand consumers and consumer behavior Nielsen measures and monitors what consumers watch (programming, advertising) and what consumers buy (categories, brands, products) on a global and local basis Nielsen has a presence in approximately 100 countries spread across Africa, Asia, Australia, Europe, Middle East, North America, South America and Russia
  4. 4. ECOSYSTEM
  5. 5. AOD PLATFORM Disaggregated Data Warehouse USER EXPERIENCEONDEMAND ENGINESOURCE DATA Flexible Adapters to accommodate multiple input source files. Disaggregated data warehouse supports ultimate flexibility. Messaging architecture supports seamless orchestration. Powerful BI and Rendering engine to provide fast, rich insights. Trips T-log POS Prod Ref Stores House holds Loyalty Cards ETLPOWEREDBYIBM MESSAGING AND ORCHESTRATION POWERED BY TIBCO FACT PROCESSING DIMENSION BUILDS PROCESSING POWERED BY IBM’S NETEZZA BI ENGINE POWERED BY JAVA RENDERING ENGINE BY EXTJS Panel ODS Scan ODS Loyalty ODS Fact Dim Dim Dim Dim Dim On-the-fly Virtual Aggregations Answer s
  6. 6. REPORTING Avg 175,000 Scan reports per month (>25% in cross cat) Avg 7,000 Panel reports per month MONTHLY USAGE Avg 5,500 unique users AND 11,000 named users on the system DATABASES Scan • Core – 159 • Custom - 141 Panel • Core – 34 • Custom - 65 PROCESSING 175B records processed per week on scan, 10X that amount on the monthly 430MM purchase txns across 100000 Nielsen panelists per week DATA VOLUMES 1.5PB of scan data - 130K stores, 4.3M UPCs, - 5 years SLAS Guaranteed system availability of 14 hours per day M-F Weekly refreshed scan data updates avail +9 6AM ET Weekly refreshed panel data updates avail +16 6AM ET METRICS
  7. 7. AOD APPLICATION
  8. 8. AOD REPORT BUILDER Behind the Smart UI • Negligible Transformations – Translating to Responsive User Interface • Highly Efficient Meta Data Change Management • Flexible Data Model – Leading to quicker feature additions
  9. 9. AOD REPORT PLAYER Behind the Smart UI • Fast Intelligent Caching – Faster render times, selective cache clearing • Can scale without compromising speed • No single point of failure in Storage
  10. 10. AOD DATA SELECTOR Behind the Smart UI • Deep Analytics – Derive Business insights with user meta data (Views) • Interactive Analytics – Ability to query meta data (N1QL)
  11. 11. AOD FRONT END
  12. 12. AOD DATA TIER Couchbase • Document storage (Json) • Unified Caching layer • Incremental Map/Reduce for search • Sharding and Replication with no single point of failure • Enterprise Support ElasticSearch • Global Search Capability • Geared towards Aggregation and Analytics • Dash boarding capabilities with Kibana
  13. 13. BUSINESS CHALLENGES
  14. 14. WHERE WE WERE? BIG Problems o Growth –Expensive scaling o Relational Fatigue o Fragmented Caching o No Analytics or Smart Search o Ad Hoc querying capability
  15. 15. WHY NOSQL Schema-less – Cost of change is low Horizontal scaling – Sharding and replication with no single point of failure Deep Analytics – Incremental map/reduce, aggregated searching Cost – Commodity hardware High Performance – Low latency and high throughput NoSQL Platform (CB+ES) Caching Layer Horizontal Scaling Schema-less For Faster Releases High Performance Smart Search Deep Analytics
  16. 16. NOSQL JOURNEY Couchbase 2.0 Mobilization, Prototyping. Elasticsearch 0.9.0 prototyped search Couchbase 2.0, 4 node cluster live in 1 data center, for document and cache storage. Elasticsearch cluster for Global Search Upgraded Couchbase 2.5.1, 16 node clusters, in 2 data centers, advanced views, Unified Analytics w/i ElasticSearch 1.4 Upgrade to Couchbase 4.0, adopt Nickel for ad hoc querying, Couchbase Lite for mobile prototyping. Enhance dash boarding capability 2012 2013 2014 2015
  17. 17. COUCHBASE USAGE MODEL  As Reverse index store using map/reduce for faster look up  For Unified analytics combining Indexes from Couchbase and Elastic  Needed a solution that keeps client reports agnostic of back end changes by updating reports of magnitude (Millions)  Provide holistic insight into report metrics and system health  As Document and Cache persistence store  Real time application uses Couchbase for responsive UI  For Ad Hoc Querying – Instant Analytics (NEW)  Ability to query key spaces and indexes using SQL-like interfaces
  18. 18. USAGE MODEL 1 DOCUMENT STORE
  19. 19. REPORT SELECTION MODEL 1 9
  20. 20. REPORT DEFINITION MODEL 2 0
  21. 21. USAGE MODEL 2 INCREMENTAL MAP/REDUCE
  22. 22. MAP REDUCE SAMPLE 2 2
  23. 23. MAP REDUCE SAMPLE 2 3
  24. 24. MAP REDUCE CODE
  25. 25. USAGE MODEL 3 UNIFIED ANALYTICS
  26. 26. 26 Reporting Data Report Audit Data Application Log Data Custom Connector Map/Reduce Log StashJDBC Connector Metrics 1.4 B Reporting Data Points 3 TB of Index Size 20 M Audit Records 5 TB of Application Log UNIFIED ANALYTICS
  27. 27. 27 UNIFIED JSON MODEL
  28. 28. 28 UNIFIED ANALYTICS
  29. 29. USAGE MODEL 4 INTERACTIVE ANALYTICS N1QL DEMO
  30. 30. Storage • 16 node CB/ES cluster in CBTS and Oldmar • Sharded data with no single point of failure • Very fast <=300ms response times Metadata Change Management • Efficient identification and application of changes to reporting meta data • Sub second incremental index creation to enable faster lookup. Caching • Fast Cache for out of JVM processing • Intelligent Cache layer for selective Cache clear Search/Analytics • Powerful Facet search with ElasticSearch • Adhoc analytics with N1QL • Rich Dashboarding capabilities with ELK stack COUCHBASE USAGE MODEL
  31. 31. 31 HEADING TOWARDS Upgrading to Couchbase 4.0 and ElasticSearch 1.6 Continue engagement with external tech groups on NOSQL Prototyping AOD mobile solution with Couchbase Lite Strengthen partnership with CouchBase
  32. 32. 32 Q & A

×