Nielsen_Couchbase_SF_2013
 

Nielsen_Couchbase_SF_2013

on

  • 894 views

 

Statistics

Views

Total Views
894
Views on SlideShare
616
Embed Views
278

Actions

Likes
0
Downloads
22
Comments
0

2 Embeds 278

http://www.couchbase.com 277
http://twimblr.appspot.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • How many have heard of Nielsen.How many of you actually work for Nielsen.Let’s transition now to a little about Nielsen
  • About me first
  • Nielsen’s mission is to help our clients have the most complete understanding of consumers worldwideUnderstanding consumers means:Who they are – psychographics, demographics, diversity and cultural variances. Family and age variances within the home.What they watch – and for this we mean what they look at and how they engage in a multi-dimensional way. Across platforms like TV, online, mobile, social, out of home. What they buy– we look across traditional and modern trade. And even how they think: we are pushing the frontier of research to understand the brain and thought patterns
  • Across Watch and Buy, the way we collect data involves a vast and extensive network of proprietary and 3rd party players that come together in an ecosystem we refer to as COLLABORATIVE DATAWe start with consumers and protecting their privacy first and foremostWe collect and assemble information across all of their day-to-day watch and buy activities online, offline, throughout their day Of course that is a VAST amount of data – FOR us that involves as we’ve heard referenced 800mm pieces of data…Our collaborative data ecosystem is the foundation of how we create value through our sheer a) breadth  - connect the dots across media and commerce and b) depth - leveraging a series of  partnerships and acquisitions, to fill in the gaps and keep Nielsen's clients at the forefront of benefits of measurement c) quality with extreme precision, accuracy, neutrality and always privacy
  • Across Watch and Buy, the way we collect data involves a vast and extensive network of proprietary and 3rd party players that come together in an ecosystem we refer to as COLLABORATIVE DATAWe start with consumers and protecting their privacy first and foremostWe collect and assemble information across all of their day-to-day watch and buy activities online, offline, throughout their day Of course that is a VAST amount of data – FOR us that involves as we’ve heard referenced 800mm pieces of data…Our collaborative data ecosystem is the foundation of how we create value through our sheer a) breadth  - connect the dots across media and commerce and b) depth - leveraging a series of  partnerships and acquisitions, to fill in the gaps and keep Nielsen's clients at the forefront of benefits of measurement c) quality with extreme precision, accuracy, neutrality and always privacy----- Meeting Notes (7/17/13 15:28) -----In the buy division, we measure and track the sales of consumer packaged goods.Data by itself is not that interesting and certainly not very beneficial to a client.What we deliver is an on the lfy processing engine. We refer to this as on-demand.
  • But what good is data on it’s own?Data is Our promise to our clients is to help them make faster, smarter more confident decisions, driven by this consumer foresightClearly with the outcome of business growth. That is our promise and what motivates us every day is helping our clients to grow
  • ----- Meeting Notes (7/17/13 15:28) -----working since beginning of year.
  • Data comes in different formats with various levels of information. Metadata can be similar between diff feeds.Data tends to add up, metadata (at this point what we store) grows as well since characteristics in dimensions certainly grows over time.Imagine we are basically storing in product dimension, each UPC sold that we can track at these retailers or that a panel member might scan (coffin joke?)
  • ----- Meeting Notes (7/17/13 15:28) -----working since beginning of year.
  • Divided between Consumer and retail.Retail drive data through POS data and manufacturers drive demand
  • ----- Meeting Notes (7/17/13 15:28) -----working since beginning of year.
  • Each report we are seeing here is completely customizable by the user. Generally we have no guard rails in place and from a shared services point of view we have no control over number of users and number of reports they can create. (SCALING)
  • Decomposition with grid to charts
  • Supported tech.very intiuitive with user helpconditions, expression, conditional formatting
  • Presentation layer of HTML5. IMPORTANT to speak of JSONmiddle tier is javal2 cache das more business sounding requests into the data
  • Front end is HTML5, SenchaExtJS. Important for data to be stored in JSONmiddle tier is javal2 cache
  • conservative on the move to each new version of extmvc in product, big diff from flexuser is dragging out set of views so mutliple cab be on paged3 for venn diagramsask who uses CIand artifactory, gradle over ant, jsjsut fits into this.
  • truly document storage whereas older version was key valremove the steps where you have to assemble a relational results set to objects or transform xml to jsonMove out of JVM, explain why (Garbage collection whoas when in JVM caching used)Why views help cache clear. How this has been a problem
  • truly document storage whereas older version was key valremove the steps where you have to assemble a relational results set to objects or transform xml to jsonMove out of JVM, explain whyWhy views help cache clear. How this has been a problem
  • store report spec, user custimzations, selectionsMCM is a big player in this
  • ----- Meeting Notes (7/17/13 15:28) -----size of data
  • Cover example of bi package and how it is split apart.what is a promise
  • Cover example of bi package and how it is split apart.what is a promise
  • ----- Meeting Notes (7/17/13 15:28) -----size of data
  • Cover example of bi package and how it is split apart.what is a promise
  • Cover example of bi package and how it is split apart.what is a promise
  • Componentize the standard ui elements and pull out dropping logic so it can be used across the business----- Meeting Notes (7/17/13 15:28) -----support everything excel can do.functional venn, force graphs. spark lines, etc.context context out in reports.

Nielsen_Couchbase_SF_2013 Nielsen_Couchbase_SF_2013 Presentation Transcript

  • WHY NIELSEN COMPANY'S GLOBAL BUY PLATFORM RELIES ON COUCHBASE DARRELL PRATT ARCHITECTURE LEADER
  • ABOUT NIELSEN
  • Help our clients have the most complete understanding of consumers worldwide. OUR MISSION
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 4 OUR ECOSYSTEM COLLABORATIVE DATA
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 5 NIELSEN ANSWERS ON DEMAND Flexibility • Dynamic “on-the-fly” processing engine • On-Demand products, markets, periods, buyer groups • User role-based reporting • Custom product definitions, hierarchies and characteristics Speed to Insights • Expedited reporting • Roadmaps and guided analysis • Dynamic reporting Integration • Consistent access channel • Internal and external data sources • Support for client business processes
  • Consumer foresight for faster, smarter, more confident decisions OUR PROMISE to drive growth
  • CONSUMER DATA
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 8 SCAN, PANEL AND LOYALTY • Scan • Point of Sales data from 1000’s of retailers • Weekly data • Disaggregated, Anonymous data • Billions of records per week • Panel • Data from more than 250,000 households across 25 countries • Similar to Nielsen Families from View • Trip and Demographic data • Millions of records a week • Loyalty • Loyalty card data from retailers • Basket level transaction data – received daily from thousands of stores • Some demographic data • 100’s of Millions of items weekly
  • WHO IS OUR CONSUMER
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 10 OUR CLIENTS • Manufacturers - Kraft, Procter & Gamble, others • Measure product success • Understand consumer behaviors • Target new products or promotions • Identify new product opportunities • Product pricing • Retailers – Safeway, Tesco, Walmart • Understand consumer buying behavior • Store performance in market • Comparison to competitors • Product pricing
  • OUR REPORTING APPLICATION
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 12 REPORT BUILDER
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 13 REPORT PLAYER
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 14 REPORT PLAYER
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 15 DATA SELECTION
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 16 FEATURES • Single front-end web application which interfaces with the disparate back-end data sources • Advanced BI capabilities • User expressions • Conditional formatting • Smart text • Smart linking of report objects • Very few limits to what user can request with regards to data • Most reports to run under 2 minutes maximum • Loading of application with most data under 5 seconds
  • APPLICATION SPECIFICS
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 18 HIGH LEVEL VIEW
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 19 ARCHITECTURAL VIEW Web Front End Portal ReportApp Workspace Admin Java Middle Tier Spring IOC + Custom Couchbase Cluster ElasticSearch Oracle RAC Netezza Tibco ESB
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 20 TECHNOLOGY STACK • UI built on Sencha Ext JS • AM Charts used for charting • D3 used in some edge case chart types • Middle tier composed of Spring MVC and Spring IOC • JSON REST endpoints through configuration • XML to JSON conversion where needed • SOA Tier using Tibco AMX 3.2 • Couchbase 2.1 – Storage, Caching and Search • Hudson, Artifactory, Gradle, Jasmine, JS Duck
  • WHY COUCHBASE
  • WE STARTED WITH A SEARCH FOR A CACHING SOLUTION
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 23 SOLUTION REQUIREMENTS • Our needs • Scalability • Shared cache/storage for separate applications • Speed • Out of JVM process • Support • As an enterprise, we need 24x7 support • Our wants • Document storage • Map/reduce views • Full text search
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 24 WHY NO-SQL • Relation Data Model Overload • Complexity of objects in system causes churn in DB models • Poor performance due to complexity • Need to get out of business of data transformations • Flexibility of data model is near number one requirement • Scalability with modest hardware and ease • Data Sharding and replication for reliability • JSON encoding • Used throughout UI, important to store as such
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 25 COUCHBASE – USAGE • Storage of native JSON data from application • User customizations of reports • Report definitions • Request instances – Data selections • BI Responses • Metadata change management • Characteristics can and do change weekly • Views created to track user usage and items affected by these changes
  • DEALING WITH REPORTING DATA
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 27 MAKE IT RESPONSIVE • Asynchronous UI • Reporting data gets BIG • Breaking up a report into objects • Asynchronously store chunked data in Couchbase • UI only requests chunk needed
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 28 MOVING DATA EFFICIENTLY • Life of a request for data moves through several systems • Web, Tibco AMX, Tibco EMS, Composite, Database • Use Couchbase as a document storage system • Enables a pass by reference methodology • Storage of data in format closest to what is displayed to user • True persistent storage and in-memory performance
  • DEALING WITH CHANGE
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 30 DIMENSIONAL DATA CHANGES • Our data is made up dimensions (Product, Market, Period, Fact…) with each dimension described by Characteristics • Characteristic data is large and changing • New products introduced • Human error on ingest • Manufacturers change their minds • Changes occur weekly if not daily • Changes here create waves throughout the system
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 31 COPING WITH CHANGE • Metadata change management process • Catch the differential changes at inception • Send those changes through system on ESB • User saved data needs to be updated or invalidated • Saved selections • Saved reports • Segment definitions • All of these items contain this characteristic data • Before Couchbase -> Stored as CLOBS in Oracle • Full table scans and programs to read all data and change where it was found • With Couchbase, MapReduce Views created to easily find items with reference to characteristic data with changes • Easy to find, easy to fix. Huge time savings
  • LOOKING FORWARD
  • Copyright©2012TheNielsenCompany.Confidentialandproprietary. 33 WHAT’S NEXT • Couchbase as a first class data storage application from mainframe acquisition • Full storage of metadata in Couchbase • Map/reduce views to capture statistics on data usage by clients • Predictive analytics using collaborative data from Couchbase views