• Save
A New Data Architecture for the App Economy - StampedeCon 2013
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

A New Data Architecture for the App Economy - StampedeCon 2013

on

  • 754 views

At the StampedeCon 2013 Big Data conference in St. Louis, Anant Jhingran, VP of Products at Apigee, discusses A New Data Architecture for the App Economy. It has been clear for quite some time that ...

At the StampedeCon 2013 Big Data conference in St. Louis, Anant Jhingran, VP of Products at Apigee, discusses A New Data Architecture for the App Economy. It has been clear for quite some time that traditional warehouses do not cut it for unstructured and semi­structured data, and therefore new systems such as NoSQL and Hadoop have emerged. But these systems throw the baby out with the bathwater. Traditional warehouses were built on the premise that applications can be simpler because the databases did a lot. Of course, the penalty for this was that the application’s world view had to fit the relational, database world view. In the new Big Data system, the primitives have been lowered so much (simple key value pair, or completely unstructured tuple structure), that the applications now have to do a lot more. We argue that there is a happy medium. We have studied the kinds of data that sits in the app economy, and the data structures that need to be built on top of NoSQL and Hadoop that considerably speed up Insights in the app economy without requiring every problem to be coded from scratch.

Statistics

Views

Total Views
754
Views on SlideShare
621
Embed Views
133

Actions

Likes
1
Downloads
0
Comments
0

2 Embeds 133

http://eventifier.co 86
http://eventifier.com 47

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

A New Data Architecture for the App Economy - StampedeCon 2013 Presentation Transcript

  • 1. 1 ©2013 Apigee. Confidential – All Rights Reserved. Apps + Data + APIs A New Data Architecture for the App Economy Anant Jhingran, Apigee
  • 2. 2 ©2013 Apigee. Confidential – All Rights Reserved. Developer User Digital Business Value Chain API APPBackend Services Internal Partner External Customer Employee Partner Existing Partner New
  • 3. 3 ©2013 Apigee. Confidential – All Rights Reserved. Digital Signals come in Three Forms in this value chain Digital Assets B&M Web Events Entities Context
  • 4. 4 ©2013 Apigee. Confidential – All Rights Reserved. •  /timestamp: •  {“timestamp”: 134578901234, •  “payload”: { •  “sending entity”: UUID1, •  “receiving entitiy”: UUID2, •  “data”: { •  “field1”: value1, •  … •  } •  } •  } •  Outside the billionaire’s club, might be more typically 30 – 50 MM/day Event Structure – generalization of “Facts” in Data Warehouse
  • 5. 5 ©2013 Apigee. Confidential – All Rights Reserved. •  POST/GET •  /users •  /developers •  /buddies •  /locations •  /products •  … •  Typical environments, ~100,000 – 1MM entities Entity Structure, generalization of “Dimensions” in Data Warehouse
  • 6. 6 ©2013 Apigee. Confidential – All Rights Reserved. Context = “Secondary Entities + Events”
  • 7. 7 ©2013 Apigee. Confidential – All Rights Reserved. ★ Time of Event Context = Other nearby relevant and interesting events Time as Context
  • 8. 8 ©2013 Apigee. Confidential – All Rights Reserved. The Rugby World Cup’s Effect on Beer Consumption in AU Context Analysis
  • 9. 9 ©2013 Apigee. Confidential – All Rights Reserved. Context = Nearby, interesting, relevant locations Location as Context
  • 10. 10 ©2013 Apigee. Confidential – All Rights Reserved. Where does a User fulfill her needs? /storelocator /product /search /buy /findinstore < 3 days < 1 day Context Analysis
  • 11. 11 ©2013 Apigee. Confidential – All Rights Reserved. Context = Complementary, supplementary and substitute entities (products, services, data) Related Entities as Context
  • 12. 12 ©2013 Apigee. Confidential – All Rights Reserved. •  /addtocart/product/12345 •  /addtocart/product/34577 •  Context is –  Product Categories –  /addtocart/product/12345?category=menscoats –  /addtocart/product/34577?category=menscoats •  Analysis is –  Promotion Effectiveness (within a 1 week window) grouped by product category (not product) Determining effectiveness of promotions
  • 13. 13 ©2013 Apigee. Confidential – All Rights Reserved. Developer Activity as Context •  Developer Activity –  Checkins, Repos, Follows •  Developer Profile –  Skills, Languages, Platforms •  Developer Network –  Follows, Followers, Watchers
  • 14. 14 ©2013 Apigee. Confidential – All Rights Reserved. Building the right APIs, Hackathons, SDKs for developers Context Analysis
  • 15. 15 ©2013 Apigee. Confidential – All Rights Reserved. Information and Use as Context Reviews Description Category Demand User Action (e.g. Purchase) Context = Information leading to decisions in end user use cases
  • 16. 16 ©2013 Apigee. Confidential – All Rights Reserved. Behavior Patterns as Context (Habits) •  User Activity on Apps establishes patterns of Behavior and Actions •  Deviations from the behavior profile are interesting also
  • 17. 17 ©2013 Apigee. Confidential – All Rights Reserved. Public Profiles and Social Activity as Context •  Social Profile, Network and Activity describe users •  Features like the Facebook Timeline for user’s preferences
  • 18. 18 ©2013 Apigee. Confidential – All Rights Reserved. Critical Technical Features
  • 19. 19 ©2013 Apigee. Confidential – All Rights Reserved. The Big Data System for the App Economy must understand… Events Entities Context DATA: ANALYSIS: Both “Batch” and “Real-Time”
  • 20. 20 ©2013 Apigee. Confidential – All Rights Reserved. •  Half Life of Data •  ETL •  Data Modeling •  Real-Time Complement Many things are Different
  • 21. 21 ©2013 Apigee. Confidential – All Rights Reserved. Half Life of Data Volume Value NOWNOW – 1 YEAR App Economy “Old” Economy
  • 22. 22 ©2013 Apigee. Confidential – All Rights Reserved. APIs displace ETL API s ET L Fed by handful of core apps Myriad apps and services Concise data Verbose data Data optimized for storage Data optimized for consumption Well-modeled business systems and data owned by enterprise Disparate, dynamic data in fast-paced mobile, social apps ecosystems Works as self-contained ‘cubes’ Works by mixing with other APIs
  • 23. 23 ©2013 Apigee. Confidential – All Rights Reserved. The new Broad Data Platform needs some new constructs Enterprise Systems" External Online Data" Data Collection Data Processing Entity and Event Model APIs API DataApp Data SQL Dimensions and Facts Joins and Aggregations ETL Map Reduce, Pig, Hive Key Value Aggregations Bulk Loads, Flume… REST, Odata? Collections, Time Series Entity Resolution, Signal Amplification,… API based access Warehousing Big Data Broad Data
  • 24. 24 ©2013 Apigee. Confidential – All Rights Reserved. Batch must also Affect Real-Time traffic, and vice-versa Big Data “Batch” Analysis ? Real-Time “Gateway”
  • 25. 25 ©2013 Apigee. Confidential – All Rights Reserved. Computer Science is about Abstractions RDBMS Map/Reduce Entities, Events and Context Abstractions Flexibility File System Abstractions Reduce the Number of Problems that can be solved But Significantly Improve Time to Value
  • 26. 26 ©2013 Apigee. Confidential – All Rights Reserved. One Possible Architectural Block Diagram RDBMS Cassandra Entities and Events in the App Economy Data Import and Access APIs CRUD and Analytical Libraries •  Tailored for “data” and use cases in the App Economy •  Built around fundamental transformations of ETL, Warehousing and Big Data Hadoop
  • 27. 27 ©2013 Apigee. Confidential – All Rights Reserved. And also requires a different approach given that context can be overwhelming Insights Data API Traffic Developer Activity Mobile App Activity
  • 28. 28 ©2013 Apigee. Confidential – All Rights Reserved. •  New Big Data Abstractions of –  Entities –  Events –  Context (secondary entities and events) •  New Data Processing Techniques –  Determining “value” of the data –  Data Stitching for enhancing signal to noise •  New Analytical Techniques –  Time Series Analysis –  Graph Traversals –  Real-Time Complement to Batch Analysis •  New Approach to Data Science Summary
  • 29. 29 ©2013 Apigee. Confidential – All Rights Reserved. Thank you.