1 ©2013 Apigee. Confidential – All Rights Reserved.
Apps + Data + APIs
A New Data Architecture for the
App Economy
Anant J...
2 ©2013 Apigee. Confidential – All Rights Reserved.
Developer User
Digital Business Value Chain
API APPBackend Services
In...
3 ©2013 Apigee. Confidential – All Rights Reserved.
Digital Signals come in Three Forms in this value chain
Digital Assets...
4 ©2013 Apigee. Confidential – All Rights Reserved.
•  /timestamp:
•  {“timestamp”: 134578901234,
•  “payload”: {
•  “send...
5 ©2013 Apigee. Confidential – All Rights Reserved.
•  POST/GET
•  /users
•  /developers
•  /buddies
•  /locations
•  /pro...
6 ©2013 Apigee. Confidential – All Rights Reserved.
Context
= “Secondary Entities + Events”
7 ©2013 Apigee. Confidential – All Rights Reserved.
★
Time of
Event
Context = Other
nearby relevant and
interesting events...
8 ©2013 Apigee. Confidential – All Rights Reserved.
The Rugby World Cup’s Effect on Beer Consumption in AU
Context
Analysis
9 ©2013 Apigee. Confidential – All Rights Reserved.
Context = Nearby,
interesting, relevant locations
Location as Context
10 ©2013 Apigee. Confidential – All Rights Reserved.
Where does a User fulfill her needs?
/storelocator
/product
/search
/...
11 ©2013 Apigee. Confidential – All Rights Reserved.
Context = Complementary, supplementary and substitute
entities (produ...
12 ©2013 Apigee. Confidential – All Rights Reserved.
•  /addtocart/product/12345
•  /addtocart/product/34577
•  Context is...
13 ©2013 Apigee. Confidential – All Rights Reserved.
Developer Activity as Context
•  Developer Activity
–  Checkins, Repo...
14 ©2013 Apigee. Confidential – All Rights Reserved.
Building the right APIs, Hackathons, SDKs for developers
Context
Anal...
15 ©2013 Apigee. Confidential – All Rights Reserved.
Information and Use as Context
Reviews
Description
Category
Demand
Us...
16 ©2013 Apigee. Confidential – All Rights Reserved.
Behavior Patterns as Context (Habits)
•  User Activity on Apps establ...
17 ©2013 Apigee. Confidential – All Rights Reserved.
Public Profiles and Social Activity as Context
•  Social Profile, Net...
18 ©2013 Apigee. Confidential – All Rights Reserved.
Critical Technical Features
19 ©2013 Apigee. Confidential – All Rights Reserved.
The Big Data System for the App Economy must understand…
Events
Entit...
20 ©2013 Apigee. Confidential – All Rights Reserved.
•  Half Life of Data
•  ETL
•  Data Modeling
•  Real-Time Complement
...
21 ©2013 Apigee. Confidential – All Rights Reserved.
Half Life of Data
Volume Value
NOWNOW – 1 YEAR
App
Economy
“Old”
Econ...
22 ©2013 Apigee. Confidential – All Rights Reserved.
APIs displace ETL
API
s
ET
L
Fed by handful of core apps Myriad apps ...
23 ©2013 Apigee. Confidential – All Rights Reserved.
The new Broad Data Platform needs some new constructs
Enterprise
Syst...
24 ©2013 Apigee. Confidential – All Rights Reserved.
Batch must also Affect Real-Time traffic, and vice-versa
Big Data “Ba...
25 ©2013 Apigee. Confidential – All Rights Reserved.
Computer Science is about Abstractions
RDBMS
Map/Reduce
Entities, Eve...
26 ©2013 Apigee. Confidential – All Rights Reserved.
One Possible Architectural Block Diagram
RDBMS Cassandra
Entities and...
27 ©2013 Apigee. Confidential – All Rights Reserved.
And also requires a different approach given that context can be
over...
28 ©2013 Apigee. Confidential – All Rights Reserved.
•  New Big Data Abstractions of
–  Entities
–  Events
–  Context (sec...
29 ©2013 Apigee. Confidential – All Rights Reserved.
Thank you.
Upcoming SlideShare
Loading in...5
×

A New Data Architecture for the App Economy - StampedeCon 2013

612

Published on

At the StampedeCon 2013 Big Data conference in St. Louis, Anant Jhingran, VP of Products at Apigee, discusses A New Data Architecture for the App Economy. It has been clear for quite some time that traditional warehouses do not cut it for unstructured and semi­structured data, and therefore new systems such as NoSQL and Hadoop have emerged. But these systems throw the baby out with the bathwater. Traditional warehouses were built on the premise that applications can be simpler because the databases did a lot. Of course, the penalty for this was that the application’s world view had to fit the relational, database world view. In the new Big Data system, the primitives have been lowered so much (simple key value pair, or completely unstructured tuple structure), that the applications now have to do a lot more. We argue that there is a happy medium. We have studied the kinds of data that sits in the app economy, and the data structures that need to be built on top of NoSQL and Hadoop that considerably speed up Insights in the app economy without requiring every problem to be coded from scratch.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
612
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

A New Data Architecture for the App Economy - StampedeCon 2013

  1. 1. 1 ©2013 Apigee. Confidential – All Rights Reserved. Apps + Data + APIs A New Data Architecture for the App Economy Anant Jhingran, Apigee
  2. 2. 2 ©2013 Apigee. Confidential – All Rights Reserved. Developer User Digital Business Value Chain API APPBackend Services Internal Partner External Customer Employee Partner Existing Partner New
  3. 3. 3 ©2013 Apigee. Confidential – All Rights Reserved. Digital Signals come in Three Forms in this value chain Digital Assets B&M Web Events Entities Context
  4. 4. 4 ©2013 Apigee. Confidential – All Rights Reserved. •  /timestamp: •  {“timestamp”: 134578901234, •  “payload”: { •  “sending entity”: UUID1, •  “receiving entitiy”: UUID2, •  “data”: { •  “field1”: value1, •  … •  } •  } •  } •  Outside the billionaire’s club, might be more typically 30 – 50 MM/day Event Structure – generalization of “Facts” in Data Warehouse
  5. 5. 5 ©2013 Apigee. Confidential – All Rights Reserved. •  POST/GET •  /users •  /developers •  /buddies •  /locations •  /products •  … •  Typical environments, ~100,000 – 1MM entities Entity Structure, generalization of “Dimensions” in Data Warehouse
  6. 6. 6 ©2013 Apigee. Confidential – All Rights Reserved. Context = “Secondary Entities + Events”
  7. 7. 7 ©2013 Apigee. Confidential – All Rights Reserved. ★ Time of Event Context = Other nearby relevant and interesting events Time as Context
  8. 8. 8 ©2013 Apigee. Confidential – All Rights Reserved. The Rugby World Cup’s Effect on Beer Consumption in AU Context Analysis
  9. 9. 9 ©2013 Apigee. Confidential – All Rights Reserved. Context = Nearby, interesting, relevant locations Location as Context
  10. 10. 10 ©2013 Apigee. Confidential – All Rights Reserved. Where does a User fulfill her needs? /storelocator /product /search /buy /findinstore < 3 days < 1 day Context Analysis
  11. 11. 11 ©2013 Apigee. Confidential – All Rights Reserved. Context = Complementary, supplementary and substitute entities (products, services, data) Related Entities as Context
  12. 12. 12 ©2013 Apigee. Confidential – All Rights Reserved. •  /addtocart/product/12345 •  /addtocart/product/34577 •  Context is –  Product Categories –  /addtocart/product/12345?category=menscoats –  /addtocart/product/34577?category=menscoats •  Analysis is –  Promotion Effectiveness (within a 1 week window) grouped by product category (not product) Determining effectiveness of promotions
  13. 13. 13 ©2013 Apigee. Confidential – All Rights Reserved. Developer Activity as Context •  Developer Activity –  Checkins, Repos, Follows •  Developer Profile –  Skills, Languages, Platforms •  Developer Network –  Follows, Followers, Watchers
  14. 14. 14 ©2013 Apigee. Confidential – All Rights Reserved. Building the right APIs, Hackathons, SDKs for developers Context Analysis
  15. 15. 15 ©2013 Apigee. Confidential – All Rights Reserved. Information and Use as Context Reviews Description Category Demand User Action (e.g. Purchase) Context = Information leading to decisions in end user use cases
  16. 16. 16 ©2013 Apigee. Confidential – All Rights Reserved. Behavior Patterns as Context (Habits) •  User Activity on Apps establishes patterns of Behavior and Actions •  Deviations from the behavior profile are interesting also
  17. 17. 17 ©2013 Apigee. Confidential – All Rights Reserved. Public Profiles and Social Activity as Context •  Social Profile, Network and Activity describe users •  Features like the Facebook Timeline for user’s preferences
  18. 18. 18 ©2013 Apigee. Confidential – All Rights Reserved. Critical Technical Features
  19. 19. 19 ©2013 Apigee. Confidential – All Rights Reserved. The Big Data System for the App Economy must understand… Events Entities Context DATA: ANALYSIS: Both “Batch” and “Real-Time”
  20. 20. 20 ©2013 Apigee. Confidential – All Rights Reserved. •  Half Life of Data •  ETL •  Data Modeling •  Real-Time Complement Many things are Different
  21. 21. 21 ©2013 Apigee. Confidential – All Rights Reserved. Half Life of Data Volume Value NOWNOW – 1 YEAR App Economy “Old” Economy
  22. 22. 22 ©2013 Apigee. Confidential – All Rights Reserved. APIs displace ETL API s ET L Fed by handful of core apps Myriad apps and services Concise data Verbose data Data optimized for storage Data optimized for consumption Well-modeled business systems and data owned by enterprise Disparate, dynamic data in fast-paced mobile, social apps ecosystems Works as self-contained ‘cubes’ Works by mixing with other APIs
  23. 23. 23 ©2013 Apigee. Confidential – All Rights Reserved. The new Broad Data Platform needs some new constructs Enterprise Systems" External Online Data" Data Collection Data Processing Entity and Event Model APIs API DataApp Data SQL Dimensions and Facts Joins and Aggregations ETL Map Reduce, Pig, Hive Key Value Aggregations Bulk Loads, Flume… REST, Odata? Collections, Time Series Entity Resolution, Signal Amplification,… API based access Warehousing Big Data Broad Data
  24. 24. 24 ©2013 Apigee. Confidential – All Rights Reserved. Batch must also Affect Real-Time traffic, and vice-versa Big Data “Batch” Analysis ? Real-Time “Gateway”
  25. 25. 25 ©2013 Apigee. Confidential – All Rights Reserved. Computer Science is about Abstractions RDBMS Map/Reduce Entities, Events and Context Abstractions Flexibility File System Abstractions Reduce the Number of Problems that can be solved But Significantly Improve Time to Value
  26. 26. 26 ©2013 Apigee. Confidential – All Rights Reserved. One Possible Architectural Block Diagram RDBMS Cassandra Entities and Events in the App Economy Data Import and Access APIs CRUD and Analytical Libraries •  Tailored for “data” and use cases in the App Economy •  Built around fundamental transformations of ETL, Warehousing and Big Data Hadoop
  27. 27. 27 ©2013 Apigee. Confidential – All Rights Reserved. And also requires a different approach given that context can be overwhelming Insights Data API Traffic Developer Activity Mobile App Activity
  28. 28. 28 ©2013 Apigee. Confidential – All Rights Reserved. •  New Big Data Abstractions of –  Entities –  Events –  Context (secondary entities and events) •  New Data Processing Techniques –  Determining “value” of the data –  Data Stitching for enhancing signal to noise •  New Analytical Techniques –  Time Series Analysis –  Graph Traversals –  Real-Time Complement to Batch Analysis •  New Approach to Data Science Summary
  29. 29. 29 ©2013 Apigee. Confidential – All Rights Reserved. Thank you.

×