Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big Data: Beyond the "Bigness" and the Technology (webcast)

5,916 views

Published on

Published in: Technology

Big Data: Beyond the "Bigness" and the Technology (webcast)

  1. 1. Big Data - Beyond the Bignessand the TechnologyApril 26, 2012Anant Jhingran @jhingranhttp://blog.apigee.comhttp://jhingran.typepad.com
  2. 2. groups.google.com/group/api-craft
  3. 3. youtube.com/apigee
  4. 4. New! IRC Channel #api-craft on freenode
  5. 5. Three themesBig Data dialog has focused on the wrong things – bigness and technology, which are both misplacedBig Data needs to focus on the right new thing – focus on data stitching from disparate data sourcesData APIs need to be front and center of any Big Data dialog – too little discussion on that
  6. 6. Big Data discussion has focused on the wrong things
  7. 7. Wrong thing #1 – focus on technology Business value Cassandra .91 HBASE TECHNOLOGY EC2 . . . “THE MEANS” DATA “THE GOLD”
  8. 8. Wrong thing #2 – focus on bigness 2 dimensions of complexity Interesting problemsdepth of analysis Big data nerds $$$ VC invest Next cool tech - webscale etc. Hype 100TB size of the data 10 PB
  9. 9. Big Data needs to focus on the new right thing
  10. 10. Circa 2005 – Data controlled within enterprise Data Warehouse Your Web Page Company Store
  11. 11. 2012 – Control shifts to edge of enterprise Social Business Networks Networks Data Your Web Page Warehouse Company Apps Store API Partners
  12. 12. Control shifts to edge of enterpriseBig Data needs to become Broad Data
  13. 13. enterprise + complementary sourcesData volume enterprise data sources old world new world
  14. 14. signal / noise Most of the bigness comes from noise The noise doesn’t matter Only the signal matters
  15. 15. signal / noise Increase signal/noise by stitching data sources
  16. 16. syndicated access ? external control ? enterprise central or de-central process? enterprise✖ Web 1.0 – Crawling . . .✖ Web 2.0 – AJAX . . .✔ Web 3.0 - APIs + control of data
  17. 17. If we give up the wrong things and take up the rightthings, what is it that we need to do?
  18. 18. Shifting from Big Data to Broad Data It’s about . . . • Accessing Data that others collect • Variety • Striking deals • Respecting the APIs • Data stitching and improving S/N ratio • Depth of analysis It’s not about . . . • Crawling • BIGNESS from any one data source
  19. 19. Data APIs are the futureSo what kind of Data APIs?
  20. 20. Data APIs are the futureMonetizable apps produce & consume dataData is the lifeblood at edge of enterpriseNeed to focus on making data consumption easy
  21. 21. Yin and a Yang of transactions and data Example APIs User management Send SMSX-APIs Add movie Do trade Get credit info Browse catalog D-APIs Get weather by Zip code Get demographics by region
  22. 22. Let’s create an information halo around APIsSee Amundsen’s Dogs, Information Halos and APIs:The epic story of your API Strategy »http://blog.apigee.com/detail/api_strategy_talk_web_2.0/
  23. 23. Give Data . . . what are your transactions, and what are your data? Do you want to be crawled or do you want to control it? Give Visibility . . . Analytics and Data go hand in hand…. . . to both your end developers and your colleagues
  24. 24. People are planting “flags” on various data domainsby collecting and stitching disparate data together Local Demographic Real-estate Business Purchases Finance Weather Internet Social Price Traffic
  25. 25. To build out a single domain, many data sources have to beaccessed and stitchedA natural stitching thing could be linked data linkeddata.org
  26. 26. Once stitched, clean APIs can be provided Data API and Analytics Cleansed, Stitched Data Sources Data Data Data (crawled, bulk Source Source Source loaded, API accessed)
  27. 27. Data API and Analytics Cleansed, Stitched Data Sources Data Data Data (crawled, bulk Source Source Source loaded, API accessed)Typically Linked Data techniques not used here
  28. 28. Can Linked Data techniques be used here? Data API and Analytics Cleansed, Stitched Data Sources Data Data Data (crawled, bulk Source Source Source loaded, API accessed)
  29. 29. Linked Data as the Data API for the domains not likelyto be very commonWhy? The interlinking of domains is not as important as thestrength of any one domain (at least for now) Local Demographic Real-estate Business Purchases Finance Weather Internet Social Price Traffic
  30. 30. If not linked data APIs, what other Data APIs mightbecome common?Our guess: APIs patterned after relational access Data API and Analytics Cleansed, Stitched Data Sources Data Data Data (crawled, bulk Source Source Source loaded, API accessed)
  31. 31. Kinds of Data APIs we are observing Imposed Hierarchy based traversal over collections http://api.worldbank.org/incomeLevels/LPrimary Key Lookup IC/countrieshttp://weather.yahooapis.com/forecastrss?w=location “Rectangle” {rows, columns} through query parameters http://api.worldbank.org/countries?p er_page=10&incomeLevel=LIC Data
  32. 32. There are many perspectives on data APIs coming from relational world http://blog.apigee.com/detail/rest_api_design_for_sql_pr rshttp://azgroups.nextslide.com/odata-begins
  33. 33. I gave a talk at MicrosoftIf NoData is not an Option,is Odata the answer?(http://bit.ly/I1P0I6)
  34. 34. What do we need for Data APIs to take off?• Practical REST and OData are good starting points• However, they cannot be available as vendor-specific implementations• The Linked Data model cannot be ignored completely• Let us, as a community, get the best of Linked Data and OData thoughts together• Let’s continue this dialog groups.google.com/group/api-craft
  35. 35. Wrapping upBig Data dialog has focused on the wrong things – bigness and technology, which are both misplacedBig Data needs to focus on the right new thing – focus on data stitching from disparate data sourcesData APIs need to be front and center of any Big Data dialog – too little discussion on that
  36. 36. THANK YOUQuestions and ideas to:@jhingran

×