Big Data: Beyond the "Bigness" and the Technology (webcast)

Big Data - Beyond the Bignessand the TechnologyApril 26, 2012Anant Jhingran @jhingranhttp://blog.apigee.comhttp://jhingran.typepad.com

groups.google.com/group/api-craft

youtube.com/apigee

New! IRC Channel #api-craft on freenode

Three themesBig Data dialog has focused on the wrong things – bigness and technology, which are both misplacedBig Data needs to focus on the right new thing – focus on data stitching from disparate data sourcesData APIs need to be front and center of any Big Data dialog – too little discussion on that

Big Data discussion has focused on the wrong things

Wrong thing #1 – focus on technology Business value Cassandra .91 HBASE TECHNOLOGY EC2 . . . “THE MEANS” DATA “THE GOLD”

Wrong thing #2 – focus on bigness 2 dimensions of complexity Interesting problemsdepth of analysis Big data nerds $$$ VC invest Next cool tech - webscale etc. Hype 100TB size of the data 10 PB

Big Data needs to focus on the new right thing

Circa 2005 – Data controlled within enterprise Data Warehouse Your Web Page Company Store

2012 – Control shifts to edge of enterprise Social Business Networks Networks Data Your Web Page Warehouse Company Apps Store API Partners

Control shifts to edge of enterpriseBig Data needs to become Broad Data

enterprise + complementary sourcesData volume enterprise data sources old world new world

signal / noise Most of the bigness comes from noise The noise doesn’t matter Only the signal matters

signal / noise Increase signal/noise by stitching data sources

syndicated access ? external control ? enterprise central or de-central process? enterprise✖ Web 1.0 – Crawling . . .✖ Web 2.0 – AJAX . . .✔ Web 3.0 - APIs + control of data

If we give up the wrong things and take up the rightthings, what is it that we need to do?

Shifting from Big Data to Broad Data It’s about . . . • Accessing Data that others collect • Variety • Striking deals • Respecting the APIs • Data stitching and improving S/N ratio • Depth of analysis It’s not about . . . • Crawling • BIGNESS from any one data source

Data APIs are the futureSo what kind of Data APIs?

Data APIs are the futureMonetizable apps produce & consume dataData is the lifeblood at edge of enterpriseNeed to focus on making data consumption easy

Yin and a Yang of transactions and data Example APIs User management Send SMSX-APIs Add movie Do trade Get credit info Browse catalog D-APIs Get weather by Zip code Get demographics by region

Let’s create an information halo around APIsSee Amundsen’s Dogs, Information Halos and APIs:The epic story of your API Strategy »http://blog.apigee.com/detail/api_strategy_talk_web_2.0/

Give Data . . . what are your transactions, and what are your data? Do you want to be crawled or do you want to control it? Give Visibility . . . Analytics and Data go hand in hand…. . . to both your end developers and your colleagues

People are planting “flags” on various data domainsby collecting and stitching disparate data together Local Demographic Real-estate Business Purchases Finance Weather Internet Social Price Traffic

To build out a single domain, many data sources have to beaccessed and stitchedA natural stitching thing could be linked data linkeddata.org

Once stitched, clean APIs can be provided Data API and Analytics Cleansed, Stitched Data Sources Data Data Data (crawled, bulk Source Source Source loaded, API accessed)

Data API and Analytics Cleansed, Stitched Data Sources Data Data Data (crawled, bulk Source Source Source loaded, API accessed)Typically Linked Data techniques not used here

Can Linked Data techniques be used here? Data API and Analytics Cleansed, Stitched Data Sources Data Data Data (crawled, bulk Source Source Source loaded, API accessed)

Linked Data as the Data API for the domains not likelyto be very commonWhy? The interlinking of domains is not as important as thestrength of any one domain (at least for now) Local Demographic Real-estate Business Purchases Finance Weather Internet Social Price Traffic

If not linked data APIs, what other Data APIs mightbecome common?Our guess: APIs patterned after relational access Data API and Analytics Cleansed, Stitched Data Sources Data Data Data (crawled, bulk Source Source Source loaded, API accessed)

Kinds of Data APIs we are observing Imposed Hierarchy based traversal over collections http://api.worldbank.org/incomeLevels/LPrimary Key Lookup IC/countrieshttp://weather.yahooapis.com/forecastrss?w=location “Rectangle” {rows, columns} through query parameters http://api.worldbank.org/countries?p er_page=10&incomeLevel=LIC Data

There are many perspectives on data APIs coming from relational world http://blog.apigee.com/detail/rest_api_design_for_sql_pr rshttp://azgroups.nextslide.com/odata-begins

I gave a talk at MicrosoftIf NoData is not an Option,is Odata the answer?(http://bit.ly/I1P0I6)

What do we need for Data APIs to take off?• Practical REST and OData are good starting points• However, they cannot be available as vendor-specific implementations• The Linked Data model cannot be ignored completely• Let us, as a community, get the best of Linked Data and OData thoughts together• Let’s continue this dialog groups.google.com/group/api-craft

Wrapping upBig Data dialog has focused on the wrong things – bigness and technology, which are both misplacedBig Data needs to focus on the right new thing – focus on data stitching from disparate data sourcesData APIs need to be front and center of any Big Data dialog – too little discussion on that

THANK YOUQuestions and ideas to:@jhingran

http://blog.apigee.com	3080
http://apigee.com	70
http://feeds.apigee.com	24
http://blog.sonoasystems.com	14
https://si0.twimg.com	3
http://blog.edit.apigee.net	2
http://webcache.googleusercontent.com	2
http://tweetedtimes.com	2
http://mktg-dev.wearepropeople.md	2
http://conversation.ecairn.com	1
http://edit.apigee.net	1
http://www.twylah.com	1
http://ip54.216-86-157.static.steadfast.net	1
http://www.hanrss.com	1
http://apigee.md	1
http://blog.local	1
http://edit.mktg.jupiter.apigee.net	1

Big Data: Beyond the "Bigness" and the Technology (webcast)

by Apigee

Accessibility

Categories

Upload Details

Usage Rights

17 Embeds 3,207

Statistics

Big Data: Beyond the "Bigness" and the Technology (webcast) Presentation Transcript