Tim Estes - Generating dynamic social networks from large scale unstructured data


Published on

Tim Estes, CEO of Digital Reasoning, delivered this presentation at the Strata Conference (Feb 2011). It discusses how large scale blog data can be mined to yield social networks of influencers, connections, discussion topics, etc.

Published in: Technology, News & Politics
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Tim Estes - Generating dynamic social networks from large scale unstructured data

  1. 1. Generating Dynamic Social Networks from Large Scale Unstructured DataEnterprise Software to Make Sense of Really Junky Data Tim Estes - CEO, Digital Reasoning
  2. 2.  What We’ll Discuss• What is a social network? • The web of relationships between entities that influences actions• Why does it matter? • To reference Aesop: “You are known by the company you keep.”• What’s required to build one algorithmically? • What’s similar, what’s the same, what’s connected
  3. 3.  What’s similar?We use patented algorithms for deducing related terms from the data… Bush White Justin Britney Nashville House Timberlake Spearspresident bush house tenn miley cyrus britney spearspresident george w gov the predators pussycat dolls the albumadministration white predators bob dylan x factorbush administration clinton oakland nine inch nails my friendsgeorge the administration milwaukee rock star mtvgeorge w president-elect st louis the timberwolves madonnageorge bush barack obama carolina sean preston lady gagabrown barack a season lanarkshire singeramerican president george w baltimore ticket prices a studentclinton kentucky nme
  4. 4.  What’s the same?Concept resolution: Roll up similar things into groups of the same (again, algorithmically) Example: Tony Blair
  5. 5.  What’s connected?Link analysis: Show who and what are connected (again, you guessed it, algorithmically) Terrorist Leader Connections
  6. 6. Let’s Put an Idea to the Test... With powerful analytics can you remove some or most of the need for a priori structure in designing and understanding social networks or other quasi- YES ontological schemas? and Can you also do it with messy unstructured data? YES
  7. 7. But first... Why do we (Digital Reasoning) care?
  8. 8. Because its what we do for a living. We make sense of the senseless. Our customers have critical needs - Digital Reasoning works primarily in the Defense and Intelligence Community making sense of noisy, unstructured data and turning it into usable entity-centric systems supporting mission critical intelligence. The data is big and bad - Little structure in content, topics all over the place, and totally different ontologies/schemas across the community. The times we live in create urgencies - We care because the better and faster we are at making sense of this kind of data, the safer our country is.
  9. 9. Why did we take a data-centric, deployed software model? Unique Environments - Given who our customers are... we can’t host their data. No one can. The solution had to be a pure deployed software model. Meaning in Hard to Reach Places - The data is basically a bunch of pieces that don’t want to be connected. People that don’t want to be found. Result? - Imagine trying to turn that kind of data in that type of architecture from a bunch of loose communication into a social network that has patterns of life, weightings of influence, and projections of probable future actions...
  10. 10. Here’s what it looks like in an architecture…
  11. 11. Now let’s show what can be learned with a little application of Entity-Oriented Analytics to a bunch of web data.
  12. 12. Test Case Web Blog+Wikipedia data (collected by Fetch) - 6M Blog URLs collected over 1Yr + - 16M unique blog messages - no unifying these, topic or author - tricky to get “good” big data from the open web. ended up using .5% of that original source. 1TB became 4GB. No a priori structure, sparse metadata, nearly all meaning emerges from analysis Let’s see what we can find out...
  13. 13. Examining connections related to “Carl Icahn” The data shows connections to and from Carl Icahn by: • people • periodicalsOn closer examination • topicsthe data tells us: • companiesCarl Icahn “is backing” astartup company that“would build” productsrelated to Barack Obama
  14. 14. Let’s examine what connections we find to “Egypt” Egypt is identified as a location, as an organization (country) and as an unassigned entity with all related connectionsOn closer examination we seeinteresting connections in theblogs for Egypt, Cairo, Issuesand the phrase “powder keg”.If we drill down into the actualblog entry we see the context ofthe connections
  15. 15. How about connections to “Steve Jobs”?One connection isconnections in The entities and interesting: Topics“Steve Jobs” to “Walt Mossberg” the blog data are vast – whichto “Kindle” is not surprising. AuthorsSynthesys shows the of authorsThe large amount reason forconnection as “pricing” popularityand topics reflect theof Steve Jobswordawe see theClicking on this as blog subjectcontext of the connection
  16. 16. Demo Platform Synthesys Platform Beta  elastic  user-driven  entity-oriented-analytics on demand
  17. 17. Observations New innovations will be algorithmic and focused on turning hard- to-use data into dynamic, evolving knowledge that can automate machine execution Architectures/solutions will have to accommodate customers that don’t want to move their data to a Public Cloud It is a true statement... “If you can connect the dots, you can connect the people”
  18. 18. So why should You care? Because there is a lot of data that doesn‘t belong on a shared grid. Such as Top Secret data, Sensitive Corporate Data, and Personal Data. Because people may want to own (Personal Computing model) vs. rent (Mainframe model) analytics Because you may not want to convert your data to fit the model of the hosted solution or map to their ontology to get the answers you need.
  19. 19. To learn more… See us at: - Strata Science Fair (Wed evening 6:45PM) - Digital Reasoning Booth #305 - www.digitalreasoning.com
  20. 20. Questions?Automated Understanding, Trusted Decisions, True Intelligence