Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ikanow oanyc summit

4,526 views

Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Ikanow oanyc summit

  1. 1. REALIZING BUSINESS VALUEFROM OPEN SOURCE DATAAND OPEN SOURCEINTELLIGENCEPresented by: Chris Morgan
  2. 2. http://bit.ly/data-vendingDATA AND ART (PRIMER)Providing value on the potential of bad news to serveout a bag of salty potato chipsharnessing the power of open data and sentiment
  3. 3. Data IntelligenceOperational LensIntelligence is information that has been transformed to meet anoperational needIntelligence
  4. 4. Intelligence CycleNo matter what methodology you use…intelligence analysis is an iterative process.
  5. 5. • Provide value to the organization – turn data intointelligence using an “operational lens”• Ensure cyclical feedback occurs duringcollection, processing, analysis, and consumption• Validate that a particular network is the rightsource of data for the questions you needansweredOpen Source Analysis Goals
  6. 6. Common PitfallsAnalyzing What Instead of WhyThe important thing is often not whatpeople are saying… but why they aresaying it.
  7. 7. Common PitfallsUsing the Wrong Analysis ToolsReporting tools rarely help dig into the why. Many commontools, reports, and metrics are misleading:– Word clouds atomize message context– Sentiment metrics are often highly inaccurate– Information in aggregate hides more than it reveals
  8. 8. Use CaseSentiment Analysishttp://bit.ly/ikanow-and-r
  9. 9. Enron Sentiment AnalysisData source~500,000 Publically available Enron emailshttp://bit.ly/ikanow-and-r
  10. 10. Enron Sentiment AnalysisHypothesisUtilize Sentiment analysis as first orderprocess to prioritize and streamline the overallanalysis processhttp://bit.ly/ikanow-and-r
  11. 11. Enron Sentiment AnalysisCaveats Sentiment was only attributed to the sender Not a complete representation of an organizations emailcorpus Counteraction of uneven coverage was estimated Not a full analysis of the set of information (objective wasto use sentiment analysis as a reduction technique)http://bit.ly/ikanow-and-r
  12. 12. Workflow• Data Ingestion Process– Extraction of entities, events, facts and some basicstatistics• Aggregation and Reduction– Aggregation of keywords with sentiment from eachemail– Average sentiment score– Follow on aggregation by email address of thesender over a given week (average sentiment score)• Visualize and Analyze– Imported into Infinit.e and R for visualizationhttp://bit.ly/ikanow-and-r
  13. 13. • Horizontal Bar– Positive sentiment = Green– Negative sentiment = Red• Chart on Left– Positive sentiment = Green– Negative sentiment = Red• Chart on Right– Heuristic – weeks withabrupt negative shiftsindicated problems inorganization– Positive sentiment = Blue– Negative sentiment = RedOne email sender’s Weekly Average Sentiment across timeWorkflow
  14. 14. Workflowclose-up snapshot of sub-set of 20 individuals emailaverage sentiment score over time
  15. 15. Individual analysis based onthe reduction of theinformation by the sentimentanalysis processWorkflow
  16. 16. Findings• Indicators and Additional Analysis– 801 weeks highlighted out of 11,500 weeks asimportant for further investigation– Keywords found could further be used to investigatestatistically the 801 weeks highlighted for manualreview– Individual evaluation of emails highlighted through areduction process (case construction)– Pipeline created for further analysis
  17. 17. Lessons Learned1. Drastically reduced thetimeline necessary for caseconstruction
  18. 18. Lessons Learned2. Multiple contexts for this typeof technique Intelligence Analysis E-Discovery Brand management Social Media Analysis
  19. 19. Lessons Learned3. Negative shifts were onlyinvestigated, analysis of the positivityside for other use cases could beapplied to different questions easily
  20. 20. Lessons Learned4. R and Infinit.e provide ainteresting technology integrationfor evaluating and reducingunstructured data
  21. 21. Chris Morgancmorgan@ikanow.comwww.ikanow.comTHANK YOUgithub.com/ikanow/infinit.e

×