0
REALIZING BUSINESS VALUEFROM OPEN SOURCE DATAAND OPEN SOURCEINTELLIGENCEPresented by: Chris Morgan
http://bit.ly/data-vendingDATA AND ART (PRIMER)Providing value on the potential of bad news to serveout a bag of salty pot...
Data IntelligenceOperational LensIntelligence is information that has been transformed to meet anoperational needIntellige...
Intelligence CycleNo matter what methodology you use…intelligence analysis is an iterative process.
• Provide value to the organization – turn data intointelligence using an “operational lens”• Ensure cyclical feedback occ...
Common PitfallsAnalyzing What Instead of WhyThe important thing is often not whatpeople are saying… but why they aresaying...
Common PitfallsUsing the Wrong Analysis ToolsReporting tools rarely help dig into the why. Many commontools, reports, and ...
Use CaseSentiment Analysishttp://bit.ly/ikanow-and-r
Enron Sentiment AnalysisData source~500,000 Publically available Enron emailshttp://bit.ly/ikanow-and-r
Enron Sentiment AnalysisHypothesisUtilize Sentiment analysis as first orderprocess to prioritize and streamline the overal...
Enron Sentiment AnalysisCaveats Sentiment was only attributed to the sender Not a complete representation of an organiza...
Workflow• Data Ingestion Process– Extraction of entities, events, facts and some basicstatistics• Aggregation and Reductio...
• Horizontal Bar– Positive sentiment = Green– Negative sentiment = Red• Chart on Left– Positive sentiment = Green– Negativ...
Workflowclose-up snapshot of sub-set of 20 individuals emailaverage sentiment score over time
Individual analysis based onthe reduction of theinformation by the sentimentanalysis processWorkflow
Findings• Indicators and Additional Analysis– 801 weeks highlighted out of 11,500 weeks asimportant for further investigat...
Lessons Learned1. Drastically reduced thetimeline necessary for caseconstruction
Lessons Learned2. Multiple contexts for this typeof technique Intelligence Analysis E-Discovery Brand management Socia...
Lessons Learned3. Negative shifts were onlyinvestigated, analysis of the positivityside for other use cases could beapplie...
Lessons Learned4. R and Infinit.e provide ainteresting technology integrationfor evaluating and reducingunstructured data
Chris Morgancmorgan@ikanow.comwww.ikanow.comTHANK YOUgithub.com/ikanow/infinit.e
Upcoming SlideShare
Loading in...5
×

Ikanow oanyc summit

3,733

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
3,733
On Slideshare
0
From Embeds
0
Number of Embeds
39
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Introduction and Topic
  • Introduction and Topic
  • No matter what methodology you use…intelligence analysis is an iterative processYou Collect the data, Store it, Analyze it, and Distribute the end results to your organization in some usable format.
  • Provide value to the organization – turn data into intelligence using an “operational lens” (answer the questions your organization is asking in other words)Ensure cyclical feedback occurs during collection, processing, analysis, and consumption (learn from the process and adjust to based on what you learn, intel gathering and analysis is not a static process)Validate that a particular network is the right source of data for the questions you need answered (i.e. is Twitter the right place to look for data related to weather?)
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Why is someone tweeting or posting? If some checks in from a store is it really because the store is so incredible that they need to share that information or is because they are trying to form an impression about their lifestyle (i.e. image shaping)?Why is much harder than What.What you learn from data can be affected as much by the tools you use to analyze data as by what is contained in the data. Picking the right tools for the job is critical.
  • Transcript of "Ikanow oanyc summit"

    1. 1. REALIZING BUSINESS VALUEFROM OPEN SOURCE DATAAND OPEN SOURCEINTELLIGENCEPresented by: Chris Morgan
    2. 2. http://bit.ly/data-vendingDATA AND ART (PRIMER)Providing value on the potential of bad news to serveout a bag of salty potato chipsharnessing the power of open data and sentiment
    3. 3. Data IntelligenceOperational LensIntelligence is information that has been transformed to meet anoperational needIntelligence
    4. 4. Intelligence CycleNo matter what methodology you use…intelligence analysis is an iterative process.
    5. 5. • Provide value to the organization – turn data intointelligence using an “operational lens”• Ensure cyclical feedback occurs duringcollection, processing, analysis, and consumption• Validate that a particular network is the rightsource of data for the questions you needansweredOpen Source Analysis Goals
    6. 6. Common PitfallsAnalyzing What Instead of WhyThe important thing is often not whatpeople are saying… but why they aresaying it.
    7. 7. Common PitfallsUsing the Wrong Analysis ToolsReporting tools rarely help dig into the why. Many commontools, reports, and metrics are misleading:– Word clouds atomize message context– Sentiment metrics are often highly inaccurate– Information in aggregate hides more than it reveals
    8. 8. Use CaseSentiment Analysishttp://bit.ly/ikanow-and-r
    9. 9. Enron Sentiment AnalysisData source~500,000 Publically available Enron emailshttp://bit.ly/ikanow-and-r
    10. 10. Enron Sentiment AnalysisHypothesisUtilize Sentiment analysis as first orderprocess to prioritize and streamline the overallanalysis processhttp://bit.ly/ikanow-and-r
    11. 11. Enron Sentiment AnalysisCaveats Sentiment was only attributed to the sender Not a complete representation of an organizations emailcorpus Counteraction of uneven coverage was estimated Not a full analysis of the set of information (objective wasto use sentiment analysis as a reduction technique)http://bit.ly/ikanow-and-r
    12. 12. Workflow• Data Ingestion Process– Extraction of entities, events, facts and some basicstatistics• Aggregation and Reduction– Aggregation of keywords with sentiment from eachemail– Average sentiment score– Follow on aggregation by email address of thesender over a given week (average sentiment score)• Visualize and Analyze– Imported into Infinit.e and R for visualizationhttp://bit.ly/ikanow-and-r
    13. 13. • Horizontal Bar– Positive sentiment = Green– Negative sentiment = Red• Chart on Left– Positive sentiment = Green– Negative sentiment = Red• Chart on Right– Heuristic – weeks withabrupt negative shiftsindicated problems inorganization– Positive sentiment = Blue– Negative sentiment = RedOne email sender’s Weekly Average Sentiment across timeWorkflow
    14. 14. Workflowclose-up snapshot of sub-set of 20 individuals emailaverage sentiment score over time
    15. 15. Individual analysis based onthe reduction of theinformation by the sentimentanalysis processWorkflow
    16. 16. Findings• Indicators and Additional Analysis– 801 weeks highlighted out of 11,500 weeks asimportant for further investigation– Keywords found could further be used to investigatestatistically the 801 weeks highlighted for manualreview– Individual evaluation of emails highlighted through areduction process (case construction)– Pipeline created for further analysis
    17. 17. Lessons Learned1. Drastically reduced thetimeline necessary for caseconstruction
    18. 18. Lessons Learned2. Multiple contexts for this typeof technique Intelligence Analysis E-Discovery Brand management Social Media Analysis
    19. 19. Lessons Learned3. Negative shifts were onlyinvestigated, analysis of the positivityside for other use cases could beapplied to different questions easily
    20. 20. Lessons Learned4. R and Infinit.e provide ainteresting technology integrationfor evaluating and reducingunstructured data
    21. 21. Chris Morgancmorgan@ikanow.comwww.ikanow.comTHANK YOUgithub.com/ikanow/infinit.e
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×