Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Impact of Open Source Search On
The Intelligence Community
Mats Bjore, Infosphere AB,
opcenter@infosphere.se, 07 OCT 10
2
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
What I Will Cover
•  Impact of Open Source Search On The
Int...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
My Background
4
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
FIRST WAKE UP CALL
10/12/10
Copyright 2010: Infosphere AB
19...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
Challenge no 1
6
TECHNOLOGY CHANGE FAST –
MINDSETS AND ORGAN...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
2nd WAKE UP CALL….
10/12/10
Copyright 2008: Infosphere AB
Co...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
Evolution of the Revolution
8
Information is
Power
Knowledge...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
Challenges for the IC (s)
•  Nature of digital information -...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
Challenges for the IC (s)
•  Nature of digital information
–...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
Early Warning
Academic
Publications
Patents
Alternative
Pres...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
1
2
Information Value Chain
DATA INFORMATION INTELLIGENCE
PA...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
IC Requirement: making sense
–  Who’s involved - How they ar...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
IC(s) Legacy and Opportunities
14
• Legacy vendors have crea...
US DoD guidance memo
•  The U.S. Department of Defense
issued a guidance memo in
October 2009 outlining the
positive aspec...
Europe
•  Since 3 February 2010, the
European Union's Open Source
Observatory and Repository
(OSOR.eu) has been providing
...
So, why the hesitation?
17
•  There are mostly academic & promotional arguments that favor
the OSS
•  Open source need to ...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
Examples
18
NATIONWIDE ALL SERVICES INTELLIGENCE PLATFORM
CO...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
Solution using Lucene as one component in a nationwide
intel...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
Solution using Lucene as one component in a nationwide
intel...
21
WithTV/Audio with speech-to-text feature using Lucene fro
search of the text
Sensemaking tools and OSS search
22
Commerical Intelligence platform using Lucene
23
LUCENE AS COMPONENT IN A REPORT CENTRIC
INTELLIGENCE PLATFORM
24
Safemed
25
Some final comments
and wrap up
26
From librarians, to searchers----
but it does not solve intelligence tasks.
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se2
8
“I see by the current issue of ‘Lab News’, Ridgeway,
that...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
Solve customers problems..
29
WE HAVE CREATED A COMPUTERIZED...
07 OCT 2010 - Mats Bjore
Infosphere AB Opcenter@infosphere.se
Contact
•  Mats Björe
•  Infosphere AB
•  mats.bjore@infosph...
Upcoming SlideShare
Loading in …5
×

Impact of open source search on the intelligence community

1,143 views

Published on

know about Impact of open source search on the intelligence community

  • Be the first to comment

Impact of open source search on the intelligence community

  1. 1. Impact of Open Source Search On The Intelligence Community Mats Bjore, Infosphere AB, opcenter@infosphere.se, 07 OCT 10 2
  2. 2. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se What I Will Cover •  Impact of Open Source Search On The Intelligence Community •  Who I am •  Defining the intelligence landscape –  Business, Government, Coalition •  Wake up, challenges opportunities •  Some policy statements and reactions •  Real world Intelligence examples 3
  3. 3. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se My Background 4
  4. 4. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se FIRST WAKE UP CALL 10/12/10 Copyright 2010: Infosphere AB 1969 1990 1994 2010
  5. 5. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se Challenge no 1 6 TECHNOLOGY CHANGE FAST – MINDSETS AND ORGANIZATIONS DON´T
  6. 6. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se 2nd WAKE UP CALL…. 10/12/10 Copyright 2008: Infosphere AB Collection AnalysisDissemination Planning
  7. 7. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se Evolution of the Revolution 8 Information is Power Knowledge is Power Sharing is Power - 1994 1995-2007 2008 Governments •  200 •  Need for intelligence Corporations •  +77 Million •  Need for information • Search and Analytics intensive • Creates isolated units even within a nation. • Collect, Store and Re-Retrieve, Analyze React and Act. • Enforces existing methods on new media • Build own systems based on existing rules and culture • Violates copyright rules to save money (sometimes) • Simple • BI intensive • Uses a mix of consultants, research reports, in-house knowledge. • Buy, Compare, Analyze and Act • Benchmark and create rules for market leadership • Live with media • Buy rights to use information • Complex ( M&A)
  8. 8. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se Challenges for the IC (s) •  Nature of digital information - From Data to Text to Media Mining 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se 9 Data mining is sorting through data to identify patterns and establish relationships. Association - looking for patterns where one event is connected to another event Sequence or path analysis - looking for patterns where one event leads to another later event Classification - looking for new patterns Clustering - finding and visually documenting groups of facts not previously known Forecasting - discovering patterns in data that can lead to reasonable predictions about the future Text mining, also known as intelligent text analysis, text data mining or knowledge-discovery in text (KDT), refers generally to the process of extracting interesting and non-trivial information and knowledge from unstructured text. Text mining is a young interdisciplinary field which draws on information retrieval, data mining, machine learning, statistics and computational linguistics. As most information (over 80%) is stored as text, text mining is believed to have a high commercial potential value. Media Mining, also known as intelligent text analysis, text data mining or knowledge-discovery in text (KDT), refers generally to the process of extracting interesting and non-trivial information and knowledge from unstructured text. Text mining is a young interdisciplinary field which draws on information retrieval, data mining, machine learning, statistics and computational linguistics. As most information (over 80%) is stored as text, text mining is believed to have a high commercial potential value.
  9. 9. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se Challenges for the IC (s) •  Nature of digital information –  From Data to Text to Media Mining –  Volumes •  Digital copycats •  Languages –  Original, Machine translated, transcribed, mixed •  Snippets –  The Moreover syndrome, Blog posts, Social media –  Location •  Internal Silos –  Mental, Security, Organizational •  External Silos –  Free and for fee - but how to connect? 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se 10
  10. 10. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se Early Warning Academic Publications Patents Alternative Press Trade Publications Research Reports ChatRooms Personal Web Sites Online New Sites News Groups News Groups Chat Rooms e-commerce sites Chat Rooms Chat Rooms Chat Rooms Chat Rooms News Groups News Groups News Groups Chat Rooms Personal Web Sites Personal Web Sites Personal Web Sites Personal Web Sites Personal Web Sites Personal Web Sties PersonalWeb Sites Personal Web Sites Chat Rooms News Groups Online New Sites Online New SitesOnline New Sites Online New Sites Online New Sites OnlineNew Sites Online New Alternative Press Alternative Press Patents Patents Patents Trade Publications Trade Publications Trade Publications Research ReportsNews Magazines Periodical Magazines Periodical Magazines News Magazines Quality of DataRaw Synthesized Timeless of DataInstantaneous Historical e-commerce sites e-commerce sites Academic Publications Patents Alternative Press Trade Publications Research Reports ChatRooms Personal Web Sites Online New Sites News SMS News Groups Chat Rooms e-commerce Chat Rooms Chat Rooms Chat Blogs Chat Rooms News Groups News Groups News Grops Chat Rooms Personal Web Sites Personal Web Sites Personal Web Sites Personal Web Sites Personal Web Sites Personal Web Sties Personal W e b Sites Personal Web Sites Chat News Groups Online New Sites Online New SitesOnline New Sites Online New Sites Online New Sites OnlineNew Sites Offline News Alternative Press Alternative Press Patents Patents Patents Trade Publications Trade Publications Trade Publications Research ReportsNews Magazines Periodical Magazines Periodical Magazines News Magazines Quality of DataRaw Synthesized Timeliness of DataInstantaneous Historical e-commerce e-commerce sites MMS Search Monitor Receive Shape Control Follow Active Passive Late Reaction The Big Challenge=TIME TO PRODUCT Twitter
  11. 11. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se 1 2 Information Value Chain DATA INFORMATION INTELLIGENCE PAST PRESENT FUTURE Contextualized Categorized Calculated Corrected Condensed Compared Connections Calculated Consequences Connections Conversations Chances Data becomes information when asked for Intelligence becomes information when not needed PASSIVE ACTIVE PROACTIVE Information becomes data when not needed INFORMATION VOLUME LEVEL OF SYNTHESIS (ANALYSIS) AND CONTEXT
  12. 12. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se IC Requirement: making sense –  Who’s involved - How they are related - Where it happened- What people are saying - Who has written about it - Who has written about related issues - What topics or categories of information are involved –  Predictions ( hypothesis based) - Fact based analytics – etc. –  Storage & Retrieval Tools that can ”document” conclusions, facts, relationships, sentiments- and that constantly be triggered, questioned, challenged and further validated by the incoming information 13
  13. 13. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se IC(s) Legacy and Opportunities 14 • Legacy vendors have created customs & relationships that are entrenched within government and the “beltway bandits” ( in every country) • 80 per cent of government IT spending in the UK goes to only five companies • Lack of knowledge that open-source equivalents to proprietary software exists.
  14. 14. US DoD guidance memo •  The U.S. Department of Defense issued a guidance memo in October 2009 outlining the positive aspects of OSS that should be considered when conducting market research on software for Department use. •  Some of the benefits noted in the memo include: •  . •  The continuous and broad peer-review enabled by publicly available source code supports software reliability and security efforts through the identification and elimination of defects that might otherwise go unrecognized by a more limited core development team. •  The unrestricted ability to modify software source code enables the Department to respond more rapidly to changing situations, missions, and future threats. •  Reliance on a particular software developer or vendor due to proprietary restrictions may be reduced by the use of OSS, which can be operated and maintained by multiple vendors, thus reducing barriers to entry and exit •  By sharing the responsibility for maintenance of OSS with other users, the Department can benefit by reducing the total cost of ownership for software, particularly compared with software for which the Department has sole •  responsibility for maintenance •  OSS is particularly suitable for rapid prototyping and experimentation, where the ability to "test drive" the software with minimal costs and administrative delays can be important. 15
  15. 15. Europe •  Since 3 February 2010, the European Union's Open Source Observatory and Repository (OSOR.eu) has been providing the public administrations with an access to more than two thousand free and open source applications. •  The OSOR is a platform where public administrations can exchange information and experiences and collaborate in developing free and open source software. The platform has managed to bring together more than 2 000 of such open source software applications in just sixteen months after its launch. •  www.OSOR.eu •  http://ec.europa.eu/idabc/ en/document/2623 •  http://cordis.europa.eu/ fp7/ict/ssai/foss- home_en.html 16
  16. 16. So, why the hesitation? 17 •  There are mostly academic & promotional arguments that favor the OSS •  Open source need to industry to prove that it can deliver cost savings compared with proprietary technology •  Provide business cases with articulated open source as cheaper than proprietary. - Shift from the academic discussion to business discussion! •  Applications without formal support and training •  Mindsets within organizations & companies •  Legal questions about licenses / uncertainties •  Perception that products change too much •  Businesses want the comfort of having a relationship with a commercial account manager from a software firm, rather than relying on the developer community for help and support •  Large organizations demand product warranties and service agreements. •  Procurement processes is not being set up in proper ways.
  17. 17. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se Examples 18 NATIONWIDE ALL SERVICES INTELLIGENCE PLATFORM COMMERICAL INTELLIGENCE APPLICATION DOCUMENTATION & SEARCH FOR A SECURITY SERVICE RISK SOULTION
  18. 18. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se Solution using Lucene as one component in a nationwide intelligence platform
  19. 19. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se Solution using Lucene as one component in a nationwide intelligence platform 20
  20. 20. 21 WithTV/Audio with speech-to-text feature using Lucene fro search of the text
  21. 21. Sensemaking tools and OSS search 22
  22. 22. Commerical Intelligence platform using Lucene 23
  23. 23. LUCENE AS COMPONENT IN A REPORT CENTRIC INTELLIGENCE PLATFORM 24
  24. 24. Safemed 25
  25. 25. Some final comments and wrap up 26
  26. 26. From librarians, to searchers---- but it does not solve intelligence tasks.
  27. 27. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se2 8 “I see by the current issue of ‘Lab News’, Ridgeway, that you’ve been working for the last 20 years on the same problem I’ve been working on for the last 20 years.” Sharing is power…
  28. 28. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se Solve customers problems.. 29 WE HAVE CREATED A COMPUTERIZED, INTERACTIVE ARTIFICIAL INTELLIGENCE PROFILING INTRANET DEVICE FOR THE UN WITH ENTITY EXTRACTION AGENTS AND VIZUALIZATION. I CALL IT THE ”OSINT-CENTER” AND IT IS RESTRICTED TO 40 COMPUTERS. WONDERFUL. MAKE SOME PHOTOCOPIES AND ROUTE IT AROUND. BUT I REALLY ONLY ASKED FOR THE NAME OF THE GENERAL SECRETARY OF THE LUCID IMAGINATION IS IT ABOUT TECHNOLOGY? OR…. BUSINESS AS IN BUSINESS AS USUAL?
  29. 29. 07 OCT 2010 - Mats Bjore Infosphere AB Opcenter@infosphere.se Contact •  Mats Björe •  Infosphere AB •  mats.bjore@infosphere.se •  +468 611 22 33 •  www.infosphere.se 30 Know more about open source search at www.lucidimagination.com

×