Your SlideShare is downloading. ×
MapR and Lucidworks Joint Webinar 2012
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

MapR and Lucidworks Joint Webinar 2012


Published on

Slides from webinar given by Ted Dunning and LucidWorks Chief Scientist, Grant Ingersoll on how search technology can be abused to implement apparently intelligent systems

Slides from webinar given by Ted Dunning and LucidWorks Chief Scientist, Grant Ingersoll on how search technology can be abused to implement apparently intelligent systems

Published in: Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • TED: We can tighten or loosen as necessary.
  • TED: I think that the agenda needs to go here because it otherwise breaks up some key flow
  • TED: This is a money slide where people should say “Wow man”. They shouldn’t understand the implications of this, but they should be very, very aware that something big just slide into the room.Tech Building Block: Not just textNot just users + queriesEmbrace Fuzziness: Esp. in Big Data, it is the only way you are going to survive.TED: I think that this should make the case for advanced that is still search at its heart. The idea that search can be radically changed should be on the next slide.
  • Search Abuse Can discuss how I started just doing free text, but then a curious thing happened, started to see people using the engine for things like: key/value, denormalized DBs, browsing engines, plagiarism detection, teaching languages, record linkage and much, much moreSearch has added more DB features over the yearsTED: We need to introduce the idea of *REVOLUTION* somewhere in here.
  • All that revolution is good, but what the heck does this have to do w/ Big Data?
  • GSI: needs a bit more meat
  • Service-Oriented ArchitectureStatelessFailover/Fault TolerantLightweight Coordination and MessagingSmart about UpdatesDocument store isDistributedScalableAnalysisBatchNear Real-Time
  • Transcript

    • 1. 1©MapR Technologies - Confidential Crowd Sourcing Reflected Intelligence Using Search and Big Data Ted Dunning Grant Ingersoll
    • 2. 2©MapR Technologies - Confidential Grant’s Background  Co-founder: – LucidWorks – Chief Scientist – Apache Mahout  Long time Lucene/Solr committer  Author: Taming Text  Background in IR and NLP – Built CLIR, QA and a variety of other search-based apps
    • 3. 3©MapR Technologies - Confidential Ted’s Background  Academia, Startups – Aptex, MusicMatch, ID Analytics, Veoh – Big data since before big  Open source – since the dark ages before the internet – Mahout, Zookeeper, Drill – bought the beer at first HUG  MapR – Chief Application Architect  Founding member of Apache Drill
    • 4. 4©MapR Technologies - Confidential Agenda  Intro  Search Evolution and Search Revolution  Reflected Intelligence Use Cases  Building a Next Generation Search and Discovery Platform – MapR – LucidWorks  1+1=3
    • 5. 5©MapR Technologies - Confidential Search is Dead, Long Live Search  Search is a system building block – text is only a part of the story  If the algorithms fit, use them!  Embrace fuzziness!  Scoring features are everywhere Content User Interaction Access Content Relationships
    • 6. 6©MapR Technologies - Confidential Search (R)evolution  Search use leads to search abuse – denormalization frees your mind – scoring is just a sparse matrix multiply  Lucene/Solr evolution – non free text usages abound – many DB-like features – noSQL before NoSQL was cool – flexible indexing – finite State Transducers FTW!  Scale  “This ain’t your father’s relevance anymore”
    • 7. 7©MapR Technologies - Confidential Add (Lots of) Water  Large-scale analysis is key to reflected intelligence – correlation analysis • based on queries, clicks, mouse tracks, even explicit feedback • produce clusters, trends, topics, SIP’s – start with engineered knowledge, refine with user feedback  Large-scale discovery features encourage experimentation  Always test, always enrich! Search DiscoveryAnalytics
    • 8. 8©MapR Technologies - Confidential Social Media Analysis in Telecom  Correlate mobile traffic analysis with social media analysis – events cause traffic micro-bursts – participants tweet the events ahead of time  Deploy operations faster to predict outages and better handle emergency situations – high cost bandwidth augmentation can be marshaled as the traffic appears – anticipation beats reaction
    • 9. 9©MapR Technologies - Confidential Provenance is 80% of value  Analysis of social media to determine advertising reach and response  In one case the same untargeted advertising was worth 5x if sold with supporting data.
    • 10. 10©MapR Technologies - Confidential Claims Analysis  Goal – Insurance claims processing and analysis – fraud analysis  Method – Combine free text search with metadata analysis to identify high risk activities across the country – Integrate with corporate workflows to detect and fix outliers in customer relations  Results – Questions that took 24-48 hours now take seconds to answer
    • 11. 11©MapR Technologies - Confidential Virginia Tech - Help the World  Grab data around crisis  Search immediately  Large-scale analysis enriches data to find ways to improve responses and understanding 
    • 12. 12©MapR Technologies - Confidential Bright Planet - Catch the Bad Guys  Online Drug Counterfeit detection  Identify commonly used language indicating counterfeits – you know it when you see it – and you know you have seen it  Feed to analyst via search-driven application – enrich based on analysts feedback
    • 13. 13©MapR Technologies - Confidential Veoh - Cross Recommendations  Cross recommendation as search – with search used to build cross recommendation!  Recommend content to people who exhibit certain behaviors (clicks, query terms, other)  (Ab)use of a search engine – but not as a search engine for content – more like a search engine for behavior
    • 14. 14©MapR Technologies - Confidential What Platform Do You Need?  Fast, efficient, scalable search – bulk and near real-time indexing – handle billions of records with sub-second search and faceting  Large scale, cost effective storage and processing capabilities  NLP and machine learning tools that scale to enhance discovery and analysis  Integrated log analysis workflows that close the loop between the raw data and user interactions
    • 15. 15©MapR Technologies - Confidential Shards 1 2 3 N Search View •Documents •Users •Logs Document Store Analytic Services •View into numeric/histo ric data •Classification •Recommendation Personalization & Machine Learning Services Classification Models In memory Replicated Multi-tenant Discovery & Enrichment Clustering, classificat ion, NLP, topic identification, searc h log analysis, user behavior Content Acquisition ETL, batch or near real-time Access APIs Data • LucidWorks Search connectors • Push Reference Architecture
    • 16. 16©MapR Technologies - Confidential MapR  MapR provides the technology leading Hadoop distribution – full eco-system distribution – integrated data platform – complete solution for data integrity  MapR clusters also provide tight integration with search technologies like LucidWorks – integration is key for effective ops
    • 17. 17©MapR Technologies - Confidential LucidWorks  LucidWorks provides the leading packaging of Apache Lucene and Solr – build your own, we support – founded by the most prominent Lucene/Solr experts  LucidWorks Search – “Solr++” • UI, REST API, MapR connectors, relevance tools, much more  LucidWorks Big Data – Big Data as a Service – Integrated LucidWorks Search, Hadoop, machine learning with prebuilt workflows for many of these tasks
    • 18. 18©MapR Technologies - Confidential LucidWorks Big Data Architecture Big Data Operating System • Administration • Provisioning • Monitoring • Configuration • Service Management • Data Management • Security System Management Uniform ReST API Search – Discovery – Analytics • LucidWorks Search • Machine Learning (classification, clustering, recommendations) • Natural Language Processing • SQL (Hive) Interface • Data Workflows (ETL, log analysis, common metrics) • Extensible • Enterprise Repository • Social Media • Databases • HDFS • Cloud (S3) • Push Content Acquisition Hadoop/HBase Search Indexes Search Logs
    • 19. 19©MapR Technologies - Confidential Easy Wins  Analyze logs from application stored in MapR  Seamlessly store search indexes in MapR – and feed to Pig, Mahout and others – use mirrors + NFS to directly deploy indexes  Snapshots make backups a snap  LucidWorks 2.5 (2013 Q1) easily connects with MapR
    • 20. 20©MapR Technologies - Confidential 1 + 1 = 3
    • 21. 21©MapR Technologies - Confidential Learn More  More information  Vote for this topic for Hadoop Summit EU:  Talk to Ted @ted_dunning  Talk to Grant @gsingers  MapR and Lucid Works