Building a Lightweight Discovery Interface for China's Patents@NYC Solr/Lucene Meetup

758 views
666 views

Published on

War stories from building GPSN, a US Federal site for searching China's patents.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
758
On SlideShare
0
From Embeds
0
Number of Embeds
196
Actions
Shares
0
Downloads
3
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Building a Lightweight Discovery Interface for China's Patents@NYC Solr/Lucene Meetup

  1. 1. BUILDING A LIGHTWEIGHT DISCOVERY INTERFACE FOR CHINESE PATENTS ! New York Solr/Lucene Meetup ERIC PUGH | epugh@o19s.com | @dep4b
  2. 2. Who am I? • Principal of OpenSource Connections - Solr/Lucene Search Consultancy http://bit.ly/OSCCommercialSummary • Member of Apache Software Foundation • SOLR-284 UpdateRichDocuments (July 07) • Fascinated by the art of software development
  3. 3. Co-Author N extEdition June!
  4. 4. Congrats to Trey and Tim!
  5. 5. Agilista
  6. 6. Selected Customers
  7. 7. Telling some stories war ^
  8. 8. • First USPTO application in “the cloud” • Simple, and discoverable • Expresses our philosophy of “Cloud meets Ocean” ! • Check it out at http:// gpsn.uspto.gov
  9. 9. Telling some stories ➡How to inject “Discovery” into your app • The Cloud to the Rescue (sorta!) • Parsers and Parsers and Parsers • Don’t be Afraid to Share!
  10. 10. Flow of understanding Data UnderstandingInformation
  11. 11. Building “Discovery” Engine UX DataTension
  12. 12. Grok data at gut level Look for outliers ! User Interviews Surveys Card Sorting Scenarios/Personas ! UX Data brainstorm Mockups Proof of concept ! !
  13. 13. Where to spend time? UX Engine Data 40% ! 20% ! 40% ! 40% ! 40% ! 20% We spent !
  14. 14. Telling some stories • How to inject “Discovery” into your app ➡The Cloud to the Rescue (sorta!) • Parsers and Parsers and Parsers • Don’t be Afraid to Share!
  15. 15. Boy meets Girl Story
  16. 16. Boy meets Girl Story Metadata Ingest Pipeline Discovery UX Content Files
  17. 17. Nothing but JS and Solr! • Updates are quarterly • User state in browser • Solr is the “RESTful” API ;-) • KISS: EmberJS + Solr
  18. 18. How we built it EmberJS Single Page Search App HTML XML JSON Server Dashboard GPSN UI (Bootsrap CSS) Browsers Mobile/ Tablet Third Party Application Servers S3 BucketSolr
  19. 19. Yes, Solr is hangout out there on the Net… • Using Jetty container security to lock down everything but the /select handler. • Yes, the /admin interface appears to load, but no panels load. • Go ahead, do a delete query! I dare you. Actually, please don’t. ;-)
  20. 20. Single 550 GB index • Solr + Index are in a Amazon AMI image. • Currently running two independent Solrs. • Optimize works! Still. • Elastic Load Balancer + AutoScale spins up more Solr’s if needed. • Threw lots of “provisioned IOPS” atVM
  21. 21. A better security proxy from Alex? https://github.com/ dergachev/solr- security-proxy
  22. 22. Spyglass • EmberJS based Widget framework • List of Results • Facets • Autocomplete • “Deploy” is just .html + .js. S3 bucket! • Tooling is a pain. EmberJS is complex! BetterthenAjaxSolr!
  23. 23. Daniel Beach’s project https://github.com/ o19s/spyglass
  24. 24. Key scaling concept behind GPSN: ! Cloud meets Ocean
  25. 25. More prosaically… Database Server Server Server Client Client Client $ $ $ $
  26. 26. Lessons Learned
  27. 27. Don’t Move Files • Copying 5 TB data up to S3 was very painful. • We used S3Funnel which is “rsync like” • We bought more network bandwidth for our office
  28. 28. Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
 –Andrew Tanenbaum, 1981
  29. 29. Data Size 0 250000 500000 750000 1000000 1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 Patent Count 277871
  30. 30. Think about DataVolume • Started with older dataset, and tasks like TIFF -> PNG conversion became progressively harder. Map/Reduce nice, need more visibility into progress.. • Should have sharded our Search Index from the beginning just to make indexing faster and cheaper process (500 gb index!) • 8 shards dropped time from 12 hours to 2 hours. Merging took 5! • We had too many steps in our pipeline
  31. 31. Building  a  Patents  Index MachineCount 0 75 150 225 300 5 days 3 days 30 Minutes 1 5 300
  32. 32. Telling some stories • How to inject “Discovery” into your app • The Cloud to the Rescue (sorta!) ➡Parsers and Parsers and Parsers • Don’t be Afraid to Share!
  33. 33. Why so many pipelines? Morphlines
  34. 34. Tika as a pipeline?
  35. 35. Lot’s of File Types • Sometimes in ZIP archives, sometimes not! • multiple XML formats as well as CSV and EDI • Purplebook,Yellowbook, Redbook,Greenbook, Questel, SIPO…
  36. 36. Tika as a pipeline! • Auto detects content type • Metadata structure has all the key/value needed for Solr • Allows us to scale up with Behemoth project (and others!).
  37. 37. Lots of files! HHHHHT APS1 ISSUE - 760106! PATN! WKU 039302717! SRC 5! APN 5328756! APT 1! ART 353! APD 19741216! TTL Golf glove! ISD 19760106! NCL 4! ECL 1 <PatentGrant>! <BibliographicData>! <GrantIdentification>! <DocumentKindCode>B1</DocumentKindCode>! <GrantNumber>06644224</GrantNumber>! <CountryCode>US</CountryCode>! <IssueDateText>2003-11-11</IssueDateText>
  38. 38. Detector to pick File public  class  GreenbookDetector  implements  Detector  {   !        private  static  Pattern  pattern  =  Pattern.compile("PATN");                    @Override          public  MediaType  detect(InputStream  stream,  Metadata  metadata)  throws  IOException  {   !                MediaType  type  =  MediaType.OCTET_STREAM;                  InputStream  lookahead  =  new  LookaheadInputStream(stream,  1024);                  String  extract  =  org.apache.commons.io.IOUtils.toString(lookahead,  "UTF-­‐8");   !                Matcher  matcher  =  pattern.matcher(extract);   !                if  (matcher.find())  {                          type  =  GreenbookParser.MEDIA_TYPE;                  }   !                lookahead.close();                                    return  type;          }         }
  39. 39. Telling some stories • How to inject “Discovery” into your app • The Cloud to the Rescue (sorta!) • Parsers and Parsers and Parsers ➡Don’t be Afraid to Share!
  40. 40. Your Search solution isn’t perfect • Allow users to export data • Most business users want to work in Excel! Accept it! • Allow other applications to build on top of it.
  41. 41. GPSN has • Lots of easy “Print to PDF” options. • Data stored in S3 as: • individual patent files • chunky downloads. • Filtering to expand or select specific data sets. • Permalinks: simple, very sharable URLs. • Underlying Solr service is exposed to public via proxy. You can query Solr yourself. • Need advance querying? Use Lucene syntax in search bar.
  42. 42. One more thought...
  43. 43. Measuring the impact of our algorithms changes is just getting harder as we get smarter.
  44. 44. www.quepid.com Quepid: Give your Queries some Love W e need betausers!
  45. 45. Thank you! ! Questions? • epugh@o19s.com • @dep4b • www.opensourceconnections.com • slideshare.com/o19s Nervous about speaking up? Ask me later!

×