Building a Lightweight Discovery Interface for China's Patents@NYC Solr/Lucene Meetup
•
2 likes•1,356 views
War stories from building GPSN, a US Federal site for searching China's patents.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Building a Lightweight Discovery Interface for China's Patents@NYC Solr/Lucene Meetup
Similar to Building a Lightweight Discovery Interface for China's Patents@NYC Solr/Lucene Meetup (20)
More from OpenSource Connections
More from OpenSource Connections (20)
Recently uploaded
Recently uploaded (20)
Building a Lightweight Discovery Interface for China's Patents@NYC Solr/Lucene Meetup
- 1. BUILDING A LIGHTWEIGHT DISCOVERY INTERFACE FOR CHINESE PATENTS ! New York Solr/Lucene Meetup ERIC PUGH | epugh@o19s.com | @dep4b
- 2. Who am I? • Principal of OpenSource Connections - Solr/Lucene Search Consultancy http://bit.ly/OSCCommercialSummary • Member of Apache Software Foundation • SOLR-284 UpdateRichDocuments (July 07) • Fascinated by the art of software development
- 4. Congrats to Trey and Tim!
- 5. Agilista
- 9. • First USPTO application in “the cloud” • Simple, and discoverable • Expresses our philosophy of “Cloud meets Ocean” ! • Check it out at http:// gpsn.uspto.gov
- 10. Telling some stories ➡How to inject “Discovery” into your app • The Cloud to the Rescue (sorta!) • Parsers and Parsers and Parsers • Don’t be Afraid to Share!
- 11. Flow of understanding Data UnderstandingInformation
- 13. Grok data at gut level Look for outliers ! User Interviews Surveys Card Sorting Scenarios/Personas ! UX Data brainstorm Mockups Proof of concept ! !
- 14. Where to spend time? UX Engine Data 40% ! 20% ! 40% ! 40% ! 40% ! 20% We spent !
- 15. Telling some stories • How to inject “Discovery” into your app ➡The Cloud to the Rescue (sorta!) • Parsers and Parsers and Parsers • Don’t be Afraid to Share!
- 16. Boy meets Girl Story
- 17. Boy meets Girl Story Metadata Ingest Pipeline Discovery UX Content Files
- 18. Nothing but JS and Solr! • Updates are quarterly • User state in browser • Solr is the “RESTful” API ;-) • KISS: EmberJS + Solr
- 19. How we built it EmberJS Single Page Search App HTML XML JSON Server Dashboard GPSN UI (Bootsrap CSS) Browsers Mobile/ Tablet Third Party Application Servers S3 BucketSolr
- 20. Yes, Solr is hangout out there on the Net… • Using Jetty container security to lock down everything but the /select handler. • Yes, the /admin interface appears to load, but no panels load. • Go ahead, do a delete query! I dare you. Actually, please don’t. ;-)
- 21. Single 550 GB index • Solr + Index are in a Amazon AMI image. • Currently running two independent Solrs. • Optimize works! Still. • Elastic Load Balancer + AutoScale spins up more Solr’s if needed. • Threw lots of “provisioned IOPS” atVM
- 22. A better security proxy from Alex? https://github.com/ dergachev/solr- security-proxy
- 23. Spyglass • EmberJS based Widget framework • List of Results • Facets • Autocomplete • “Deploy” is just .html + .js. S3 bucket! • Tooling is a pain. EmberJS is complex! BetterthenAjaxSolr!
- 25. Key scaling concept behind GPSN: ! Cloud meets Ocean
- 28. Lessons Learned
- 29. Don’t Move Files • Copying 5 TB data up to S3 was very painful. • We used S3Funnel which is “rsync like” • We bought more network bandwidth for our office
- 30. Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. –Andrew Tanenbaum, 1981
- 31. Data Size 0 250000 500000 750000 1000000 1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011 Patent Count 277871
- 32. Think about DataVolume • Started with older dataset, and tasks like TIFF -> PNG conversion became progressively harder. Map/Reduce nice, need more visibility into progress.. • Should have sharded our Search Index from the beginning just to make indexing faster and cheaper process (500 gb index!) • 8 shards dropped time from 12 hours to 2 hours. Merging took 5! • We had too many steps in our pipeline
- 33. Building a Patents Index MachineCount 0 75 150 225 300 5 days 3 days 30 Minutes 1 5 300
- 34. Telling some stories • How to inject “Discovery” into your app • The Cloud to the Rescue (sorta!) ➡Parsers and Parsers and Parsers • Don’t be Afraid to Share!
- 35. Why so many pipelines? Morphlines
- 36. Tika as a pipeline?
- 37. Lot’s of File Types • Sometimes in ZIP archives, sometimes not! • multiple XML formats as well as CSV and EDI • Purplebook,Yellowbook, Redbook,Greenbook, Questel, SIPO…
- 38. Tika as a pipeline! • Auto detects content type • Metadata structure has all the key/value needed for Solr • Allows us to scale up with Behemoth project (and others!).
- 39. Lots of files! HHHHHT APS1 ISSUE - 760106! PATN! WKU 039302717! SRC 5! APN 5328756! APT 1! ART 353! APD 19741216! TTL Golf glove! ISD 19760106! NCL 4! ECL 1 <PatentGrant>! <BibliographicData>! <GrantIdentification>! <DocumentKindCode>B1</DocumentKindCode>! <GrantNumber>06644224</GrantNumber>! <CountryCode>US</CountryCode>! <IssueDateText>2003-11-11</IssueDateText>
- 40. Detector to pick File public class GreenbookDetector implements Detector { ! private static Pattern pattern = Pattern.compile("PATN"); @Override public MediaType detect(InputStream stream, Metadata metadata) throws IOException { ! MediaType type = MediaType.OCTET_STREAM; InputStream lookahead = new LookaheadInputStream(stream, 1024); String extract = org.apache.commons.io.IOUtils.toString(lookahead, "UTF-‐8"); ! Matcher matcher = pattern.matcher(extract); ! if (matcher.find()) { type = GreenbookParser.MEDIA_TYPE; } ! lookahead.close(); return type; } }
- 41. Telling some stories • How to inject “Discovery” into your app • The Cloud to the Rescue (sorta!) • Parsers and Parsers and Parsers ➡Don’t be Afraid to Share!
- 42. Your Search solution isn’t perfect • Allow users to export data • Most business users want to work in Excel! Accept it! • Allow other applications to build on top of it.
- 43. GPSN has • Lots of easy “Print to PDF” options. • Data stored in S3 as: • individual patent files • chunky downloads. • Filtering to expand or select specific data sets. • Permalinks: simple, very sharable URLs. • Underlying Solr service is exposed to public via proxy. You can query Solr yourself. • Need advance querying? Use Lucene syntax in search bar.
- 45. Measuring the impact of our algorithms changes is just getting harder as we get smarter.
- 46. www.quepid.com Quepid: Give your Queries some Love W e need betausers!
- 47. Thank you! ! Questions? • epugh@o19s.com • @dep4b • www.opensourceconnections.com • slideshare.com/o19s Nervous about speaking up? Ask me later!