SlideShare a Scribd company logo
1 of 14
Download to read offline
A mini-google for a private phpBB forum
Background Recently joined a non-technical forum Set up by a friend of the founder – hasn’t been seen for 1-2 years Running on shared hosting Search limited to past years content to keep things fast
3 steps Mirror site locally Insert content in a database Release sphinx
The Problems It’s private
The Problems It’s private  It’s not well formed
The Problems It’s private  It’s not well formed Page info contained in query string viewtopic.php?f=30&t=10170
Wget Old faithful – first tool I reach for when I want to download anything from a website Simple for simple tasks but with the flexibility to handle more complex tasks
Wget – Logging in Ability to import a Netscape style cookies file Didn’t work for me Wrong format? Wrongly configured wget? phpBB checking user agent / ip address? Log in directly in wget Multi-step process
Wget – are we there yet? There is massive redundancy in the link structure Every post has an individual link which pulls in the entire topic Can’t exclude based on query string
Zend_HTTP Most of my time spent with PHP and more recently Zend framework There is a lot to be said for using a tool you are familiar with
Zend_HTTP
Scraping HTML First needed to correct errors Tidy extension SimpleXML Need to change xmlns to ns Still doesn’t work in all cases
Releasing Sphinx Ridiculously simple Simple config file adapted from example Runs for ~30s for ~90k posts Added a simply database query to beef up web interface from the example  Only downside – memory footprint
Future tasks Keep index updated Implemented but could be more efficient Exercise to learn python

More Related Content

Viewers also liked

getting your feet wet with jquery
getting your feet wet with jquerygetting your feet wet with jquery
getting your feet wet with jqueryBenjamin Sterling
 
Getting Your Feet Wet With jQuery
Getting Your Feet Wet With jQueryGetting Your Feet Wet With jQuery
Getting Your Feet Wet With jQueryBenjamin Sterling
 
Interactive WebMap Dundee Vineyards, Oregon
Interactive WebMap Dundee Vineyards, OregonInteractive WebMap Dundee Vineyards, Oregon
Interactive WebMap Dundee Vineyards, OregonDonnych Diaz
 
Montinore Estates Slide Show
Montinore Estates Slide ShowMontinore Estates Slide Show
Montinore Estates Slide ShowDonnych Diaz
 
Purple Martins Nesting Sites
Purple Martins Nesting SitesPurple Martins Nesting Sites
Purple Martins Nesting SitesDonnych Diaz
 
EPA Reported Chemical Releases in Zipcode 97124
EPA Reported Chemical Releases in Zipcode 97124EPA Reported Chemical Releases in Zipcode 97124
EPA Reported Chemical Releases in Zipcode 97124Donnych Diaz
 

Viewers also liked (6)

getting your feet wet with jquery
getting your feet wet with jquerygetting your feet wet with jquery
getting your feet wet with jquery
 
Getting Your Feet Wet With jQuery
Getting Your Feet Wet With jQueryGetting Your Feet Wet With jQuery
Getting Your Feet Wet With jQuery
 
Interactive WebMap Dundee Vineyards, Oregon
Interactive WebMap Dundee Vineyards, OregonInteractive WebMap Dundee Vineyards, Oregon
Interactive WebMap Dundee Vineyards, Oregon
 
Montinore Estates Slide Show
Montinore Estates Slide ShowMontinore Estates Slide Show
Montinore Estates Slide Show
 
Purple Martins Nesting Sites
Purple Martins Nesting SitesPurple Martins Nesting Sites
Purple Martins Nesting Sites
 
EPA Reported Chemical Releases in Zipcode 97124
EPA Reported Chemical Releases in Zipcode 97124EPA Reported Chemical Releases in Zipcode 97124
EPA Reported Chemical Releases in Zipcode 97124
 

Recently uploaded

Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationKnoldus Inc.
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsYoss Cohen
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 

Recently uploaded (20)

Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog Presentation
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platforms
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 

Dedicated search for a private phpBB forum using sphinx

  • 1. A mini-google for a private phpBB forum
  • 2. Background Recently joined a non-technical forum Set up by a friend of the founder – hasn’t been seen for 1-2 years Running on shared hosting Search limited to past years content to keep things fast
  • 3. 3 steps Mirror site locally Insert content in a database Release sphinx
  • 5. The Problems It’s private It’s not well formed
  • 6. The Problems It’s private It’s not well formed Page info contained in query string viewtopic.php?f=30&t=10170
  • 7. Wget Old faithful – first tool I reach for when I want to download anything from a website Simple for simple tasks but with the flexibility to handle more complex tasks
  • 8. Wget – Logging in Ability to import a Netscape style cookies file Didn’t work for me Wrong format? Wrongly configured wget? phpBB checking user agent / ip address? Log in directly in wget Multi-step process
  • 9. Wget – are we there yet? There is massive redundancy in the link structure Every post has an individual link which pulls in the entire topic Can’t exclude based on query string
  • 10. Zend_HTTP Most of my time spent with PHP and more recently Zend framework There is a lot to be said for using a tool you are familiar with
  • 12. Scraping HTML First needed to correct errors Tidy extension SimpleXML Need to change xmlns to ns Still doesn’t work in all cases
  • 13. Releasing Sphinx Ridiculously simple Simple config file adapted from example Runs for ~30s for ~90k posts Added a simply database query to beef up web interface from the example Only downside – memory footprint
  • 14. Future tasks Keep index updated Implemented but could be more efficient Exercise to learn python