Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Crawlers and Web UI with full stack JS - PiterJS #1

1,586 views

Published on

Crawl and scrape thousands of bidding contest from hundreds of sources globally every day

Published in: Software
  • Be the first to comment

Crawlers and Web UI with full stack JS - PiterJS #1

  1. 1. Crawlers and Web UI with full stack JS
  2. 2. What is Oppex? • World’s largest source of public sector bidding contests • Crawl and scrape thousands of bidding contest from hundreds of sources globally every day
  3. 3. Stack NodeJS Solr MongoDB React Flux NodeJS + Express Oppex Engine Oppex UI
  4. 4. Oppex Engine • Crawlers • Postprocessors • Data storage • Notifications • Infrastructure
  5. 5. Oppex Engine • Crawlers • Postprocessors • Data storage • Notifications • Infrastructure • Clustered Node.js workers • Kue + Redis for job queues • Pre-defined crawler strategies • Reverse html templates
  6. 6. Oppex Engine • Crawlers • Postprocessors • Data storage • Notifications • Infrastructure • Translation via Google Translate API & Yandex Translate API • Geocoding Google Places API & MapQuest API • Currency parsing, email & phone extraction etc.
  7. 7. Oppex Engine • Crawlers • Postprocessors • Data storage • Notifications • Infrastructure • MongoDB for raw data storage • Solr for search indexing
  8. 8. Oppex Engine • Crawlers • Postprocessors • Data storage • Notifications • Infrastructure • User-saved Solr queries ran at predefined intervals • Mandrill for emails • Socket.io
  9. 9. Oppex Engine • Crawlers • Postprocessors • Data storage • Notifications • Infrastructure • Docker virtualization • Backups to AWS S3 + Glacier • Logstash for logging • Elasticsearch + Kibana for monitoring • Automated deployment scripts
  10. 10. Oppex UI • React + Flux • Babel • Webpack • Immutable.js • Stylus
  11. 11. Oppex UI • React + Flux • Babel • Webpack • Immutable.js • Stylus • Great for large and complex apps • Stop fiddling with DOM • Encourages modular design • Ensures data integrity across the app
  12. 12. Oppex UI • React + Flux • Babel • Webpack • Immutable.js • Stylus • Turns ES6 code into ES5 • Imports • Classes • Arrow functions • Promises
  13. 13. Oppex UI • React + Flux • Babel • Webpack • Immutable.js • Stylus • Isomorphic apps • CommonJS or Bower even on client • Hot reload for client code or stylesheets
  14. 14. Oppex UI • React + Flux • Babel • Webpack • Immutable.js • Stylus • Never copy objects/ collections ever again • Deep trees with easy access to nodes • Lazy Seq for chaining collection methods without intermediate stage
  15. 15. Oppex UI • React + Flux • Babel • Webpack • Immutable.js • Stylus • The best css preprocessor • More logic processing than Less • Pure Node, no Ruby required unlike Sass
  16. 16. Thank you! • Mikko Lehmuskoski • mikko@oppex.com • +358 40 750 8307

×