Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How we use Bottle and Elasticsearch


Published on

An Adventure in Sinar Project

Published in: Technology
  • Be the first to comment

  • Be the first to like this

How we use Bottle and Elasticsearch

  1. 1. How We Use Bottle and Elasticsearch An Adventure in Sinar Project
  2. 2. A background of Malaysian BillWatcher● The big idea, is to let citizen knows what is happening in parliament, bills being debated and pass● No it doesnt know the bills that is not debated yet.● Its is scraper based, just because the parliament site tend to be stand alone, and nobody bother to go there.● And cheat on twitter notification
  3. 3. The result● Didnt quite worked out.● That is not the point of this talk● You can talk to us later on this
  4. 4. What we use?● We use bottle micro framework.● Elasticsearch via pyes● sqlalchemy for db abstraction(seriously thinking to move to mongodb instead, idea?)● We use bootstrap for css and their js plugin● with jquery● Beaufifulsoup for scraping(that is our data source)We only going to cover pyes and bottle(talk to me later for everything else)
  5. 5. Bottle● A micro framework● Similar to flask!!!● Less feature though● But it is OKAY.
  6. 6. Bottle a views● Watcher/blob/master/billwatcher/● Watcher/blob/master/billwatcher/
  7. 7. What happen?● Bottle can return a dict, and it automatically output json, if the object in dict is compatible● Or with View Decorator, the dict can be feed to a template
  8. 8. bottle the template● Watcher/blob/master/billwatcher/views/list.tpl●
  9. 9. What happen?● Bottle read template from a dict/object● like most templating language does.● Have some logic like other, different syntax same idea● Dont have good inheritance though
  10. 10. elasticsearch!!!● OK I lied, this is written in JAVA● But hey, it got RESTful API● and pyes did a lot of the stuff for us already
  11. 11. Before index● before we do that● we dont need to create a schema● But we do it because we can control the search priority
  12. 12. Indexer● Watcher/blob/master/billwatcher/
  13. 13. then we index● First flatten your result● convert to dict● index
  14. 14. The indexer● Watcher/blob/master/billwatcher/● Watcher/blob/master/billwatcher/
  15. 15. now we search!!!!● There is some field inside the search result● Watcher/blob/master/billwatcher/
  16. 16. The future● Do localization● Find new data source● Move template to jinja● Seriously thinking of moving to mongodb● Seriously still trying to see a better way to scrape websites