How we use Bottle and Elasticsearch


Published on

An Adventure in Sinar Project

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

How we use Bottle and Elasticsearch

  1. 1. How We Use Bottle and Elasticsearch An Adventure in Sinar Project
  2. 2. A background of Malaysian BillWatcher● The big idea, is to let citizen knows what is happening in parliament, bills being debated and pass● No it doesnt know the bills that is not debated yet.● Its is scraper based, just because the parliament site tend to be stand alone, and nobody bother to go there.● And cheat on twitter notification
  3. 3. The result● Didnt quite worked out.● That is not the point of this talk● You can talk to us later on this
  4. 4. What we use?● We use bottle micro framework.● Elasticsearch via pyes● sqlalchemy for db abstraction(seriously thinking to move to mongodb instead, idea?)● We use bootstrap for css and their js plugin● with jquery● Beaufifulsoup for scraping(that is our data source)We only going to cover pyes and bottle(talk to me later for everything else)
  5. 5. Bottle● A micro framework● Similar to flask!!!● Less feature though● But it is OKAY.
  6. 6. Bottle a views● Watcher/blob/master/billwatcher/● Watcher/blob/master/billwatcher/
  7. 7. What happen?● Bottle can return a dict, and it automatically output json, if the object in dict is compatible● Or with View Decorator, the dict can be feed to a template
  8. 8. bottle the template● Watcher/blob/master/billwatcher/views/list.tpl●
  9. 9. What happen?● Bottle read template from a dict/object● like most templating language does.● Have some logic like other, different syntax same idea● Dont have good inheritance though
  10. 10. elasticsearch!!!● OK I lied, this is written in JAVA● But hey, it got RESTful API● and pyes did a lot of the stuff for us already
  11. 11. Before index● before we do that● we dont need to create a schema● But we do it because we can control the search priority
  12. 12. Indexer● Watcher/blob/master/billwatcher/
  13. 13. then we index● First flatten your result● convert to dict● index
  14. 14. The indexer● Watcher/blob/master/billwatcher/● Watcher/blob/master/billwatcher/
  15. 15. now we search!!!!● There is some field inside the search result● Watcher/blob/master/billwatcher/
  16. 16. The future● Do localization● Find new data source● Move template to jinja● Seriously thinking of moving to mongodb● Seriously still trying to see a better way to scrape websites