How we use Bottle and Elasticsearch

6,431 views

Published on

An Adventure in Sinar Project

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
6,431
On SlideShare
0
From Embeds
0
Number of Embeds
2,533
Actions
Shares
0
Downloads
21
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

How we use Bottle and Elasticsearch

  1. 1. How We Use Bottle and Elasticsearch An Adventure in Sinar Project
  2. 2. A background of Malaysian BillWatcher● The big idea, is to let citizen knows what is happening in parliament, bills being debated and pass● No it doesnt know the bills that is not debated yet.● Its is scraper based, just because the parliament site tend to be stand alone, and nobody bother to go there.● And cheat on twitter notification
  3. 3. The result● Didnt quite worked out.● That is not the point of this talk● You can talk to us later on this
  4. 4. What we use?● We use bottle micro framework.● Elasticsearch via pyes● sqlalchemy for db abstraction(seriously thinking to move to mongodb instead, idea?)● We use bootstrap for css and their js plugin● with jquery● Beaufifulsoup for scraping(that is our data source)We only going to cover pyes and bottle(talk to me later for everything else)
  5. 5. Bottle● A micro framework● Similar to flask!!!● Less feature though● But it is OKAY.
  6. 6. Bottle a views● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/pages.py● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/pages.py
  7. 7. What happen?● Bottle can return a dict, and it automatically output json, if the object in dict is compatible● Or with View Decorator, the dict can be feed to a template
  8. 8. bottle the template● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/views/list.tpl●
  9. 9. What happen?● Bottle read template from a dict/object● like most templating language does.● Have some logic like other, different syntax same idea● Dont have good inheritance though
  10. 10. elasticsearch!!!● OK I lied, this is written in JAVA● But hey, it got RESTful API● and pyes did a lot of the stuff for us already
  11. 11. Before index● before we do that● we dont need to create a schema● But we do it because we can control the search priority
  12. 12. Indexer● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/indexer.py
  13. 13. then we index● First flatten your result● convert to dict● index
  14. 14. The indexer● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/indexer.py● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/loader.py
  15. 15. now we search!!!!● There is some field inside the search result● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/pages.py
  16. 16. The future● Do localization● Find new data source● Move template to jinja● Seriously thinking of moving to mongodb● Seriously still trying to see a better way to scrape websites

×