• Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
3,674
On Slideshare
0
From Embeds
0
Number of Embeds
7

Actions

Shares
Downloads
10
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. How We Use Bottle and Elasticsearch An Adventure in Sinar Project
  • 2. A background of Malaysian BillWatcher● The big idea, is to let citizen knows what is happening in parliament, bills being debated and pass● No it doesnt know the bills that is not debated yet.● Its is scraper based, just because the parliament site tend to be stand alone, and nobody bother to go there.● And cheat on twitter notification
  • 3. The result● Didnt quite worked out.● That is not the point of this talk● You can talk to us later on this
  • 4. What we use?● We use bottle micro framework.● Elasticsearch via pyes● sqlalchemy for db abstraction(seriously thinking to move to mongodb instead, idea?)● We use bootstrap for css and their js plugin● with jquery● Beaufifulsoup for scraping(that is our data source)We only going to cover pyes and bottle(talk to me later for everything else)
  • 5. Bottle● A micro framework● Similar to flask!!!● Less feature though● But it is OKAY.
  • 6. Bottle a views● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/pages.py● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/pages.py
  • 7. What happen?● Bottle can return a dict, and it automatically output json, if the object in dict is compatible● Or with View Decorator, the dict can be feed to a template
  • 8. bottle the template● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/views/list.tpl●
  • 9. What happen?● Bottle read template from a dict/object● like most templating language does.● Have some logic like other, different syntax same idea● Dont have good inheritance though
  • 10. elasticsearch!!!● OK I lied, this is written in JAVA● But hey, it got RESTful API● and pyes did a lot of the stuff for us already
  • 11. Before index● before we do that● we dont need to create a schema● But we do it because we can control the search priority
  • 12. Indexer● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/indexer.py
  • 13. then we index● First flatten your result● convert to dict● index
  • 14. The indexer● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/indexer.py● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/loader.py
  • 15. now we search!!!!● There is some field inside the search result● https://github.com/Sinar/Malaysian-Bill- Watcher/blob/master/billwatcher/pages.py
  • 16. The future● Do localization● Find new data source● Move template to jinja● Seriously thinking of moving to mongodb● Seriously still trying to see a better way to scrape websites