From Hk0weather to Open Data

1,175 views

Published on

First edition presenting at BarCampHK 2013.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,175
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
9
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

From Hk0weather to Open Data

  1. 1. 從 hk0weather 到 Open Data, Open Government From hk0weather to Open Data, Open Government Sammy Fung BarCampHK 2013 http://slidesha.re/11ZPKvx
  2. 2. Sammy Fung● Blogger: sammy.hk● Coder: perl -> php -> python● Linux User: server + desktop● “Cloud”: OpenStack.● Smartphone: iPhone 3G -> Android● Organizer: Linux/Open Source Communities
  3. 3. Agenda● My Products: WeatherHK, TCTrack● My new opensource projects: hk0weather● My comments to Open Data and Open Government● Q&A, Open Discussion to Floor.
  4. 4. My Products● WeatherHK● TCTrack
  5. 5. WeatherHK● http://twitter.com/weatherhk● Interview by MetroPop in 2009.
  6. 6. WeatherHK● http://twitter.com/weatherhk● Backend: Python + Scrapy + Database + Twitter + NNTP......● Frontend: Twitter + Newsgroup● hourly current weather report● weather forecast report● tropical signal warning
  7. 7. My Products● WeatherHK● TCTrack
  8. 8. TCTrack● http://sammy.hk/projects/tctrack/tctrack.php● http://twitter.com/tctrack● Plot TC current and forecast tracks over Google Map.● Tweet JTWC updates for Northwest Pacific.● Source: JTWC, HKO.
  9. 9. TCTrack● http://sammy.hk/projects/tctrack/tctrack.php● http://twitter.com/tctrack● Probably first tctrack map in HK using GoogleMap● Use of GMap: TCTrack -> Weather Underground Hong Kong -> HKO
  10. 10. g0v.tw● g0v.tw 台灣零時政府● clkao 高嘉良 suggested to implement open source weather tools for Hong Kong.
  11. 11. hk0weather● https://github.com/sammyfung/hk0weather● Open Source Hong Kong Weather Project.● convert to JSON data from HKO webpages.● python + scrapy● 1st version: from current weather report, extracting temperture and humidity from 20+ weather stations, export in json format.
  12. 12. hk0weather● https://github.com/sammyfung/hk0weather● $ virtualenv hk0weatherenv● $ source hk0weatherenv/bin/activate● $ pip install scrapy● $ git clone https://github.com/sammyfung/hk0weather.git● $ cd hk0weather● $ scrapy crawl currwx -t json -o testresult
  13. 13. hk0weather[{"humidity": 80, "station": "hko", "temperture": 17, "time": 1360785720},{"station": "kingspark", "temperture": 16, "time": 1360785720},{"station": "wongchukhang", "temperture": 17, "time": 1360785720},{"station": "takwuling", "temperture": 16, "time": 1360785720},{"station": "laufaushan", "temperture": 15, "time": 1360785720},{"station": "taipo", "temperture": 16, "time": 1360785720},{"station": "shatin", "temperture": 17, "time": 1360785720},{"station": "tuenmun", "temperture": 17, "time": 1360785720},{"station": "tseungkwano", "temperture": 16, "time": 1360785720},{"station": "saikung", "temperture": 16, "time": 1360785720},{"station": "cheungchau", "temperture": 17, "time": 1360785720},{"station": "cheungchau", "temperture": 17, "time": 1360785720},{"station": "tsingyi", "temperture": 17, "time": 1360785720},{"station": "shekkong", "temperture": 15, "time": 1360785720},{"station": "tsuenwanhokoon", "temperture": 15, "time": 1360785720},{"station": "tsuenwanshingmunvalley", "temperture": 17, "time": 1360785720},{"station": "hongkongpark", "temperture": 17, "time": 1360785720},{"station": "shaukeiwan", "temperture": 16, "time": 1360785720},{"station": "kowlooncity", "temperture": 16, "time": 1360785720},{"station": "happyvalley", "temperture": 18, "time": 1360785720},{"station": "wongtaisin", "temperture": 17, "time": 1360785720},{"station": "stanley", "temperture": 16, "time": 1360785720},{"station": "kwuntong", "temperture": 15, "time": 1360785720},{"station": "shamshuipo", "temperture": 17, "time": 1360785720}]
  14. 14. hk0weather● Future Planning:● Add more weather reports.● Getting ideas and/or cooperate with pro Weather hobbists.● Remarks:● Development of hk0weather is started from ZERO, its code is different than my twitter @weatherhk.
  15. 15. Why python + scrapy ?● python: my current favourite programming language for few years.● scrapy: web crawling framework written in Python.
  16. 16. Scrapy● scrapy: web crawling framework written in Python.● HtmlXPathSelector● Output: built-in JSON, CSV, XML.● Python: import re
  17. 17. Items.pyclass Hk0WeatherItem(Item): time = Field() station = Field() temperture = Field() humidity = Field() pass
  18. 18. Items.pyclass Hk0WeatherItem(Item): time = Field() station = Field() temperture = Field() humidity = Field() pass
  19. 19. Currwx.py start_urls = (http://www.weather.gov.hk/wxinfo/currwx/currentc.htm, )
  20. 20. Currwx.pydef parse(self, response): laststation = temperture = int() stations = [] hxs = HtmlXPathSelector(response) report = hxs.select(//div[@id="ming"])
  21. 21. libhk0class hk0: stations = [ (u 天 文 台 , hko), (u 京 士 柏 , kingspark), (u 黃 竹 坑 , wongchukhang), (u 打 鼓 嶺 , takwuling), (u 流 浮 山 , laufaushan),
  22. 22. libhk0class hk0: def gettime(self, report): … def hk0current(self, report): …
  23. 23. Challenge● Challenge on first day of hk0weather release.● Facebook comment to blog post: 「其實 HKO 有個年費 plan 係嘔兩個好靚仔嘅 XML 。而只 要你肯問,你件嘢又非商業用途, HKO 都會免 費俾你用」
  24. 24. Challenge● I replied after googling for it.● google 找到相關 XML schema 示範資料,但 ....● 1. 在於 Open Data 角度,暫時看不到有網頁說明任何人以非商 業用途使用資訊,可以免費獲得登入戶口。而我本身是「已登 記」使用 HKO 網頁資料作非商業用途的使用 / 轉載者,我一直收 到來自 HKO 的更新電郵都是說明 HTML 格式的更新, " 印象中 " HKO 並沒有發出過 XML 和邀請開 XML 使用戶口。● 2. 而 ( 以我覺得 ) 以業餘氣象發燒友的角度來看,目前 HKO XML schema 仍有不足之處,我已想到一些我想要的數據,仍是只能 從 HTML 版本取得。● Challenge: FAIL! XD
  25. 25. Open Data from Govenment● My Chinese Blog Post 「香港政府機構開放資料 Open Data 情況」 on 2013/1/17.● Data.One released on 2011/3/31.● Weather at Data.One provides 7 dataset URLs, returns RSS (XML) format (Eng/TChi/SChi)● One word: Useless.● Data.One dataset is different with HKO own XML metioned in previous “challenge”.
  26. 26. Open Data from Govenment● Example - Current local weather report:● Plain text report in RSS.● Difference to quote report content: – Website: HTML tags – Data.One: RSS description tags● Other weather data is missing, eg. Area temperture updates per each 12 mins.
  27. 27. Open Data from Govenment● Weather at Data.One is report but not data.● Weather RSS is already released by HKO before launch of Data.One.● Technically, json/xml format is better readable by computer programs.
  28. 28. Open Data from Govenment
  29. 29. Improving Legco Vote Result Data ?● Legcovotes.net is created by Hong Kong netitizens(?).● Only 20 famous vote results are included.● It is possible to let public to input other vote results by hand, and submissions should be verified by legcovotes.net authoritative.● Including other data, eg. Minutes in plain text or paragraphs related to a counciler.
  30. 30. Thank you.● This presentation will be uploaded to slideshare.● Licensed in Creative Commons BY-NC-SA.● http://slidesha.re/11ZPKvx● http://sammy.hk

×