This document is a series of notes about Python web scraping and networking tools written on September 25, 2009. It introduces the twill browser testing tool, Python network libraries like urllib and httplib, how to make GET and POST requests in Python, handling cookies with cookielib, and the basics of HTTP including GET, POST, HEAD, and PUT/DELETE requests. It concludes with some ideas for projects including web games and scraping data from sites into SQL databases.

1. 5
qingfeng@me.com
2009-09-26
2009 9 25

2. Python & Network
✤
✤ ,
✤
2009 9 25

3. twill
✤ http://twill.idyll.org/
✤ python setup.py install
✤ twill-sh
2009 9 25

4. twill
go http://www.renren.com/SysHome.do
showforms
fv 1 email qingfeng@me.com
fv 1 password xxx
submit
show
2009 9 25

5. twill
go http://upload.renren.com/addphotoPlain.do?
id=327316943
showforms
formfile 3 photo1 /Users/fengqing/Downloads/p1.jpg
submit
2009 9 25

6. Python Network Library
✤ urllib
✤ httplib
✤ urllib2
✤ cookiejar
2009 9 25

7. urllib.urlopen(“http://www.sina.com.cn”).read()
2009 9 25

8. f = open(“c:/sina.html”,”w”)
html = urllib.urlopen(“http://www.sina.com.cn”).read()
f.write(html)
f.close()
2009 9 25

9. cookiejar = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
params = {'email':'qingfeng@me.com','password':'xxx','origURL':"http://
www.renren.com/SysHome.do"}
data = urllib.urlencode(params)
fobj = opener.open('http://passport.renren.com/PLogin.do', data)
fobj = opener.open('http://renren.com/profile.do?id=260622588')
print fobj.read()
2009 9 25

10. HTTP
✤ GET
✤ POST
✤ HEAD
✤ PUT/DELETE
2009 9 25

11. GET
✤ telnet www.renren.com 80
✤ GET / HTTP/1.0
2009 9 25

12. Cookie
✤
✤ Cookie
✤ Cookie &
2009 9 25

13. Try
✤ ,
✤ SQL
✤ WebGame
2009 9 25

14. Free Time
✤ , ,
✤
✤
✤
✤ /
✤
2009 9 25