PyCon 2013 : Scripting to PyPi to GitHub and More

3,921 views
3,827 views

Published on

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,921
On SlideShare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
45
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

PyCon 2013 : Scripting to PyPi to GitHub and More

  1. 1. Script to PyPi to Github @__mharrison__ http://hairysun.com
  2. 2. About Me● 12 years Python● Worked in HA, Search, Open Source, BI and Storage● Author of multiple Python Books
  3. 3. ContinuumsMore like perl-TMTOWTDI
  4. 4. Agenda● Project Development – Versioning – Configuration – Logging – File input – Shell invocation● Environment Layout * virtualenv * pip● Project layout
  5. 5. Agenda (2)● Documentation● Automation/Makefile● Packaging – setup.py – PyPi● Testing● Github● Travis CI● Poachplate
  6. 6. Begin
  7. 7. Warning● Starting from basic Python knowledge● Hands on – (short) lecture – (short) code – repeat until time is gone
  8. 8. ProjectCreate pycat. Pythonic implementation ofcat
  9. 9. Scripting
  10. 10. hello world cat.pyimport sysfor line in sys.stdin: print line,
  11. 11. hello world - Python 3import sysfor line in sys.stdin: print (line, end=)
  12. 12. 2 or 3?
  13. 13. 2 or 3?● Better legacy/library support for 2● Possible to support both
  14. 14. hello worldimport sysfor line in sys.stdin: sys.stdout.write(line)
  15. 15. Assignmentcreate cat.py
  16. 16. Single File or Project?
  17. 17. LayoutCan depend on distribution mechanism:● Single file● Zip file – Python entry point – Splat/run● System package● PyPi/Distribute/pip package
  18. 18. Single File● chmod and place in $PATH● add #!/usr/bin/env python
  19. 19. Single File (2)● No reuse
  20. 20. Zip File● PYTHONPATH=cat.zip python -m __main__● tar -zxvf cat.zip; cd cat; python cat.py
  21. 21. Zip File (2)● No reuse● No stacktrace
  22. 22. System Package● emerge -av qtile (rpm|apt|brew)
  23. 23. System Package (2)● Requires root● At mercy of packager (maybe worse than PyPi)● Reuse● Limited to single version● python -m modulename
  24. 24. Pip Package“Best practice” is combo of: ● Distribute● Virtualenv● Pip
  25. 25. Pip Package (2)$ virtualenv catenv$ source catenv/bin/activate$ pip install pycat
  26. 26. Pip Package (3)● Multiple versions● Reuse● Effort to create setup.py
  27. 27. Minimal Example LayoutProject/ README.txt project/ __init__.py other.py ... setup.py
  28. 28. Better Example LayoutProject/ .gitignore doc/ Makefile index.rst README.txt Makefile bin/ runfoo.py project/ __init__.py other.py ... setup.py
  29. 29. Assignmentcreate layout for cat.py
  30. 30. Semantic Versioning
  31. 31. Versioninghttp://semver.orgFormal spec for versioning projects
  32. 32. Python VersioningPEP 386N.N[.n]+[{a|b|c}N[.N]+][.postN][.devN]1.0a1 < 1.0a2 < 1.0b2.post345
  33. 33. setup.pyfrom distutils.core import setupsetup(name=PyCat,.... version=1.0)
  34. 34. Where to store version?In module.__version__ (might causeimporting issues)
  35. 35. VersionIf using Sphinx for docs, be sure to update:docs/ conf.py
  36. 36. argparseap = argparse.ArgumentParser(version=1.0)...ap.print_version()
  37. 37. AssignmentAdd version to project
  38. 38. Configuration
  39. 39. ● Python Configuration – Django (settings.py) – Python (site.py, setup.py)● JSON/YAML – Google App Engine● Environment Variables – Python (PYTHONPATH)● .ini (ConfigParser, ConfigObj) – matplotlib – ipython● Command line options (argparse)● Sqlite blob● Shelve/pickle blob
  40. 40. Configuration (2)Are not mutually exclusive
  41. 41. Configuration (3)(Unix) Hierarchy:● System rc (run control) (/etc/conf.d)● User rc (~/.config/app/...)● Environment variables● Command line optionshttp://www.faqs.org/docs/artu/ch10s02.htmlFilesystem Hierarchy Standard:http://www.pathname.com/fhs/
  42. 42. Configuration (4)● Plain text config is easily approachable● Careful with Python config on process run by root
  43. 43. AssignmentAdd configuration -n to show line numbers
  44. 44. Logging
  45. 45. Logginglogging module provides feature-richlogging
  46. 46. Logging (2)import logginglogging.basicConfig(level=logging.ERROR, filename=.log)...logging.error(Error encountered in...)
  47. 47. Assignment Add configuration--verbose to log file being “catted”
  48. 48. Dealing with File Input
  49. 49. “Files”Dealing with?● Filename● file object● string data
  50. 50. FilenameSomewhat analogous to an Iterable.● Can open/iterate many times● Implementation depends on file● Need to manage closing file
  51. 51. File ObjectSomewhat analogous to an Iterator.● Can iterate once (unless seeked)● Can accept file, StringIO, socket, generator, etc● Memory friendly - scalable
  52. 52. String DataNo iterator analogy.● Memory hog - less scalable
  53. 53. Stdlib ExamplesModule String Data File Filenamejson loads loadpickle loads loadxml.etree.Element fromstring parse parseTreexml.dom.minidom parseString parse parseConfigParser cp.readfp cp.read(filenames )csv reader DictReaderpyyaml 3rd party load, safe_load load, safe_load
  54. 54. Stdlib Take-aways● mostly functions● file interface is required, others optional● parse or load
  55. 55. Example>>> import sys>>> def parse(fin):... for line in upper(fin):... sys.stdout.write(line)>>> def upper(iterable):... for item in iterable:... yield str(item).upper()
  56. 56. Create file to parse>>> with open(/tmp/data, w)as fout:... fout.write(line1n )... fout.write(line2n )
  57. 57. Filename to file>>> filename = /tmp/data>>> with open(filename) as fin:... parse(fin)LINE1LINE2
  58. 58. String data to file>>> data = "stringn datan ">>> import StringIO>>>parse(StringIO.StringIO(data))STRINGDATA
  59. 59. Parse Iterable>>> data = [foon , barn ]>>> parse(data)FOOBAR
  60. 60. More file benefits● Combine with generators to filter, tweak● Easier to test
  61. 61. AssignmentAdd a parse function
  62. 62. Invoking Shell Commands
  63. 63. Reading output>>> import subprocess>>> p = subprocess.Popen(id -u, shell=True,stdout=subprocess.PIPE, stderr=subprocess.PIPE)>>> p.stdout.read()1000n>>> p.returncode # None means not done>>> print p.wait()0
  64. 64. Feeding stdinCan use communicate orp2.stdin.write w/ flush/close.>>> p2 = subprocess.Popen(wc -l, shell=True,stdout=subprocess.PIPE, stdin=subprocess.PIPE,stderr=subprocess.PIPE)>>> out, err = p2.communicate(foon barn )#p.stdin.flush()>>> out2n>>> p2.returncode0
  65. 65. Chaining scripts.Chaining is pretty straightforward make sure to close stdinhttp://stackoverflow.com/questions/1595492/blocks-send-input-to-python-subprocess-pipeline>>> p3 = subprocess.Popen(sort, shell=True,... stdout=subprocess.PIPE,... stdin=subprocess.PIPE)>>> p4 = subprocess.Popen(uniq, shell=True,... stdout=subprocess.PIPE,... stdin=p3.stdout,... close_fds=True) # hangs w/o close_fds>>> p3.stdin.write(1n 2n 1n )>>> p3.stdin.flush(); p3.stdin.close();>>> p4.stdout.read()1n2n
  66. 66. Chaining scripts and pythoncat 0-2, add 10 to them (in python) and wc-l results.>>> import os>>> p5 = subprocess.Popen(cat, shell=True,stdout=subprocess.PIPE, stdin=subprocess.PIPE, close_fds=True)>>> def p6(input):... add 10 to line in input ... for line in input:... yield %d%s %(int(line.strip())+10, os.linesep)
  67. 67. Chaining scripts and python (2)>>> p7 = subprocess.Popen(wc -l, shell=True,stdout=subprocess.PIPE, stdin=subprocess.PIPE, close_fds=True)>>> [p5.stdin.write(str(x)+os.linesep) for x in xrange(3)]>>> p5.stdin.close()>>> [p7.stdin.write(x) for x in p6(p5.stdout.xreadlines())]>>> p7.stdin.close()>>> p7.stdout.read()3n
  68. 68. Environment
  69. 69. Python Environment● System Python or “Virtual” Python● Installation method
  70. 70. Which PythonSystem Python● Requires root● Only one version of library● System packages may be out of date
  71. 71. Which Python (2)“Virtual” Python ● Runs as user● Specify version● Sandboxed from system● Create multiple sandboxes
  72. 72. Which Python (3)Install virtualenv
  73. 73. virtualenvInstallation:● System package $ sudo apt-get install python-virtualenv● With pip $ pip install virtualenv● $ easy_install virtualenv● $ wget https://raw.github.com/pypa/virtualenv/master/ virtualenv.py; python virtualenv.py
  74. 74. virtualenv (2)Create virtual environment:$ virtualenv env_name
  75. 75. Directory Structureenv_name/ bin/ activate python pip lib/ python2.7/ site-packages/
  76. 76. virtualenv (3)Activate virtual environment:$ source env_name/bin/activate
  77. 77. virtualenv (4)Windows activate virtual environment:> pathtoenvScriptsactivateMay require PS C:>Set-ExecutionPolicy AllSigned
  78. 78. virtualenv (5)Comes with pip to install packages:$ pip install sqlalchemy
  79. 79. virtualenv (6)$ which python # not /usr/bin/python/home/matt/work/courses/script-pypi-github/env/bin/python
  80. 80. virtualenv (7)>>> sys.path[, ...,/home/matt/work/courses/script-pypi-github/env/lib/python2.7/site-packages]
  81. 81. virtualenv (8)Use deactivate shell function to resetPATH:$ deactivate
  82. 82. Assignmentcreate virtualenv env
  83. 83. pip
  84. 84. pipRecursive Acronym - Pip Installs Packages● install● upgrade● uninstall● “pin” versions● requirements.txt
  85. 85. pip (2)Install:$ pip install sqlalchemy
  86. 86. pip (3)Upgrade:$ pip install --upgrade sqlalchemy
  87. 87. pip (4)Uninstall:$ pip uninstall sqlalchemy
  88. 88. pip (5)“Pin” version:$ pip install sqlalchemy==0.7
  89. 89. pip (6)Requirements file:$ pip install -r requirements.txt
  90. 90. pip (7)Requirements file $ cat requirements.txt:sqlalchemy==0.7foobarbazlib>=1.0
  91. 91. pip (8)Create requirement file from env:$ pip freeze > req.txt
  92. 92. pip (9)● pip docs say to install from virtualenv● virtualenv docs say to install from pip
  93. 93. pip (10)Install from directory:pip install --download packages -rrequirements.txtpip install --no-index--find-links=file://full/path/to/packages -r requirements.txt
  94. 94. distribute, setuptools, distutils● pip wraps distribute adds uninstall, req.txt● distribute fork of setuptools● setuptools unmaintained, adds eggs, dependencies, easy_install● distutils stdlib packaging library
  95. 95. Assignment use pip to installpackage from internet
  96. 96. More Layout
  97. 97. Better Example LayoutProject/ .gitignore doc/ Makefile index.rst README.txt Makefile bin/ runfoo.py project/ __init__.py other.py ... setup.py
  98. 98. .gitignore*.py[cod]# C extensions*.so# Packages*.egg*.egg-infodistbuildeggspartsbinvarsdist
  99. 99. develop-eggs.installed.cfg .gitignoreliblib64__pycache__# Installer logspip-log.txt# Unit test / coverage reports.coverage.toxnosetests.xml# Translations*.mohttps://github.com/github/gitignore
  100. 100. .gitignore (2)See blaze-core for example of C andDocs.https://github.com/ContinuumIO/blaze-core
  101. 101. From requests:.coverage .gitignore (3)MANIFESTcoverage.xmlnosetests.xmljunit-report.xmlpylint.txttoy.pyviolations.pyflakes.txtcover/docs/_buildrequests.egg-info/*.pyc*.swpenv/.workont.pyt2.pyhttps://github.com/kennethreitz/requests
  102. 102. ● Other(for django) localsettings.py tips● *~ (emacs)● Run git status and add outliers to .gitignore● Make settings global: git config --global core.excludesfile Python.gitignore git config --global core.excludesfile Python.gitignore
  103. 103. Assignmentadd (premptive) .gitignore
  104. 104. Documentation
  105. 105. DocumentationTwo types:● Developer docs (README, INSTALL, HACKING, etc)● End user
  106. 106. DeveloperREADME - main entry point for project● Brief intro● Links/Contact● License
  107. 107. READMEFor github integration name it README.rstor README.md
  108. 108. LICENSEInclude text of license. Templates athttp://opensource.org/licenses/index.html
  109. 109. LicensingSome include dunder meta in project docstring (requests__init__.py)::copyright: (c) 2013 by Kenneth Reitz.:license: Apache 2.0, see LICENSE for moredetails.(note IANAL)
  110. 110. Licensing (2)Some include dunder meta in project (requests__init__.py):__title__ = requests__version__ = 1.1.0__build__ = 0x010100__author__ = Kenneth Reitz__license__ = Apache 2.0__copyright__ = Copyright 2013 Kenneth Reitz(note IANAL)
  111. 111. Other files● AUTHORS● HISTORY/CHANGELOG● TODO
  112. 112. Assignmentcreate simple README
  113. 113. End User DocsSphinx is a tool that makes it easy to createintelligent and beautiful documentation,written by Georg Brandl and licensedunder the BSD license.http://sphinx-doc.org
  114. 114. Suggested setupProject/ doc/ # sphinx stuff Makefile
  115. 115. Sphinx in 4 Lines$ cd docs$ sphinx-quickstart$ $EDITOR index.rst$ make html
  116. 116. Assignmentwrite docs at a later time :)
  117. 117. Makefile
  118. 118. Running commands often: ● nosetests (plus options) Motivation ● create sdist ● upload to PyPi ● create virtualenv ● install dependencies ● cleanup cruft ● create TAGS ● profile ● sdist ● PyPi - register and upload ● creating pngs from svgs ● docs ● Python 3 testing ● etc...
  119. 119. Makefile● Knows about executing (build) commands● Knows about dependencies
  120. 120. ExampleTo test:● Make virtualenv● install code dependencies● install nose (+coverage)● run tests
  121. 121. Clean checkout$ make testvs$ virtualenv env$ env/bin/activate$ pip install -r deps.txt$ pip install nose coverage.py$ nosestests
  122. 122. Enough makeknowledge to be dangerous
  123. 123. Makefile (1)Syntax of Makefile:file: dependentfile<TAB>Command1...<TAB>CommandN
  124. 124. Makefile (2)Running (runs Makefile by default):$ make file# will build dependentfile ifnecessary# then build file
  125. 125. Makefile (3)Example:foo: foo.c foo.h<TAB>cc -c foo.c<TAB>cc -o foo foo.o
  126. 126. Makefile (4)Running (echoes commands by default -sfor silent):$ makecc -c foo.ccc -o foo foo.o
  127. 127. Makefile (5)Subsequent runs do nothing:$ makemake: `foo is up to date.
  128. 128. Makefile (6)Add a clean command:.PHONY: cleanclean:<TAB>rm foo.o foo
  129. 129. Makefile (7)Since clean isnt a file, need to use .PHONYto indicate that to make. (If you had a filenamed clean it wouldnt try to build it).
  130. 130. Makefile (8)(Simply Expanded) Variables (expanded when set):BIN := env/binPY := $(BIN)/pythonNOSE := $(BIN)/nosetests.PHONY: buildbuild: env<TAB>$(PY) setup.py sdist
  131. 131. Makefile (9)(Recursively Expanded) Variables (expanded when used):FILE = fooDATA = $(FILE)# If DATA expanded would be fooFILE = bar# If DATA expanded would be bar
  132. 132. Makefile (10)Shell functions:.PHONY: pwdpwd:<TAB>pushd /etc
  133. 133. Makefile (11)Invoking:$ make pwdpushd /etcmake: pushd: Command not foundmake: *** [pwd] Error 127(pushd is a bash function)
  134. 134. Makefile (12)Shell functions:SHELL := /bin/bash.PHONY: pwdpwd:<TAB>pushd /etc
  135. 135. Makefile (13)Multiple commands:SHELL := /bin/bash.PHONY: pwdpwd:<TAB>pushd /etc<TAB>pwd<TAB>popd
  136. 136. Makefile (14)Multiple commands:$ make pwdpushd /etc/etc /tmp/foopwd/tmp/foopopd/bin/bash: line 0: popd: directory stack empty
  137. 137. Makefile (15)Each tab indented command runs in itsown process. Use ; and put in one line oruse for line continuation
  138. 138. Makefile (16)Multiple commands (use line continuation ):SHELL := /bin/bash.PHONY: pwd2pwd2:<TAB>pushd /etc; <TAB>pwd; <TAB>popd
  139. 139. Makefile (17)Shell variables:.PHONY: pathpath:<TAB>echo $PATH
  140. 140. Makefile (18)Make thinks they are make variables:$ make pathecho ATHATH
  141. 141. Makefile (19)$ needs to be escaped with $:.PHONY: pathpath:<TAB>echo $$PATH
  142. 142. Makefile (18)Now it works:$ make pathecho $PATH/tmp/maketest
  143. 143. Makefiles for Python Projects
  144. 144. Inspired by Rick Hardings Talkhttp://pyvideo.org/video/1354/starting-your-project-right-setup-and-automationhttps://github.com/mitechie/pyohio_2012
  145. 145. Makefile for PythonMake virtualenv:env:<TAB>virtualenv env
  146. 146. Makefile for Python (2)Make dependencies:.PHONY: depsdeps: env<TAB>$(PIP) install -rrequirements.txt
  147. 147. Makefile for Python (3)Testing with nose:.PHONY: testtest: nose deps<TAB>$(NOSE)# nose depends on the nosetests binarynose: $(NOSE)$(NOSE): env<TAB>$(PIP) install nose
  148. 148. Contrary Opinions“Dynamic languages dont need anythinglike make, unless they have somecompile-time interface dependenciesbetween modules”http://stackoverflow.com/questions/7580939/why-are-there-no-makefiles-for-automation-in-python-projects
  149. 149. Other options● paver● fabric● buildout
  150. 150. Packaging
  151. 151. setup.py overloaded● create sdist (source distribution)● upload to pypi● install package
  152. 152. setup.py wartrequire keyword of distutils doesntdownload reqs only documents them. Userequirements.txt in combo with pip.
  153. 153. setup.py exampleFrom requests:setup( name=requests, version=requests.__version__, description=Python HTTP for Humans., long_description=open(README.rst).read() +nn + open(HISTORY.rst).read(), author=Kenneth Reitz, author_email=me@kennethreitz.com, url=http://python-requests.org,
  154. 154. setup.py example (2)packages=packages,package_data={: [LICENSE, NOTICE],requests: [*.pem]},package_dir={requests: requests},include_package_data=True,install_requires=requires,license=open(LICENSE).read(),zip_safe=False,
  155. 155. setup.py example (3) classifiers=( Development Status :: 5 - Production/Stable, Intended Audience :: Developers, Natural Language :: English, License :: OSI Approved :: Apache Software License, Programming Language :: Python, Programming Language :: Python :: 2.6, Programming Language :: Python :: 2.7, Programming Language :: Python :: 3, # Programming Language :: Python :: 3.0, Programming Language :: Python :: 3.1, Programming Language :: Python :: 3.2, Programming Language :: Python :: 3.3, ),)
  156. 156. setup.py modulesIf project consists of a few modules thismay be easiest
  157. 157. setup.py packagesNeed to explicitly list all packages, not justroot
  158. 158. setup.py scriptsAdd executable Python files here
  159. 159. setup.py non-Python files● add files to MANIFEST.in (include in package)● add files to package_data in setup.py (include in install) Not recursive
  160. 160. MANIFEST.in language● include|exclude pat1 pat2 ...● recursive-(include|exclude) dir pat1 pat2 ...● global-(include|exclude) dir pat1 pat2 ...● prune dir● graft dirhttp://docs.python.org/release/1.6/dist/sdist-cmd.html#sdist-cmd
  161. 161. setup.py classifiersAlmost 600 different classifiers.Not used by pip to enforce versions. For UIonly
  162. 162. Create sdist$ python setup.py sdist
  163. 163. PyPiValidate setup.py:$ python setup.py check
  164. 164. PyPi RegisterClick on “Register” on right hand boxhttps://pypi.python.org/pypi?%3Aaction=register_form
  165. 165. PyPi Upload$ python setup.py sdist registerupload
  166. 166. PyPi Upload (2)$ python setup.py sdist register upload...Creating tar archiveremoving rst2odp-0.2.4 (and everything under it)running registerrunning checkWe need to know who you are, so please choose either: 1. use your existing login, 2. register as a new user, 3. have the server generate a new password for you (and email it to you),or 4. quitYour selection [default 1]:1
  167. 167. PyPi Upload (3)Username: mharrisonPassword:Registering rst2odp to http://pypi.python.org/pypiServer response (200): OKI can store your PyPI login so future submissions will be faster.(the login will be stored in /home/matt/.pypirc)Save your login (y/N)?yrunning uploadSubmitting dist/rst2odp-0.2.4.tar.gz to http://pypi.python.org/pypiServer response (200): OK
  168. 168. PyPi NoteThough PyPi packages are signed there isno verification step during packageinstallation
  169. 169. Non PyPi URL$ pip install --no-index -fhttp://dist.plone.org/thirdparty/-U PIL
  170. 170. Personal PyPihttps://github.com/benliles/djangopypi
  171. 171. PyPiMakefile integration:# --------- PyPi ----------.PHONY: buildbuild: env<TAB>$(PY) setup.py sdist.PHONY: uploadupload: env<TAB>$(PY) setup.py sdist register upload
  172. 172. Testing
  173. 173. TestingAdd tests to your project
  174. 174. Testing (2)● use doctest or unittest
  175. 175. Testing (3)Use nose to run:$ env/bin/nosetests..-------------Ran 2 tests in 0.007sOK
  176. 176. Testing (4)Makefile integration:NOSE := env/bin/nosetests# --------- Testing ----------.PHONY: testtest: nose deps<TAB>$(NOSE)# nose depends on the nosetests binarynose: $(NOSE)$(NOSE): env<TAB>$(PIP) install nose
  177. 177. Github
  178. 178. GithubJust a satisfied user
  179. 179. Why Github?Not bazaar nor mercurial
  180. 180. Why Github? (2)Code is a first class object
  181. 181. GithubDont check in keys/passwords!http://ejohn.org/blog/keeping-passwords-in-source-control/#postcomment
  182. 182. A Branching Strategyhttp://github.com/nvie/gitflow
  183. 183. Travis CI
  184. 184. Travis CICI (continuous integration) for Github
  185. 185. Travis CI (2)Illustrates power of (web) hooks
  186. 186. 5 Steps● Sign-in with github● Sync repos on Profile page● Enable repo● Create a .travis.yml file on github● Push a commit on github
  187. 187. travis.ymllanguage: pythonpython: - "3.3" - "2.7" - "2.6" - "pypy"# command to run testsscript: make test
  188. 188. More Infohttp://about.travis-ci.org/
  189. 189. Poachplate
  190. 190. poachplateExample of most of what we have talkedabout todayhttps://github.com/mattharrison/poachplate
  191. 191. Thats allQuestions? Tweet or email me matthewharrison@gmail.com @__mharrison__ http://hairysun.com

×