IPython Open Source Academia WrapupIPythonA modern vision of interactive computingFernando Pérezhttp://fperez.org, @fperez...
IPython Open Source Academia WrapupOutline1 IPython: Interactive Python2 The Life of an Open Source Project3 Academia vs O...
In the beginning, IBM said...Let there be FORTRAN
In the beginning, IBM said...Let there be FORTRAN
Beyond (Floating Point) Number CrunchingHardwarefloating pointArbitrary precisionintegersRationalsInterval arithmeticSymbo...
The purpose of computing is insight, not numbers.Richard Hamming, 1962
IPython Open Source Academia WrapupThe computer as microscopeExploratory: Problem’s definition evolves as we understand it....
IPython: part of a Rich EcosystemIPythonNetworkX
IPython Open Source Academia WrapupThe Lifecycle of a Scientific Idea (schematically)1 Individual exploratory work2 Collabo...
IPython Open Source Academia WrapupThe Lifecycle of a Scientific Idea (schematically)1 Individual exploratory work2 Collabo...
IPython’s goal:Fluid transitions in all these steps
Demo
IPython Open Source Academia WrapupPillar #1: An architecture for interactive computingFP (UC Berkeley) IPython 3/20/13 11...
IPython Open Source Academia WrapupPillar #2: the Notebook FormatJSON but version control-friendlyEasy for machine process...
IPython Open Source Academia WrapupOutline1 IPython: Interactive Python2 The Life of an Open Source Project3 Academia vs O...
Documented protocols and formats:a growing ecosystem around IPython
An Emacs Notebook Client!Takafumi Arakakihttp://tkf.github.com/emacs-ipython-notebook
Microsoft Visual Studio 2010 integrated consoleDino Viehland and Shahrokh Mortazavi (Microsoft)http://pytools.codeplex.com
A vim client to control an IPython kernel/consolePaul Ivanov (Berkeley)https://github.com/ivanov/vim-ipython
Notebooks on Windows Azure CloudShahrokh Mortazavi (Microsoft), B.G., F.P.http://bit.ly/JQeojD
Star Cluster: IPython parallel+Notebook on Amazon EC2Justin Riley (MIT)http://web.mit.edu/star/cluster
NBViewer: easy notebook sharingMatthias Bussonnierhttp://nbviewer.ipython.org
Other projects using IPythonScientificEPD: Enthought Python Distribution.Anaconda: Continuum Python Distribution.Sage: open...
How did we get here?A brief history of IPythonOctober 2001: “just a little afternoon hack”My own $PYTHONSTARTUP:ipython-0....
Summer 2009: NIH-funded cleanup by Brian.March 2010: prototype networked shell using ØMQ2-day sprint with BrianEnthought f...
An important plothttp://www.ohloh.net/p/ipython
(Incomplete) Cast of CharactersBrian Granger - Physics, Cal State San Luis ObispoMin Ragan-Kelley - Nuclear Engineering, U...
IPython Open Source Academia WrapupOutline1 IPython: Interactive Python2 The Life of an Open Source Project3 Academia vs O...
Support at the edges of academic fundingEnthought, Austin, TX: Lots!Microsoft: WinHPC support, Visual Studio integration, ...
Open Source:skills, tools and practices we need!A culture where things get done.Wildly collaborativeReproducible by necess...
Reward Structure in academia:we punish all of the aboveDepartmental boundaries: interdisciplinary work is a great buzzword...
NumFOCUS: Open Code, Better SciencePromote the health of our open source scientific computingecosystemSupport the developme...
IPython Open Source Academia WrapupOutline1 IPython: Interactive Python2 The Life of an Open Source Project3 Academia vs O...
The future of IPython: a 2-year roadmapSpring/summer 2013: IPython 1.0Notebook document management (nbconvert)JavaScript i...
In closing: our vision of scientific computingBuild on the right abstractionsThe kernel: unify interactive and parallel com...
In closing: our vision of scientific computingBuild on the right abstractionsThe kernel: unify interactive and parallel com...
In closing: our vision of scientific computingBuild on the right abstractionsThe kernel: unify interactive and parallel com...
John D. Hunter, 1968-2012: http://matplotlib.orgMemorial fund: http://numfocus.org/johnhunter
Upcoming SlideShare
Loading in …5
×

IPython: A Modern Vision of Interactive Computing (PyData SV 2013)

2,328 views

Published on

Slides for a talk given by Fernando Pérez at PyData Silicon Valley 2013

Published in: Technology, Education

IPython: A Modern Vision of Interactive Computing (PyData SV 2013)

  1. 1. IPython Open Source Academia WrapupIPythonA modern vision of interactive computingFernando Pérezhttp://fperez.org, @fperez_orgFernando.Perez@berkeley.eduHenry H. Wheeler Jr. Brain Imaging Center, UC BerkeleyPyData 2013, Silicon ValleyMarch 20, 2013
  2. 2. IPython Open Source Academia WrapupOutline1 IPython: Interactive Python2 The Life of an Open Source Project3 Academia vs Open Source4 WrapupFP (UC Berkeley) IPython 3/20/13 2 / 34
  3. 3. In the beginning, IBM said...Let there be FORTRAN
  4. 4. In the beginning, IBM said...Let there be FORTRAN
  5. 5. Beyond (Floating Point) Number CrunchingHardwarefloating pointArbitrary precisionintegersRationalsInterval arithmeticSymbolic manipulationFORTRANExtended precisionfloating pointText processingDatabasesGraphical userinterfacesWeb interfacesHardwarecontrolMulti-languageintegrationData formats: HDF5, XML, ...
  6. 6. The purpose of computing is insight, not numbers.Richard Hamming, 1962
  7. 7. IPython Open Source Academia WrapupThe computer as microscopeExploratory: Problem’s definition evolves as we understand it.No ‘requirements’ to build an application against.Mathematica, Maple, Matlab, IDL, etc.All have an interactive environment.Applications LanguagesFP (UC Berkeley) IPython 3/20/13 6 / 34
  8. 8. IPython: part of a Rich EcosystemIPythonNetworkX
  9. 9. IPython Open Source Academia WrapupThe Lifecycle of a Scientific Idea (schematically)1 Individual exploratory work2 Collaborative development3 Parallel production runs (HPC, cloud, ...)4 Publication (with reproducible results!)5 Education6 Goto 1.The Problem with most toolsBarriers and discontinuities in workflow in between all the stepsFP (UC Berkeley) IPython 3/20/13 8 / 34
  10. 10. IPython Open Source Academia WrapupThe Lifecycle of a Scientific Idea (schematically)1 Individual exploratory work2 Collaborative development3 Parallel production runs (HPC, cloud, ...)4 Publication (with reproducible results!)5 Education6 Goto 1.The Problem with most toolsBarriers and discontinuities in workflow in between all the stepsFP (UC Berkeley) IPython 3/20/13 8 / 34
  11. 11. IPython’s goal:Fluid transitions in all these steps
  12. 12. Demo
  13. 13. IPython Open Source Academia WrapupPillar #1: An architecture for interactive computingFP (UC Berkeley) IPython 3/20/13 11 / 34
  14. 14. IPython Open Source Academia WrapupPillar #2: the Notebook FormatJSON but version control-friendlyEasy for machine processing, fixable by hand if need be.Lots of hooks for metadataNot Python-specific (Ruby, JS notebooks exist, R, Julia planned)Produce Markdown, reST, LATEX, HTML, etc...An open format for sharing, publishing andarchiving executable computational workFP (UC Berkeley) IPython 3/20/13 12 / 34
  15. 15. IPython Open Source Academia WrapupOutline1 IPython: Interactive Python2 The Life of an Open Source Project3 Academia vs Open Source4 WrapupFP (UC Berkeley) IPython 3/20/13 13 / 34
  16. 16. Documented protocols and formats:a growing ecosystem around IPython
  17. 17. An Emacs Notebook Client!Takafumi Arakakihttp://tkf.github.com/emacs-ipython-notebook
  18. 18. Microsoft Visual Studio 2010 integrated consoleDino Viehland and Shahrokh Mortazavi (Microsoft)http://pytools.codeplex.com
  19. 19. A vim client to control an IPython kernel/consolePaul Ivanov (Berkeley)https://github.com/ivanov/vim-ipython
  20. 20. Notebooks on Windows Azure CloudShahrokh Mortazavi (Microsoft), B.G., F.P.http://bit.ly/JQeojD
  21. 21. Star Cluster: IPython parallel+Notebook on Amazon EC2Justin Riley (MIT)http://web.mit.edu/star/cluster
  22. 22. NBViewer: easy notebook sharingMatthias Bussonnierhttp://nbviewer.ipython.org
  23. 23. Other projects using IPythonScientificEPD: Enthought Python Distribution.Anaconda: Continuum Python Distribution.Sage: open source mathematics.PyRAF: Space Telescope Science InstituteCASA: Nat. Radio Astronomy ObservatoryGanga: CERNPyMAD: neutron spectrom., Laue LangevinSardana: European Synchrotron RadiationASCEND: eng. modeling (Carnegie Mellon).JModelica: dynamical systems.DASH: Denver Aerosol Sources and Health.Trilinos: Sandia National Lab.DoD: baseline configuration.NiPype: computational pipelines, MIT.PyIMSL Studio, by Visual Numerics....Web/OtherVisual Studio 2010: MS.Django.Turbo Gears.Pylons web frameworkZope and Plone CMS.Axon Shell, BBCKamaelia.Schevo database.Pitz: distributedtask/bug tracking.iVR (interactive VirtualReality).Movable Python(portable Pythonenvironment)....
  24. 24. How did we get here?A brief history of IPythonOctober 2001: “just a little afternoon hack”My own $PYTHONSTARTUP:ipython-0.0.1.py: 259 lines.In [N]: prompts and _N results cache.IPP (Interactive Python Prompt) by Janko Hauser (Oceanography)LazyPython by Nathan Gray (CS Caltech)2002: Ignore John Hunter’s Gnuplot support patches... let there be matplotlib(actually finish my PhD!)2005: Brian Granger, Min Ragan-KelleyFirst parallel tools, Twisted-based2005-2008: Ville Vainio, Gaël Varoquaux, Laurent DufréchouCore maintenance, Wx integration.
  25. 25. Summer 2009: NIH-funded cleanup by Brian.March 2010: prototype networked shell using ØMQ2-day sprint with BrianEnthought funds Qt console. Min ports parallel code to ØMQCore architecture ready, foundation for NotebookFall 2010James Gao at Berkeley builds (5th!) Notebook Prototype.Summer 2011Brian rebuids James’ prototype into today’s Notebook.
  26. 26. An important plothttp://www.ohloh.net/p/ipython
  27. 27. (Incomplete) Cast of CharactersBrian Granger - Physics, Cal State San Luis ObispoMin Ragan-Kelley - Nuclear Engineering, UC BerkeleyMatthias Bussonnier - Physics, Institut Curie, ParisBrad Froehle - Mathematics, UC BerkeleyPaul Ivanov - Neuroscience, UC Berkeley.Robert Kern - EnthoughtThomas Kluyver - Biology, U. SheffieldJonathan March- EnthoughtEvan Patterson - Physics, Caltech/EnthoughtJörgen Stenarson - Elect. Engineering, Sweden.Stefan van der Walt - UC BerkeleyJohn Hunter - TradeLink Securities, Chicago.Prabhu Ramachandran - Aerospace Engineering, IIT Bombay.Satra Ghosh- MIT NeuroscienceGaël Varoquaux - Neurospin (Orsay, France)Ville Vainio - CS, Tampere University of Technology, FinlandBarry Wark - Neuroscience, U. Washington.Ondrej Certik - Physics, U Nevada RenoDarren Dale - CornellJustin Riley - MITMark Voorhies - UC San FranciscoNicholas Rougier - INRIA Nancy Grand EstThomas Spura - Fedora projectMany more! (~220 commit authors)
  28. 28. IPython Open Source Academia WrapupOutline1 IPython: Interactive Python2 The Life of an Open Source Project3 Academia vs Open Source4 WrapupFP (UC Berkeley) IPython 3/20/13 26 / 34
  29. 29. Support at the edges of academic fundingEnthought, Austin, TX: Lots!Microsoft: WinHPC support, Visual Studio integration, Azure(thanks to Shahrokh Mortazavi).DoD/DRC Inc: funding through Sept. 2012 (thanks to JoseUnpingco and Chris Keees).NIH: via NiPy grantNSF: via Sage compmath grantGoogle: summer of code 2005, 2010.Tech-X Corp., Boulder, CO: Parallel/notebook (previous versions)Recent stable funding (2 years, 7 people, J. Taylor):
  30. 30. Open Source:skills, tools and practices we need!A culture where things get done.Wildly collaborativeReproducible by necessityVersion control, testing, documentation, public peer review, etc.
  31. 31. Reward Structure in academia:we punish all of the aboveDepartmental boundaries: interdisciplinary work is a great buzzword,not such a great career path.Computational heritage is built on codenot on citationsContinuous evolution vs publication milestonesAuthorship in collaborative works vs the first-author paper.Scholarship and intellectual effort embedded in the code.
  32. 32. NumFOCUS: Open Code, Better SciencePromote the health of our open source scientific computingecosystemSupport the development of multiple projects.Community-created and driven.A neutral ground for industry, academia and government to supportscientific open source.501(c)3 - donations are tax-exempt in the USAhttp://numfocus.org
  33. 33. IPython Open Source Academia WrapupOutline1 IPython: Interactive Python2 The Life of an Open Source Project3 Academia vs Open Source4 WrapupFP (UC Berkeley) IPython 3/20/13 31 / 34
  34. 34. The future of IPython: a 2-year roadmapSpring/summer 2013: IPython 1.0Notebook document management (nbconvert)JavaScript internals cleanupFall 2013Interactive JavaScript APIWith callbacks to remote kernels.2014Multiuser serverSimple to deployTrusted (shell OK) Unix users in a lab, group, class, etc.https://github.com/ipython/ipython/wiki/Roadmap:-IPython
  35. 35. In closing: our vision of scientific computingBuild on the right abstractionsThe kernel: unify interactive and parallel computing→ you only have one brain!A single protocol: many kernels, many clients.Communications and loggingthe protocol is the notebook file format.Insight and communication (Hamming)“Literate computing” vs “literate programming”.Build a community and an ecosystem“How to Scale a Code in the Human Dimension”, M. Turk,http://arxiv.org/abs/1301.7064.
  36. 36. In closing: our vision of scientific computingBuild on the right abstractionsThe kernel: unify interactive and parallel computing→ you only have one brain!A single protocol: many kernels, many clients.Communications and loggingthe protocol is the notebook file format.Insight and communication (Hamming)“Literate computing” vs “literate programming”.Build a community and an ecosystem“How to Scale a Code in the Human Dimension”, M. Turk,http://arxiv.org/abs/1301.7064.
  37. 37. In closing: our vision of scientific computingBuild on the right abstractionsThe kernel: unify interactive and parallel computing→ you only have one brain!A single protocol: many kernels, many clients.Communications and loggingthe protocol is the notebook file format.Insight and communication (Hamming)“Literate computing” vs “literate programming”.Build a community and an ecosystem“How to Scale a Code in the Human Dimension”, M. Turk,http://arxiv.org/abs/1301.7064.
  38. 38. John D. Hunter, 1968-2012: http://matplotlib.orgMemorial fund: http://numfocus.org/johnhunter

×