Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Computable Content

277 views

Published on

Lessons learned from 3 (going on 4) generations of Jupyter use cases at O'Reilly Media. In particular, about "Oriole" tutorials which combine video with Jupyter notebooks, Docker containers, backed by services managed on a cluster by Marathon, Mesos, Redis, and Nginx.

https://conferences.oreilly.com/fluent/fl-ca/public/schedule/detail/62859
https://conferences.oreilly.com/velocity/vl-ca/public/schedule/detail/62858

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Computable Content

  1. 1. Computable  Content:  
 Lessons  Learned Paco  Nathan  @pacoid   Director,  Learning  Group  @  O’Reilly  Media   2017-­‐06-­‐21
  2. 2. An  observation 2
  3. 3. Oriole 3
  4. 4. One  approach 4
  5. 5. Jupyter  use  @  O’Reilly  Media ▪ Embracing  Jupyter  notebooks  at  O’Reilly
 oreilly.com/ideas/jupyter-­‐at-­‐oreilly   ▪ Learning  alongside  innovators,  thought-­‐by-­‐thought,  in  context
 oreilly.com/ideas/oreilly-­‐oriole-­‐learn-­‐alongside-­‐innovators-­‐ thought-­‐by-­‐thought-­‐in-­‐context   ▪ Oriole  online  tutorials
 safaribooksonline.com/oriole/   ▪ How  do  you  learn?
 oreilly.com/learning/how-­‐do-­‐you-­‐learn 5
  6. 6. For  example ▪ A  unique  new  medium  blends  code,  data,  text,  
 and  video  into  a  narrated  learning  experience
 with  computable  content   ▪ Purely  browser-­‐based  UX;  zero  installation  
 required   ▪ Substantially  higher  engagement  metrics   ▪ Opens  the  door  for  live  coding  in  assessments 6 oreilly.com/learning/regex-­‐golf-­‐with-­‐peter-­‐norvig
  7. 7. Motivations O’Reilly  needed  a  way  for  authors  to  use  Jupyter  notebooks  to  create   professional  publications.  We  also  wanted  to  integrate  video  narration   into  the  UX.  The  result  is  a  unique  new  medium  called  Oriole:   ▪ Context  as  a  “unit  of  thought”   ▪ Code  and  video  sync’ed  together   ▪ Each  web  session  get  its  own  Docker  container  in  the  cloud   ▪ 100%  HTML  experience,  no  download/install/config  needed   ▪ Jupyter  notebooks  used  in  the  middleware   ▪ Leverage  interactive,  data-­‐driven  graphics 7
  8. 8. Outcomes 8 ▪ Tutorials  are  now  much  quicker  to  publish  than   “traditional”  books  and  videos   ▪ Less  time  required  for  innovators  in  programming,   data  science,  devops,  design,  etc.  –  who  tend  to  be   really  busy  people   ▪ Audience  gets  direct,  hands-­‐on,  contextualized   experience  across  a  wide  variety  of  programming   environments
  9. 9. Limitations 9 ▪ Notebook  kernels  run  REPLs,  so  older  languages   were  not  feasible   ▪ Brief  code  blocks  with  tangible  outcomes  –  
 precludes  business  topics,  systems  engineering,  etc   ▪ What  materials  will  fit  within  a  Docker  container?
  10. 10. Third  iteration  of  Jupyter  @  O’Reilly 10 1. notebooks  as  supplemental  material  to   other  published  work   2. notebooks  published  as  HTML,  as  articles   3. computable  content,  containerized   notebooks  +  video  narratives   4. hosted  notebooks
  11. 11. Long-­‐term  goal:     make  learning  materials  more  powerful  by   integrating  compute  engines  +  data  services 11
  12. 12. Project  Jupyter 12
  13. 13. Project  Jupyter ▪ The  evolution  of  iPython  notebooks,  applied   to  a  range  of  different  programming  languages   and  environments   ▪ https://jupyter.org/   ▪ https://github.com/ipython/ipython/wiki/ IPython-­‐kernels-­‐for-­‐other-­‐languages 13
  14. 14. Example  Notebook 14
  15. 15. Projects 15 ▪ JupyterHub
 github.com/jupyterhub/jupyterhub   ▪ Jupyter  in  Education
 groups.google.com/forum/#!forum/ jupyter-­‐education   ▪ JupyterLab
 github.com/jupyterlab/jupyterlab   ▪ Jupyter  Kernels
 github.com/ipython/ipython/wiki/ IPython-­‐kernels-­‐for-­‐other-­‐languages
  16. 16. A  suite  of  network  protocols Think  of  Jupyter,  at  its  core,  as  a  suite  of   network  protocols:   Jupyter  is  to  the  remote  semantics  of  a  REPL   as…   HTTP  is  to  the  remote  semantics  of  file  share 16
  17. 17. A  suite  of  network  protocols 17 Code%runs% in%a%REPL Kernel Edi0ng%+% Results Notebook Network Protocol
  18. 18. History,  Context 18
  19. 19. Notebook  metaphor Wolfram  Research  introduced   notebooks  in  1988  for  working  
 with  Mathematica 19
  20. 20. Related  work 20
  21. 21. Literate  programming Don  Knuth
 literateprogramming.com/   Paraphrased:
 Instead  of  telling  computers  what  to   do,  tell  other  people  what  you  want   the  computers  to  do 21
  22. 22. Speech  acts PyCon  2016  Keynote,  Lorena  Barba   youtu.be/ckW1xuGVpug?t=35m11s
 (video)   figshare.com/articles/ PyCon2016_Keynote/3407779
 (slides)   Highly  recommended:  speech  acts  
 (based  on  Winograd  and  Flores)  
 as  theory  here 22
  23. 23. Best  Practices 23
  24. 24. The  following  lessons  learned  in  using  Jupyter   notebooks  +  video  for  learning  materials  apply   well  in  many  situations  for  data  science  teams   working  across  an  organization 24
  25. 25. Teaching  with  Jupyter  –  1  of  2 ▪ focus  on  a  concise  “unit  of  thought”   ▪ invest  the  time  and  editorial  effort  to  create  a  good  intro   ▪ keep  your  narrative  simple  and  reasonably  linear   ▪ “chunk”  the  text  and  code  into  understandable  parts   ▪ alternate  between  text,  code,  output,  further  links,  etc.   ▪ code  cells  should  be  brief  (<  10  lines),  must  show  output 25
  26. 26. Teaching  with  Jupyter  –  2  of  2 ▪ load  data+libraries  from  the  container,  not  the  network   ▪ clear  all  output  then  “Run  All”  –  or  it  didn’t  happen   ▪ video  narratives:  there’s  text,  and  there’s  subtext...   ▪ pause  after  each  “beat”:  smile,  breathe,  let  people  follow  you   For  JVM  people:  stop  thinking  only  about  IDEs,  Ivy,  Maven,  etc.  (ibid,  Knuth1984)
 BUILD  UBER  JARS,  LOAD  LIBS  FROM  CONTAINER,  NOT  THE  NETWORK!
 (apologies  for  shousng) 26
  27. 27. Sharing  is  caring In  data  science,  we  see  the  benefits  to  teams  for  shared   insights,  storytelling,  etc.   Meanwhile  domain  expertise  is  generally  more  important   than  knowledge  about  tools   There’s  a  value  for  developers  to  use  notebooks  in  lieu  of   IDEs  in  some  cases  –  what  are  those  cases?   GitHub  now  renders  notebooks,  so  they  can  be  used  for   documentation,  reporting,  etc.   Digital  Object  Identifiers  (DOI)  can  be  assigned  through   Zenodo,  making  notebooks  citable  for  academic  publication 27
  28. 28. Authoring  and  Scale-­‐Out 28
  29. 29. Launchbot.io 29
  30. 30. Achieving  scale ▪ Launchbot.io  allows  a  notebook  author  to   build  a  container  that  includes  the  required   Jupyter  kernel,  installed  libraries,  datasets,   etc.   ▪ Install  Docker  on  your  laptop   ▪ Backend  uses  Git  and  DockerHub  to  manage   containers   ▪ For  scale,  deploy  to  DC/OS  or  a  cloud 30
  31. 31. “A  notebook,  a  container,  and  ~20  minutes  
  of  informal  video  walk  into  a  bar…” 31
  32. 32. System  architecture 32  Tutorial               Middleware  Cluster                
  33. 33. O’Reilly  Strata   NY,  Sep  25-­‐28
 SG,  Dec  4-­‐7   O’Reilly  ArXficial  Intelligence   NY,  Jun  26-­‐29
 SF,  Sep  17-­‐20   JupyterCon   NY,  Aug  22-­‐25 33
  34. 34. 34 Learn  Alongside
 Innovators Just  Enough  Math Building  Data   Science  Teams Hylbert-­‐Speys How  Do  You  Learn? periodic  newslewer  with  updates,  
 events,  conference  summaries…   liber118.com/pxn/
 @pacoid

×