Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jupyter Kernel: How to Speak in Another Language

1,799 views

Published on

A PyCon TW 2016 speech about Jupyter Kernels

Published in: Technology

Jupyter Kernel: How to Speak in Another Language

  1. 1. Jupyter Kernel How to Speak in Another Language
  2. 2. How to Make a Jupyter Kernel:
  3. 3. 1: 5 ømq sockets
  4. 4. 2: Implement Protocol
  5. 5. 3. Holy Shit, YOU RUINED EVERYTHING, LIKE EVERYTHING YOU DO
  6. 6. 4. This Kernel Is Like Your Life
  7. 7. 5. Begin a Lot of Things, Left Undone
  8. 8. 6. And Nobody Loves You
  9. 9. Understanding Jupyter Kernel
  10. 10. Agenda • Jupyter, IPython, notebook, console, client, kernel • The Interactive Computing Protocol • Implementing a Kernel • Live Demo
  11. 11. About this guy • 廖偉涵 Adrian Liaw • github.com/adrianliaw • High School Homeschooler • A Pythonista • Student of Udacity’s
 Data Analyst Nanodegree
  12. 12. About this guy • Amateur Pianist • Classical Music Lover • Coffee Shop Lover
  13. 13. IPYTHON ENHANCED PYTHON SHELL
  14. 14. IPYTHON NOTEBOOK WEB-BASED NOTEBOOK ENVIRONMENT
  15. 15. The Big Split™
  16. 16. Jupyter Notebook (With R Kernel)
  17. 17. -Project Jupyter “Project Jupyter was born out of the IPython Project in 2014 as it evolved to support interactive data science and scientific computing across all programming languages.”
  18. 18. CLIENT KERNEL ØMQ Sockets
  19. 19. CLIENT KERNEL Execute “10 + 3”
  20. 20. CLIENT KERNEL Result: 13
  21. 21. CLIENT KERNEL CLIENT
  22. 22. What’s Inside IPython • The Interactive IPython Shell (No ØMQ) • Magic Commands • Auto Word Completer • Beautiful Traceback • Shell History Management
  23. 23. Uh, And Kernel?
  24. 24. IPyKernel • Jupyter’s “Kernel 0” • ØMQ Handlers • github.com/ipython/ipykernel
  25. 25. What’s Inside Jupyter • Web-based Notebook Interface & nbconvert • Interactive Qt Console • Terminal-based Console • Spec for Notebook Document Format • Spec for Interactive Computing Protocol
  26. 26. CLIENT KERNEL ØMQ Sockets
  27. 27. Interactive Computing Protocol • i.e. Jupyter Messaging Protocol • Communication Between Kernel and Client • Based on ØMQ and JSON
  28. 28. DEALER ROUTER CLIENT KERNEL Shell SUB PUBIOPub DEALER ROUTER stdin DEALER ROUTER Control REQ REP Heartbeat
  29. 29. Connection File { "transport": "tcp", "ip": "127.0.0.1", "shell_port": 56166, "iopub_port": 56167, "stdin_port": 56168, "control_port": 56169, "hb_port": 56170, "signature_scheme": "hmac-sha256", "key": "<hashed key>" }
  30. 30. Message Format { 'header' : { 'msg_id' : str, 'username' : str, 'session' : str, 'date': str, 'msg_type' : str, 'version' : '5.0', }, 'parent_header' : dict, 'metadata' : dict, 'content' : dict, }
  31. 31. http://bit.ly/JupyterMessaging
  32. 32. { "header": { "msg_type": "kernel_info_request", ... }, "content": {}, } CLIENT KERNEL Shell
  33. 33. CLIENT KERNEL Shell { "header": { "msg_type": "kernel_info_reply", ... }, "parent_header": {...}, "content": { "banner": "Python 3.5.1 ... IPython 4.0.0 ...", "implementation": "ipython", "implementation_version": "4.0.0", "language_info": { "name": "python", "version": "3.5.1", ... }, }, }
  34. 34. CLIENT KERNELShell { "header": { "msg_type": "execute_request", ... }, "content": { "code": "print('HAI')", "silent": False, "store_history": True, }, }
  35. 35. CLIENT KERNELIOPub { "header": {"msg_type": "stream", ...}, "parent_header": {...}, "content": { "name": "stdout", "text": "HAIn", }, }
  36. 36. CLIENT KERNELShell { "header": { "msg_type": "execute_reply", ... }, "parent_header": {...}, "content": { "execution_count": 1, "status": "ok", "user_expressions": {}, }, }
  37. 37. CLIENT KERNELShell { "header": {"msg_type": "execute_request", ...}, "content": { "code": "pd.read_csv('foo.csv')", "silent": False, "store_history": True, }, }
  38. 38. CLIENT KERNELIOPub { "header": {"msg_type": "execute_result", ...}, "parent_header": {...}, "content": { "execution_count": 5, "data": { "text/plain": " col1 col2n A 3n....", "text/html": "<div><table>...</table></div>", }, }, }
  39. 39. CLIENT KERNELShell { "header": { "msg_type": "execute_reply", ... }, "parent_header": {...}, "content": { "execution_count": 5, "status": "ok", "user_expressions": {}, }, }
  40. 40. CLIENT KERNELShell { "header": {"msg_type": "execute_request", ...}, "content": { "code": "read.csv('table.csv')", "silent": False, "store_history": True, }, }
  41. 41. CLIENT KERNELIOPub { "header": {"msg_type": "execute_result", ...}, "parent_header": {...}, "content": { "execution_count": 1, "data": { "text/plain": " col1 col2n A 3n....", "text/html": "<table>...</table>", }, }, }
  42. 42. CLIENT KERNELShell { "header": { "msg_type": "execute_reply", ... }, "parent_header": {...}, "content": { "execution_count": 1, "status": "ok", "user_expressions": {}, }, }
  43. 43. MAKING A KERNEL
  44. 44. Kernel Types • Native Kernel • Python Wrapper Kernel • REPL Wrapper Kernel
  45. 45. IRKernel 11. zmqctx <<- zmq.ctx.new() 12. sockets <<- list( 13. hb = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$REP), 14. iopub = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$PUB), 15. control = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$ROUTER), 16. stdin = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$ROUTER), 17. shell = zmq.socket(zmqctx, .pbd_env$ZMQ.ST$ROUTER))
  46. 46. IRKernel IRuby IHaskell cling IJulia IScala Lua Kernel Clojure-Kernel ITorch IJavascript ipykernel
  47. 47. Wrapper Kernel • Python-driven • ipykernel.kernelbase.Kernel • http://bit.ly/WrapperKernel
  48. 48. from ipykernel.kernelbase import Kernel class MyKernel(Kernel): def do_execute def do_complete def do_history def do_shutdown
  49. 49. Hy Kernel Redis Kernel Xonsh C Kernel Matlab Kernel SAS Kernel
  50. 50. REPL Wrapper from pexpect.replwrap import REPLWrapper
  51. 51. Octave Kernel Jupyter Powershell Bash Kernel Mongo Kernel Scilab Kernel
  52. 52. LIVE DEMO
  53. 53. Q&A
  54. 54. Reference • P.3, P.4, P.17, P.20, P.50 images: Project Jupyter • P.6, P.7, P.8 images: Molg H. • P.16: Project Jupyter • P.11 image: Toomore Chiang • P.12 image: Virginia Cheng • P.13 image: ipython • P.45 image: R

×