Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Doing the Impossible


Published on

If someone tells you that it is impossible to reimplement some piece of software to be compatible with a newer technology, that is most often not truth.
This is the story about porting RestrictedPython, one of the core packages of Zope to be compatible with Python 3.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Doing the Impossible

  1. 1. Doing the Impossible Porting RestrictedPython to Python 3 The way to Python 3 for Plone? Alexander Loechel PloneConf 2017 - Barcelona 1
  2. 2. –Johnny Appleseed “Type a quote here.” 2 End of Life for Python 2 by 2020 PyCon (April 12th 2020)
  3. 3. PloneConf 2014 Bristol - Plone Roadmap Discussion “Plone on Python 3 
 is impossible, there are too 
 many blockers in Zope. Especially RestrictedPython & AccessControl” 3
  4. 4. Clarke's First Law – Arthur C. Clarke “When a distinguished but elderly scientist states that something is possible, he is almost certainly right. 
 When he states that something is impossible, he is very probably wrong.” 4
  5. 5. – 2015 - Start looking into RestrictedPython “Why should RestrictedPython be the major blocker for 
 Plone on Python 3?” 5
  6. 6. Hanno Schlichting - Zope Release Manager „Every piece of Zope that was not adopted by Plone is literally dead.“ 6
  7. 7. „Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.“ 7 - Brian Kernighan Almost no documentation low test coverage
  8. 8. Python community - around 2000 „Where Zope leads, Python follows“ 8 Zope specific leftovers in the Python 2 Standard Library Deprecated Python Standard Library Modules: • compiler Package (no described upgrade path, almost no documentation)
  9. 9. Clarke's Second Law – Arthur C. Clarke “The only way of discovering the limits of the possible is to venture a little way past them into the impossible.” 9
  10. 10. Clarke's Third Law – Arthur C. Clarke “Any sufficiently 
 advanced technology 
 is indistinguishable 
 from magic.” 10
  11. 11. • TTW (Through-The-Web) • PythonScripts • DocumentTemplate • Workflows • Zope ZCatalog 11
  12. 12. What is RestrictedPython and what is it not? 12
  13. 13. A Workshop on Python at NIST
 SPAM 1 - First PyCon (1994) - Topics I've assembled a strawman list of topics for discussion. New topics
 and comments on the topics listed are certainly welcome. A.Requirements for a "Safe" Python interpreter B. A standard GUI module interface definition for Python C.The requirements for persistent objects in Python D.A Python engineering graphs package E. The standard Python WWW interface F. Embedding Python in a WWW client G.Technical information management using Python H.Support for dynamic loading of foreign language modules in Python I. Replacing make, rcs, and cvs with Python J. An Electronic Data Interchange library for Python K. Discussing the formation of a Python Consortium 13
  14. 14. Rich Hickey “Most of the biggest problem 
 in software are problems of misconception.” 14
  15. 15. No Sandbox Sandboxes most often don’t work 15
  16. 16. Limited, „Safe“ Subset of the Python Programming Language / Grammar • No more Turing Complete • Common Technique (Ada RavenScar Profile) 16
  17. 17. The Way to RestrictedPython 4.0 and Python 3 17
  18. 18. - John Johnson “First, solve the problem. Then, write the code.” 18
  19. 19. • Python 2 Standard Library Module compiler and its Class ast • Not fully documented • No upgrade path for Python 3 described • manual Byte-Code generator • Interpreter specifics —> Only CPython 2 compatible 19
  20. 20. If it is not documented it is not usable
 if it is not tested it did not work
 if it is not checked into version control it did not exists
 if it is not repeatable it is not science • Compiler knowledge was necessary to port RestrictedPython • I did have a degree in Informatics (Computer Science) and 
 learned how to build and manipulate compilers, so give it a try. 20
  21. 21. Plone Open Garden 2015 Start looking into RestrictedPython 21
  22. 22. Kent Beck “Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” 22 Reading & 
 understanding the Code
  23. 23. Louis Sreygley “Without requirements or design, programming is the art of adding bugs to an empty text file.” 23 Documentation First + Test Driven Development
  24. 24. • compiler.ast => ast.AST • builtin compile() function since Python 2.6 accepts ast.AST as input and compiles to Python Byte-Code —> Interpreter independence • compile(source, filename, mode [, flags=ast.PyCF_ONLY_AST [, dont_inherit]]) 
 or ast.parse() return ast.AST With Python 2.6+ / 3.4+ RestrictedPython could be easier to implement 24
  25. 25. Got Access to Zope Foundation Repositories via Tres Seaver at Plone Symposium Tokyo 2015 “Access Required.” 25
  26. 26. >>> import this The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren't special enough to break the rules. … PEP 20 - 19. Aug. 2004 26 Apply Python’s PEP8 and Plone’s Coding Conventions —> Make the Code more readable
  27. 27. – Richard Feynman “The first principle is that you must not fool yourself — and you are the easiest person to fool.” 27 Writing Tests & raising test coverage —> Functional Test of all Python Syntax cases
  28. 28. tox & . • tox: local tests of multiple python versions • pytest: modern Python testing framework, that allows: • parameterisation (inputs and functions) • skip_if • assert Talk: Modern Python Testing 28
  29. 29. Test example (pytest) c_exec = (c_exec‘, [ 
 ]) @pytest.mark.parametrize(*c_exec) def test_(c_exec): result = c_exec('a = max([1, 2, 3])')
 assert result.errors == () loc = {} exec(result.code, {}, loc) assert loc['a'] == 3 assert result.used_names == {'max': True} 29
  30. 30. Alpine City Sprint 2016 / 2017 Join in of Gocept & other Zope Core Developers 30
  31. 31. Ian Hickson „Things that are impossible
 just take longer.“ 31
  32. 32. May 2017 Release of RestrictedPython 4.0a1 —> 4.0.b2 Welcome to the Python 3 Wonderland32
  33. 33. Philip J. Eby - Python Core Developer - Author PEP 333/3333 - WSGI „Those who do not study Zope, are condemned to reinvent it.“ 33
  34. 34. Bjarne Stroustrup “If you think it’s simple, 
 then you have 
 misunderstood the problem.” 34
  35. 35. My Wish Make RestrictedPython more known enable other Projects and Frameworks to use 
 a „Safe“ Python Interpreter through the Web
  36. 36. John Gall - Systemantics: How Systems Really Work and How They Fail A complex system that works is invariably
 found to have evolved from a simple system
 that worked. A complex system designed from scratch never works and cannot be patched up to make it work. 
 You have to start over with a working simple system. 36
  37. 37. Lessons Learned while Porting RestrictedPython • Don’t take »„impossible“ to port / fix« statements serious • Adopt modern tools and frameworks that helps: • tox & pytest —> Update Best Practices for Plone development 37
  38. 38. Some further Talk recommendations • Zope on Python 3
 Hanno Schlichting - Friday 14:30-15:15 - Auditori • Modern Python Testing
 Alexander Loechel - Thursday 14:30-15:15 - Sala d'actes • subtemplates in bobtemplates.plone or on the way to plonecli 
 Maik Derstappen - Friday 15:25-15:55 - Sala d'actes 38
  39. 39. Clarke's Law of Revolutionary Ideas – Arthur C. Clarke Every revolutionary idea 
 — in science, politics, art, or whatever — 
 seems to evoke three stages of reaction. 
 They may be summed up by the phrases: 1. "It's completely impossible — don't waste my time" 2. "It's possible, but it's not worth doing" 3. "I said it was a good idea all along" 39