Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Applying the Unix Philosophy to Django projects: a report from the real world

1,348 views

Published on

Software architecture is often overlooked: pushed by deadlines and hurry, we tend to make our applications a giant ball of mud. Here is where the unix philosophy comes to help.

In this talk I will give an overview of the unix philosophy and I will explain how I have applied it to django development for the OpenWISP and NetJSON projects (a set of software modules and standard formats that can be used to deploy wireless networks and implement a public wifi service) with encouraging results. In this talk I will cover the benefits and the downside of such approach, showing example implementations that are being tested in the real world.

Published in: Technology
  • Be the first to comment

Applying the Unix Philosophy to Django projects: a report from the real world

  1. 1. Unix Philosophy & Django projects Applying the Unix Philosophy to Django projects a report from the real world
  2. 2. Who am I? Federico Capoano (A.K.A. Nemesis) OpenWISP core developer Working on public wif at Cineca since 2012
  3. 3. What is OpenWISP? OpenWISP is a set of software modules that can be used to deploy and manage wireless networks (public wif, backbone, mesh networks)
  4. 4. OpenWISP 1 OpenWISP 1: started in 2008-2016 Focused on public wif for Italian cities Fast growth of features to accommodate requirements
  5. 5. Problems of OpenWISP 1 ● monolithic ruby on rails apps
  6. 6. Problems of OpenWISP 1 ● monolithic ruby on rails apps ● tight coupling
  7. 7. Problems of OpenWISP 1 ● monolithic ruby on rails apps ● tight coupling ● duplicated logic
  8. 8. Problems of OpenWISP 1 ● monolithic ruby on rails apps ● tight coupling ● duplicated logic ● policy not separated from mechanisms
  9. 9. Problems of OpenWISP 1 ● monolithic ruby on rails apps ● tight coupling ● duplicated logic ● policy not separated from mechanisms ● complexity
  10. 10. Negative consequences (OpenWISP 1) These problems caused several very negative consequences
  11. 11. 1. "Giant ball of mud" problems Many features in a single codebase Hard to add new features in a clean way Hard to maintain over time (eg: upgrade dependencies, fx bugs)
  12. 12. 2. Inflexibility Hard to extend without changing the core Hard to adapt it and reuse it in different contexts Specialized features available in the codebase are not easily reusable
  13. 13. 3. Little or no contributions Complexity scared away potential external contributors We only got low quality patches which were never merged
  14. 14. 4. Costs and risk of death Increased costs to keep the project alive Many years of work risked to fade into oblivion
  15. 15. 5. Duplication of efforts Little reusability means: reinventing the wheel no ecosystem of interoperable tools
  16. 16. Nikola is not happy
  17. 17. This can't be... There must be a better way!
  18. 18. The Unix Philosophy What is the unix philosophy?
  19. 19. 1. Approach to software development ● Bottom-up
  20. 20. 1. Approach to software development ● Bottom-up ● Pragmatic
  21. 21. 1. Approach to software development ● Bottom-up ● Pragmatic ● Grounded in experience
  22. 22. 1. Approach to software development ● Bottom-up ● Pragmatic ● Grounded in experience ● Founded in empirical skepticism
  23. 23. 2. Mature Evolved in the Unix community since 1970
  24. 24. 3. Change tolerant Born during the digital revolution 1970-2010 It kept pace with the dramatic evolution of technology
  25. 25. The Unix Philosophy in short What are the main concepts of the Unix philosophy?
  26. 26. "Do one thing and do it well" ● Break down big problems in smaller problems
  27. 27. "Do one thing and do it well" ● Break down big problems in smaller problems ● Write simple programs that solve one problem well
  28. 28. "Do one thing and do it well" ● Break down big problems in smaller problems ● Write simple programs that solve one problem well ● Write programs to work together
  29. 29. "Do one thing and do it well" ● Break down big problems in smaller problems ● Write simple programs that solve one problem well ● Write programs to work together ● Value simplicity and maintainability over performance and complexity
  30. 30. What advantages does it bring? What are the main advantages of applying the Unix Philosophy?
  31. 31. 1. Saner development Solving one problem at time is simpler
  32. 32. 2. Maintainability Maintaining a simple program is less expensive
  33. 33. 3. Usability A simple program is easier to use
  34. 34. 4. Readability A simple program is easier to read and understand
  35. 35. 5. Agility A simple program will be more easily readapted to new requirements
  36. 36. 6. Reuse Being easier to use, understand, read and maintain it will be readapted to work in different contexts and will attract more contributors
  37. 37. 7. Ecosystem Many such programs form an ecosystem of interoperable tools (like basic lego building blocks) and Allow to solve complex problems faster
  38. 38. ME GUSTA
  39. 39. The Unix Philosophy in detail The 17 rules of the Unix Philosophy From "The art of unix programming" by Eric Raymond
  40. 40. 1. Rule of Modularity Write simple parts connected by clean interfaces source
  41. 41. 2. Rule of Clarity Clarity is better than cleverness source
  42. 42. 3. Rule of Composition Design programs to be connected to other programs source
  43. 43. 4. Rule of Separation Separate policy from mechanism; separate interfaces from engines source
  44. 44. 5. Rule of Simplicity Design for simplicity; add complexity only where you must source
  45. 45. 6. Rule of Parsimony Write a big program only when it is clear by demonstration that nothing else will do source
  46. 46. 7. Rule of Transparency Design for visibility to make inspection and debugging easier source
  47. 47. 8. Rule of Robustness Software is said to be robust when it performs well under unexpected conditions which stress the designer's assumptions, as well as under normal conditions source
  48. 48. 9. Rule of Representation Fold knowledge into data so program logic can be stupid and robust source
  49. 49. 10. Rule of Least Surprise In interface design, always do the least surprising thing source
  50. 50. 11. Rule of Silence When a program has nothing surprising to say, it should say nothing source
  51. 51. 12. Rule of Repair When you must fail, fail noisily and as soon as possible source
  52. 52. 13. Rule of Economy Programmer time is expensive; conserve it in preference to machine time source
  53. 53. 14. Rule of Generation Avoid hand-hacking; write programs to write programs when you can source
  54. 54. 15. Rule of Optimization Prototype before polishing; get it working before you optimize it source
  55. 55. 16. Rule of Diversity Distrust all claims for “one true way” source
  56. 56. 17. Rule of Extensibility Design for the future, because it will be here sooner than you think source
  57. 57. How to apply it to Django? How to apply the Unix philosophy to django projects? Let's see some real world examples from OpenWISP 2
  58. 58. 1. Rule of Modularity ● Develop main features as reusable django apps
  59. 59. 1. Rule of Modularity ● Develop main features as reusable django apps ● One django app for each group of related features
  60. 60. 1. Rule of Modularity ● Develop main features as reusable django apps ● One django app for each group of related features ● Document public API
  61. 61. 1. Rule of Modularity ● Develop main features as reusable django apps ● One django app for each group of related features ● Document its public API ● Include a license and a changelog
  62. 62. 1. Rule of Modularity ● Develop main features as reusable django apps ● One django app for each group of related features ● Document its public API ● Include a license and a changelog ● Publish your app on pypi and djangopackages.org
  63. 63. 1. Real world examples ● Confgurations of routers and VPNs: django-netjsonconfg
  64. 64. 1. Real world examples ● Confgurations of routers and VPNs: django-netjsonconfg ● PKI management (x509 certifcates): django-x509
  65. 65. 1. Real world examples ● Confgurations of routers and VPNs: django-netjsonconfg ● PKI management (x509 certifcates): django-x509 ● Multi-tenancy: contributed to django-organizations reuse existing projects when possible!
  66. 66. 2. Rule of Clarity Explicit is better than implicit This is already a widely accepted concept in the python world
  67. 67. 3. Rule of Composition Combine, extend and customize django reusable apps in your fnal django project
  68. 68. 3. Real world examples ● openwisp-users: extends django-organizations
  69. 69. 3. Real world examples ● openwisp-users: extends django-organizations ● openwisp-controller: depends on openwisp-users and extends django-netjsonconfg and django-x509
  70. 70. 3. Real world examples ● openwisp-users: extends django-organizations ● openwisp-controller: depends on openwisp-users and extends django-netjsonconfg and django-x509 ● Final result handled by ansible-openwisp2 in an (almost) transparent manner
  71. 71. 4. Rule of Separation ● implement mechanisms as libraries
  72. 72. 4. Rule of Separation ● implement mechanisms as libraries ● implement policy as a confguration
  73. 73. 4. Rule of Separation ● implement mechanisms as libraries ● implement policy as a confguration ● make these libraries highly confgurable
  74. 74. 4. Rule of Separation ● implement mechanisms as libraries ● implement policy as a confguration ● make these libraries highly confgurable ● make these libraries work with data
  75. 75. 4. Rule of Separation ● implement mechanisms as libraries ● implement policy as a confguration ● make these libraries highly confgurable ● make these libraries work with data ● clearly defne input and output
  76. 76. 4. Real world examples ● netjsonconfg: python library for generating router & VPN confgurations from NetJSON objects
  77. 77. 4. Real world examples ● netjsonconfg: python library for generating router & VPN confgurations from NetJSON objects ● django-netjsonconfg: web interface that uses netjsonconfg under the hood
  78. 78. >>> from netjsonconfig import OpenWrt >>> router = OpenWrt({ ... "general": {"hostname": "HomeRouter"} ... }) >>> print(router.render()) package system config system 'system' option hostname 'HomeRouter' option timezone 'UTC' option zonename 'UTC'
  79. 79. 5. Rule of Simplicity ● start with very basic features
  80. 80. 5. Rule of Simplicity ● start with very basic features ● release your project early (even if you feel it's incomplete)
  81. 81. 5. Rule of Simplicity ● start with very basic features ● release your project early (even if you feel it's incomplete) ● add one feature at time as your understanding grows
  82. 82. 5. Rule of Simplicity ● start with very basic features ● release your project early (even if you feel it's incomplete) ● add one feature at time as your understanding grows ● add complexity only when necessary
  83. 83. 5. Real world examples as of April 2017:
  84. 84. 5. Real world examples as of April 2017: ● 24 releases of netjsonconfg
  85. 85. 5. Real world examples as of April 2017: ● 24 releases of netjsonconfg ● 25 releases of django-netjsonconfg
  86. 86. 5. Real world examples as of April 2017: ● 24 releases of netjsonconfg ● 25 releases of django-netjsonconfg ● 5 releases of django-x509
  87. 87. 5. Real world examples as of April 2017: ● 24 releases of netjsonconfg ● 25 releases of django-netjsonconfg ● 5 releases of django-x509 OpenWISP 2 still lacks some features of OpenWISP 1
  88. 88. 6. Rule of Parsimony Prefer creating new reusable apps when adding big features unless doing this complicates things a lot with no real advantages
  89. 89. 6. Real world examples PKI management in new app: django-x509 VPN confgurations added to existing app: django-netjsonconfg
  90. 90. 7. Rule of Transparency Log unexpected events using the python logging facility
  91. 91. Logging of bad requests in django-netjsonconfg import logging logger = logging.getLogger(__name__) def invalid_response(request, error, status): logger.warning(error, extra={'request': request, 'stack': True}) return ControllerResponse(error, status=status)
  92. 92. 7. Real world examples Provide good default logging Provide support for sentry Take a look at a real example
  93. 93. 8. Rule of Robustness Add constraints to your reusable django apps only when necessary: ● avoid very strict validation rules ● provide confgurable settings
  94. 94. 8. Real world examples ● settings in django-netjsonconfg ● settings in django-x509
  95. 95. 9. Rule of Representation Fold complex information in data structures Process these data structures with algorithms Try to make algorithms framework-agnostic (if possible)
  96. 96. 9. Real world examples in OpenWISP 2, confguration of routers is implemented as a single text feld formatted as NetJSON (in OpenWISP 1 each available confguration had its own database table and model, very hard to maintain and evolve)
  97. 97. # NetJSON DeviceConfiguration example { "general": {"hostname": "HomeRouter"}, "interfaces": [ { "name": "eth0", "type": "ethernet", "addresses": [ { "address": "192.168.1.1", "mask": 24, "proto": "static", "family": "ipv4" } ] } ] }
  98. 98. 10. Rule of Least Surprise This has become a widely accepted concept in the IT industry
  99. 99. 11. Rule of Silence Just don't make your apps annoying It's not that hard :-P
  100. 100. 12. Rule of Repair Errors should never pass silently Unless explicitly silenced (from the zen of python) Let's add: fail fast, noisily and early
  101. 101. from django.core.exceptions import ImproperlyConfigured from .settings import REGISTRATION_ENABLED, SHARED_SECRET if REGISTRATION_ENABLED and not SHARED_SECRET: msg = 'NETJSONCONFIG_SHARED_SECRET not set!' raise ImproperlyConfigured(msg) Full real world example available in django-netjsonconfg
  102. 102. 13. Rule of Economy When Unix was born this was a radical idea: assembler was the norm, C was considered a higher level language Python is a consequence of the success of that radical idea Embrace this concept
  103. 103. 14. Rule of Generation When you fnd yourself writing lots of boilerplate code, try to use meta-programming or code generators
  104. 104. 14. meta-programmed urlpatterns urls.py in openwisp-controller
  105. 105. 14. Auto-generated UI from json-schema JSON-Schema of this UI available on github
  106. 106. 15. Rule of Optimization ● write a frst early version of your django app ● do not forget to write automated tests ● refactor and polish
  107. 107. 16. Rule of Diversity Maintain a skeptic attitude towards dogmas and fads Measure the effectiveness of an approach before embracing it
  108. 108. 17. Rule of Extensibility Provide ways to extend and customize the behaviour of modules without the need of changing the core code
  109. 109. 17. Extensibility: abstract models ● Provide abstract models in your most important modules
  110. 110. 17. Extensibility: abstract models ● Provide abstract models in your most important modules ● Store these in a python fle which does not import concrete models (otherwise other django apps won't be able to import them)
  111. 111. 17. Extensibility: abstract models abstract models in django-netjsonconfg example usage
  112. 112. 17. Extensibility: abstract models abstract models in django-x509 example usage
  113. 113. 17. Extensibility: base admin ● Provide base admin classes in your main modules (avoid duplication)
  114. 114. 17. Extensibility: base admin ● Provide base admin classes in your main modules (avoid duplication) ● Store them in a python fle which does not import concrete models (otherwise other django-apps won't be able to import them)
  115. 115. 17. Extensibility: base admin base admin classes django-netjsonconfg example usage
  116. 116. 17. Extensibility: base views ● If your reusable django app has views, provide generic views that can be extended ● Store them in a python fle which does not import concrete models (otherwise other django-apps won't be able to import them)
  117. 117. 17. Extensibility: base views base views in django-netjsonconfg example usage
  118. 118. 17. Extensibility: reusable urls If your app provides views that can be extended, third party apps will have to redefne their URLs You may want to avoid this required duplication by providing a mechanism to import urls
  119. 119. 17. Extensibility: reusable urls reusable urls in django-netjsonconfg example usage: from django_netjsonconfig.utils import get_controller_urls from . import views # customized views # creates new url patterns hooked to customized views urlpatterns = get_controller_urls(views)
  120. 120. 17. Extensibility: AppConfig If your reusable django app relies on signal connection for some features, provide a base AppConfig class
  121. 121. 17. Extensibility: AppConfig base AppConfig class in django-netjsonconfg from django_netjsonconfig.apps import OpenWispAppConfig class MyOwnApp(OpenWispAppConfig): name = 'yourapp.config' label = 'config' def __setmodels__(self): # these are your custom models from .models import Config, VpnClient self.config_model = Config self.vpnclient_model = VpnClient
  122. 122. Positive results Positive results achieved in OpenWISP
  123. 123. 1. Faster release cycle & evolution We release new features more often The project is evolving rapidly
  124. 124. 2. Easier maintainance Once a bug in a specifc module is replicated fxing it is easier compared to the work needed to fx bugs in OpenWISP 1
  125. 125. 3. More derivative work OpenWISP 2 was released officially less than 1 year ago (as of April 2017) notwithstanding that, there are already a couple of derivative works
  126. 126. 4. Growth The user base has been growing rapidly Users send feedback and patches
  127. 127. 4. Growth: mailing list
  128. 128. 4. Growth: GSOC 2017 OpenWISP has also been accepted as a mentoring organization for the Google Summer of Code 2017
  129. 129. Disadvantages Is not a bed of roses
  130. 130. 1. Integration issues Combining more django apps may result in integration issues Changes to project confguration can result in bugs that are not caught by unit tests
  131. 131. 1. Solution Integration tests are important in this case Shortcut: I was able to import tests of a base app and repeat them in the extension app This easy fx was good enough for my case
  132. 132. 2. Big features When I needed to introduce a new major feature I needed to change at least a couple of modules
  133. 133. 2. Solution many times this will be necessary If you have to change many modules every time, refactor your code to be more loosely coupled
  134. 134. 3. Repository management overhead ● working with many repositories can be overwhelming
  135. 135. 3. Repository management overhead ● working with many repositories can be overwhelming ● more git tags, pypi releases, versioning, changelogs
  136. 136. 3. Repository management overhead ● working with many repositories can be overwhelming ● more git tags, pypi releases, versioning, changelogs ● more announcements for new releases
  137. 137. 3. Solutions ● import several projects in a single window of your editor
  138. 138. 3. Solutions ● import several projects in a single window of your editor ● automate versioning and changelogs (I haven't done this yet)
  139. 139. 3. Solutions ● import several projects in a single window of your editor ● automate versioning and changelogs (I haven't done this yet) ● do not send announcements for minor releases which contain no real advantage for end users
  140. 140. 4. Documentation fragmentation We have many repositories with their own READMEs and docs We don't have a single documentation website yet
  141. 141. 4. Solutions I added links to specifc module docs from the website I often have to send those links on the mailing list But a central comprehensive documentation website would be better (we don't have this yet)
  142. 142. 5. Bug reports Users don't know where to send bug reports and they write to the mailing list
  143. 143. 5. Solution ● For the moment I reply by pointing to them where to open issues
  144. 144. 5. Solution ● For the moment I reply by pointing to them where to open issues ● But a single issue tracker for all the repositories would be better (even though it may not really solve the "which module?" problem)
  145. 145. 6. Dispersion of popularity We no longer have a single popular repository with many github stars
  146. 146. 6. Solution Stop caring about this Github stars are not an accurate indicator of the success of a project
  147. 147. Conclusions The creators of Unix are not newcomers Following their heritage is a good idea
  148. 148. Delve deep Further resources on the Unix Philosophy and related concepts: ● the art of unix programming ● the pragmatic programmer
  149. 149. Practice Start to gradually apply these concepts in your projects EG: extract big features to separate projects
  150. 150. NO to dogmas ● Practice empirical (scientifc) skepticism
  151. 151. NO to dogmas ● Practice empirical (scientifc) skepticism ● Test your ideas
  152. 152. NO to dogmas ● Practice empirical (scientifc) skepticism ● Test your ideas ● Measure your results
  153. 153. Read existing code Read the code of OpenWISP @openwisp organization on github (or other projects like @openstack)
  154. 154. Unix Philosophy for the win!
  155. 155. Thank you Ideas, critical feedback, suggestions? Talk to me! Find me also on: ● twitter (@nemesisdesign) ● github (@nemesisdesign) ● linkedin (Federico Capoano) You can fnd the slides on slideshare

×