Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Is Python still production ready ? Ludovic Gasc

396 views

Published on

PyParis 2017
http://pyparis.org

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Is Python still production ready ? Ludovic Gasc

  1. 1. Is Python still production ready? For Humans facing software
  2. 2. Warning 1. Clue/spoiler alert: We are at PyParis... 2. Because it’s a keynote, you won’t see in-depth technical details 3. First time I give a keynote 4. Please be gentle, I’m French Catalan with a strong accent ;-)
  3. 3. Is Python too old to be used on production in 2017 ? Python: 1990 (27 years ago) Go: 2009 (8 years ago) Rust: 2010 (7 years ago) COBOL: 1959 (58 years ago) Creation date
  4. 4. Maybe, maybe not… (Dr.) Who knows ? Python: 2017 (This year) Go: 2017 (This year) Rust: 2017 (This year) COBOL: 2014 (3 years ago) Last release
  5. 5. How to select the “right” future proof technology ?
  6. 6. Wait a minute... 1. An environment that provides services are actually used 2. If the production goes down, you will receive calls from users (= Best monitoring tool ever) What is a production environment ?
  7. 7. What do we need to have a production ? 1. Server(s) instance(s) 2. First line support 3. Eventually an On-call 24/7 4. Monitoring Warning: It can be the same person for all roles
  8. 8. Wait a minute... Because: 1. 1. Money (Companies) 2. 2. Help others (Non-profit)Why do we need to maintain a production ?
  9. 9. Wait a minute...
  10. 10. What’s wrong in this presentation ?
  11. 11. Ordering wrong priorities of
  12. 12. Wrong ordering of priorities How to detect the important and urgent points of your project ?
  13. 13. Let’s start from the beginning: You 1. What are your personal priorities, motivations ? Are these priorities compatible to build a project ? a. Happiness b. Help humanity, the nature c. Change the world d. Generate Money e. Make fun of others f. Make others suffer g. Burn cars or chariots
  14. 14. What is your project ? 1. Try to forget technical tools 2. Ideally, observe your future users 3. Try to list technical constraints you have 4. Then, only at this moment, match with your toolbox 5. Avoid over-engineering of your project
  15. 15. What are the important points to build a production ? 1. How to safely upgrade your production ? 2. How to create reproducible tests to detect regressions ? 3. How to safely “upgrade” your users ?
  16. 16. How to safely upgrade your production ? 1. Big bang upgrades are very bad for the stress 2. You must be able to upgrade only part of your production 3. You must be able to rebuild your entire stack quickly 4. Use proper tools: Devops (Ex: Ansible) virtual OS (VMs, containers)
  17. 17. How to create reproducible tests to detect regressions ? 1. Take statistics of what is really used on your production 2. Test first the critical features 3. Be sure that your tests are as close as possible to the reality of your users 4. Prioritize! Impossible to test everything at the first step
  18. 18. How to safely “upgrade” your users ? 1. They use your user interface (GUI) 2. Incompatibility changes 3. Major releases that break the habits (Ex: Microsoft)
  19. 19. Priority: the tasks complicated to change after the release 1. GUI + User eXperience (UX) 2. Hardware specifications 3. APIs signatures 4. Foundations of your product
  20. 20. Humans upgrade process for the complicated points 1.GUI rendering => Newsletter, documentation 2.Hardware specifications => New price list 3.APIs signatures => Retro Compatibility, API versioning 4.And for foundations ?
  21. 21. Foundations => One architect for decision 1. Explore several paths 2. But often, only one path can be taken 3. You must have a person to validate 4. Democracy is the worst decision system for that: You need a technical production solution, not a politic solution
  22. 22. Foundations changes: Scientific approach 1. Analyze your global problem 2. Try to catch differences between the new trends and what you need for your application 3. Work on your architecture: You aren’t Google or Facebook 4. You don’t need to be trendy, you need to be efficient for your problems
  23. 23. Machine upgrade process for the complicated points 1. GUI rendering => Builtin with Web technologies 2. Hardware specifications => End Of Life support 3. APIs signatures => Different HTTP endpoints 4. Foundations => “Be able to change the engine of a flying plane”
  24. 24. Be able to change the engine of a flying plane ? WAT? Think your architecture where each element can be replaced, even Python Have man-in-the-middle with external world to manipulate: HTTP: HAProxy SIP: Kamailio Avoid the big bangs, migrate step by step by endpoint
  25. 25. How to build products ? 1. Quick prototype with Python 2. Validate the business model 3. Catch bugs in your production, not from your dreams 4. Change architecture and/or used libraries 5. #Troll_alert: Write automated tests suite ;-)
  26. 26. Which points are really important ? You will make mistakes, then: 1. Don’t focus on fictive bugs at the beginning: 99% of chance you’ll discover other bigger issues 2. Avoid over-engineering 3. Lazier is better than fancy 4. Be able to change the engine of a flying plane
  27. 27. What else for your architecture ? 1. Keep several implementation patterns in mind: a. Async b. Threads c. Multi-worker 2. Understand if you have IO-bound and/or CPU-bound problems 3. Re-use a maximum lower layers (Virtualization…) 4. Be able to split your problematic in “islands”: Virtual Machines (VM) or containers are cheap
  28. 28. The world becomes more and more async 1. What is async ? 2. The world is natively async 3. Languages are more and more async native: Javascript, Go, Python, C#... 4. WebSockets and HTTP/2 fit async pattern
  29. 29. Why do async in my Python ? 1. Be a time traveler 2. Become (almost) immortal 3. Follow my AsyncIO tutorial in E109/110 tomorrow at 14:00
  30. 30. But… keep a balance: 1. Don’t try to avoid all bugs if upgrades are easy 2. Don’t change too much your existing stack 3. If it works and handles the load, don’t change 4. Bugfixes are the first class citizen 5. Features are the second class citizen
  31. 31. In your context, Is Python production ready ? Or not ?
  32. 32. By the way, what does it mean “production ready” ?
  33. 33. For me, “production ready” means: 1. Almost no maintenance cost 2. Easy to repair (aka. upgrade) 3. Replicate and share the knowledge in several brains 4. At least one person must have a global overview of the architecture
  34. 34. How to select the “right” future proof technology ? 1. 1. No unique answer 2. 2. Several answers are possible 3. 3. Answer change depends when you ask the question (Ex: Heavy vs Web GUI In 90’s and 2010’s) 4. 4. Conclusion: Don’t waste too much your time on that
  35. 35. First conclusion 1. 1. Build things 2. 2. Understand lower layers 3. 3. You make mistakes 4. 4. Try to learn from your mistakes 5. 5. Goto 1.
  36. 36. Spoiler alert 1. 1. You will fail 2. 2. It will take a lot of time to learn and test 3. 3.there is no Royal Road Euclid 4. 3. We have unequal brains
  37. 37. We have unequal brains 1. For the “Tour de France” (Bicycle competition): a. Few will be on the top b. Some will arrive in the last positions c. Most of the population cannot achieve that 2. To compare with Development: a. Few need 2 hours to do a development task b. Some need 2 days for the same task c. Most of the population cannot achieve that
  38. 38. Know your (technical) limits 1. 1. Competence != Confidence a. Clearly two independent variables 2. 2. Use a tool != Build a tool a. Exponential difference 3. 3. Be humble a. a. A lot of people are smarter than you b. b. Conferences are very good to discuss with smarter people
  39. 39. Try to rationalize your real expertise/competence 1. What have you already coded ? 2. What is the complexity ? 3. How many people use your product ? 4. How many times do you need to do the “fireman” ?
  40. 40. Handle your frustrations 1. 1. You have learnt to crawl, to walk and to run in this order 2. 2. You must find what are the crawl/walk/run steps you need to learn a new technology 3. 3. It’s possible you’ll never reach the end of the learning curve The road is long, long, long, long...
  41. 41. Try to match your expectations with the reality When you build your house 1. You see easily what is the progression 2. If you fail in the architecture, gravity will crash your house 3. You can easily find workers When you build your application 1. Very hard to see the progression 2. Not really a good governor to see if you made a “good” or a “bad” architecture 3. Workers are complicated to find And about Dev workers...
  42. 42. Conservative 1. 1. Follow News/Twitter/conferences 2. 2. Take care about new trends 3. 3. Don’t hesitate to trash knowledge to learn new tools 1.Take the leader tool 2.Never reboot a project 3.Don’t care about IT trends Early Adopter
  43. 43. We are developers, we are the kings of the new world 1. Software is everywhere now 2. Tools are more and more complex 3. Less and less people will have the global overview 4. In fact, we are the Gods, more powerful than Illuminatis 5. However...
  44. 44. Look the “system administrator” role 1. In 90’s, they are the kings in companies: They can decide what softwares can be used in a company 2. 20 years later, most companies use cloud solutions: a. Better service continuity b. No ego problems anymore with sysadmins to change things: You pay for a service directly c. With a good devops toolbox, pretty easy to replace a lot of manual tasks originally done by system administrators d. Only few system administrators are becoming devops: Not the same job, not the same skills, more developer oriented
  45. 45. Be humble 1. 1. Because it’s your role = you are paid for that 2. 2. Because you could be easily replaced by: a. a. More conciliant people b. b. More skilled people c. c. On-the-shelf solution d. d. AI ? e. e. Me ;) Help other people in the company to sell your product
  46. 46. What is your goal for this conference ? 1. Be up-to-date about Python ecosystem ? 2. Discover new tools ? 3. Discuss with people who have the same problems ? 4. Show to others you are a “king” because you are in a conference ? Be present != Be competent ;-) 5. Why people are here ?
  47. 47. Is Python still production ready for me ? Definitively Yes
  48. 48. Yes for me, because ? (#1) Now apply “production ready” definition on Python itself: 1. The learning curve is the smallest possible 2. Python has clearly a big community a. Look this event and others 3. A lot of libraries in PyPI are production ready: a. Documented, tested, maintained with love
  49. 49. Yes for me, because ? (#2) 1. Between 1.0 and 3.6, Python has evolved: a. Functional pattern support b. A Reboot with Python 3.0 c. Async pattern support d. Multi-paradigm in general 2. We have a “Benevolent Dictator for Life”: Guido a. For the final decisions in the debates 3. Efficiency
  50. 50. What is efficiency ? 1. In 90’s, machines matter a. CPU cost > Human brain 2. In 3rd Millennium, Human matter a. CPU cost <<<<<< Human brain 3. Prioritize the human cost than the CPU cost 4. However, you must not follow the “fleur au fusil” approach
  51. 51. Benchmark your product 1. Have a staging instance with real data from production 2. Benchmarks to measure maximum capacity 3. Plan future optimizations 4. “Steal” ideas from others a. during a conference for example 5. Open your chakras about biases
  52. 52. The collective unconscious (aka. “Python is slow”) 1. Same patterns as the human prejudice a. Color, gender... 2. We self-maintain this climate by self-flagellation 3.And about benchmarks themselves:
  53. 53. Spoiler alert v2 (Web scraping) http://blog.dataweave.in/post/81760696759/web-scraping-at-scale-python-vs-go
  54. 54. Spoiler alert v3 (SQL queries) https://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python/
  55. 55. What to believe ? 1.Spoiler alert v4: Nobody 2. A benchmark is True and False at the same time: a. True in the context where it was launched b. False because: i. Different of your production needs ii. As everything in life, mistakes can be made 3. Take your own measures on your production
  56. 56. Where can you start ? 1. Read “High Performance Python” 2. Read Victor Stinner’s blog: https://haypo.github.io/ 3. Improve your measures toolbox 4. Be careful about auto-magic efficient new solutions 5. Nevertheless, Python has a big toolbox to handle efficiency problems 6. Measure, benchmark and restart
  57. 57. But, by the way, do you have efficiency problems? 1. Performance is an important governor, BUT nevertheless secondary. 2. You must have first (a lot of) users. 3. Efficiency VS development time priority
  58. 58. Two large time consuming families CPU-bound Parsing Computation Machine learning … I/O bound WebServices Databases
  59. 59. What do Scientific Pythonists use? CPU bound: Cython PyPy Numba Pandas
  60. 60. What do Network Pythonists use? I/O bound with AsyncIO: Uvloop asyncpg ujson ...
  61. 61. And by the way, Python, is it fast or slow? 1. I don’t know 2. On the other hand, Python is now fast enough for us to make money (dev time and CPU) ;-) 3. While keeping an eye on our future growth
  62. 62. Conclusion #2 (Aka. hunting prejudices) https://twitter.com/glyph/status/735390252865622016
  63. 63. By the way, is Python still production ready for you ? 1. If you are at PyParis, we already know the answer. 2. But this keynote goal was also to give: a. Some clues to have better overview for your work b. Some remarks about the microcosm of a conference
  64. 64. Conclusion #3 1. You will make mistakes (architecture, management ...) 2. Do as little as possible 3. Reboot when you spend more time in maintaining than adding features 4. Do not be afraid to change the engine
  65. 65. Don’t forget 1. We all have different priorities and needs 2. Unique response or solution doesn’t exist 3. Be careful about new tools: You will need to maintain during a long time when your project is online. 4. Be tolerant about others solutions… 5. ...while it doesn’t block you to work
  66. 66. Enjoy the conference ! See you soon at: 1. EuroPython (Jul 2017) 2. PyCon-FR (Sep 2017) 3. Python-FOSDEM (Feb 2018) And a lot of other events: https://www.python.org/events
  67. 67. Thank you Questions ? Twitter: @GMLudo
  68. 68. Who I am ? Creator of API-Hour (Daemon framework for AsyncIO) Aiosip (SIP implementation in AsyncIO) Co-maintainer of Panoramisk (Asterisk binding for AsyncIO) Contributor in several AsyncIO libraries like aiohttp I did a lot of benchmarks to try to understand where are the bottlenecks Now, I’m contributor of https://www.techempower.com/benchmarks/ Twitter: @GMLudo

×