Python For Large Company?

1. PythonIn Large Companies? SébastienTandel sebastien.tandel@corp.terra.com.br sebastien.tandel@gmail.com

2. Plan About Terra The 7 steps Prototype Define the Goals Integration Some Libs Prove It Works Evangelize Next Steps Conclusions

3. About Terra : Web Portal Largest Latin American web portal Located in 18 countries 1000s of servers Brazil : ~7M unique visitors / day ~70M pageviews / day

4. About Terra Source: Nielsen NetView (June 2009)

5. About Terra : Email Plaftorm I’m part of the email team. Some stats : +10M mailboxes +30M inbound emails per day +30M outbound emails per day avg : 300 mail/s, peak : 600 mail/s Systems Main systems : SMTP, LMTP, POP, IMAP, Webmail Total of +30 systems to design/develop/maintain Main languages used C / C++

6. About Terra <ul><li>Several “official” languages at Terra :</li></ul>PHP, C, C++, Java, C#, Erlang <ul><li>Average # is one per team!

7. No official “scripting” language (Python, Perl or other)</li></ul>Why? From what I hear Performance Integration with others systems (& legacy) Costs / benefits? Buzzword fear Labor market

8. Flash Python Overview Python is … Interpreted Dynamically Typed Really Concise Multi-paradigm : procedural, OO, functional Exceptions : helpful for robustness, debug (no strace ;)) Garbage Collector : don’t worry about allocation/free

9. Step 1 : Prototype

10. Step 1 : Prototype Buggy system re-written as prototype in Python Surprise! Worked a lot better than its C cousin Prototype is now in production! Spread the word about this rewrite around me Some technical people liked the idea One has not been so enthusiast … my manager Cons: no integration with homemade systems Just one example

11. Step 1 : Prototype Introducing new ideas is a long and though way

12. Step 2 : Define the Goals Performance critical systems : postfix, lmtp, imap / pop

13. Step 2 : Define the Goals <ul><li>Performance critical systems</li></ul>Web-based systems Webmail, ECP

14. Step 2 : Define the Goals <ul><li>Performance critical systems

15. Web-based systems</li></ul>Backend systems : spamreporter, cleaner, clx trainer, base trainer, mfsbuilder, migrador, nnentrega, smigol, …

17. Web-based systems

18. Backend systems</li></ul>Almost inexistent systems (though interesting ones) : Mailboxes stats, logs analysis (stats and user behavior characterization)

21. Backend systems

22. Stats / User behavior characterization, …</li></ul>System / Integration tests scripts

25. Backend systems

26. Stats / User behavior characterization, …

27. System / Integration tests scripts</li></ul>The Grail : <ul><li>Python can be used for ALL except Performance Critical Systems</li></li></ul><li>Step 3 : Integration

28. Step 3 : Integration Python could be used with every systems but how can I interface with the homemade systems (legacy) ? 

29. Step 3 : Integration Various way to create Python Bindings : Python C API: the “hard” way

30. Step 3 : Integration Various way to create Python Bindings : Python C API: the “hard” way swig : the lazy way won’t create a Pythonic API for you

31. Step 3 : Integration Various way to create Python Bindings : Python C API : the “hard” way swig : the lazy way ctypes: the stupidly easy way from ctypes import cdll l = cdll.LoadLibrary(“libc.so.6”) l.mkdir(“python-mkdir-test”)

32. Step 3 : Integration Various way to create Python Bindings : Python C API : the “hard” way swig : the lazy way ctypes : the stupidly easy way Cython : write python, compile with gcc

33. Step 3 : Integration Wrote bindings to interface with all major internal systems (thanks to ctypes) With pythonic API! 

34. Step 3 : Integration from trrauth import TrrAuth auth = TrrAuth(“IMAP”)auth.open_userpass(“standel”, “1q2w3e”, “terra”) auth.attributes = [ “short_name”, “id_perm”, “antispam” ]

35. Step 3 : Integration from trrauth import TrrAuth auth = TrrAuth(“IMAP”)auth.open_userpass(“standel”, “1q2w3e”, “terra”) auth.attributes = [ “short_name”, “id_perm”, “antispam” ] print auth.short_name, “:”, auth.id_perm

36. Step 3 : Integration from trrauth import TrrAuth auth= TrrAuth(“IMAP”)auth.open_userpass(“standel”, “1q2w3e”, “terra”) auth.attributes = [ “short_name”, “id_perm”, “antispam” ] for attr, value in auth: print attr, “:”, value

37. Step 4 : Some Libs

38. Step 4 : Some LibsMaster / Slave Master responsible for : Forking the slaves Reading a “list” of tasks Distribution of the tasks to the slaves Slave responsible for : Execution of the task Return execution status to the master Key characteristics : Slave death detection Handle unhandled exceptions(+ hook) Master <-> slave protocol allows temporary error code Timeout of the tasks

39. Step 4 : Some LibsMaster / Slave One neat characteristic : System might got bug in prod w/ minimal impact If unhandled exception occurs Only one slave dies It is detected and master will fork a new one (if needed) The lib handles the exception : Default behavior : prints to console User defined (callback) : e.g. write the stack trace to a file! Cherry on the cake : getting specific production data about faulty task

40. from robustpools.process_pool import master_task_list from robustpools.process_pool import slave_task_list m_config = { 'INFINITE_LOOP' : 0 } class list_task(object): def __init__(self, list, num, timeout_validity=600): self.__num = num def _id(self): return self.__num id = property(_id) class list_slave(slave_task_list): def __init__(self): super(list_slave, self).__init__(list_task) def run(self, task): print task.id return 0, "ok” list = xrange(10) m = master_task_list(list, num_slave=5, slave_class=list_slave, config=m_config) m.start()

41. Step 4 : Some LibsTCP Sockets Pool Manage connections to a pool of servers send in a round-robin/priority way to each server Detect connection errors Retry to connect Number of retries limited => after mark as dead Retry again later with exponential backoff

42. Step 5 : Prove It Works

43. Step 5 : Prove It Works Prove = collect data … How? Write integrated systems using bindings and libs of previous steps. Show it works  Performance Productivity

44. Step 5 : Prove It Works Performance, one obvious thought : C/C++ PINCS Performance is not C, Stupid!

45. Step 5 : Prove It WorksPerformance Some of the rewrites works faster than C/C++ cousins Why? OS / Systems limits Libs (legacy) Algorithms Software Architecture Infrastructure

46. Step 5 : Prove It WorksProductivity BTW, pure performance so important? Time to Market much more important Adopt Lean Thinking and eliminate every possible waste <ul><li>Writing too much code is a big waste in several ways</li></ul>Loose time when writing Increase # bugs More time to maintain More time to know code base (think to new employees) <ul><li>Impact Overall Productivity</li></li></ul><li>Step 5 : Prove It WorksProductivity http://page.mi.fu-berlin.de/prechelt/Biblio/jccpprt2_advances2003.pdf

47. Step 5 : Prove It WorksProductivity http://page.mi.fu-berlin.de/prechelt/Biblio/jccpprt2_advances2003.pdf

48. Step 5 : Prove It WorksProductivity http://www.ohloh.net

51. Step 5 : Prove It WorksProductivity

52. Step 5 : Prove It WorksProductivity Some existing C/C++ systems re-written in Python Original C/C++ versions total of ~20.000 LOC In Python, 4-6x less code ! The previous numbers do not seem to lie 

53. Step 5 : Prove It WorksProductivity Oh, parsing an email? Any idea in C/C++?

54. Step 5 : Prove It WorksProductivity <ul><li>parsing an email</li></ul>from email import message_from_file fh = open(filename, “r”) mail = message_from_file(fh) fh.close()

55. Step 5 : Prove It WorksProductivity <ul><li>parsing an email</li></ul>content types of parts? Any idea in C/C++ ? from email import message_from_file def get_mail(filename): fh = open(filename, “r”) mail = message_from_file(fh) fh.close() return mail mail = get_mail(filename)

56. Step 5 : Prove It WorksProductivity <ul><li>parsing an email

57. content types of parts</li></ul>from email import message_from_file def get_mail(filename): fh = open(filename, “r”) mail = message_from_file(fh) fh.close() return mail mail = get_mail(filename) for part in mail.walk(): print part.get_content_type()

59. content types of parts

60. getting headers?

61. Any idea in C/C++?</li></ul>from email import message_from_file def get_mail(filename): fh = open(filename, “r”) mail = message_from_file(fh) fh.close() return mail mail = get_mail(filename) for part in mail.walk(): print part.get_content_type()

63. content types of parts

64. getting headers

65. Python libs are just that simple!</li></ul>… and there are a lot! from email import message_from_file def get_mail(filename): fh = open(filename, “r”) mail = message_from_file(fh) fh.close() return mail mail = get_mail(filename) for part in mail.walk(): print part.get_content_type() print mail[“From”] print mail[“Subject”]

66. Step 5 : Prove It WorksPerformance (Again?) For equivalent architecture (libs, algorithm, infrastructure) C is a best performer than Python!  Python Is Not C, Stupid!

67. Step 5 : Prove It WorksPerformance (Again?) Bottleneck discovered! PINCS! : think first to architecture!

68. Step 5 : Prove It WorksPerformance (Again?) Bottleneck discovered! <ul><li>PINCS! : think first to architecture!</li></ul>Ctypes/ Swig : python bindings Write your bottleneck in C / C++, use it in your python app

69. Step 5 : Prove It WorksPerformance (Again?) Bottleneck discovered! <ul><li>PINCS! : think first to architecture!

70. Ctypes : absurdly easy python bindings</li></ul>Cython: write python, obtain a gcc compiled lib

72. Ctypes : absurdly easy python bindings

73. Cython: write python, obtain a gcc compiled lib</li></ul>Psyco: JIT for python Just an additional module import in your code 2 – 100x times faster than normal Python Requires a bit more memory

75. Ctypes : absurdly easy python bindings

76. Cython: write python, obtain a gcc compiled lib

77. Psyco: JIT for python

78. Unladden Swallow : Google Project

79. Produce a version of Python at least 5x faster

80. Every patch goes to Python (no fork!)</li></li></ul><li>Step 6 : Evangelize

81. Step 6 : Evangelize Once having stopped and look at what have been accomplished … Show it, Evangelize!

82. Step 6 : Evangelize Because introducing a “new technology” is not just about teaching something to users. You’ve got to play the role of evangelist! Innovators (3.5%) New stuffs? they’re in!

83. Step 6 : Evangelize Because introducing a “new technology” is not just about teaching something to users. You’ve got to play the role of evangelist! <ul><li>Innovators (3.5%) </li></ul>Early-adopters (12.5%) Open to new ideas but check before

84. Step 6 : Evangelize Because introducing a “new technology” is not just about teaching something to users. You’ve got to play the role of evangelist! <ul><li>Innovators (3.5%)

85. Early-adopters (12.5%)</li></ul>Early majority (35%) First, they must see the idea working

87. Early-adopters (12.5%)

88. Early majority (35%)</li></ul>Late majority (35%) Accept after lot of pressure, or imposed

90. Early-adopters (12.5%)

91. Early majority (35%)

92. Late majority (35%)</li></ul>Laggard (14%) Never accept (why would I want to change?)

93. Step 6 : Evangelize During work, I constantly spoke (a lot) to others Presentation on Python made for all Present to a large audience what has been done Open discussion Poster resuming what has been done Wiki page documenting Python stuffs Specific mailing-list related to Python

94. Step 6 : Evangelize lot of work and slow process but I won some allies Some technical people are convinced that Python is useful Some managers are convinced that Python could be a good thing for Terra Starting evaluation in some specific cases

95. Step 7 : Next Steps

96. Step 7 : Next Steps Proven that Python could be useful in some cases. Don’t forget my Grail! The way has not ended … I’m lobbying to start using Python for web development. And again, I made a prototype

97. Step 7 : Next Steps Django = THE Python MVC web framework : Model : By describing data, no code written (SQLAlchemy) Automatic creation of tables (if needed), Data accessed through objects, No SQL needed! View : access models to get the data render the output through templates <ul><li>loose coupling interface <-> code!</li></ul>Controller : REST through url parsing

98. Step 7 : Next Steps Login : Module auth already exists. Easy to tell django that authentication is required @login_required def list_abook(request, username): … login_requiredis a python decorator

99. Step 7 : Next Steps Caching information (memcache, bd, file, …) 4 levels : Per site : one config line Per view : one python decorator @cache_page(60 * 15) def list_abook(request, username): … In templates : maybe better to let this one out!  Low-level cache access : cache.get(id) cache.set(id, value, timeout)

100. Step 7 : Next Steps Address book Web Service Retrieve address book of one user, Add an account, Add an entry to the address book of a user, View all the address book entries, Output in HTML, JSON and CSV < 100 LOC 2 hours (w/o knowing the framework) Not one line of SQL just usefulcode

101. Conclusions One year and a half … and Evangelization is not done yet! Email Team : Several systems have been written in Python and works really fine … even with the Terra high load! Web project should start right now People are starting using/learning it inside the company Some teams are starting evaluating Python Some Terra employees here at this conference!

102. THANKS! Any Questions?

Python For Large Company?

Sébastien Tandel

Python For Large Company?