Downloaded 32 times
![# cat ~/repo/thrift_example/TARGETS
thrift_library(
name = "myservice_thrift",
languages = ["ruby", "php", "cpp", "python"],
thrift_srcs = {"myservice.thrift": ["Calculator"]},
)
python_binary(
name = "myservice_server",
main_module = "thrift_example.server",
srcs = ["server.py"],
deps = [":myservice_thrift-py"],
)
python_binary(
name = "myservice_client",
main_module = "thrift_example.client",
srcs = ["client.py"],
deps = [":myservice_thrift-py"],
)](https://image.slidesharecdn.com/pycon2014-ireland-angelofailla-public-141013051322-conversion-gate02/85/Python-at-Facebook-15-320.jpg)
![pallotron@dev:~/repo $ fbconfig thrift_example/
Configuring 20 targets
Writing 208/208 rules to makefile [100%]
Done
pallotron@dev:~/repo $ fbmake opt
thrift/compiler/thriftl.ll
common/memory/JEMalloc.cpp
[…]
Linking _build/opt/thrift/compiler/thrift...
_build/opt/thrift/compiler/thrift
thrift_example/myservice.thrift
_build/opt/thrift_example/myservice_thrift-Calculator-pyremote.lpar
_build/opt/thrift_example/myservice_client.lpar
_build/opt/thrift_example/myservice_server.lpar
_build/opt/thrift_example/myservice_server.par
_build/opt/thrift_example/myservice_thrift-Calculator-pyremote.par
_build/opt/thrift_example/myservice_client.par](https://image.slidesharecdn.com/pycon2014-ireland-angelofailla-public-141013051322-conversion-gate02/85/Python-at-Facebook-16-320.jpg)
![1: def consume():
2:
3: handle = dict()
4: ctc = ConfigeratorThriftClient()
5: fname = "example/config"
6:
7: def on_update(name):
8: handle['config'] = ctc.getJSONConfigContent(name)
9:
10: ctc.startMonitoringConfig(fname)
11: ctc.setUpdateCallback(on_update)
12: on_update(fname)
13:
14: return handle](https://image.slidesharecdn.com/pycon2014-ireland-angelofailla-public-141013051322-conversion-gate02/85/Python-at-Facebook-21-320.jpg)
Uploaded byAngelo Failla
Python at Facebook
The document outlines how Facebook utilizes Python, emphasizing its central repository for back-end systems and its significance as the third most-used programming language. It details the implementation of Thrift for service definition, the structure of .par files, and the processes for building and running Python-based client-server applications. Additionally, it discusses configuration management, code review practices, and an automation project called 'Keanu' to test system capacity under real-world conditions.
More Related Content
Python at Facebook
- 2. Python @ Facebook Angelo Failla Production Engineer @ Facebook Dublin October 11th, 2014 -- PyCon Ireland 2014
- 3. 1 How dowe use Python at Facebook? 2 Description of a production system based on Python 3 Questions Agenda
- 4. How do weuse Python at Facebook?
- 6. • Centralized repofor back-end systems • A LOT of code. • All Python code hosted there • 3rd most used language (PHP/Hack => C++ => Python) • Still a LOT of code :P
- 7. ONE BUILD TOOL TO RULE THEM ALL Idreamlikecrazy@ https://www.flickr.com/photos/purple-lover/13583362554 (cc 2.0 license)
- 8. Everything is builtfrom HEAD C++ binaries are statically linked http://cdn.instapop.com/assets/memes/Skeptical%20Baby/4302/thumb.jpeg?1383838705
- 9. How about thesame in Python? We use .par files PAR file Short shell script header Zipped content __main__.py .py, .pyc files .so and other files needed by the application
- 10. Thrift (https://thrift.apache.org/) ina nutshell Write myservice.thrift Thrift compiler generates language bindings Client/Server app imports generated code
- 11. # cat ~/repo/thrift_example/myservice.thrift namespace py thrift_example.myservice service Calculator { i32 add(1: i32 num1, 2: i32 num2) }
- 12. $ thrift --genpy myservice.thrift $ tree gen-py/ gen-py/ ├── __init__.py └── myservice ├── Calculator.py ├── Calculator-remote ├── constants.py ├── __init__.py └── ttypes.py
- 13. # cat ~/fbcode/thrift_example/server.py # import lines removed, see: http://tinyurl.com/p6a7cuw 1: class CalculatorHandler(Calculator.Iface): 2: def add(self, num1, num2): 3: return num1 + num2 4: 5: handler = CalculatorHandler() 6: socket = TSocket.TServerSocket(port=9090) 7: tfactory = TTransport.TBufferedTransportFactory() 8: pfactory = TBinaryProtocol.TBinaryProtocolFactory() 9: server = TServer.TSimpleServer( 10: handler, socket, tfactory, pfactory) 11: server.serve()
- 14. # cat ~/fbcode/thrift_example/client.py # import lines removed, see: http://tinyurl.com/p6zz9ex 1: socket = TSocket.TSocket('localhost', 9090) 2: transport = TTransport.TBufferedTransport(socket) 3: protocol = TBinaryProtocol.TBinaryProtocol(transport) 4: client = Calculator.Client(protocol) 5: transport.open() 6: 7: sum = client.add(1, 2) 8: print(‘1 + 2 = %d’ % sum) 9: 10: transport.close()
- 15. # cat ~/repo/thrift_example/TARGETS thrift_library( name = "myservice_thrift", languages = ["ruby", "php", "cpp", "python"], thrift_srcs = {"myservice.thrift": ["Calculator"]}, ) python_binary( name = "myservice_server", main_module = "thrift_example.server", srcs = ["server.py"], deps = [":myservice_thrift-py"], ) python_binary( name = "myservice_client", main_module = "thrift_example.client", srcs = ["client.py"], deps = [":myservice_thrift-py"], )
- 16. pallotron@dev:~/repo $ fbconfigthrift_example/ Configuring 20 targets Writing 208/208 rules to makefile [100%] Done pallotron@dev:~/repo $ fbmake opt thrift/compiler/thriftl.ll common/memory/JEMalloc.cpp […] Linking _build/opt/thrift/compiler/thrift... _build/opt/thrift/compiler/thrift thrift_example/myservice.thrift _build/opt/thrift_example/myservice_thrift-Calculator-pyremote.lpar _build/opt/thrift_example/myservice_client.lpar _build/opt/thrift_example/myservice_server.lpar _build/opt/thrift_example/myservice_server.par _build/opt/thrift_example/myservice_thrift-Calculator-pyremote.par _build/opt/thrift_example/myservice_client.par
- 18. Python as aDSL
- 19.
- 20. JSON Review andCommit Git Repo MAGIC! Proxy disk Application Thrift schema: struct Job { 1: string name, 2: Scheduling scheduling, } struct Scheduling { 1: int priority, } Python config: importThrift(file) myConfig = Job( name = “ajobname”, scheduling = Scheduling( priority = 3) ) export(myConfig) Python validator: def validate_job(job): ensureNotEmpty(job.name) ensurePositive( job.scheduling.priority) addValidator(myConfig, validate_job) Distribution Client lib Composition
- 21. 1: def consume(): 2: 3: handle = dict() 4: ctc = ConfigeratorThriftClient() 5: fname = "example/config" 6: 7: def on_update(name): 8: handle['config'] = ctc.getJSONConfigContent(name) 9: 10: ctc.startMonitoringConfig(fname) 11: ctc.setUpdateCallback(on_update) 12: on_update(fname) 13: 14: return handle
- 22. Benefits of using configerator
- 23. Code review onconfiguration changes Automated validation http://www.jasonawesome.com/wp-content/uploads/2010/06/sally-code-review-300x256.png http://sounddesignlive.com/wp-content/uploads/2013/08/sound-design-live-automate_ all_the_things.jpeg
- 24. Clear representation ofcomplex config http://www.visualthinkingmagic.com/wp-content/uploads/2011/08/complex-simple.gif No service restart required to pick up changes http://social.microsoft.com/Forums/getfile/25914/
- 25.
- 26. • Code *is*peer reviewed • We use Phabricator • Originally developed @ FB by Evan Priestley • Go check www.phabricator.org • Supports git/svn/hg
- 27. • (Some) PEPs enforced and push blocking • Common code dir for accelerated development • Unit tests results are clearly visible in our code review tool http://memegenerator.net/Bill-Lumbergh-Office-Space
- 28. Suggestions for reviewers: • Reviewing code is not writing code • Nitpicks are not enough • Stop typing and have a conversation • Build trust • Find people to emulate
- 29. Making diffs easierto be reviewed: • One and only one thesis • Do No Harm • Mix and match reviewers • Write good test plans • You are not your code
- 30. Production Python at Facebook
- 31. PROJECT KEANU •Automated load tests of regions/clusters/hosts • Implements feedback control loop • No simulated traffic • Goal is to find out “breaking point” == real capacity • Runs daily • Peek time • When no incidents • Skips drained clusters
- 32.
- 33.
Editor's Notes
- #21 The three basic components of Configerator are: Configuration composition: it’s the part of the framework that deals with how configuration "source" is composed. Configuration consists of a flat file, which is code reviewed and checked into Git. The source file is validated and compiled into a JSON file, which is also code reviewed and checked into Git. Configuration distribution: A system to detect config file updates in Git, and promptly make them available to all interested clients (usually in approx. 10 seconds for average-sized configs). Uses local disk for fallback if upstream system is unavailable. A Client library: provides a mean to access configs and get notification on configuration updates.
- #22 Configerator provides a library to subscribe and consume configuration (and it supports our main languages, PHP, C++, Python and Go soon). In this picture you see a snippet of Python code that use that library (the ConfigeratorThriftClient class)
- #24 Code review of config changes Configuration changes are subtle and can break things in a very bad way, core reviewing config changes helps. Both source and compiled config files are code-reviewed in Git, changes/history can be much easier to visualize and understand. Automated validation You can import a common library that can be used to make simple tests like validating that a variable is a positive integer, or that some string matches a given regular expression, but you can also write your own complex business logic (people in the Traffic team use configerator to configure internal/external VIPs and drain/undrain traffic on clusters).
- #25 Abstract complexity Using Python as a DSL enables you to abstract away complexity. You only need to know the dial of knobs of an application to configure it, all the nitty gritty complex details are hidden from non-eng or Engs not familiar with the system. Configs can be composed and included in each other, and can be structured hierarchically. Config hierarchies may be independent of the hierarchy of the service that uses it. Client fallback If the configerator infrastructure is down configuration files are still being served (as files are stored in the machine’s file system)
- #29 Code reviews are really important part of our engineering process. However it can become onerous with certain attitudes and disposition by code reviewers. If approached with the wrong attitude it can become counterproductive, demoralize and slow down things unnecessarily. The goal, reviewing code is not writing code Good reviewers should keep in mind that the goal of a reviewers is not to make it so that the code you review looks as if you wrote it. The goal is to agreed on overall direction and ensure certain quality bars are hit. Suggestions are fine, common style guidelines are fine. Nit are not enough If you review comments only include nitpicks or minor style issues that is not a sufficient bar to request changes. Caring about nits is important, you should care, but you shouldn’t allow it to unnecessarily slow down things. You can make those comments still accepting the diff. Trust that your colleague are respectful of the code base and that they will do the right thing. If you don’t trust your colleagues than you and your team should work on that. Stop typing and have a conversation You should avoid long winded discussion on diffs whenever possible. If you are not reaching convergence quickly in a review don’t spent hours debating on comments, just go and talk to the author of the diff Build trust Often the root cause of code review issues is the a lack of trust. If that’s the case no matter how good the code is, you’ll have hard time reviewing it. Especially true when changing code that someone else owns. You can assume people trust reviewers to be good engineers and make sane decisions but you cannot assume they trust reviewers to have all the knowledge they have when making those decisions. So you should build trust and the best way to do is is to talk and be ready to listen and be open to change. If you think the discussion is going to be controversial you should go to the discussion assuming your are wrong That doesn’t mean you should not defend your position, they are important but not more important than others. Find people to emulate. The best way to learn how to review code effectively is to find people who do a good job and emulate and learn from them. It’s a soft skill that is developed with time, good reviewers always make suggestions respectfully, they focus on the more salient and important part of a diff, and never slow people down.
- #30 Making diffs easier to be reviewed: One and Only one thesis: Each diff you send out should endeavor to do only one thing. Do No Harm: your code should follow the Hippocratic Oath: first and foremost do no harm, each diff you send should be a net improvement to the codebase (adding more useful code or remove unused code, or refactoring old code) It is a good reason to mix and match the reviewers to whom you send code. That’s because teammate have a high likelihood to have a direct interest in getting your code shipped. On the other hands non-teammates may have better knowledge of the right way to write the code, but have less of an idea about your end goal, hence why it is always a good idea to mix and match. Write a good test plan You are not your code: code reviews are better when both parties go into the process realizing that only the integrity of the code is at stake, not the one of the coder or reviewer