Imagine you’re a new engineer at a workplace who has to learn a new unfamiliar codebase. After you acquire a copy of the repo, what is your next step? How do you dissect a new unfamiliar codebase to understand its inner workings? Come see a guided walkthrough of reading the widely used python-requests project, which gets over 18,000 downloads per day and powers many of the world’s REST-based APIs.
This was a talk given at PyCon 2016.
Let's read code: the python-requests librarySusan Tan
- The document discusses reading through the python-requests codebase by cloning the requests repository, setting up a local development environment, and examining unit tests and code snippets.
- It demonstrates using the requests.get method in a unit test, looking at the HTTPDigestAuth class definition, and using the httpbin fixture and httpbin.org site to test authentication.
- Key points are that httpbin is used extensively in unit tests to make requests, sessions persist request parameters and use connection pooling, and the requests documentation provides helpful references.
The document discusses HTTP requests and responses. It explains that a request contains a start line with the method, URL and HTTP version, followed by headers providing additional information, and an optional message body. A response contains a status line with the protocol version and status code, followed by headers including caching information, and an optional message body. Content negotiation headers like Accept and Content-Type are used to select the appropriate representation format.
Reliable Python REST API (by Volodymyr Hotsyk) - Web Back-End Tech Hangout - ...Innovecs
On Saturday, 12 of April, regular quarterly meeting of Tech Hangout Community took place in Creative Space 12, the cultural and educational center based in Kiev! The event was held under the motto «One day of inspiring talks on Web Back-End». This time Python, Ruby and PHP developers gathered to make peace and learn the Force.
*TECH HANGOUT COMMUNITY was found in 2012 by the developers for the developers for knowledge and experience sharing. Such meetings are the part of Innovecs Educational Project that actively develops sphere of internal trainings and knowledge exchange program among professionals. This Initiative was born within the walls of Innovecs and has proved to be extremely popular and high-demand. In a short period of time it gained its own Facebook group with more than 90 members, blog with more than 40 posts and constant quarterly external meeting of Tech hangout community with more than 80 participants. The concept of the event proposes a 30-minute report on the topic previously defined, and the discussion in a roundtable session format.
Join to discuss - https://www.facebook.com/groups/techhangout/
The document discusses developing REST APIs with Python and Django Rest Framework (DRF). It explains the basics of REST, why it is used, and how to build a REST API with DRF including serializers, views, URLs, permissions, versioning, documentation, and testing. DRF allows building web APIs with Django that are highly configurable and have little boilerplate code. It also supports non-ORM data sources.
The document discusses how RSpec achieves "zero monkey patching mode" in version 3.0. It describes how RSpec previously extensively used monkey patching but has worked to reduce this in recent versions. It details how RSpec loads spec files, builds example groups and hierarchies, runs examples, and makes expectations without monkey patching core objects or classes. Key aspects covered include the rspec executable, loading files, building example groups, running examples, and the expect-based syntax.
A few general pointers for Perl programmers starting out to write tests using Perl6. This describes a few of the differences in handling arrays vs. hashes, comparing objects, flattening, and value vs. immutable object contents.
The $path to knowledge: What little it take to unit-test Perl.Workhorse Computing
Metadata-driven lazyness, Perl, and Jenkins provide a nice mix for automated testing. With Perl the only thing required to start testing is a files path, from there the possibilities are endless. Using Symbol's qualify_to_ref makes it easy to validate @EXPORT & @EXPORT_OK, knowing the path makes it easy to use "perl -wc" to get diagnostics.
The beautiful thing is all of it can be lazy... er, "automated". And repeatable. And simple.
Let's read code: the python-requests librarySusan Tan
- The document discusses reading through the python-requests codebase by cloning the requests repository, setting up a local development environment, and examining unit tests and code snippets.
- It demonstrates using the requests.get method in a unit test, looking at the HTTPDigestAuth class definition, and using the httpbin fixture and httpbin.org site to test authentication.
- Key points are that httpbin is used extensively in unit tests to make requests, sessions persist request parameters and use connection pooling, and the requests documentation provides helpful references.
The document discusses HTTP requests and responses. It explains that a request contains a start line with the method, URL and HTTP version, followed by headers providing additional information, and an optional message body. A response contains a status line with the protocol version and status code, followed by headers including caching information, and an optional message body. Content negotiation headers like Accept and Content-Type are used to select the appropriate representation format.
Reliable Python REST API (by Volodymyr Hotsyk) - Web Back-End Tech Hangout - ...Innovecs
On Saturday, 12 of April, regular quarterly meeting of Tech Hangout Community took place in Creative Space 12, the cultural and educational center based in Kiev! The event was held under the motto «One day of inspiring talks on Web Back-End». This time Python, Ruby and PHP developers gathered to make peace and learn the Force.
*TECH HANGOUT COMMUNITY was found in 2012 by the developers for the developers for knowledge and experience sharing. Such meetings are the part of Innovecs Educational Project that actively develops sphere of internal trainings and knowledge exchange program among professionals. This Initiative was born within the walls of Innovecs and has proved to be extremely popular and high-demand. In a short period of time it gained its own Facebook group with more than 90 members, blog with more than 40 posts and constant quarterly external meeting of Tech hangout community with more than 80 participants. The concept of the event proposes a 30-minute report on the topic previously defined, and the discussion in a roundtable session format.
Join to discuss - https://www.facebook.com/groups/techhangout/
The document discusses developing REST APIs with Python and Django Rest Framework (DRF). It explains the basics of REST, why it is used, and how to build a REST API with DRF including serializers, views, URLs, permissions, versioning, documentation, and testing. DRF allows building web APIs with Django that are highly configurable and have little boilerplate code. It also supports non-ORM data sources.
The document discusses how RSpec achieves "zero monkey patching mode" in version 3.0. It describes how RSpec previously extensively used monkey patching but has worked to reduce this in recent versions. It details how RSpec loads spec files, builds example groups and hierarchies, runs examples, and makes expectations without monkey patching core objects or classes. Key aspects covered include the rspec executable, loading files, building example groups, running examples, and the expect-based syntax.
A few general pointers for Perl programmers starting out to write tests using Perl6. This describes a few of the differences in handling arrays vs. hashes, comparing objects, flattening, and value vs. immutable object contents.
The $path to knowledge: What little it take to unit-test Perl.Workhorse Computing
Metadata-driven lazyness, Perl, and Jenkins provide a nice mix for automated testing. With Perl the only thing required to start testing is a files path, from there the possibilities are endless. Using Symbol's qualify_to_ref makes it easy to validate @EXPORT & @EXPORT_OK, knowing the path makes it easy to use "perl -wc" to get diagnostics.
The beautiful thing is all of it can be lazy... er, "automated". And repeatable. And simple.
Variable interpolation is a standard way to BASH your head. This talk looks at interpolation, eval, ${} handling and "set -vx" to debug basic variable handling.
This document discusses an approach to data-driven testing of objects using Object::Exercise. Key points include:
- Object::Exercise standardizes repetitive test code into a reusable framework.
- Tests are defined declaratively as data, replacing hardwired test code. This avoids issues like testing the tests.
- Tests can validate return values, check for exceptions, and perform more complex checks using regular expressions and code references.
- The approach also allows overriding parts of an object under test by replacing methods that initialize test data. This enables flexible testing of different configurations.
Object Trampoline: Why having not the object you want is what you need.Workhorse Computing
Overview of Trampoline Objects in Perl with examples for lazy construction, lazy module use, added sanity checks. This version includes corrections from the original presented at OSCON 2013 and comments.
The document discusses static code analysis tools and their ability to find security vulnerabilities. It notes that while tools can find some issues, they lack context and have many false positives. Advanced static analysis requires modeling the program flow and variable states, but fully capturing semantics is difficult. Overall, static tools are better for finding simple issues but struggle with deeper design flaws. Manual review is still needed to find many vulnerabilities.
This document discusses automated testing of Puppet code. It describes using RSpec-puppet for unit testing Puppet manifests and ensuring resources are properly configured. It also discusses performing "after testing" by executing tests in the user environment to check the final configuration and catch issues not found by Puppet reports. Post-provision checks (PPC) is presented as a framework for writing these types of integration tests and automating their execution after Puppet runs via tools like TeamCity.
Face it, backticks are a pain. BASH $() construct provides a simpler, more effective approach. This talk uses examples from automating git branches and command line processing with getopt(1) to show how $() works in shell scripts.
perl often doesn't get updated because people don't have a way to know if their current code works with the new one. The problem is that they lack unit tests. This talk describes how simple it is to generate unit tests with Perl and shell, use them to automate solving problems like missing modules, and test a complete code base.
The document discusses smoke testing, which are simple preliminary tests done after a software deployment to check for failures. It recommends conducting smoke tests that are fast, test key URLs and parameters, and validate status codes, response times, and that a response body is returned. The document then provides details on implementing concurrent smoke tests in PHP using a library that runs tests concurrently and collects results. It also outlines the server architecture used to route requests and deploy code, with smoke tests running before new code is switched to live.
This document discusses various techniques for writing doctests, including:
- Summarizing how doctests allow testing functions through documentation strings.
- Handling unpredictable output like object IDs using options like ELLIPSIS.
- Working around whitespace issues using tags like <BLANKLINE>.
- Specifying test locations within modules or external files.
- Documenting functions in external files like .rst files.
- Running tests on modules, files, or via a unittest suite.
- Isolating test globals and contexts to avoid interference across tests.
The document discusses monitoring and analyzing memory usage in Raku processes. It describes using the getrusage(2) system call to retrieve resident set size (RSS) and other memory statistics for a process. It then presents the ProcStats module, which allows periodically sampling getrusage(2) data over time to track RSS and detect changes from an initial baseline. The module outputs differences in memory statistics compared to the first sample, ignoring unchanged values. This provides a concise way to monitor for increases in a process's memory footprint over time.
Plack is an interface for web request handlers that simplifies the interface and makes code more portable. It allows developers to focus on request handling rather than API specifics. Plack addresses issues with traditional CGI and mod_perl approaches by running handlers outside of servers in a standardized way. This encapsulation improves performance, debugging, and code reuse across different server implementations. Plack includes modules for common tasks like routing, middleware, and running PSGI applications on various web servers.
Short introduction to the basics of Perl testing and some resources for further reading. Includes basics of Test::Simple, Test::More, using Inline for multi-language testing, testing coverage, testing tests.
Introducing the Eve REST API Framework.
FOSDEM 2014, Brussels
PyCon Sweden 2014, Stockholm
PyCon Italy 2014, Florence
Python Meetup, Helsinki
EuroPython 2014, Berlin
Três conceitos que farão a diferença nos seus appsGuilherme Rambo
Nesta talk Guilherme irá apresentar três padrões que adotou e que mudaram significativamente a qualidade dos seus apps: Abstrair a internet dos seus view controllers; calcular diferenças entre coleções de models e lidar com navegação entre telas.
Performance benchmarks are all too often inaccurate. This talk introduces some things to look for in setting up and running benchmarks to make them effective.
This document introduces Flask, a Python microframework for building web applications and APIs. It shows how to create a basic "Hello World" Flask app in 3 steps: creating the Flask object, defining a route, and returning a JSON response. It then demonstrates various response types like simple text, generators, and files. It covers routing concepts like dynamic routes, default values, and REST verbs. More advanced topics discussed include static file serving, database connections, exception handling, and using blueprints to organize routes.
This document introduces the Flask micro web framework. It discusses that Flask provides URL routing, request and response objects, template engines and other features for web development. Flask is simple and extensible, using Werkzeug and Jinja2. It does not include an ORM or form validation, but supports extensions. The document provides examples of basic routing, using request objects, templates and the development server. It also discusses using SQLAlchemy, WTForms and common patterns like MVC with Flask projects.
Building a Perl5 smoketest environment in Docker using CPAN::Reporter::Smoker. Includes an overview of "smoke testing", shell commands to contstruct a hybrid environment with underlying O/S image and data volumes for /opt, /var/lib/CPAN. This allows maintaining the Perly smoke environemnt without having to rebuild it.
Curious about how to take that first step from CCNP to Programmer? Hey guess what, you don't even have to make an entire career out of programming, you can use it to AUGMENT your existing job responsibilities and make your life easier. This deck won't teach you everything you need to know, but it should help making that first step a little bit easier.
The SQL OFFSET keyword is evil. It basically behaves like SLEEP in other programming langauges: the bigger the number, the slower the execution.
Fetching results in a page-by-page fashion in SQL doesn't require OFFSET at all but an even simpler SQL clause. Besides being faster, you don't have to cope with drifting results if new data is inserted between two page fetches.
Variable interpolation is a standard way to BASH your head. This talk looks at interpolation, eval, ${} handling and "set -vx" to debug basic variable handling.
This document discusses an approach to data-driven testing of objects using Object::Exercise. Key points include:
- Object::Exercise standardizes repetitive test code into a reusable framework.
- Tests are defined declaratively as data, replacing hardwired test code. This avoids issues like testing the tests.
- Tests can validate return values, check for exceptions, and perform more complex checks using regular expressions and code references.
- The approach also allows overriding parts of an object under test by replacing methods that initialize test data. This enables flexible testing of different configurations.
Object Trampoline: Why having not the object you want is what you need.Workhorse Computing
Overview of Trampoline Objects in Perl with examples for lazy construction, lazy module use, added sanity checks. This version includes corrections from the original presented at OSCON 2013 and comments.
The document discusses static code analysis tools and their ability to find security vulnerabilities. It notes that while tools can find some issues, they lack context and have many false positives. Advanced static analysis requires modeling the program flow and variable states, but fully capturing semantics is difficult. Overall, static tools are better for finding simple issues but struggle with deeper design flaws. Manual review is still needed to find many vulnerabilities.
This document discusses automated testing of Puppet code. It describes using RSpec-puppet for unit testing Puppet manifests and ensuring resources are properly configured. It also discusses performing "after testing" by executing tests in the user environment to check the final configuration and catch issues not found by Puppet reports. Post-provision checks (PPC) is presented as a framework for writing these types of integration tests and automating their execution after Puppet runs via tools like TeamCity.
Face it, backticks are a pain. BASH $() construct provides a simpler, more effective approach. This talk uses examples from automating git branches and command line processing with getopt(1) to show how $() works in shell scripts.
perl often doesn't get updated because people don't have a way to know if their current code works with the new one. The problem is that they lack unit tests. This talk describes how simple it is to generate unit tests with Perl and shell, use them to automate solving problems like missing modules, and test a complete code base.
The document discusses smoke testing, which are simple preliminary tests done after a software deployment to check for failures. It recommends conducting smoke tests that are fast, test key URLs and parameters, and validate status codes, response times, and that a response body is returned. The document then provides details on implementing concurrent smoke tests in PHP using a library that runs tests concurrently and collects results. It also outlines the server architecture used to route requests and deploy code, with smoke tests running before new code is switched to live.
This document discusses various techniques for writing doctests, including:
- Summarizing how doctests allow testing functions through documentation strings.
- Handling unpredictable output like object IDs using options like ELLIPSIS.
- Working around whitespace issues using tags like <BLANKLINE>.
- Specifying test locations within modules or external files.
- Documenting functions in external files like .rst files.
- Running tests on modules, files, or via a unittest suite.
- Isolating test globals and contexts to avoid interference across tests.
The document discusses monitoring and analyzing memory usage in Raku processes. It describes using the getrusage(2) system call to retrieve resident set size (RSS) and other memory statistics for a process. It then presents the ProcStats module, which allows periodically sampling getrusage(2) data over time to track RSS and detect changes from an initial baseline. The module outputs differences in memory statistics compared to the first sample, ignoring unchanged values. This provides a concise way to monitor for increases in a process's memory footprint over time.
Plack is an interface for web request handlers that simplifies the interface and makes code more portable. It allows developers to focus on request handling rather than API specifics. Plack addresses issues with traditional CGI and mod_perl approaches by running handlers outside of servers in a standardized way. This encapsulation improves performance, debugging, and code reuse across different server implementations. Plack includes modules for common tasks like routing, middleware, and running PSGI applications on various web servers.
Short introduction to the basics of Perl testing and some resources for further reading. Includes basics of Test::Simple, Test::More, using Inline for multi-language testing, testing coverage, testing tests.
Introducing the Eve REST API Framework.
FOSDEM 2014, Brussels
PyCon Sweden 2014, Stockholm
PyCon Italy 2014, Florence
Python Meetup, Helsinki
EuroPython 2014, Berlin
Três conceitos que farão a diferença nos seus appsGuilherme Rambo
Nesta talk Guilherme irá apresentar três padrões que adotou e que mudaram significativamente a qualidade dos seus apps: Abstrair a internet dos seus view controllers; calcular diferenças entre coleções de models e lidar com navegação entre telas.
Performance benchmarks are all too often inaccurate. This talk introduces some things to look for in setting up and running benchmarks to make them effective.
This document introduces Flask, a Python microframework for building web applications and APIs. It shows how to create a basic "Hello World" Flask app in 3 steps: creating the Flask object, defining a route, and returning a JSON response. It then demonstrates various response types like simple text, generators, and files. It covers routing concepts like dynamic routes, default values, and REST verbs. More advanced topics discussed include static file serving, database connections, exception handling, and using blueprints to organize routes.
This document introduces the Flask micro web framework. It discusses that Flask provides URL routing, request and response objects, template engines and other features for web development. Flask is simple and extensible, using Werkzeug and Jinja2. It does not include an ORM or form validation, but supports extensions. The document provides examples of basic routing, using request objects, templates and the development server. It also discusses using SQLAlchemy, WTForms and common patterns like MVC with Flask projects.
Building a Perl5 smoketest environment in Docker using CPAN::Reporter::Smoker. Includes an overview of "smoke testing", shell commands to contstruct a hybrid environment with underlying O/S image and data volumes for /opt, /var/lib/CPAN. This allows maintaining the Perly smoke environemnt without having to rebuild it.
Curious about how to take that first step from CCNP to Programmer? Hey guess what, you don't even have to make an entire career out of programming, you can use it to AUGMENT your existing job responsibilities and make your life easier. This deck won't teach you everything you need to know, but it should help making that first step a little bit easier.
The SQL OFFSET keyword is evil. It basically behaves like SLEEP in other programming langauges: the bigger the number, the slower the execution.
Fetching results in a page-by-page fashion in SQL doesn't require OFFSET at all but an even simpler SQL clause. Besides being faster, you don't have to cope with drifting results if new data is inserted between two page fetches.
The document describes a network testing course that teaches how to develop custom network tools and scripts using Python, TCL, Expect and Scapy. The course contains 30 video tutorials totaling over 85 minutes, 3 ebooks and covers topics like packet crafting, protocol headers, network monitoring tools, network automation with TCL/Expect, and building troubleshooting tools with Python and Scapy. The goal is to help network engineers, administrators and testers automate networking activities and develop custom testing tools.
Device Programmability with Cisco Plug-n-Play SolutionCisco DevNet
Cisco Open Plug-n-Play solution allows customers to reduce the costs associated with deployment/installation of network devices, increase the speed and reduce the complexity of deployments without compromising the security. Using Cisco Plug-n-Play solution, customers can do Zero Touch Installs of Cisco gear in various deployment scenarios and deployment locations.
Watch the DevNet 2052 replay from the Cisco Live On-Demand Library at: https://www.ciscolive.com/online/connect/sessionDetail.ww?SESSION_ID=91108&backBtn=true
Check out more and register for Cisco DevNet: http://ow.ly/jCNV3030OfS
NetDevOps for the Network Dude: How to get started with API's, Ansible and Py...Cisco DevNet
This document provides an agenda and overview for a presentation on network automation using APIs, Ansible, and Python. The presentation introduces network programmability and automation tools like Ansible, discusses using infrastructure as code approaches, and provides examples of automating network device configurations and modules using Python and Jinja templates. It aims to help network engineers get started with network automation.
Open Device Programmability: Hands-on Intro to RESTCONF (and a bit of NETCONF)Cisco DevNet
In this small group, hands-on workshop session you'll learn how to write your first Python application that uses YANG, NETCONF and , RESTCONF to access operational and configuration data on a device.
Watch the DevNet 2044 replay from the Cisco Live On-Demand Library at: https://www.ciscolive.com/online/connect/sessionDetail.ww?SESSION_ID=92725&backBtn=true
Check out more and register for Cisco DevNet: http://ow.ly/jCNV3030OfS
Vikash Agrawal gave a presentation on Python to the Google Technology Users Group in Manipal on October 31, 2011. The presentation covered the history of Python, its key features including indentation-based syntax and lack of delimiters, popular uses of Python at companies like Google, YouTube and NASA, and differences from languages like C++ and Java such as no pointers and inclusion of a garbage collector. It also discussed important versions of Python like 2.6, 2.7 and 3.0 and demonstrated basic coding in Python. The presentation concluded with a question and answer session.
The document discusses FakeWeb, a Ruby gem that allows stubbing and mocking HTTP requests in tests without making actual requests. It provides examples of registering fake responses for URIs with different bodies, status codes, headers, and number of requests. FakeWeb makes it possible to test code that makes web requests without needing a live connection.
This document summarizes and compares Ruby HTTP client libraries. It discusses the sync and async APIs of 16 libraries including Net::HTTP, HTTPClient, and Faraday. It covers their compatibility, supported features like keep-alive connections, and performance based on benchmarks. The document recommends libraries based on priorities like speed, HTML handling, API clients, and SSL support. It encourages readers to check the detailed feature matrix and report any errors found.
This document provides an overview of quality assurance for PHP projects. It discusses the importance of revision control, documentation, testing, and automation in QA. Revision control systems like SVN and Git are recommended for team development and tracking code versions. PHP Lint is introduced for syntax checking files from the command line. Documenting code with PHPDoc is suggested. Unit testing forms and models is demonstrated using PHPUnit. Validation, filtering, and sanitizing user input is important for protection against exploits. Overall the document promotes establishing strong quality practices like testing, revision control, and documentation for PHP projects.
This document discusses tools for testing web services over HTTP in Python. It introduces HTTPie, a command line tool for making HTTP requests, and Behave, a behavior-driven development tool that uses the Gherkin language to write human-readable test cases. The document provides examples of using HTTPie to debug services and Behave steps to test authentication on a sample API.
This document discusses quality assurance (QA) for PHP projects. It introduces various QA tools and techniques including syntax checking, documentation, testing, version control and code coverage. Screenshots are provided to illustrate concepts like detecting bugs early, observing behavior and preventing mistakes. The document also includes exercises for attendees to practice setting up version control with Git, running syntax checks with PHP Lint, generating documentation with phpDocumentor, and testing models with PHPUnit.
This workshop is a hands-on training where a real Zend Framework application is used as an example to start improving QA using tools to test, document and perform software metric calculations to indicate where the software can be improved. I also explain the reports produced by a CI system.
Everyone talks about raising the bar on quality of code, but it's always hard to start implementing it when you have no clue where to start. With this talk I'm shooing that there are many levels developers can improve themselves by using the right tools. In this talk I'll go over each tool with examples how to use them against your codebase. A must attend talk for every developer that wants to scale up their quality. Most PHP developers deploy code that does what the customer requested but they don't have a clue about the quality of the product they deliver. Without this knowledge, maintenance can be a hell and very expensive. In this workshop I cover unit testing, code measuring, performance testing, debugging and profiling and give tips and tricks how to continue after this workshop.
This document outlines exercises for learning REST APIs with Python in 90 minutes. It introduces REST APIs and JSON format and how to make GET and POST requests using curl, requests module, and Python scripts. Exercises include making GET requests to retrieve headers and JSON data, handling HTTPS and basic authentication, and posting data to create a GitHub gist. The homework is to implement full CRUD operations for gists API, use OAuth for GitHub authentication, and get Jenkins job details in Python scripts.
Everyone talks about raising the bar on quality of code, but it's always hard to start implementing it when you have no clue where to start. With this talk I'm shooing that there are many levels developers can improve themselves by using the right tools. In this talk I'll go over each tool with examples how to use them against your codebase. A must attend talk for every developer that wants to scale up their quality. Most PHP developers deploy code that does what the customer requested but they don't have a clue about the quality of the product they deliver. Without this knowledge, maintenance can be a hell and very expensive. In this workshop I cover unit testing, code measuring, performance testing, debugging and profiling and give tips and tricks how to continue after this workshop.
Everyone talks about raising the bar on quality of code, but it's always hard to start implementing it when you have no clue where to start. With this talk I'm shooing that there are many levels developers can improve themselves by using the right tools. In this talk I'll go over each tool with examples how to use them against your codebase. A must attend talk for every developer that wants to scale up their quality. Most PHP developers deploy code that does what the customer requested but they don't have a clue about the quality of the product they deliver. Without this knowledge, maintenance can be a hell and very expensive. In this workshop I cover unit testing, code measuring, performance testing, debugging and profiling and give tips and tricks how to continue after this workshop.
Testing http calls with Webmock and VCRKerry Buckley
Webmock and VCR are tools for stubbing and recording HTTP requests in tests. Webmock allows fine-grained control over stubbing but requires the remote server, while VCR records and replays requests without the server through cassettes. The document provides information on setting up and using both tools to stub or record HTTP requests in different testing frameworks for reliable and isolated tests.
As more and more web applications integrate with third-party APIs and other external data, processing those external resources in the background more and more important. A simple job runner is a great start, however as your load increases, you very quickly outgrew that simplistic queuing system. We will cover where getting started using Resque and Redis, how to test your jobs, when it makes sense to use Resque, implementations of Resque in other languages, and look how I've used Resque.
Tips and tricks for building api heavy ruby on rails applicationsTim Cull
The document provides tips for building API-heavy Ruby on Rails applications. It discusses using APIs from Instagram, CafePress, Spreadsheets, Google Docs, and others. It covers authentication challenges, using background jobs, effective testing strategies like mocking HTTP requests, and different approaches to OAuth authentication used by APIs like Instagram, Freshbooks, Xero, and Evernote. Code examples are provided for common API patterns like making requests, parsing responses, and implementing OAuth flows.
Django is a high-level Python web framework that allows for fast development, automates repetitive tasks, and follows best practices. It provides features like an admin interface, elegant URL design, templates, and is powerful yet easy to use for web development. The document discusses Django's history, installation, basic tutorial on creating models, views, URLs, and templates.
This document discusses using semantic web technologies with REST APIs. It provides a case study of calculating issue close times from GitHub and Jira APIs. The code starts simply but gets more complex to handle different API representations. Adding semantics through common schemas helps parse issues into a uniform object to simplify the code. The document advocates for combining REST, hypermedia, and semantics for understanding and discovery.
Dev Jumpstart: Build Your First App with MongoDBMongoDB
This document provides an overview of MongoDB and how to build a simple blogging application with it. It describes MongoDB as a document database where documents are analogous to JSON objects and collections are analogous to tables in a relational database. It demonstrates how to install and run MongoDB, insert sample data for users and blog posts, query the data, and update documents. The document also lists various MongoDB drivers and provides pointers for next steps in learning about data modeling, replication, performance, and scaling in MongoDB.
This document discusses testing APIs and external services. It presents two main approaches: stubbing, where external dependencies are mocked to make tests faster and isolated, and recording interactions using VCR to automatically record real interactions for playback in future test runs. VCR allows tests to run quickly without real external connections while still accurately testing service integration. The document also covers more advanced VCR features like matching requests more flexibly, handling multiple requests, dynamic cassettes, and automatically updating cassettes over time.
Gohan : YAML-based REST API Service Definition Language
API Definition Generation (including Swagger)
DB Table Generation & OR Mapping
Support Custom Logic using Gohan Script (Javascript, and Go)
Extensible Role-Based Access Control
etcd integration
Make an object be searchable is difficult? Taking control of search result's ranking is even more challenging.
In BarCamp Bangkhen 2015 (Nov 22, 2015), Suparit Krityakien, Co-founder & Software Architecture of Wongnai (http://www.wongnai.com) shared his experiences on applying Wongnai ranking algorithm to take control of our search engine's scoring systematically as well as what can be tuned to make it smarter.
This presentation is to furnish you with fundamental knowledge required when tweaking ranking on your search.
Rants and Ruminations From A Job Applicant After 💯 CS Job Interviews in Silic...Susan Tan
What is it like to interview at 1 tech company? Stressful and tiring. What is it like to interview at 100 tech companies? In early September 2016, I quit an uninspiring full-time remote software job and talked to 100 employers in the Bay Area to find the best fit. Listen to my rants and ruminations about that journey.
This 30-minute talk was given at PyCon 2017 in Portland, DjangoCon Europe 2017 in Florence, and a condensed 10-minute version was given at Codeland 2017 in NYC.
How to Upgrade to the Newest Shiniest Django VersionSusan Tan
This document summarizes Susan's presentation on upgrading Django projects. The presentation covered:
- Why upgrades are important for getting new features, fixes and security updates.
- The steps involved in upgrading include running tests, fixing broken tests one by one, checking that fewer tests fail with each new version, verifying the UI still works, and editing deployment files.
- Challenges include dependencies breaking, release notes documenting changes that require code updates, and the repetitive nature of fixing tests across multiple versions.
- The key takeaways are to upgrade one version at a time, run and fix tests with each version, read release notes, break tasks into small steps, and use checklists to track progress
How do I run multiple python apps in 1 command line under 1 WSGI app?Susan Tan
This document discusses running multiple Python web applications under a single WSGI server. It presents a Flask "Hello World" app and a Django "Hello World" app that run on different ports. It then shows how to use a WSGI dispatcher middleware to combine the two apps so they run on the same port from a single command line. Sample code is provided to demonstrate configuring WSGI to dispatch requests between the Flask and Django apps.
You already know how to use “git status”, “git push”, and “git add” for your personal projects. You know how to work on a team project with git version control. How do you achieve the next level of git mastery and become as productive as possible? Mistakes happen very frequently when coding. Sometimes they can be terrifying mistakes. You’ll learn git commands to un-do your coding mistakes effortlessly. How do you re-order, edit, remove, and squash commits? You’ll understand the differences between all the "git r*” commands: "git rebase", "git revert", "git reset", and "git reflog" and know when and why to use them. We’ll cover how to set up your git environment for a productive workflow, different ways to undo your mistakes in git, and finally, different ways to automate an entire git workflow.
These are the slides for a talk that I gave at PyTN 2015 on a Sunday morning.
Have you ever wished that you can write Python in the browser? Brython is an experimental Javascript library that enables you to leverage the elegance of Python on the client side. Novice programmers who are familiar with Javascript will learn about Brython, how Brython measures up against Javascript, and why getting client-side Python adopted as a mainstream tool is really difficult.
Ready to Unlock the Power of Blockchain!Toptal Tech
Imagine a world where data flows freely, yet remains secure. A world where trust is built into the fabric of every transaction. This is the promise of blockchain, a revolutionary technology poised to reshape our digital landscape.
Toptal Tech is at the forefront of this innovation, connecting you with the brightest minds in blockchain development. Together, we can unlock the potential of this transformative technology, building a future of transparency, security, and endless possibilities.
Instagram has become one of the most popular social media platforms, allowing people to share photos, videos, and stories with their followers. Sometimes, though, you might want to view someone's story without them knowing.
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024APNIC
Ellisha Heppner, Grant Management Lead, presented an update on APNIC Foundation to the PNG DNS Forum held from 6 to 10 May, 2024 in Port Moresby, Papua New Guinea.
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptxBrad Spiegel Macon GA
Brad Spiegel Macon GA’s journey exemplifies the profound impact that one individual can have on their community. Through his unwavering dedication to digital inclusion, he’s not only bridging the gap in Macon but also setting an example for others to follow.
3. This is a talk about
1. how to actively read a new Python codebase
2. reading thru the python-requests codebase
“Indeed, the ratio of time spent reading versus
writing is well over 10 to 1. We are constantly
reading old code as part of the effort to write
new code. ...[Therefore,] making it easy to read
makes it easier to write.”
Robert C. Martin, Clean Code: A Handbook of Agile Software Craftsmanship
4. Set up your editor to
• jump into any method or class definition
• search files by keywords
• get call hierarchy of any given method
or class
Note: I’ll be using Sublime Text with python-requests library.
Step 0: Prepare Your Editor
5. Step 1: Git clone and open the repo
$ git clone https://github.com/
kennethreitz/requests
$ cd requests
$ subl requests
6. Step 2: Set up local dev environment to
get into mindset of a contributor
python test_requests.py works too.
requests is on permanent feature freeze.
Source: http://docs.python-requests.org/en/latest/dev/todo/#development-dependencies
7. What it’s like to read a large codebase
Once you’ve set up your editor & local dev
environment…
8. Goal for today —
Figure out how this code snippet works
>>> r = requests.get('https://api.github.com/
user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos':
77}
9. Step 3: Look at unit tests
• Over 1,600 lines of code in
test_requests.py. Where to look
first?
• Use git grep or keyword search
for “requests.get”
14. def test_DIGEST_HTTP_200_OK_GET(self, httpbin):
auth = HTTPDigestAuth('user', 'pass')
url = httpbin('digest-auth', ‘auth', 'user',
'pass')
r = requests.get(url, auth=auth)
assert r.status_code == 200
r = requests.get(url)
assert r.status_code == 401
s = requests.session()
s.auth = HTTPDigestAuth('user', 'pass')
r = s.get(url)
assert r.status_code == 200
test_requests.pyWhat’s a session?
15. def test_DIGEST_HTTP_200_OK_GET(self, httpbin):
auth = HTTPDigestAuth('user', 'pass')
url = httpbin('digest-auth', ‘auth', 'user',
'pass')
r = requests.get(url, auth=auth)
assert r.status_code == 200
r = requests.get(url)
assert r.status_code == 401
s = requests.session()
s.auth = HTTPDigestAuth('user', 'pass')
r = s.get(url)
assert r.status_code == 200
test_requests.py
Let’s look at class definition
16. class HTTPDigestAuth(AuthBase):
"""Attaches HTTP Digest Authentication to the given Request
object."""
def __init__(self, username, password):
self.username = username
self.password = password
# Keep state in per-thread local storage
self._thread_local = threading.local()
def init_per_thread_state(self):
# Ensure state is initialized just once per-thread
…
def build_digest_header(self, method, url):
…
def handle_redirect(self, r, **kwargs):
…
def handle_401(self, r, **kwargs):
…
def __call__(self, r):
…
What is the HTTPDigestAuth class?
auth.py
18. def test_DIGEST_HTTP_200_OK_GET(self, httpbin):
auth = HTTPDigestAuth('user', 'pass')
url = httpbin('digest-auth', ‘auth',
'user', 'pass')
r = requests.get(url, auth=auth)
assert r.status_code == 200
r = requests.get(url)
assert r.status_code == 401
s = requests.session()
s.auth = HTTPDigestAuth('user', 'pass')
r = s.get(url)
assert r.status_code == 200
test_requests.py
Let’s look at method definition
19. def test_DIGEST_HTTP_200_OK_GET(self, httpbin):
auth = HTTPDigestAuth('user', 'pass')
url = httpbin('digest-auth', ‘auth',
'user', 'pass')
This is httpbin() method in conftest.py:
import pytest
from requests.compat import urljoin
def prepare_url(value):
# Issue #1483: Make sure the URL always has a trailing
slash
httpbin_url = value.url.rstrip('/') + '/'
def inner(*suffix):
return urljoin(httpbin_url, ‘/'.join(suffix))
return inner
@pytest.fixture
def httpbin(httpbin):
return prepare_url(httpbin)
20.
21. I’m still confused by what httpbin()
method is doing.
Next steps:
• look up “httpbin” in official request
docs.
• If that fails, then use a debugger.
We’ll do both.
27. • httpbin is everywhere in unit tests in the
requests repo every time a request is
made.
• This is a BIG step forward in our
understanding of unit tests in this repo.
httpbin endpoints and requests unit
test
28. import pdb
pdb.set_trace()
Use pdbpp debugger
Let’s inspect variable “url” in that previous unit test.
“pdbpp is a million times better than ipdb” —a co-worker
35. def get(url, params=None, **kwargs):
"""Sends a GET request.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary or bytes to be sent in the query
string for the :class:`Request`.
:param **kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
kwargs.setdefault('allow_redirects', True)
return request('get', url, params=params, **kwargs)
This is the get method
What’s happening here?
• Set default dict of key-value pairs to allow redirects by
default
• Returns a request. What’s a request?
api.py
36. def request(method, url, **kwargs):
"""Constructs and sends a :class:`Request <Request>`.
:param method: method for the new :class:`Request` object.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
:param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
:param json: (optional) json data to send in the body of the :class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
:param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
:param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': ('filename', fileobj)}``)
for multipart encoding upload.
:param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
:param timeout: (optional) How long to wait for the server to send data
before giving up, as a float, or a :ref:`(connect timeout, read
timeout) <timeouts>` tuple.
:type timeout: float or tuple
:param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
:type allow_redirects: bool
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
:param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided.
Defaults to ``True``.
:param stream: (optional) if ``False``, the response content will be immediately downloaded.
:param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
:return: :class:`Response <Response>` object
:rtype: requests.Response
Usage::
>>> import requests
>>> req = requests.request('GET', 'http://httpbin.org/get')
<Response [200]>
"""
# By using the 'with' statement we are sure the session is closed, thus we
# avoid leaving sockets open which can trigger a ResourceWarning in some
# cases, and look like a memory leak in others.
with sessions.Session() as session:
return session.request(method=method, url=url, **kwargs)
api.py
This is the request method
37. def request(method, url, **kwargs):
with sessions.Session() as session:
return session.request(method=method, url=url, **kwargs)
What are sessions??
api.py
This is the same request method without
docstrings or comments
38. class Session(SessionRedirectMixin):
"""A Requests session.
Provides cookie persistence, connection-pooling, and configuration.
Basic Usage::
>>> import requests
>>> s = requests.Session()
>>> s.get('http://httpbin.org/get')
<Response [200]>
Or as a context manager::
>>> with requests.Session() as s:
>>> s.get('http://httpbin.org/get')
<Response [200]>
"""
__attrs__ = [
'headers', 'cookies', 'auth', 'proxies', 'hooks', 'params', 'verify',
'cert', 'prefetch', 'adapters', 'stream', 'trust_env',
'max_redirects',
]
def __init__(self):
#: A case-insensitive dictionary of headers to be sent on each
#: :class:`Request <Request>` sent from this
#: :class:`Session <Session>`.
self.headers = default_headers()
#: Default Authentication tuple or object to attach to
#: :class:`Request <Request>`.
self.auth = None
#: Dictionary mapping protocol or protocol and host to the URL of the proxy
#: (e.g. {'http': 'foo.bar:3128', 'http://host.name': 'foo.bar:4012'}) to
#: be used on each :class:`Request <Request>`.
self.proxies = {}
#: Event-handling hooks.
self.hooks = default_hooks()
#: Dictionary of querystring data to attach to each
#: :class:`Request <Request>`. The dictionary values may be lists for
#: representing multivalued query parameters.
self.params = {}
#: Stream response content default.
self.stream = False
#: SSL Verification default.
self.verify = True
#: SSL certificate default.
self.cert = None
….
….
……
…….…
sessions.py
R
eally
long
class
definition
of“Sessions”
40. What’s a session?
• an object that persists parameters across
requests
• makes use of urllib3’s connection pooling
• has all methods of request API
• provides default data to request object
• note: requests has well written docs
41. def request(method, url, **kwargs):
with sessions.Session() as session:
return session.request(method=method, url=url, **kwargs)
What is this request() in Session
class?
api.py
This is the same request method without
docstrings or comments
42. sessions.py
def request(self, method, url,
params=None,
data=None,
headers=None,
cookies=None,
files=None,
auth=None,
timeout=None,
allow_redirects=True,
proxies=None,
hooks=None,
stream=None,
verify=None,
cert=None,
json=None):
# Create the Request.
req = Request(
method = method.upper(),
url = url,
headers = headers,
files = files,
data = data or {},
json = json,
params = params or {},
auth = auth,
cookies = cookies,
hooks = hooks,
)
prep = self.prepare_request(req)
proxies = proxies or {}
settings = self.merge_environment_settings(
prep.url, proxies, stream, verify, cert
)
# Send the request.
send_kwargs = {
'timeout': timeout,
'allow_redirects': allow_redirects,
}
send_kwargs.update(settings)
resp = self.send(prep, **send_kwargs)
return resp
What is this request() in Session
class?
45. class Request(RequestHooksMixin):
"""A user-created :class:`Request <Request>` object.
Used to prepare a :class:`PreparedRequest <PreparedRequest>`, which is sent to the server.
:param method: HTTP method to use.
:param url: URL to send.
:param headers: dictionary of headers to send.
:param files: dictionary of {filename: fileobject} files to multipart upload.
:param data: the body to attach to the request. If a dictionary is provided, form-encoding will take place.
:param json: json for the body to attach to the request (if files or data is not specified).
:param params: dictionary of URL parameters to append to the URL.
:param auth: Auth handler or (user, pass) tuple.
:param cookies: dictionary or CookieJar of cookies to attach to this request.
:param hooks: dictionary of callback hooks, for internal usage.
Usage::
>>> import requests
>>> req = requests.Request('GET', 'http://httpbin.org/get')
>>> req.prepare()
<PreparedRequest [GET]>
"""
def __init__(self, method=None, url=None, headers=None, files=None,
data=None, params=None, auth=None, cookies=None, hooks=None, json=None):
# Default empty dicts for dict params.
data = [] if data is None else data
files = [] if files is None else files
headers = {} if headers is None else headers
params = {} if params is None else params
hooks = {} if hooks is None else hooks
self.hooks = default_hooks()
for (k, v) in list(hooks.items()):
self.register_hook(event=k, hook=v)
self.method = method
self.url = url
self.headers = headers
self.files = files
self.data = data
self.json = json
self.params = params
self.auth = auth
self.cookies = cookies
def __repr__(self):
return '<Request [%s]>' % (self.method)
def prepare(self):
"""Constructs a :class:`PreparedRequest <PreparedRequest>` for transmission and returns it."""
p = PreparedRequest()
p.prepare(
method=self.method,
url=self.url,
headers=self.headers,
files=self.files,
data=self.data,
json=self.json,
params=self.params,
auth=self.auth,
cookies=self.cookies,
hooks=self.hooks,
)
return p
models.py
request arguments
that were passed in
This is Request
class definition
48. def prepare_request(self, request):
"""Constructs a :class:`PreparedRequest <PreparedRequest>` for
transmission and returns it. The :class:`PreparedRequest` has settings
merged from the :class:`Request <Request>` instance and those of the
:class:`Session`.
:param request: :class:`Request` instance to prepare with this
session's settings.
"""
cookies = request.cookies or {}
# Bootstrap CookieJar.
if not isinstance(cookies, cookielib.CookieJar):
cookies = cookiejar_from_dict(cookies)
# Merge with session cookies
merged_cookies = merge_cookies(
merge_cookies(RequestsCookieJar(), self.cookies), cookies)
# Set environment's basic authentication if not explicitly set.
auth = request.auth
if self.trust_env and not auth and not self.auth:
auth = get_netrc_auth(request.url)
p = PreparedRequest()
p.prepare(
method=request.method.upper(),
url=request.url,
files=request.files,
data=request.data,
json=request.json,
headers=merge_setting(request.headers, self.headers,
dict_class=CaseInsensitiveDict),
params=merge_setting(request.params, self.params),
auth=merge_setting(auth, self.auth),
cookies=merged_cookies,
hooks=merge_hooks(request.hooks, self.hooks),
)
return p
sessions.py
setup stuff
happens
“PreparedRequests
class”? What is the
class method
“prepare()”?
49. class PreparedRequest(RequestEncodingMixin, RequestHooksMixin):
"""The fully mutable :class:`PreparedRequest <PreparedRequest>` object,
containing the exact bytes that will be sent to the server.
Generated from either a :class:`Request <Request>` object or manually.
Usage::
>>> import requests
>>> req = requests.Request('GET', 'http://httpbin.org/get')
>>> r = req.prepare()
<PreparedRequest [GET]>
>>> s = requests.Session()
>>> s.send(r)
<Response [200]>
"""
def __init__(self):
#: HTTP verb to send to the server.
self.method = None
#: HTTP URL to send the request to.
self.url = None
#: dictionary of HTTP headers.
self.headers = None
# The `CookieJar` used to create the Cookie header will be stored here
# after prepare_cookies is called
self._cookies = None
#: request body to send to the server.
self.body = None
#: dictionary of callback hooks, for internal usage.
self.hooks = default_hooks()
def prepare(self, method=None, url=None, headers=None, files=None,
data=None, params=None, auth=None, cookies=None, hooks=None, json=None):
"""Prepares the entire request with the given parameters."""
self.prepare_method(method)
self.prepare_url(url, params)
self.prepare_headers(headers)
self.prepare_cookies(cookies)
self.prepare_body(data, files, json)
self.prepare_auth(auth, url)
# Note that prepare_auth must be last to enable authentication schemes
# such as OAuth to work on a fully prepared request.
# This MUST go after prepare_auth. Authenticators could add a hook
self.prepare_hooks(hooks)
models.py
W
hat are
Prepared
Requests?
Lots more
layers of
abstraction!
51. 3. send request
4. return response
Let’s dissect requests/sessions.py
def send(self, request, **kwargs):
"""Send a given PreparedRequest."""
….[LONG METHOD DEFINITION HERE]….
return r
send()request object response object
sessions.py
52. def send(self, request, **kwargs):
"""Send a given PreparedRequest."""
# Set defaults that the hooks can utilize to ensure they
always have
# the correct parameters to reproduce the previous request.
kwargs.setdefault('stream', self.stream)
kwargs.setdefault('verify', self.verify)
kwargs.setdefault('cert', self.cert)
kwargs.setdefault('proxies', self.proxies)
# It's possible that users might accidentally send a Request
object.
# Guard against that specific failure case.
if not isinstance(request, PreparedRequest):
raise ValueError('You can only send PreparedRequests.')
checked_urls = set()
while request.url in self.redirect_cache:
checked_urls.add(request.url)
new_url = self.redirect_cache.get(request.url)
if new_url in checked_urls:
break
request.url = new_url
# Set up variables needed for resolve_redirects and
dispatching of hooks
allow_redirects = kwargs.pop('allow_redirects', True)
stream = kwargs.get('stream')
hooks = request.hooks
# Get the appropriate adapter to use
adapter = self.get_adapter(url=request.url)
# Start time (approximately) of the request
start = datetime.utcnow()
# Send the request
r = adapter.send(request, **kwargs)
# Total elapsed time of the request (approximately)
r.elapsed = datetime.utcnow() - start
# Response manipulation hooks
r = dispatch_hook('response', hooks, r, **kwargs)
# Persist cookies
if r.history:
# If the hooks create history then we want those cookies
too
for resp in r.history:
extract_cookies_to_jar(self.cookies, resp.request,
resp.raw)
extract_cookies_to_jar(self.cookies, request, r.raw)
# Redirect resolving generator.
gen = self.resolve_redirects(r, request, **kwargs)
# Resolve redirects if allowed.
history = [resp for resp in gen] if allow_redirects else []
# Shuffle things around if there's history.
if history:
# Insert the first (original) request at the start
history.insert(0, r)
# Get the last request made
r = history.pop()
r.history = history
if not stream:
r.content
return r
3. send request
4. return response
# Get the appropriate adapter to use
adapter = self.get_adapter(url=request.url)
# Start time (approximately) of the request
start = datetime.utcnow()
# Send the request
r = adapter.send(request, **kwargs)
sessions.py
What is the send method
doing in adapters.py?
What’s an adapter?
53. What’s an adapter?
“This adapter provides the default Requests
interaction with HTTP and HTTPS using the
powerful urllib3 library.”
—“Transport Adapters” in requests advanced docs
54. class HTTPAdapter(BaseAdapter):
"""
The built-in HTTP Adapter for urllib3.
Provides a general-case interface for Requests sessions to contact HTTP and
HTTPS urls by implementing the Transport Adapter interface. This class will
usually be created by the :class:`Session <Session>` class under the
covers.
:param pool_connections: The number of urllib3 connection pools to cache.
:param pool_maxsize: The maximum number of connections to save in the pool.
:param int max_retries: The maximum number of retries each connection
should attempt. Note, this applies only to failed DNS lookups, socket
connections and connection timeouts, never to requests where data has
made it to the server. By default, Requests does not retry failed
connections. If you need granular control over the conditions under
which we retry a request, import urllib3's ``Retry`` class and pass
that instead.
:param pool_block: Whether the connection pool should block for connections.
Usage::
>>> import requests
>>> s = requests.Session()
>>> a = requests.adapters.HTTPAdapter(max_retries=3)
>>> s.mount('http://', a)
"""
This is HTTPAdapter class, the interface for urllib3
adapters.py
55. """
requests.adapters
~~~~~~~~~~~~~~~~~
This module contains the transport adapters that Requests uses to define
and maintain connections.
"""
import os.path
import socket
from .models import Response
from .packages.urllib3.poolmanager import PoolManager, proxy_from_url
from .packages.urllib3.response import HTTPResponse
from .packages.urllib3.util import Timeout as TimeoutSauce
from .packages.urllib3.util.retry import Retry
from .compat import urlparse, basestring
from .utils import (DEFAULT_CA_BUNDLE_PATH, get_encoding_from_headers,
prepend_scheme_if_needed, get_auth_from_url, urldefragauth,
select_proxy)
from .structures import CaseInsensitiveDict
from .packages.urllib3.exceptions import ClosedPoolError
from .packages.urllib3.exceptions import ConnectTimeoutError
from .packages.urllib3.exceptions import HTTPError as _HTTPError
from .packages.urllib3.exceptions import MaxRetryError
from .packages.urllib3.exceptions import NewConnectionError
from .packages.urllib3.exceptions import ProxyError as _ProxyError
from .packages.urllib3.exceptions import ProtocolError
from .packages.urllib3.exceptions import ReadTimeoutError
from .packages.urllib3.exceptions import SSLError as _SSLError
from .packages.urllib3.exceptions import ResponseError
from .cookies import extract_cookies_to_jar
from .exceptions import (ConnectionError, ConnectTimeout, ReadTimeout, SSLError,
ProxyError, RetryError)
from .auth import _basic_auth_str
DEFAULT_POOLBLOCK = False
DEFAULT_POOLSIZE = 10
DEFAULT_RETRIES = 0
DEFAULT_POOL_TIMEOUT = None
the imports at top of requests/adapters.py
urllib3 is imported here
adapters.py
56. def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
"""Sends PreparedRequest object. Returns Response object.
:param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
:param stream: (optional) Whether to stream the request content.
:param timeout: (optional) How long to wait for the server to send
data before giving up, as a float, or a :ref:`(connect timeout,
read timeout) <timeouts>` tuple.
:type timeout: float or tuple
:param verify: (optional) Whether to verify SSL certificates.
:param cert: (optional) Any user-provided SSL certificate to be trusted.
:param proxies: (optional) The proxies dictionary to apply to the request.
"""
conn = self.get_connection(request.url, proxies)
self.cert_verify(conn, request.url, verify, cert)
url = self.request_url(request, proxies)
self.add_headers(request)
chunked = not (request.body is None or 'Content-Length' in request.headers)
if isinstance(timeout, tuple):
try:
connect, read = timeout
timeout = TimeoutSauce(connect=connect, read=read)
except ValueError as e:
# this may raise a string formatting error.
err = ("Invalid timeout {0}. Pass a (connect, read) "
"timeout tuple, or a single float to set "
"both timeouts to the same value".format(timeout))
raise ValueError(err)
else:
timeout = TimeoutSauce(connect=timeout, read=timeout)
try:
if not chunked:
resp = conn.urlopen(
method=request.method,
url=url,
body=request.body,
headers=request.headers,
redirect=False,
assert_same_host=False,
preload_content=False,
decode_content=False,
retries=self.max_retries,
timeout=timeout
)
# Send the request.
else:
if hasattr(conn, 'proxy_pool'):
conn = conn.proxy_pool
low_conn = conn._get_conn(timeout=DEFAULT_POOL_TIMEOUT)
try:
low_conn.putrequest(request.method,
url,
skip_accept_encoding=True)
for header, value in request.headers.items():
low_conn.putheader(header, value)
low_conn.endheaders()
for i in request.body:
low_conn.send(hex(len(i))[2:].encode('utf-8'))
low_conn.send(b'rn')
low_conn.send(i)
low_conn.send(b'rn')
low_conn.send(b'0rnrn')
# Receive the response from the server
try:
# For Python 2.7+ versions, use buffering of HTTP
# responses
r = low_conn.getresponse(buffering=True)
except TypeError:
# For compatibility with Python 2.6 versions and back
r = low_conn.getresponse()
resp = HTTPResponse.from_httplib(
r,
pool=conn,
connection=low_conn,
preload_content=False,
decode_content=False
)
except:
# If we hit any problems here, clean up the connection.
# Then, reraise so that we can handle the actual exception.
low_conn.close()
raise
except (ProtocolError, socket.error) as err:
raise ConnectionError(err, request=request)
except MaxRetryError as e:
if isinstance(e.reason, ConnectTimeoutError):
# TODO: Remove this in 3.0.0: see #2811
if not isinstance(e.reason, NewConnectionError):
raise ConnectTimeout(e, request=request)
if isinstance(e.reason, ResponseError):
raise RetryError(e, request=request)
raise ConnectionError(e, request=request)
except ClosedPoolError as e:
raise ConnectionError(e, request=request)
except _ProxyError as e:
raise ProxyError(e)
except (_SSLError, _HTTPError) as e:
if isinstance(e, _SSLError):
raise SSLError(e, request=request)
elif isinstance(e, ReadTimeoutError):
raise ReadTimeout(e, request=request)
else:
raise
adapters.py
Size
5
font. This
is
the
definition
of send() m
ethod. Over
100
lines
long
and
it can’t fit this
slide.
Exercise
left to
reader to
read
thru
this
send() m
ethod.
57. def send(self, request, stream=False, timeout=None, verify=True, cert=None,
proxies=None):
…
import pdp
pdb.set_trace()
return self.build_response(request, resp)
Let’s place a debugger in
adapters.py and run the same unit
test again.
py.test test_requests.py::TestRequests::test_DIGEST_HTTP_200_OK_GET
Run this same unit test on command line
adapters.py
58. Use pytest debugger to see output of send() method
[48] > /Users/susantan/Projects/requests/requests/
adapters.py(455)send()
-> return self.build_response(request, resp)
(Pdb++) request.url
‘http://127.0.0.1:58948/digest-auth/auth/user/pass'
(Pdb++) resp
<requests.packages.urllib3.response.HTTPResponse object
at 0x102c15110>
(Pdb++) our_version_of_response_object =
self.build_response(request, resp)
(Pdb++) our_version_of_response_object.json()
{u'authenticated': True, u'user': u’user'}
(Pdb++) our_version_of_response_object.status_code
200
59. Goal for today —
Figure out how this code snippet works
>>> r = requests.get('https://api.github.com/
user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos':
77}
63. def request(method, url, **kwargs):
with sessions.Session() as session:
return session.request(method=method, url=url,
**kwargs)
api.py
This is the request method
This is the get method
def get(url, params=None, **kwargs):
kwargs.setdefault('allow_redirects', True)
return request('get', url, params=params,
**kwargs)
65. A mental map of files and associated
function calls
adapters.py
sessions.py
models.py
api.py
File names
test_requests.py
request(), get(), session.request()
class Request(), class
PreparedRequest()
class Request(), prepare_request(), send()
send()
method or class names
test_DIGEST_HTTP_200_OK_GET()
66. No walkthrough of a codebase is the
same for any person. An Alternative —
In [4]: import requests
In [4]: resp = requests.post('http://httpbin.org/post',
data={'name':'Susan'})
[14] > /Users/susantan/Projects/requests/requests/
adapters.py(346)send()
-> conn = self.get_connection(request.url, proxies)
(Pdb++) l
341 """
342
343 import pdb
344 pdb.set_trace()
345
346 -> conn = self.get_connection(request.url, proxies)
347
348 self.cert_verify(conn, request.url, verify, cert)
349 url = self.request_url(request, proxies)
350 self.add_headers(request)
Set breakpoints!
67. We really know “requests.get(url)” works in great
depth.
Takeaways
r = requests.get('https://api.github.com/
user', auth=('user', 'pass'))
68. • Talk to core devs or maintainers
• git blame
What to do when you get
really stuck on figuring out
codebase?
69. Use your favorite python shell and debugger to
explore a small code sample.
What to do when you get
really stuck on figuring out
codebase?