This document discusses various tools for debugging memory issues in Python, Cython, and C/C++ programs running on Linux, including top, ps, objgraph, guppy, memory_profiler, valgrind, gdb, strace, and ltrace. It provides an example of using these tools to find and fix a memory leak, showing massif output before and after the fix.
Paul Tarjan ( http://github.com/ptarjan ) presented this to the Hadoop User Group at the Yahoo! Sunnyvale campus on 11/18/09. Paul describes his solution for building a Hadoop Record Reader in Python.
[GSoC 2017] gopy: Updating gopy to support Python3 and PyPyDong-hee Na
gopy is an excellent tool which generates (and compiles) a CPython extension module from a go package. And I hope more developers could make full use of gopy to migrate their go code into python code. To make gopy more advanced, It is necessary to provide APIs for various Python compiler versions, such as CPython 2/3 and PyPy. This can be improved with CFFI or ctypes. Moreover, many go’s implementations/features are not yet implemented in gopy. So we need to implement implementations such as slices, interfaces, and maps in the go.
My goal is to update gopy by using CFFI to support Python3 and PyPy and write detailed documents
Paul Tarjan ( http://github.com/ptarjan ) presented this to the Hadoop User Group at the Yahoo! Sunnyvale campus on 11/18/09. Paul describes his solution for building a Hadoop Record Reader in Python.
[GSoC 2017] gopy: Updating gopy to support Python3 and PyPyDong-hee Na
gopy is an excellent tool which generates (and compiles) a CPython extension module from a go package. And I hope more developers could make full use of gopy to migrate their go code into python code. To make gopy more advanced, It is necessary to provide APIs for various Python compiler versions, such as CPython 2/3 and PyPy. This can be improved with CFFI or ctypes. Moreover, many go’s implementations/features are not yet implemented in gopy. So we need to implement implementations such as slices, interfaces, and maps in the go.
My goal is to update gopy by using CFFI to support Python3 and PyPy and write detailed documents
The C++ programming language has a history going back to 1979, when Bjarne Stroustrup was doing work for his Ph.D. thesis. One of the languages Stroustrup had the opportunity to work with was a language called Simula, which as the name implies is a language primarily designed for simulations.
Matplotlib has wonderfully served the Python community as the cornerstone of scientific graphics. Recently, many additional Python plotting options have surfaced, aimed to make it easier to create graphics that are interactive and web-publishable. These slides outline some of the new options with links to easy-to-follow, IPython notebooks.
This presentation is about using Boost.Python library to create modules with С++.
Presentation by Andriy Ohorodnyk (Lead Software Engineer, GlobalLogic, Lviv), delivered GlobalLogic C++ TechTalk in Lviv, September 18, 2014.
More details -
http://www.globallogic.com.ua/press-releases/lviv-cpp-techtalk-coverage
The Python Debugger pdb is one of the most important tools available to python-developers. Knowing how to apply the many features in your daily routine makes you more productive and boosts the understanding of the code you otherwise bang your head against. This talk explores the features of pdb, some of its alternatives and showcases some advanced tipps and tricks.
Eclipse Neon Democamp Budapest - VIATRA 1.3 releaseÁbel Hegedüs
Overview of VIATRA 1.3 features, and a collection of links for related tutorials, tools, applications and other material. Presented at the Eclipse Neon Democamp at Budapest.
The C++ programming language has a history going back to 1979, when Bjarne Stroustrup was doing work for his Ph.D. thesis. One of the languages Stroustrup had the opportunity to work with was a language called Simula, which as the name implies is a language primarily designed for simulations.
Matplotlib has wonderfully served the Python community as the cornerstone of scientific graphics. Recently, many additional Python plotting options have surfaced, aimed to make it easier to create graphics that are interactive and web-publishable. These slides outline some of the new options with links to easy-to-follow, IPython notebooks.
This presentation is about using Boost.Python library to create modules with С++.
Presentation by Andriy Ohorodnyk (Lead Software Engineer, GlobalLogic, Lviv), delivered GlobalLogic C++ TechTalk in Lviv, September 18, 2014.
More details -
http://www.globallogic.com.ua/press-releases/lviv-cpp-techtalk-coverage
The Python Debugger pdb is one of the most important tools available to python-developers. Knowing how to apply the many features in your daily routine makes you more productive and boosts the understanding of the code you otherwise bang your head against. This talk explores the features of pdb, some of its alternatives and showcases some advanced tipps and tricks.
Eclipse Neon Democamp Budapest - VIATRA 1.3 releaseÁbel Hegedüs
Overview of VIATRA 1.3 features, and a collection of links for related tutorials, tools, applications and other material. Presented at the Eclipse Neon Democamp at Budapest.
What's it like to be an app? - a Made by Many experience prototyping workshop...Made by Many
On Wednesday 14th November 2012, Rory Hamilton, Charlotte Hillenbrand and Cath Richardson from Made by Many ran an experience prototyping workshop as part of Internet Week Europe.
Participants created and tested a mobile app proposition within the space of a few hours.
To find out more, visit http://madebymany.com
Talk at PyCon2022 over building binary packages for Python. Covers an overview and an in-depth look into pybind11 for binding, scikit-build for creating the build, and build & cibuildwheel for making the binaries that can be distributed on PyPI.
Fast intro of a basic set of tools you can use to diagnose your software’s performance for programs written in Python. It's just an overall view so I really recommend to check them by yourself :).
Euro python2011 High Performance PythonIan Ozsvald
I ran this as a 4 hour tutorial at EuroPython 2011 to teach High Performance Python coding.
Techniques covered include bottleneck analysis by profiling, bytecode analysis, converting to C using Cython and ShedSkin, use of the numerical numpy library and numexpr, multi-core and multi-machine parallelisation and using CUDA GPUs.
Write-up with 49 page PDF report: http://ianozsvald.com/2011/06/29/high-performance-python-tutorial-v0-1-from-my-4-hour-tutorial-at-europython-2011/
Python is a great programming language. It is a complete tutorial of using this programming language.
This slides is split into two parts, and it is the first part. Another part is at: http://www.slideshare.net/moskytw/programming-with-python-adv.
Digital RSE: automated code quality checks - RSE group meetingHenry Schreiner
Given at a local RSE group meeting. Covers code quality practices, focusing on Python but over multiple languages, with useful tools highlighted throughout.
Presenting adhocr (abbreviation for Ad-hoc copy and run) as a simple, but powerful UNIX administrator tool. If you like to retrieve data or execute commands on lots of systems simultaneously then this tool is your friend. There is no need to exchange your ssh keys as the power behind adhocr is the expect tool (language). For example, it is plain easy to use adhocr to distribute your public ssh key to all your systems. The real power of adhocr is the central point of logging, which is perfect for \'grep\'ing into stuff you\'re looking for.
You also have the ability to execute commands via the \'sudo su -\' way, which is a blessing in environments where root is not permitted to login directly.
You can even use it monitoring your systems thanks to the powerful error catching.
Introduction to Docker at SF Peninsula Software Development Meetup @GuidewiredotCloud
Docker is an open-source project to easily create lightweight, portable, self-sufficient containers from any application. The same container that a developer builds and tests on a laptop can run at scale, in production, on VMs, bare metal, OpenStack clusters, public clouds and more.
2. Background
Memory was found to be leaking somewhere in our Python scripts
There are a bunch of tools I had never used before to find and diagnose the problem
3. Good Ol’ Top (or htop)
Run your code and watch resident memory increase
● top -> shift+F -> n -> enter
● top -> shift+M
● watch -n 1 ps -o rss -C PROCESSNAME
● watch -n 1 ps -o rss -p PID
9. Linux Memory System Calls
brk - set the program break, adds memory pages to the top of the heap
sbrk - increments the program break
mmap - map independent page of memory
munmap - unmaps memory pages
strace -e brk,mmap,munmap -p $(pidof PROCNAME)
Compiler flag -O0 removes optimizations