Threads and Callbacks for Embedded PythonYi-Lung Tsai
Python is a great choice to be customized plug-ins for existing applications. Extending existing applications with Python program is also practical. For large systems, multi-thread programming is ubiquitous along with asynchronous programming, such as event routing. This presentation focuses on dealing with threads and callbacks while embedding Python in other applications.
Calling C++ function from Pyhon
OS (Windows 7 also works) Windows 8.1
Compiler Visual Studio 2013 Update4
Python (Other version may be incompatible) python-3.3.5.msi (x86) [1]
SWIG (Unzip and copy swigwin-3.0.2 folder to C:\) swigwin-3.0.2 [2]
Threads and Callbacks for Embedded PythonYi-Lung Tsai
Python is a great choice to be customized plug-ins for existing applications. Extending existing applications with Python program is also practical. For large systems, multi-thread programming is ubiquitous along with asynchronous programming, such as event routing. This presentation focuses on dealing with threads and callbacks while embedding Python in other applications.
Calling C++ function from Pyhon
OS (Windows 7 also works) Windows 8.1
Compiler Visual Studio 2013 Update4
Python (Other version may be incompatible) python-3.3.5.msi (x86) [1]
SWIG (Unzip and copy swigwin-3.0.2 folder to C:\) swigwin-3.0.2 [2]
The OSI Superboard II was the computer on which I first learned to program back in 1979. Python is why programming remains fun today. In this tale of old meets new, I describe how I have used Python 3 to create a cloud computing service for my still-working Superboard--a problem complicated by it only having 8Kb of RAM and 300-baud cassette tape audio ports for I/O.
Notes about moving from python to c++ py contw 2020Yung-Yu Chen
Python is fast to write and deliver results, but it is not a good choice when you cannot sacrifice any runtime. It is the time to switch to C++. But we certainly don’t want to give up the productivity available to Python, and we worry about the complexity of C++. The way to go is to design the backbone system in C++ and expose the API in Python. Then we can enjoy the capabilities coming from the complex compiler and scripting it just like Python.
Despite being a slow interpreter, Python is a key component in high-performance computing (HPC). Python is easy to use. C++ is fast. Together they are a beautiful blend. A new tool, pybind11, makes this approach even more attractive to HPC code. It focuses on the niceties C++11 brings in. Beyond the syntactic sugar around the Python C API, it is interesting to see how pybind11 handles the vast difference between the two languages, and what matters to HPC.
Arduino C maXbox web of things slide showMax Kleiner
RGB LED in Arduino
• Generating QR Code
• 3D Printing
• Web Video Cam
• Digi Clock with Real Time Clock
• Android SeekBar to Arduino LED Matrix
The second part is about new ideas, prototyping and new technologies that are in the lab. It’s
about research papers, and software philosophy, and about researchers worldwide.
Python is a great programming language. It is a complete tutorial of using this programming language.
This slides is split into two parts, and it is the second part. Another part is at: http://www.slideshare.net/moskytw/programming-with-python-basic.
Python is a great programming language. It is a complete tutorial of using this programming language.
This slides is split into two parts, and it is the first part. Another part is at: http://www.slideshare.net/moskytw/programming-with-python-adv.
Talk I gave to The Coding Machine about PHP 8. Overview of the RFC process and which QoL, features and notable BC breaks before finishing on some performance improvements.
source{d} is building the open-source components to enable large-scale code analysis and machine learning on source code. Their powerful tools can ingest all of the world’s public git repositories turning code into ASTs ready for machine learning and other analyses, all exposed through a flexible and friendly API. Francesc will show you how to run machine learning on source code with a series of live demos.
Writing concurrent code is becoming more and more important to leverage the parallelism of multicore architectures. The C++11 library introduced futures and promises as a first step towards task-based programming. However, the C++ support of concurrency is still very limited. Other languages, like C# and Python, provide some forms of resumable functions or coroutines and in C#, the async/await pattern enables to write functions that suspend their execution while waiting for a computation or I/O to complete.This talk will describe a proposal for the addition of resumable function and async/await in C++17. We will focus on the implementation of resumable function on Windows, and we'll play with a first prototype of their implementation in the Visual Studio 2015 Preview. Finally, we will see how resumable functions can also be used to implement (lazy) generators, similar to the one provided by "yield" statements in C#.
Sometimes you see code that is perfectly OK according to the definition of the language, but which is flawed because it breaks too many established idioms and conventions. On the other hand, a solid piece of code is something that looks like it is written by an experienced person who cares about professionalism in programming.
A presentation at Norwegian Developer Conference 2010
The OSI Superboard II was the computer on which I first learned to program back in 1979. Python is why programming remains fun today. In this tale of old meets new, I describe how I have used Python 3 to create a cloud computing service for my still-working Superboard--a problem complicated by it only having 8Kb of RAM and 300-baud cassette tape audio ports for I/O.
Notes about moving from python to c++ py contw 2020Yung-Yu Chen
Python is fast to write and deliver results, but it is not a good choice when you cannot sacrifice any runtime. It is the time to switch to C++. But we certainly don’t want to give up the productivity available to Python, and we worry about the complexity of C++. The way to go is to design the backbone system in C++ and expose the API in Python. Then we can enjoy the capabilities coming from the complex compiler and scripting it just like Python.
Despite being a slow interpreter, Python is a key component in high-performance computing (HPC). Python is easy to use. C++ is fast. Together they are a beautiful blend. A new tool, pybind11, makes this approach even more attractive to HPC code. It focuses on the niceties C++11 brings in. Beyond the syntactic sugar around the Python C API, it is interesting to see how pybind11 handles the vast difference between the two languages, and what matters to HPC.
Arduino C maXbox web of things slide showMax Kleiner
RGB LED in Arduino
• Generating QR Code
• 3D Printing
• Web Video Cam
• Digi Clock with Real Time Clock
• Android SeekBar to Arduino LED Matrix
The second part is about new ideas, prototyping and new technologies that are in the lab. It’s
about research papers, and software philosophy, and about researchers worldwide.
Python is a great programming language. It is a complete tutorial of using this programming language.
This slides is split into two parts, and it is the second part. Another part is at: http://www.slideshare.net/moskytw/programming-with-python-basic.
Python is a great programming language. It is a complete tutorial of using this programming language.
This slides is split into two parts, and it is the first part. Another part is at: http://www.slideshare.net/moskytw/programming-with-python-adv.
Talk I gave to The Coding Machine about PHP 8. Overview of the RFC process and which QoL, features and notable BC breaks before finishing on some performance improvements.
source{d} is building the open-source components to enable large-scale code analysis and machine learning on source code. Their powerful tools can ingest all of the world’s public git repositories turning code into ASTs ready for machine learning and other analyses, all exposed through a flexible and friendly API. Francesc will show you how to run machine learning on source code with a series of live demos.
Writing concurrent code is becoming more and more important to leverage the parallelism of multicore architectures. The C++11 library introduced futures and promises as a first step towards task-based programming. However, the C++ support of concurrency is still very limited. Other languages, like C# and Python, provide some forms of resumable functions or coroutines and in C#, the async/await pattern enables to write functions that suspend their execution while waiting for a computation or I/O to complete.This talk will describe a proposal for the addition of resumable function and async/await in C++17. We will focus on the implementation of resumable function on Windows, and we'll play with a first prototype of their implementation in the Visual Studio 2015 Preview. Finally, we will see how resumable functions can also be used to implement (lazy) generators, similar to the one provided by "yield" statements in C#.
Sometimes you see code that is perfectly OK according to the definition of the language, but which is flawed because it breaks too many established idioms and conventions. On the other hand, a solid piece of code is something that looks like it is written by an experienced person who cares about professionalism in programming.
A presentation at Norwegian Developer Conference 2010
Linux has this great tool called strace, on OSX there’s a tool called dtruss - based on dtrace. Dtruss is great in functionality, it gives pretty much everything you need. It is just not as nice to use as strace. However, on Linux there is also ltrace for library tracing. That is arguably more useful because you can see much more granular application activity. Unfortunately, there isn’t such a tool on OSX. So, I decided to make one - albeit a simpler version for now. I called it objc_trace.
O'Reilly Velocity New York 2016 presentation on modern Linux tracing tools and technology. Highlights the available tracing data sources on Linux (ftrace, perf_events, BPF) and demonstrates some tools that can be used to obtain traces, including DebugFS, the perf front-end, and most importantly, the BCC/BPF tool collection.
Fast as C: How to Write Really Terrible JavaCharles Nutter
For years we’ve been told that the JVM’s amazing optimizers can take your running code and make it “fast” or “as fast as C++” or “as fast as C”…or sometimes “faster than C”. And yet we don’t often see this happen in practice, due in large part to (good and bad) development patterns that have taken hold in the Java world.
In this talk, we’ll explore the main reasons why Java code rarely runs as fast as C or C++ and how you can write really bad Java code that the JVM will do a better job of optimizing. We’ll take some popular microbenchmarks and burn them to the ground, monitoring JIT logs and assembly dumps along the way.
100 bugs in Open Source C/C++ projects Andrey Karpov
This article demonstrates capabilities of the static code analysis methodology. The readers are offered to study the samples of one hundred errors found in open-source projects in C/C++.
Any large modern application consists of numerous third-party libraries, and I'd like to discuss the topic of our trust in these libraries. In books and articles, there are lots of debates about code quality, testing methods, development methodologies, and so on. But I don't remember anyone discussing the quality of bricks applications are built from. So let's talk about it today. For example, there is the Medicine Insight Segmentation and Registration Toolkit (ITK). I find it to be implemented pretty well. At least, I have noticed just a few bugs in its code. But I cannot say the same about the code of the third-party libraries used there. So the question is: how much can we trust such systems? Much food for thought.
A talk on debugging of Python applications given at a local KharkivPy event.
A brief introduction into a set of tools that allow Python developers to debug common issues in their applications.
JVM Mechanics: When Does the JVM JIT & Deoptimize?Doug Hawkins
HotSpot promises to do the "right" thing for us by identifying our hot code and compiling "just-in-time", but how does HotSpot make those decisions?
This presentation aims to detail how HotSpot makes those decisions and how it corrects its mistakes through a series of demos that you run yourself.
Similar to WAD : A Module for Converting Fatal Extension Errors into Python Exceptions (20)
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
How world-class product teams are winning in the AI era by CEO and Founder, P...
WAD : A Module for Converting Fatal Extension Errors into Python Exceptions
1. PY-MA
WAD:
A Module for Converting Fatal Extension Errors into Python
Exceptions
David M. Beazley
Department of Computer Science
University of Chicago
beazley@cs.uchicago.edu
March 6, 2001
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 1 beazley@cs.uchicago.edu
2. Python Extension Building
A popular use of Python
• Hand-written extensions.
• FPIG
• pyfort
• SIP
• BPL
• CXX
• Extension Classes
• GRAD
• SWIG
• (Apologies to anyone I missed)
Extension building is fun
• Python as control language for C,C++, or Fortran.
• Rapid development and prototyping
• Nice user interfaces
But, debugging of extensions is problematic
• At the very least, it’s annoying.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 2 beazley@cs.uchicago.edu
3. A Python Error
% python spam.py
Traceback (most recent call last):
File "spam.py", line 15, in ?
blah()
File "spam.py", line 12, in blah
bar()
File "spam.py", line 9, in bar
foo()
File "spam.py", line 6, in foo
spam(3)
File "spam.py", line 3, in spam
doh(n)
NameError: There is no variable named 'doh'
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 3 beazley@cs.uchicago.edu
4. An Extension Error
% python spam.py
Segmentation Fault (core dumped)
%
or
% python spam.py
Bus Error (core dumped)
or
% python spam.py
Assertion failed: n > 0, file debug.c, line 54
Abort (core dumped)
%
Well, obviously something “bad” happened
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 4 beazley@cs.uchicago.edu
5. Common Failure Modes
Uninitialized Data
• Improper initialization of libraries.
• Forgetting to call an initialization function?
• Calling functions in the wrong order?
Improper argument checking
• Passing of NULL pointers.
• Improper conversion of Python objects to C.
Failed assertions
• Library may make extensive use of assert().
• This is good, but it causes execution to abort.
Weird stuff
• Illegal instructions.
• Bus error. Memory alignment problems.
Math errors
• Floating point exception (SIGFPE).
• Of course, this only happens after 50 hours of computation.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 5 beazley@cs.uchicago.edu
6. GDB Traceback
(gdb) where #31 0xff165544 in Tcl_UplevelObjCmd (dummy=0x1, interp=0x20e658, objc=1,
#0 0xff1d9bf0 in __sigprocmask () from /usr/lib/libthread.so.1 objv=0x24ec80)
#1 0xff1ce628 in _resetsig () from /usr/lib/libthread.so.1 at ./../generic/tclProc.c:614
#2 0xff1cdd18 in _sigon () from /usr/lib/libthread.so.1 #32 0xff13e98c in TclExecuteByteCode (interp=0x20e658, codePtr=0x2a0b70)
#3 0xff1d0e8c in _thrp_kill () from /usr/lib/libthread.so.1 at ./../generic/tclExecute.c:845
#4 0xfee49b10 in raise () from /usr/lib/libc.so.1 #33 0xff122bf8 in Tcl_EvalObjEx (interp=0x20e658, objPtr=0x274d50, flags=0)
#5 0xfee3512c in abort () from /usr/lib/libc.so.1 at ./../generic/tclBasic.c:2723
#6 0xfee353d0 in _assert () from /usr/lib/libc.so.1 #34 0xff165afc in TclObjInterpProc (clientData=0x1, interp=0x20e658, objc=0,
#7 0xfeee13ec in abort_crash () from /u0/beazley/Projects/WAD/WAD/Test/./ objv=0xffbeebd8) at ./../generic/tclProc.c:1001
debugmodule.so #35 0xff15e18c in EvalObjv (interp=0x20e658, objc=2, objv=0xffbeebd8,
#8 0xfeee28ec in _wrap_abort_crash () command=0xffbef024 "n tkButtonUp .1907556n", length=25, flags=0)
from /u0/beazley/Projects/WAD/WAD/Test/./debugmodule.so at ./../generic/tclParse.c:932
#9 0x281c8 in call_builtin (func=0x1cc4f0, arg=0x1f9424, kw=0x0) at ceval.c:2650 #36 0xff15e7d0 in Tcl_EvalEx (interp=0x20e658,
#10 0x28094 in PyEval_CallObjectWithKeywords (func=0x1cc4f0, arg=0x1f9424, kw=0x0) script=0xffbef024 "n tkButtonUp .1907556n", numBytes=25, flags=-4264800)
at ceval.c:2618 at ./../generic/tclParse.c:1393
#11 0x26764 in eval_code2 (co=0x1d37e0, globals=0x0, locals=0x1d37cf, args=0x1cc4f0, #37 0xff15e9c0 in Tcl_Eval (interp=0x20e658,
argcount=1762552, kws=0x0, kwcount=0, defs=0x0, defcount=0, owner=0x0) at string=0xffbef024 "n tkButtonUp .1907556n") at ./../generic/tclParse.c:1512
ceval.c:1951 #38 0xff1243d0 in Tcl_GlobalEval (interp=0x20e658,
#12 0x263a0 in eval_code2 (co=0x1d3858, globals=0x0, locals=0x1cc4f0, args=0x19b1a4, command=0xffbef024 "n tkButtonUp .1907556n") at ./../generic/tclBasic.c:4139
argcount=1883008, kws=0x1d7318, kwcount=0, defs=0x0, defcount=0, owner=0x0) #39 0xff221a40 in Tk_BindEvent (bindingTable=0xffbef024, eventPtr=0x29ffa0,
at ceval.c:1850 tkwin=0x2790a8, numObjects=2045728, objectPtr=0xffbef170) at ./../generic/
#13 0x263a0 in eval_code2 (co=0x1d3e50, globals=0x0, locals=0x19b1a4, args=0x1a7374, tkBind.c:1784
argcount=1883128, kws=0x0, kwcount=0, defs=0x0, defcount=0, owner=0x0) at #40 0xff226450 in TkBindEventProc (winPtr=0x2790a8, eventPtr=0x29ffa0)
ceval.c:1850 at ./../generic/tkCmds.c:244
#14 0x285e0 in call_function (func=0x1a73a4, arg=0x18f114, kw=0x0) at ceval.c:2772 #41 0xff22c218 in Tk_HandleEvent (eventPtr=0x29ffa0) at ./../generic/tkEvent.c:737
#15 0x28080 in PyEval_CallObjectWithKeywords (func=0x1a73a4, arg=0x18f114, kw=0x0) #42 0xff22c61c in WindowEventProc (evPtr=0x29ff98, flags=-1) at ./../generic/
at ceval.c:2616 tkEvent.c:1072
#16 0x680b0 in builtin_apply (self=0x0, args=0x0) at bltinmodule.c:88 #43 0xff15bb54 in Tcl_ServiceEvent (flags=-1) at ./../generic/tclNotify.c:607
#17 0x281c8 in call_builtin (func=0x1910c8, arg=0x1f9b54, kw=0x0) at ceval.c:2650 #44 0xff15beec in Tcl_DoOneEvent (flags=-1) at ./../generic/tclNotify.c:846
#18 0x28094 in PyEval_CallObjectWithKeywords (func=0x1910c8, arg=0x1f9b54, kw=0x0) #45 0x99314 in EventHook () at ./_tkinter.c:2020
at ceval.c:2618 #46 0xbaf30 in rl_read_key () at input.c:374
#19 0x26764 in eval_code2 (co=0x1f3948, globals=0x0, locals=0x1f38f0, args=0x1910c8, #47 0xac920 in readline_internal_char () at readline.c:454
argcount=1733540, kws=0x0, kwcount=0, defs=0x0, defcount=0, owner=0x2436e4) #48 0xaca64 in readline_internal_charloop () at readline.c:507
at ceval.c:1951 #49 0xaca94 in readline_internal () at readline.c:521
#20 0x285e0 in call_function (func=0x24374c, arg=0x1a606c, kw=0x0) at ceval.c:2772 #50 0xac704 in readline (prompt=0x1cbd9c ">>> ") at readline.c:349
#21 0x28080 in PyEval_CallObjectWithKeywords (func=0x261414, arg=0x18f114, kw=0x0) #51 0x8249c in call_readline (prompt=0x1cbd9c ">>> ") at ./readline.c:462
at ceval.c:2616 #52 0x21ae0 in PyOS_Readline (prompt=0x1cbd9c ">>> ") at myreadline.c:118
#22 0x98064 in PythonCmd (clientData=0x1cc8e0, interp=0x20e658, argc=0, #53 0x205a0 in tok_nextc (tok=0x27abd0) at tokenizer.c:192
argv=0xffbee060) #54 0x20fb4 in PyTokenizer_Get (tok=0x27abd0, p_start=0xffbef8c4, p_end=0xffbef8c0)
at ./_tkinter.c:1274 at tokenizer.c:516
#23 0xff122064 in TclInvokeStringCommand (clientData=0x278538, interp=0x20e658, #55 0x20274 in parsetok (tok=0x27abd0, g=0x17026c, start=256, err_ret=0xffbef9b0)
objc=1, at parsetok.c:128
objv=0x24ec84) at ./../generic/tclBasic.c:1752 #56 0x20158 in PyParser_ParseFile (fp=0x18ebe8, filename=0xbf628 "<stdin>",
#24 0xff13e98c in TclExecuteByteCode (interp=0x20e658, codePtr=0x2a0cd0) g=0x17026c,
at ./../generic/tclExecute.c:845 start=256, ps1=0x1cbd9c ">>> ", ps2=0x25a7e4 "... ", err_ret=0xffbef9b0)
#25 0xff122bf8 in Tcl_EvalObjEx (interp=0x20e658, objPtr=0x2370c8, flags=0) at parsetok.c:75
at ./../generic/tclBasic.c:2723 #57 0x3a9c0 in PyRun_InteractiveOne (fp=0x18ebe8, filename=0xbf628 "<stdin>")
#26 0xff258220 in TkInvokeButton (butPtr=0x279188) at ./../generic/tkButton.c:1457 at pythonrun.c:514
#27 0xff257698 in ButtonWidgetObjCmd (clientData=0x279188, interp=0x20e658, objc=2, #58 0x3a8bc in PyRun_InteractiveLoop (fp=0x18ebe8, filename=0xbf628 "<stdin>")
objv=0x295e00) at ./../generic/tkButton.c:835 at pythonrun.c:478
#28 0xff15e18c in EvalObjv (interp=0x20e658, objc=2, objv=0x295e00, #59 0x3a7ac in PyRun_AnyFileEx (fp=0x18ebe8, filename=0xbf628 "<stdin>", closeit=0)
command=0xff182128 "", at pythonrun.c:453
length=0, flags=262144) at ./../generic/tclParse.c:932 #60 0x3a76c in PyRun_AnyFile (fp=0x18ebe8, filename=0xbf628 "<stdin>") at
#29 0xff15e2b8 in Tcl_EvalObjv (interp=0x20e658, objc=2, objv=0x295e00, pythonrun.c:444
flags=262144) #61 0x1ff20 in Py_Main (argc=3, argv=0xffbefc74) at main.c:297
at ./../generic/tclParse.c:1019 #62 0x1f90c in main (argc=3, argv=0xffbefc74) at python.c:10
#30 0xff122928 in Tcl_EvalObjEx (interp=0x20e658, objPtr=0x2370e0, flags=262144) (gdb)
at ./../generic/tclBasic.c:2565
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 6 beazley@cs.uchicago.edu
7. GDB Traceback
(gdb) where #31 0xff165544 in Tcl_UplevelObjCmd (dummy=0x1, interp=0x20e658, objc=1,
#0 0xff1d9bf0 in __sigprocmask () from /usr/lib/libthread.so.1 objv=0x24ec80)
#1 0xff1ce628 in _resetsig () from /usr/lib/libthread.so.1 at ./../generic/tclProc.c:614
#2 0xff1cdd18 in _sigon () from /usr/lib/libthread.so.1 #32 0xff13e98c in TclExecuteByteCode (interp=0x20e658, codePtr=0x2a0b70)
#3 0xff1d0e8c in _thrp_kill () from /usr/lib/libthread.so.1 at ./../generic/tclExecute.c:845
#4 0xfee49b10 in raise () from /usr/lib/libc.so.1 #33 0xff122bf8 in Tcl_EvalObjEx (interp=0x20e658, objPtr=0x274d50, flags=0)
#5 0xfee3512c in abort () from /usr/lib/libc.so.1 at ./../generic/tclBasic.c:2723
#6 0xfee353d0 in _assert () from /usr/lib/libc.so.1 #34 0xff165afc in TclObjInterpProc (clientData=0x1, interp=0x20e658, objc=0,
#7 0xfeee13ec in abort_crash () from /u0/beazley/Projects/WAD/WAD/Test/./ objv=0xffbeebd8) at ./../generic/tclProc.c:1001
debugmodule.so #35 0xff15e18c in EvalObjv (interp=0x20e658, objc=2, objv=0xffbeebd8,
#8 0xfeee28ec in _wrap_abort_crash () command=0xffbef024 "n tkButtonUp .1907556n", length=25, flags=0)
from /u0/beazley/Projects/WAD/WAD/Test/./debugmodule.so at ./../generic/tclParse.c:932
#9 0x281c8 in call_builtin (func=0x1cc4f0, arg=0x1f9424, kw=0x0) at ceval.c:2650 #36 0xff15e7d0 in Tcl_EvalEx (interp=0x20e658,
#10 0x28094 in PyEval_CallObjectWithKeywords (func=0x1cc4f0, arg=0x1f9424, kw=0x0) script=0xffbef024 "n tkButtonUp .1907556n", numBytes=25, flags=-4264800)
at ceval.c:2618 at ./../generic/tclParse.c:1393
#11 0x26764 in eval_code2 (co=0x1d37e0, globals=0x0, locals=0x1d37cf, args=0x1cc4f0, #37 0xff15e9c0 in Tcl_Eval (interp=0x20e658,
argcount=1762552, kws=0x0, kwcount=0, defs=0x0, defcount=0, owner=0x0) at string=0xffbef024 "n tkButtonUp .1907556n") at ./../generic/tclParse.c:1512
ceval.c:1951 #38 0xff1243d0 in Tcl_GlobalEval (interp=0x20e658,
(gdb) where
#12 0x263a0 in eval_code2 (co=0x1d3858, globals=0x0, locals=0x1cc4f0, args=0x19b1a4, command=0xffbef024 "n tkButtonUp .1907556n") at ./../generic/tclBasic.c:4139
at ceval.c:1850
#0 0xff1d9bf0 in __sigprocmask () from /usr/lib/libthread.so.1
argcount=1883008, kws=0x1d7318, kwcount=0, defs=0x0, defcount=0, owner=0x0) #39 0xff221a40 in Tk_BindEvent (bindingTable=0xffbef024, eventPtr=0x29ffa0,
tkwin=0x2790a8, numObjects=2045728, objectPtr=0xffbef170) at ./../generic/
#1 0xff1ce628 in _resetsig () from /usr/lib/libthread.so.1
#13 0x263a0 in eval_code2 (co=0x1d3e50, globals=0x0, locals=0x19b1a4, args=0x1a7374,
argcount=1883128, kws=0x0, kwcount=0, defs=0x0, defcount=0, owner=0x0) at
tkBind.c:1784
#40 0xff226450 in TkBindEventProc (winPtr=0x2790a8, eventPtr=0x29ffa0)
ceval.c:1850 #2 0xff1cdd18 in _sigon () from /usr/lib/libthread.so.1 at ./../generic/tkCmds.c:244
#14 0x285e0 in call_function (func=0x1a73a4, arg=0x18f114, kw=0x0) at ceval.c:2772 #41 0xff22c218 in Tk_HandleEvent (eventPtr=0x29ffa0) at ./../generic/tkEvent.c:737
#3 0xff1d0e8c in _thrp_kill () from /usr/lib/libthread.so.1
#15 0x28080 in PyEval_CallObjectWithKeywords (func=0x1a73a4, arg=0x18f114, kw=0x0) #42 0xff22c61c in WindowEventProc (evPtr=0x29ff98, flags=-1) at ./../generic/
at ceval.c:2616
#4 0xfee49b10 in raise () from /usr/lib/libc.so.1
#16 0x680b0 in builtin_apply (self=0x0, args=0x0) at bltinmodule.c:88
tkEvent.c:1072
#43 0xff15bb54 in Tcl_ServiceEvent (flags=-1) at ./../generic/tclNotify.c:607
#5 0xfee3512c in abort () from /usr/lib/libc.so.1
#17 0x281c8 in call_builtin (func=0x1910c8, arg=0x1f9b54, kw=0x0) at ceval.c:2650
#18 0x28094 in PyEval_CallObjectWithKeywords (func=0x1910c8, arg=0x1f9b54, kw=0x0)
#44 0xff15beec in Tcl_DoOneEvent (flags=-1) at ./../generic/tclNotify.c:846
#45 0x99314 in EventHook () at ./_tkinter.c:2020
at ceval.c:2618#6 0xfee353d0 in _assert () from /usr/lib/libc.so.1 #46 0xbaf30 in rl_read_key () at input.c:374
#19 0x26764 in eval_code2 (co=0x1f3948, globals=0x0, locals=0x1f38f0, args=0x1910c8, #47 0xac920 in readline_internal_char () at readline.c:454
#7 0xfeee13ec in abort_crash () from /u0/beazley/Projects/WAD/WAD/Test/
argcount=1733540, kws=0x0, kwcount=0, defs=0x0, defcount=0, owner=0x2436e4) #48 0xaca64 in readline_internal_charloop () at readline.c:507
at ceval.c:1951
./debugmodule.so
#20 0x285e0 in call_function (func=0x24374c, arg=0x1a606c, kw=0x0) at ceval.c:2772
#49 0xaca94 in readline_internal () at readline.c:521
#50 0xac704 in readline (prompt=0x1cbd9c ">>> ") at readline.c:349
at ceval.c:2616
#8 0xfeee28ec in _wrap_abort_crash ()
#21 0x28080 in PyEval_CallObjectWithKeywords (func=0x261414, arg=0x18f114, kw=0x0) #51 0x8249c in call_readline (prompt=0x1cbd9c ">>> ") at ./readline.c:462
#52 0x21ae0 in PyOS_Readline (prompt=0x1cbd9c ">>> ") at myreadline.c:118
from /u0/beazley/Projects/WAD/WAD/Test/./debugmodule.so
#22 0x98064 in PythonCmd (clientData=0x1cc8e0, interp=0x20e658, argc=0, #53 0x205a0 in tok_nextc (tok=0x27abd0) at tokenizer.c:192
argv=0xffbee060) #54 0x20fb4 in PyTokenizer_Get (tok=0x27abd0, p_start=0xffbef8c4, p_end=0xffbef8c0)
#9 0x281c8 in call_builtin (func=0x1cc4f0, arg=0x1f9424, kw=0x0) at
at ./_tkinter.c:1274 at tokenizer.c:516
objc=1,
ceval.c:2650
#23 0xff122064 in TclInvokeStringCommand (clientData=0x278538, interp=0x20e658, #55 0x20274 in parsetok (tok=0x27abd0, g=0x17026c, start=256, err_ret=0xffbef9b0)
at parsetok.c:128
#10 0x28094 in PyEval_CallObjectWithKeywords (func=0x1cc4f0,
objv=0x24ec84) at ./../generic/tclBasic.c:1752
#24 0xff13e98c in TclExecuteByteCode (interp=0x20e658, codePtr=0x2a0cd0)
#56 0x20158 in PyParser_ParseFile (fp=0x18ebe8, filename=0xbf628 "<stdin>",
g=0x17026c,
arg=0x1f9424, kw=0x0)
at ./../generic/tclExecute.c:845 start=256, ps1=0x1cbd9c ">>> ", ps2=0x25a7e4 "... ", err_ret=0xffbef9b0)
#25 0xff122bf8 in Tcl_EvalObjEx (interp=0x20e658, objPtr=0x2370c8, flags=0) at parsetok.c:75
at ceval.c:2618
at ./../generic/tclBasic.c:2723 #57 0x3a9c0 in PyRun_InteractiveOne (fp=0x18ebe8, filename=0xbf628 "<stdin>")
#26 0xff258220 in TkInvokeButton (butPtr=0x279188) at ./../generic/tkButton.c:1457 at pythonrun.c:514
#27 0xff257698 in ButtonWidgetObjCmd (clientData=0x279188, interp=0x20e658, objc=2, #58 0x3a8bc in PyRun_InteractiveLoop (fp=0x18ebe8, filename=0xbf628 "<stdin>")
objv=0x295e00) at ./../generic/tkButton.c:835 at pythonrun.c:478
#28 0xff15e18c in EvalObjv (interp=0x20e658, objc=2, objv=0x295e00, #59 0x3a7ac in PyRun_AnyFileEx (fp=0x18ebe8, filename=0xbf628 "<stdin>", closeit=0)
command=0xff182128 "", at pythonrun.c:453
length=0, flags=262144) at ./../generic/tclParse.c:932 #60 0x3a76c in PyRun_AnyFile (fp=0x18ebe8, filename=0xbf628 "<stdin>") at
#29 0xff15e2b8 in Tcl_EvalObjv (interp=0x20e658, objc=2, objv=0x295e00, pythonrun.c:444
flags=262144) #61 0x1ff20 in Py_Main (argc=3, argv=0xffbefc74) at main.c:297
at ./../generic/tclParse.c:1019 #62 0x1f90c in main (argc=3, argv=0xffbefc74) at python.c:10
#30 0xff122928 in Tcl_EvalObjEx (interp=0x20e658, objPtr=0x2370e0, flags=262144) (gdb)
at ./../generic/tclBasic.c:2565
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 7 beazley@cs.uchicago.edu
8. Debugging Problems
General problem
• Traditional debugger doesn’t know anything about Python scripts.
• Mostly provides information about the implementation of Python.
• Can’t fully answer question of “how did I get here?”
• A problem if you have a lot of Python code.
Sometimes it is hard to reproduce a problem
• Run-time environment may be complex.
• Problems may be due to timing or precise event sequences.
• Problem may only occur after a long period of time.
Other issues
• Requires users to run a separate application (very unpython).
• Requires users to have a C development environment installed.
• Assumes users know how to use the C debugger.
Claim: I think you can do better
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 8 beazley@cs.uchicago.edu
9. WAD
Wrapped Application Debugger
• Idea: Maybe you could turn fatal extension errors into Python Exceptions
• Seg faults, bus errors, illegal instructions, failed assertions, and math errors.
Python Python
DeathException
Extension WAD Extension
Code Code
Death
Demo
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 9 beazley@cs.uchicago.edu
10. WAD Demo
% python
>>> import debug
>>> debug.seg_crash()
Segmentation fault (core dumped)
%
% python
>>> import debug
>>> import libwadpy
WAD Enabled
>>> debug.seg_crash()
Traceback (most recent call last):
File "<stdin>", line 1, in ?
SegFault: [ C stack trace ]
#2 0x000281c0 in call_builtin(func=0x1cbaf0,arg=0x18f114,kw=0x0) in 'ceval.c', line
2650
#1 0xfeee26b8 in _wrap_seg_crash(self=0x0,args=0x18f114) in 'pydebug.c', line 510
#0 0xfeee1258 in seg_crash(0x1,0xfeef2d48,0x19a9f8,0x0,0x7365675f,0x5f5f6469) in
'debug.c', line 18
/u0/beazley/Projects/WAD/WAD/Test/debug.c, line 18
int seg_crash() {
int *a = 0;
=> *a = 3;
return 1;
}
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 10 beazley@cs.uchicago.edu
11. Big Picture
WAD
• WAD is a dynamically loadable Python extension module.
• Converts catastrophic errors to Python exceptions.
Key features
• No modifications to Python
• No modifications to extensions.
• No recompilation.
• No relinking.
• No separate debugger required (gdb, dbx, etc.)
• No C, C++, Fortran development environment needed.
• No added performance penalty.
The rest of this talk
• Using WAD
• Gory implementation details
• Limitations
• Future directions.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 11 beazley@cs.uchicago.edu
12. Using WAD
1. Explicit import
import libwadpy
2. Implicit linking
ld -shared $(OBJS) -o foomodule.so -lwadpy
• Automatically loads WAD when the extension is loaded.
What WAD provides
• 4 new Python exceptions (SegFault, BusError, AbortError, IllegalInstruction)
• Exceptions are added to __builtin__ module.
• A new Python type (WadObject). Returned as an exception value.
• Otherwise, no public functions, constants, or variables (libwadpy is empty).
• Also: WAD is completely self contained
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 12 beazley@cs.uchicago.edu
13. Exception Handling with WAD
Just like ordinary Python exception handling
• Except that you get a much more interesting exception value
try:
naughty bits
except SegFault, s:
t = s.args[0] # Get trace object
print t # Prints stack trace
len(t) # Number of stack frames
f = s[3] # Returns a stack frame
f.__FILE__ # Source file
f.__LINE__ # Source line
f.__EXE__ # Object file
f.__PC__ # Program counter
f.__STACK__ # Raw stack frame
...
f.name # Value of parameter or local name
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 13 beazley@cs.uchicago.edu
14. Implementation Overview
Unix signal handling
• SIGSEGV
• SIGBUS
• SIGABRT
• SIGFPE
• SIGILL
Process introspection
• Discovering program context.
• Reading of object files
• Collection of debugging data
Abort and return to Python
• How do you actually get back to the interpreter?
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 14 beazley@cs.uchicago.edu
15. Control Flow
>>> foo.spam()
Python Internals
call_builtin()
Extension
Code
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 15 beazley@cs.uchicago.edu
16. Control Flow (cont)
>>> foo.spam()
Python Internals
call_builtin()
WAD
1. Examine call stack
Extension 2. Collect debugging information.
Code 3. Look for safe place to return.
SIGSEGV
4. Raise Python exception
5. Abort execution.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 16 beazley@cs.uchicago.edu
17. Control Flow (cont)
>>> foo.spam()
Traceback (most recent call last):
File "spam.py", line 22, in ?
foo.spam()
SegFault: [ C stack trace ]
Python Internals
call_builtin()
NULL
WAD
1. Examine call stack
Extension 2. Collect debugging information.
Code 3. Look for safe place to return.
SIGSEGV
4. Raise Python exception
5. Abort execution.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 17 beazley@cs.uchicago.edu
18. Signal Handling
Traditional Signal Handling
void seg_handler(int signo) {
...
printf("Aiiee!!!!");
...
return;
}
void foo() {
...
signal(SIGSEGV, seg_handler);
...
SIGSEGV naughty bits
...
}
• Signal handler executes on error.
• Unfortunately, execution resumes at point of error (and repeats).
• Note: Python signal module can’t handle SEGSEGV and related signals.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 18 beazley@cs.uchicago.edu
19. Signal Handling
Advanced Signal Handling
void seg_handler(signo, siginfo, context) {
printf("Aiiee!!!!");
...
modify context
return;
}
void foo() { bar() {
... ...
sigaction(SIGSEGV, ...); nice bits
... ...
SIGSEGV naughty bits }
...
}
• Rarely used form of sigaction() allows signal handler to modify context
• Includes all CPU registers, program counter (PC), stack pointer (SP)
• Changes take effect on return from signal handler.
• Normally used to implement user-level thread libraries.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 19 beazley@cs.uchicago.edu
20. WAD: In a Nutshell
Signal handling + context rewriting
• Signal handler collects process information.
• Raise Python exception.
• Rewrite process context so that Python interpreter regains control.
• Return from signal handler.
Issues
• How do you perform process introspection?
• How do you figure out where to return in Python?
• How do you abort execution without breaking the universe?
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 20 beazley@cs.uchicago.edu
21. Finding Program Context
stack trace (PC) call stack
0x0001f7c4 ...
0x0001f904
0x0001fedc
0x0003a7c8 0xfee350e4
0x0003ac1c
0x0003b77c
0x0003b7a8
0x0003b7f8 0xfee49b08
0x000237e8
0x0002675c
0x0002808c
0x000281c0 Stack
0xfeee241c
0xfeee1178 Frame
PC 0xff1d0e84
0x000bee2c
0xfee350e4
0xfee49b08
0xff1d0e84
0xff1cdd10 0xff1cdd10
signal 0xff1d9bf0 SP
1. Generate raw stack trace
• A very simple while loop.
• Get sequence of PC, SP values and stack frames.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 21 beazley@cs.uchicago.edu
22. Finding Program Context
stack trace (PC) process memory map
0x0001f7c4
0x0001f904 python
0x0001fedc
0x0003a7c8
0x0003ac1c
0x0003b77c
0x0003b7a8
0x0003b7f8
0x000237e8
0x0002675c
0x0002808c
0x000281c0
0xfeee241c
0xfeee1178
0x000bee2c
0xfee350e4 libc.so
0xfee49b08 foomodule.so
0xff1d0e84
0xff1cdd10 libthread.so
signal 0xff1d9bf0
2. Read process memory map from /proc
• Get base/bounds for Python executable, all shared libraries, heap, stack, etc.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 22 beazley@cs.uchicago.edu
23. Program Context
stack trace (PC) process memory map
0x0001f7c4
0x0001f904 python
0x0001fedc
0x0003a7c8
0x0003ac1c
0x0003b77c
0x0003b7a8
0x0003b7f8
0x000237e8
0x0002675c
0x0002808c
0x000281c0
0xfeee241c
0xfeee1178
0x000bee2c
0xfee350e4 libc.so
0xfee49b08 foomodule.so
0xff1d0e84
0xff1cdd10 libthread.so
signal 0xff1d9bf0
3. Map stack trace to memory map
• Determines the module associated with each stack frame.
• Note: memory map also used to validate the stack trace.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 23 beazley@cs.uchicago.edu
24. Program Context
C stack trace (PC) process memory map symbol names
0x0001f7c4 _start
0x0001f904 python main
0x0001fedc Py_Main
0x0003a7c8 PyRun_AnyFileEx
0x0003ac1c PyRun_SimpleFileEx
0x0003b77c PyRun_FileEx
0x0003b7a8 run_err_node
0x0003b7f8 run_node
0x000237e8 PyEval_EvalCode
0x0002675c eval_code2
0x0002808c PyEval_CallObjectW
0x000281c0 call_builtin
0xfeee241c _wrap_spam
0xfeee1178 spam
0x000bee2c __eprintf
0xfee350e4 libc.so abort
0xfee49b08 foomodule.so raise
0xff1d0e84 _thrp_kill
0xff1cdd10 libthread.so _sigon
signal 0xff1d9bf0 __sigprocmask
4. Map to symbolic names
• Read ELF symbol table from object files in memory map
• Symbols defined by a simple (name, base, size) triple.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 24 beazley@cs.uchicago.edu
25. Gathering Debugging Information
Items of interest
• Source filename
• Source line number
• Function parameters (names, values)
• Local variables (names, values).
Debugging information is stored in object files
• If code compiled with -g
• However, debugging data is not loaded into memory during execution.
Collection strategy
• Load all object files found in process memory map.
• Search for debugging data for each symbol in the stack trace.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 25 beazley@cs.uchicago.edu
26. Gathering Debugging Information
Stabs data (simplified)
Type Value Desc String
source file 0x64 /home/beazley/blah/
0x64 foo.c
...
int spam(int n) { 0x80 int:t(0,1)=bs4;0;32
0x80 short:t(0,2)=bs2;0;16
short temp; ...
...
error assert(n > 0); 0x24 10 spam:F(0,1)
0xa0 10 n:p(0,1)
...
0x80 11 temp:(0,2)
} 0x44 16 12
0x44 28 13
...
0x44 112 23
Symbol offsets ...
Line numbers
STABS
• Language neutral specification of source information.
• Includes locations, types, functions, parameters, locals, line numbers, etc.
• Decoding is a major head explosion (and that’s all I will say about it).
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 26 beazley@cs.uchicago.edu
27. Final Result
Get a C data structure representing program state
name "spam"
source "/u0/beazley/WAD/Test/spam.c"
line 42
args [ ("n", -1) ]
stack < raw stack data >
next
name "wrap_spam"
source "/u0/beazley/WAD/Test/spam_wrap.c"
line 1782
args [ ("self", 0x0), ("args", 0x1782308) ]
stack < raw stack data >
next
name "call_builtin"
source "/public/software/Python-2.0/Python/ceval.c"
line 2650
args [("func",0x1cc2d8),("self",0x0),("args,...)]
stack < raw stack data >
next
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 27 beazley@cs.uchicago.edu
28. Returning to Python
Step 1: Examine stack trace for a suitable return point
Call Stack Return Table
_start Function name Value
main
Py_Main
PyRun_AnyFileEx call_builtin 0
PyRun_SimpleFileEx PyObject_GetattrString 0
PyRun_FileEx PyObject_SetattrString -1
run_err_node ...
run_node
PyEval_EvalCode
eval_code2
PyEval_CallObjectW
call_builtin = Return value = 0 (NULL)
_wrap_spam
spam
__eprintf • Table contains Python functions that call ext. code.
abort • Search looks for first function found on stack.
raise
_thrp_kill • Return value used when raising exceptions (e.g.,
_sigon return NULL on error.)
__sigprocmask
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 28 beazley@cs.uchicago.edu
29. Returning to Python
Step 2: Raise an exception
• If no valid Python return function, print stack trace and exit.
• Otherwise, raise Python exception.
• SegFault, AbortError, BusError, IllegalInstruction, Floating-
PointError
Exception value
• Is a special Python type WadObject
• Contains entire stack trace and all data collected.
• Is really just a wrapper around the C data structure described earlier.
• str() and repr() methods simply dump the stack trace as a string.
• Other methods provide access to raw data.
try:
# some naughty extension code
except SegFault, s:
print "Whoa!"
print s # Dump a stack trace
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 29 beazley@cs.uchicago.edu
30. Returning to Python
Step 3: Modify process context and return
• Chop off the call stack and return with an error/exception
Call Stack
_start _start
main main
Py_Main Py_Main
PyRun_AnyFileEx PyRun_AnyFileEx
PyRun_SimpleFileEx PyRun_SimpleFileEx
PyRun_FileEx PyRun_FileEx
run_err_node run_err_node
run_node run_node
PyEval_EvalCode PyEval_EvalCode
eval_code2 eval_code2
PyEval_CallObjectW PyEval_CallObjectW
call_builtin call_builtin
_wrap_spam
spam
__eprintf NULL, AbortError
abort return from signal
raise
_thrp_kill
_sigon
__sigprocmask
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 30 beazley@cs.uchicago.edu
31. A "Slight" Complication
Return mechanism is similar to:
• setjmp/longjmp in C
• C++ exception handling.
However...
• Python is not instrumented or modified in any way.
• There is no corresponding setjmp() call.
• There is no matching try { ... } clause in C++.
This means...
• We are returning to some "arbitrary" location in the Python executable.
• Never designed with such a non-local procedure return in mind.
This is a bit of a problem:
• Corrupted CPU registers.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 31 beazley@cs.uchicago.edu
32. The Register Save Problem
CPU Registers Call Stack
foo() {
foo
... locals
call bar
...
}
bar() {
...
do stuff
...
}
Each procedure uses CPU registers.
•Temporaries, local variables, memory addressing, etc.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 32 beazley@cs.uchicago.edu
33. Register Save (cont)
CPU Registers Call Stack
foo() {
foo
... locals
save_caller_regs
call bar
restore_caller_regs
...
}
bar() {
...
do stuff
...
}
Caller-save
•Must save certain registers before calling a new procedure.
•Restore after procedure returns.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 33 beazley@cs.uchicago.edu
34. Register Save (cont)
CPU Registers Call Stack
foo() {
foo
... locals
save_caller_regs
call bar
restore_caller_regs
... bar
locals
}
bar() {
save_callee_regs
...
do stuff
...
restore_callee_regs
}
Callee-save
•Procedures save registers they plan to overwrite.
•Restore values prior to returning
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 34 beazley@cs.uchicago.edu
35. Register Save (cont)
CPU Registers Call Stack
foo() {
foo
... locals
save_caller_regs
call bar
restore_caller_regs
... bar
locals
} return
bar() {
save_callee_regs
...
do stuff
...
restore_callee_regs
}
Procedure return
•Callee restores registers
•Caller restores registers
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 35 beazley@cs.uchicago.edu
36. Register Save (cont)
CPU Registers Call Stack
foo() {
foo
... locals
save_caller_regs
call bar
restore_caller_regs
... bar
locals
} corrupted
bar() {
save_callee_regs
...
SIGSEGV naughty bits
... abort
restore_callee_regs
}
Aborted return
•Callee-saved register values are lost (never restored)
•Corrupts CPU state in caller on return (this is usually bad)
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 36 beazley@cs.uchicago.edu
37. Register Restoration
Solution: SPARC
•Each procedure gets a fresh set of CPU registers (i.e., a “window”)
•To restore state: simply roll back the register windows
Solution: i386
•Manually inspect machine code of function prologues
•Figure out where callee-save registers are saved on call-stack
•Restore values while walking up the call stack.
blah:
55 pushl %ebp
89 e5 movl %esp,%ebp
83 ec 2c subl $0x2c,%esp Size of locals
57 pushl %edi
56 pushl %esi
Saved registers
53 pushl %ebx
• Only a heuristic. Might get it wrong, but the return to Python may still work.
• Not as bad as it sounds---implementation is fairly simple.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 37 beazley@cs.uchicago.edu
38. The Auto-Initialization Hack
One final bit...
• How do you get WAD to initialize itself when linked to extensions?
class WadInit {
public:
WadInit() {
wad_initialize();
}
};
static WadInit winit;
• Dynamic link/loader automatically invokes C++ static constructors on import.
• Constructors are invoked before any extension code executes.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 38 beazley@cs.uchicago.edu
39. Implementation Details
Implementation
• Mostly ANSI C, some assembly, some C++
• ~1500 semicolons
• Most code related to introspection (debugging, symbol tables, etc...)
• Core is Python independent (only 166 semicolons related to Python).
• Execution is isolated (own stack and memory management).
• Does not rely upon third party libraries (e.g., libbfd).
Compatibility
• Sun Sparc Solaris
• i386 Linux (recent kernels).
• Python 1.5 and newer (class based exceptions)
• Miscellaneous compatibility issues on Linux.
• Also supports Tcl.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 39 beazley@cs.uchicago.edu
40. Limitations
Non-local return, aborted execution
• May leak memory
• No destruction of objects in C++.
• May interact poorly with C++ exceptions.
• May result in unreleased system resources (files, sockets, etc.).
• May result in deadlock (if holding locks when error occurs).
Unrecoverable errors
• Extensions that destroy or corrupt Python interpreter data.
• Stack overflow (results in corrupted call-stack).
Compiler optimization
• False reporting of debugging data, source files, and lines.
• Incorrect register recovery (-fomit-frame-pointer)
Compatibility
• Mixing threads and signals is extremely problematic.
• WAD requires fully functional signal implementation.
• Some versions of Linux, Linux+Threads do not work.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 40 beazley@cs.uchicago.edu
41. More Limitations
Debugging information
• Only simple datatypes are currently understood.
• No special C++ support (classes, name demangling, etc.)
• No understanding of structures.
Things that just don’t work
• Breakpoints
• Single-step execution.
• Restart
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 41 beazley@cs.uchicago.edu
42. Related Work
Surprisingly little literature on this topic
• PyDebug.
• Programming environments for Common Lisp (FFI).
• Asynchronous exception handling (ML, Haskell)
• Rn (A mixed interpreter-compiled system for Fortran)
• Modifications to gdb for debugging Common Lisp (WCL).
• Java mixed-mode debugging (Java + JNI). ???
• Perl (sigtrap module can print perl stack trace on fatal error).
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 42 beazley@cs.uchicago.edu
43. Future Directions
Better error recovery and data reporting
• Make the WAD core as generic as possible.
• Better heuristics for certain errors (corrupted call stack, corrupted heap).
• Improved collection of debugging information.
Support for more platforms
• Obviously. Maybe. Not.
Integration with Python debugger, IDEs?
• Demo.
Other languages
• Tcl, Ruby, Perl, etc. (Tcl works now).
Bizarre execution modes?
• Restarts?
• Breakpoints?
• Code patching?
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 43 beazley@cs.uchicago.edu
44. Conclusions
Extension programming
• A lot of people are building extensions.
• Debugging has always been a little annoying.
Conventional wisdom
• Modify an existing debugger to understand Python.
• Why reinvent the wheel (especially debuggers)?
Why not reevaluate the situation?
• Traditional debugging model is awkward for extension programming.
• Exception handling approach is cool and fits in nicely with Python scripts.
• Simply knowing where code crashed is enough to fix a lot of bugs.
• The exception approach is also nice when distributing extensions.
Bottom line: WAD is mostly a proof of concept
• Common extension errors can be handled within Python.
• Can extend Python exception handling to compiled extensions.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 44 beazley@cs.uchicago.edu
45. More Information
http://systems.cs.uchicago.edu/wad
• This is work in progress.
• Not ready for prime time yet.
• Many related problems to work on.
• Volunteers welcome.
• I’m also looking for students.
WAD : 9th International Python Conference, Long Beach, California, March 6, 2001 45 beazley@cs.uchicago.edu