Debugging Your Python With GDB (FTW!)
In this post we’ll take a look at how to debug your Python code using GDB. It is a handy thing to understand, especially if you’re confronted with an unexpected SEGV or other less than helpful error. I do realize there is some awesome python-gdb.py integration with GDB. I purposefully ignored that. Sometimes all ya have is the binary.
As an unfortunate note, I started doing this using Python 3.3, but at some point, I switched to 2.6 accidentally. I’ve migrated the earlier pieces to 2.6. If anyone smarter than I notices an inconsistency, this is why. I’m fairly certain I’ve cleaned it all up.
Finally, the GDB formatting is mine. I attempted to make it slightly more readable. Hope it helps.
Update: One thing I forgot to mention was the set of GDB macros that come with the Python source. That automates a good bit of the mechanics outlined here. Thanks, Evgeny!
How Does Python Evaluate Code?
First, a little bit of background. Python implements a stack-based virtual machine. Python byte code manipulates that stack during normal execution. For example, let’s take a look at a small application disassembled into byte code:
a = 1 b = 2 c = a + b print c
This is a fairly trivial example that should show us a good sampling of the “instruction set.” We’re going to skim over this bit as understanding all of the byte code operations really isn’t a necessity here. When we use the dis module, we see that the following code is generated:
jeff@martian:~/cpython$ /usr/bin/python -mdis add.py
1 0 LOAD_CONST 0 (1)
3 STORE_NAME 0 (a)
2 6 LOAD_CONST 1 (2)
9 STORE_NAME 1 (b)
3 12 LOAD_NAME 0 (a)
15 LOAD_NAME 1 (b)
18 BINARY_ADD
19 STORE_NAME 2 (c)
4 22 LOAD_NAME 2 (c)
25 PRINT_ITEM
26 PRINT_NEWLINE
27 LOAD_CONST 2 (None)
30 RETURN_VALUE
This is fairly self explanatory. We see at position 1 that the constants 1 & 2 are placed into a & b. Next, they’re placed on the stack and BINARY_ADD is called, which triggers the addition of two number objects. Next, STORE_NAME saves the value of the add operation (from the top of the stack) to the location c. Finally, we load c and call the print operations. In Python 3, this would simply call the print function, via CALL_FUNCTION. For an overview of how Python generates bytecode from Python code, see Python/compile.c. The comment at the top of the file is quite helpful.
Using Python 2.6 as a reference point, all of this happens at Python/ceval.c. The function handling byte code execution is named PyEval_EvalFrameEx. Generally, this is a big switch statement. I use the term switch loosely as it is actually a collection of computed goto labels on both Mac OS and Linux (Visual Studio doesn’t allow that).
Looking at this function, you’ll see various entries such as this;
case POP_TOP:
v = POP();
Py_DECREF(v);
goto fast_next_opcode;
This is the implementation for the POP_TOP instruction. The POP macro returns the top value of the stack and the subsequent Py_DECREF(v) decrements the reference count. At this point, that could trigger execution of v->ob_type->tp_del & v->ob_type->tp_dealloc, if the reference count of v (v->ob_refcnt) has reached zero. As an aside, note that Python checks for events/thread switches every sys.getcheckinterval() instructions. If the corresponding implementation of an instruction is complex (and doesn’t release the GIL), we can be left waiting here.
Now, we come to the function we’re interested in:
PyObject * PyEvalCodeEx(PyObject *co, PyObject *globals, PyObject
*locals, PyObject **args, int argcount, PyObject **kws, int
kwcount, PyObject **defs, int defcount, PyObject *closure);
Essentially, this function builds a frame from the code object being executed and relies on PyEval_PyEvalFrameEx to handle bytecode instruction evaluation. The code object contains references to globals, locals, nested scopes (free vars/cell vars, depending on the angle), etc. PyEvalCodeEx “transforms” that into a PyFrameObject.
It is this code object evaluation function we’re interested in as functions and methods are generally boiled down to code objects.
Python Data Structure Data Structures
Now that we’ve covered where to look, we need to take a look at what to look for. This means building a bit of an understanding around a few data structures.
Type Objects
All of Python’s classes (well, almost) are represented by PyTypeObject objects, which is defined in Python/Include/Object.h. This structure contains a whole lot of fields. Most of these fields will be pretty familiar looking as this is generally how “dunder”, or __methods__ , are implemented. Standard, generic values are used (see PyType_Ready) if you don’t setup your own. This is a long structure, but including it here is relevant:
typedef struct _typeobject {
PyObject_VAR_HEAD
const char *tp_name; /* For printing, in format "." */
Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */
/* Methods to implement standard operations */
destructor tp_dealloc;
printfunc tp_print;
getattrfunc tp_getattr;
setattrfunc tp_setattr;
cmpfunc tp_compare;
reprfunc tp_repr;
/* Method suites for standard classes */
PyNumberMethods *tp_as_number;
PySequenceMethods *tp_as_sequence;
PyMappingMethods *tp_as_mapping;
/* More standard operations (here for binary compatibility) */
hashfunc tp_hash;
ternaryfunc tp_call;
reprfunc tp_str;
getattrofunc tp_getattro;
setattrofunc tp_setattro;
/* Functions to access object as input/output buffer */
PyBufferProcs *tp_as_buffer;
/* Flags to define presence of optional/expanded features */
long tp_flags;
const char *tp_doc; /* Documentation string */
/* Assigned meaning in release 2.0 */
/* call function for all accessible objects */
traverseproc tp_traverse;
/* delete references to contained objects */
inquiry tp_clear;
/* Assigned meaning in release 2.1 */
/* rich comparisons */
richcmpfunc tp_richcompare;
/* weak reference enabler */
Py_ssize_t tp_weaklistoffset;
/* Added in release 2.2 */
/* Iterators */
getiterfunc tp_iter;
iternextfunc tp_iternext;
/* Attribute descriptor and subclassing stuff */
struct PyMethodDef *tp_methods;
struct PyMemberDef *tp_members;
struct PyGetSetDef *tp_getset;
struct _typeobject *tp_base;
PyObject *tp_dict;
descrgetfunc tp_descr_get;
descrsetfunc tp_descr_set;
Py_ssize_t tp_dictoffset;
initproc tp_init;
allocfunc tp_alloc;
newfunc tp_new;
freefunc tp_free; /* Low-level free-memory routine */
inquiry tp_is_gc; /* For PyObject_IS_GC */
PyObject *tp_bases;
PyObject *tp_mro; /* method resolution order */
PyObject *tp_cache;
PyObject *tp_subclasses;
PyObject *tp_weaklist;
destructor tp_del;
/* Type attribute cache version tag. Added in version 2.6 */
unsigned int tp_version_tag;
#ifdef COUNT_ALLOCS
/* these must be last and never explicitly initialized */
Py_ssize_t tp_allocs;
Py_ssize_t tp_frees;
Py_ssize_t tp_maxalloc;
struct _typeobject *tp_prev;
struct _typeobject *tp_next;
#endif
} PyTypeObject;
The typedef (typedefs? Anyone know the plural of #typedef?) above (i.e. PyNumberMethods) are the C-level equivalent of the double underscore methods required to implement a certain protocol (programmatic interface). They expand into method collections:
typedef struct {
lenfunc mp_length;
binaryfunc mp_subscript;
objobjargproc mp_ass_subscript;
} PyMappingMethods;
These translate into len, subscript, and subscript assignment.
Instances
All Python instances are all implemented as pointers to PyObject values, which is defined as:
typedef struct _object {
PyObject_HEAD
} PyObject;
PyObject_HEAD, by default, expands to include only a pointer to the object’s type (type objects have a type of type!) and the reference count.
/* PyObject_HEAD defines the initial segment of every PyObject. */
#define PyObject_HEAD \
_PyObject_HEAD_EXTRA \
Py_ssize_t ob_refcnt; \
struct _typeobject *ob_type;
Wait! Where is all of the per-instance data you say? For classes that do not define __slots__, there is a dictoffset member of the corresponding PyTypeObject structure. This provides the address, via offset from the end of the PyObject structure, that contains a Python dictionary. This is the __dict__ used to store per instance information. If __slots__ is defined, then dictoffset is NULL and the slot values are stored at the end of the PyObject structure and accessed via descriptors. Generic structures are passed around via casting (and turned back into concrete values via the same method).
Somewhat related bonus Python trivia: The class dictionary is actually a PyDictProxy_Type that refers to the type’s tp_dict field. You can’t edit it directly.
To clarify, assuming we have a type NinjaTurtle that is represented by PyTypeObject *ninja, then for an instance donatello, the following is true: (PyObject *)donatello->ob_type = ninja; Good. So, naturally, to perform an init call, the corresponding code would like like the following:
donatello->ob_type->tp_init((PyObject *)donatello);
In fact, this is almost exactly what happens when a type is called directly (ala class instantiation: MyClass()).
Code Objects
Let’s look at one final object, the code object. This is represented by a structure defined in code.h. It is rather simple object (though note the first member).
/* Bytecode object */
typedef struct {
PyObject_HEAD
int co_argcount; /* #arguments, except *args */
int co_nlocals; /* #local variables */
int co_stacksize; /* #entries needed for evaluation stack */
int co_flags; /* CO_..., see below */
PyObject *co_code; /* instruction opcodes */
PyObject *co_consts; /* list (constants used) */
PyObject *co_names; /* list of strings (names used) */
PyObject *co_varnames; /* tuple of strings (local variable names) */
PyObject *co_freevars; /* tuple of strings (free variable names) */
PyObject *co_cellvars; /* tuple of strings (cell variable names) */
/* The rest doesn't count for hash/cmp */
PyObject *co_filename; /* string (where it was loaded from) */
PyObject *co_name; /* string (name, for reference) */
int co_firstlineno; /* first source line number */
PyObject *co_lnotab; /* string (encoding addr<->lineno mapping) See
Objects/lnotab_notes.txt for details. */
void *co_zombieframe; /* for optimization only (see frameobject.c) */
PyObject *co_weakreflist; /* to support weakrefs to code objects */
} PyCodeObject;
From here, we can switch into Python. Note the above fields and then have a peek at a function’s func_code attribute (__code__ in 3.x):
>>> >>> def f(): pass ... [66987 refs] >>> import pprint [67863 refs] >>> pprint.pprint(dir(f.func_code)) ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_kwonlyargcount', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames'] [67870 refs] >>>
Perfect. Now we’ve made the connection between Python and C. Now we can take a look at the actual debugging process.
GDB’ing the Py.
We’ll use the same small bit of code we used above as our test script. We’re referencing /usr/bin/python here, which may vary on your system.
First, we’ll start the interpreter. Note that we’re debugging Python itself, not the script passed to it. GDB will not start if we pass in the Python script as the executable.
jeff@martian:~/cpython$ gdb /usr/bin/python GNU gdb (GDB) 7.4-gg1 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux". Reading symbols from /usr/bin/python... Reading symbols from /usr/lib/debug/usr/bin/python...done. done.
Now we’ll set the appropriate args for the execution of Python — our script. Note that nothing is running at this point.
(gdb) set args add.py
Now, since we want to see how to pick apart the location of our Python code from the C level, we’ll set a breakpoint at PyEval_EvalCodeEx. This forces GDB to up and stop when it gets to our function.
(gdb) break PyEval_EvalCodeEx Breakpoint 1 at 0x80e1f53: file ../../../Python/ceval.c, line 2767. (gdb)
Note that if the correct source is available, this gets much easier as there is Python+GDB integration available via python-gdb.py. Now, we can run the executable:
(gdb) run Starting program: /usr/bin/python add.py Breakpoint 1, PyEval_EvalCodeEx (co=0xf7de7338, globals=0xf7df313c, locals=0xf7df313c, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../../../Python/ceval.c:2767 2767 ../../../Python/ceval.c: No such file or directory.
Understanding the Object Representation
From here, we can examine the code in question. First, let’s print the value of the first argument to PyEval_EvalCodeEx. From our prototype above, we know this is a code object:
(gdb) p *co
$1 = {ob_refcnt = 1, ob_type = 0x81a1e60, co_argcount = 0,
co_nlocals = 0, co_stacksize = 1, co_flags = 64,
co_code = 0xf7dedd40, co_consts = 0xf7dedd0c,
co_names = 0xf7dc102c, co_varnames = 0xf7dc102c,
co_freevars = 0xf7dc102c, co_cellvars = 0xf7dc102c,
co_filename = 0xf7de4200, co_name = 0xf7dedd60,
co_firstlineno = 1, co_lnotab = 0xf7dc10b0, co_zombieframe = 0x0}
(gdb)
Here, we see the ob_refcnt and the ob_type. If we cast this to a PyObject *, you’ll see that it only prints that information.
(gdb) p *(PyObject *)co
$4 = {ob_refcnt = 1, ob_type = 0x81a1e60}
(gdb)
Ok, let’s step ahead until we see something interesting. We’ll “GDB continue” until we have an args=<value> which is not 0×0, or NULL. We’ll look at the following frame:
Breakpoint 1, PyEval_EvalCodeEx (co=0xf7d8cc80, globals=0xf7d8a35c, locals=0x0, args=0x81bfe7c, argcount=0, kws=0x81bfe7c, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../../../Python/ceval.c:2767 2767 in ../../../Python/ceval.c (gdb) info frame Stack level 0, frame at 0xfffec7a0: eip = 0x80e1f53 in PyEval_EvalCodeEx (../../../Python/ceval.c:2767); saved eip 0x80e0cd2 called by frame at 0xfffec890 source language c. Arglist at 0xfffec798, args: co=0xf7d8cc80, globals=0xf7d8a35c, locals=0x0, args=0x81bfe7c, argcount=0, kws=0x81bfe7c, kwcount=0, defs=0x0, defcount=0, closure=0x0 Locals at 0xfffec798, Previous frame's sp is 0xfffec7a0 Saved registers: ebx at 0xfffec78c, ebp at 0xfffec798, esi at 0xfffec790, edi at 0xfffec794, eip at 0xfffec79c (gdb)
First, let’s have a look at the co value again:
(gdb) p *co
$10 = {ob_refcnt = 2, ob_type = 0x81a1e60, co_argcount = 0,
co_nlocals = 0, co_stacksize = 1,
co_flags = 99, co_code = 0xf7d8e688,
co_consts = 0xf7d8ddac, co_names = 0xf7dc102c,
co_varnames = 0xf7dc102c, co_freevars = 0xf7dc102c,
co_cellvars = 0xf7dc102c,
co_filename = 0xf7d8cc38, co_name = 0xf7d8ddc0,
co_firstlineno = 51, co_lnotab = 0xf7d8dde0,
co_zombieframe = 0x0}
Building a Python Friendly Backtrace
Now we can deduce where exactly this code comes from. We can pull the line number, the function name, and the file!
(gdb) p co->co_firstlineno $16 = 51 (gdb) x/s ((PyStringObject)*co->co_name)->ob_sval 0xf7d8ddd4: "_g" (gdb) x/s ((PyStringObject)*co->co_filename)->ob_sval 0xf7d8cc4c: "/usr/lib/python2.6/types.py" (gdb)
So, types.py, line 51, function _g. Let’s take a look:
jeff@martian:~$ head /usr/lib/python2.6/types.py -n 51 | tail -n 1 def _g():
Excellent. This is where our Python function lives! There’s no point in going into it, however, this gives us a starting point to determine where a problem lives.
Looking up Argument Types and Values
Furthermore, we can pull out information about the arguments passed as well. Let’s go back and determine what the type is. Remember our ‘info frame’ gave us an args parameter?
(gdb) p *args $21 = (PyObject *) 0x0
Drat! Null. This function takes no arguments. Let’s jump down a few more frames until we find a function that includes an argument.
Breakpoint 1, PyEval_EvalCodeEx (co=0xf7d9f8d8, globals=0xf7d8a9bc, locals=0x0, args=0xf7d9e1c8, argcount=4, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../../../Python/ceval.c:2767 2767 in ../../../Python/ceval.c (gdb) info frame Stack level 0, frame at 0xfffefa50: eip = 0x80e1f53 in PyEval_EvalCodeEx (../../../Python/ceval.c:2767); saved eip 0x813e70e called by frame at 0xfffefac0 source language c. Arglist at 0xfffefa48, args: co=0xf7d9f8d8, globals=0xf7d8a9bc, locals=0x0, args=0xf7d9e1c8, argcount=4, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0 Locals at 0xfffefa48, Previous frame's sp is 0xfffefa50 Saved registers: ebx at 0xfffefa3c, ebp at 0xfffefa48, esi at 0xfffefa40, edi at 0xfffefa44, eip at 0xfffefa4c (gdb)
Here we go. Now, using the above “trick”, we learn that this is line 78 in method __new__ in abc.py:
(gdb)p co->co_firstlineno $24 = 78 (gdb) x/s ((PyStringObject)*co->co_name)->ob_sval 0xf7dc4694: "__new__" (gdb) x/s ((PyStringObject)*co->co_filename)->ob_sval 0xf7d9f8a4: "/usr/lib/python2.6/abc.py" (gdb)
Perfect. Now, since __new__ is (sometimes) indicative of a metaclass — and we’re looking at code from the Abstract Base Class module which I happen to know goes metaclass crazy — we should have a class, a name, a bases tuple, and an object dictionary. Let’s look at the object types:
(gdb) x/s args[0]->ob_type.tp_name 0x81590e5 <.LC33+5012>: "type" (gdb) x/s args[1]->ob_type.tp_name 0x8158d74 <.LC33+4131>: "str" (gdb) x/s args[2]->ob_type.tp_name 0x8158f43 <.LC33+4594>: "tuple" (gdb) x/s args[3]->ob_type.tp_name 0x8156ea5 <.LC16+1319>: "dict" (gdb)
Perfect! We’ve found the location of the code executing and the types of arguments that it takes. What if we wanted to see, for example, the actual name passed in instead of the “str” type? Simple. We just repeat what we’ve already learned:
(gdb) x/s (*(PyStringObject *)args[1]).ob_sval 0xf7d96054: "Hashable" (gdb) p (*(PyStringObject *)args[1]).ob_refcnt $38 = 8 (gdb)
Now we know, without looking at a line of Python, that this is the __new__ method of the metaclass for the Hashable ABC and the name of the class has a reference count of 8.
Accessing Dictionaries
Finally, what about something more detailed? Let’s look at the dictionary passed here.
(gdb) p *((PyDictObject*)args[3])
$51 = {ob_refcnt = 3, ob_type = 0x81854a0, ma_fill = 4, ma_used = 4, ma_mask = 7,
ma_table = 0xf7d8aa60, ma_lookup = 0x808c70c , ma_smalltable = {
{me_hash = 435549560, me_key = 0xf7dc44e0, me_value = 0xf7d9b4fc},
{me_hash = 0, me_key = 0x0, me_value = 0x0},
{me_hash = 1333480578, me_key = 0xf7dc2a20, me_value = 0xf7d9d5a0},
{me_hash = -1120181165,me_key = 0xf7dc2688, me_value = 0xf7dc132c},
{me_hash = 1733367940, me_key = 0xf7d942f0, me_value = 0x81c3e64},
{me_hash = 0, me_key = 0x0, me_value = 0x0},
{me_hash = 0, me_key = 0x0, me_value = 0x0},
{me_hash = 0, me_key = 0x0, me_value = 0x0}}}
(gdb)
What’s all of this me business? Let’s look at one of the items in the hash table representing the dictionary.
(gdb) p *((PyTypeObject*)((PyDictObject*)args[3])->ma_smalltable[2].me_key.ob_type)
$64 = {ob_refcnt = 71, ob_type = 0x818a940, ob_size = 0, tp_name = 0x8158d74 "str",
tp_basicsize = 24, tp_itemsize = 1, tp_dealloc = 0x809d982 ,
tp_print = 0x809d74c , tp_getattr = 0, tp_setattr = 0, tp_compare = 0,
tp_repr = 0x809ec77 , tp_as_number = 0x8187fe0, tp_as_sequence = 0x8188080,
tp_as_mapping = 0x81880a8, tp_hash = 0x809c5a9 , tp_call = 0, tp_str = 0x809e602 ,
tp_getattro = 0x8091091 ,
tp_setattro = 0x8090e1a , tp_as_buffer = 0x81880b4, tp_flags = 136713723,
tp_doc = 0x81880e0
"str(object) -> string\n\nReturn a nice string representation of the object.\n
If the argument is a string, the return value is the same object.",
tp_traverse = 0,
tp_clear = 0, tp_richcompare = 0x809cd84 , tp_weaklistoffset = 0, tp_iter = 0,
tp_iternext = 0, tp_methods = 0x8188180, tp_members = 0x0, tp_getset = 0x0,
tp_base = 0x8187c00, tp_dict = 0xf7dc34f4, tp_descr_get = 0, tp_descr_set = 0,
tp_dictoffset = 0, tp_init = 0x80ac582 , tp_alloc = 0x80ad345 ,
tp_new = 0x80a2fcc , tp_free = 0x8094510 , tp_is_gc = 0, tp_bases = 0xf7dc4f0c,
tp_mro = 0xf7dc7fa4, tp_cache = 0x0, tp_subclasses = 0x0,
tp_weaklist = 0xf7dc7fcc, tp_del = 0, tp_version_tag = 0}
(gdb)
Excellent. The key type is a string. What’s the value?
(gdb) x/s ((PyStringObject *)((PyTypeObject*)((
PyDictObject*)args[3])->ma_smalltable[2].me_value)).ob_sval
0xf7d9d5b4: "_abcoll"
(gdb)
The value of this entry is the string “_abcoll.” Note that the key type doesn’t reference the value type. I left out the step in which I looked up the value’s type.
Closing Notes
The most important step in understanding how to do this is having Python source available. You’re debugging a C program here; you want to access structure members and fields. Given the above knowledge, you should be able to walk through and display information about almost any Python object in memory. A big help.
What about the shared libraries?
If you’re referencing shared object files that aren’t in standard library paths, you can add them to your GDB shared object search path from your local directory as follows:
for i in $(find . -name *.so)
do
dirname $i;
done | sort | uniq | tr \\n : | sed -e 's#\./#'$PWD'#g'
And then…
(gdb) set solib-search-path <the above output>
As always, you should ensure these are the same versions that you’re running or that may be referenced in a core.
What if I Have a Core File?
You’ll use it like you would with any other debug session:
gdb -c <core> /usr/bin/python
All of the standard commands should work at that point: up, down, select, frame, etc…
How do I Get a Core File?
You can force a binary to drop a core by ensuring that the ulimit is set appropriately via ulimit -Sc unlimited. If your core files aren’t where you expect, see man core.
In: development, linux, open source, python
on April 20, 2012 at 3:55 am
Permalink
You’ve missed completely .gdbinit macros that come with the Python sources, they make it a completely different way of debugging:
http://hg.python.org/releasing/2.7.3/file/tip/Misc/gdbinit
on April 20, 2012 at 5:43 am
Permalink
Thanks! I knew about this, they just completely slipped my mind. Updated.