gdb/python: stop using PyObject_IsInstance in py-disasm.c

The PyObject_IsInstance function can return -1 for errors, 0 to
indicate false, and 1 to indicate true.

I noticed in python/py-disasm.c that we treat the result of
PyObject_IsInstance as a bool.  This means that if PyObject_IsInstance
returns -1, then this will be treated as true.  The consequence of
this is that we will invoke undefined behaviour by treating the result
from the _print_insn call as if it was a DisassemblerResult object,
even though PyObject_IsInstance raised an error, and the result might
not be of the required type.

I could fix this by taking the -1 result into account, however,
gdb.DisassemblerResult cannot be sub-classed, the type doesn't have
the Py_TPFLAGS_BASETYPE flag.  As such, we can switch to using
PyObject_TypeCheck instead, which only return 0 or 1, with no error
case.

I have also taken the opportunity to improve the error message emitted
if the result has the wrong type.  Better error message make debugging
issues easier.

I've added a test which exposes the problem when using
PyObject_IsInstance, and I've updated the existing test for the
improved error message.

Approved-By: Tom Tromey <tom@tromey.com>

0 files changed, 0 insertions, 0 deletions