gdb/python: don't use PyObject_IsInstance in gdbpy_is_color

The gdbpy_is_color function uses PyObject_IsInstance, and converts the
return from PyObject_IsInstance to a bool.

Unfortunately, PyObject_IsInstance can return -1, 0, or 1, for error,
failure, or success respectively.  When converting to a bool both -1
and 1 will convert to true.

Additionally, when PyObject_IsInstance returns -1 an error will be
set.

What this means is that, if gdbpy_is_color is called with a non
gdb.Color object, and the PyObject_IsInstance check raises an error,
then (a) GDB will continue as if the object is a gdb.Color object,
which is likely going to invoke undefined behaviour, see
gdbpy_get_color for example, and (b) when GDB eventually returns to
the Python interpreter, due to an error being set, we'll see:

  Python Exception <class 'SystemError'>: PyEval_EvalFrameEx returned a result with an error set
  Error occurred in Python: PyEval_EvalFrameEx returned a result with an error set

However, after the previous commit, gdb.Color can no longer be
sub-classed, this means that fixing the above problems is easy, we can
replace the PyObject_IsInstance check with a PyObject_TypeCheck, the
PyObject_TypeCheck function only returns 0 or 1, there's no -1 error
case.

It's also worth noting that PyObject_TypeCheck is the function that is
more commonly used within GDB's Python API implementation, include the
py-color.c use there were only 4 PyObject_IsInstance uses.  Of the
remaining 3, 2 are fine, and one other (in py-disasm.c) is also
wrong.  I'll address that in a separate patch.

There's also a new test included which exposes the above issue.

Approved-By: Tom Tromey <tom@tromey.com>

0 files changed, 0 insertions, 0 deletions