gdb/python: don't use PyObject_IsInstance in gdbpy_is_color

The gdbpy_is_color function uses PyObject_IsInstance, and converts the
return from PyObject_IsInstance to a bool.

Unfortunately, PyObject_IsInstance can return -1, 0, or 1, for error,
failure, or success respectively.  When converting to a bool both -1
and 1 will convert to true.

Additionally, when PyObject_IsInstance returns -1 an error will be
set.

What this means is that, if gdbpy_is_color is called with a non
gdb.Color object, and the PyObject_IsInstance check raises an error,
then (a) GDB will continue as if the object is a gdb.Color object,
which is likely going to invoke undefined behaviour, see
gdbpy_get_color for example, and (b) when GDB eventually returns to
the Python interpreter, due to an error being set, we'll see:

  Python Exception <class 'SystemError'>: PyEval_EvalFrameEx returned a result with an error set
  Error occurred in Python: PyEval_EvalFrameEx returned a result with an error set

However, after the previous commit, gdb.Color can no longer be
sub-classed, this means that fixing the above problems is easy, we can
replace the PyObject_IsInstance check with a PyObject_TypeCheck, the
PyObject_TypeCheck function only returns 0 or 1, there's no -1 error
case.

It's also worth noting that PyObject_TypeCheck is the function that is
more commonly used within GDB's Python API implementation, include the
py-color.c use there were only 4 PyObject_IsInstance uses.  Of the
remaining 3, 2 are fine, and one other (in py-disasm.c) is also
wrong.  I'll address that in a separate patch.

There's also a new test included which exposes the above issue.

Approved-By: Tom Tromey <tom@tromey.com>

1 files changed, 2 insertions, 1 deletions

diff --git a/gdb/python/py-color.c b/gdb/python/py-color.c
index fb4b80e..c48d14e 100644
--- a/gdb/python/py-color.c
+++ b/gdb/python/py-color.c
@@ -64,7 +64,8 @@ create_color_object (const ui_file_style::color &color)
 bool
 gdbpy_is_color (PyObject *obj)
 {
-  return PyObject_IsInstance (obj, (PyObject *) &colorpy_object_type);
+  gdb_assert (obj != nullptr);
+  return PyObject_TypeCheck (obj, &colorpy_object_type) != 0;
 }
 
 /* See py-color.h.  */