From 37f5e02ef5be1ba47e80df4af690df26ee1dac83 Mon Sep 17 00:00:00 2001 From: Andrew Burgess Date: Tue, 22 Apr 2025 19:56:13 +0100 Subject: gdb/python: don't use PyObject_IsInstance in gdbpy_is_color The gdbpy_is_color function uses PyObject_IsInstance, and converts the return from PyObject_IsInstance to a bool. Unfortunately, PyObject_IsInstance can return -1, 0, or 1, for error, failure, or success respectively. When converting to a bool both -1 and 1 will convert to true. Additionally, when PyObject_IsInstance returns -1 an error will be set. What this means is that, if gdbpy_is_color is called with a non gdb.Color object, and the PyObject_IsInstance check raises an error, then (a) GDB will continue as if the object is a gdb.Color object, which is likely going to invoke undefined behaviour, see gdbpy_get_color for example, and (b) when GDB eventually returns to the Python interpreter, due to an error being set, we'll see: Python Exception : PyEval_EvalFrameEx returned a result with an error set Error occurred in Python: PyEval_EvalFrameEx returned a result with an error set However, after the previous commit, gdb.Color can no longer be sub-classed, this means that fixing the above problems is easy, we can replace the PyObject_IsInstance check with a PyObject_TypeCheck, the PyObject_TypeCheck function only returns 0 or 1, there's no -1 error case. It's also worth noting that PyObject_TypeCheck is the function that is more commonly used within GDB's Python API implementation, include the py-color.c use there were only 4 PyObject_IsInstance uses. Of the remaining 3, 2 are fine, and one other (in py-disasm.c) is also wrong. I'll address that in a separate patch. There's also a new test included which exposes the above issue. Approved-By: Tom Tromey --- gdb/python/py-color.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) (limited to 'gdb/python') diff --git a/gdb/python/py-color.c b/gdb/python/py-color.c index fb4b80e..c48d14e 100644 --- a/gdb/python/py-color.c +++ b/gdb/python/py-color.c @@ -64,7 +64,8 @@ create_color_object (const ui_file_style::color &color) bool gdbpy_is_color (PyObject *obj) { - return PyObject_IsInstance (obj, (PyObject *) &colorpy_object_type); + gdb_assert (obj != nullptr); + return PyObject_TypeCheck (obj, &colorpy_object_type) != 0; } /* See py-color.h. */ -- cgit v1.1