5.26. Debugging With the GNU Project Debugger (GDB) Remotely

GDB allows you to examine running programs, which in turn helps you to understand and fix problems. It also allows you to perform post-mortem style analysis of program crashes. GDB is available as a package within the Yocto Project and is installed in SDK images by default. See the "Images" chapter in the Yocto Project Reference Manual for a description of these images. You can find information on GDB at http://sourceware.org/gdb/.

Tip

For best results, install debug (-dbg) packages for the applications you are going to debug. Doing so makes extra debug symbols available that give you more meaningful output.

Sometimes, due to memory or disk space constraints, it is not possible to use GDB directly on the remote target to debug applications. These constraints arise because GDB needs to load the debugging information and the binaries of the process being debugged. Additionally, GDB needs to perform many computations to locate information such as function names, variable names and values, stack traces and so forth - even before starting the debugging process. These extra computations place more load on the target system and can alter the characteristics of the program being debugged.

To help get past the previously mentioned constraints, you can use gdbserver, which runs on the remote target and does not load any debugging information from the debugged process. Instead, a GDB instance processes the debugging information that is run on a remote computer - the host GDB. The host GDB then sends control commands to gdbserver to make it stop or start the debugged program, as well as read or write memory regions of that debugged program. All the debugging information loaded and processed as well as all the heavy debugging is done by the host GDB. Offloading these processes gives the gdbserver running on the target a chance to remain small and fast.

Because the host GDB is responsible for loading the debugging information and for doing the necessary processing to make actual debugging happen, you have to make sure the host can access the unstripped binaries complete with their debugging information and also be sure the target is compiled with no optimizations. The host GDB must also have local access to all the libraries used by the debugged program. Because gdbserver does not need any local debugging information, the binaries on the remote target can remain stripped. However, the binaries must also be compiled without optimization so they match the host's binaries.

To remain consistent with GDB documentation and terminology, the binary being debugged on the remote target machine is referred to as the "inferior" binary. For documentation on GDB see the GDB site.

The following steps show you how to debug using the GNU project debugger.

  1. Configure your build system to construct the companion debug filesystem:

    In your local.conf file, set the following:

         IMAGE_GEN_DEBUGFS = "1"
         IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
                        

    These options cause the OpenEmbedded build system to generate a special companion filesystem fragment, which contains the matching source and debug symbols to your deployable filesystem. The build system does this by looking at what is in the deployed filesystem, and pulling the corresponding -dbg packages.

    The companion debug filesystem is not a complete filesystem, but only contains the debug fragments. This filesystem must be combined with the full filesystem for debugging. Subsequent steps in this procedure show how to combine the partial filesystem with the full filesystem.

  2. Configure the system to include gdbserver in the target filesystem:

    Make the following addition in either your local.conf file or in an image recipe:

         IMAGE_INSTALL_append = “ gdbserver"
                        

    The change makes sure the gdbserver package is included.

  3. Build the environment:

    Use the following command to construct the image and the companion Debug Filesystem:

         $ bitbake image
                        

    Build the cross GDB component and make it available for debugging. Build the SDK that matches the image. Building the SDK is best for a production build that can be used later for debugging, especially during long term maintenance:

         $ bitbake -c populate_sdk image
                        

    Alternatively, you can build the minimal toolchain components that match the target. Doing so creates a smaller than typical SDK and only contains a minimal set of components with which to build simple test applications, as well as run the debugger:

         $ bitbake meta-toolchain
                        

    A final method is to build Gdb itself within the build system:

         $ bitbake gdb-cross-architecture
                        

    Doing so produces a temporary copy of cross-gdb you can use for debugging during development. While this is the quickest approach, the two previous methods in this step are better when considering long-term maintenance strategies.

    Note

    If you run bitbake gdb-cross, the OpenEmbedded build system suggests the actual image (e.g. gdb-cross-i586). The suggestion is usually the actual name you want to use.

  4. Set up the debugfs

    Run the following commands to set up the debugfs:

         $ mkdir debugfs
         $ cd debugfs
         $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2
         $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2
                        

  5. Set up GDB

    Install the SDK (if you built one) and then source the correct environment file. Sourcing the environment file puts the SDK in your PATH environment variable.

    If you are using the build system, Gdb is located in build-dir/tmp/sysroots/host/usr/bin/architecture/architecture-gdb

  6. Boot the target:

    For information on how to run QEMU, see the QEMU Documentation.

    Note

    Be sure to verify that your host can access the target via TCP.

  7. Debug a program:

    Debugging a program involves running gdbserver on the target and then running Gdb on the host. The example in this step debugs gzip:

         root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
                        

    For additional gdbserver options, see the GDB Server Documentation.

    After running gdbserver on the target, you need to run Gdb on the host and configure it and connect to the target. Use these commands:

         $ cd directory-holding-the-debugfs-directory
         $ arch-gdb
    
         (gdb) set sysroot debugfs
         (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
         (gdb) target remote IP-of-target:1234
                        

    At this point, everything should automatically load (i.e. matching binaries, symbols and headers).

    Note

    The Gdb set commands in the previous example can be placed into the users ~/.gdbinit file. Upon starting, Gdb automatically runs whatever commands are in that file.

  8. Deploying without a full image rebuild:

    In many cases, during development you want a quick method to deploy a new binary to the target and debug it, without waiting for a full image build.

    One approach to solving this situation is to just build the component you want to debug. Once you have built the component, copy the executable directly to both the target and the host debugfs.

    If the binary is processed through the debug splitting in OpenEmbedded, you should also copy the debug items (i.e. .debug contents and corresponding /usr/src/debug files) from the work directory. Here is an example:

         $ bitbake bash
         $ bitbake -c devshell bash
         $ cd ..
         $ scp packages-split/bash/bin/bash target:/bin/bash
         $ cp -a packages-split/bash-dbg/* path/debugfs