Troubleshooting

File and Line Breakpoints Are Not Getting Hit

First you must make sure that your source files were compiled with debug information. Typically this means passing -g to the compiler when compiling your source file.

When setting breakpoints in implementation source files (.c, cpp, cxx, .m, .mm, etc), LLDB by default will only search for compile units whose filename matches. If your code does tricky things like using #include to include source files:

$ cat foo.c
#include "bar.c"
#include "baz.c"
...

This will cause breakpoints in “bar.c” to be inlined into the compile unit for “foo.c”. If your code does this, or if your build system combines multiple files in some way such that breakpoints from one implementation file will be compiled into another implementation file, you will need to tell LLDB to always search for inlined breakpoint locations by adding the following line to your ~/.lldbinit file:

$ echo "settings set target.inline-breakpoint-strategy always" >> ~/.lldbinit

This tells LLDB to always look in all compile units and search for breakpoint locations by file and line even if the implementation file doesn’t match. Setting breakpoints in header files always searches all compile units because inline functions are commonly defined in header files and often cause multiple breakpoints to have source line information that matches many header file paths.

If you set a file and line breakpoint using a full path to the source file, like Xcode does when setting a breakpoint in its GUI on macOS when you click in the gutter of the source view, this path must match the full paths in the debug information. If the paths mismatch, possibly due to passing in a resolved source file path that doesn’t match an unresolved path in the debug information, this can cause breakpoints to not be resolved. Try setting breakpoints using the file basename only.

If you are using an IDE and you move your project in your file system and build again, sometimes doing a clean then build will solve the issue.This will fix the issue if some .o files didn’t get rebuilt after the move as the .o files in the build folder might still contain stale debug information with the old source locations.

How Do I Check If I Have Debug Symbols?

Checking if a module has any compile units (source files) is a good way to check if there is debug information in a module:

(lldb) file /tmp/a.out
(lldb) image list
[  0] 71E5A649-8FEF-3887-9CED-D3EF8FC2FD6E 0x0000000100000000 /tmp/a.out
      /tmp/a.out.dSYM/Contents/Resources/DWARF/a.out
[  1] 6900F2BA-DB48-3B78-B668-58FC0CF6BCB8 0x00007fff5fc00000 /usr/lib/dyld
....
(lldb) script lldb.target.module['/tmp/a.out'].GetNumCompileUnits()
1
(lldb) script lldb.target.module['/usr/lib/dyld'].GetNumCompileUnits()
0

Above we can see that “/tmp/a.out” does have a compile unit, and “/usr/lib/dyld” does not.

We can also list the full paths to all compile units for a module using python:

(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
>>> m = lldb.target.module['a.out']
>>> for i in range(m.GetNumCompileUnits()):
...   cu = m.GetCompileUnitAtIndex(i).file.fullpath
/tmp/main.c
/tmp/foo.c
/tmp/bar.c
>>>

This can help to show the actual full path to the source files. Sometimes IDEs will set breakpoints by full paths where the path doesn’t match the full path in the debug info and this can cause LLDB to not resolve breakpoints. You can use the breakpoint list command with the –verbose option to see the full paths for any source file and line breakpoints that the IDE set using:

(lldb) breakpoint list --verbose

How Do I Find Out Which Features My Copy Of LLDB Has?

Some features such as XML parsing are optional and must be enabled when LLDB is built. To check which features your copy of LLDB has enabled, use the version command from within LLDB:

(lldb) version -v

Note

This feature was added in LLDB 22. If you are using an earlier version, you can use one of the methods below.

If your LLDB has a scripting langauge enabled, you can also use this command to print the same information:

(lldb) script lldb.debugger.GetBuildConfiguration()

This command will fail if no scripting langauge was enabled. In that case, you can instead check the shared library dependencies of LLDB.

For example on Linux you can use the following command:

$ readelf -d <path-to-lldb> | grep NEEDED
0x0000000000000001 (NEEDED)             Shared library: [liblldb.so.22.0git]
0x0000000000000001 (NEEDED)             Shared library: [libxml2.so.2]
0x0000000000000001 (NEEDED)             Shared library: [libedit.so.2]
<...>

The output above shows us that this particular copy of LLDB has XML parsing (libxml2) and editline (libedit) enabled.

Note

readelf -d as used above only shows direct dependencies of the binary. Libraries loaded by a library will not be shown. An example of this is Python. lldb requires liblldb and it is liblldb that would require libpython. The same readelf command can be used on liblldb to see if it does depend on Python.

ldd will show you the full dependency tree of lldb but do not use it unless you trust the lldb binary. As some versions of ldd may execute the binary in the process of inspecting it.

On Windows, use dumpbin /dependents <path-to-lldb>. The same caveat from Linux applies to Windows. To find dependencies like Python, you need to run dumpbin on liblldb.dll too.

On MacOS, use otool -l <path-to-lldb>.

Why Do I See More, Less, Or Different Registers Than I Expected?

The registers you see in LLDB are defined by information either provided by the debug server you are connected to, or in some cases, guessed by LLDB.

If you are not seeing the registers you expect, the first step is to figure out which method is being used to read the register information. They are presented here in the order that LLDB will try them.

Target Definition Script

These scripts tell LLDB what registers exist on the debug server without having to ask it. You can check if you are using one by checking the setting:

(lldb) settings show plugin.process.gdb-remote.target-definition-file

In most cases you will not be using such a script.

If you are using one, or want to write one, you can learn about them by reading the *_target_definition.py files in the Python examples folder.

Tip

We recommend that before attempting to write a target definition script to solve your issues, you look into the other methods first.

Target Description Format XML

The most commonly used method is the debug server sending LLDB an XML document in the Target Description Format.

LLDB can only process this if XML parsing was enabled when it was built. A previous section of this document explains how to check this. If XML is not enabled and the debug server has offered an XML document, we highly recommend switching to a build of LLDB with XML parsing enabled.

Note

If LLDB was offered an XML document but could not use it, you will see a warning in your debug session to alert you to this situation.

If your LLDB has XML support, next check whether the debug server offered this XML document. Enable the GDB remote packet log, then connect to the debug server as you normally would.

(lldb) log enable gdb-remote packets

This will produce a lot of output. Scroll back to just after you connected to the debug server. Look for lines like these:

lldb             < 104> send packet: $qSupported:xmlRegisters=<...>
lldb             < 260> read packet: $<...>qXfer:features:read+<...>

The sent packet is LLDB telling the debug server that it can parse XML definitions. The read (received) packet is the debug server telling LLDB that it is allowed to request Target Description XML.

If you do not see either of these, then one or both of LLDB or the debug server does not support XML. This is not a fatal problem but may result in degraded register information.

Switching to a debug server with XML support may not be possible. For example if you have hardware debug tools that cannot be changed. Consult your local experts for advice on this.

If you did see those packets, scroll further down to confirm that the XML was read. You should see output that looks like an XML document:

lldb             <  43> send packet: $qXfer:features:read:target.xml<...>
lldb             <27583> read packet: $l<?xml version="1.0"?>
<target version="1.0">
<...>

Even if the XML document was read by LLDB, it, or parts of it may not be valid. LLDB is permissive when parsing, so you will not get a fatal error. You need to check the GDB Remote process log for details of any parsing problems:

(lldb) log enable gdb-remote process

Note

There will always be some messages about register information because we log details of successful parsing too. If registers are present but presented in an unexpected way, check these log messages to see if LLDB has misinterpreted the register information.

If those messages do not tell you what is wrong, your last option is to read the document yourself by copying it out of the gdb-remote packets output. Start by checking the basic XML structure of nested elements and begin/end markers are correct, then compare it to the Target Description specification.

If reading XML is not possible, or it fails to produce any valid register information, LLDB falls back to the next method.

qRegisterInfo

When XML reading fails, LLDB will try an lldb-server specific packet called qRegisterInfo. This packet contains all the same register information as the XML would, except advanced formatting information. That information tells LLDB how to format registers as complex types like structures.

So if you are using lldb connected to lldb-server or debugserver, it is likely you will not notice the lack of XML. Connected to anything else, qRegisterInfo will not work and LLDB will fall back to the final method.

To check if LLDB is using qRegisterInfo you can check the gdb-remote packets and gdb-remote process logs.

Fallback Register Layouts

The final method is that LLDB will assume that it can use the general register read packet and make an assumption about the offset of each register in the response.

This is often done with older debug servers, or ones running with very limited resources such as those inside of operating system kernels.

This requires that LLDB and the debug server be in sync which is not always the case if you are connecting LLDB to anything but lldb-server or debugserver. Work has been done to make LLDB more compatible, but it is impossible to predict what might be in use in the wild.

Symptoms of a mismatch are missing, overlapping or incorrect register values. Often this will cause other features like backtracing to fail.

In this case, assuming you cannot change the debug server, you do not have much choice but to find a debug client that does match up. Often GDB will work better, it has a longer history of compatability fixes.

If you must use LLDB, you could patch it to match the debug server. The fallback layouts are stored in lldb/source/Plugins/Process/gdb-remote/GDBRemoteRegisterFallback.cpp.