Adding Target Support#

LLDB supports many combinations of architecture, operating system and other system components. In this document we describe the considerations and requirements for porting LLDB to a combination of those things which we will refer to as a “target” within this document.

Note

The terms Target and Platform are used throughout LLDB, often referring to a subset of what this document calls a “target”. Unfortunately there is no more specific word to use here.

This document provides some hints on implementation, but because every target is unique, we expect developers to learn about implementation from existing targets.

This document starts with the end of the process, proposing your target for upstream inclusion. It is worth reading through this even if that is not one of your goals, as you will have to tackle the same topics even in a downstream implementation.

Upstreaming Proposal Process#

You must send an RFC to the LLDB Discourse forum before upstreaming new target support. This RFC must be accepted in some form by the community before any changes can be merged upstream that are specific to your target.

This RFC follows the normal rules of the LLVM community decision making process.

We require an RFC for target support upstreaming because:

  • We want to ensure there is some level of public discussion on the topic. It is important that details, even those obvious to the community at the time, are written down so that future readers may learn from them.

  • These discussions form the basis for future proposals and the assessment of those proposals. Every proposal will be different in some way, and by contrasting current proposals with the previous proposals we can do a better job assessing them.

RFCs are not required to be:

  • In a set format. Make a logical argument in whatever way you think fit.

  • Exhaustively detailed. Include what you think is relevant and the community will ask for the rest.

  • Answering all the same questions, or making all the same points, as previous proposals. Compare your target with existing targets, but not everything will, or has to, apply to yours.

  • A commitment to the maximum theoretical level of support (more on this later). If your resources are limited, say so, and that will be taken into account (and vice versa, committing to a lot of work does not guarantee acceptance).

Removal Proposal Process#

Removing target support also requires an RFC. Reasonable effort should be made to involve the original contributors and users of the target support.

Note

You may propose the removal of a subset of the target. For instance, removing support for an operating system on architecture A, without removing support for that operating system on architecture B.

The content of the RFC should cover the same topics as an upstreaming proposal, but you will be arguing for the opposite conclusion.

We reserve the right to change our standards over time, so a removal proposal is not necessarily a refutation of each point made in the upstreaming proposal. For example, what was an acceptable level of testing in the past may no longer be acceptable.

If standards change and are to be applied to existing targets, target stakeholders should be given the opportunity to meet those standards rather than going straight to removal.

Expectations Of Upstream Code#

Listed below are some examples of factors considered when considering accepting code upstream. These are examples and RFC authors are free to add their own, leave some out, or explain why they do not apply to their proposal.

If you do use these points, they need to come with an answer and evidence to justify the answer, rather than simply “yes this applies to my target”. In other words, your proposal must stand alone without requiring readers to read this document as well.

The first set covers your motivation for your target being upstream, rather than on a fork:

  • Will it help you distribute an LLDB that includes this target support? For example if there is an existing community for this target, and how would they acquire LLDB?

  • Will it enable a wider community than your own forks would? For example for it to be included in Linux distribution packaging.

  • Will it improve support for other targets by being there? For example if we already support Operating System X on Architecture Y, adding Architecture Z support may improve both as a side effect.

  • Will it help you keep your changes in sync? For example if it involves fundamental changes to LLDB, or you have a very small amount of maintenance resources in your community.

  • What other costs (or benefits) do you incur staying on a fork? For example, your company might already have a fork.

The next set is about whether you, your community, or the LLDB community, can adequately maintain the code upstream:

  • Who will be the maintainers for this target? Ideally there will be more than one, who is present in the LLVM community and can be contacted in a few different ways.

  • How often will it be tested, where, by whom and who will be responsible for it?

  • Who will address problems with it? Will it always be the named maintainers? Is it so common that anyone in upstream LLDB can deal with it, or will only employees of a specific company work on it.

  • When it breaks, how easy will it be for the upstream LLDB community to continue their work without disruption?

  • If upstream contributors want to reproduce issues on your target, how can they access it? Can it be emulated or virtualized? Does it require them to sign a license? Do you offer access for open source projects? (and does that include employees of other companies)

The Extent of Target Support#

The factors above are part of a cost benefit analysis that the LLDB community will do, prioritising the health of the community and the project. This means that there is no set level or type of commitment required for a new target.

Each case will have unique aspects, so you need to tell the community how much impact your target will have on LLDB. Below are some questions you can start with, along with stereotypical “big” and “small” answers.

Note that these are intentionally not maximums and minimums. The impact of some targets will be bigger than “big” or smaller than “small”.

  • How many users will use LLDB with this target?

    • Big: millions of developers worldwide.

    • Small: you and a small community of target users.

  • How many changes will it require?

    • Big: large changes to all parts of LLDB. On the level of the existing code for C and C++ support.

    • Small: small changes to enable existing support from LLVM, changes to packet parsing (see the MSP430 case study below).

  • What parts and features of LLDB will work on, or with, this target?

    • Big: lldb, lldb-server and advanced features like shared libraries.

    • Small: just lldb and only basic features like continue, stop, reading memory and registers.

  • How often is it tested?

    • Big: per-commit testing of LLDB, following the upstream llvm-project.

    • Small: per release of a downstream community or individual developer’s tools.

  • Who will maintain it?

    • Big: there are several listed maintainers for this target, who are employed by a company with significant investments in the target.

    • Small: there is a single maintainer listed for this target.

  • When it breaks, who will be affected?

    • Big: every single developer and user of LLDB.

    • Small: you and your target’s community.

  • How do I reproduce a problem on this target?

    • Big: all components are open source and can be run anywhere by using simulations with very little setup.

    • Small: you contact the maintainer and they do it for you.

  • Who will fix problems?

    • Big: the maintainers and a large group of contributors.

    • Small: only the listed maintainer.

Target Support Case Studies#

These case studies are to give you starting points for your proposals. Consider how your target compares to these.

Apple Targets#

Apple targets have possibly the most extensive support in LLDB.

  • Changes from LLVM’s main branch are continuously tested. Results are accessible publicly, and reported to all LLDB contributors.

  • Many employees contribute upstream and are maintainers for these targets.

  • Target-specific problems are either solved by the upstream community with maintainer input, or by the maintainers themselves.

  • Apple-specific features either do not impact other targets, or, when they do, they are designed and maintained in collaboration with the community.

  • Many people unrelated to Apple itself use Apple hardware, which is available at retail. So it is fairly easy to find someone who can reproduce an issue.

  • LLDB gets a lot of secondary testing downstream, and exposure to Apple’s own developer community.

FreeBSD#

FreeBSD is an example of quite self-contained support, managed by the project’s community.

  • FreeBSD is open source and available to anyone to build, modify, run on hardware, emulate or virtualize.

  • FreeBSD support is mostly in the native target parts of lldb-server, so issues with it rarely impact any other target.

  • It is quite similar to Linux, so it does not cause large scale changes to fundamental assumptions LLDB makes.

  • It has at least one maintainer and sometimes contributions to upstream LLDB from the FreeBSD community.

  • Issues are solved by the maintainer and the FreeBSD community.

  • LLDB on FreeBSD is tested each time FreeBSD updates the version used in the base system, which is roughly per upstream LLVM release.

Linux#

Linux is an example where the Linux community as a whole does not do all the work of Linux support. Some architectures have a wide contributor base and others have company-specific contributors as well.

  • Linux is open source, freely available, and can be installed on hardware, virtualised, emulated, and so on. Most problems can be reproduced in more than one way.

  • Many architectures are tested by the architecture vendor, others are tested on community-provided hardware.

  • LLDB is a commonly available Linux package, so further testing happens during various Linux distributions’ release cycles.

  • Support for a new architecture is quite easy to isolate, so problems do not impact other Linux architectures, or other operating systems.

  • Linux is very popular, therefore its choice of standards informs many core features of LLDB.

  • It has more than one listed maintainer and many contributors. Issues are resolved by the community.

MSP430#

MSP430 is handled as a bare-metal (no operating system) target in LLDB. So it is the most minimal example.

  • LLVM already has MSP430 support, so changes to enable it in LLDB were minimal.

  • It does not use lldb-server, and lldb only required minor adjustments to be compatible with the commonly used debug server.

  • An ABI plugin was added, which is isolated to MSP430 only.

  • It is not systematically tested anywhere, but gets some use by MSP430 developers.

  • It has no documented maintainer.

  • We rely on users to report issues with it, and would likely guide them to fix them themselves.

Components Of Target Support#

This is a very high-level view. We recommend you combine this with reading the changes made for recently added targets, as each target is going to be slightly different.

Assuming that:

  • You want to support a combination of a new architecture and a new operating system.

  • LLVM already supports your architecture and operating system.

  • You want to port most of the LLDB features, which means porting both lldb and lldb-server.

Then this is a list of components you will need to write. If your target is similar to others you may be able to reuse existing components and we encourage you to do so.

First the components concerned with operating systems:

  • A Platform plugin. For example PlatformLinux.

  • A HostInfo plugin. For example HostInfoLinux.

  • A native process plugin (native means it runs on your target). For example NativeProcessLinux.

  • Signal information. For example LinuxSignals.

  • A dynamic loader plugin. For example DynamicLoaderPOSIXDYLD.

  • An object file plugin. For example ObjectFileELF.

Then the components concerned with architecture (though the distinction is often inexact):

  • ABI plugin. For example ABISysV_arm64.

  • Architecture plugin. For example ArchitectureAArch64.

  • Register definitions and register context (a context is a collection of registers). For example NativeRegisterContextLinux_arm64.

  • Unwinding support for backtracing. LLDB needs to be taught about any architecture-specific directives.

  • Instruction emulation. For example EmulateInstructionARM64. Most targets need to emulate a small number of instructions for unwinding. If your target does not have hardware single-step, you will need to emulate many times more than that.

The order of implementation will vary in each case. Each component does not need to be fully implemented for you to start work on the next.

If you are going to be running lldb on your target, then an obvious first step is to get lldb to build there without any changes. This lldb will only be useful for remotely debugging other targets, but you will at least know that the build system is compatible.

Next, try lldb with any existing debug servers for your target. If they are similar to lldb-server or gdbserver, this can flush out some obvious issues in lldb.

Then begin porting lldb-server to your target. We recommend that you get the test suite to run as soon as possible, however bad the results are.

As you have seen above, there are a lot of moving parts to a debugger. So having some set of results to measure progress is very important. Sometimes a change will get one test to pass, sometimes hundreds, and it is easy to regress if you are not careful.

Target Hardware Features#

Some hardware features can make porting easier or harder. Below is a non-exhaustive list of features and their impacts on porting LLDB.

Hardware Single Step#

If the target lacks this feature, you will have to implement software single stepping. This is much more complex and involves emulating any instruction that could modify the program counter.

Instruction Bundles and Sequences#

This is any situation where to resume the program you have to replay some previous instructions. LLDB needs to know the extent of the sequence.

For example, an atomic sequence may implement an atomic operation by looping until success. When stepping through the sequence normally, this check will always fail (due to the debug exceptions) and cause it to loop forever.

You can teach LLDB to find the sequence start point, and replay the whole thing as if it were one step.

If you have instruction bundles, check how breakpoints behave and where in the bundles they can be placed.

Single Instruction Equivalents of Sequences#

The opposite of the previous point. If your target has single instructions for what is normally a sequence, this reduces the work needed in LLDB. Single instruction atomics are a common example.

Runtime Register Resizing#

Anything like AArch64’s Scalable Vector Extension (SVE) registers. At each stop event the registers may have a different size.

Support for this is currently SVE specific as it requires lldb to know which registers scale and what to derive their size from.

Registers That Come And Go At Runtime#

If you have registers that are not present for the entire program runtime you will need to decide how to present that.

The one example we support right now is AArch64’s Scalable Matrix Extension (SME) ZA register. This register can be switched off when not in use. We handle this by showing a fake zero value at these times, with a separate mode bit in another register so users can tell a real zero from a fake zero.

The more fundamental and the more numerous the registers are, the more likely you are to confuse users by showing them even when they are unusable. For instance if you have two execution modes that use separate register sets, showing both all the time may be confusing for users.

Registers With The Same Name In Different Contexts#

When you have banked registers or a copy of a register for each execution mode, it usually only has one name. You need to decide if it makes sense to allow users to access each one separately.

For example, AArch64’s SME extension adds a “streaming mode”. SVE registers exist in the normal mode and the streaming mode. However programs only ever use one or the other, and the values are cleared when the mode is switched. So there is no reason to let users write to the inactive mode’s registers and we just show 1 set with the normal naming. That set always refers to the active mode.

However if you have overlapping sets that can hold their own values, you may want to make the normal register name the active set, and have a way to address the other sets.

Execution Modes That Change Instruction Encoding#

Arm (meaning Armv7 and prior) has 2 execution modes: Arm and Thumb. If you attempt to execute Thumb mode code in Arm mode, it will not work. Programs can mix the two modes by using special mode switching branches.

The debugger has to be aware of what mode the inferior is in so that it can correctly compare addresses, place breakpoints and use the correct breakpoint instruction encoding.

In Arm’s case, the information comes from markers in the program file, and the bottom bit of the program counter. This is often a source of bugs because many parts of the debugger have to know that this bit is not part of the instruction address.

Variable Length Instruction Encoding#

LLDB already supports a wide variety of encoding strategies, so variable length encoding is not much more work than fixed length.

Of the current targets we have:

  • Intel which is variable length.

  • Arm (Armv7 and prior) with Arm (32-bit), Thumb 1 (16-bit) and Thumb 2 (a mix of 16 and 32-bit).

  • RISC-V which is variable, usually 32-bit or the 16-bit compressed instructions.

  • AArch64 which is fixed length, always 32-bit.

Hardware Breakpoints and Watchpoints#

Code breakpoints can be implemented in software by replacing an instruction with a software breakpoint instruction.

However, if you want to only stop in certain situations (a single address space, a single execution mode, and so on), code breakpoints implemented in hardware will be much faster.

Doing it in software means you have to context switch between the debug stub and the debugger to filter every stop event. Which is slow even when locally debugging. Put the debug server on the end of a high latency connection and the slow down is multiplied.

In addition, hardware code breakpoints can be set in read-only memory. Which is important for code executing out of ROM, which is common on embedded targets.

Watchpoints are used to wait for a specific type of access to a specific range of memory. Doing this in software is possible but very invasive so use hardware watchpoints if you can.

For comparison, one way to implement a software watchpoint is to unmap the memory around a location and then filter the memory faults to find the access you are looking for. This requires a lot of traffic between debugger and debug server, and needs to be implemented for every supported operating system.

Whereas a hardware watchpoint is usually a few registers programmed with hardware specific values, and most operating systems expose those registers directly to userspace.

Note

Hardware often has many more features than LLDB makes use of. What we make use of is decided by how useful it will be to how many users and how understandable it will be when presented in the LLDB interface.

Accurate Breakpoints and Watchpoints#

Ideally your break and watchpoint exceptions contain enough information to know exactly what caused them. Which seems obvious from a software level, but if you check the architecture specifications you will find that they often allow a range of behaviours.

For instance, if you watch a 4 byte chunk of memory and an instruction writes 8 bytes over it, is the hardware required to report an address in the watched 4 bytes? Or can it report an address in the second 4 bytes, because it is still part of the write that triggered the watchpoint?

LLDB will assume accuracy unless told otherwise, and if there are untraceable situations users will have to figure out which watch or breakpoints to manually disable so that they can continue.

Addresses That Are More Than Just Numbers#

Though many programming languages have rules that prevent the use of pointers in some integer-like ways, the reality is that a lot of hardware treats them as numbers.

This changes with capabilities (for example CHERI), mode bits (Arm’s Thumb) and re-use of non-address bits (AArch64’s TBI, MTE and PAC). A pointer is no longer just a number that refers to a memory location, it contains extra information.

Where the size of the pointer is equal to the bit width of the architecture (64-bit pointers on AArch64 for example), LLDB can probably handle it. LLDB already has the concept of “non-address bits” that must be removed to get the memory address from a pointer.

If pointers are like capabilities where their size is greater than that of a memory address, you will need to change the type LLDB uses to store addresses which is a lot more work.