I'm trying to debug a Raspberry Pi 5 using a debug probe, OpenOCD, and GDB. I'm running code on all four cores, so I'm trying to setup SMP to get a GDB thread for each core.
Previously, I tried debugging each core without SMP (added an inferior for each core in GDB and connected to four different ports). This setup worked but had created some issues. One of them was that jumping to breakpoints would create a halt instruction in the code that OpenOCD would not change back to the original instruction. These halt instructions made debugging tricky. To get past them I would have to manually set my program counter and change the halt instruction back to the original instruction (if it was going to run again). I'd like to have a debugging setup where I don't need to worry about that problem.
I think I have something missing or wrong in my OpenOCD config file for the Raspberry Pi 5. I found this similar question where the person has the same behavior as me, but I've already added the -rtos hwthread flags to the targets.
Here's my config file:
# rpi5 chip name
set _CHIP_NAME bcm2712
# the rpi5 has four cortex-a76 cores
set _CORES 4
# Use SMP since we are running code on multiple cores
set _USE_SMP 1
#
set _DAP_TAPID 0x4ba00477
# Specify that the debug probe is using the SWD protocol
transport select swd
# Related to creating a new debug access port
swd newdap $_CHIP_NAME cpu -expected-id $_DAP_TAPID -irlen 4
# Sets clock speed in KHz
adapter speed 4000
# Create a debug access port
dap create $_CHIP_NAME.dap -chain-position $_CHIP_NAME.cpu
# TODO: don't know why this is needed or if it's messing up the SMP config
# MEM-AP for direct access
# target create $_CHIP_NAME.ap mem_ap -dap $_CHIP_NAME.dap -ap-num 0
# these addresses are obtained from the ROM table via 'dap info 0' command
# Each address if for is for a specific core
#
# Debug controller base address for each core
set _DBG_BASE {0x80010000 0x80110000 0x80210000 0x80310000}
#
set _CTI_BASE {0x80020000 0x80120000 0x80220000 0x80320000}
# Used for building the 'target smp' command for each core
set _SMP_COMMAND "target smp"
# Perform configuration for each of the four cores
for { set _CORE 0 } { $_CORE < $_CORES } { incr _CORE } {
# Set the CTI name for each core
set _CTI_NAME $_CHIP_NAME.cti$_CORE
# Set the target name for each core
set _TARGET_NAME $_CHIP_NAME.cpu$_CORE
# Add each target name to the SMP command
set _SMP_COMMAND "$_SMP_COMMAND $_TARGET_NAME"
# Create an ARM Cross-Trigger Interface for each core
# CTI is control core states and access registers
cti create $_CTI_NAME -dap $_CHIP_NAME.dap -ap-num 0 -baseaddr [lindex $_CTI_BASE $_CORE]
# Create debug target for GDB that refers to a JTAG TAP
# aarch64: this is an ARMv8-A core with an MMU
# -dap: debug access port to use for this target
target create $_TARGET_NAME aarch64 -dap $_CHIP_NAME.dap
#
# Configure the debug target
#
# Set the CTI for this core
$_TARGET_NAME configure -cti $_CTI_NAME
# Set the address of the debug controller for this core
$_TARGET_NAME configure -dbgbase [lindex $_DBG_BASE $_CORE]
# Set the access port for the target
$_TARGET_NAME configure -ap-num 0
# This is needed for GDB to correctly inspect the SMP system (section 20.7)
$_TARGET_NAME configure -rtos hwthread
}
# Run the SMP command to add each target to the SMP config
eval $_SMP_COMMAND
# Print to verify command
echo "Running SMP Command: $_SMP_COMMAND"
# Set the current target to the primary core before GDB starts
targets $_CHIP_NAME.cpu0
Here's the output I get when I start OpenOCD and connect with GDB:
Open On-Chip Debugger 0.12.0
Licensed under GNU GPL v2
For bug reports, read
http://openocd.org/doc/doxygen/bugs.html
Info : Hardware thread awareness created
Info : Hardware thread awareness created
Info : Hardware thread awareness created
Info : Hardware thread awareness created
Running SMP Command: target smp bcm2712.cpu0 bcm2712.cpu1 bcm2712.cpu2 bcm2712.cpu3
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : Using CMSIS-DAPv2 interface with VID:PID=0x2e8a:0x000c, serial=E6633861A32A5938
Info : CMSIS-DAP: SWD supported
Info : CMSIS-DAP: Atomic commands supported
Info : CMSIS-DAP: Test domain timer supported
Info : CMSIS-DAP: FW Version = 2.0.0
Info : CMSIS-DAP: Interface Initialised (SWD)
Info : SWCLK/TCK = 0 SWDIO/TMS = 0 TDI = 0 TDO = 0 nTRST = 0 nRESET = 0
Info : CMSIS-DAP: Interface ready
Info : clock speed 4000 kHz
Info : SWD DPIDR 0x2ba01477
Info : bcm2712.cpu0: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2712.cpu1: hardware has 6 breakpoints, 4 watchpoints
Info : bcm2712.cpu2: hardware has 6 breakpoints, 4 watchpoints
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from ./fsix_files/fsix.arm64-rpi5.elf...
Info : bcm2712.cpu3: hardware has 6 breakpoints, 4 watchpoints
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/ubuntu/Documents/rpi5_debug/fsix_files/fsix.arm64-rpi5.elf.
Use `info auto-load python-scripts [REGEXP]' to list them.
Info : starting gdb server for bcm2712.cpu0 on 3333
Info : Listening on port 3333 for gdb connections
Breakpoint 1 at 0x80000
Breakpoint 2 at 0x8087c
Remote debugging using :3333
Info : accepting 'gdb' connection on tcp/3333
Info : bcm2712.cpu0 cluster 0 core 0 multi core
Info : bcm2712.cpu1 cluster 1 core 0 multi core
bcm2712.cpu1 halted in AArch64 state due to debug-request, current mode: EL3H
cpsr: 0x200002cd pc: 0x1c14
MMU: disabled, D-Cache: disabled, I-Cache: enabled
Info : bcm2712.cpu2 cluster 2 core 0 multi core
bcm2712.cpu2 halted in AArch64 state due to debug-request, current mode: EL3H
cpsr: 0x200002cd pc: 0x1c14
MMU: disabled, D-Cache: disabled, I-Cache: enabled
Info : bcm2712.cpu3 cluster 3 core 0 multi core
bcm2712.cpu3 halted in AArch64 state due to debug-request, current mode: EL3H
cpsr: 0x200002cd pc: 0x1c14
MMU: disabled, D-Cache: disabled, I-Cache: enabled
bcm2712.cpu0 halted in AArch64 state due to debug-request, current mode: EL2H
cpsr: 0x000003c9 pc: 0x80000
MMU: disabled, D-Cache: disabled, I-Cache: disabled
Info : New GDB Connection: 1, Target bcm2712.cpu0, state: halted
Right now I only get one thread when running info threads in GDB:
(gdb) info threads
Id Target Id Frame
* 1 Thread 1 "bcm2712.cpu3" (Name: bcm2712.cpu3, state: debug-request) 0x0000000000080000 in _start ()
I'd like to get a thread per core when running info threads like this:
(gdb) info threads
Id Target Id Frame
* 1 Thread 1.1 (CPU#0 [running]) 0x0000000000080000 in _start ()
2 Thread 1.2 (CPU#1 [running]) 0x000000000000030c in ?? ()
3 Thread 1.3 (CPU#2 [running]) 0x000000000000030c in ?? ()
4 Thread 1.4 (CPU#3 [running]) 0x000000000000030c in ?? ()
Running monitor targets shows that OpenOCD is aware of the four cores.
(gdb) monitor targets
TargetName Type Endian TapName State
-- ------------------ ---------- ------ ------------------ ------------
0* bcm2712.cpu0 aarch64 little bcm2712.cpu halted
1 bcm2712.cpu1 aarch64 little bcm2712.cpu halted
2 bcm2712.cpu2 aarch64 little bcm2712.cpu halted
3 bcm2712.cpu3 aarch64 little bcm2712.cpu halted
I appreciate any help! I'm new to OpenOCD. I've read a good amount of the OpenOCD User Guide, but am kind of stumped.
I needed to specify a coreid when creating targets. I added this line to add it to each target.
$_TARGET_NAME configure -coreid $_CORE
One of the coreid descriptions in section 11.3 Target Configuration of the OpenOCD User Guide states:
This value coreid is currently also used in other contexts as a general CPU index, e.g. in SMP nodes or to select a specific CPU in a chip.
After I was able to view all the cores as threads, I had an issue where I couldn't step instructions in the secondary cores. I think this is related to the target state for each of the cores.
To fix this I'm running monitor targets <core number> to select a core. Then I run monitor step to put a core in the single-step state. Then I'm allowed to single step in GDB.