Determining the library which causes "Illegal instruction" on a Pi Zero W (armv6), and fixing the build

I understand that a lot of compilation issues on the Pi Zeros are due to the fact that they use armv6, whereas the newer Raspberry Pi's like the 3 A+ and B+ use armv7. However, I do not understand how to find the offending library in an application that is causing the issue, and if there is perhaps a simple fix for the problem.

Background:

I am trying to port an application from a Linux Desktop environment to the Pi Zero (running armv6). I successfully ported it to the Pi 3 B and B+. That is, I compiled the code, and checked that it is producing the correct output.

However, the Pi Zero implementation compiles, but just spits out a single message when run:

Illegal instruction

This is most likely due to some command that is not compatible with armv6, but I cannot figure out which command that is. I would like to start by determining which library is the problem child. Please tell me how I would diagnose that.

Extra Info:

I have checked that the compiler is not the issue. How? I made a simple hello world program, and compiled it for the Pi Zero:

#include<iostream>

int main(int argc, char *argv[]){
   std::cout << "Hello World!" << std::endl;
   return 0;
}

So the compiler itself doesn't seem to be the issue.

More details:

If I run readelf -A myapp, my understanding is that the output is reporting that the app is indeed compiled for armv6:

Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "6"
  Tag_CPU_arch: v6
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_FP_arch: VFPv2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_rounding: Needed
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6

Here is the readelf -A for one of the shared libraries:

Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "6"
  Tag_CPU_arch: v6
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_FP_arch: VFPv4
  Tag_Advanced_SIMD_arch: NEONv1 with Fused-MAC
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_HardFP_use: Deprecated
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6

Solution

To identify a fault such as Illegal instruction you can run the program under a debugger capable of interacting with the operating system's fault handler.

On a Linux system such as the pi, that would be gdb. You may need to install this, on a debian-derived distribution that would be sudo apt-get install gdb

Then run your program

gdb myprog

or if your program needs command line arguments

gdb myprog --args some_argument another_argument

Once gdb starts up type run, and the program will execute near normally until it reaches the illegal instruction, at which point you will be dumped back at the gdb prompt with a hopefully informative error message.

There you can explore with commands such as backtrace or if the programmer has associated source, list. If the fault is at an address gdb can see as being mapped as from a file it should show you that - you can also get at the mapping information via the gdb command info files or by looking in /proc/[PID]/maps

If for some reason you can't run the program live under gdb, you can research how to enable core dumps for your system, and then load the program and the core dump into gdb for post-mortem analysis.

Depending on system configuration, if running the program on its own without a debugger, you may also see information about the fault in the output of dmesg or another system log.