I am building latest poky and seeing random build failures like these :
| {standard input}: Assembler messages:
| {standard input}:31007: Warning: end of file not at end of a line; newline inserted
| {standard input}: Error: .size expression for _ZN5clang5arcmt29getFileRemappingsFromFileListERSt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_ESaIS9_EEN4llvm8ArrayRefINSD_9StringRefEEEPNS
_18DiagnosticConsumerE.cold does not evaluate to a constant
| g++: fatal error: Killed signal terminated program cc1plus
| compilation terminated.
and this :
| {standard input}: Assembler messages:
| {standard input}:4411: Warning: end of file not at end of a line; newline inserted
| {standard input}:4686: Error: no such instruction: `_zn4llvm19dyn_cast_if_presentipn5clang14typesourceinfoens_12pointerunionijs3_pns1_9fielddeclepns1_17indirectfi'
| g++: fatal error: Killed signal terminated program cc1plus
| compilation terminated.
and this :
| {standard input}: Assembler messages:
| {standard input}:123544: Warning: end of file not at end of a line; newline inserted
| {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive
| g++: fatal error: Killed signal terminated program cc1plus
| compilation terminated.
Without doing anything if I kept re-building, build passed after 3rd attempt. When I investidated, there is an indication at this SO answer that this happens because of memory pressure. So I re-ran the build and I kept seeing the following messages in dmesg :
[1465456.949609] systemd-journald[2059095]: Under memory pressure, flushing caches.
[1465464.860330] systemd-journald[2059095]: Under memory pressure, flushing caches.
[1465466.860938] systemd-journald[2059095]: Under memory pressure, flushing caches.
Here's my system details, while the yocto build is happening in parallel :
$ free
total used free shared buff/cache available
Mem: 32903080 13796868 10145592 748 9847580 19106212
Swap: 2097148 1966256 130892
and its a 36 core Ubuntu 24 LTS VM running on ESXI server. Could this be possible that all these random errors are because of memory pressure as the logs are indicating ?
EDIT I am also seeing lot of OOM kill happening :
[1466329.947738] Out of memory: Killed process 1507187 (cc1plus) total-vm:1716500kB, anon-rss:1627136kB, file-rss:288kB, shmem-rss:0kB, UID:1000 pgtables:3340kB oom_score_adj:0
Despite having massive memory/CPU configuration on host machine, bitbake is destroying itself. This seems a bug to me. bitbake shouldn't be this much greedy that it crosses limits and becomes a victim to OOM kill. Is my understanding correct here? What is the fix for this? Limiting number of build threads?
That's not bitbake being oomd but gcc. Try setting BB_NUMBER_THREADS to say 16 and PARALLEL_MAKE to -j16 to reduce the amount of parallelism.
You didn't say what recipes you're building but if it's something like webkit then yes it will eat all your RAM and ask for more if you have too many link jobs (which can take many gigs each) at once.
You can also investigate the pressure monitoring functionality in recent releases, but I've never used it so can't comment. It's BB_MAX_PRESSURE_MEMORY and https://www.youtube.com/watch?v=aWq9KQI6dAU should be useful.