I'm running to cuda-memcheck to debug my code and the output is as follows
========= Program hit cudaErrorCudartUnloading (error 29) due to "driver shutting down" on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2e40d3]
========= Host Frame:./nmt [0x53526]
========= Host Frame:./nmt [0xfbd9]
terminate called after throwing an instance of '========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 [0x3c259]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 [0x3c2a5]
========= Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main + 0xfc) [0x21ecc]
thrust::system::system_error'
========= Host Frame:./nmt [0x530a]
=========
what(): driver shutting down
========= Error: process didn't terminate successfully
========= Internal error (20)
========= No CUDA-MEMCHECK results found
Is it possible to tell from the line Host Frame:./nmt [0x53526] where is broken in the code? If so, how can I do that?
As @talonmies indicated (I suspect he will not mind if I post a CW answer), the cuda-memcheck tool provides additional stack back tracing capability, which can be enabled with the --show-backtrace switch added to the command line.
The back trace may consist of both host and device functions (i.e. host and device back traces.)
If the application has been also compiled with host debug symbol information (e.g. -g on linux) then cuda-memcheck can show function names for the host functions in the host backtrace.
Additional usage information is available in the documentation.