Can 64-bit library work in a 32-bit application? For example, my application GUI uses 32-bit Qt. And my business core is a 64-bit library. The OS is 64-bit. Can they work together and how?
In short: You can't link a 32-bit app to a 64-bit library.
You can run a 32-bit application, using 32-bit shared libraries on a 64-bit OS (at least all the popular 32-/64-bit processors such as AMD, Intel and Sparc). But that doesn't involve any libraries.
Longer answer: I was involved (on the outskirts) of some of the teams that designed the 64-bit Linux kernel for x86. There were briefly (compared to the whole project, the discussions lasted quite a few hours) some discussion as to how you could technically make this work. The short summary of this is that in 64-bit there are registers that aren't available in 32-bit. There is also the problem of memory addresses and the extra 32-bits in registers. All of these CAN be solved supposing the library itself "knows" that it's a 32-bit compatible library. But then we basically have a 64-bit library that is written to be a 32-bit library, and we've kind of lost the point.
The "more registers" may not apply to some processors, but the bigger address/bit-range of registers definitely apply to ALL 32- and 64-bit compatible processors. And I'm not aware of any single processor that allows a 32-bit code calling a 64-bit shared library or static library. It just doesn't work unless the code is specifically written to cope with that, which defeats the purpose of having a generic 64-bit library to support 32-bit apps.
Edit:
The above discusses linking one executable unit, e.g. an executable file, a shared library or a static library. That has to be all "one bitness", either 32 or 64 - no mixing.
When a process that talks to another process (e.g. a GUI app which displays status from a non-GUI process that), as long as the two processes use the same protocol [and typically, IPC doesn't allow passing of pointers anyway, so 32-/64-bit conversion isn't as big an issue], you can have one process that is 32-bit and another that is 64-bit.