I want to compress a .app folder to a .dmg
I used bzip2 (-format UDBZ) to do that but the filesize is still too large and I know it can be done better. Is there a better compression algorythm available? Can I use 7zip (LZMA)?
I'm afraid not. The official documentation of hdiutil is here, and only zlib and bzip2 are available.
How big is it, by the way? What is taking up most of the space? Try to shrink down the size of .app itself.
By the way, from my personal interaction with non-techie users, I can testify that packing .app into .dmg is a very confusing concept for them... They don't understand what a disk image is, and what it means it is mounted as a disk. It would be easier for them to understand if .app is just zipped or bzip2'ed, which would be automatically decompressed when downloaded from the web.