Copying text outside of Vim with set mouse=a enabled

After enabling set mouse=a, text copied inside of Vim will not paste outside of Vim. Is there a way to fix this?

Here, selecting text with the mouse turns on visual mode and disables the Copy option in the popup menu:

Press Shift while selecting with the mouse. This will make mouse selection behave as if mouse=a was not enabled.

Note: this trick also applies to "middle button paste": if you want to paste in Vim text that was selected outside, press Shift while clicking the middle button. Just make sure that insert mode is activated when you do that (you may also want to :set paste to avoid unexpected effects).

OS X (Mac): hold Alt/Option while selecting (source)