I am trying to emulate the functionality of alpha compositing in Matlab or to be more specific the compose types CopyOpacity and Over from ImageMagick. First goal is to mask out a region with heavy aliasing edges like the black regions in the images in the set (image set with aliasing edges). This should be done with a compatible gray scale mask were black regions should be eliminated and white should be preserved see gray scale mask. Important is the continuous transition between black and white resulting in a continuous transition between transparent and opaque, see screenshot with transparent regions).
There has been two approaches to compose a foreground FG with and background BF with a gray scale mask CM_mask_blur_alpha but the results were not as expected (see an image set with post-processing with ImageMagick as reference for a pure Matlab procedure).
1) In the first code the gray scale mask is unexpectedly treated as a binary mask resulting in unacceptable aliasing effects at the edges of the formerly black regions (image set for first approach):
FG = uint8(CM_mask_blur_alpha .* FG + (1 - CM_mask_blur_alpha) .* BG);
2) The second approach lead to a visible continuous transition between a foreground and the background but there are aliasing effects from the FG remaining (image set for second approach):
FG = uint8(bsxfun(@times, CM_mask_blur_alpha, FG) + bsxfun(@times, (1 - CM_mask_blur_alpha), BG));
It seems that a one-step approach is not working so I am looking for a two-step approach like in ImageMagick with a masking out of the black regions resulting in an intermediate image with transparency and as second step a compose over of this intermediate image over an background. It is partly a problem setting as in MATLAB: Applying transparent mask over an RGB image and blending with another but there is no gray scale mask and I could not adapt solution parts like the generation of an alpha channel with values derived from the gray mask.
It turns out that the problem was not the compositing part but the mask. If the mask is enlarged with a morphological operation (imdilate) with a huge value (se = strel('disk',9)) and then blurred (imgaussfilt) with a relatively large value (like sigma=3) the aliased edges were masked out with a smooth transition.
whiteImage = 255 * ones(cm_out_h, cm_out_h, 'uint8');
conformal = maketform('custom', 2, 2, [], @conformalInverse_0001, []);
CM_mask = imtransform(whiteImage, conformal, ..., 'FillValues', 0);
% MORPHOLOGY: ENLARGE BLACK WITH imdilate
CM_mask = imcomplement(CM_mask);
se = strel('disk',9);
CM_mask = imdilate(CM_mask,se);
CM_mask = imcomplement(CM_mask);
% BLUR
CM_mask_blur = imgaussfilt(CM_mask, 3);
% ALPHA
CM_mask_blur_alpha = double(CM_mask_blur)/255;
CM_mask_blur_alpha = im2double(CM_mask_blur_alpha);