Add rounded corners + transparency background from import video (.mp4) to exported video (.mov) in Swift

Problem

I have a .mp4 file in which I want to add rounded corners and make the background (behind the rounded corners) transparant. I got two issues with my current code:

1. For some reason, the CALayer I apply with the rounded corners makes the whole video black, despite setting the backgroundColor to clear. Adding a large borderWidth helps, but I am curious if there is a better alternative, it's a bit of a hack. Link to code: https://github.com/Jasperav/VideoWithRoundedTransparantCorners/blob/fdc534a7a4541c58e04d48c1ed5c62fe9ad6e497/VideoWithRoundedTransparantCornersTests/VideoEditor.swift#L277
2. The background of the rounded corners is black, not transparant. How to make it transparant?

This is an example of the output video when having a borderWidth of 50 on the CALayer property maskLayer (having no borderWidth makes the entire video black), there is a big black box which shouldn't be there:

Setting the borderWidth to greatestFiniteMagnitude fixes the big black box, but it's ugly and I think I am misusing the CALayer by doing this.

I also do not know how to make the edges transparant instead of black.

Reproduction

To reproduce the problem fairly easy, I created this repo with a single test. Running that test will create the .mov file with the problems described above. Below is the code as well.

Repo (just run the test, check the logging for output and play the video): https://github.com/Jasperav/VideoWithRoundedTransparantCorners/tree/main/VideoWithRoundedTransparantCornersTests

Code (VideoEditor.swift):

import AVFoundation
import Foundation
import Photos
import AppKit
import QuartzCore
import OSLog

let logger = Logger()

class VideoEditor {
    func export(
        url: URL,
        outputDir: URL,
        size: CGSize
    ) async -> String? {
        do {
            let (asset, video) = try await resizeVideo(videoAsset: AVURLAsset(url: url), targetSize: size, isKeepAspectRatio: false, isCutBlackEdge: false)
            
            try await exportVideo(outputPath: outputDir, asset: asset, videoComposition: video)

            return nil
        } catch let error as YGCVideoError {
            switch error {
            case .videoFileNotFind:
                return NSLocalizedString("video_error_video_file_not_found", comment: "")
            case .videoTrackNotFind:
                return NSLocalizedString("video_error_no_video_track", comment: "")
            case .audioTrackNotFind:
                return NSLocalizedString("video_error_no_audio_track", comment: "")
            case .compositionTrackInitFailed:
                return NSLocalizedString("video_error_could_not_create_composition_track", comment: "")
            case .targetSizeNotCorrect:
                return NSLocalizedString("video_error_wrong_size", comment: "")
            case .timeSetNotCorrect:
                return NSLocalizedString("video_error_wrong_time", comment: "")
            case .noDir:
                return NSLocalizedString("video_error_no_dir", comment: "")
            case .noExportSession:
                return NSLocalizedString("video_error_no_export_session", comment: "")
                case .exporterError(let exporterError):
                return String.localizedStringWithFormat(NSLocalizedString("video_error_exporter_error", comment: ""), exporterError)
            }
        } catch {
            assertionFailure()

            return error.localizedDescription
        }
    }

    private enum YGCVideoError: Error {
        case videoFileNotFind
        case videoTrackNotFind
        case audioTrackNotFind
        case compositionTrackInitFailed
        case targetSizeNotCorrect
        case timeSetNotCorrect
        case noDir
        case noExportSession
        case exporterError(String)
    }

    private enum YGCTimeRange {
        case naturalRange
        case secondsRange(Double, Double)
        case cmtimeRange(CMTime, CMTime)

        func validateTime(videoTime: CMTime) -> Bool {
            switch self {
            case .naturalRange:
                return true
            case let .secondsRange(begin, end):
                let seconds = CMTimeGetSeconds(videoTime)
                if end > begin, begin >= 0, end < seconds {
                    return true
                } else {
                    return false
                }
            case let .cmtimeRange(_, end):
                if CMTimeCompare(end, videoTime) == 1 {
                    return false
                } else {
                    return true
                }
            }
        }
    }

    private enum Way {
        case right, left, up, down
    }

    private func orientationFromTransform(transform: CGAffineTransform) -> (orientation: Way, isPortrait: Bool) {
        var assetOrientation = Way.up
        var isPortrait = false

        if transform.a == 0, transform.b == 1.0, transform.c == -1.0, transform.d == 0 {
            assetOrientation = .right
            isPortrait = true
        } else if transform.a == 0, transform.b == -1.0, transform.c == 1.0, transform.d == 0 {
            assetOrientation = .left
            isPortrait = true
        } else if transform.a == 1.0, transform.b == 0, transform.c == 0, transform.d == 1.0 {
            assetOrientation = .up
        } else if transform.a == -1.0, transform.b == 0, transform.c == 0, transform.d == -1.0 {
            assetOrientation = .down
        }

        return (assetOrientation, isPortrait)
    }

    private func videoCompositionInstructionForTrack(track: AVCompositionTrack, videoTrack: AVAssetTrack, targetSize: CGSize) async throws -> AVMutableVideoCompositionLayerInstruction {
        let instruction = AVMutableVideoCompositionLayerInstruction(assetTrack: track)
        let transform = try await videoTrack.load(.preferredTransform)
        let naturalSize = try await videoTrack.load(.naturalSize)
        let assetInfo = orientationFromTransform(transform: transform)
        var scaleToFitRatio = targetSize.width / naturalSize.width

        if assetInfo.isPortrait {
            scaleToFitRatio = targetSize.width / naturalSize.height

            let scaleFactor = CGAffineTransform(scaleX: scaleToFitRatio, y: scaleToFitRatio)

            instruction.setTransform(transform.concatenating(scaleFactor), at: CMTime.zero)
        } else {
            let scaleFactor = CGAffineTransform(scaleX: scaleToFitRatio, y: scaleToFitRatio)

            var concat = transform.concatenating(scaleFactor).concatenating(CGAffineTransform(translationX: 0, y: targetSize.width / 2))

            if assetInfo.orientation == .down {
                let fixUpsideDown = CGAffineTransform(rotationAngle: CGFloat.pi)
                let yFix = naturalSize.height + targetSize.height
                let centerFix = CGAffineTransform(translationX: naturalSize.width, y: yFix)

                concat = fixUpsideDown.concatenating(centerFix).concatenating(scaleFactor)
            }

            instruction.setTransform(concat, at: CMTime.zero)
        }

        return instruction
    }

    private func exportVideo(outputPath: URL, asset: AVAsset, videoComposition: AVMutableVideoComposition?) async throws {
        let fileExists = FileManager.default.fileExists(atPath: outputPath.path())

        logger.debug("Output dir: \(outputPath), exists: \(fileExists)")

        if fileExists {
            do {
                try FileManager.default.removeItem(atPath: outputPath.path())
            } catch {
                logger.error("remove file failed")
            }
        }

        let dir = outputPath.deletingLastPathComponent().path()

        logger.debug("Will try to create dir: \(dir)")

        try? FileManager.default.createDirectory(atPath: dir, withIntermediateDirectories: true)

        var isDirectory = ObjCBool(false)

        guard FileManager.default.fileExists(atPath: dir, isDirectory: &isDirectory), isDirectory.boolValue else {
            logger.error("Could not create dir, or dir is a file")

            throw YGCVideoError.noDir
        }

        guard let exporter = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHighestQuality) else {
            logger.error("generate export failed")

            throw YGCVideoError.noExportSession
        }

        exporter.outputURL = outputPath
        exporter.outputFileType = .mov
        exporter.shouldOptimizeForNetworkUse = false

        if let composition = videoComposition {
            exporter.videoComposition = composition
        }

        await exporter.export()

        logger.debug("Status: \(String(describing: exporter.status)), error: \(exporter.error)")

        if exporter.status != .completed {
            throw YGCVideoError.exporterError(exporter.error?.localizedDescription ?? "NO SPECIFIC ERROR")
        }
    }

    private func resizeVideo(videoAsset: AVURLAsset,
                             targetSize: CGSize,
                             isKeepAspectRatio: Bool,
                             isCutBlackEdge: Bool) async throws -> (AVMutableComposition, AVMutableVideoComposition)
    {
        guard let videoTrack = try await videoAsset.loadTracks(withMediaType: .video).first else {
            throw YGCVideoError.videoTrackNotFind
        }


        guard let audioTrack = try await videoAsset.loadTracks(withMediaType: .audio).first else {
            throw YGCVideoError.audioTrackNotFind
        }

        let resizeComposition = AVMutableComposition(urlAssetInitializationOptions: nil)

        guard let compositionVideoTrack = resizeComposition.addMutableTrack(withMediaType: AVMediaType.video, preferredTrackID: videoTrack.trackID) else {
            throw YGCVideoError.compositionTrackInitFailed
        }
        guard let compostiionAudioTrack = resizeComposition.addMutableTrack(withMediaType: AVMediaType.audio, preferredTrackID: audioTrack.trackID) else {
            throw YGCVideoError.compositionTrackInitFailed
        }

        let duration = try await videoAsset.load(.duration)

        try compositionVideoTrack.insertTimeRange(CMTimeRangeMake(start: CMTime.zero, duration: duration), of: videoTrack, at: CMTime.zero)
        try compostiionAudioTrack.insertTimeRange(CMTimeRangeMake(start: CMTime.zero, duration: duration), of: audioTrack, at: CMTime.zero)

        let originTransform = try await videoTrack.load(.preferredTransform)
        let info = orientationFromTransform(transform: originTransform)
        let naturalSize = try await videoTrack.load(.naturalSize)
        let videoNaturaSize: CGSize = if info.isPortrait, info.orientation != .up {
            CGSize(width: naturalSize.height, height: naturalSize.width)
        } else {
            naturalSize
        }

        if videoNaturaSize.width < targetSize.width, videoNaturaSize.height < targetSize.height {
            throw YGCVideoError.targetSizeNotCorrect
        }

        let fitRect: CGRect = if isKeepAspectRatio {
            AVMakeRect(aspectRatio: videoNaturaSize, insideRect: CGRect(origin: CGPoint.zero, size: targetSize))
        } else {
            CGRect(origin: CGPoint.zero, size: targetSize)
        }

        let mainInstruction = AVMutableVideoCompositionInstruction()

        mainInstruction.timeRange = CMTimeRange(start: CMTime.zero, end: duration)

        let layerInstruction = AVMutableVideoCompositionLayerInstruction(assetTrack: videoTrack)

        let finalTransform: CGAffineTransform = if info.isPortrait {
            if isCutBlackEdge {
                originTransform.concatenating(CGAffineTransform(scaleX: fitRect.width / videoNaturaSize.width, y: fitRect.height / videoNaturaSize.height))
            } else {
                originTransform.concatenating(CGAffineTransform(scaleX: fitRect.width / videoNaturaSize.width, y: fitRect.height / videoNaturaSize.height)).concatenating(CGAffineTransform(translationX: fitRect.minX, y: fitRect.minY))
            }

        } else {
            if isCutBlackEdge {
                originTransform.concatenating(CGAffineTransform(scaleX: fitRect.width / videoNaturaSize.width, y: fitRect.height / videoNaturaSize.height))
            } else {
                originTransform.concatenating(CGAffineTransform(scaleX: fitRect.width / videoNaturaSize.width, y: fitRect.height / videoNaturaSize.height)).concatenating(CGAffineTransform(translationX: fitRect.minX, y: fitRect.minY))
            }
        }
        layerInstruction.setTransform(finalTransform, at: CMTime.zero)
        mainInstruction.layerInstructions = [layerInstruction]

        let videoComposition = AVMutableVideoComposition()
        
        videoComposition.frameDuration = CMTimeMake(value: 1, timescale: 30)
        
        let videoLayer = CALayer()
        
        videoLayer.frame = CGRect(x: 0, y: 0, width: targetSize.width, height: targetSize.height)
        videoLayer.backgroundColor = .clear
        
        let maskLayer = CALayer()
        
        maskLayer.frame = videoLayer.bounds
        maskLayer.cornerRadius = 100
        maskLayer.masksToBounds = true
        maskLayer.borderWidth = 50
        maskLayer.backgroundColor = .clear

        videoLayer.mask = maskLayer
        
        videoComposition.animationTool = AVVideoCompositionCoreAnimationTool(postProcessingAsVideoLayer: videoLayer, in: videoLayer)
        
        if isCutBlackEdge, isKeepAspectRatio {
            videoComposition.renderSize = fitRect.size
        } else {
            videoComposition.renderSize = targetSize
        }

        videoComposition.instructions = [mainInstruction]

        return (resizeComposition, videoComposition)
    }
}

Question

• How do I apply rounded corners to a video and make the 'background'/edges of the rounded corners transparant?
• Why do I currently have a big black box in the middle, which is only removed when setting a huge borderWidth?

What I want

This is how I want the final output to be, when the black edges (with the huge red arrows) are transparant.

Update

When using a CAShapeLayer, I can only see the shape, but I am having trouble to 'cut' out the edges to be transparant. I tried adding sublayers, but I couldn't get it working.

This is the output (you can see the black path):

Based on this CAShapeLayer:

let shapeLayer = CAShapeLayer()

shapeLayer.frame = CGRect(x: 0, y: 0, width: targetSize.width, height: targetSize.height)
shapeLayer.path = NSBezierPath(roundedRect: CGRect(x: 0, y: 0, width: targetSize.width, height: targetSize.height), xRadius: cornerRadius, yRadius: cornerRadius).cgPath
shapeLayer.fillColor = .clear
shapeLayer.strokeColor = .black

let parentLayer = CALayer()
let videoLayer = CALayer()

parentLayer.frame = CGRect(x: 0, y: 0, width: targetSize.width, height: targetSize.height)
videoLayer.frame = parentLayer.bounds
parentLayer.addSublayer(videoLayer)
parentLayer.addSublayer(shapeLayer)

let animationTool = AVVideoCompositionCoreAnimationTool(postProcessingAsVideoLayer: videoLayer, in: parentLayer)

extract.videoComposition.animationTool = animationTool

How can I cut out the edges to be transparant?

Couple issues...

First, you are using the layer mask incorrectly. Get rid of the border, and set the mask layer background color to any opaque color - such as .white.

Second, you need to use AVAssetExportPresetHEVCHighestQualityWithAlpha as the AVAssetExportSession preset (instead of AVAssetExportPresetHighestQuality).

Here are the changes I made to your VideoEditor class...

---

in func exportVideo(...):

    //guard let exporter = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHighestQuality) else {
    //    logger.error("generate export failed")
    //
    //    throw YGCVideoError.noExportSession
    //}

    // need to use AVAssetExportPresetHEVCHighestQualityWithAlpha preset
    guard let exporter = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetHEVCHighestQualityWithAlpha) else {
        logger.error("generate export failed")
        
        throw YGCVideoError.noExportSession
    }
    

---

in func resizeVideo(...):

    // get rid of border
    //maskLayer.borderWidth = 50
    
    // do not use .clear ... use any opaque color
    maskLayer.backgroundColor = .white // .clear

---

When I load and playback the output video on a view with a yellow background (in a quick iOS app) I get this: