iosswiftreplaykit

Trouble with screen recording in iOS


I've got some trouble with screen recording using ReplayKit

So, the strangeness begins in the startCapture's handler closure

Every time it starts to capture and sampleType is .video, the self.videoWriter?.startSession is called, and instantaneously videoWriter.status becomes failure (3)

This behaviour is not throwing any exceptions until I finish recording:

Optional("Cannot Encode")

Optional("Optional(Error Domain=AVFoundationErrorDomain Code=-11834 "Cannot Encode" UserInfo={AVErrorMediaSubTypeKey=(\n 778924083\n), NSLocalizedFailureReason=The encoder required for this media cannot be found., AVErrorMediaTypeKey=soun, NSLocalizedDescription=Cannot Encode })") Optional("<AVAssetWriter: 0x283ef0920, outputURL = file:///var/mobile/Containers/Data/Application/A85826BA-DCBE-428B-AB9E-84D77F234FF6/Documents/Replays/capture92949FA1-F65D-48A0-8140-207BC892C204.mp4, outputFileType = public.mpeg-4>")

Error Domain=PHPhotosErrorDomain Code=3302 "(null)"

Here is my code:

let settings = [
    AVFormatIDKey: Int(AudioFormat.kAudioFormatMPEGLayer3.rawValue),
    AVNumberOfChannelsKey: 2,
    AVSampleRateKey: 44100,
    AVEncoderBitRateKey: 128000
]
let audioInput = AVAssetWriterInput(
    mediaType: .audio,
    outputSettings: settings
)

audioInput.expectsMediaDataInRealTime = true
videoWriter.add(audioInput)

self.videoWriterInput = audioInput
self.videoWriter = videoWriter
let videoSettings: [String: Any] = [
   AVVideoCodecKey: AVVideoCodecType.h264,
   AVVideoWidthKey: passingSize.width,
   AVVideoHeightKey: passingSize.height
]

self.videoWriterInput = AVAssetWriterInput(
    mediaType: AVMediaType.video,
    outputSettings: videoSettings
)

self.appAudioWriterInput = AVAssetWriterInput(
    mediaType: .audio,
    outputSettings: settings
)
self.videoWriter.add(appAudioWriterInput!)


self.recorder.startCapture(
    handler: { (sampleBuffer, sampleType, passedError) in 
       switch sampleType {
       case .video:
           self.handleSampleBuffer(sampleBuffer: sampleBuffer)
       case .audioApp:
           self.add(sample: sampleBuffer, to: appAudioWriterInput)
       default:
           break
       }
   },
   completionHandler: { error in
       if let error = error {
          print(">>> Error!")
          errorHandler(error: error)
       }
    }
)

private func handleSampleBuffer(sampleBuffer: CMSampleBuffer) {
    if videoWriter?.status == AVAssetWriter.Status.unknown {
        serialQueue.async {
            if !self.isRecording {
                self.videoWriter?.startWriting()
                self.videoWriter?.startSession(
                    atSourceTime: .zero
                )
                self.isRecording = true
            }
        }
    } else if videoWriter?.status == AVAssetWriter.Status.writing &&
        videoWriterInput?.isReadyForMoreMediaData == true {
            serialQueue.async {
                self.videoWriterInput?.append(sampleBuffer)
            }
    }
}

I'm not familiar with iOS development and I think that I'm missing something obvious.

Things I've tried:

  1. Clean XCode caches
  2. Run app on both IPhone and Simulator
  3. Check PHPhotoLibrary.authorizationStatus() and AVCaptureDevice.authorizationStatus before recording
  4. Experiment with audio and video recording settings

Here is my modifications to Info.plist

<key>NSPhotoLibraryAddUsageDescription</key>
<string>The app requires access to Photos to save media to it.</string>
<key>NSPhotoLibraryUsageDescription</key>
<string>The app requires access to Photos to interact with it.</string>
<key>PHPhotoLibraryPreventAutomaticLimitedAccessAlert</key>
<true/>

Solution

  • The solution was pretty simple: I just passed wrong variable to self.videoWriterInput

    Proper initialization should be something like this:

    let screenSize: CGRect = UIScreen.main.bounds
            
    let videoSettings: [String: Any] = [
        AVVideoCodecKey: AVVideoCodecType.h264,
        AVVideoWidthKey: screenSize.width,
        AVVideoHeightKey: screenSize.height,
    ]
    
    let newVideoWriterInput = AVAssetWriterInput(
        mediaType: AVMediaType.video,
        outputSettings: videoSettings
    )
    
            
    newVideoWriterInput.expectsMediaDataInRealTime = true
    self.videoWriter?.add(newVideoWriterInput)
    self.videoWriterInput = newVideoWriterInput // THE ERROR WAS HERE
            
    let settings: [String : Any] = [
        AVNumberOfChannelsKey : 2,
        AVFormatIDKey : kAudioFormatMPEG4AAC,
        AVSampleRateKey: 44100,
        AVEncoderAudioQualityKey: AVAudioQuality.high.rawValue
    ]
    
    let audioInput = AVAssetWriterInput(
        mediaType: .audio,
        outputSettings: settings
    )
    
    audioInput.expectsMediaDataInRealTime = true
    self.videoWriter?.add(audioInput)
    self.appAudioWriterInput = audioInput