Audio playback interrupts with brief gaps of silence in Swift

I have a WebSocket connection that sends 256KB chunks of binary audio data. I need to decode this binary format, write it into an AVAudioPCMBuffer, and schedule it for playback using AVAudioPlayerNode. However, when playback starts, I experience small gaps of silence, likely between chunks. I need the chunks to be played seamlessly, one after another, without interruptions.

How can I achieve seamless real-time audio playback in Swift without saving to a file? I need the audio to start playing as soon as the first chunks are received, without waiting for the entire stream to finish.

    func receiveMessage() {
        webSocketTask?.receive { [weak self] result in
            switch result {
            case .failure(let error):
                print("WebSocket error: \(error)")
            
            case .success(let message):
                switch message {
                case .data(let data):
                    self?.onChunkIncoming(data);
                    
                    if !(self?.hasStartedPlaying ?? true) {
                        self?.hasStartedPlaying = true;
                        self?.player.play();
                    }
                case .string(let text):
                    if text == "END" {
                        self?.closeConnection()
                    }
                @unknown default:
                    fatalError("Unknown format")
                }
            }
            
            self?.receiveMessage()
        }
    }

    private func onChunkIncoming(_ chunk: Data) {
        // Creating Audio PCM  Buffer
        guard let audioBuffer = self.decodeAudioData(chunk) else {
          NSLog("[AudioStreaming] failed auido buffer")
          return
        }
        
        print("Chunk is here")

        self.addToBuffer(buffer: audioBuffer)
    }

    private func addToBuffer(buffer: AVAudioPCMBuffer) {
        guard buffer.format.isEqual(outputFormat) else {
            NSLog("[AudioStreaming] no equal \(buffer.format) \(String(describing: outputFormat))")
            return;
        }
        
        self.buffersInQueue += 1;
        
        player.scheduleBuffer(buffer, at: nil, options: []) {
            DispatchQueue.main.async { [weak self] in
                guard let self else { return }

                self.buffersInQueue -= 1
                print("Chunk played")
                  
                if self.buffersInQueue == 0 {
                    print("Last chunk played")
                }
            }
        }
    }

I identified and resolved the audio stuttering issue in my decodeAudioData function. The problem was caused by incorrectly calculating the number of samples to be copied into AVAudioPCMBuffer. Initially, I used frameCount as the byte count, but I overlooked that the audio is stereo with two channels. To correctly process the data, the total number of samples must be calculated by multiplying frameCount by the number of channels.

data.withUnsafeBytes { (bufferPointer: UnsafeRawBufferPointer) in
    if let memory = bufferPointer.baseAddress?.assumingMemoryBound(to: Int16.self) {
       inputBuffer.int16ChannelData?.pointee.update(from: memory, count: Int(frameCount * inputFormat.channelCount))
    }
}

Thanks to Gordon Childs for pointing out that I was recreating the audio converter each time. To avoid losing chunks of audio when switching back to MP3, it's best to instantiate the converter once in init(), rather than recreating it each time.