AudioConverter number of packets is wrong

I've set up a class to convert audio from one format to another given an input and output AudioStreamBasicDescription. When I convert Linear PCM from the mic to iLBC, it works and gives me 6 packets when I give it 1024 packets from the AudioUnitRender function. I then send those 226 bytes via UDP to the same app running on a different device. The problem is that when I use the same class to convert back into Linear PCM for giving to an audio unit input, the AudioConverterFillComplexBuffer function doesn't give 1024 packets, it gives 960... This means that the audio unit input is expecting 4096 bytes (2048 x 2 for stereo) but I can only give it 3190 or so, and so it sounds really crackly and distorted...

If I give AudioConverter 1024 packets of LinearPCM, convert to iLBC, convert back to LinearPCM, surely I should get 1024 packets again?

Audio converter function:

-(void) doConvert {

    // Start converting
    if (converting) return;
    converting = YES;

    while (true) {

        // Get next buffer
        id bfr = [buffers getNextBuffer];
        if (!bfr) {
            converting = NO;
            return;
        }

        // Get info
        NSArray* bfrs = ([bfr isKindOfClass:[NSArray class]] ? bfr : @[bfr]);
        int bfrSize = 0;
        for (NSData* dat in bfrs) bfrSize += dat.length;

        int inputPackets = bfrSize / self.inputFormat.mBytesPerPacket;
        int outputPackets = (inputPackets * self.inputFormat.mFramesPerPacket) / self.outputFormat.mFramesPerPacket;

        // Create output buffer
        AudioBufferList* bufferList = (AudioBufferList*) malloc(sizeof(AudioBufferList) * self.outputFormat.mChannelsPerFrame);
        bufferList -> mNumberBuffers = self.outputFormat.mChannelsPerFrame;
        for (int i = 0 ; i < self.outputFormat.mChannelsPerFrame ; i++) {
            bufferList -> mBuffers[i].mNumberChannels = 1;
            bufferList -> mBuffers[i].mDataByteSize = 4*1024;
            bufferList -> mBuffers[i].mData = malloc(bufferList -> mBuffers[i].mDataByteSize);
        }

        // Create input buffer
        AudioBufferList* inputBufferList = (AudioBufferList*) malloc(sizeof(AudioBufferList) * bfrs.count);
        inputBufferList -> mNumberBuffers = bfrs.count;
        for (int i = 0 ; i < bfrs.count ; i++) {
            inputBufferList -> mBuffers[i].mNumberChannels = 1;
            inputBufferList -> mBuffers[i].mDataByteSize = [[bfrs objectAtIndex:i] length];
            inputBufferList -> mBuffers[i].mData = (void*) [[bfrs objectAtIndex:i] bytes];
        }

        // Create sound data payload
        struct SoundDataPayload payload;
        payload.data = inputBufferList;
        payload.numPackets = inputPackets;
        payload.packetDescriptions = NULL;
        payload.used = NO;

        // Convert data
        UInt32 numPackets = outputPackets;
        OSStatus err = AudioConverterFillComplexBuffer(converter, acvConverterComplexInput, &payload, &numPackets, bufferList, NULL);
        if (err)
            continue;

        // Check how to output
        if (bufferList -> mNumberBuffers > 1) {

            // Output as array
            NSMutableArray* array = [NSMutableArray arrayWithCapacity:bufferList -> mNumberBuffers];
            for (int i = 0 ; i < bufferList -> mNumberBuffers ; i++)
                [array addObject:[NSData dataWithBytes:bufferList -> mBuffers[i].mData length:bufferList -> mBuffers[i].mDataByteSize]];

            // Save
            [convertedBuffers addBuffer:array];

        } else {

            // Output as data
            NSData* newData = [NSData dataWithBytes:bufferList -> mBuffers[0].mData length:bufferList -> mBuffers[0].mDataByteSize];

            // Save
            [convertedBuffers addBuffer:newData];

        }

        // Free memory
        for (int i = 0 ; i < bufferList -> mNumberBuffers ; i++)
            free(bufferList -> mBuffers[i].mData);

        free(inputBufferList);
        free(bufferList);

        // Tell delegate
        if (self.convertHandler)
            //dispatch_async(dispatch_get_main_queue(), self.convertHandler);
            self.convertHandler();

    }

}

Formats when converting to iLBC:

// Get input format from mic
UInt32 size = sizeof(AudioStreamBasicDescription);
AudioStreamBasicDescription inputDesc;
AudioUnitGetProperty(self.ioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, 1, &inputDesc, &size);

// Set output stream description
size = sizeof(AudioStreamBasicDescription);
AudioStreamBasicDescription outputDescription;
memset(&outputDescription, 0, size);
outputDescription.mFormatID         = kAudioFormatiLBC;
OSStatus err = AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &size, &outputDescription);

Formats when converting from iLBC:

// Set input stream description
size = sizeof(AudioStreamBasicDescription);
AudioStreamBasicDescription inputDescription;
memset(&inputDescription, 0, size);
inputDescription.mFormatID        = kAudioFormatiLBC;
AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &size, &inputDescription);

// Set output stream description
UInt32 size = sizeof(AudioStreamBasicDescription);
AudioStreamBasicDescription outputDesc;
AudioUnitGetProperty(unit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, 0, &outputDesc, &size);

You have to use an intermediate buffer to save up enough bytes from enough incoming packets to exactly match the number requested by the audio unit input. Depending on any one UDP packet in compressed format to be exactly the right size won't work.

The AudioConverter may buffer samples and change the packet sizes depending on the compression format.