time_base working for video but undesired effects for audio

I'm trying to change video play speed with the ffmpeg c api. So far everything is working except changes of the time base when applied to the audio stream don't speed up the audio. Instead, it cuts out parts of the audio. Here's roughly what I have:

for (uint64_t i = 0; i < in_ctx->nb_streams; i++) { // Copy over streams and codec parameters
    AVStream* in_stream = in_ctx->streams[i];
    AVStream* out_stream = avformat_new_stream(out_ctx, NULL);

    in_stream->time_base.den += 20000;

    if (!out_stream) {
        return fatal_error();
    }

    avcodec_parameters_copy(out_stream->codecpar, in_stream->codecpar);
}
if (avio_open(&out_ctx->pb, argv[2], AVIO_FLAG_WRITE) < 0 ||
    avformat_write_header(out_ctx, NULL) < 0) {
    return fatal_error();
}

while (av_read_frame(in_ctx, packet) >= 0) {
    AVStream* in_stream = in_ctx->streams[packet->stream_index];
    AVStream* out_stream = out_ctx->streams[packet->stream_index];

    av_packet_rescale_ts(packet, in_stream->time_base, out_stream->time_base);
    if (av_interleaved_write_frame(out_ctx, packet) < 0) {
        return fatal_error();
    }
}

You have to use the atempo filter or similar to reform the audio stream.

Changing the effective timestamps works for video since all pixels in a frame are meant to be painted at the given timestamp. An audio packet is a sequence of samples. Changing packet timestamp alters when that packet is supposed to get presented but not the duration taken to play each sample in the sequence, which is 1/samplerate. So even when one compresses the timestamps of audio packets in time, the decoder waits till the current packet is done playing. If the playback delay for the next packet has crossed whatever threshold or sync requirement is used by the player, expect it to get dropped.