Converting mp4 files without frames into mp3 using python / moviepy

So what I'm trying to do is convert the mp4 files I get from downloading YouTube videos with pytube to mp3 files using moviepy. However, those mp4 files do not contain any frames which raises KeyError: 'video_fps' in the ffmpeg_reader.

Is it even possible to do that with moviepy or do I need to use a different tool? I guess I could also just download the mp4 files with video but that would waste resources especially for large playlists.

Here is the code I used:

from moviepy.video.io.VideoFileClip import VideoFileClip
import pytube
    
def downloadPlaylist(url):
        playlist = pytube.Playlist(url)
    
        for video in playlist.videos:
            filename = video.streams.get_audio_only().download()
            clip = VideoFileClip(filename)
            clip.audio.write_audiofile(filename[:-4] + ".mp3")
            clip.close()

So just changing VideoFileClip(filename) to AudioFileClip(filename) was the solution:

    clip = AudioFileClip(filename)
    clip.write_audiofile(filename[:-4] + ".mp3")
    clip.close()