Audio format where silence would not affect file size

I'm looking for an audio format where a silence of a couple of hours at the beginning does not affect the overall file size. Has anyone any idea which one to use and what settings I have to use? I tried m4a, ogg and mp3 so far with no luck. An audio sample with 4 hours of silence in the beginning leads to a 400 MB file in some formats.

Of course, dealing with it programmatically would be the more sensible and SO way, something like SoX and the silence/pad effects. After all, any bit of silence is identical to any other bit of silence, trying to compress it is a bit of waste of effort.

Having said that, I was a little curious about this myself so I had a go at comparing how well the different codecs fared at compressing pure digital silence.

I created two test files. The first was a 44.1kHz 16bit 30 minutes long stereo WAVE file containing uncorrelated brown noise at -10.66 dBFS RMS. The second file was the same, except padded with 210 minutes of silence, making the total duration 240 minutes (or 4 hours). Next I encoded the files to various lossy and lossless codecs and looked at the size difference between the padded and unpadded files to gauge how efficiently the silence was encoded.

 codec    noise  noise.silence      diff  ratio
   wav    317.5         2540.0    2222.5    8.0
he-aac     14.6          116.5     101.9    8.0
vorbis     36.4          237.1     200.7    6.5
   mp3     38.2          217.2     179.0    5.7
  opus     27.0           81.6      54.6    3.0
   tta    213.8          544.1     330.3    2.5
   aac     54.0          131.7      77.7    2.4
    wv    211.3          444.1     232.8    2.1
  alac    212.5          393.7     181.2    1.9
  flac    211.5          404.8     193.3    1.9
   als    209.7          384.2     174.5    1.8
   ofr    209.3          356.9     147.6    1.7

Codecs used:

Lossless
wav:    WAVE
tta:    True Audio v3.4.1
wv:     WavPack v4.80.0 (wavpack -x)
alac:   Apple Lossless
ofr:    OptimFROG v5.100 (ofr --preset 2)
als:    MPEG-4 Audio Lossless Coding v23 (mp4alsRM23 -a -b -o50)
flac:   Free Lossless Audio Codec v1.3.1 (flac -8)

Lossy vbr
mp3:    LAME MP3 v3.99.5 (lame -h -V2)
opus:   Opus v1.1.2 (opusenc --bitrate 128 --framesize 40)
aac:    Advanced Audio Codec v2.0 (afconvert -f 'm4af' -d aac -q 127 -s 3 -u vbrq 100)
vorbis: Vorbis aoTuV b5.5 (oggenc -q 5)

Lossy cbr
he-aac: High-Efficiency AAC v1 (afconvert -f 'm4af' -d aach -q 127 -s 0 -b 64000)