@font-face is causing slow load times

Lately i started noticing the website loads very slow at some point.

I've investigated this with firebug and when the page takes very long to load this is showing up:

It seems totally random, but the requests to webfont.woff just waits and waits forever. If i try to reach this resourse manually, it works fine and responses within 100ms. The page i'm talking about is http://wtf.maikelzweerink.nl, and the custom fonts come from the main domain maikelzweerink.nl.

The face-fonts are declared at default.css, also from the main domain:

@font-face {
    font-family: 'Proximanova Regular';
    src: url('//maikelzweerink.nl/general/font/Proximanova-Regular/webfont.eot');
    src: url('//maikelzweerink.nl/general/font/Proximanova-Regular/webfont.eot?#iefix') format('embedded-opentype'),
         url('//maikelzweerink.nl/general/font/Proximanova-Regular/webfont.woff') format('woff'),
         url('//maikelzweerink.nl/general/font/Proximanova-Regular/webfont.ttf') format('truetype'),
         url('//maikelzweerink.nl/general/font/Proximanova-Regular/webfont.svg#ProximaNovaRgRegular') format('svg');
}

Even with the correct HTTP headers

In chrome the same deal (click for larger):

I used base64 enconding to embed the font inside the CSS to reduce the amount of requests. The base64 has the downside of costing more initial bandwith but hey, it works! This fixed it.

Ofcourse i'm not really happy with a CSS file that is 180KB in size :/

Edit: It turned out to be the ISP that was the problem. When I mannualy loaded the font it retrieved it from cache (dammit chrome!). Somehow these fonts where not directly available on the ISP service and requests/reads from the harddisk would take a while.